Data Sharing versus Data Security
Two of my favorite topics that seem to contradict each other are the sharing of data and the securing of it (especially crypto). I'll be pitching both at the upcoming VM and Linux Workshop next month at NC A&T in Greensboro. What's really amazing at this time is how hot both topics are and yet how poorly understood they are.
The VM and Linux Workshop is an highly technical and surprisingly inexpensive conference. This year there will be the added value of Velocity Software's performance seminar two days prior to the workshop (at the same location).
Data Sharing
The glut of mobile computing and popularity "cloud computing" illuminates the need for data sharing ... even amplifies it. For just one example, think about your list-o-contacts on your smart phone. Handy, eh? But not so handy if you have to go into contortions to synchronize the info with other parts of your digital life. Gotta have data synch, and that's data sharing.
Full disclosure: the presentation will specifically cover filesystem sharing, though we'll touch on data synch and other such content manglement. Networked filesystems were more common until recently. In a context where applications use files, network sharing of filesystems makes a lot of sense. But even where apps use methods other than files for their working data, they need files for basic operation. And here's the kicker: most of the operating system needs files. So ... sharing of filesystems makes a lot of sense in more contexts than you might think.
Sharing filesystems as block devices or memory is cooler: It is efficient, effective, even elegant. Sharing virtual disk or virtual ROM is obvious to some but still not practiced as widely as it should be. And sharing disk images works in SAN land as well as in the virtual world. "It's everywhere you want to be."
Data Security
We're talking crypto, but not just crypto. (Controlling access is crucial.)
I mentioned smart phone and contacts. Data sec balances data sharing so that your contacts remain yours.
The presentation will cover a mere handful of the data access issues. (You could have a week long seminar discussing all related topics!) For the focus topic of SSL, it's vital to protect the server's secret key.I'll get into how that works, why its important, and related tools (PGP/GPG and SSH and others). Ciphers and hashes and keys ... oh my!
Truth be told, PKI is a mess, but it gives us a framework, a reference. The heavy lifting is done by SSL (and TLS) which is really slick, well defined, and robust. But the underlying cryptography is a moving target. As soon as you learn one cipher or hash, you find it has been defeated. So keeping things plug-compatible is essential.
Both of these topics border on obsession for me. They're so geeky cool. More people should use them. Even where they're not applicable, they should be considered ... part of your "doctor's bag" of remedies and treatments. I'm excited to be talking at the Workshop again.
-- R; <><
Tuesday, May 27, 2014
Wednesday, May 7, 2014
Sir Santa Seen in Super Set Script
Wednesday, March 5, 2014
Dave Tuttle's Memoirs
Dave Tuttle's Memoirs
I suppose some people think they're in it just for the money or just for the tech or just for the business. Maybe. But human nature consistently shows us that we need something more. Ultimately, it's that "image of God" creativity, even for those who think we're not terribly creative (and those who know we're not terribly god-like or even godly).
Why do you do what you do?
In software, especially in FOSS, why do you code the code you code?
Intrigue
I attended a Git workshop this past Monday evening. (Yeah, yeah ... it's about time. Hey! It's not like I don't use code manglement, CVS, SVN, etc.) The class was taught by Aidan Feldman of github. Great stuff! Excellent quick workshop; got us all actually using Git on github. Anyway, he mentioned that Linus created Git. I asked if the legend was true ... that Linus did it over a weekend. The bastion of all human knowledge says Git was done in four days and fully operational in a matter of weeks. Ahhh... that's the good stuff.
Before Linus, there were other heroes: Vint and Vixie, Gettys and Garbee and Grace, Barber and Farber, John and John and Jonn (Eckert and Mauchly -vs- von Neumann, rivals), Brooks and Bartik, Kay and Cray, Englebart. I'm grasping at straws and leaving out dozens ... hundreds. But you get the idea.
Intrigue inspires and FOSS fosters freedom.
VM/CMS
In my day job, I work with CMS. No, not that CMS, this CMS. It's IBM's "Conversational Monitor System" (fka "Cambridge Monitor System" to show some history). CMS is a single user, single-tasking operating system hosted by the oldest viable hypervisor: VM/CP the "Control Program".
Hey ... hypervisor ... you know what that is! It's that thing Rosenblum created. Yeah.
Not exactly. Didn't mean to drip into sarcasm. If I haven't lost the audience, then what follows may be more inspiration than irritation.
VM/CP is arguably the best hypervisor, and CMS (in spite of its warts) has been able to focus on a handful of incredible things, including CMS UPDATE. (We did mention code management, you recall.) I love this stuff!
Virtualization rocks! With CP under it, CMS is truly useful. XEDIT, REXX, Pipelines, a CISC processor and a full-featured assembler (plus C and other languages), that multi-level source code update tool, even OpenVM (yes, there's a shell, and yes, that part is multi-tasking).
History of VM
In the glory days of VM, Melinda Varian put together the often cited and repeatedly praised "VM and the VM Community: Past, Present, and Future". Her work ranks up there with Tracy Kidder's "The Soul of a New Machine", though less well known (and without the assist of a publisher).
I was already a fan of VM when I saw Melinda present at a SHARE conference. Hearing the history, the stories of how the system came to be, it all gave me chills. This is the system IBM tried to kill. Over and over.
The ultimate insult (to IBM) happened when Linux was ported to the mainframe. Linux works great on any platform, but is a teriffic candidate for virtualization, especially on high-redundancy hardware (like mainframes). There were two ports, by the way, one by IBM and another by Linas. No, not that Linus, and not that one either, this Linas. But that's a whole nutha story. (more intrigue!)
IBM sales guy: "Remember that system we told you to get rid of?"
IBM customer: "You mean VM? Sure. We got rid of it, like you told us."
IBM sales guy: "Well, you're gonna want it back."
IBM customer: [facepalm]
Addendum
The 1991 revision of Melinda's paper had a couple of addendums. I picked up printed copies. [remember paper?] One addendum was "Dave Tuttle's Memoirs". This is not usually seen alongside the newer revisions (of Melinda's work).
At one job, we were hosting lotso Linux on top of z/VM. (That company had never gotten rid of VM anyway.) Trying to inspire my teammates, I made copies of Tuttle's Memoirs and passed them around. [remember printed copies?] Some of the guys actually read it. These were good guys. I don't think they were in it for the money, maybe for the tech. We weren't coding, but I hope they were intrigued.
Given the myriad search engines available, you can probably find Mr. Tuttle's work for yourself. Here's one I found (other than my paper copy), starting about page 135:
http://www.cs.tufts.edu/~nr/cs257/archive/melinda-varian/neuvm.pdf
Enjoy! Be intrigued. Be inspired.
Someday I'd like to meet Dave Tuttle in person.
-- R; <><
I suppose some people think they're in it just for the money or just for the tech or just for the business. Maybe. But human nature consistently shows us that we need something more. Ultimately, it's that "image of God" creativity, even for those who think we're not terribly creative (and those who know we're not terribly god-like or even godly).
Why do you do what you do?
In software, especially in FOSS, why do you code the code you code?
Intrigue
I attended a Git workshop this past Monday evening. (Yeah, yeah ... it's about time. Hey! It's not like I don't use code manglement, CVS, SVN, etc.) The class was taught by Aidan Feldman of github. Great stuff! Excellent quick workshop; got us all actually using Git on github. Anyway, he mentioned that Linus created Git. I asked if the legend was true ... that Linus did it over a weekend. The bastion of all human knowledge says Git was done in four days and fully operational in a matter of weeks. Ahhh... that's the good stuff.
Before Linus, there were other heroes: Vint and Vixie, Gettys and Garbee and Grace, Barber and Farber, John and John and Jonn (Eckert and Mauchly -vs- von Neumann, rivals), Brooks and Bartik, Kay and Cray, Englebart. I'm grasping at straws and leaving out dozens ... hundreds. But you get the idea.
Intrigue inspires and FOSS fosters freedom.
VM/CMS
In my day job, I work with CMS. No, not that CMS, this CMS. It's IBM's "Conversational Monitor System" (fka "Cambridge Monitor System" to show some history). CMS is a single user, single-tasking operating system hosted by the oldest viable hypervisor: VM/CP the "Control Program".
Hey ... hypervisor ... you know what that is! It's that thing Rosenblum created. Yeah.
Not exactly. Didn't mean to drip into sarcasm. If I haven't lost the audience, then what follows may be more inspiration than irritation.
VM/CP is arguably the best hypervisor, and CMS (in spite of its warts) has been able to focus on a handful of incredible things, including CMS UPDATE. (We did mention code management, you recall.) I love this stuff!
Virtualization rocks! With CP under it, CMS is truly useful. XEDIT, REXX, Pipelines, a CISC processor and a full-featured assembler (plus C and other languages), that multi-level source code update tool, even OpenVM (yes, there's a shell, and yes, that part is multi-tasking).
History of VM
In the glory days of VM, Melinda Varian put together the often cited and repeatedly praised "VM and the VM Community: Past, Present, and Future". Her work ranks up there with Tracy Kidder's "The Soul of a New Machine", though less well known (and without the assist of a publisher).
I was already a fan of VM when I saw Melinda present at a SHARE conference. Hearing the history, the stories of how the system came to be, it all gave me chills. This is the system IBM tried to kill. Over and over.
The ultimate insult (to IBM) happened when Linux was ported to the mainframe. Linux works great on any platform, but is a teriffic candidate for virtualization, especially on high-redundancy hardware (like mainframes). There were two ports, by the way, one by IBM and another by Linas. No, not that Linus, and not that one either, this Linas. But that's a whole nutha story. (more intrigue!)
IBM sales guy: "Remember that system we told you to get rid of?"
IBM customer: "You mean VM? Sure. We got rid of it, like you told us."
IBM sales guy: "Well, you're gonna want it back."
IBM customer: [facepalm]
Addendum
The 1991 revision of Melinda's paper had a couple of addendums. I picked up printed copies. [remember paper?] One addendum was "Dave Tuttle's Memoirs". This is not usually seen alongside the newer revisions (of Melinda's work).
At one job, we were hosting lotso Linux on top of z/VM. (That company had never gotten rid of VM anyway.) Trying to inspire my teammates, I made copies of Tuttle's Memoirs and passed them around. [remember printed copies?] Some of the guys actually read it. These were good guys. I don't think they were in it for the money, maybe for the tech. We weren't coding, but I hope they were intrigued.
Given the myriad search engines available, you can probably find Mr. Tuttle's work for yourself. Here's one I found (other than my paper copy), starting about page 135:
http://www.cs.tufts.edu/~nr/cs257/archive/melinda-varian/neuvm.pdf
Enjoy! Be intrigued. Be inspired.
Someday I'd like to meet Dave Tuttle in person.
-- R; <><
Friday, January 17, 2014
Ho, Ho, Zero
Ho, Ho, Zero
Sir Santa tried to stuff the bag with an annual gift or two, but was late. The gifts are also a bit slim this year.
Of note, CD0, pronounced "see dee zero". This is a rootstrapping tool for NORD Linux. The slim part is that the instructions are missing. But the purpose of the thing is to supply a shell and minimal environment. When combined with GCC and BINUTILS (from CD2), the scripts at http://www.casita.net/pub/nord/build/, and a helping hand from Perl (from CD1), you can "rootstrap" a fresh system.
This rootstrapping thing (using the current CD0) has been tested on S390 and I386. The CD image also contains support for PPC, which is expected to work just fine, but is untested.
The other gift in the bag is a minor update to UFT making it leak less info in secure contexts. Seems to work just fine, but we'll have to talk about that in another post.
-- R; <><
Sir Santa tried to stuff the bag with an annual gift or two, but was late. The gifts are also a bit slim this year.
Of note, CD0, pronounced "see dee zero". This is a rootstrapping tool for NORD Linux. The slim part is that the instructions are missing. But the purpose of the thing is to supply a shell and minimal environment. When combined with GCC and BINUTILS (from CD2), the scripts at http://www.casita.net/pub/nord/build/, and a helping hand from Perl (from CD1), you can "rootstrap" a fresh system.
This rootstrapping thing (using the current CD0) has been tested on S390 and I386. The CD image also contains support for PPC, which is expected to work just fine, but is untested.
The other gift in the bag is a minor update to UFT making it leak less info in secure contexts. Seems to work just fine, but we'll have to talk about that in another post.
-- R; <><
Wednesday, January 8, 2014
Climate Change on the Net
Climate Change on the Net
There's a storm brewing. Winds of change have shaken loose the low hanging fruit and blown open the barn doors. But fruit which stays on the ground will rot and doors left open will let livestock out and predators in.
Some points to note ...
Even prior to consumer availability of IPv6, we were seeing more "things" on the internet. In my own house, when we replaced our aging Zenith television with a stylish Samsung LCD, we wound up with three "devices" on our home LAN which prior counterparts were not networked. They speak a variety of convenience protocols. Behind my firewall, they're supposedly safe. Supposedly.
Electronics in vehicles and appliances have increased and gotten steadily more sophisticated. The idiot light says "service required", so you visit your mechanic (or the auto parts store), they plug-in their OBD reader and tell you what's really wrong. Soon, your fridge will tell you to buy milk (or will order milk all by itself). Kinda scary. But also very geeky.
Cloud computing is the buzzword du jour. It's progressive in that we can buy (rent) disposable complete systems for almost any computing work imaginable. But we deploy complete package bundles, and that doesn't always scale. Default distributions are not by default secure.
So how's cloud retro?
It takes us back to the "service bureau" days.
That's not a bad thing, just ironic. It reverses the trend over several decades of owning the hardware, but it fulfills part of the MULTICS vision of computing as a utility.
A Northern Wind
NORD is a simple "open source" operating system following the standard recipe. It uses the 'make install' logic defined by individual package authors and supplements that with a ready-to-run scheme. Other than that, it's just Linux.
Unlike some Linux distributions, NORD tries to be more Unix than Linux. But it is pragmatic enough to use the Linux kernel, GNU core utilities, and other Linux packages.
The software and computing industry is increasingly distressed about security. (Privacy too, but mostly with the overall hardness against attack.) Consider a recent Wired Magazine article by Bruce Schneier (with help from Jim Gettys, I'm told) ...
The Internet of Things is Wildly Insecure
NORD is particularly suited to embedded use. (Like a generic underpinning for things like OpenWrt.) For simplicity, and giving a boost to security, it starts less services and daemons than other systems.
I use NORD daily for a variety of things, mostly related to software development. (I have a day job.) At one time, NORD's unnamed predecessor served as one of my primary internet-facing servers. It was reliable and secure.
But NORD needs help. It doesn't have an installer. That's by design, but it needs a way to bootstrap onto new hardware. Someone who can assemble a quick stamp utility for I386 and S390 would really be welcome.
An old reference doc for NORD is in Google space ...
NORD Scratch Linux
Recently, friends familiar with the project have lent a hand, one in particular. Their contributions are huge, breaking NORD out of the "one man's toy" mode into something reproducible. Thanks.
NORD is not the only source based system available. If more people, especially the civic minded hacker brain trust, would use systems like LFS, the digital world would be better overall.
There's a companion project, not so much help on the systems front but a boost to security. UFT (Unsolicited File Transfer) has gotten new attention. While its original design was more for interoperability than privacy, it turns out to be great for both goals.
Join the movement.
-- R; <><
There's a storm brewing. Winds of change have shaken loose the low hanging fruit and blown open the barn doors. But fruit which stays on the ground will rot and doors left open will let livestock out and predators in.
Some points to note ...
- IPv6, enabling "the internet of things"
- pervasive embedded computing (not your father's Oldsmobile)
- cloud computing, futuristic and retro
Even prior to consumer availability of IPv6, we were seeing more "things" on the internet. In my own house, when we replaced our aging Zenith television with a stylish Samsung LCD, we wound up with three "devices" on our home LAN which prior counterparts were not networked. They speak a variety of convenience protocols. Behind my firewall, they're supposedly safe. Supposedly.
Electronics in vehicles and appliances have increased and gotten steadily more sophisticated. The idiot light says "service required", so you visit your mechanic (or the auto parts store), they plug-in their OBD reader and tell you what's really wrong. Soon, your fridge will tell you to buy milk (or will order milk all by itself). Kinda scary. But also very geeky.
Cloud computing is the buzzword du jour. It's progressive in that we can buy (rent) disposable complete systems for almost any computing work imaginable. But we deploy complete package bundles, and that doesn't always scale. Default distributions are not by default secure.
So how's cloud retro?
It takes us back to the "service bureau" days.
That's not a bad thing, just ironic. It reverses the trend over several decades of owning the hardware, but it fulfills part of the MULTICS vision of computing as a utility.
A Northern Wind
NORD is a simple "open source" operating system following the standard recipe. It uses the 'make install' logic defined by individual package authors and supplements that with a ready-to-run scheme. Other than that, it's just Linux.
Unlike some Linux distributions, NORD tries to be more Unix than Linux. But it is pragmatic enough to use the Linux kernel, GNU core utilities, and other Linux packages.
The software and computing industry is increasingly distressed about security. (Privacy too, but mostly with the overall hardness against attack.) Consider a recent Wired Magazine article by Bruce Schneier (with help from Jim Gettys, I'm told) ...
The Internet of Things is Wildly Insecure
NORD is particularly suited to embedded use. (Like a generic underpinning for things like OpenWrt.) For simplicity, and giving a boost to security, it starts less services and daemons than other systems.
I use NORD daily for a variety of things, mostly related to software development. (I have a day job.) At one time, NORD's unnamed predecessor served as one of my primary internet-facing servers. It was reliable and secure.
But NORD needs help. It doesn't have an installer. That's by design, but it needs a way to bootstrap onto new hardware. Someone who can assemble a quick stamp utility for I386 and S390 would really be welcome.
An old reference doc for NORD is in Google space ...
NORD Scratch Linux
Recently, friends familiar with the project have lent a hand, one in particular. Their contributions are huge, breaking NORD out of the "one man's toy" mode into something reproducible. Thanks.
NORD is not the only source based system available. If more people, especially the civic minded hacker brain trust, would use systems like LFS, the digital world would be better overall.
There's a companion project, not so much help on the systems front but a boost to security. UFT (Unsolicited File Transfer) has gotten new attention. While its original design was more for interoperability than privacy, it turns out to be great for both goals.
Join the movement.
-- R; <><
| NORD Mascot |
Wednesday, November 6, 2013
Console - Lose the Password, Reap Better Security
Console - Lose the Password, Reap Better Security
Often, that which seems contrary to what we want is really the best way to get it.
No More Passwords
You may have heard talk about being more secure by not using passwords. Specifically, if we use means other than traditional password access then the target systems and services may be better defended. In English, using a PKI certificate, an SSH key, or some other form of credentials is typically more secure than ye olde username/password pair.
I agree with this view.
For many of my systems, there is no password.
I don't mean simply that the password is unknown. I certainly don't mean that the password is blank. What I'm saying is that there is no usable password. Anything typed in will fail. To access these boxes, the user (typically moi) must take one of the other routes.
It works.
Unusable Passwords
A quick way to render root's password unusable is to code an asterisk in the password field (either /etc/passwd then 'pwconv' or /etc/shadow directly). This is the norm for service accounts (bin, daemon, mail, nobody, ftp). Best practice is to employ 'sudo' for all root work. Sign on as yourself (with SSH or whatever), then 'sudo' as needed. There's an audit trail. How novel.
So ... you don't really need a root password anyway, now do you?
But I propose something even more radical.
I wear the sysadmin hat daily. Whether for development or for hobby and home, I get to play "root". (Some of us actually enjoy this, in small doses.)
Virtualization is virtually everywhere. Cool!
But one of the most annoying things about virtualization is the double-signon effect. In my sysadmin mode, I get confronted with this regularly. I'm signed onto the host, but the guest throws a password prompt. [sigh] Not surprising; makes sense; but is in the way of real work.
It's worse than "in the way", more than a hassle. For the guest to process your secondary sign-on, it must have ... drum roll ... a password. Now wait just a minute. We're trying to do away with passwords!! Those of us trying to evolve beyond passwords find this situation positively primordial. (a prime ordeal) What to do??
No More Login
The modest proposal: throw a shell on the guest console directly.
Shocked? You should be. (Unless you've heard it before.)
It's not a new idea. Some of us have been recommending this for several years. Do this in concert with unusable passwords. Replace 'getty' with a shell. (The security guys usually don't "get it" about the nature of 'getty' so they throw a hissy fit and we sysadmins are forced to acquiesce.) But it's a good idea, and in the long run more secure (for virtual consoles) than a login prompt.
With an operational shell, not a login prompt, you get immediately to work. There is no fumbling around at the guest console. There is no password to be concerned about.
I wish the security guys (or any objectors) would follow this line of reasoning. Someone accessing a virtual console has already been vetted by the host. Someone with control of a guest (even if a password prompt is in effect) can do so much more, whether good or bad. So if they #1 have been verified and #2 are in a position to completely re-image the guest, then a no-login-required shell is perfectly reasonable. Combine that with the unusable passwords trick and your virtual systems are hardened on one more front.
Depending on the physical security of your data center, this crazy concept might be a good idea for physical consoles too. The security issue is the same: you're sitting in front of a physical machine with full ability to reboot, re-image, anything. We presume that your data center is not the public library, where password-protected consoles still make sense even though operators freely Ctrl-Alt-Del. Results not typical. Use with caution. Your mileage may vary. Do not fold, spindle or mutilate. And I am not a lawyer.
If your shop is small, then you may *be* the security guy. You could schedule a meeting with yourself, discuss the pros and cons, and convince yourself that this is in fact a good idea. (But talking to yourself is a red flag for anyone in that role.)
For Example
For most of my guest systems, I replace 'getty' with a no-login-required script of some sort. The contents of /sbin/conshell are roughly as follows.
Yeah yeah ... there are spiffier ways to code the conditionals. (Where is Jon Miller when I need him?) I've left this as-is because it works on a wide range of shells.
Thanks for reading. Stay safe.
-- R; <><
Often, that which seems contrary to what we want is really the best way to get it.
No More Passwords
You may have heard talk about being more secure by not using passwords. Specifically, if we use means other than traditional password access then the target systems and services may be better defended. In English, using a PKI certificate, an SSH key, or some other form of credentials is typically more secure than ye olde username/password pair.
I agree with this view.
For many of my systems, there is no password.
I don't mean simply that the password is unknown. I certainly don't mean that the password is blank. What I'm saying is that there is no usable password. Anything typed in will fail. To access these boxes, the user (typically moi) must take one of the other routes.
It works.
Unusable Passwords
A quick way to render root's password unusable is to code an asterisk in the password field (either /etc/passwd then 'pwconv' or /etc/shadow directly). This is the norm for service accounts (bin, daemon, mail, nobody, ftp). Best practice is to employ 'sudo' for all root work. Sign on as yourself (with SSH or whatever), then 'sudo' as needed. There's an audit trail. How novel.
So ... you don't really need a root password anyway, now do you?
But I propose something even more radical.
I wear the sysadmin hat daily. Whether for development or for hobby and home, I get to play "root". (Some of us actually enjoy this, in small doses.)
Virtualization is virtually everywhere. Cool!
But one of the most annoying things about virtualization is the double-signon effect. In my sysadmin mode, I get confronted with this regularly. I'm signed onto the host, but the guest throws a password prompt. [sigh] Not surprising; makes sense; but is in the way of real work.
It's worse than "in the way", more than a hassle. For the guest to process your secondary sign-on, it must have ... drum roll ... a password. Now wait just a minute. We're trying to do away with passwords!! Those of us trying to evolve beyond passwords find this situation positively primordial. (a prime ordeal) What to do??
No More Login
The modest proposal: throw a shell on the guest console directly.
Shocked? You should be. (Unless you've heard it before.)
It's not a new idea. Some of us have been recommending this for several years. Do this in concert with unusable passwords. Replace 'getty' with a shell. (The security guys usually don't "get it" about the nature of 'getty' so they throw a hissy fit and we sysadmins are forced to acquiesce.) But it's a good idea, and in the long run more secure (for virtual consoles) than a login prompt.
With an operational shell, not a login prompt, you get immediately to work. There is no fumbling around at the guest console. There is no password to be concerned about.
I wish the security guys (or any objectors) would follow this line of reasoning. Someone accessing a virtual console has already been vetted by the host. Someone with control of a guest (even if a password prompt is in effect) can do so much more, whether good or bad. So if they #1 have been verified and #2 are in a position to completely re-image the guest, then a no-login-required shell is perfectly reasonable. Combine that with the unusable passwords trick and your virtual systems are hardened on one more front.
Depending on the physical security of your data center, this crazy concept might be a good idea for physical consoles too. The security issue is the same: you're sitting in front of a physical machine with full ability to reboot, re-image, anything. We presume that your data center is not the public library, where password-protected consoles still make sense even though operators freely Ctrl-Alt-Del. Results not typical. Use with caution. Your mileage may vary. Do not fold, spindle or mutilate. And I am not a lawyer.
If your shop is small, then you may *be* the security guy. You could schedule a meeting with yourself, discuss the pros and cons, and convince yourself that this is in fact a good idea. (But talking to yourself is a red flag for anyone in that role.)
For Example
For most of my guest systems, I replace 'getty' with a no-login-required script of some sort. The contents of /sbin/conshell are roughly as follows.
CON=console
if [ ! -z "$1" ] ; then CON="$1" ; fi
if [ "$CON" = `basename $CON` ] ; then CON=/dev/$CON ; fi
PS1='\$ ' ; export PS1
exec sh -i 0<$CON 1>$CON 2>$CON
Yeah yeah ... there are spiffier ways to code the conditionals. (Where is Jon Miller when I need him?) I've left this as-is because it works on a wide range of shells.
Thanks for reading. Stay safe.
-- R; <><
Friday, October 18, 2013
Lessons 67 and 68 - Internet Outage
Lessons 67 and 68
So far, I've lost 8 hours and $150. But we have internet again.
It was Tuesday. I am normally at the office on Tuesdays. But my teammate was out and I have a full plate, so I thought I'd save the commute time and work from home. While sitting with my wife and contemplating the day before us, I noticed the lights blink. We both heard a beep. Right away I knew two or three appliances had reset, including a server or two. No biggie, so I thought.
When we have an outage, I try to learn from the mistakes. Whether I missed a step or some system has let me down, it's a healthy challenge to review and adjust.
Mistake Number 1 - delayed UPS battery replacement
The beep was from the UPS which covers our "important" computer gear:
the cablemodem, the router, and the main server. The BSL1079 (lead/acid "gel cell") failed long ago, but I had been using an aging car battery. This was not simply putting things off. The car battery, even aging, has more than ten times the capacity of the normal UPS battery. But either the battery had aged more than I knew or my spit-n-bailin-wire rigging had loosened up. This was totally my fault.
Lesson: just order the [expletive deleted] normal battery and do high capacity as a separate project.
Mistake Number 2 - mixing services ... and service levels
Some time back, all our stuff hubbed off a server called "main".
NFS, YP/NIS, SMB, NTP, DNS, SMTP, IMAP, internal HTTP, and notably DHCP. Most of these services have been doled out to dedicated appliances or to service providers. The exception is DHCP. So when the primary server powered up, it had these old filesystems to check. (Things did not come down clean, so an integrity check is warranted.) DHCP had to wait until that was done.
The filesystems are still used, but with a lower service level requirement. DHCP has a much higher service level requirement, especially with increased WiFi. So the idea that a high priority service is waiting behind a lower priority service is bass-ackards (as we say in Texas). This will change. My fault, there's history.
Lesson: consider service requirements and plan accordingly. (DHCP will move)
Mistake Number 3 (not mine) - deceptive diagnostics (and this was the worst)
With the server back, and the network units functioning normally (I thought), I checked on our IPv6 tunnel server. This is a Xen virtual machine hosted by the same physical box as "main". Native IPv6 is not available yet where we are, but the SixXS tunnel does nicely. But the tunnel wasn't starting.
Turned out that IPv4 connectivity was still down.
Turned out that the router had no DHCP lease from our ISP.
After multiple (controlled) on/off cycles of both the cablemodem and router, the relationship was still "we're not talking". Activity lights, yes, but all zeros for the external address. Plugged in a laptop directly to the cablemodem; got a lease! So that indicated clearly the router has failed. Clearly.
This NetGear router has been giving us a little trouble on the WiFi side. Dunno if it is just RF interference or perhaps something we can blame on the internet provider. There are gaps in 802.11 coverage inside the house, dead zones. So off I went to Best Buy, returning with a shiny new LinkSys "AC" model, with A/B/G/N backward compatibility.
The new router also failed to obtain a DHCP lease. Huh?!?
Pause and reflect: Old router, no lease. Laptop, yes lease.
But new router, also no lease. The cablemodem is just not smart enough (one would think) to distinguish between a "computer" and a "router". I had already tried cloning the MAC address of the laptop to the old router so it would look to the provider's DHCP server "just like the laptop" (which had succeeded in DHCP). No joy.
This is where I lost the rest of the day ... multiple reboots, power on/off cycles, WiFi reassociations, and DHCP transactions (on the "inside"). Cablemodem worked with two computers, failed on one, failed on both routers. The new router got a DHCP lease on our internal LAN, and then so did the old router. [sigh] (I could have gotten a better priced on the new router if I were not in a rush from the outage.)
What a waste of time.
What finally worked was to put our old "firewall" on the cablemodem. This is a Linux box with two ethernet ports. That's one I got right. And the "mistake" was misleading cues from Time/Warner Cable's device.
Lesson: hang onto what worked before, at least one generation.Maybe consider changing internet providers!
Mistake Number 4 - too much reliance on internet (?)
Uhh... maybe not.
We do *business* online.
For most transactions, using the Internet is no less legitimate, even more reliable, than using the telephone. So what are you gonna tell me? Too much reliance on the phone?
I grant you that if we rely on internet for these things we need to have reliable alternatives, whether a procedural habit fall-back to voice or perhaps redundant internet service. Yeah ... that's it. Redundant internet service. In a prior job, my employer paid for internet and we paid for a second line, keeping personal and "work" on separate channels. WHEN (not if) there was a failure on one, I would switch all traffic to the other.
The idea that Joe Suburbia have two internet contracts is silly.
The idea that my family have two internet contracts is less so, but still pricey.
Lesson: be prepared, and have alternatives.
I wrote this in a hurry because I thought it should be dispatched quickly. Hopefully it's not too jumbled. Hopefully your internet experience is more effective.
-- R; <><
So far, I've lost 8 hours and $150. But we have internet again.
It was Tuesday. I am normally at the office on Tuesdays. But my teammate was out and I have a full plate, so I thought I'd save the commute time and work from home. While sitting with my wife and contemplating the day before us, I noticed the lights blink. We both heard a beep. Right away I knew two or three appliances had reset, including a server or two. No biggie, so I thought.
When we have an outage, I try to learn from the mistakes. Whether I missed a step or some system has let me down, it's a healthy challenge to review and adjust.
Mistake Number 1 - delayed UPS battery replacement
The beep was from the UPS which covers our "important" computer gear:
the cablemodem, the router, and the main server. The BSL1079 (lead/acid "gel cell") failed long ago, but I had been using an aging car battery. This was not simply putting things off. The car battery, even aging, has more than ten times the capacity of the normal UPS battery. But either the battery had aged more than I knew or my spit-n-bailin-wire rigging had loosened up. This was totally my fault.
Lesson: just order the [expletive deleted] normal battery and do high capacity as a separate project.
Mistake Number 2 - mixing services ... and service levels
Some time back, all our stuff hubbed off a server called "main".
NFS, YP/NIS, SMB, NTP, DNS, SMTP, IMAP, internal HTTP, and notably DHCP. Most of these services have been doled out to dedicated appliances or to service providers. The exception is DHCP. So when the primary server powered up, it had these old filesystems to check. (Things did not come down clean, so an integrity check is warranted.) DHCP had to wait until that was done.
The filesystems are still used, but with a lower service level requirement. DHCP has a much higher service level requirement, especially with increased WiFi. So the idea that a high priority service is waiting behind a lower priority service is bass-ackards (as we say in Texas). This will change. My fault, there's history.
Lesson: consider service requirements and plan accordingly. (DHCP will move)
Mistake Number 3 (not mine) - deceptive diagnostics (and this was the worst)
With the server back, and the network units functioning normally (I thought), I checked on our IPv6 tunnel server. This is a Xen virtual machine hosted by the same physical box as "main". Native IPv6 is not available yet where we are, but the SixXS tunnel does nicely. But the tunnel wasn't starting.
Turned out that IPv4 connectivity was still down.
Turned out that the router had no DHCP lease from our ISP.
After multiple (controlled) on/off cycles of both the cablemodem and router, the relationship was still "we're not talking". Activity lights, yes, but all zeros for the external address. Plugged in a laptop directly to the cablemodem; got a lease! So that indicated clearly the router has failed. Clearly.
This NetGear router has been giving us a little trouble on the WiFi side. Dunno if it is just RF interference or perhaps something we can blame on the internet provider. There are gaps in 802.11 coverage inside the house, dead zones. So off I went to Best Buy, returning with a shiny new LinkSys "AC" model, with A/B/G/N backward compatibility.
The new router also failed to obtain a DHCP lease. Huh?!?
Pause and reflect: Old router, no lease. Laptop, yes lease.
But new router, also no lease. The cablemodem is just not smart enough (one would think) to distinguish between a "computer" and a "router". I had already tried cloning the MAC address of the laptop to the old router so it would look to the provider's DHCP server "just like the laptop" (which had succeeded in DHCP). No joy.
This is where I lost the rest of the day ... multiple reboots, power on/off cycles, WiFi reassociations, and DHCP transactions (on the "inside"). Cablemodem worked with two computers, failed on one, failed on both routers. The new router got a DHCP lease on our internal LAN, and then so did the old router. [sigh] (I could have gotten a better priced on the new router if I were not in a rush from the outage.)
What a waste of time.
What finally worked was to put our old "firewall" on the cablemodem. This is a Linux box with two ethernet ports. That's one I got right. And the "mistake" was misleading cues from Time/Warner Cable's device.
Lesson: hang onto what worked before, at least one generation.Maybe consider changing internet providers!
Mistake Number 4 - too much reliance on internet (?)
Uhh... maybe not.
We do *business* online.
For most transactions, using the Internet is no less legitimate, even more reliable, than using the telephone. So what are you gonna tell me? Too much reliance on the phone?
I grant you that if we rely on internet for these things we need to have reliable alternatives, whether a procedural habit fall-back to voice or perhaps redundant internet service. Yeah ... that's it. Redundant internet service. In a prior job, my employer paid for internet and we paid for a second line, keeping personal and "work" on separate channels. WHEN (not if) there was a failure on one, I would switch all traffic to the other.
The idea that Joe Suburbia have two internet contracts is silly.
The idea that my family have two internet contracts is less so, but still pricey.
Lesson: be prepared, and have alternatives.
I wrote this in a hurry because I thought it should be dispatched quickly. Hopefully it's not too jumbled. Hopefully your internet experience is more effective.
-- R; <><
Subscribe to:
Posts (Atom)
