Call it Lennarx

Lennarx Assimilation


I took the mic at a certain conference, Q&A after the speaker made his pitch, shared my concerns about the topic. There were probably 300 people in that session. The speaker was obviously pro where I was con. Most of the audience were supporters, enthusiastic just to hear. No one spoke up for my side. (i.e., to second my statement or raise their own questions or worries) Possibly not one else was on "my side".
I felt alone.


More recently, my colleague Phil said "it's good to be contrarian". That's comforting, if not effective. I'd really like to bring actual change and not merely whine about the badness.My wife would surely appreciate if I were less contrarian in general.


Linux and SystemD


Speaking of whining, there's lots of that over SystemD. Time and again I hear from people who hate it. But it's not going away. The criticisms are legimitate, even if opinion. For example, SystemD assimilating the logging function is gross overreach, but not technically insurmountable. I can say I dislike the fact that logging is rolled into SystemD but can't prove that it doesn't work. So I (and others, I know they're out there) sit here under the rule of Lennart.


Then my friend Russ sent this ...

http://lkml.iu.edu//hypermail/linux/kernel/1408.1/02496.html

I don't know Christopher Barry, but I agree with everything he said. (He said it better than I could have and enumerated more facts than I would have.)


Call it Lennarx


If the kernel started by Linus is Linux, then the system daemon started by Lennart is Lennarx. Why not?


The pro-SystemD crowd would just as soon see the rest of us get over it, shut up, and accept "this is Linux". I've never understood (or never cared about) the benefits. They've never understood (or never cared about) the costs of it, what the rest of us lost. And what did we lose? Simplicity, for one thing. This is not a "who moved my cheese?" story.

For me, there's also the loss of interoperability. (SysV INIT is generally compatible with other Unix systems.) And I was told that one advantage of SystemD is faster boot times.Hasn't happened. Rebooting takes just as long, sometimes longer. But think about it ... faster boot time? WHY do we want to improve something we don't really want to DO so often?


There is Hope


Though we're often accused of tilting at windmills, there is real hope. The majority will no doubt continue with SystemD but the alternatives are out there. Hearing from people like Christopher Barry prove that we're not alone. Most of us also use the BSDs. And that's kinda the point: it's more about Unix and FOSS built for that API.


FOSS runs everywhere. FOSS fans have always had to fight FUD. This latest borg battle has made major inroads. It has assimilated some of our friends. But the remnant remains.


-- R; <><

NORD Updates for 2017

NORD Updates for 2017


Here's the latest status and package updates for NORD.


NORD is in production use for both internal and external service at casita.net, and is presently reliable and stable. But it's smart to keep current.


NORD Updates - core operating system


Most of the NORD systems at casita.net use a shared operating system. For virtual machines, the OS resides on a shared virtual disk and "client" guests all boot from that disk and use it for the core system. To perform maintenance on the OS, there are swappable copies of this disk.  When applying mantenance, mount the alternate disk, 'chroot' into that filesystem, mount any required support, and run 'nord-build-csc' for each package of interest. (Tool chain is usually not part of the core OS in NORD, so would be mounted from a separate disk or filesystem.)


Packages updated lately include (listed here with versions) ...

• bash-4.4
• coreutils-8.27
• curl-7.54.1
• dash-0.5.9.1
• diffutils-3.6
• file-5.31
• findutils-4.6.0
• gawk-4.1.4
• gettext-0.19.8
• grep-3.1
• gzip-1.8
• m4-1.4.18
• nano-2.8.6
• readline-7.0
• rsync-3.1.2
• sed-4.4
• texinfo-6.4
• wget-1.19
• xz-5.2.3
• zlib-1.2.11

For each, the stub makefile can be found from ...

http://www.casita.net/pub/cscratch/package.mk

The syntax of 'nord-build-csc' does not include the version. The version is in the stub makefile.


After updating core packages, simply unmount the 'chroot' environment. Swizzle the alternate disk (now with the latest core packages) into the place of the production disk when the time is right and reboot the client virtual machines.



NORD Updates - supplemental software


Supplemental packages on NORD are handled primarily via Chicory. (In fact, a major purpose of NORD is to serve as a showcase for Chicory and as a build environment.) Most of the core OS packages in NORD can also be built with Chicory. All of the above updates are reflected in the Chicory build of those packages. For each, see the "wrapper" makefile ...

http://www.casita.net/pub/package/package-version.mak

Chicory packages updated recently include ...

• bind-9.11.1
• db-6.0.20
• gcc-4.2.4
• gcc-4.8.5
• gnucobol-1.1
• jansson-2.10
• libevent-2.0.22
• libevent-2.1.8
• libressl-2.5.5
• libressl-2.6.0
• musl-1.1.16
• openssl-0.9.8k
• openvpn-2.3.15
• python-2.5.2
• regina-3.9.1
• screen-4.6.1

This is not an exhaustive list.


On NORD, use 'nord-build-opt' or 'chicory-build' to build Chicory packages. 


The updated core packages have been reflected in a master spreadsheet for NORD. The supplemental packages have not, but soon will. This spreadsheet, along with other docs, resides in Google space, which is considered sub-optimal since it is proprietary. There is also a Github respository for NORD which includes a growing body of NORD build scripts. The official repository is http://www.casita.net/pub/nord and http://www.casita.net/nord.  (casita.net is available both HTTP and HTTPS and does not require HSTS.)


-- R; <><

Identity Based Encryption

Identity Based Encryption

We were readying the house for visitors and cleaning-up. In the loft, on a bookshelf,  I found a house key. It fit the door to our house, but didn't unlock it.
Where did this key come from? What door does it open? 

It occurred to me that some kind of identifier would help. We've got several keys for other houses and they're all tagged. This key happened to not be tagged. 

This is one aspect of  Identity Based Encryption. With crypto, there are keys. What files, messages, or systems do the keys unlock? The identity is more important than the key itself. 

It's not a stretch to say that a key without an identity is worthless. Indeed, a key under your control without an identity that others would recognize may seem more  secure. The fact that others cannot know what door (file, message, system) it unlocks may be a kind of security. 

But you know what it's for. 

My employer sells "identity based encryption", both asymmetric (IBE) and symmetric (IBSE). It's good stuff. It makes sense. 

This unlabeled house key brings the story to life. Keys need identities so their purpose can be known. 

-- R; <><

 

NORD and stali

NORD and stali

My friend Skippy sent email suggesting I check out stali. I had seen it before. Good stuff, if perhaps a bit knee-jerk.

Can't help but think that stali springs from a Linux-centric experience, even though the developers clearly value simplicity. And I'm thrilled to see another team fighting the good fight against bloat.

How does it relate (if at all) to NORD?
I took the framework of stali goals and wrote a NORD philosophy (section of the intro doc):

• Follow the Unix philosophy.
• Target i386, s390, ppc, and arm hardware.
• Separate easily-replaced core system from optional packages.
• Follow Linux FHS where it makes sense.
• Don’t use SystemD.
• Make as much static as is reasonable. (minimize shared library dependencies)
• Achieve simplicity and stability. (good rescue or embedded system)
• Achieve runs-from-ROM.
• Minimize security attack surfaces.
• Include a hand selected collection of the standard tools.
• Upgrade/install using RSYNC; no package manager needed.

In recent weeks, I needed a clean development system (again!). Had trouble building Squid Proxy on I386. (It built just fine on S390.) So I went about cycling through the core packages (again!), also updated the kernel headers, and tried a re-build of GLIBC. Got stuck. Still stuck. But most things build and re-build just fine.

I've been reviewing some Chicory-built packages to re-do them with static linkage. That will be an ongoing process. 

-- R; <><

NORD Rationale

NORD Rationale


A few days ago (this is mid January 2017), I added a rationale section to the NORD Linux intro document. I've given a lot of thought to the "why should anyone bother?" question with respect to NORD. The project has become an obsession, but I find objective reasons to continue using it. It's not just a hobby but a tool for other work.

Two systems handling web traffic and other services for casita.net are NORD systems. Those are Buckeyes and ltroth1. There are other systems where NORD runs in 'chroot' handling some workloads within that jail. So the environment has become significant within this domain.


Showcase for Other Projects


NORD didn't start out as yet another distro. Projects like simply recompiling the Linux kernel added to a collection. The collection grew into a usable system. It reached critical mass and could sustain real work. That seems to be still its primary purpose. It is the stage where other projects perform.

CSCRATCH is the project which [re]builds the core operating system. It's unique to NORD (though it's not exclusively for Linux). Chicory is more widely effective. Both are just wrappers around the standard recipe.

NORD is my primary platform for hardening activities: reliability, auditability, servicability, as well as penetration defense. There are also growing concerns about trusting trust. (Sure, I'm paranoid. But am I paranoid enough?)


Serious about Source


I've been working with Free and Open Source software for most of my career. At this point, the pay-for and proprietary software that puts food on my table depends inseparably on FOSS. 

Ironically, the rise in use of FOSS throughout the industry has not led to a corresponding use of source code at delivery points. The Linux distributors and forward looking software vendors have done such a good job of embracing FOSS and making it drop-in usable that their customers don't need to actually use the source. But it's tech debt. 

As long as the providers do the right thing, their customers can proceed with their own business and focus on more important details of that operation. And most vendors/distributors are doing the right thing and will likely continue. But take note, be aware, and get ready. Consider source code as part of your business continuity plan. 

NORD can be acquired and used without ever worrying about source or compiling or building. (Compared to "real" distributions it's really rough and I presume the consumer knows traditional Unix.) But NORD is designed to be re-built in a pinch. NORD can assimilate an update or  patch faster than any other Linux implementation. (In recent history was Shell Shock which NORD handled no delays: as soon as patches were available, just recompile.) 

Summary in Three Ss

Rationale for NORD solidifies into simplicity, showcase, and source code. 

Now I need to get others interested. A half dozen friends have lent a hand over the course of this saga. A couple of them have been able to make use of the deliverables. We need more participants. (We at least need people to hammer on the build logic, test the results, find bugs and maybe squash them.) 

Wanna help?


-- R; <><

The Death of Veracity

The Death of Veracity

Your vocabulary word for today, students, is "veracity". You may notice that Wikipedia re-directs to the page for "honesty". But this post is about a computer, not about moral integrity.

How we came to name the machine "veracity" is a sentimental saga for another season. The soggy sadness of today is the fallout from Veracity (the machine) having silently succumbed to an as-yet unknown hardware failure.

In recent weeks I've gotten a number of alarms "you need to take backups". We seem to have had a rash of machine failures. Veracity's demise is the second outage this week alone. (The other was "the day the WiFi died" which I'll describe in a separate post.)

This is a disaster recovery story, a D/R tail with a happy ending. 

Veracity and Virtualization

The good news is that the systems hosted on veracity appear to be intact. One of them, zechariah by name, is our primary IPv6 gateway server. It is up again, after copying of its system disk and root disk to another virtualization host. Whatever failed on Veracity, thankfully the disk was okay.

The other guest, Jeremiah, followed soon after. It acts as our "main" server (email, files, print, DNS, DHCP). But I had gotten lax about backups. The D/R plan for jeremiah was that if it failed we'd switch the IP address for main over from Jeremiah to Nehemiah. While it lived, Nehemiah contained regular backups of Jeremiah's content. We did switch between the two once or twice in those days.

This method of using a similar-but-not-identical system for failover goes back before we had virtual machines on our little network. Where physical systems are involved, the historical plan for D/R is to have another system with the same or better capability standing ready to pick up the load. I was introduced to virtualization in 1982 but pervasive PC-think prevented me from applying tried and true V12N methods to personal systems. Bummer.

It began to dawn on me that we don't need a secondary server for a virtual system. All we really need is a copy of that system, a clone. Call it replication. Then when disaster strikes, bring up the clone on a designated alternate hypervisor: no moving around of IP addresses, no quirks from the subtle differences between the recovery system and the real system. A copy of a virtual machine is a perfect substitute because it's not actually a substitute. They're more identical than twins. 

Replication and Recovery

Zecharian and Jeremiah are in better shape now than they were before the mishap. The host hardware to which they got moved has KVM. Previously they were Xen guests. Not complaining about Xen, but the change forced me to make some adjustments that had been put off, things that needed to be done anyway. They were already configured to share the O/S (another fun rabbit trail, maybe another blog post). They share a common system disk image, now corrected for KVM. (They each have their own root filesystem.) Once the KVM changes were done for one, the other instantly got the same benefit.

I had more success recovering Zechariah and Jeremiah than with this blog post. (Don't get me started about how the Blogger app likes to lose draft updates.) 

NORD to the Rescue

As it happened, I had a newer kernel for NORD than that of the SUSE release Jeremiah and Zechariah run. As it happened, I already had a KVM-bootable system disk. So I copied NORD's kernel to the SUSE system disk and re-stamped the bootstrap. Generally, one simply brings in the kernel and related modules. As long as the kernel and modules are as-new-as and within the same generation the userland parts should have few problems. Works.

Note: this is a good reason to sym-link /lib/modules to /boot/modules and commit to a replaceable /boot volume It's a feature of NORD but trivial with any Linux distro.

KVM performance sucks relative to that of Xen. Any time you can use para-virtualization (instead of full hardware simulation) you're going to see better performance. Xen was originally para-virt-only and continues to strongly support para-virtualization. But we're using KVM for the sake of manageability. The guests can run with no knowledge of the hypervisor. (We can always switch to para-virt later, selectively per device.) And these guests aren't doing heavy multi-media work. Performance is sufficient. Presence is paramount.

You can see Zechariah for yourself. (You'll need IPv6 to hit it.) The web content is presently unchaged, demonsntrating the effect of a replicated virtual machine. An update with part of this story will be forthcoming. Jeremiah's connectivity is more controlled, not generally reachable from outside.

-- R; <><

misdirected redirection

misdirected redirection

 ... or HTTPS everywhere getting us nowhere


I'm a long time believer in crypto. (Look me up. I'm in the Web-of-Trust.) I've done SSL stack development (in assembler). My day job is helping customers integrate field-level encryption. And I look forward to a safer, more secure, heavily crypto-laden internet.
But I'm concerned about the rush to HTTPS everywhere.


Okay, "getting us nowhere" is an exaggeration, but it makes for a decent sound-alike to English speakers.


To be specific, there are cases where plain HTTP makes more sense. Sometimes it's just for performance sake. Don't presume that every object must be fetched via SSL/TLS. Some content is actually not a security risk. I could ask "Why would you burden not-at-risk clear traffic with encryption?". But I won't ask. To ask would be rhetorical and that would be ASSuming things that might not be correct.


And then there's automation. For better or worse, our world runs on automation, much of which is not encrypted. Not now anyway. Much not now; some not ever. And unencrypted operation is not inherently evil.


Rule number one: don't break stuff.


Instead of rhetorically asking or assuming, I'll tell you: think about what should and should not be protected with TLS/SSL. Don't blindly bloat our treasured traffic with careless crypto. Think. Be selective about which sessions and services actually needs the extra work. (And it will be extra work, and it won't be your burden alone. Choices you make affect other people, always.)


Okay, "don't break stuff" is trumped by security (and by bugs). But hear my point that getting it right is hard work. Blanket solutions aren't solutions, and solving even the most urgent problems by wanton breakage is to follow one problem with another.


It's as if someone (make that plural, many someones) asked (rhetorically), "Why would you not encrypt everything?". The question ASSumes that there's no good reason to have cleartext on the net. But there are good reasons. I'll cite only one because I'm tired, presently annoyed, and generally cranky of late. Here's a classic, "Why would you ever need more than 640K?".


Important note: That's not how the quote goes and Bill Gates never actually said such a thing. (Someone did say something once to John Sculley about floppies being all Apple would ever need in response to a question about Macintosh networking. Wanna guess who?) Please hear my second point that there is no one-size-fits-all for software, or for hardware, or for clothing.

My case is the chicken-and-egg situation of trying to build OpenSSL from source. One must download the source before one can build the source. How does that happen? Easy, use 'curl' or 'wget' or some similar tool, point to openssl.org, get the tarball. Explode the downloaded tarball and follow the standard recipe. But if you don't have SSL working then you can't use HTTPS.

So far my case doesn't sound like a problem. Here's the problem. Some bright "HTTPS everywhere" aficionado decided that nobody should be using HTTP. When you hit openssl.org via HTTP you (now) get re-directed to HTTPS. If you're downloading OpenSSL for the sake of implementing OpenSSL ... and you don't already have some kind of SSL or TLS ... this is a difficult situation.

They broke stuff. They broke my stuff. Now it's personal.

Building systems from source is important. Or I don't know, maybe it's not important. (Seems like it's important to some people, but they're getting hard to find.) I build from source for several reasons, partly because I'm a control freak, partly because I'm a tinker, and partly because I don't trust systems built by other people. Trust ... it's always about trust. (Systems built by other people: I do use them, but kind of like Google. I use Google but I don't trust them. Been saying that for several years now. And look, here I am blogging on a Google property.)

The automation in question actually has SSL. The problem is that it has an embryonic infrastructure with an empty PKI trust store. This is not to say that it doesn't have a solid trust chain. It just doesn't (at the point of fetching OpenSSL source) have a cache of root certificates for the World Wide Web. So when we hit a site like openssl.org (via HTTPS) the server certificate fails to verify. (Plain HTTP is fine, was fine, until mister "my solution works for everyone" did the re-direct re-design on their site.)

Gimme back HTTP!

It's the re-direction that's the problem.
I said HTTP because I meant HTTP. The protocol has 301 and 302. Oy vey, another great feature now fallen victim to abuse by ill-conceived implementation. (There's a long and growing list of those.) The files didn't move. (Would be nice if some people used 301 to replace 404, ya think?)

This is the second time in recent months that I've run up against someone having disabled a perfectly reasonable function because they knew better than the rest of us. I guess we gotta kill stupidity one bad idea at a time.

Let's encrypt. Let's encrypt widely. Let's encrypt carefully.


-- R; <><