Sunday, December 3, 2017



I grew up using VM/CMS and wrote a few programs for that platform. While most of my world is Unixy, there are some things in *CMS just begging to be ported. One is 'XMITMSG' (the command) and "APPLMSG" (the assembler macro). Together they form the interface to the message handler used by most well-developed CMS programs.

Just Say It

'echo', Say, printf(), ... so many ways to write a line of output. And since we all want to keep it simple, who would ask for the complexity of a "message handler" just to display verbiage? Years ago, I wrote a certain client program that was really well received. That was fun! Someone actually liked my program. Got lots of feedback and feature requests. One requested change kinda put me off, "Use XMITMSG.". What the heck is XMITMSG? Why would I want to use it? "You can change the message repository for other languages.". Okay, now that is interesting, and cool, but sounds like a lot of work. "It's easy!", and it turned out it was.

I submitted to the desires of my users, changed out the "Say" statements, replacing them with calls to the 'XMITMSG' program. Had to put all the verbal content into a message repository, but it worked. I was hooked! Not too long after, someone contributed a message repository for Portuguese. Wow. So my client was the only internationalized program of its kind.

More Strucured

It's not just about internationalization. Using the message handler meant that human readable output was better managed. Forcing myself to use 'XMITMSG' meant I had to think just a little bit harder about what each print operation was actually doing. I'm naturally un-structured, which seems to be contrary for techies, especially coders, so the utility was a real help for me in development. On other platforms, I really miss that CMS message handler.

I cobbled up a crude 'xmitmsg' for Unix. It's still out there, but last time I tinkered with it there were compiler issues. (C has changed, and so have I. A re-write was in order. See below.) 

Beyond organization, going through a central function also means you can direct traffic more effectively. What if you want something logged instead of displayed? What if you want certain output muted? These things are done for free with a proper display handler.

But We can Hash and Replace

Later, I was on a team developing a multi-platform product. Part of it ran on CMS. On the VM side, we had this built-in function. It would be trivial to do it on AIX and Solaris and Linux. (I had already done it and could do it better in a formal context.)

The Unix guys would not go for it. One in particular, a brilliant engineer named Mike, conveyed to me that it just didn't make sense from his perspective. (Almost like a hygiene thing and calling out messages by number was somehow unclean.) Instead they wanted to hash the messages and do string replacement. Not sure what they were using: sounded sorta like GetText but probably a fore-runner. I lost that battle. The product shipped. Move along. Nothing to see here.

And Then Came FUZIX

More than two years ago, I added an XMITMSG work-alike project to my GitHub space. Nuthin happnin; no time. But I've been prompted to get back into GitHub.

One of the projects I follow (for certain values of "follow"; who has time??) is FUZIX. Small is beautiful. FUZIX is a Unix like system for 8-bit processors. Remember the 6502? Yeah, that one. It's an ongoing effort with lots of contributors doing 'make'. One of the contributors posted the output of a failing build which contained this:

fallo en las instrucciones para el objetivo 'stty.rel'

Which in English is:

recipe for target 'stty.rel' failed

This is a perfect candidate for XMITMSG.
This is the kind of thing the message handle project is intended to address.


The project is called "xmitmsgx", note the trailing "x", XMITMSG for POSIX.
It works.
It needs work.

Check it out. Maybe do a pull request and contribute some code.

The initial cut is compatible with CMS 'XMITMSG', same message source syntax, same message code format, stuff like that. But let's think outside the box and realize that other input and output are doable. Meanwhile, it helps to have a known working model to compare against.

-- R; <><

*CMS is the "Conversational Monitor System" component of z/VM and previous incarnations of that operating system. The acronym is so over-used that most readers probably think of something else when they see it. In this post, I'm explicitly talking about VM/CMS even if I leave off the "VM/" prefix. Capisce?

Sunday, November 5, 2017

Call it Lennarx

Lennarx Assimilation

I took the mic at a certain conference, Q&A after the speaker made his pitch, shared my concerns about the topic. There were probably 300 people in that session. The speaker was obviously pro where I was con. Most of the audience were supporters, enthusiastic just to hear. No one spoke up for my side. (i.e., to second my statement or raise their own questions or worries) Possibly not one else was on "my side".
I felt alone.

More recently, my colleague Phil said "it's good to be contrarian". That's comforting, if not effective. I'd really like to bring actual change and not merely whine about the badness.My wife would surely appreciate if I were less contrarian in general.

Linux and SystemD

Speaking of whining, there's lots of that over SystemD. Time and again I hear from people who hate it. But it's not going away. The criticisms are legimitate, even if opinion. For example, SystemD assimilating the logging function is gross overreach, but not technically insurmountable. I can say I dislike the fact that logging is rolled into SystemD but can't prove that it doesn't work. So I (and others, I know they're out there) sit here under the rule of Lennart.

Then my friend Russ sent this ...

I don't know Christopher Barry, but I agree with everything he said. (He said it better than I could have and enumerated more facts than I would have.)

Call it Lennarx

If the kernel started by Linus is Linux, then the system daemon started by Lennart is Lennarx. Why not?

The pro-SystemD crowd would just as soon see the rest of us get over it, shut up, and accept "this is Linux". I've never understood (or never cared about) the benefits. They've never understood (or never cared about) the costs of it, what the rest of us lost. And what did we lose? Simplicity, for one thing. This is not a "who moved my cheese?" story.

For me, there's also the loss of interoperability. (SysV INIT is generally compatible with other Unix systems.) And I was told that one advantage of SystemD is faster boot times.Hasn't happened. Rebooting takes just as long, sometimes longer. But think about it ... faster boot time? WHY do we want to improve something we don't really want to DO so often?

There is Hope

Though we're often accused of tilting at windmills, there is real hope. The majority will no doubt continue with SystemD but the alternatives are out there. Hearing from people like Christopher Barry prove that we're not alone. Most of us also use the BSDs. And that's kinda the point: it's more about Unix and FOSS built for that API.

FOSS runs everywhere. FOSS fans have always had to fight FUD. This latest borg battle has made major inroads. It has assimilated some of our friends. But the remnant remains.

-- R; <><

Monday, October 30, 2017

NORD Updates for 2017

NORD Updates for 2017

Here's the latest status and package updates for NORD.

NORD is in production use for both internal and external service at, and is presently reliable and stable. But it's smart to keep current.

NORD Updates - core operating system

Most of the NORD systems at use a shared operating system. For virtual machines, the OS resides on a shared virtual disk and "client" guests all boot from that disk and use it for the core system. To perform maintenance on the OS, there are swappable copies of this disk.  When applying mantenance, mount the alternate disk, 'chroot' into that filesystem, mount any required support, and run 'nord-build-csc' for each package of interest. (Tool chain is usually not part of the core OS in NORD, so would be mounted from a separate disk or filesystem.)

Packages updated lately include (listed here with versions) ...

  • bash-4.4
  • coreutils-8.27
  • curl-7.54.1
  • dash-
  • diffutils-3.6
  • file-5.31
  • findutils-4.6.0
  • gawk-4.1.4
  • gettext-0.19.8
  • grep-3.1
  • gzip-1.8
  • m4-1.4.18
  • nano-2.8.6
  • readline-7.0
  • rsync-3.1.2
  • sed-4.4
  • texinfo-6.4
  • wget-1.19
  • xz-5.2.3
  • zlib-1.2.11

For each, the stub makefile can be found from ...

The syntax of 'nord-build-csc' does not include the version. The version is in the stub makefile.

After updating core packages, simply unmount the 'chroot' environment. Swizzle the alternate disk (now with the latest core packages) into the place of the production disk when the time is right and reboot the client virtual machines.

NORD Updates - supplemental software

Supplemental packages on NORD are handled primarily via Chicory. (In fact, a major purpose of NORD is to serve as a showcase for Chicory and as a build environment.) Most of the core OS packages in NORD can also be built with Chicory. All of the above updates are reflected in the Chicory build of those packages. For each, see the "wrapper" makefile ...

Chicory packages updated recently include ...

  • bind-9.11.1
  • db-6.0.20
  • gcc-4.2.4
  • gcc-4.8.5
  • gnucobol-1.1
  • jansson-2.10
  • libevent-2.0.22
  • libevent-2.1.8
  • libressl-2.5.5
  • libressl-2.6.0
  • musl-1.1.16
  • openssl-0.9.8k
  • openvpn-2.3.15
  • python-2.5.2
  • regina-3.9.1
  • screen-4.6.1

This is not an exhaustive list.

On NORD, use 'nord-build-opt' or 'chicory-build' to build Chicory packages. 

The updated core packages have been reflected in a master spreadsheet for NORD. The supplemental packages have not, but soon will. This spreadsheet, along with other docs, resides in Google space, which is considered sub-optimal since it is proprietary. There is also a Github respository for NORD which includes a growing body of NORD build scripts. The official repository is and  ( is available both HTTP and HTTPS and does not require HSTS.)

-- R; <><

Saturday, July 1, 2017

Identity Based Encryption

Identity Based Encryption

We were readying the house for visitors and cleaning-up. In the loft, on a bookshelf,  I found a house key. It fit the door to our house, but didn't unlock it.
Where did this key come from? What door does it open? 

It occurred to me that some kind of identifier would help. We've got several keys for other houses and they're all tagged. This key happened to not be tagged. 

This is one aspect of  Identity Based Encryption. With crypto, there are keys. What files, messages, or systems do the keys unlock? The identity is more important than the key itself. 

It's not a stretch to say that a key without an identity is worthless. Indeed, a key under your control without an identity that others would recognize may seem more  secure. The fact that others cannot know what door (file, message, system) it unlocks may be a kind of security. 
But you know what it's for. 

My employer sells "identity based encryption", both asymmetric (IBE) and symmetric (IBSE). It's good stuff. It makes sense. 

This unlabeled house key brings the story to life. Keys need identities so their purpose can be known. 

-- R; <><

Monday, April 24, 2017

NORD and stali

NORD and stali

My friend Skippy sent email suggesting I check out stali. I had seen it before. Good stuff, if perhaps a bit knee-jerk.

Can't help but think that stali springs from a Linux-centric experience, even though the developers clearly value simplicity. And I'm thrilled to see another team fighting the good fight against bloat.

How does it relate (if at all) to NORD?
I took the framework of stali goals and wrote a NORD philosophy (section of the intro doc):

  • Follow the Unix philosophy.
  • Target i386, s390, ppc, and arm hardware.
  • Separate easily-replaced core system from optional packages.
  • Follow Linux FHS where it makes sense.
  • Don’t use SystemD.
  • Make as much static as is reasonable. (minimize shared library dependencies)
  • Achieve simplicity and stability. (good rescue or embedded system)
  • Achieve runs-from-ROM.
  • Minimize security attack surfaces.
  • Include a hand selected collection of the standard tools.
  • Upgrade/install using RSYNC; no package manager needed.

In recent weeks, I needed a clean development system (again!). Had trouble building Squid Proxy on I386. (It built just fine on S390.) So I went about cycling through the core packages (again!), also updated the kernel headers, and tried a re-build of GLIBC. Got stuck. Still stuck. But most things build and re-build just fine.

I've been reviewing some Chicory-built packages to re-do them with static linkage. That will be an ongoing process. 

-- R; <><

Sunday, January 15, 2017

NORD Rationale

NORD Rationale

A few days ago (this is mid January 2017), I added a rationale section to the NORD Linux intro document. I've given a lot of thought to the "why should anyone bother?" question with respect to NORD. The project has become an obsession, but I find objective reasons to continue using it. It's not just a hobby but a tool for other work.

Two systems handling web traffic and other services for are NORD systems. Those are Buckeyes and ltroth1. There are other systems where NORD runs in 'chroot' handling some workloads within that jail. So the environment has become significant within this domain.

Showcase for Other Projects

NORD didn't start out as yet another distro. Projects like simply recompiling the Linux kernel added to a collection. The collection grew into a usable system. It reached critical mass and could sustain real work. That seems to be still its primary purpose. It is the stage where other projects perform.

CSCRATCH is the project which [re]builds the core operating system. It's unique to NORD (though it's not exclusively for Linux). Chicory is more widely effective. Both are just wrappers around the standard recipe.

NORD is my primary platform for hardening activities: reliability, auditability, servicability, as well as penetration defense. There are also growing concerns about trusting trust. (Sure, I'm paranoid. But am I paranoid enough?)

Serious about Source

I've been working with Free and Open Source software for most of my career. At this point, the pay-for and proprietary software that puts food on my table depends inseparably on FOSS. 

Ironically, the rise in use of FOSS throughout the industry has not led to a corresponding use of source code at delivery points. The Linux distributors and forward looking software vendors have done such a good job of embracing FOSS and making it drop-in usable that their customers don't need to actually use the source. But it's tech debt. 

As long as the providers do the right thing, their customers can proceed with their own business and focus on more important details of that operation. And most vendors/distributors are doing the right thing and will likely continue. But take note, be aware, and get ready. Consider source code as part of your business continuity plan. 

NORD can be acquired and used without ever worrying about source or compiling or building. (Compared to "real" distributions it's really rough and I presume the consumer knows traditional Unix.) But NORD is designed to be re-built in a pinch. NORD can assimilate an update or  patch faster than any other Linux implementation. (In recent history was Shell Shock which NORD handled no delays: as soon as patches were available, just recompile.) 

Summary in Three Ss

Rationale for NORD solidifies into simplicity, showcase, and source code. 

Now I need to get others interested. A half dozen friends have lent a hand over the course of this saga. A couple of them have been able to make use of the deliverables. We need more participants. (We at least need people to hammer on the build logic, test the results, find bugs and maybe squash them.) 
Wanna help?

-- R; <><

Sunday, August 21, 2016

The Death of Veracity

The Death of Veracity

Your vocabulary word for today, students, is "veracity". You may notice that Wikipedia re-directs to the page for "honesty". But this post is about a computer, not about moral integrity.

How we came to name the machine "veracity" is a sentimental saga for another season. The soggy sadness of today is the fallout from Veracity (the machine) having silently succumbed to an as-yet unknown hardware failure.

In recent weeks I've gotten a number of alarms "you need to take backups". We seem to have had a rash of machine failures. Veracity's demise is the second outage this week alone. (The other was "the day the WiFi died" which I'll describe in a separate post.)

This is a disaster recovery story, a D/R tail with a happy ending. 

Veracity and Virtualization

The good news is that the systems hosted on veracity appear to be intact. One of them, zechariah by name, is our primary IPv6 gateway server. It is up again, after copying of its system disk and root disk to another virtualization host. Whatever failed on Veracity, thankfully the disk was okay.

The other guest, Jeremiah, followed soon after. It acts as our "main" server (email, files, print, DNS, DHCP). But I had gotten lax about backups. The D/R plan for jeremiah was that if it failed we'd switch the IP address for main over from Jeremiah to Nehemiah. While it lived, Nehemiah contained regular backups of Jeremiah's content. We did switch between the two once or twice in those days.

This method of using a similar-but-not-identical system for failover goes back before we had virtual machines on our little network. Where physical systems are involved, the historical plan for D/R is to have another system with the same or better capability standing ready to pick up the load. I was introduced to virtualization in 1982 but pervasive PC-think prevented me from applying tried and true V12N methods to personal systems. Bummer.

It began to dawn on me that we don't need a secondary server for a virtual system. All we really need is a copy of that system, a clone. Call it replication. Then when disaster strikes, bring up the clone on a designated alternate hypervisor: no moving around of IP addresses, no quirks from the subtle differences between the recovery system and the real system. A copy of a virtual machine is a perfect substitute because it's not actually a substitute. They're more identical than twins. 

Replication and Recovery

Zecharian and Jeremiah are in better shape now than they were before the mishap. The host hardware to which they got moved has KVM. Previously they were Xen guests. Not complaining about Xen, but the change forced me to make some adjustments that had been put off, things that needed to be done anyway. They were already configured to share the O/S (another fun rabbit trail, maybe another blog post). They share a common system disk image, now corrected for KVM. (They each have their own root filesystem.) Once the KVM changes were done for one, the other instantly got the same benefit.

I had more success recovering Zechariah and Jeremiah than with this blog post. (Don't get me started about how the Blogger app likes to lose draft updates.) 

NORD to the Rescue

As it happened, I had a newer kernel for NORD than that of the SUSE release Jeremiah and Zechariah run. As it happened, I already had a KVM-bootable system disk. So I copied NORD's kernel to the SUSE system disk and re-stamped the bootstrap. Generally, one simply brings in the kernel and related modules. As long as the kernel and modules are as-new-as and within the same generation the userland parts should have few problems. Works.

Note: this is a good reason to sym-link /lib/modules to /boot/modules and commit to a replaceable /boot volume It's a feature of NORD but trivial with any Linux distro.

KVM performance sucks relative to that of Xen. Any time you can use para-virtualization (instead of full hardware simulation) you're going to see better performance. Xen was originally para-virt-only and continues to strongly support para-virtualization. But we're using KVM for the sake of manageability. The guests can run with no knowledge of the hypervisor. (We can always switch to para-virt later, selectively per device.) And these guests aren't doing heavy multi-media work. Performance is sufficient. Presence is paramount.

You can see Zechariah for yourself. (You'll need IPv6 to hit it.) The web content is presently unchaged, demonsntrating the effect of a replicated virtual machine. An update with part of this story will be forthcoming. Jeremiah's connectivity is more controlled, not generally reachable from outside.

-- R; <><