Git and markdown and CD0

Git and markdown and CD0

Sir Russ the Just prevails upon me: I'm increasingly using Git for projects.
What's so great about Git?

 Git

Not personally stuck on any one content manglement system. (The acronym CMS is a whole other story.) But Git seems to have advantages for collaborative coding. Hearing the hype, my radar perked up when the GitHub guys offered a free intro class (now five years ago). 

At work we lean primarily on Subversion. But Git looms large. My teammate, the Nomad, gave me his take on the matter. For him, Git works better when you need to branch, test, merge, and commit remotely. (So maybe that helps nomadic life.) The corporately sponsored GitHub server is available to anyone with credentials, not merely our division (and with no special dispensation from our over-worked local devops team). Normalized access is a major plus. Anyone on my team can use it too.

Markdown

Speaking of remote operation, at work we also use Confluence. This popular documentation tool must be incredibly chatty. I haven't analyzed the traffic. All I can say is that (for me, 100% remote) it's ssslllooowww with a capital SSS. (Evidently not slow for those at HQ.) If Confluence pages under the covers were plaintext (HTML or other markup), we could RSYNC to/from the server and make changes easily. But Confluence wants to be interactive by way of your web browser. (Something about hammers and nails and "everybody has Internet Exploder", so what were Confluence designers thinking?)

I don't know what I don't know about Confluence. Gentle reader, please be gentle if you have myriad clues that I lack about the wonders of Confluence and how it can be made to work better. Maybe you'll say, "yes, Rick, you can RSYNC from/to Confluence".
Meanwhile, enter Markdown.

Another nudge from Russ is to get me free from the talons of Google, specifically their documentation service. (Let's not talk about the fact that this post has its genesis on another Google property.) Russ says I should use markdown for project docs. I offered no resistance: markdown is plaintext, it's portable, it's simple. At work, markdown docs on the corporate GitHub server are a refreshing alternative to Confluence pages. For hobbies, I find myself converting Google docs to markdown as time allows.

CD0

All of this comes to a head because NORD has a "GitHub project" presence which is now getting more TLC. This is where I'm burning most of that doco conversion time, moving files from Google Drive to GitHub. But being GitHub, this is also where the non-doc files have been mirroed for a long time. The effect is a convergence of doc files and operational files into a common space. Good!

What's lacking is a spreadsheet, or something that looks/tastes/smells like a spreadsheet, which can serve both for documentation and operation. As NORD evolved, it helped to have a place to publicly list what packages are current. (NORD eschews formal package management, which would include inventory, for the sake of sticking closer to the individual package sources.) Using the common spreadsheet as a reference, one can get a local NORD installation up to date programmatically. Markdown does support tables, so that might be a workable alternative to a proper spreadsheet.

What's CD0?

NORD is comprised of a core operating system with "optional" packages drawn in on demand. The optional packages are built with Chicory so they can reside anywhere, but as a starting point there are collections under /opt named CD1 through CD5. (Maybe more later.) The naming is intentional because eash collection could literally reside on a CD or DVD or in an ISO-9660 image. In any case, separate filesystems is ovten reasonable.

CD0 is a special case. The packages in CD0 are the Chicory build of core packages (normally built with CSCRATCH). CD0 doesn't have the entire core package set, but by design has enough packages for "rootstrapping". Supplement packages from CD0 with those from CD1, CD2, CD3, and a fresh NORD core system root can be derived. Once the new NORD core system has been built, CD0 is no longer needed.

Summary, only tangential to Git and Markdown, the current CD0 set is ...

• autoconf-2.69
• automake-1.15
• bash-4.4
• bison-3.0.4
• bzip2-1.0.6
• coreutils-8.27
• curl-7.54.1
• dash-0.5.9.1
• diffutils-3.6
• file-5.31
• findutils-4.6.0
• flex-2.5.39
• gawk-4.1.4
• gettext-0.19.8
• grep-3.1
• gzip-1.8
• libtool-2.4.6
• m4-1.4.18
• make-3.79.1
• nano-2.8.6
• ncurses-6.0
• patch-2.7.5
• pdksh-5.2.14
• readline-7.0
• rsync-3.1.2
• screen-4.6.1
• sed-4.4
• sharutils-4.15.2
• tar-1.22
• texinfo-6.4
• wget-1.19
• which-2.21
• xz-5.2.3
• zlib-1.2.11

That's 34 packages. Some of them are behind current releases for one reason or another. With Chicory, both old and new can co-exist on the system, but this set is intended for rootstrapping. Now to put the list into table form.

-- R; <><

XMITMSG for POSIX

XMITMSG for POSIX

I grew up using VM/CMS and wrote a few programs for that platform. While most of my world is Unixy, there are some things in *CMS just begging to be ported. One is 'XMITMSG' (the command) and "APPLMSG" (the assembler macro). Together they form the interface to the message handler used by most well-developed CMS programs.

Just Say It

'echo', Say, printf(), ... so many ways to write a line of output. And since we all want to keep it simple, who would ask for the complexity of a "message handler" just to display verbiage? Years ago, I wrote a certain client program that was really well received. That was fun! Someone actually liked my program. Got lots of feedback and feature requests. One requested change kinda put me off, "Use XMITMSG.". What the heck is XMITMSG? Why would I want to use it? "You can change the message repository for other languages.". Okay, now that is interesting, and cool, but sounds like a lot of work. "It's easy!", and it turned out it was.

I submitted to the desires of my users, changed out the "Say" statements, replacing them with calls to the 'XMITMSG' program. Had to put all the verbal content into a message repository, but it worked. I was hooked! Not too long after, someone contributed a message repository for Portuguese. Wow. So my client was the only internationalized program of its kind.

More Strucured

It's not just about internationalization. Using the message handler meant that human readable output was better managed. Forcing myself to use 'XMITMSG' meant I had to think just a little bit harder about what each print operation was actually doing. I'm naturally un-structured, which seems to be contrary for techies, especially coders, so the utility was a real help for me in development. On other platforms, I really miss that CMS message handler.

I cobbled up a crude 'xmitmsg' for Unix. It's still out there, but last time I tinkered with it there were compiler issues. (C has changed, and so have I. A re-write was in order. See below.) 

Beyond organization, going through a central function also means you can direct traffic more effectively. What if you want something logged instead of displayed? What if you want certain output muted? These things are done for free with a proper display handler.

But We can Hash and Replace

Later, I was on a team developing a multi-platform product. Part of it ran on CMS. On the VM side, we had this built-in function. It would be trivial to do it on AIX and Solaris and Linux. (I had already done it and could do it better in a formal context.)

The Unix guys would not go for it. One in particular, a brilliant engineer named Mike, conveyed to me that it just didn't make sense from his perspective. (Almost like a hygiene thing and calling out messages by number was somehow unclean.) Instead they wanted to hash the messages and do string replacement. Not sure what they were using: sounded sorta like GetText but probably a fore-runner. I lost that battle. The product shipped. Move along. Nothing to see here.

And Then Came FUZIX

More than two years ago, I added an XMITMSG work-alike project to my GitHub space. Nuthin happnin; no time. But I've been prompted to get back into GitHub.

One of the projects I follow (for certain values of "follow"; who has time??) is FUZIX. Small is beautiful. FUZIX is a Unix like system for 8-bit processors. Remember the 6502? Yeah, that one. It's an ongoing effort with lots of contributors doing 'make'. One of the contributors posted the output of a failing build which contained this:

fallo en las instrucciones para el objetivo 'stty.rel'

Which in English is:

recipe for target 'stty.rel' failed

This is a perfect candidate for XMITMSG.
This is the kind of thing the message handle project is intended to address.

XMITMSGX

The project is called "xmitmsgx", note the trailing "x", XMITMSG for POSIX.
It works.
It needs work.

https://github.com/trothr/xmitmsgx/

Check it out. Maybe do a pull request and contribute some code.

The initial cut is compatible with CMS 'XMITMSG', same message source syntax, same message code format, stuff like that. But let's think outside the box and realize that other input and output are doable. Meanwhile, it helps to have a known working model to compare against.

-- R; <><

*CMS is the "Conversational Monitor System" component of z/VM and previous incarnations of that operating system. The acronym is so over-used that most readers probably think of something else when they see it. In this post, I'm explicitly talking about VM/CMS even if I leave off the "VM/" prefix. Capisce?

Call it Lennarx

Lennarx Assimilation


I took the mic at a certain conference, Q&A after the speaker made his pitch, shared my concerns about the topic. There were probably 300 people in that session. The speaker was obviously pro where I was con. Most of the audience were supporters, enthusiastic just to hear. No one spoke up for my side. (i.e., to second my statement or raise their own questions or worries) Possibly not one else was on "my side".
I felt alone.


More recently, my colleague Phil said "it's good to be contrarian". That's comforting, if not effective. I'd really like to bring actual change and not merely whine about the badness.My wife would surely appreciate if I were less contrarian in general.


Linux and SystemD


Speaking of whining, there's lots of that over SystemD. Time and again I hear from people who hate it. But it's not going away. The criticisms are legimitate, even if opinion. For example, SystemD assimilating the logging function is gross overreach, but not technically insurmountable. I can say I dislike the fact that logging is rolled into SystemD but can't prove that it doesn't work. So I (and others, I know they're out there) sit here under the rule of Lennart.


Then my friend Russ sent this ...

http://lkml.iu.edu//hypermail/linux/kernel/1408.1/02496.html

I don't know Christopher Barry, but I agree with everything he said. (He said it better than I could have and enumerated more facts than I would have.)


Call it Lennarx


If the kernel started by Linus is Linux, then the system daemon started by Lennart is Lennarx. Why not?


The pro-SystemD crowd would just as soon see the rest of us get over it, shut up, and accept "this is Linux". I've never understood (or never cared about) the benefits. They've never understood (or never cared about) the costs of it, what the rest of us lost. And what did we lose? Simplicity, for one thing. This is not a "who moved my cheese?" story.

For me, there's also the loss of interoperability. (SysV INIT is generally compatible with other Unix systems.) And I was told that one advantage of SystemD is faster boot times.Hasn't happened. Rebooting takes just as long, sometimes longer. But think about it ... faster boot time? WHY do we want to improve something we don't really want to DO so often?


There is Hope


Though we're often accused of tilting at windmills, there is real hope. The majority will no doubt continue with SystemD but the alternatives are out there. Hearing from people like Christopher Barry prove that we're not alone. Most of us also use the BSDs. And that's kinda the point: it's more about Unix and FOSS built for that API.


FOSS runs everywhere. FOSS fans have always had to fight FUD. This latest borg battle has made major inroads. It has assimilated some of our friends. But the remnant remains.


-- R; <><

NORD Updates for 2017

NORD Updates for 2017


Here's the latest status and package updates for NORD.


NORD is in production use for both internal and external service at casita.net, and is presently reliable and stable. But it's smart to keep current.


NORD Updates - core operating system


Most of the NORD systems at casita.net use a shared operating system. For virtual machines, the OS resides on a shared virtual disk and "client" guests all boot from that disk and use it for the core system. To perform maintenance on the OS, there are swappable copies of this disk.  When applying mantenance, mount the alternate disk, 'chroot' into that filesystem, mount any required support, and run 'nord-build-csc' for each package of interest. (Tool chain is usually not part of the core OS in NORD, so would be mounted from a separate disk or filesystem.)


Packages updated lately include (listed here with versions) ...

• bash-4.4
• coreutils-8.27
• curl-7.54.1
• dash-0.5.9.1
• diffutils-3.6
• file-5.31
• findutils-4.6.0
• gawk-4.1.4
• gettext-0.19.8
• grep-3.1
• gzip-1.8
• m4-1.4.18
• nano-2.8.6
• readline-7.0
• rsync-3.1.2
• sed-4.4
• texinfo-6.4
• wget-1.19
• xz-5.2.3
• zlib-1.2.11

For each, the stub makefile can be found from ...

http://www.casita.net/pub/cscratch/package.mk

The syntax of 'nord-build-csc' does not include the version. The version is in the stub makefile.


After updating core packages, simply unmount the 'chroot' environment. Swizzle the alternate disk (now with the latest core packages) into the place of the production disk when the time is right and reboot the client virtual machines.



NORD Updates - supplemental software


Supplemental packages on NORD are handled primarily via Chicory. (In fact, a major purpose of NORD is to serve as a showcase for Chicory and as a build environment.) Most of the core OS packages in NORD can also be built with Chicory. All of the above updates are reflected in the Chicory build of those packages. For each, see the "wrapper" makefile ...

http://www.casita.net/pub/package/package-version.mak

Chicory packages updated recently include ...

• bind-9.11.1
• db-6.0.20
• gcc-4.2.4
• gcc-4.8.5
• gnucobol-1.1
• jansson-2.10
• libevent-2.0.22
• libevent-2.1.8
• libressl-2.5.5
• libressl-2.6.0
• musl-1.1.16
• openssl-0.9.8k
• openvpn-2.3.15
• python-2.5.2
• regina-3.9.1
• screen-4.6.1

This is not an exhaustive list.


On NORD, use 'nord-build-opt' or 'chicory-build' to build Chicory packages. 


The updated core packages have been reflected in a master spreadsheet for NORD. The supplemental packages have not, but soon will. This spreadsheet, along with other docs, resides in Google space, which is considered sub-optimal since it is proprietary. There is also a Github respository for NORD which includes a growing body of NORD build scripts. The official repository is http://www.casita.net/pub/nord and http://www.casita.net/nord.  (casita.net is available both HTTP and HTTPS and does not require HSTS.)


-- R; <><

Identity Based Encryption

Identity Based Encryption

We were readying the house for visitors and cleaning-up. In the loft, on a bookshelf,  I found a house key. It fit the door to our house, but didn't unlock it.
Where did this key come from? What door does it open? 

It occurred to me that some kind of identifier would help. We've got several keys for other houses and they're all tagged. This key happened to not be tagged. 

This is one aspect of  Identity Based Encryption. With crypto, there are keys. What files, messages, or systems do the keys unlock? The identity is more important than the key itself. 

It's not a stretch to say that a key without an identity is worthless. Indeed, a key under your control without an identity that others would recognize may seem more  secure. The fact that others cannot know what door (file, message, system) it unlocks may be a kind of security. 

But you know what it's for. 

My employer sells "identity based encryption", both asymmetric (IBE) and symmetric (IBSE). It's good stuff. It makes sense. 

This unlabeled house key brings the story to life. Keys need identities so their purpose can be known. 

-- R; <><

 

NORD and stali

NORD and stali

My friend Skippy sent email suggesting I check out stali. I had seen it before. Good stuff, if perhaps a bit knee-jerk.

Can't help but think that stali springs from a Linux-centric experience, even though the developers clearly value simplicity. And I'm thrilled to see another team fighting the good fight against bloat.

How does it relate (if at all) to NORD?
I took the framework of stali goals and wrote a NORD philosophy (section of the intro doc):

• Follow the Unix philosophy.
• Target i386, s390, ppc, and arm hardware.
• Separate easily-replaced core system from optional packages.
• Follow Linux FHS where it makes sense.
• Don’t use SystemD.
• Make as much static as is reasonable. (minimize shared library dependencies)
• Achieve simplicity and stability. (good rescue or embedded system)
• Achieve runs-from-ROM.
• Minimize security attack surfaces.
• Include a hand selected collection of the standard tools.
• Upgrade/install using RSYNC; no package manager needed.

In recent weeks, I needed a clean development system (again!). Had trouble building Squid Proxy on I386. (It built just fine on S390.) So I went about cycling through the core packages (again!), also updated the kernel headers, and tried a re-build of GLIBC. Got stuck. Still stuck. But most things build and re-build just fine.

I've been reviewing some Chicory-built packages to re-do them with static linkage. That will be an ongoing process. 

-- R; <><

NORD Rationale

NORD Rationale


A few days ago (this is mid January 2017), I added a rationale section to the NORD Linux intro document. I've given a lot of thought to the "why should anyone bother?" question with respect to NORD. The project has become an obsession, but I find objective reasons to continue using it. It's not just a hobby but a tool for other work.

Two systems handling web traffic and other services for casita.net are NORD systems. Those are Buckeyes and ltroth1. There are other systems where NORD runs in 'chroot' handling some workloads within that jail. So the environment has become significant within this domain.


Showcase for Other Projects


NORD didn't start out as yet another distro. Projects like simply recompiling the Linux kernel added to a collection. The collection grew into a usable system. It reached critical mass and could sustain real work. That seems to be still its primary purpose. It is the stage where other projects perform.

CSCRATCH is the project which [re]builds the core operating system. It's unique to NORD (though it's not exclusively for Linux). Chicory is more widely effective. Both are just wrappers around the standard recipe.

NORD is my primary platform for hardening activities: reliability, auditability, servicability, as well as penetration defense. There are also growing concerns about trusting trust. (Sure, I'm paranoid. But am I paranoid enough?)


Serious about Source


I've been working with Free and Open Source software for most of my career. At this point, the pay-for and proprietary software that puts food on my table depends inseparably on FOSS. 

Ironically, the rise in use of FOSS throughout the industry has not led to a corresponding use of source code at delivery points. The Linux distributors and forward looking software vendors have done such a good job of embracing FOSS and making it drop-in usable that their customers don't need to actually use the source. But it's tech debt. 

As long as the providers do the right thing, their customers can proceed with their own business and focus on more important details of that operation. And most vendors/distributors are doing the right thing and will likely continue. But take note, be aware, and get ready. Consider source code as part of your business continuity plan. 

NORD can be acquired and used without ever worrying about source or compiling or building. (Compared to "real" distributions it's really rough and I presume the consumer knows traditional Unix.) But NORD is designed to be re-built in a pinch. NORD can assimilate an update or  patch faster than any other Linux implementation. (In recent history was Shell Shock which NORD handled no delays: as soon as patches were available, just recompile.) 

Summary in Three Ss

Rationale for NORD solidifies into simplicity, showcase, and source code. 

Now I need to get others interested. A half dozen friends have lent a hand over the course of this saga. A couple of them have been able to make use of the deliverables. We need more participants. (We at least need people to hammer on the build logic, test the results, find bugs and maybe squash them.) 

Wanna help?


-- R; <><