Git and markdown and CD0

Git and markdown and CD0

Sir Russ the Just prevails upon me: I'm increasingly using Git for projects.
What's so great about Git?

 Git

Not personally stuck on any one content manglement system. (The acronym CMS is a whole other story.) But Git seems to have advantages for collaborative coding. Hearing the hype, my radar perked up when the GitHub guys offered a free intro class (now five years ago). 

At work we lean primarily on Subversion. But Git looms large. My teammate, the Nomad, gave me his take on the matter. For him, Git works better when you need to branch, test, merge, and commit remotely. (So maybe that helps nomadic life.) The corporately sponsored GitHub server is available to anyone with credentials, not merely our division (and with no special dispensation from our over-worked local devops team). Normalized access is a major plus. Anyone on my team can use it too.

Markdown

Speaking of remote operation, at work we also use Confluence. This popular documentation tool must be incredibly chatty. I haven't analyzed the traffic. All I can say is that (for me, 100% remote) it's ssslllooowww with a capital SSS. (Evidently not slow for those at HQ.) If Confluence pages under the covers were plaintext (HTML or other markup), we could RSYNC to/from the server and make changes easily. But Confluence wants to be interactive by way of your web browser. (Something about hammers and nails and "everybody has Internet Exploder", so what were Confluence designers thinking?)

I don't know what I don't know about Confluence. Gentle reader, please be gentle if you have myriad clues that I lack about the wonders of Confluence and how it can be made to work better. Maybe you'll say, "yes, Rick, you can RSYNC from/to Confluence".
Meanwhile, enter Markdown.

Another nudge from Russ is to get me free from the talons of Google, specifically their documentation service. (Let's not talk about the fact that this post has its genesis on another Google property.) Russ says I should use markdown for project docs. I offered no resistance: markdown is plaintext, it's portable, it's simple. At work, markdown docs on the corporate GitHub server are a refreshing alternative to Confluence pages. For hobbies, I find myself converting Google docs to markdown as time allows.

CD0

All of this comes to a head because NORD has a "GitHub project" presence which is now getting more TLC. This is where I'm burning most of that doco conversion time, moving files from Google Drive to GitHub. But being GitHub, this is also where the non-doc files have been mirroed for a long time. The effect is a convergence of doc files and operational files into a common space. Good!

What's lacking is a spreadsheet, or something that looks/tastes/smells like a spreadsheet, which can serve both for documentation and operation. As NORD evolved, it helped to have a place to publicly list what packages are current. (NORD eschews formal package management, which would include inventory, for the sake of sticking closer to the individual package sources.) Using the common spreadsheet as a reference, one can get a local NORD installation up to date programmatically. Markdown does support tables, so that might be a workable alternative to a proper spreadsheet.

What's CD0?

NORD is comprised of a core operating system with "optional" packages drawn in on demand. The optional packages are built with Chicory so they can reside anywhere, but as a starting point there are collections under /opt named CD1 through CD5. (Maybe more later.) The naming is intentional because eash collection could literally reside on a CD or DVD or in an ISO-9660 image. In any case, separate filesystems is ovten reasonable.

CD0 is a special case. The packages in CD0 are the Chicory build of core packages (normally built with CSCRATCH). CD0 doesn't have the entire core package set, but by design has enough packages for "rootstrapping". Supplement packages from CD0 with those from CD1, CD2, CD3, and a fresh NORD core system root can be derived. Once the new NORD core system has been built, CD0 is no longer needed.

Summary, only tangential to Git and Markdown, the current CD0 set is ...

• autoconf-2.69
• automake-1.15
• bash-4.4
• bison-3.0.4
• bzip2-1.0.6
• coreutils-8.27
• curl-7.54.1
• dash-0.5.9.1
• diffutils-3.6
• file-5.31
• findutils-4.6.0
• flex-2.5.39
• gawk-4.1.4
• gettext-0.19.8
• grep-3.1
• gzip-1.8
• libtool-2.4.6
• m4-1.4.18
• make-3.79.1
• nano-2.8.6
• ncurses-6.0
• patch-2.7.5
• pdksh-5.2.14
• readline-7.0
• rsync-3.1.2
• screen-4.6.1
• sed-4.4
• sharutils-4.15.2
• tar-1.22
• texinfo-6.4
• wget-1.19
• which-2.21
• xz-5.2.3
• zlib-1.2.11

That's 34 packages. Some of them are behind current releases for one reason or another. With Chicory, both old and new can co-exist on the system, but this set is intended for rootstrapping. Now to put the list into table form.

-- R; <><

XMITMSG for POSIX

XMITMSG for POSIX

I grew up using VM/CMS and wrote a few programs for that platform. While most of my world is Unixy, there are some things in *CMS just begging to be ported. One is 'XMITMSG' (the command) and "APPLMSG" (the assembler macro). Together they form the interface to the message handler used by most well-developed CMS programs.

Just Say It

'echo', Say, printf(), ... so many ways to write a line of output. And since we all want to keep it simple, who would ask for the complexity of a "message handler" just to display verbiage? Years ago, I wrote a certain client program that was really well received. That was fun! Someone actually liked my program. Got lots of feedback and feature requests. One requested change kinda put me off, "Use XMITMSG.". What the heck is XMITMSG? Why would I want to use it? "You can change the message repository for other languages.". Okay, now that is interesting, and cool, but sounds like a lot of work. "It's easy!", and it turned out it was.

I submitted to the desires of my users, changed out the "Say" statements, replacing them with calls to the 'XMITMSG' program. Had to put all the verbal content into a message repository, but it worked. I was hooked! Not too long after, someone contributed a message repository for Portuguese. Wow. So my client was the only internationalized program of its kind.

More Strucured

It's not just about internationalization. Using the message handler meant that human readable output was better managed. Forcing myself to use 'XMITMSG' meant I had to think just a little bit harder about what each print operation was actually doing. I'm naturally un-structured, which seems to be contrary for techies, especially coders, so the utility was a real help for me in development. On other platforms, I really miss that CMS message handler.

I cobbled up a crude 'xmitmsg' for Unix. It's still out there, but last time I tinkered with it there were compiler issues. (C has changed, and so have I. A re-write was in order. See below.) 

Beyond organization, going through a central function also means you can direct traffic more effectively. What if you want something logged instead of displayed? What if you want certain output muted? These things are done for free with a proper display handler.

But We can Hash and Replace

Later, I was on a team developing a multi-platform product. Part of it ran on CMS. On the VM side, we had this built-in function. It would be trivial to do it on AIX and Solaris and Linux. (I had already done it and could do it better in a formal context.)

The Unix guys would not go for it. One in particular, a brilliant engineer named Mike, conveyed to me that it just didn't make sense from his perspective. (Almost like a hygiene thing and calling out messages by number was somehow unclean.) Instead they wanted to hash the messages and do string replacement. Not sure what they were using: sounded sorta like GetText but probably a fore-runner. I lost that battle. The product shipped. Move along. Nothing to see here.

And Then Came FUZIX

More than two years ago, I added an XMITMSG work-alike project to my GitHub space. Nuthin happnin; no time. But I've been prompted to get back into GitHub.

One of the projects I follow (for certain values of "follow"; who has time??) is FUZIX. Small is beautiful. FUZIX is a Unix like system for 8-bit processors. Remember the 6502? Yeah, that one. It's an ongoing effort with lots of contributors doing 'make'. One of the contributors posted the output of a failing build which contained this:

fallo en las instrucciones para el objetivo 'stty.rel'

Which in English is:

recipe for target 'stty.rel' failed

This is a perfect candidate for XMITMSG.
This is the kind of thing the message handle project is intended to address.

XMITMSGX

The project is called "xmitmsgx", note the trailing "x", XMITMSG for POSIX.
It works.
It needs work.

https://github.com/trothr/xmitmsgx/

Check it out. Maybe do a pull request and contribute some code.

The initial cut is compatible with CMS 'XMITMSG', same message source syntax, same message code format, stuff like that. But let's think outside the box and realize that other input and output are doable. Meanwhile, it helps to have a known working model to compare against.

-- R; <><

*CMS is the "Conversational Monitor System" component of z/VM and previous incarnations of that operating system. The acronym is so over-used that most readers probably think of something else when they see it. In this post, I'm explicitly talking about VM/CMS even if I leave off the "VM/" prefix. Capisce?

Call it Lennarx

Lennarx Assimilation


I took the mic at a certain conference, Q&A after the speaker made his pitch, shared my concerns about the topic. There were probably 300 people in that session. The speaker was obviously pro where I was con. Most of the audience were supporters, enthusiastic just to hear. No one spoke up for my side. (i.e., to second my statement or raise their own questions or worries) Possibly not one else was on "my side".
I felt alone.


More recently, my colleague Phil said "it's good to be contrarian". That's comforting, if not effective. I'd really like to bring actual change and not merely whine about the badness.My wife would surely appreciate if I were less contrarian in general.


Linux and SystemD


Speaking of whining, there's lots of that over SystemD. Time and again I hear from people who hate it. But it's not going away. The criticisms are legimitate, even if opinion. For example, SystemD assimilating the logging function is gross overreach, but not technically insurmountable. I can say I dislike the fact that logging is rolled into SystemD but can't prove that it doesn't work. So I (and others, I know they're out there) sit here under the rule of Lennart.


Then my friend Russ sent this ...

http://lkml.iu.edu//hypermail/linux/kernel/1408.1/02496.html

I don't know Christopher Barry, but I agree with everything he said. (He said it better than I could have and enumerated more facts than I would have.)


Call it Lennarx


If the kernel started by Linus is Linux, then the system daemon started by Lennart is Lennarx. Why not?


The pro-SystemD crowd would just as soon see the rest of us get over it, shut up, and accept "this is Linux". I've never understood (or never cared about) the benefits. They've never understood (or never cared about) the costs of it, what the rest of us lost. And what did we lose? Simplicity, for one thing. This is not a "who moved my cheese?" story.

For me, there's also the loss of interoperability. (SysV INIT is generally compatible with other Unix systems.) And I was told that one advantage of SystemD is faster boot times.Hasn't happened. Rebooting takes just as long, sometimes longer. But think about it ... faster boot time? WHY do we want to improve something we don't really want to DO so often?


There is Hope


Though we're often accused of tilting at windmills, there is real hope. The majority will no doubt continue with SystemD but the alternatives are out there. Hearing from people like Christopher Barry prove that we're not alone. Most of us also use the BSDs. And that's kinda the point: it's more about Unix and FOSS built for that API.


FOSS runs everywhere. FOSS fans have always had to fight FUD. This latest borg battle has made major inroads. It has assimilated some of our friends. But the remnant remains.


-- R; <><

NORD Updates for 2017

NORD Updates for 2017


Here's the latest status and package updates for NORD.


NORD is in production use for both internal and external service at casita.net, and is presently reliable and stable. But it's smart to keep current.


NORD Updates - core operating system


Most of the NORD systems at casita.net use a shared operating system. For virtual machines, the OS resides on a shared virtual disk and "client" guests all boot from that disk and use it for the core system. To perform maintenance on the OS, there are swappable copies of this disk.  When applying mantenance, mount the alternate disk, 'chroot' into that filesystem, mount any required support, and run 'nord-build-csc' for each package of interest. (Tool chain is usually not part of the core OS in NORD, so would be mounted from a separate disk or filesystem.)


Packages updated lately include (listed here with versions) ...

• bash-4.4
• coreutils-8.27
• curl-7.54.1
• dash-0.5.9.1
• diffutils-3.6
• file-5.31
• findutils-4.6.0
• gawk-4.1.4
• gettext-0.19.8
• grep-3.1
• gzip-1.8
• m4-1.4.18
• nano-2.8.6
• readline-7.0
• rsync-3.1.2
• sed-4.4
• texinfo-6.4
• wget-1.19
• xz-5.2.3
• zlib-1.2.11

For each, the stub makefile can be found from ...

http://www.casita.net/pub/cscratch/package.mk

The syntax of 'nord-build-csc' does not include the version. The version is in the stub makefile.


After updating core packages, simply unmount the 'chroot' environment. Swizzle the alternate disk (now with the latest core packages) into the place of the production disk when the time is right and reboot the client virtual machines.



NORD Updates - supplemental software


Supplemental packages on NORD are handled primarily via Chicory. (In fact, a major purpose of NORD is to serve as a showcase for Chicory and as a build environment.) Most of the core OS packages in NORD can also be built with Chicory. All of the above updates are reflected in the Chicory build of those packages. For each, see the "wrapper" makefile ...

http://www.casita.net/pub/package/package-version.mak

Chicory packages updated recently include ...

• bind-9.11.1
• db-6.0.20
• gcc-4.2.4
• gcc-4.8.5
• gnucobol-1.1
• jansson-2.10
• libevent-2.0.22
• libevent-2.1.8
• libressl-2.5.5
• libressl-2.6.0
• musl-1.1.16
• openssl-0.9.8k
• openvpn-2.3.15
• python-2.5.2
• regina-3.9.1
• screen-4.6.1

This is not an exhaustive list.


On NORD, use 'nord-build-opt' or 'chicory-build' to build Chicory packages. 


The updated core packages have been reflected in a master spreadsheet for NORD. The supplemental packages have not, but soon will. This spreadsheet, along with other docs, resides in Google space, which is considered sub-optimal since it is proprietary. There is also a Github respository for NORD which includes a growing body of NORD build scripts. The official repository is http://www.casita.net/pub/nord and http://www.casita.net/nord.  (casita.net is available both HTTP and HTTPS and does not require HSTS.)


-- R; <><

Identity Based Encryption

Identity Based Encryption

We were readying the house for visitors and cleaning-up. In the loft, on a bookshelf,  I found a house key. It fit the door to our house, but didn't unlock it.
Where did this key come from? What door does it open? 

It occurred to me that some kind of identifier would help. We've got several keys for other houses and they're all tagged. This key happened to not be tagged. 

This is one aspect of  Identity Based Encryption. With crypto, there are keys. What files, messages, or systems do the keys unlock? The identity is more important than the key itself. 

It's not a stretch to say that a key without an identity is worthless. Indeed, a key under your control without an identity that others would recognize may seem more  secure. The fact that others cannot know what door (file, message, system) it unlocks may be a kind of security. 

But you know what it's for. 

My employer sells "identity based encryption", both asymmetric (IBE) and symmetric (IBSE). It's good stuff. It makes sense. 

This unlabeled house key brings the story to life. Keys need identities so their purpose can be known. 

-- R; <><

 

NORD and stali

NORD and stali

My friend Skippy sent email suggesting I check out stali. I had seen it before. Good stuff, if perhaps a bit knee-jerk.

Can't help but think that stali springs from a Linux-centric experience, even though the developers clearly value simplicity. And I'm thrilled to see another team fighting the good fight against bloat.

How does it relate (if at all) to NORD?
I took the framework of stali goals and wrote a NORD philosophy (section of the intro doc):

• Follow the Unix philosophy.
• Target i386, s390, ppc, and arm hardware.
• Separate easily-replaced core system from optional packages.
• Follow Linux FHS where it makes sense.
• Don’t use SystemD.
• Make as much static as is reasonable. (minimize shared library dependencies)
• Achieve simplicity and stability. (good rescue or embedded system)
• Achieve runs-from-ROM.
• Minimize security attack surfaces.
• Include a hand selected collection of the standard tools.
• Upgrade/install using RSYNC; no package manager needed.

In recent weeks, I needed a clean development system (again!). Had trouble building Squid Proxy on I386. (It built just fine on S390.) So I went about cycling through the core packages (again!), also updated the kernel headers, and tried a re-build of GLIBC. Got stuck. Still stuck. But most things build and re-build just fine.

I've been reviewing some Chicory-built packages to re-do them with static linkage. That will be an ongoing process. 

-- R; <><

NORD Rationale

NORD Rationale


A few days ago (this is mid January 2017), I added a rationale section to the NORD Linux intro document. I've given a lot of thought to the "why should anyone bother?" question with respect to NORD. The project has become an obsession, but I find objective reasons to continue using it. It's not just a hobby but a tool for other work.

Two systems handling web traffic and other services for casita.net are NORD systems. Those are Buckeyes and ltroth1. There are other systems where NORD runs in 'chroot' handling some workloads within that jail. So the environment has become significant within this domain.


Showcase for Other Projects


NORD didn't start out as yet another distro. Projects like simply recompiling the Linux kernel added to a collection. The collection grew into a usable system. It reached critical mass and could sustain real work. That seems to be still its primary purpose. It is the stage where other projects perform.

CSCRATCH is the project which [re]builds the core operating system. It's unique to NORD (though it's not exclusively for Linux). Chicory is more widely effective. Both are just wrappers around the standard recipe.

NORD is my primary platform for hardening activities: reliability, auditability, servicability, as well as penetration defense. There are also growing concerns about trusting trust. (Sure, I'm paranoid. But am I paranoid enough?)


Serious about Source


I've been working with Free and Open Source software for most of my career. At this point, the pay-for and proprietary software that puts food on my table depends inseparably on FOSS. 

Ironically, the rise in use of FOSS throughout the industry has not led to a corresponding use of source code at delivery points. The Linux distributors and forward looking software vendors have done such a good job of embracing FOSS and making it drop-in usable that their customers don't need to actually use the source. But it's tech debt. 

As long as the providers do the right thing, their customers can proceed with their own business and focus on more important details of that operation. And most vendors/distributors are doing the right thing and will likely continue. But take note, be aware, and get ready. Consider source code as part of your business continuity plan. 

NORD can be acquired and used without ever worrying about source or compiling or building. (Compared to "real" distributions it's really rough and I presume the consumer knows traditional Unix.) But NORD is designed to be re-built in a pinch. NORD can assimilate an update or  patch faster than any other Linux implementation. (In recent history was Shell Shock which NORD handled no delays: as soon as patches were available, just recompile.) 

Summary in Three Ss

Rationale for NORD solidifies into simplicity, showcase, and source code. 

Now I need to get others interested. A half dozen friends have lent a hand over the course of this saga. A couple of them have been able to make use of the deliverables. We need more participants. (We at least need people to hammer on the build logic, test the results, find bugs and maybe squash them.) 

Wanna help?


-- R; <><

The Death of Veracity

The Death of Veracity

Your vocabulary word for today, students, is "veracity". You may notice that Wikipedia re-directs to the page for "honesty". But this post is about a computer, not about moral integrity.

How we came to name the machine "veracity" is a sentimental saga for another season. The soggy sadness of today is the fallout from Veracity (the machine) having silently succumbed to an as-yet unknown hardware failure.

In recent weeks I've gotten a number of alarms "you need to take backups". We seem to have had a rash of machine failures. Veracity's demise is the second outage this week alone. (The other was "the day the WiFi died" which I'll describe in a separate post.)

This is a disaster recovery story, a D/R tail with a happy ending. 

Veracity and Virtualization

The good news is that the systems hosted on veracity appear to be intact. One of them, zechariah by name, is our primary IPv6 gateway server. It is up again, after copying of its system disk and root disk to another virtualization host. Whatever failed on Veracity, thankfully the disk was okay.

The other guest, Jeremiah, followed soon after. It acts as our "main" server (email, files, print, DNS, DHCP). But I had gotten lax about backups. The D/R plan for jeremiah was that if it failed we'd switch the IP address for main over from Jeremiah to Nehemiah. While it lived, Nehemiah contained regular backups of Jeremiah's content. We did switch between the two once or twice in those days.

This method of using a similar-but-not-identical system for failover goes back before we had virtual machines on our little network. Where physical systems are involved, the historical plan for D/R is to have another system with the same or better capability standing ready to pick up the load. I was introduced to virtualization in 1982 but pervasive PC-think prevented me from applying tried and true V12N methods to personal systems. Bummer.

It began to dawn on me that we don't need a secondary server for a virtual system. All we really need is a copy of that system, a clone. Call it replication. Then when disaster strikes, bring up the clone on a designated alternate hypervisor: no moving around of IP addresses, no quirks from the subtle differences between the recovery system and the real system. A copy of a virtual machine is a perfect substitute because it's not actually a substitute. They're more identical than twins. 

Replication and Recovery

Zecharian and Jeremiah are in better shape now than they were before the mishap. The host hardware to which they got moved has KVM. Previously they were Xen guests. Not complaining about Xen, but the change forced me to make some adjustments that had been put off, things that needed to be done anyway. They were already configured to share the O/S (another fun rabbit trail, maybe another blog post). They share a common system disk image, now corrected for KVM. (They each have their own root filesystem.) Once the KVM changes were done for one, the other instantly got the same benefit.

I had more success recovering Zechariah and Jeremiah than with this blog post. (Don't get me started about how the Blogger app likes to lose draft updates.) 

NORD to the Rescue

As it happened, I had a newer kernel for NORD than that of the SUSE release Jeremiah and Zechariah run. As it happened, I already had a KVM-bootable system disk. So I copied NORD's kernel to the SUSE system disk and re-stamped the bootstrap. Generally, one simply brings in the kernel and related modules. As long as the kernel and modules are as-new-as and within the same generation the userland parts should have few problems. Works.

Note: this is a good reason to sym-link /lib/modules to /boot/modules and commit to a replaceable /boot volume It's a feature of NORD but trivial with any Linux distro.

KVM performance sucks relative to that of Xen. Any time you can use para-virtualization (instead of full hardware simulation) you're going to see better performance. Xen was originally para-virt-only and continues to strongly support para-virtualization. But we're using KVM for the sake of manageability. The guests can run with no knowledge of the hypervisor. (We can always switch to para-virt later, selectively per device.) And these guests aren't doing heavy multi-media work. Performance is sufficient. Presence is paramount.

You can see Zechariah for yourself. (You'll need IPv6 to hit it.) The web content is presently unchaged, demonsntrating the effect of a replicated virtual machine. An update with part of this story will be forthcoming. Jeremiah's connectivity is more controlled, not generally reachable from outside.

-- R; <><

misdirected redirection

misdirected redirection

 ... or HTTPS everywhere getting us nowhere


I'm a long time believer in crypto. (Look me up. I'm in the Web-of-Trust.) I've done SSL stack development (in assembler). My day job is helping customers integrate field-level encryption. And I look forward to a safer, more secure, heavily crypto-laden internet.
But I'm concerned about the rush to HTTPS everywhere.


Okay, "getting us nowhere" is an exaggeration, but it makes for a decent sound-alike to English speakers.


To be specific, there are cases where plain HTTP makes more sense. Sometimes it's just for performance sake. Don't presume that every object must be fetched via SSL/TLS. Some content is actually not a security risk. I could ask "Why would you burden not-at-risk clear traffic with encryption?". But I won't ask. To ask would be rhetorical and that would be ASSuming things that might not be correct.


And then there's automation. For better or worse, our world runs on automation, much of which is not encrypted. Not now anyway. Much not now; some not ever. And unencrypted operation is not inherently evil.


Rule number one: don't break stuff.


Instead of rhetorically asking or assuming, I'll tell you: think about what should and should not be protected with TLS/SSL. Don't blindly bloat our treasured traffic with careless crypto. Think. Be selective about which sessions and services actually needs the extra work. (And it will be extra work, and it won't be your burden alone. Choices you make affect other people, always.)


Okay, "don't break stuff" is trumped by security (and by bugs). But hear my point that getting it right is hard work. Blanket solutions aren't solutions, and solving even the most urgent problems by wanton breakage is to follow one problem with another.


It's as if someone (make that plural, many someones) asked (rhetorically), "Why would you not encrypt everything?". The question ASSumes that there's no good reason to have cleartext on the net. But there are good reasons. I'll cite only one because I'm tired, presently annoyed, and generally cranky of late. Here's a classic, "Why would you ever need more than 640K?".


Important note: That's not how the quote goes and Bill Gates never actually said such a thing. (Someone did say something once to John Sculley about floppies being all Apple would ever need in response to a question about Macintosh networking. Wanna guess who?) Please hear my second point that there is no one-size-fits-all for software, or for hardware, or for clothing.

My case is the chicken-and-egg situation of trying to build OpenSSL from source. One must download the source before one can build the source. How does that happen? Easy, use 'curl' or 'wget' or some similar tool, point to openssl.org, get the tarball. Explode the downloaded tarball and follow the standard recipe. But if you don't have SSL working then you can't use HTTPS.

So far my case doesn't sound like a problem. Here's the problem. Some bright "HTTPS everywhere" aficionado decided that nobody should be using HTTP. When you hit openssl.org via HTTP you (now) get re-directed to HTTPS. If you're downloading OpenSSL for the sake of implementing OpenSSL ... and you don't already have some kind of SSL or TLS ... this is a difficult situation.

They broke stuff. They broke my stuff. Now it's personal.

Building systems from source is important. Or I don't know, maybe it's not important. (Seems like it's important to some people, but they're getting hard to find.) I build from source for several reasons, partly because I'm a control freak, partly because I'm a tinker, and partly because I don't trust systems built by other people. Trust ... it's always about trust. (Systems built by other people: I do use them, but kind of like Google. I use Google but I don't trust them. Been saying that for several years now. And look, here I am blogging on a Google property.)

The automation in question actually has SSL. The problem is that it has an embryonic infrastructure with an empty PKI trust store. This is not to say that it doesn't have a solid trust chain. It just doesn't (at the point of fetching OpenSSL source) have a cache of root certificates for the World Wide Web. So when we hit a site like openssl.org (via HTTPS) the server certificate fails to verify. (Plain HTTP is fine, was fine, until mister "my solution works for everyone" did the re-direct re-design on their site.)

Gimme back HTTP!

It's the re-direction that's the problem.
I said HTTP because I meant HTTP. The protocol has 301 and 302. Oy vey, another great feature now fallen victim to abuse by ill-conceived implementation. (There's a long and growing list of those.) The files didn't move. (Would be nice if some people used 301 to replace 404, ya think?)

This is the second time in recent months that I've run up against someone having disabled a perfectly reasonable function because they knew better than the rest of us. I guess we gotta kill stupidity one bad idea at a time.

Let's encrypt. Let's encrypt widely. Let's encrypt carefully.


-- R; <><

Off-site Backup

Off-site Backup

I've been pressed for time. Scheduling one thing to finish when another is due to start has become essential. On this particular evening, I was hungry, but needed to walk the dog. So I checked the time required for my pizza: 18 to 21 minutes in the conventional oven. Perfect! I could let this cook while exercising the canine.

As I left the house, it occurred to me that I was leaving the hot oven unattended: wife was visiting family, son was on campus, daughter was at work. The risk was small, but it's the kind of risk that we avoid. I was reminded of what happened to Bdale Garbee a year or three ago.

The Garbee family lost everything. Their home was consumed by the same Colorado fire the rest of us heard about on the nightly news. See the video. Bdale's account is enlightening.

For me, this was just a ten minute mental exercise, but here's how it went. All that was most precious was off-premises: family in distributed locations, dog with me. What remained, except for a few heirlooms, was replaceable. But the data? What if we happened to lose the data? Maybe we depend too much on computers to hold our "data".

Until maybe three months ago, I did have off-site backup. Probably seems like overkill for a residential "data center", but I'm a hobbyist. We hobbyists do things for fun that others do only for pay. Dad was more than happy to let me park a surplus desktop-turned-server at his house. A little Linux, shiny new SATA drive, some SixXS to avoid NAT, and a touch of RSYNC. Voi-la! Instant off-site backup.

That was when he had a house. Now he's in an apartment. The facility doesn't provide wired internet; everything is WiFi. And the hack I had rigged to piggy-back that server off of his Windows box began to fail. (First to go was an old 8-port hub in the Rube Goldberg scheme I concocted.) I retrieved the machine several weeks before my lonesome pizza and puppy party.

Surplus hardware is great! You get extended life (from an investment someone made, if not you yourself) and you get low cost service for all kinds of things. Here I had Fedora with LVM and a decent sized platter stack. It was more than just remote storage; it was also a remote point-of-presence. (Helps for those when-not-if times that something funky is happening with the web. And Netflix can just chill because the bandwidth is way to low for regional masking.)

The point of the post: think about a surplus box built to your own specifications as a means to have your own off-site backup or similar service.

But my off-site machine, and its spinning rust, was back home now. Any catastrophe which might wipe out my primary systems would do just as much to my spare. Scary!

When pooch and I got back from our walk, the house was not in flames. There was no smoke nor out-of-control cookery. Instead I was greeted by the aroma of a nearly finished Red Baron Supreme with thin crust. Yesss!! But I am reminded that I need to get serious about the remote box (or maybe two?) and re-deploy real soon now.

-- R; <><

Going Dark

Going Dark


Two scary stories made the GMA headlines Thursday morning. The first breaking news was about the coincidental glitches yesterday (knocking down WSJ, halting NYSE, and grounding Continental/United). People, it really could be just a coincidence . (More on that later.) The second nail biter concerned FBI director James Comey's warning: crypto cripples canvassing. I often wonder why I take my cues from ABC, and you would be reasonable to ask "why?". But this was a real story; here's why.


It's called "going dark" when two parties (good or bad) utilize end-to-end encryption. Directory Comey wants a way around it for cases where one of the parties deserves surveillance. A lot of people in my world are doing a facepalm. This is old news. It's the same thing we've heard from the gubmint before. Computer security professionals call it "exceptional access" where a police agency (anyone really) can bypass encryption, somewhat like a traditional wiretap.


When Crypto is Outlawed


It's cliche, but you get the idea. When encryption is outlawed, only outlaws will have encryption. This debate is not new. Techies have been fighting in support of cryptography for decades. If you're not a computer geek, you probably did not know. (If you're not a computer geek, you probably don't *want* to know.)


Today's chilling warning is frustrating: I kind of like Director Comey, but his testimony before congress appears to be without proof.
 

"Comey didn't offer any evidence regarding FBI investigations thwarted by strong encryption technology. Still, he said, he felt like researchers and tech companies haven't given the government's request a fair shot before dismissing it.  ..."
-- Yahoo news 

I'd like to introduce Mr. Comey to Lynn Wheeler, Bruce Schneier, Jacob Appelbaum, ... and other computer experts. Yes, James, they have given the government's request a lot of thought, and "dismissed" is the wrong word to describe their response. (I know Wheeler personally. Schneier and Appelbaum have high profile reputations in the crypto world, as do Diffie, Rivest, and other contributors to the MIT paper. These people are the best and brightest on this topic.)


When Guns are Outlawed


There's a similar debate about gun control. I have many friends and family who are in the "right to carry" camp. Guns are dangerous, but lots of things are dangerous. My pro-carry peeps get it that the bad guys will have guns. It's better to arm (and train!) the good guys.


Substitute "cryptography" for "guns", or substitute "privacy" there. When guns/crypto/privacy is outlawed only outlaws will have guns/crypto/privacy. The same argument applies (and has yet to be countered) that the bad guys will still have and use guns and crypto even after the law abiding public are disarmed.


The difference (for purpose of analogy) between guns and encryption is that guns are physical and cryptography is mathematics. Any back-door means of bypassing crypto is like an invisible hand on (radio control of) the safety on your Glock. Doesn't matter if the manufacturer swears they'll only ever give that control to legit law enforcement, the mechanism itself is a detriment. Experience has proven time and again that exceptional access leads to unthinkable flaws in the resulting products.



Breaking and Entering


The matter is very personal! I may be damaging my professional reputation by revealing this: I've been hit. It happened a long time ago, and I've learned a lot since then. Someone broke in to one of my computers. Was I surprised, scared? Heck yeah! So I did some forensics and I'm confident about the extent of the breach and have worked hard to keep things secure going forward.


Do I want strong encryption? You betcha!!

But a back door? No way! The front door is enough trouble to guard, thank you.


The FBI wants back doors in the apps and systems (phones). By analogy, these apps and smart phones are akin to a house with cheap siding, too many windows, and no Tyvek. They don't even need the barrel of a tank turret to pierce the wall, just kick it. In other words, the infrastructure surrounding two end-to-end encrypted devices is fully exposed and easily watched.



Good Morning America


It was a GMA headline that jolted me awake. (Coffee had not kicked in yet.) Whether ABC, CBS, NBC, Fox, CNN, ... attention grabbing headlines sell. There's no way "you" are going to get the straight scoop about crypto tech from a major news outlet. It's not in their best interest to give you the full fair-and-balanced story. (Not even Fox, much as they tout that slogan.) If you want to really understand, you may have to do some journalistic gymnastics of your own.


Balanced against the consumerism of ABC is an article on Huffington Post. Not my favorite rag by a long shot (and the author supports "IP enforcement" raising an eyebrow), but truth is truth and Ms Espinel got this one right: back doors weaken defenses.


Good Grief America!


Being a software guy, with decades of experience in encryption, I knew the GMA story told only the most seductive fraction. Director Comey would do better to mute some of the conversation he wants to start. It's not in our best interest to be arguing details in the news media where terrorists can watch.


The fact is, even when a bad guy scrambles the message, he still leaves tracks. Maybe my homey Comey should pal-up with the NSA. Whether they can un-scramble the eggs or not, they're the first people he should ask how-to. The exposed infrastructure (mentioned above) is leaking info all over the place. Google knows it. Apple knows it. The NSA and CIA know it. Chances are the FBI knows it too.


Everyone (not just FBI): quit looking for short-cuts.
Going dark does not make the parties invisible, just opaque. 
Arm the law abiding majority, including the cops, and get busy with actual police work to stop the criminals.


-- R; <><

Bad Idea #8 - BTRFS Subvols

Bad Idea #8 - BTRFS Subvols

Got your attention? 

If you like Btrfs then I expect so. 

But take heart: the problem here is not subvolumes. This is not a slam on subvols nor on Btrfs itself. The problem is subvolume abuse. Hopefully you'll keep reading and not click away.

 

Like so many other things, Btrfs subvols are more good than bad, but side effects can have unexpected results, unintended consequences. Here's a quick story of one case of a particularly annoying side effect. 

 

I'm embracing Btrfs for a lot of reasons, so I chose it as the root filesystem type when doing a recent Linux installation. Snapshot capability was also selected by default. I knew about that feature but did not know about some of its consequences. Then when signing on to the shiny new system, I found more than a dozen subvols mounted in support of snapshot. 

This broke my RSYNC plan. 

 

My usual habit is to install the new system in a virtual machine, then copy ... er, uh ... take a snapshot (ha!). The copy gets put into a dedicated logical volume. I then copy the new kernel and initrd to a common boot partition used by all systems. This works really well: I get a variety of distros and releases and yet can 'zypper' or 'yum' as needed within each. 

 

This time, there were Btrfs subvolumes. The subvols showed as individual mount points, so the "-x" flag did the right thing by not descending across mounts. We wouldn't want to snag /proc or /sys when snapping that copy of the new op sys, so we really need that "-x". But we do want the normal files, all of them. We don't want to have to construct a myriad 'rsync' invocations to make it work.

 

The system I installed, not knowing to de-select snapshots, wound up with 14 subvols. It looks like a case of some developer trying to do with subvols what we all used to do with LVM. (Scaled up just a bit since that number of "volumes" is beyond what anyone bothered with before we had subvols.) Now ... the layering violation (subvols versus LVM) is a whole nutha topic. But you can see from this count: 15 invocations of 'rsync' versus just one.

Perhaps just a nuisance?
Fairly, though, a nuisance with questionable requirement.

 

These "bad idea" rants sound horribly whiny. There is an upside. Reader, please understand, it's really about KISS. In this particular complaint, subvolume abuse goes against the grain of simpler filesystem handling. 

 

Where has simplicity gone? 

We in the Linux and open source world love to pick on Windows, yet we're now creating the same complicated crap, monolithic mechanisms with unforeseen flaws. 

 

-- R; <><

KVM Across the Board

KVM Across the Board

Let's talk about hardware.
Let's talk about physical hardware versus emulated hardware.
Let's talk about virtual hardware.

Emulation

Emulation is where you run a program to simulate a machine. A popular PC emulator is BOCHS. You can use BOCHS to boot any operating system which would run on I386 hardware. BOCHS can run on any hardware, I386 or other. BOCHS can run on z Series (IBM mainframe). BOCHS can run on p Series (IBM RS/6K). BOCHS can run on ARM (your cell phone).

There are many emulators for many architectures.
Mainframe aficionados commonly use Hercules to emulate a mainframe where BOCHS emulates a PC.

There is a steep performance penalty in emulation. Each emulated CPU instruction requires dozens or hundreds of physical CPU instructions. Processors this century are so fast that we sometimes don't notice. (The pain of this performance penalty varies widely for varying workloads.) 

Virtualization

Virtualization is better. 

Virtualization is hard to implement but easy to understand. A "guest" runs on the underlying physical machine until there is an exception (privileged operation, I/O, a page fault, or similar). Emulation does come into play (because not all devices and hardware facilities are virtualized). But virtualization is different from emulation in a big way. With virtualization, the guest runs on the bare metal most of the time, not interpreted by a program. Clearly more efficient.

Something which hosts guest operating systems via virtualization is called a hypervisor. The premier hypervisor is z/VM. z/VM only runs on IBM z Series hardware.

Let's talk about KVM

Since VMware made its debut in the late 1990s, virtualization has been more than just a mainframe phenomenon. Until that time, the only serious hypervisor was IBM's VM (variously VM/ESA, VM/XA, VM/SP, and other monickers, now z/VM). Before VMware, the best we got (in quantity) apart from z/VM was emulation. But now there are many.

We cannot know (unless we measure each, which is time consuming) how much true virtualzation each hypervisor really does. There is the concept of paravirtualization, which is beyond the scope of one little blogger post. Let's talk about that L8R.

An incomplete list (just for discussion) of hypervisors includes z/VM, VMware, Xen, KVM, and products from Sun/Oracle, Microsoft, Apple, and other vendors.

But let's talk about KVM.

KVM started life in the PC world, specifically the Linux PC arena. We had Xen, but it was paravirt at best. Advances in chip logic from AMD and INTeL allowed for more effective virtualization.So Qumranet developed KVM and managed to get it accepted into the Linux kernel. Thus it is the kernel virtual machine facility. Nice!

I got wind that KVM was being made to work on z Systems. (Being a late bloomer, the info was after-the-fact.) I also found out that KVM had been ported to p Systems. (That's the PowerPC chip.)

Now ... this is a really cool development: the same hypervisor across multiple hardware architectures. I have imagined this and longed for it for two decades! Wow. Can't do that with VMware. Can't do that with Oracle, Apple, Microsoft. Can't even do that with IBM z/VM. 

Let's talk about IBM

This little post is in response to a thread on LinkedIn. Specifically, I'm trying to answer Jon Butler's question if KVM is enabled for the mainframe hardware. The answer is yes.

The mainframe is amazing and has lots of bells and whistles. At the center is the same basic idea of a general purpose processor (variously called a CP or GP, with a Linux-specific spin called IFL, loosely "S390"). z/VM runs on it. KVM runs on it. 

KVM runs on: I386 (and X86_64), PPC, S390, and (soon) ARM

But then there's z/VM. 

IBM has this incredible hypervisor. I don't have time to describe the orders of magnitude difference between z/VM and the others.

z/VM runs on: S390

There is ample historical evidence that IBM has wanted to kill z/VM for a long long time. Maybe it threatens their cash cow (z/OS, nee MVS). 

Many respected colleagues have expressed frustration with IBM in recent months over positioning of their shiny new z13 machine. A z13 can run hundreds, nay thousands, of Linux guests without breaking a sweat. It can over-commit memory and still deliver flawless service with sub-second response time. It can do all this by the magic of z/VM.

So what's wrong?

What's wrong is that IBM marketing hints the amazing z13 does all this with the help of KVM. That's simply not true. It's like the marketing guys don't know the difference between "z/" and "K". 

z/VM is not all that expensive (to you). Might be expensive to IBM.
z/VM is, however, very different from KVM. No doubt some customers with pre-disposed warm fuzzies for KVM will embrace that and expect the results touted for z/VM. Not gonna happen.

What Shall We Do?

I love z/VM. There's nothing like it. It takes full advantage of z hardware and then adds its own incredible features.

I love KVM. It breaks out across hardware architectures. But it's a baby, just starting to crawl.

My fear is that IBM will finally lop-off their VM product (like some say they've wanted to do for decades) and leave us with only KVM.

So what shall we do?

First, be clear about the story. If an IBMer rattles on about z13 and only includes KVM, raise your hand. Call them on the carpet about it. Second, if they do proceed to ditch z/VM (which experienced engineers believe would be a major mistake), then as customers demand that they port all the features of z/VM to KVM: the monitor stream, the performance controls, DCSS, IUCV, the unquantifyable* features of CP/CMS.

*Not sure "unquantifyable" is a word, but you get the idea. 

-- R; <><

Bad Idea #75 - Mandatory Indentation

Bad Idea #75 - Mandatory Indentation

I've done it!
I've finally learned Python.
Could not put it off any longer and what a cool language! 

Found out about the origin of the name. Guessing Guido wanted the Holy Grail of languages. I'm also a fan of the comedy troupe, and MP/HG is one of my favorite films, so you might expect me to love the language just for that. But no. Had to continue with my bad attitude.

Anyone notice that the Dutch are taking over the computing world? And in case GvR ever runs across this, I have been to Haarlem. Beautiful town. 

I didn't like Python.
It's an aesthetics thing.
(I dislike Perl for the same reason, resulting in similar self injury.) 

Thou Shalt Bathe

Unlike my friend JEP, I have no allergy to curly braces, so for me there's no need for indentation (as far as making the compiler/interpreter happy). But I do indent. Always have. C'mon ... that's the most basic coding hygiene. (I bathe too.)

What offends me about Python is that indentation is part of the language, that indentation is mandatory. (Somewhere in the history of 'make' is a similar soggy saga w/r/t indentation. I will cite another time.)

Look ... I indent as a matter of course, so when I refrain from indenting there's a reason for it. But with Python I don't have that liberty. The language forces me to indent like a parent telling their kid to brush their teeth.

A Pearl of a Language

Now that I've dived in, this water's fine. Really. 

Writing new code is easy as shell. Yet it is much better structured (than shell) so the results are less of a hack and more like a Mack (truck). It has a strong collection of libraries, including a spiffy OS interface, and supports "the right" set of data types.

Took less than a day to port one of my Tcl scripts to Python. Easy! And fun!

Yet, like any gem, it does have flaws. 

Maybe "flaws" is too strong of a word. Call them "qualities". 

I've been so apologetic about REXX over the years. Python has similar quirks. (And MFC gets a pass on quirks because REXX was created in a vacuum before the glut of scripting languages we now carry.) Some aspects of Python take a bit of getting used to ... in a way that reminds me of REXX.

I already lost one job offer due to lack of Perl skills. 

Hopefully I won't lose any others from lack of Python prowess. 

Now looking forward to more Python. 

-- R; <><

Kitchen Code

Kitchen Code - how to *not* cook software

Ah, the joy of writing a new program from scratch. Well, mostly from scratch. We use bits and pieces of other programs and mix it all together, season it just right. Poof ... perfection. That's what programming is all about, right?

Hardly.

Real programming means you're going to come back to a program later. Maybe you need to add a feature. Maybe you need to squish a bug. And maybe it's not as easy as the first time in that stew. This after-the-first-release kind of programming is not as exhilarating as "green field" work.

Other hands in the Soup

As if re-working your own code wasn't bad enough, turns out you sometimes have to work on someone else's creation. This can actually be a good thing. This can actually be fun, believe it or not. (You've surely heard about the wonder of Open Source Software. Lots o team work there.)

Coding as part of a team takes discipline. It takes more time at first. (But often takes less time in the fix-up and fancy-up phase.) It's more rewarding.

Working on multi-author code is better ... as long as all authors agree on standards. There are two dangers, problems, with team code that I call "Kitchen Code". Oh how I wish we could all be free from them.

A pot of Spaghetti

Spaghetti Code is the perfect term for a program which jumps around. If you've been into programming for any decent length of time, you have surely heard the phrase. You have probably even seen some spaghetti code. You may have even written some. You'll be forgiven. Just don't do it again!

I love lower level languages. Maybe that's why I love C. Some people call C a "glorified assembler". To me that'd be a compliment. But even C provides strong entry/return fencing. Get into real assembly language and there's real temptation.

Assembler provides the most efficient coding possible. It's also tedious. I don't know if it's the temptation to optimize or that coders tire of the tedium, but it's really common to ... heck ... just branch already! So efficient. So easy. But don't.

No! You must be strong padawan!

Dry-Erase board covered with Post-It Notes*

Noodle cooks have another tendency: disorganized data.

On the other wall from the stove with that spaghetti pot is the white board. There are sticky notes on it. The notes have little bits of information someone needs.

Notes are great. Scratchpads are awesome. But be sure to boil off the excess liquid and get something solid. By "solid" I don't mean one goopy blob of common storage. Organize it. Scope it. Don't expose the whole beefy lot of bits and bytes to every routine in your casserole.

Large structs with global exposure are like rum in the skillet. Someone gets burned.

But *do* use *real* Post-It notes (er, uh ... comments) to explain what you were thinking. They're like margin notes in your recipe book. Do it.

So remember: when it comes to software, avoid spaghetti and contain the data.

-- R; <><

*Post-It is a registered trade mark of 3M. But we like 3M and we trust they'll appreciate the free advertising and not give us crap for using their TM here.

FuzixOS: Because Small Is Beautiful

FuzixOS: Because Small Is Beautiful

Linux kernel developer Alan Cox has come up with an excellent new toy. In recent months, he has reported working with several older hardware platforms ... just for fun. And now, he has published FuzixOS on Github.

In the open source world, I've been noticing a loss of simplicity lately. Fuzix is one alternative to the bloat. It's a collection of features from UZI and various Unix for 8-bit systems. Yes, in terms of anti-bloat, this is extreme. But it's cool!

Can't help but think about NORD in this context. The prime motivator for NORD is the need for simplicity. If Fuzix gets a little more robust, maybe some of the FOSS packages in NORD can be tried on Fuzix. ("Portable Apps" will require sym-links, but the CSCRATCH parts will likely work straight away.) It might even be possible to use the Fuzix kernel for a "real" NORD build.


-- R; <><

Impending Internet

Impending Internet

I heard a respected software engineer say something like, "I saw the advent of the web, and that turned out to be right. I saw Linux on virtualization, and that turned out to be right.". Personally I felt a little vindicated because I had seen the same things. But I was frustrated by his comment because this guy also does not see some impending changes of our landscape.

Impending ... as if something is yet to be.
Case in point, IPv6. I've been pitching IPv6 for the platforms I support for several years. But a number of my peers just don't like it. They don't want to "do" IPv6 until they have to. I gotta think that the procrastination is from ignorance: I suspect that they think IPv6 is difficult. It's not.

IPv6 is not impending. It's now. "It's soooo yesterday!"

As for me, I hate being forced into something, even if it's something that I like, something cool or fun. I got into IPv6 with help from the Orc almost four years ago. Had been trying on my own to crack into IPv6 for the better part of a decade. In the end, I got by with a little help from my friend.

Then today's news, not specifically about IPv6, but tangentially related. I heard about a Chinese government hack against SSL. The targeted network is their CERNET education and research network, which it turns out is entirely IPv6 (CERNET2, at least). Here's my chance to say it: told ya. The Asian tech sector is way ahead of Europe and the Americas on this front.

Most of the major shops have a complete IPv6 rollout underway, many nearly complete. So ... there's plenty of examples of getting this done. C'mon, try with a little help from your friends.




-- R; <><

Proposing Powerful Portable Pipelines

Proposing Powerful Portable Pipelines

IBM calls the package CMS/TSO Pipelines. A certain group of us affectionately call it "Hartmann Pipes". It's standard equipment on VM/CMS, but a lot of people upgrade to the author's version. On TSO (part of z/OS), it's optional. Those who learn even the simplest of its capabilities tend to get really hooked on it.

When you're excited about something, you talk about it. But discussing Hartmann Pipelines with the uninitiated is difficult. Most people have seen command line pipes (Unix, Linux, even Windows) so they wonder "what's the big deal?". How is this implementation different from other pipelining schemes? Aye, there's the rub. I'll get into that detail later. And some readers will need that rationale before any of this makes sense. First ... the proposal.

There are two audiences. There are those who don't know CMS Pipelines. I gotta sell you on this wonderful thing. There are those who do know CMS Pipelines. I gotta sell you on this particular implementation. It includes a compromise; I gotta sell you on the compromise. Maybe from the two groups there will be enough volunteers to achieve critical mass.

The Proposal

There have been several ports of pipelines. Prior efforts require one or more substantial support environments. The idea here is to use pure C. Interaction between the stages is summarily handled by these functions ...

• output(p,d,l) /* blocks until record is consumed */
• peekto(p,b,l) /* examines a record without consuming */
• readto(p,b,l) /* consumes a record (and gets contents) */

output() is used by a producer stage. The producer is blocked until the record is consumed. p is a pipe struct pointer, referencing a connection, in this case an output connector. d points to the data, but it's not necessarily a NULL terminated character string. There are no sacred characters. l says how long the record is.

peekto() and readto() are used by a consumer stage. They're identical except that peekto() leaves the record pending, and leaves the producer stage blocked. p is a similar pipe struct pointer to that used for output(), except that it references an input connector. b points to a buffer and l indicates its size.

These are easily implemented on any POSIX system using a pair of traditional pipes. We'll have one Unix style pipe pointed "downstream" sending data and stats, and another pointed "upstream" sending control signals. Some of the control signals are ...

• tell me about the record
• give me the data, used by both peekto() and readto()
• unblock, "got it", producer can move along, used by readto()

The producer accepts these controls via the upstream pipe and sends meta data or data on via downstream pipe. The producer blocks until it gets a "got it" signal from the consumer. The consumer blocks until the producer has a record to send.

The "connector" struct starts with two file descriptors ...

struct PIPECONN {
    int  ctrl;    /* control from consumer to producer */
    int  data;    /* data from producer to consumer */
    ... other elements ...
        }

The data pipe alternately carries metadata based on signals on the ctrl pipe.





The Compromise

The above operations are under control of the host operating system. Some time back, I mentioned this idea to the Piper (Hartmann himself). His take on it was not clear. CMS Pipelines has its own dispatcher. (CMS originally did not have built-in multi-tasking.) I've mentioned the idea to several serious plumbers. They unanimously reject it, citing Hartmann's dispatcher as an essential requirement. But is it?

About the time I first published this article, David Craig shared his excursion into Plan 9 (that's a whole nutha journal entry) including discussion about Plan 9 security. I found this ...

"... weak but easy-to-use security can be more effective
than strong but difficult-to-use security if it is more likely to be used."

The point goes for more than just security. A weak (or not well tuned) but easy-to-use dispatcher can be more effective if it leads to a simpler implementation. So that's to the Pipes knowing audience. Let each stage be a unique process.

Now a word to the uninitiated.


A Record by Any Other Name

Hartmann Pipes convey records. "Eww ... such a mainframe concept." Well, yes, in fact. John Hartmann invented Pipelines on a mainframe and CMS/TSO Pipes runs on two mainframe operating systems.

Why do we fight over these things? If it were called a packet or a message or perhaps a block then you wouldn't have a problem with it. I won't enumerate the reasons for using quantized data (records) leaving examples such as MQ and TCP as ample justification.

The value of the Hartmann implementation includes ...

• record structure (if any) is retained
• "delaying the record" can be tolerated or avoided, as needed
• an entire record can be examined before it is consumed, which allows for in-flight changes to the pipeline based on content
• a stage can be added, to input or output, between two connected stages
• multiple streams can run concurrently, and fan-in or fan-out
• output can be looped back to input

We could really use this on systems other than only CMS and TSO.


-- R: <><


Originally published 2014-July-13. 

Data Sharing versus Data Security

Data Sharing versus Data Security

Two of my favorite topics that seem to contradict each other are the sharing of data and the securing of it (especially crypto). I'll be pitching both at the upcoming VM and Linux Workshop next month at NC A&T in Greensboro. What's really amazing at this time is how hot both topics are and yet how poorly understood they are.

The VM and Linux Workshop is an highly technical and surprisingly inexpensive conference. This year there will be the added value of Velocity Software's performance seminar two days prior to the workshop (at the same location).

Data Sharing

The glut of mobile computing and popularity "cloud computing" illuminates the need for data sharing ... even amplifies it. For just one example, think about your list-o-contacts on your smart phone. Handy, eh? But not so handy if you have to go into contortions to synchronize the info with other parts of your digital life. Gotta have data synch, and that's data sharing.

Full disclosure: the presentation will specifically cover filesystem sharing, though we'll touch on data synch and other such content manglement. Networked filesystems were more common until recently. In a context where applications use files, network sharing of filesystems makes a lot of sense. But even where apps use methods other than files for their working data, they need files for basic operation. And here's the kicker: most of the operating system needs files. So ... sharing of filesystems makes a lot of sense in more contexts than you might think.

Sharing filesystems as block devices or memory is cooler: It is efficient, effective, even elegant. Sharing virtual disk or virtual ROM is obvious to some but still not practiced as widely as it should be. And sharing disk images works in SAN land as well as in the virtual world. "It's everywhere you want to be."

Data Security

We're talking crypto, but not just crypto. (Controlling access is crucial.)

I mentioned smart phone and contacts. Data sec balances data sharing so that your contacts remain yours.

The presentation will cover a mere handful of the data access issues. (You could have a week long seminar discussing all related topics!) For the focus topic of SSL, it's vital to protect the server's secret key.I'll get into how that works, why its important, and related tools (PGP/GPG and SSH and others). Ciphers and hashes and keys ... oh my!

Truth be told, PKI is a mess, but it gives us a framework, a reference. The heavy lifting is done by SSL (and TLS) which is really slick, well defined, and robust. But the underlying cryptography is a moving target. As soon as you learn one cipher or hash, you find it has been defeated. So keeping things plug-compatible is essential. 

Both of these topics border on obsession for me. They're so geeky cool. More people should use them. Even where they're not applicable, they should be considered ... part of your "doctor's bag" of remedies and treatments. I'm excited to be talking at the Workshop again.

-- R; <><

Sir Santa Seen in Super Set Script

Sir Santa Seen in Super Set Script

or

Fun with Unicode

We suspect this is what Sir Santa really looks like. He even has a Unicode point of his own!

But he's not much of a fan of Android, perhaps since they have "issues".