Going Dark

Going Dark


Two scary stories made the GMA headlines Thursday morning. The first breaking news was about the coincidental glitches yesterday (knocking down WSJ, halting NYSE, and grounding Continental/United). People, it really could be just a coincidence . (More on that later.) The second nail biter concerned FBI director James Comey's warning: crypto cripples canvassing. I often wonder why I take my cues from ABC, and you would be reasonable to ask "why?". But this was a real story; here's why.


It's called "going dark" when two parties (good or bad) utilize end-to-end encryption. Directory Comey wants a way around it for cases where one of the parties deserves surveillance. A lot of people in my world are doing a facepalm. This is old news. It's the same thing we've heard from the gubmint before. Computer security professionals call it "exceptional access" where a police agency (anyone really) can bypass encryption, somewhat like a traditional wiretap.


When Crypto is Outlawed


It's cliche, but you get the idea. When encryption is outlawed, only outlaws will have encryption. This debate is not new. Techies have been fighting in support of cryptography for decades. If you're not a computer geek, you probably did not know. (If you're not a computer geek, you probably don't *want* to know.)


Today's chilling warning is frustrating: I kind of like Director Comey, but his testimony before congress appears to be without proof.
 

"Comey didn't offer any evidence regarding FBI investigations thwarted by strong encryption technology. Still, he said, he felt like researchers and tech companies haven't given the government's request a fair shot before dismissing it.  ..."
-- Yahoo news 

I'd like to introduce Mr. Comey to Lynn Wheeler, Bruce Schneier, Jacob Appelbaum, ... and other computer experts. Yes, James, they have given the government's request a lot of thought, and "dismissed" is the wrong word to describe their response. (I know Wheeler personally. Schneier and Appelbaum have high profile reputations in the crypto world, as do Diffie, Rivest, and other contributors to the MIT paper. These people are the best and brightest on this topic.)


When Guns are Outlawed


There's a similar debate about gun control. I have many friends and family who are in the "right to carry" camp. Guns are dangerous, but lots of things are dangerous. My pro-carry peeps get it that the bad guys will have guns. It's better to arm (and train!) the good guys.


Substitute "cryptography" for "guns", or substitute "privacy" there. When guns/crypto/privacy is outlawed only outlaws will have guns/crypto/privacy. The same argument applies (and has yet to be countered) that the bad guys will still have and use guns and crypto even after the law abiding public are disarmed.


The difference (for purpose of analogy) between guns and encryption is that guns are physical and cryptography is mathematics. Any back-door means of bypassing crypto is like an invisible hand on (radio control of) the safety on your Glock. Doesn't matter if the manufacturer swears they'll only ever give that control to legit law enforcement, the mechanism itself is a detriment. Experience has proven time and again that exceptional access leads to unthinkable flaws in the resulting products.



Breaking and Entering


The matter is very personal! I may be damaging my professional reputation by revealing this: I've been hit. It happened a long time ago, and I've learned a lot since then. Someone broke in to one of my computers. Was I surprised, scared? Heck yeah! So I did some forensics and I'm confident about the extent of the breach and have worked hard to keep things secure going forward.


Do I want strong encryption? You betcha!!

But a back door? No way! The front door is enough trouble to guard, thank you.


The FBI wants back doors in the apps and systems (phones). By analogy, these apps and smart phones are akin to a house with cheap siding, too many windows, and no Tyvek. They don't even need the barrel of a tank turret to pierce the wall, just kick it. In other words, the infrastructure surrounding two end-to-end encrypted devices is fully exposed and easily watched.



Good Morning America


It was a GMA headline that jolted me awake. (Coffee had not kicked in yet.) Whether ABC, CBS, NBC, Fox, CNN, ... attention grabbing headlines sell. There's no way "you" are going to get the straight scoop about crypto tech from a major news outlet. It's not in their best interest to give you the full fair-and-balanced story. (Not even Fox, much as they tout that slogan.) If you want to really understand, you may have to do some journalistic gymnastics of your own.


Balanced against the consumerism of ABC is an article on Huffington Post. Not my favorite rag by a long shot (and the author supports "IP enforcement" raising an eyebrow), but truth is truth and Ms Espinel got this one right: back doors weaken defenses.


Good Grief America!


Being a software guy, with decades of experience in encryption, I knew the GMA story told only the most seductive fraction. Director Comey would do better to mute some of the conversation he wants to start. It's not in our best interest to be arguing details in the news media where terrorists can watch.


The fact is, even when a bad guy scrambles the message, he still leaves tracks. Maybe my homey Comey should pal-up with the NSA. Whether they can un-scramble the eggs or not, they're the first people he should ask how-to. The exposed infrastructure (mentioned above) is leaking info all over the place. Google knows it. Apple knows it. The NSA and CIA know it. Chances are the FBI knows it too.


Everyone (not just FBI): quit looking for short-cuts.
Going dark does not make the parties invisible, just opaque. 
Arm the law abiding majority, including the cops, and get busy with actual police work to stop the criminals.


-- R; <><

Bad Idea #8 - BTRFS Subvols

Bad Idea #8 - BTRFS Subvols

Got your attention? 

If you like Btrfs then I expect so. 

But take heart: the problem here is not subvolumes. This is not a slam on subvols nor on Btrfs itself. The problem is subvolume abuse. Hopefully you'll keep reading and not click away.

 

Like so many other things, Btrfs subvols are more good than bad, but side effects can have unexpected results, unintended consequences. Here's a quick story of one case of a particularly annoying side effect. 

 

I'm embracing Btrfs for a lot of reasons, so I chose it as the root filesystem type when doing a recent Linux installation. Snapshot capability was also selected by default. I knew about that feature but did not know about some of its consequences. Then when signing on to the shiny new system, I found more than a dozen subvols mounted in support of snapshot. 

This broke my RSYNC plan. 

 

My usual habit is to install the new system in a virtual machine, then copy ... er, uh ... take a snapshot (ha!). The copy gets put into a dedicated logical volume. I then copy the new kernel and initrd to a common boot partition used by all systems. This works really well: I get a variety of distros and releases and yet can 'zypper' or 'yum' as needed within each. 

 

This time, there were Btrfs subvolumes. The subvols showed as individual mount points, so the "-x" flag did the right thing by not descending across mounts. We wouldn't want to snag /proc or /sys when snapping that copy of the new op sys, so we really need that "-x". But we do want the normal files, all of them. We don't want to have to construct a myriad 'rsync' invocations to make it work.

 

The system I installed, not knowing to de-select snapshots, wound up with 14 subvols. It looks like a case of some developer trying to do with subvols what we all used to do with LVM. (Scaled up just a bit since that number of "volumes" is beyond what anyone bothered with before we had subvols.) Now ... the layering violation (subvols versus LVM) is a whole nutha topic. But you can see from this count: 15 invocations of 'rsync' versus just one.

Perhaps just a nuisance?
Fairly, though, a nuisance with questionable requirement.

 

These "bad idea" rants sound horribly whiny. There is an upside. Reader, please understand, it's really about KISS. In this particular complaint, subvolume abuse goes against the grain of simpler filesystem handling. 

 

Where has simplicity gone? 

We in the Linux and open source world love to pick on Windows, yet we're now creating the same complicated crap, monolithic mechanisms with unforeseen flaws. 

 

-- R; <><

KVM Across the Board

KVM Across the Board

Let's talk about hardware.
Let's talk about physical hardware versus emulated hardware.
Let's talk about virtual hardware.

Emulation

Emulation is where you run a program to simulate a machine. A popular PC emulator is BOCHS. You can use BOCHS to boot any operating system which would run on I386 hardware. BOCHS can run on any hardware, I386 or other. BOCHS can run on z Series (IBM mainframe). BOCHS can run on p Series (IBM RS/6K). BOCHS can run on ARM (your cell phone).

There are many emulators for many architectures.
Mainframe aficionados commonly use Hercules to emulate a mainframe where BOCHS emulates a PC.

There is a steep performance penalty in emulation. Each emulated CPU instruction requires dozens or hundreds of physical CPU instructions. Processors this century are so fast that we sometimes don't notice. (The pain of this performance penalty varies widely for varying workloads.) 

Virtualization

Virtualization is better. 

Virtualization is hard to implement but easy to understand. A "guest" runs on the underlying physical machine until there is an exception (privileged operation, I/O, a page fault, or similar). Emulation does come into play (because not all devices and hardware facilities are virtualized). But virtualization is different from emulation in a big way. With virtualization, the guest runs on the bare metal most of the time, not interpreted by a program. Clearly more efficient.

Something which hosts guest operating systems via virtualization is called a hypervisor. The premier hypervisor is z/VM. z/VM only runs on IBM z Series hardware.

Let's talk about KVM

Since VMware made its debut in the late 1990s, virtualization has been more than just a mainframe phenomenon. Until that time, the only serious hypervisor was IBM's VM (variously VM/ESA, VM/XA, VM/SP, and other monickers, now z/VM). Before VMware, the best we got (in quantity) apart from z/VM was emulation. But now there are many.

We cannot know (unless we measure each, which is time consuming) how much true virtualzation each hypervisor really does. There is the concept of paravirtualization, which is beyond the scope of one little blogger post. Let's talk about that L8R.

An incomplete list (just for discussion) of hypervisors includes z/VM, VMware, Xen, KVM, and products from Sun/Oracle, Microsoft, Apple, and other vendors.

But let's talk about KVM.

KVM started life in the PC world, specifically the Linux PC arena. We had Xen, but it was paravirt at best. Advances in chip logic from AMD and INTeL allowed for more effective virtualization.So Qumranet developed KVM and managed to get it accepted into the Linux kernel. Thus it is the kernel virtual machine facility. Nice!

I got wind that KVM was being made to work on z Systems. (Being a late bloomer, the info was after-the-fact.) I also found out that KVM had been ported to p Systems. (That's the PowerPC chip.)

Now ... this is a really cool development: the same hypervisor across multiple hardware architectures. I have imagined this and longed for it for two decades! Wow. Can't do that with VMware. Can't do that with Oracle, Apple, Microsoft. Can't even do that with IBM z/VM. 

Let's talk about IBM

This little post is in response to a thread on LinkedIn. Specifically, I'm trying to answer Jon Butler's question if KVM is enabled for the mainframe hardware. The answer is yes.

The mainframe is amazing and has lots of bells and whistles. At the center is the same basic idea of a general purpose processor (variously called a CP or GP, with a Linux-specific spin called IFL, loosely "S390"). z/VM runs on it. KVM runs on it. 

KVM runs on: I386 (and X86_64), PPC, S390, and (soon) ARM

But then there's z/VM. 

IBM has this incredible hypervisor. I don't have time to describe the orders of magnitude difference between z/VM and the others.

z/VM runs on: S390

There is ample historical evidence that IBM has wanted to kill z/VM for a long long time. Maybe it threatens their cash cow (z/OS, nee MVS). 

Many respected colleagues have expressed frustration with IBM in recent months over positioning of their shiny new z13 machine. A z13 can run hundreds, nay thousands, of Linux guests without breaking a sweat. It can over-commit memory and still deliver flawless service with sub-second response time. It can do all this by the magic of z/VM.

So what's wrong?

What's wrong is that IBM marketing hints the amazing z13 does all this with the help of KVM. That's simply not true. It's like the marketing guys don't know the difference between "z/" and "K". 

z/VM is not all that expensive (to you). Might be expensive to IBM.
z/VM is, however, very different from KVM. No doubt some customers with pre-disposed warm fuzzies for KVM will embrace that and expect the results touted for z/VM. Not gonna happen.

What Shall We Do?

I love z/VM. There's nothing like it. It takes full advantage of z hardware and then adds its own incredible features.

I love KVM. It breaks out across hardware architectures. But it's a baby, just starting to crawl.

My fear is that IBM will finally lop-off their VM product (like some say they've wanted to do for decades) and leave us with only KVM.

So what shall we do?

First, be clear about the story. If an IBMer rattles on about z13 and only includes KVM, raise your hand. Call them on the carpet about it. Second, if they do proceed to ditch z/VM (which experienced engineers believe would be a major mistake), then as customers demand that they port all the features of z/VM to KVM: the monitor stream, the performance controls, DCSS, IUCV, the unquantifyable* features of CP/CMS.

*Not sure "unquantifyable" is a word, but you get the idea. 

-- R; <><

Bad Idea #75 - Mandatory Indentation

Bad Idea #75 - Mandatory Indentation

I've done it!
I've finally learned Python.
Could not put it off any longer and what a cool language! 

Found out about the origin of the name. Guessing Guido wanted the Holy Grail of languages. I'm also a fan of the comedy troupe, and MP/HG is one of my favorite films, so you might expect me to love the language just for that. But no. Had to continue with my bad attitude.

Anyone notice that the Dutch are taking over the computing world? And in case GvR ever runs across this, I have been to Haarlem. Beautiful town. 

I didn't like Python.
It's an aesthetics thing.
(I dislike Perl for the same reason, resulting in similar self injury.) 

Thou Shalt Bathe

Unlike my friend JEP, I have no allergy to curly braces, so for me there's no need for indentation (as far as making the compiler/interpreter happy). But I do indent. Always have. C'mon ... that's the most basic coding hygiene. (I bathe too.)

What offends me about Python is that indentation is part of the language, that indentation is mandatory. (Somewhere in the history of 'make' is a similar soggy saga w/r/t indentation. I will cite another time.)

Look ... I indent as a matter of course, so when I refrain from indenting there's a reason for it. But with Python I don't have that liberty. The language forces me to indent like a parent telling their kid to brush their teeth.

A Pearl of a Language

Now that I've dived in, this water's fine. Really. 

Writing new code is easy as shell. Yet it is much better structured (than shell) so the results are less of a hack and more like a Mack (truck). It has a strong collection of libraries, including a spiffy OS interface, and supports "the right" set of data types.

Took less than a day to port one of my Tcl scripts to Python. Easy! And fun!

Yet, like any gem, it does have flaws. 

Maybe "flaws" is too strong of a word. Call them "qualities". 

I've been so apologetic about REXX over the years. Python has similar quirks. (And MFC gets a pass on quirks because REXX was created in a vacuum before the glut of scripting languages we now carry.) Some aspects of Python take a bit of getting used to ... in a way that reminds me of REXX.

I already lost one job offer due to lack of Perl skills. 

Hopefully I won't lose any others from lack of Python prowess. 

Now looking forward to more Python. 

-- R; <><

Kitchen Code

Kitchen Code - how to *not* cook software

Ah, the joy of writing a new program from scratch. Well, mostly from scratch. We use bits and pieces of other programs and mix it all together, season it just right. Poof ... perfection. That's what programming is all about, right?

Hardly.

Real programming means you're going to come back to a program later. Maybe you need to add a feature. Maybe you need to squish a bug. And maybe it's not as easy as the first time in that stew. This after-the-first-release kind of programming is not as exhilarating as "green field" work.

Other hands in the Soup

As if re-working your own code wasn't bad enough, turns out you sometimes have to work on someone else's creation. This can actually be a good thing. This can actually be fun, believe it or not. (You've surely heard about the wonder of Open Source Software. Lots o team work there.)

Coding as part of a team takes discipline. It takes more time at first. (But often takes less time in the fix-up and fancy-up phase.) It's more rewarding.

Working on multi-author code is better ... as long as all authors agree on standards. There are two dangers, problems, with team code that I call "Kitchen Code". Oh how I wish we could all be free from them.

A pot of Spaghetti

Spaghetti Code is the perfect term for a program which jumps around. If you've been into programming for any decent length of time, you have surely heard the phrase. You have probably even seen some spaghetti code. You may have even written some. You'll be forgiven. Just don't do it again!

I love lower level languages. Maybe that's why I love C. Some people call C a "glorified assembler". To me that'd be a compliment. But even C provides strong entry/return fencing. Get into real assembly language and there's real temptation.

Assembler provides the most efficient coding possible. It's also tedious. I don't know if it's the temptation to optimize or that coders tire of the tedium, but it's really common to ... heck ... just branch already! So efficient. So easy. But don't.

No! You must be strong padawan!

Dry-Erase board covered with Post-It Notes*

Noodle cooks have another tendency: disorganized data.

On the other wall from the stove with that spaghetti pot is the white board. There are sticky notes on it. The notes have little bits of information someone needs.

Notes are great. Scratchpads are awesome. But be sure to boil off the excess liquid and get something solid. By "solid" I don't mean one goopy blob of common storage. Organize it. Scope it. Don't expose the whole beefy lot of bits and bytes to every routine in your casserole.

Large structs with global exposure are like rum in the skillet. Someone gets burned.

But *do* use *real* Post-It notes (er, uh ... comments) to explain what you were thinking. They're like margin notes in your recipe book. Do it.

So remember: when it comes to software, avoid spaghetti and contain the data.

-- R; <><

*Post-It is a registered trade mark of 3M. But we like 3M and we trust they'll appreciate the free advertising and not give us crap for using their TM here.