KVM Across the Board

KVM Across the Board

Let's talk about hardware.
Let's talk about physical hardware versus emulated hardware.
Let's talk about virtual hardware.

Emulation

Emulation is where you run a program to simulate a machine. A popular PC emulator is BOCHS. You can use BOCHS to boot any operating system which would run on I386 hardware. BOCHS can run on any hardware, I386 or other. BOCHS can run on z Series (IBM mainframe). BOCHS can run on p Series (IBM RS/6K). BOCHS can run on ARM (your cell phone).

There are many emulators for many architectures.
Mainframe aficionados commonly use Hercules to emulate a mainframe where BOCHS emulates a PC.

There is a steep performance penalty in emulation. Each emulated CPU instruction requires dozens or hundreds of physical CPU instructions. Processors this century are so fast that we sometimes don't notice. (The pain of this performance penalty varies widely for varying workloads.) 

Virtualization

Virtualization is better. 

Virtualization is hard to implement but easy to understand. A "guest" runs on the underlying physical machine until there is an exception (privileged operation, I/O, a page fault, or similar). Emulation does come into play (because not all devices and hardware facilities are virtualized). But virtualization is different from emulation in a big way. With virtualization, the guest runs on the bare metal most of the time, not interpreted by a program. Clearly more efficient.

Something which hosts guest operating systems via virtualization is called a hypervisor. The premier hypervisor is z/VM. z/VM only runs on IBM z Series hardware.

Let's talk about KVM

Since VMware made its debut in the late 1990s, virtualization has been more than just a mainframe phenomenon. Until that time, the only serious hypervisor was IBM's VM (variously VM/ESA, VM/XA, VM/SP, and other monickers, now z/VM). Before VMware, the best we got (in quantity) apart from z/VM was emulation. But now there are many.

We cannot know (unless we measure each, which is time consuming) how much true virtualzation each hypervisor really does. There is the concept of paravirtualization, which is beyond the scope of one little blogger post. Let's talk about that L8R.

An incomplete list (just for discussion) of hypervisors includes z/VM, VMware, Xen, KVM, and products from Sun/Oracle, Microsoft, Apple, and other vendors.

But let's talk about KVM.

KVM started life in the PC world, specifically the Linux PC arena. We had Xen, but it was paravirt at best. Advances in chip logic from AMD and INTeL allowed for more effective virtualization.So Qumranet developed KVM and managed to get it accepted into the Linux kernel. Thus it is the kernel virtual machine facility. Nice!

I got wind that KVM was being made to work on z Systems. (Being a late bloomer, the info was after-the-fact.) I also found out that KVM had been ported to p Systems. (That's the PowerPC chip.)

Now ... this is a really cool development: the same hypervisor across multiple hardware architectures. I have imagined this and longed for it for two decades! Wow. Can't do that with VMware. Can't do that with Oracle, Apple, Microsoft. Can't even do that with IBM z/VM. 

Let's talk about IBM

This little post is in response to a thread on LinkedIn. Specifically, I'm trying to answer Jon Butler's question if KVM is enabled for the mainframe hardware. The answer is yes.

The mainframe is amazing and has lots of bells and whistles. At the center is the same basic idea of a general purpose processor (variously called a CP or GP, with a Linux-specific spin called IFL, loosely "S390"). z/VM runs on it. KVM runs on it. 

KVM runs on: I386 (and X86_64), PPC, S390, and (soon) ARM

But then there's z/VM. 

IBM has this incredible hypervisor. I don't have time to describe the orders of magnitude difference between z/VM and the others.

z/VM runs on: S390

There is ample historical evidence that IBM has wanted to kill z/VM for a long long time. Maybe it threatens their cash cow (z/OS, nee MVS). 

Many respected colleagues have expressed frustration with IBM in recent months over positioning of their shiny new z13 machine. A z13 can run hundreds, nay thousands, of Linux guests without breaking a sweat. It can over-commit memory and still deliver flawless service with sub-second response time. It can do all this by the magic of z/VM.

So what's wrong?

What's wrong is that IBM marketing hints the amazing z13 does all this with the help of KVM. That's simply not true. It's like the marketing guys don't know the difference between "z/" and "K". 

z/VM is not all that expensive (to you). Might be expensive to IBM.
z/VM is, however, very different from KVM. No doubt some customers with pre-disposed warm fuzzies for KVM will embrace that and expect the results touted for z/VM. Not gonna happen.

What Shall We Do?

I love z/VM. There's nothing like it. It takes full advantage of z hardware and then adds its own incredible features.

I love KVM. It breaks out across hardware architectures. But it's a baby, just starting to crawl.

My fear is that IBM will finally lop-off their VM product (like some say they've wanted to do for decades) and leave us with only KVM.

So what shall we do?

First, be clear about the story. If an IBMer rattles on about z13 and only includes KVM, raise your hand. Call them on the carpet about it. Second, if they do proceed to ditch z/VM (which experienced engineers believe would be a major mistake), then as customers demand that they port all the features of z/VM to KVM: the monitor stream, the performance controls, DCSS, IUCV, the unquantifyable* features of CP/CMS.

*Not sure "unquantifyable" is a word, but you get the idea. 

-- R; <><

Bad Idea #75 - Mandatory Indentation

Bad Idea #75 - Mandatory Indentation

I've done it!
I've finally learned Python.
Could not put it off any longer and what a cool language! 

Found out about the origin of the name. Guessing Guido wanted the Holy Grail of languages. I'm also a fan of the comedy troupe, and MP/HG is one of my favorite films, so you might expect me to love the language just for that. But no. Had to continue with my bad attitude.

Anyone notice that the Dutch are taking over the computing world? And in case GvR ever runs across this, I have been to Haarlem. Beautiful town. 

I didn't like Python.
It's an aesthetics thing.
(I dislike Perl for the same reason, resulting in similar self injury.) 

Thou Shalt Bathe

Unlike my friend JEP, I have no allergy to curly braces, so for me there's no need for indentation (as far as making the compiler/interpreter happy). But I do indent. Always have. C'mon ... that's the most basic coding hygiene. (I bathe too.)

What offends me about Python is that indentation is part of the language, that indentation is mandatory. (Somewhere in the history of 'make' is a similar soggy saga w/r/t indentation. I will cite another time.)

Look ... I indent as a matter of course, so when I refrain from indenting there's a reason for it. But with Python I don't have that liberty. The language forces me to indent like a parent telling their kid to brush their teeth.

A Pearl of a Language

Now that I've dived in, this water's fine. Really. 

Writing new code is easy as shell. Yet it is much better structured (than shell) so the results are less of a hack and more like a Mack (truck). It has a strong collection of libraries, including a spiffy OS interface, and supports "the right" set of data types.

Took less than a day to port one of my Tcl scripts to Python. Easy! And fun!

Yet, like any gem, it does have flaws. 

Maybe "flaws" is too strong of a word. Call them "qualities". 

I've been so apologetic about REXX over the years. Python has similar quirks. (And MFC gets a pass on quirks because REXX was created in a vacuum before the glut of scripting languages we now carry.) Some aspects of Python take a bit of getting used to ... in a way that reminds me of REXX.

I already lost one job offer due to lack of Perl skills. 

Hopefully I won't lose any others from lack of Python prowess. 

Now looking forward to more Python. 

-- R; <><

Kitchen Code

Kitchen Code - how to *not* cook software

Ah, the joy of writing a new program from scratch. Well, mostly from scratch. We use bits and pieces of other programs and mix it all together, season it just right. Poof ... perfection. That's what programming is all about, right?

Hardly.

Real programming means you're going to come back to a program later. Maybe you need to add a feature. Maybe you need to squish a bug. And maybe it's not as easy as the first time in that stew. This after-the-first-release kind of programming is not as exhilarating as "green field" work.

Other hands in the Soup

As if re-working your own code wasn't bad enough, turns out you sometimes have to work on someone else's creation. This can actually be a good thing. This can actually be fun, believe it or not. (You've surely heard about the wonder of Open Source Software. Lots o team work there.)

Coding as part of a team takes discipline. It takes more time at first. (But often takes less time in the fix-up and fancy-up phase.) It's more rewarding.

Working on multi-author code is better ... as long as all authors agree on standards. There are two dangers, problems, with team code that I call "Kitchen Code". Oh how I wish we could all be free from them.

A pot of Spaghetti

Spaghetti Code is the perfect term for a program which jumps around. If you've been into programming for any decent length of time, you have surely heard the phrase. You have probably even seen some spaghetti code. You may have even written some. You'll be forgiven. Just don't do it again!

I love lower level languages. Maybe that's why I love C. Some people call C a "glorified assembler". To me that'd be a compliment. But even C provides strong entry/return fencing. Get into real assembly language and there's real temptation.

Assembler provides the most efficient coding possible. It's also tedious. I don't know if it's the temptation to optimize or that coders tire of the tedium, but it's really common to ... heck ... just branch already! So efficient. So easy. But don't.

No! You must be strong padawan!

Dry-Erase board covered with Post-It Notes*

Noodle cooks have another tendency: disorganized data.

On the other wall from the stove with that spaghetti pot is the white board. There are sticky notes on it. The notes have little bits of information someone needs.

Notes are great. Scratchpads are awesome. But be sure to boil off the excess liquid and get something solid. By "solid" I don't mean one goopy blob of common storage. Organize it. Scope it. Don't expose the whole beefy lot of bits and bytes to every routine in your casserole.

Large structs with global exposure are like rum in the skillet. Someone gets burned.

But *do* use *real* Post-It notes (er, uh ... comments) to explain what you were thinking. They're like margin notes in your recipe book. Do it.

So remember: when it comes to software, avoid spaghetti and contain the data.

-- R; <><

*Post-It is a registered trade mark of 3M. But we like 3M and we trust they'll appreciate the free advertising and not give us crap for using their TM here.

FuzixOS: Because Small Is Beautiful

FuzixOS: Because Small Is Beautiful

Linux kernel developer Alan Cox has come up with an excellent new toy. In recent months, he has reported working with several older hardware platforms ... just for fun. And now, he has published FuzixOS on Github.

In the open source world, I've been noticing a loss of simplicity lately. Fuzix is one alternative to the bloat. It's a collection of features from UZI and various Unix for 8-bit systems. Yes, in terms of anti-bloat, this is extreme. But it's cool!

Can't help but think about NORD in this context. The prime motivator for NORD is the need for simplicity. If Fuzix gets a little more robust, maybe some of the FOSS packages in NORD can be tried on Fuzix. ("Portable Apps" will require sym-links, but the CSCRATCH parts will likely work straight away.) It might even be possible to use the Fuzix kernel for a "real" NORD build.


-- R; <><

Impending Internet

Impending Internet

I heard a respected software engineer say something like, "I saw the advent of the web, and that turned out to be right. I saw Linux on virtualization, and that turned out to be right.". Personally I felt a little vindicated because I had seen the same things. But I was frustrated by his comment because this guy also does not see some impending changes of our landscape.

Impending ... as if something is yet to be.
Case in point, IPv6. I've been pitching IPv6 for the platforms I support for several years. But a number of my peers just don't like it. They don't want to "do" IPv6 until they have to. I gotta think that the procrastination is from ignorance: I suspect that they think IPv6 is difficult. It's not.

IPv6 is not impending. It's now. "It's soooo yesterday!"

As for me, I hate being forced into something, even if it's something that I like, something cool or fun. I got into IPv6 with help from the Orc almost four years ago. Had been trying on my own to crack into IPv6 for the better part of a decade. In the end, I got by with a little help from my friend.

Then today's news, not specifically about IPv6, but tangentially related. I heard about a Chinese government hack against SSL. The targeted network is their CERNET education and research network, which it turns out is entirely IPv6 (CERNET2, at least). Here's my chance to say it: told ya. The Asian tech sector is way ahead of Europe and the Americas on this front.

Most of the major shops have a complete IPv6 rollout underway, many nearly complete. So ... there's plenty of examples of getting this done. C'mon, try with a little help from your friends.




-- R; <><

Proposing Powerful Portable Pipelines

Proposing Powerful Portable Pipelines

IBM calls the package CMS/TSO Pipelines. A certain group of us affectionately call it "Hartmann Pipes". It's standard equipment on VM/CMS, but a lot of people upgrade to the author's version. On TSO (part of z/OS), it's optional. Those who learn even the simplest of its capabilities tend to get really hooked on it.

When you're excited about something, you talk about it. But discussing Hartmann Pipelines with the uninitiated is difficult. Most people have seen command line pipes (Unix, Linux, even Windows) so they wonder "what's the big deal?". How is this implementation different from other pipelining schemes? Aye, there's the rub. I'll get into that detail later. And some readers will need that rationale before any of this makes sense. First ... the proposal.

There are two audiences. There are those who don't know CMS Pipelines. I gotta sell you on this wonderful thing. There are those who do know CMS Pipelines. I gotta sell you on this particular implementation. It includes a compromise; I gotta sell you on the compromise. Maybe from the two groups there will be enough volunteers to achieve critical mass.

The Proposal

There have been several ports of pipelines. Prior efforts require one or more substantial support environments. The idea here is to use pure C. Interaction between the stages is summarily handled by these functions ...

• output(p,d,l) /* blocks until record is consumed */
• peekto(p,b,l) /* examines a record without consuming */
• readto(p,b,l) /* consumes a record (and gets contents) */

output() is used by a producer stage. The producer is blocked until the record is consumed. p is a pipe struct pointer, referencing a connection, in this case an output connector. d points to the data, but it's not necessarily a NULL terminated character string. There are no sacred characters. l says how long the record is.

peekto() and readto() are used by a consumer stage. They're identical except that peekto() leaves the record pending, and leaves the producer stage blocked. p is a similar pipe struct pointer to that used for output(), except that it references an input connector. b points to a buffer and l indicates its size.

These are easily implemented on any POSIX system using a pair of traditional pipes. We'll have one Unix style pipe pointed "downstream" sending data and stats, and another pointed "upstream" sending control signals. Some of the control signals are ...

• tell me about the record
• give me the data, used by both peekto() and readto()
• unblock, "got it", producer can move along, used by readto()

The producer accepts these controls via the upstream pipe and sends meta data or data on via downstream pipe. The producer blocks until it gets a "got it" signal from the consumer. The consumer blocks until the producer has a record to send.

The "connector" struct starts with two file descriptors ...

struct PIPECONN {
    int  ctrl;    /* control from consumer to producer */
    int  data;    /* data from producer to consumer */
    ... other elements ...
        }

The data pipe alternately carries metadata based on signals on the ctrl pipe.





The Compromise

The above operations are under control of the host operating system. Some time back, I mentioned this idea to the Piper (Hartmann himself). His take on it was not clear. CMS Pipelines has its own dispatcher. (CMS originally did not have built-in multi-tasking.) I've mentioned the idea to several serious plumbers. They unanimously reject it, citing Hartmann's dispatcher as an essential requirement. But is it?

About the time I first published this article, David Craig shared his excursion into Plan 9 (that's a whole nutha journal entry) including discussion about Plan 9 security. I found this ...

"... weak but easy-to-use security can be more effective
than strong but difficult-to-use security if it is more likely to be used."

The point goes for more than just security. A weak (or not well tuned) but easy-to-use dispatcher can be more effective if it leads to a simpler implementation. So that's to the Pipes knowing audience. Let each stage be a unique process.

Now a word to the uninitiated.


A Record by Any Other Name

Hartmann Pipes convey records. "Eww ... such a mainframe concept." Well, yes, in fact. John Hartmann invented Pipelines on a mainframe and CMS/TSO Pipes runs on two mainframe operating systems.

Why do we fight over these things? If it were called a packet or a message or perhaps a block then you wouldn't have a problem with it. I won't enumerate the reasons for using quantized data (records) leaving examples such as MQ and TCP as ample justification.

The value of the Hartmann implementation includes ...

• record structure (if any) is retained
• "delaying the record" can be tolerated or avoided, as needed
• an entire record can be examined before it is consumed, which allows for in-flight changes to the pipeline based on content
• a stage can be added, to input or output, between two connected stages
• multiple streams can run concurrently, and fan-in or fan-out
• output can be looped back to input

We could really use this on systems other than only CMS and TSO.


-- R: <><


Originally published 2014-July-13. 

Data Sharing versus Data Security

Data Sharing versus Data Security

Two of my favorite topics that seem to contradict each other are the sharing of data and the securing of it (especially crypto). I'll be pitching both at the upcoming VM and Linux Workshop next month at NC A&T in Greensboro. What's really amazing at this time is how hot both topics are and yet how poorly understood they are.

The VM and Linux Workshop is an highly technical and surprisingly inexpensive conference. This year there will be the added value of Velocity Software's performance seminar two days prior to the workshop (at the same location).

Data Sharing

The glut of mobile computing and popularity "cloud computing" illuminates the need for data sharing ... even amplifies it. For just one example, think about your list-o-contacts on your smart phone. Handy, eh? But not so handy if you have to go into contortions to synchronize the info with other parts of your digital life. Gotta have data synch, and that's data sharing.

Full disclosure: the presentation will specifically cover filesystem sharing, though we'll touch on data synch and other such content manglement. Networked filesystems were more common until recently. In a context where applications use files, network sharing of filesystems makes a lot of sense. But even where apps use methods other than files for their working data, they need files for basic operation. And here's the kicker: most of the operating system needs files. So ... sharing of filesystems makes a lot of sense in more contexts than you might think.

Sharing filesystems as block devices or memory is cooler: It is efficient, effective, even elegant. Sharing virtual disk or virtual ROM is obvious to some but still not practiced as widely as it should be. And sharing disk images works in SAN land as well as in the virtual world. "It's everywhere you want to be."

Data Security

We're talking crypto, but not just crypto. (Controlling access is crucial.)

I mentioned smart phone and contacts. Data sec balances data sharing so that your contacts remain yours.

The presentation will cover a mere handful of the data access issues. (You could have a week long seminar discussing all related topics!) For the focus topic of SSL, it's vital to protect the server's secret key.I'll get into how that works, why its important, and related tools (PGP/GPG and SSH and others). Ciphers and hashes and keys ... oh my!

Truth be told, PKI is a mess, but it gives us a framework, a reference. The heavy lifting is done by SSL (and TLS) which is really slick, well defined, and robust. But the underlying cryptography is a moving target. As soon as you learn one cipher or hash, you find it has been defeated. So keeping things plug-compatible is essential. 

Both of these topics border on obsession for me. They're so geeky cool. More people should use them. Even where they're not applicable, they should be considered ... part of your "doctor's bag" of remedies and treatments. I'm excited to be talking at the Workshop again.

-- R; <><