Shrinking the World and Compressing Time

Yesterday I wrote about the VM Community. The backbone of that community has been electronic communication. Now, the fortunate folks out there who never knew a world before the internet, blogs, RSS, etc. may be thinking 'boooring'. I'd ask them to think about how long this has been around and the ground breaking that was involved.

The first electronic conferencing or forums that I was aware of was VMSHARE. David N. Smith of TYMSHARE introduced it at a SHARE conference in 1976. VMSHARE was one of the earliest electronic conferences. Dave wrote the software for this and convinced his management to supply the system and network. One of the driving forces behind it was the desire for greater community and discussing ideas in between conferences. Later that decade, in 1979, VMSHARE was extended to Europe. You can still view the archives.

Here is my first entry to the VMSAHRE world, the NEWGUY FORUM:

 I just got lost for 20 minutes reading some of the old appends. :-) The SHARE community would give me a few new nicknames.

Over time, various electronic conferences and bulletin boards would become available as networks increased and expanded. Internally at IBM, conferencing or "forums" would become common place and was based on powerful software running on VM. This would be used for all sorts of topics and communication within IBM. One of the forums that I recall with a heavy heart, was one that was spontaneously created on January 28, 1986 to share information on the space shuttle Challenger disaster. At that time, IBM Federal Systems was still part of IBM and had a role in the space shuttle program.

Another software product that would appear was LISTSERV from L-Soft. Under Eric Thomas' leadership and vision, it would become a formal product in 1994. While today it runs on a variety of platforms, it started and continues to run on VM. The mailing lists managed by LISTSERV would replace VMSHARE with various lists supporting TCP/IP, Pipelines, VM, and of course Linux on System z. VMESA-L was a popular list during the hay days of VM/ESA. To avoid future name changes, the list changed to IBMVM after z/VM came along. If you aren't on these lists, check out the VM Home Page for information on them.

Perhaps I'm not objective enough, but one of the things I've noticed, and others confirm, is that the VM related discussion groups tend to be friendlier than your typical newsgroups. Flaming appends are very rare, as is trolling. The group as a whole does a great job of policing itself. Newcomers are made to feel welcome and encouraged to join the discussions. The few times I can recall that a moderator got involved in the past few years was to remind us the discussion was getting a bit silly and off topic.

The IBMVM list is like a lifeline for novices and veterans alike. VM is used around the world. So there is always someone awake who can help me or you.

.BB

The VM Community

The VM Community encompasses a large group of diverse people: IBMers, Vendors, Customers, and Consultants. Within each of those groups, are more subgroups, developers, support, sales, etc. all bound together by the common technology known as VM.

As I mentioned earlier, my view of this community's scope grew over the years. It's amazing to look at how various people and groups have contributed so much to VM and the use of it. This involved contributing code and features, helping one another use the product, and influencing IBM.

Some relative newcomers to VM were surprised lately when I spoke of the OCO (Object Code Only) wars. They were even more surprised when I started describing how much our customers modified and enhanced the product with their own code. And I really floored them when I told them that a number still use modifications. Beyond the modifications are hundreds of user written programs, scripts, tools, white papers, redbooks, and presentations.

While some shops only have a single system programmer, I point out that they're never alone. You have this community backing you. And it's a community with  credentials. We tend to measure experience in decades rather than years. That's not to say there aren't people that are new to VM. We have a number of them as well, and it's a nice blend. As a new system programmer, there will seldom be a time where you'll face a problem that hasn't been faced before. And you'll be able to find someone to help as well.

One of the things that impresses me is that this VM Community crosses borders, parties, and other lines that tend to divide. I'll see system programmers from two different competing corporations exchange ideas. I'll see people agreeing with one another on an electronic forum where I know they voted for different candidates. It's just one of the refreshing things about the VM Community. Of course there is good nature ribbing and joking at times. That's part of being in a family, or so my mother told me, the youngest of four children.

That's not to say there aren't moments of tension. Particularly between IBM and its customers. But like a good community, we work it out and adapt. I mentioned in an earlier post that Paul Van Leer taught me to represent IBM to the customer and the customer to IBM. So there have been a few times where I felt tension from both sides in carrying out that mission statement. I'm thankful those moments have been few.

There are a number of definitions for community. The following one is from the The American Heritage® Science Dictionary. I probably could have opened with this definition. Actually, I could have opened and closed with it.

Community: A group of organisms or populations living and interacting with one another in a particular environment. The organisms in a community affect each other's abundance, distribution, and evolutionary adaptation. Depending on how broadly one views the interaction between organisms, a community can be small and local, as in a pond or tree, or regional or global as in a biome.

Influential People: Joe Tingley

I had various people showing me the ropes when I started with IBM: Greg Gasper, Eric Strom, Richard Shultz, and others. But I want to talk about Joe Tingley, as he had a significant impact on me, in multiple ways. The project where I learned the most with Joe was the introduction of AVS into, I believe VM/SP 6. AVS (APPC/VM VTAM Support) allowed programs utilizing APPC/VM for communication to work across VTAM, an SNA based network. At the time, IP was out there, but in business SNA was king. (Have I reached my quota of acronyms for the day?)

As this was new function, it was more difficult than just doing a comparison measurement. You never know how many measurements one will have to make to validate the line item. But it's usually safe to say that the most difficult measurement is the first one. That's an area where Joe always thrived, the difficult measurements and configurations. It was a challenge he fed off. So the first thing I learned from Joe was to not shy away from challenges.

As I mentioned, sometimes new code doesn't work. While Joe wasn't a programmer, he had a good vision for problem determination. How do you narrow down on the problem. He would encourage me to try to get down to as small a window on the problem: line of code, module, driver change, etc.. This became instilled in me and was beneficial, both for working on customer problems as well as new code. Developers always like you better when you can say "I think the problem is related to this segment of code in HCPxxx" rather than "it don't work good."

Often when working with Joe, I would hear "come on Wild Bill, we need to talk to fill in the blank." (Perhaps we'll explain that nickname another time.) Joe knew a lot of people and we'd go to them directly. I learned the importance of dialogue because often in these problems an email wasn't efficient. Email, in Joe's book, was for giving managers status. This also shortened the path in getting an answer to a problem that was blocking progress. It also helped me as a new person, because people got to meet me. That way I wasn't just a name on the copylist of an email.

The gaining efficiency by using the right communication method was indicative of Joe making the best use of all resources. He was very good at, let's call it scrounging or scavenging machine time, disk space, and other resources. He'd be very creative in that manner. Related to this was his work ethic. If there was 20 minutes left in a machine shot, some people would say, well that's not enough to do a whole lot, I'll try again another day. Joe would use that 20 minutes, and more if the next shift didn't show up on time. He worked hard, and played hard.

Joe would leave VM performance as we were scaled down, but eventually would come back to VM Development Lab doing both system test and performance. Again looking for a challenge, he took on the various cryptographic tests. He didn't call me Wild Bill as much but he still continued to encourage me. Joe would continue to work while fighting cancer. And he fought with an attitude that inspired many for several years.

In May of this year, Joe passed away. I consider him a hero, who is missed by family, friends, and co-workers. Thanks Joe for all you taught me.

Notebook Number 1

Oh boy, are you folks in trouble now. I'm moving offices in the near future and in process of sorting through things. I found my stack of notebooks from over my career. The first thing I noticed is that I think my handwriting was much neater in 1985.

So here's the entry from May 12, 1986, I took the liberty of spelling out a few things that would have been in an odd shorthand:

  Building 8 Raised Floor 4361-5 / 8 Meg
  (let Eric know when finished w/4361-5)
  1. Change uCode and IML
  2. Restore System VMSRES --> 240  VMPK01 --> 241
      VME R2 w/Fortran VMPK02 --> 330  VMPK03 --> 331
  3. RUNFOP FSMIXFOP 20 20 1 182
         061158-065658  Looks Good
  4. Install Read Chain modules, Regen CMS & CP
  5. RUNFOP FSMIXFOP 20 20 1 182
         075653-084153     sleepers
  6. RUNFOP FSMIXFOP 20 20 1 182
        092419-100919  sleepers

This was a good memory for me. As I got a chance to help evaluate some new algorithms on chaining of reads. We might as well go right through this. Building 8 was one of the Glendale Lab buildings. The processor had 8 megabytes of real memory, and was one of the larger configurations at the time. I believe in that time frame, we did benchmarks with the minimum memory requirement of 768KB.

Doing performance runs usually involved dedicated hardware and that meant getting machine shots. In those days, we submitted requests like the week before and someone built a schedule for the following week. This meant planning was important, and there were often deals being made or people looking for spare time. So the second line was a reminder that I was to let Eric know that the machine was free when I finished my run. I don't recall if I had borrowed time from him or he was looking for spare time. Yes, this is the same Eric that would later be my manager.

Step 1 involved changing the microcode. That particular raised floor was shared with Glendale Engineering Processor Lab. Those guys would play with all sorts of things. So to avoid a surprise, we'd often put our own microcode disks in and re-IML (initial machine load or initial microcode load). I remember those were big floppy disks, maybe 8", I'm not sure.

Disk was also at a premium, so we'd often have to restore systems. That's what I was doing in my step 2. What I'd be doing is first a base measurement and then install the new prototype and do a couple measurements. Then we compare the differences. The "RUNFOP" was an exec that did the setup and ran a workload using a tool called FSID (Full Screen Internal Driver). It was a cool tool. It was modifications to CP that allowed it to create simulated virtual machines with hooks in modules like DMKGRF. I believe Galen Miller was the originator. There would later be VM/XA and VM/ESA versions of FSID. For many years, I owned the VM/ESA version. It helped me learn a fair amount about the system as it required mods to 3270 handling as well as the dispatcher and scheduler.

 The parameters meant we'd run 20 userids simulating a CMS workload. So 20 userids in 8MB. The "sleepers" comment was that occasionally some of the simulated userids got out of sync and appeared to sleep or hang for some period of time. It could also mean that there was a problem with the system or the script of commands being executed.

The other thing I noticed from the notes is that the "061158-065658" were timestamps of the measurement. For me to have the measurement period start at 06:11, meant I must have gotten up pretty early. Just the price of working on a great system. :-)

Mail Call

I remember in college checking the mail. We had those little boxes at the mail station in the student union, you know the kind with the little window. I remember getting excited when I you could see some of the light from the other side was blocked, meaning there was some mail in there. It was even better when it was a package slip!! When you had a care package from home, everyone became your best friend.

With my new job at IBM in 1985 was I still happy to see mail in my apartment mailbox. At work we also had 'mail folders' out near the secretarial area. I probably checked that at least twice a day. In those days we still got a lot of paper memos, many with 'buck slips' attached. Those little slips with everyone's name on it and you crossed off your name and bucked it on to the next. Who had someone in their department that was infamous for having the buck slip items get stuck in their mail folder?

Then there was, be still my heart, notes delivered via CMS NOTE command. I could write to people in my own department or around the world via NOTE. The VM systems were interconnected and magic would get them from here to there and back again. We also had tools such as NAMES and RDRLIST to handle contacts and look at the notes others sent you. This was really fun. I mean, this really helped facilitate business.

Due to a tragic error (i.e. user error) some of my earliest mail is no longer available. This is the oldest one in my NOTEBOOK files and triggered a lot of memories:

Later I would be introduced to PROFS (Professional Office System) which would evolve into IBM OfficeVision which ran on a few IBM platforms, but ran best on VM (OV/VM). OV not only included mail and contacts, but also calendar and document facilitation. Oh, and those calendars and documents could be shared and had built in management functions. Hmm, perhaps Lotus and others didn't invent collaboration software after all!

The dependency of corporations on OV/VM was measured by how they reacted when there was a problem. I would see how the "Profs" system being down would go from "the what?" to "How can we get anything done with this running slowly?!!"  Corporations became more and more dependent on this collaboration software. The buck slip started to fade away. The marketing sound bite used in that era was "9 million people log onto OV/VM each day". By today's standards, that may not seem like a lot, but in 1991 that was real penetration in the market place. I believe the largest single VM system in the OV/VM usage was around 20,000 seats. Amazing for that era, and still respectable by today's standards.

The OV/VM team when I worked with them was based out of an IBM lab near Dallas. They were a good team and worked well with the VM team. They cared about performance, and not just speeds and feeds. I remember working with them to implement data collection in the VM monitor data stream.

Besides OV/VM there were a number of other mail related VM tools. Internally within IBM, many people used Mailbox. Outside of IBM, the most well known one would be MAILBOOK or the predecessor, RiceMail. Additionally, z/VM supports SMTP and a very impressive implementation of IMAP.

Over the years, mail and electronic mail became less fun. It became part of "work". The ever increasing snowball rolling down the hill after me. I started capturing the data for the graph shown below a number of years ago for a productivity study of sorts. Can you pick out the months where we had a major deliverable or a critical customer situation? I shouldn't complain about my email load, as I know others get far more than I do.

Email History

I recall a fair amount of gnashing of teeth as we migrated off of OV/VM onto the newly acquired IBM Lotus Notes. The stability of it did improved significantly when it was moved to run on Linux on z/VM. Hooray! My mail is back on z/VM. The numbers above don't include the messages I get from IBMVM and LINUX-390 listserver lists. Those still go to my z/VM userid reader and I don't always consider them work.

Be Our Guest

It's time to talk a little about the guests that have run on VM. I mean what is a host without guests? When I started with VM, it was during the time that the product was more CMS focused, but even then guests were a part of the ecosystem. Earlier blogs mentioned z/VM as a guest, so I won't talk about that too much today.

The first guest I was exposed to was VSE/Systems Package, or simply VSE/SP. Like VM, VSE has a rich history and dates back a few years earlier than VM. VSE/SP was an evolution from DOS/VSE. The product as I knew it was more job or transaction oriented than the interactive CMS. This was real business software. Its use by customers as a guest of VM was very extensive, and there were a lot of programs focused on improving the synergy. I recall a lot of effort on CICS/VM, which was meant to extend the CICS (Customer Information Control System) transaction server into the VM environment as well. Beyond that, there was a lot of work to improve the synergy of VSE and VM. The ability to have VSE and VM handshake on page faults for example. VSE was, and is, an excellent guest.
The VM and VSE interaction was an exciting time. Not only was I exposed to new and interesting technology, but also new people and the Boeblingen Lab. We continue to work with some of those people still on VSE and others on Linux. In particular, I remember Dr. Wolfgang Kraemer, the VSE Performance Leader. I really liked Dr. Kraemer as he was really good at what he did and fun. I still get an involuntary smile whenever I think of him calling me "Billy Boy" with his German accent.

The next guest I would hear about was MVS. It would be described to me as a larger cousin of VSE, offering more function and more scaling. It would later become a component of OS/390, which would evolve into z/OS. Since the introduction of logical partitions to System z, it's become rare for production z/OS workloads to run on VM. However, it is on VM where development of the z/OS system and many of the related software occurs both inside and outside of IBM. The z/OS family has never embraced handshaking technology to the degree VSE did. Part of that is tied to wanting to run the same on VM as it did on bare metal or logical partitions. The height of our involvement with MVS guests may have been in support of Y2K testing. Many customers ramped up their MVS guests, including virtual sysplexes during that time frame. As z/OS Development and z/VM Development often face similar challenges in implementation of new hardware, I've gotten to work with a number of them over the years. Their attention to detail and RAS are impressive.

The 1980s and 1990s also had other guests that I recall. MUSIC (Multi-User System for Interactive Computing aka McGill University System for Interactive Computing) was developed by our friends at McGill University. It was leading edge software that allowed students and professors to create and run their own programs interactively. Another unique guest was MUMPS (Massachusetts General  Hospital Utility Multi-Programming System) . I never ran it myself, but I recall customers who did. Some may classify it as a language environment rather than an operating system, I don't know enough to argue those points. It was used predominantly in the healthcare industry. And then there was also AIX/370, a port of the Unix oriented AIX that today runs on System p.

I must not forget to mention TPF (Transaction Processing Facility) of which the current family is z/TPF. This guest also has a long history going back to the Airlines Control Program (ACP). It's all about fast and efficient transactions. TPF is known for high volume reservation and authorization processing in the travel and finance industries. Like MVS, it is very rare for production TPF workloads to run on z/VM, but there is a lot of testing done on VM. A fascinating aspect of TPF on VM are the vendor additions or extensions to VM, most notably VPARS and VTAPE from VSSI. Which make TPF an even more effective guest.

And then there is Linux. In the late 1990s, I would hear of different discussions of porting it to System z. Then quickly the discussions, turned to rumors, and then reality. Of all the guests, Linux has been the largest game changer. As many of my friends were telling me I should look for another job, that VM was a dead end, there was new life breathed into the VM product. Linux also has the cutest mascot of the guests as well, which doesn't hurt. Linux is more like VSE than some of the other guests. Many of the former VSE team is involved with it in Boeblingen. It's used heavily in production environments. And it has great handshaking with VM.

I joke with a friend that a bed and breakfast might be my next career path. Perhaps the idea intrigues me since I've really enjoyed getting to know the guests staying at Hotel VM.

Pesky End Users, Not

In an earlier blog, there was a comment from James Vincent about being just a 'pesky end user', and I said I'd get back to that. The IBM heritage through the Watson family was customer oriented and embodied by the second of the three original IBM Basic Beliefs:

A desire to have the best customer service of any company in the world.

For the first year or two of my career, customers were like antelope. I had read about them and seen them on television, but never in person. I wouldn't see one until the announce day for VM/SP 5. At that time, it was customary to send programmers from the labs out to branch offices for announce day. I was one of the lucky ones selected, and was sent to Nashville for three days of meetings with customers and IBMers in the field. The customers looked a lot like me, only a little more comfortable in suits. On the last day, a customer jokingly said, "you must be from a development lab, you're wearing a blue shirt." At the time, I was still paying off college loans, so my wardrobe was limited. All the IBMers from the branch office would have fresh white shirts each and every day, so I stood out on that last day. I really should look for my notes from that trip, as I can't remember which customers I visited. I just remember it opened my eyes.

SHARE and other conferences would continue to make an impact on me. Meeting customers and hearing how they actually used our product put a face on both the customer and the business. It was no longer just about implementing a design, supporting an architecture, or adding a new feature. It was about giving our customers what they needed to be productive and successful.

Last year we arranged a couple meetings where customers spoke to the z/VM Development Lab. At first I wasn't sure how many on our team would be able to take time to attend. I was very pleased to see a great turnout for each event. I was even more encouraged to hear some of the feedback as my co-workers got a glimpse of what our customers' lives are like.

One of the most frequent comments was about the awe of realizing the importance of z/VM in a customer enterprise. The idea that millions of transactions, millions of dollars were dependent on z/VM gave them a sense of pride. Hearing of critical monitoring systems running on z/VM gave them a sense of responsibility bordering on duty. While discussing a customer usage of z/VM with one young programmer, he admitted that it made him a little nervous to think of how much people were relying on the code he wrote. I was encouraged, as I knew he got it.

I also heard comments of respect for how much a z/VM system programmer has to know besides z/VM, the various products and systems that make up an overall application solution and business process. The names of some of the products were new to us, let alone how they worked.  We live in our well defined world of z/VM, but system programmers have to journey through different lands of software and vendors.

z/VM system programmers are able to juggle multiple applications and various types of workloads. Their own customers on these z/VM systems aren't simulated workloads, but often people who get vocal and angry when things do not work. They have naming conventions and schedules and change control systems to understand. Most work for corporations in the business of making money. The systems that support or generate revenue are kept running in harmony by these skilled and diligent people.

It's also fun to see how customers use or extend the system in ways we didn't expect. The degree of integration they create is amazing. The example that popped into my head just now is the creative way the NAMES file construct has been used over the years. I've seen customers use it as a mini database, tie it into a disaster recovery solution, or provide a problem tracking system based partly on it. There are also modifications to various components of z/VM, tons of REXX and Pipelines based tools, and exploitation of various user exits.

After hearing and seeing how our customers use z/VM to make the world run, I don't think of them as 'pesky end users'. I think of them more as musicians who take the instrument we craft, and create rich tones from it. It is then that I'm in the audience, listening with pride.

Influential People: Wes Ernsberger

I've worked with a large number of people over the years in the core VM performance team, but I believe Wes Ernsberger is one of the folks that is on top in the longevity list. Wes already had several years in performance when I joined IBM in 1985. That was in the peak of resources, when there were multiple departments committed to VM performance. The headcount for performance decreased significantly after the consolidation of VM from Kingston and Endicott to just Endicott. There would be more reductions during the gold rush also known as the client/server era. The 1990s would see an end to the IBM full employment practice. Reductions and re-balancing of resources in the mid to late 1990s would show the VM performance team drop to a six-pack, and then three: myself, Bill Guzior, and Wes Ernsberger. Fortunately Linux arrived and the team began to grow again, moving towards much healthier numbers.

The era of three was a very tightly-knit team. I remember we had a rule that both Wes and I weren't permitted to vacation at the same time, just so we could ensure coverage for problems. Wes and I would do the bulk of the design evaluation and data analysis. Bill G would keep the systems running and do the bulk of the measurements. VM wasn't strategic at the time, which had downsides. But we were also very free to lead ourselves, determine priorities, and do the most important things. I really enjoyed those years. Wes and Bill are great teammates.

Wes retired a number of years ago now. The first month he was gone, it was very common to hear "I miss Wes." in the hallways. Years later, I still find myself saying it on at least a monthly basis.

Occasionally there is someone in the Development Lab or else where that does something for performance. If it is truly heroic, we bestow upon them official VM Performance Hero Badges. I believe the last set we gave out was for the team that worked through the enhancements to Limit Shares (new SET SRM LIMITHARD settings). When Wes was leaving, I commented that he was beyond being a hero. He was a superhero. The kind that saves the day time and time again, but takes no credit and appears to be the guy next door.

Wes taught me several things, a couple through direct conversation, but mostly through example and being a role model.

A criticism of Wes when it came time to give out awards was often that he hadn't put in as much overtime as others that were up for an award. This was wrong for a couple reasons. Wes was incredible at projecting schedules. We would get a line item to evaluate. He'd look through the basics and start building a plan. He was thorough in this, thinking through the dependencies and factoring in contingencies. Time and time again he'd be very accurate. While others had to deal with the stress of churn in their schedules, Wes was calm and on track. As a result, he very seldom had to work huge amounts of overtime to make up for not planning well.

Related to his use of time, was Wes' work ethic. I have never worked with anyone with as high a true productivity rate. I don't recall Wes ever spending more than 10 or 15 minutes on a distraction such as hanging out at the coffee machine, talking about the football or baseball game from the night before, or just goofing off. It wasn't that Wes was anti-social, he was just focused on work while he was there. Five days a week, one of the first in the office. And then most often you could find him there for a couple hours on Saturday morning.

The idea of planning went beyond just schedules. I'm told that back when runs had to be done by turning over jobs to operators to run or run yourself during a weekly run window, Wes really made sure he had done everything right ahead of time to ensure he got the most out of that run window. Nothing would be worse than turning in a job and getting back the results to find there was a stupid mistake that ruined the run, meaning, you had to wait for the next window to re-run it.

Wes taught me a lot about design analysis. It was fun. Working with design early on to understand the goals and start to create building blocks to understand the potential performance, or lack thereof for a new line item. Through Wes I learned how to break things down into the basic components and build simple spreadsheet models of things. These were great tools for setting and refining expectations.

He really was a great teacher. When the performance team started growing in size again. Wes did a lot of the mentoring and teaching of new people on the team. Patient. He was so patient with everyone. He even built a plan prior to retirement about who would get each piece of his work. He walked everyone through the delegated tasks. I've never seen work transfer done so effectively. I can't thank him enough for that.

Wes had many other fine traits, but I'll end with one I admired the most: he was open to criticism or suggestions. For many of the years that I led the VM Performance Evaluation team, I would write up feedback on each of the team members. I'd provide that feedback to the member and their manager usually just prior to rankings, or "contribution assessment sessions". I felt it was a privilege and an obligation to do this for my team. I remember after doing this one year, Wes stopped over to my office. He thanked me for the positive words, and then asked, "What can I do better? I like knowing I'm doing good things, but knowing what I'm doing wrong is more valuable."   And I knew he was serious. Likewise, he would let me know where I could improve as well. From that, we were a better team. VM was a better product. Earth was a safer planet for computing.  And isn't that what being a superhero is all about?

Correcting Mestakes

Oh, did I misspell mistake? That reminds me. As a relatively new hire, I was looking at some of the workloads we used to evaluate CMS performance. While looking over the commands, I saw one named XXXXX. I scratched my head as I wasn't sure what this command did. I looked around and couldn't find anything on this mysterious command. I finally spoke with one of the more senior members who was an expert in CMS performance, Gary Hine.

Gary smiled, "Ohhh, that's one of the more common commands people use." I looked back at him the same way I used to look back at my older brothers when I thought they were spinning a tale, and asked what it did. Still smiling Gary replied, "Nothing". He then went on to explain that one of the more common commands was a typo which was an invalid command. They had picked this up at some point when studying real users. (Though I know my rate of typos is proportional to how many people are watching me type). I found it interesting that we actually measured the performance of mistakes. It made sense. The quicker it would execute, the quicker the user would realize they made a mistake and correct it.

In System z, it's one of the things we think about a lot (perhaps still not enough, but a lot). That is, what happens when there is a mistake? How obvious is it that there was a problem, and more importantly how obvious is the method to correct things? A few years ago Richard Lewis and Chuck Morse started doing hands-on Labs at SHARE and other conferences. I started attending those sessions to help, but also to learn. I got a whole new perspective seeing a student who made a mistake updating CP Config and trying to sort out what they did wrong. Oh, you typed PAG001 instead of PG0001. The hard part in writing documentation is often clearing your mind so you only know what the reader will know before they start to read the documentation. Yes, not so easy.

The other area where I remember us doing significant performance work in the early days that surprised me was HELP. These days HELP isn't used that often relative to the number of transactions running through a z/VM system. In the past though, it was used quite heavily and we made various improvements over the years. I'm biased, but I still think it's easier in some regards than some other help systems.

For example, the software for this blog. I want to find out... well heck. I can't find any help on this gui based blog editor. So there goes my example. But we've all been there. Lets look at Thunderbird email client instead:

1. Click help 
2. Click help contents
3. Wait for browser to start
4. In search menu, type in 'help set font'
5. Scratch head
6. In search menu, type in 'set font'
7. Look at dozen options provided
8. Pick one
9. Back to thunderbird
10. Click Tools
11. Click Options
12. Click Composition
13. Click General
14. Set fonts
15. See if that did what I wanted

Seems like a lot of clicking.

z/VM HELP has short comings, but it's fairly concise and gives me much of what I need. And the number of steps in navigating has never been so many that I forgot what I needed before I found it.

While assisting in one of the labs I mentioned earlier, I saw a student make a mistake and get an error message. I showed them that they could type HELP message_number and presto. More smiles.

Knowing we all make mistakes, learning from them, and helping others avoid or correct mistakes is something VM has been doing for a long time.

Speaking of help. A shout out to the various people that let me know about typos and other corrections. It is appreciated.

It Depends; and with Linux It Depends Even More

Does anyone know who David Bradley is? He's the one of the designers on the original IBM PC who designed Control+Alt+Delete to trigger a soft reboot. His line that captured his 15 minutes of fame was:

"I may have invented Control-Alt-Delete, but Bill Gates made it famous."

I didn't invent "It Depends". I actually first heard it from Marty Zimelis when he was with Amdahl doing education. I believe it was rule number one on his list of Performance Analyst Rules. (I need to find and post the other ones). But I know some people think I own the royalties on it. When Linux came on the scene, I did add the "with Linux it depends even more."

Now, I know some people consider "It Depends" to be a cop out or a way to avoid answering a question. I don't mean for it to be that. It's really just a preamble for me which translates into:

Be careful, there is not necessarily a single or unqualified answer to your question. I am going to ask you to think about some things and understand the environment and factors that play in that environment. I will be very sad if you just take one aspect of what I say and run off and misuse that information. If my response involves a range of values, you need to know why the answer may lean towards one or the other end of that range.

As you can imagine, it's much easier to say "it depends" than the above. Some of you have caught on and after hearing "it depends", ask what are the dependencies?

At some point in my career, I was faced with the reality that there are a set of people out there that actually listen to what I say. I became more aware of the idea that what I said had weight. I began to be more cautious about giving absolutes, and quicker to point out whether what I was sharing was conjecture, provable, or proven fact. Being clear on the difference of those has become very important to me in problem determination.

I've been in situations with major problems. Then Jimmy will say something in the hallway such as,  "I wonder if it's the network?".  Some listener will repeat it as "Jimmy thinks the problem is the network.", which becomes "Jimmy identified the problem as the network."  Heavy sigh.

I often hear that z/VM or System z is too complex or difficult. And I often think, compared to what? I've never met anyone of average intelligence that had trouble navigating around the CMS environment and editing in a week or two; writing useful REXX scripts in a month; understanding the fundamentals of z/VM system programming in a few months. I hear the 3270 is too difficult. Seriously? It's the same keyboard as every other system, just less mousing. I lose this argument a lot with some people. Oh well.

(I'd really like to see a study on the ratio of time spent typing vs. clicking for the average z/VM system programmer, Linux Admin, and Windows Server Administrator. My conjecture is that the ratios would be closer than some people believe.)

Please don't get me wrong. I'm all for simplification. However, there's a difference between putting a GUI frontend on SET SRM values and simplification. What we want to avoid is eye candy for sake of eye candy; and making it simpler to screw up the system.

I always had trouble putting my finger on why some people seem to have so much trouble with Linux on z/VM. Until one day, Steve Gracin from IBM ATS gave me the answer. "It doesn't matter what system or interface you're using, you have to know what you're doing." And in some cases, we're just not willing to learn that. (See early blog entries about the value of mentors and community). And in other cases, we're not willing to admit we don't understand. Most systems that do significant things are complex and therefore, it's our jobs to understand aspects of the system.

So back to "it depends". Often, that's the start of thinking about what factors come into play. What objectives are there? The beauty of Linux is that it is so flexible; the challenge of Linux is it is so flexible. I work with customers running 256MB virtual machines and virtual machines over 100GBs. What is right for one, may be wrong for the other. And someone needs to think about the difference. Understanding the dependencies of ones system is a key aspect to knowing what you're doing. And if you want a job where thinking isn't required, I suggest systems programming, on any platform, is not for you.

Customers, you should know you're not the only ones I give the "it depends" line. Inside, I do the same thing in planning and design meetings. A new solution or line item is proposed and I know it will have different degrees of benefit for different customers. The answer to "what is the benefit to the customer?" is "it depends". Now, what are those factors. Understanding those, helps us to understand the scope of benefit to customers and the return on the risk. Some of our less successful ventures have been where we built everything and then asked the question.

So I hope you all have a better appreciation for what's behind "it depends".

How a Cat Works

"If you try and take a cat apart to see how it works, the first thing you have on your hands is a non-working cat.” - Douglas Adams

Yesterday, I discussed my love for second-level systems, and how the use of them makes us understand z/VM and gives us a place to experiment and learn. Earlier in this blog, I mentioned the aspect of architecture purity. I'd like to take a step down into an aspect of z/VM that not everyone has the need or time to examine. But it's powerful stuff that allows you to see inside the cat without taking it apart.

DISPLAY, TRACE, and STORE. These three CP commands are the keys to magical places. I didn't learn these reading a book, but again by seeing them used early in my job.

I seem to recall seeing DISPLAY used for both virtual and real to check eye catchers in modules. In Performance we'd often get new code to test and actually looking at it in memory was best way to make sure it was what we thought it was suppose to be.

I remember for some VTAM performance measurements, we'd want to disable some of the timer loops in the VTAM code. I remember being shown how to run a trace to see the addresses of where the code that triggered the loop existed, and thinking that's pretty cool. But then my mentor went on to disable the loop by storing branch into that code. This to me was like milking a cow while it was moving. And a whole new perspective opened up to me.

I would smile as I asked a developer how a particular diagnose or instruction like STSI works. Some, instead of getting out the full Principles of Operation, would just grab a virtual machine, drop into CP, store the opcode in memory perhaps change a register, and begin execution at that instruction. Geeky and yet fashionable.

In the late 1990s and early 2000s it seemed everyone working on VM had been working long enough that everyone knew these things. Linux changed that. A couple years ago a Linux developer from one of the distributions found himself working on System z. At a SHARE conference, he asked me if there were any utilities in z/VM to help with problem determination. He worked on an area of the kernel where the problems were often at a point where the kernel wasn't fully up or that tracing from Linux just wasn't feasible. So at a break between sessions, we sat down and I just started showing him some of the things you can do with Display, Trace, and Store. His eyes lit up. I had just armed him with various methods to solve a myriad of possible problems.

There are many examples. One last one I'll mention was working on the raised floor with Geoff Blandy. He took it a step too far when he started modifying the CP that we were running on at the time, with Store Host. Now I was milking the cow while riding on it; and I really didn't think that was particularly safe. At first it was just one or two instructions. This will be quicker than rebuilding he assured me. I think when we were done, he had put in a few dozen instructions this way while I kept an eye on RTM to see if anything was running. Somehow it all worked; or at least didn't crash. Of course I wasn't thinking of the Display Host when at the end he asked, "Ok, just take what we did and modify the source".

This approach is a bit unorthodox, but then so was Geoff in many ways. I did learn from him, and I do have my share of EXECs that modify code or constants in z/VM. All thanks to the keys of Display, Trace, and Store. A powerful aspect of the product that over the years has contributed not just to z/VM, but z/OS, VSE, Linux on System z, actually every OS that ever ran on it.

Spending Time in the Fort

Growing up we all had moments where we pretended to be a fireman, or policeman, or the Lone Ranger, or Captain Kirk, or a System Programmer with our own z196. They tell me playing is one of the ways we learn. I've found it interesting over the years to watch how different people learn.

I believe VM helps us learn how VM works, as well as System z in general. One of my early assignments was to build my own second-level system. And it's still something I recommend to our new hires no matter which discipline (development, system test, performance).  You learn a lot about how the system is 'put together' in the process. Once you have your second level system, you have a new proving ground or playground.

Most of us in the Development Lab these days are still somewhat focused or specialized. A second level system allows us to explore other aspects. For example, we most often think about a z/VM system that is up and running, and these days they just run and run. So we don't often see initialization or shutdown. Both are important to understand and our second-level systems give us a chance to see and experience that. I've heard that some customers schedule shutdown/IPLs just so there operators get a chance to practice, particularly IPLing the system. Lots of things can be learned or practiced by doing this on second-level system.

Once I had my second-level system, I was able to try almost any command, especially the harmful ones. Seeing is believing. Does cold start really erase all the user spool files? Well, by golly, it does. Good thing I did that second-level. Occasionally, I need to give a customer a procedure to follow. While our documentation is good and I've been around a long time, I still often test it out second level. This way I can match the customer level better, avoid customizations that may be on my production first level, and validate things closer. One time last year when I skipped this, I missed an important aspect of the procedure related to privilege classes.

Beyond this, I found my second-level system gave me a reason to build automation or customization around it. It's interesting to wander around the Lab and see that a lot of people have written their own tooling for their second level systems. Some of it is incredibly elaborate. Using temporary disks or virtual disks in storage for their page space or perhaps also spool space. Various programs to generate loads or start of one-off applications. Just amazing the creativity that occurs in that space.

For those that love VM, their second-level system is a place of fun. At times it makes me think of the fort we built in the neighbors barn. The hours of creativity and fun spent in that fort, and in my second-level system. Spending time in both of them has made me late for dinner more than once.

Influential People: Eric Strom

I first met Eric as I was being introduced to folks in the VM Performance departments (yes departments - at the time there were two in Endicott and multiple ones in Kingston). He immediately busted on me because I went to Pitt and he was a Penn State grad, the two being major rivals at the time. I learned a lot from Eric about the discipline of measurements and analysis. A few years later Eric would move into management and I would be working for him. At the risk of giving him a big head, I consider him to be the most influential manager in my career.

Eric believed in me more than I believed in myself. He was the manager that decided I needed customer exposure and put me in to be a rep to SHARE, which would change me significantly. He's the model of the manager who challenged his people, but didn't set them up for failure. Years later we would joke about me going to SHARE and being so green. I asked him if he had realized how I was not ready to fill the shoes of Paul Van Leer. Eric said he knew it, but he also knew that I didn't realize what I was getting into yet, so it would be okay.

Eric would continue to push me, give me opportunities, but also be in my corner when needed. He helped me improve my communication, introduced me to other technical leaders, and when necessary, tell me to suck it up.

After managing VM Performance, I believe his next job was in the crit sit office (critical customer situations). We would occasionally work together on situations; and he would continue to encourage me. He would move around over the next ten or fifteen years, back into management, back out. Eventually he would find himself in performance again, only this time for z/OS.

I also credit Eric with being the key to IBM embracing the Internet. Now you may be saying "Whoa Bill, now that's a stretch!". Just give me a minute to explain. In the late 1980s and early 1990s there was a very bright young man named David Grossman working in VM Performance. Coming out of Michigan, he was a Unix-weenie at heart, and a very self-motivated, hard worker. He would tell Eric that while he enjoyed the performance work, his heart wasn't really in it. Eric already knew that, but also knew that most managers would kill to have someone so talented. What happened was Dave gave Eric a number of quality years in VM, and then Eric worked hard to find him a great job as one of the IBM liaisons at Cornell University in a type of super computer or theory lab.

Hold on, I'm coming to the part about the internet. Dave was working away at Cornell in 1994 during the winter Olympics (Dave was also a great athlete, often riding bike between Endicott and Ithaca for fun after work). Dave discovered that a web site run by another company was taking the IBM Olympic data and serving it up on their own web site. That launched him into action to get IBM executive attention and ultimately IBM into the e-game. I can't do the story justice, but there was a excellent Harvard Business Review article that describes it all. 

All of that as a result of mutual trust and loyalty between a manager and their employee.

Some of you might recognize the name Strom, as in Earl Strom, long time NBA official. In 1995, Eric gave the Hall of Fame acceptance speech for his father, who had passed away the year before. Eric recently found that someone put a copy of it on YouTube; and I watched it the other night. Eric spoke of his father's love of the game and his respect for the players and others involved in the game.

While I'm a horrible basketball player, I'd like to think Eric coached me some as a software engineer.
Love and respect in one's work is possible for NBA officials, and software engineers.

Love and respect. Trust and Loyalty.

A Boy and his Toys

I always have a few toys in the office. I've been reminded you have to get older, but you don't have to grow up. I like to have fun. But that's not the only reason I keep some of these in my office. You see, I work with some brilliant people, the kind of people that have incredible memories and amazing cognitive skills. Their minds work faster than mine. So when I have a complex topic to discuss with them, I ask them to stop over. And then I make sure some of the toys are out in plain sight. This has worked well over the years for several people. Damian Osisek is my favorite target in this respect. He'll start working on the maze or a puzzle while I quiz him about some aspect of architecture, cache congruency classes, super scalar processing, or SIE implementation details. I liken this to knee-capping a processor to lower the capacity rating. The amazing thing is, Damian will solve puzzles during the discussion. While I own some of the puzzles, I can't solve them even when focused solely on them. I'm fortunate to work with people that need to be knee-capped. Oh, that sounds bad, but you know what I mean.

I appreciate vendor give-aways at conferences and trade-shows, some of my better toys. You might be able to pick out the Sterling Software Yo-Yo in the picture above. Carol Everitt recently found it and gave it to me. It used to light up. We're working on restoring that aspect of it. :)

I also have had some LEGOs in my office from time to time. (No surprise to some of  you.) These have come in handy as well. I once used them to illustrate a point back when there was some discussion about how to rank or appraise individuals, and the whole idea of measuring contribution. I handed a single LEGO brick to the manager and suggested they build something. They caught on quickly that you can't build anything too interesting with a single brick. But a set (team) of bricks is what allows greater things to be built. Last I looked, the manager still had the LEGO on their desk.

The Cornfield Workload

Nostalgia time again. How many remember the old orange-covered Technical Bulletins that the System Centers used to produce? (How many actually remember when we had regional Area System Centers?). Back in those days, the VM Performance report was a collaboration of the VM Performance Evaluation team in the Development Lab and one of the System Centers. We'd pull together the bulk of the data and information and then turn it over for additional writing and editing and formalizing to one or more people in the centers.

Seeing my name on one of the inside pages listing the 'contributors' was awesome. And Mom got a copy that year for Christmas. Having a last name that starts close to the beginning of the alphabet has pros and cons. On one hand, I was one of the first names on the list. On the other hand, when people were looking for someone to help answer questions, I was one of the first names on the list. I recall getting an email shortly after we published a performance report for one of the last VM/SP releases. Someone in the field had read the report and wanted more information on the CORNFIELD benchmark. Yes, the CORNFIELD benchmark. Don't worry, I had not heard of it either. I wrote back asking which CORNFIELD benchmark? To which they replied, "The one you mention in the VM/SP Performance Report".

Up to that point, I had thought I wasn't new to performance anymore and knew pretty much what all we did. But suddenly there is a whole benchmark that I had never even heard about and which apparently was in our Performance Report. I started asking around. I got some ideas, but they were all stretches. There was a workload we ran for VSE called PACE COB. Someone suggested maybe they think COB as in corn on the cob? (I actually don't recall what the PACE COB meant. I think it used POWER and I remember we eventually replaced it with the DYNAPACE workload). Then someone discussed maybe it was part of the CMS interactive workloads we ran to load test VM with a simulation of multiple CMS users doing program development. One of the simulations included using XEDIT to create a FORTRAN programs, compile them, and run them. Get ready for this. One of the FORTRAN programs was one that computed the coverage when spreading manure on fields. Maybe cornfields?!!

I went back to my new friend in the field with my two new possibilities. Which were both wrong. To make things easier (and perhaps because he was starting to think he should have started at the other end of the alphabet), he pointed me to the exact page in the report where we talk about CORNFIELD. The page corresponded to a section where we reported on CMS performance, actually providing pathlength and page reference data on various CMS commands. I started reading through the table of commands: SET, ACCESS, COMPARE, CORNFIELD, ... 

CORNFIELD?!!! Did you figure it out yet? This was in early days, at least for me, of proof or spell checking. The software decided COPYFILE should have been CORNFIELD. I laugh about it now, but at the time, many of us looked at each other asking how did we miss that? It was a good lesson learned.

Thinking back on this story, I also reflect fondly on the desire of those involved to report accurately and completely on the performance of a new release. We took pride in it. We worked well with other teams since in those days not everyone could 'publish' information. It was an expense so you wanted to get it right. These days the requirements to publish information are down to just 'can you operate a kyebored?'. Or maybe there aren't any requirements.

The LEGO of System z Software (aka Pipelines)

As I look back over the things that impacted my productivity as a VM performance analyst and tester, I find myself coming back to CMS Pipelines. Read Melinda Varian's paper on the history of z/VM for more details on how IBM's John Hartmann came up with the idea, and how it found its way into the product.

I had heard tell of IBMers using Pipelines in late 1980s, but it was always bit-jockey types and I had assumed it was the same ones who got giddy over the idea of using a plugboard to program or using dials on control panels to set IPL addresses. In other words, people who were either way smarter than me or loved this stuff way more than me (or perhaps both). I didn't really start to embrace Pipelines until it was incorporated into VM/ESA V1R1.1 (CMS 8).

My love for Pipes grew exponential in that first year. At first, my reaction was, "This is interesting". Then I started to see what others were doing with it, and my reaction moved towards, "This is cool". Finally, as I began to use it for heavy lifting in my analysis work, "This is awesome".

I still remember trying to do some analysis to reduce some of VM/ESA's master processor usage in early 1990s, this was the time frame where larger n-ways were starting to appear on the horizon. The use of OV/VM was becoming heavier as well, and made many requests of CP that were master only related. It was an important area to improve. I started to think that to understand some of master only processing I would need to mine more data out of the CP trace table. I could gather large amounts of data through TRSAVE, but then what? This wasn't like dump analysis where the answer is typically within the last 200 trace entries. This was a place where Pipes became my friend as I would massage the data in various ways. Continuing to narrow in on the culprits.

This Pipes-aided analysis led to two changes in VM/ESA V1R2.1 that significantly reduced the master processor usage. That led to one of my first formal IBM awards. I'd like to thank the academy, my parents, and Pipes; without their support I would not have reached the solution as easily.

Pipes would come to the rescue at various times after that. I would write a little utility to read the CP trace table on a running systems and produce some stats that was useful for performance analysis in some extreme cases. The MONVIEW and Friends utilities now shipped as samples in z/VM are Pipes-based. Some of which actually involved user-written stages. At one point John even provided a Pipelines stage to directly read *Monitor data, appropriately named STARMON. Now I was giddy!

I never rose to Master Plumber status in the fraternity of Pipelines coders, but I loved them just the same. I admit playing with LEGO bricks also. The beauty is anyone can build what their mind sees. The child that creates a LEGO model car made up of 9 bricks and the man-child that builds a 10,000 brick replica of the United States capital building each go one brick at a time. And there would be much more stress if the bricks hadn't been designed to work together as they do. The possibilities are amazing. Just five 8-studded bricks of the same color can be put together in over 10 million configurations. Perhaps that's why no two Pipes programs are exactly alike.

I also just noticed. Both CMS Pipelines and LEGO bricks originated in Denmark. Keep plumbing and keep building.

Monday Morning Monitor Club

In the late 1990s, I held a meeting on Monday mornings. We had a different topic each week usually focused on the VM monitor system service, but also covering a variety of related performance topics. I would take portions of those meeting topics and roll them into presentations for SHARE called VM Performance Tidbits. At some point, I think we moved the meetings to Tuesday, but continued to call it the Monday Morning Monitor Club (MMMC).

I have fond memories of those times when people gathered to learn and share. We have less time to do that these days, which is a shame. The MMMC included performance people and CP developers, and service folks as well. Seems odd in ways that people would gather on a regular basis to talk about such a focused topic. (It didn't seem to get any hits when I included it in my online dating profile: enjoys hiking and dissecting z/VM monitor records.) And yet, there were faithful members showing up each week.

VM historically has done performance measurement well. Thanks to collaboration from the community. The current z/VM monitor system service is fairly elegant, though many of us take it for granted. Within the z/VM Development Lab the Performance Evaluation Team has a presentation that they give to developers about when/how/why to create monitor data, called What is Good Monitor Data? (yes, the jokers like to swizzle that to What Good is Monitor Data?). Despite the jokes, the topic is taken seriously.

Most of what one needs to know about the performance of a z/VM system is in the monitor data stream. It is a very complete picture of resources used by the virtual machines and the z/VM system itself. In addition, includes data on delays to virtual machines and their effect on one another. The VM Monitor system service also reaches up and down for additional data. Various interfaces exist for z/VM to reach down into the firmware and hardware to get data; as well as the appldata interface which allows it to reach up into the virtual machine at very low overhead.

The VM Monitor system service is a low overhead data collector. One nice approach it uses is to place the data in memory shared between the Control Program and one or more virtual machines. This allows CP to put the data right in the shared memory and avoid data moves for each reader of the data. The appldata interface mentioned earlier works off a similar principle by allowing CP to have access to guest memory and pull that information from the guest directly.

The VM Monitor system service is very flexible in terms of what data you can ask to be collected: which virtual machines, devices, types of data, etc.. Whoa. I'm getting carried away on this subject. But do you see why I'd look forward to a Monday Morning?

The key the reflection here is the VM system has the idea of being able to capture this great data to help solve problems and provide information for planning. It's another piece of the VM DNA. The designers realized that the system would have problems and data would be needed to help in the analysis. This was also a key part of a white paper from the SHARE VM Performance Project in the 1980s. Now they could have taken the approach used else where to deal with these scenarios:

• Reboot and see if the problem goes away.
• Buy a bigger/faster machine
• "We were unable to recreate that problem, can you send us your private data and 10 TB of disks."

I know I'm biased, but being able to do what we can do with 1 hour of monitor data makes me happy. I know there will always be a field or two missing; and we don't always get first failure data capture. But the idea that we strive for those is important. And it's something that none of the college text books touched. I learned it and lived it in VM.

Influential People: Virg Meredith

Within IBM, if you've been involved with VM performance in any way, you probably have worked with Virg Meredith. He was one of the technical leaders when I started in 1985. His office is where anyone who got stuck on a problem usually ended up going. The bonus was that almost every answer came with a story. "Well, let me tell you...", he would often start out.

Besides understanding performance, I learned a number of other things from Virg, some of them associated with interesting quips:

• Something always changes. Virg tells of a group that came to him claiming that they ran a program and it ran very fast. They did not change anything, and ran it a second time. This time performance was poor. Virg asked them, "If you didn't change anything, why did you run it again?"
• "The right answer to the wrong question, is still the wrong answer."
• I'm not sure if it's because Virg is from the 'show me' state or not, but he prefers real data over hearing what people think the numbers were. Getting real data is worth dozens of conference calls.
• When there were meetings with executives, Virg always invited the team that helped collect the data and set up the environment. He knew it was a team.
• He spent as much time helping others with their careers as he did with his own.

There are a ton of Virgisms. One of the current team members is collecting them.

A fellow performance team member would refer to Virg as the Velvet Steamroller. Virg never hurt anyone, didn't raise his voice; but he was going to let you know where things stood. He'd leave an impression on you, but not leave any marks. Virg is very soft spoken, so much that on conference calls, we sometimes give him a mic of his own. In ways, its interesting. People will be talking over one another, but then Virg will start talking. Everyone has to quiet down in order to hear.

Now, the quiet doesn't mean Virg doesn't have some competitive spirit in him. I played softball with him for a couple of years, so I know. I remember one game where he had a very nice hit. The kind that makes you think right away, easy double, maybe a triple. There had been a man on first who watched a little too long before running. As the base runner passed second, Virg was right on his heels, urging him onwards, knowing he couldn't pass him. I honestly don't remember how the play ended, but I remember laughing at the scene.

The Tree

When I turned 25, Virg informed me that he owned a softball glove older than me. He still plays, but I've long since stopped. That same year Virg was added to the IBM Quarter Century Club. As is the custom, many of us wrote congratulatory letters. I can't find my copy of the letter I wrote him now, but I remember writing in it about how having him around gave me great confidence and assurance. I told him of a large evergreen tree next to my parents house. You could see the tree from up the street. I knew whenever I saw the tree, that I was home; and it would be ok.

That tree is still there, and Virg still sits down the hall, doing VM performance. And when we go to work and see the light on in his office, we know everything will be ok.

No Room at the Inn

z/VM has some of the best customers in the known universe. As I look back, I see how fortunate I've been at times to work with various people and the acts of kindness that they have shown me. One particular story sticks out in my mind.

I believe it was April of 1995. We had a customer in the Boston area that was having performance problems with Focus databases on the relatively new VM/ESA Version 1 Release 2.2. We had come up with a modification to correct this. The plan was simple:

1. Drive the six hours to Boston
2. Apply the fix
3. Run a test
4. Go to lunch to celebrate the solution
5. Drive back to Endicott

Around 5:30pm, I realized the plan would need some adjustment. While the drive to Boston was uneventful, the fix did not improve things. I'd be staying the night in Boston. But wait, did I mention this was the day of the Boston Marathon? The half million people or so that showed up to see Cosmas Ndeti of Kenya win the marathon with a time of 2:09:22 had taken all the rooms in the city that were within the IBM corporate hotel limit.

The customer system programmer to the rescue. He was a virtual bachelor that day as his wife was out of town visiting relatives. He invited me home to stay in the guest room. He gave me a ride so I would not have to endure Boston driving. I remember on the drive home, I saw a station wagon (who remembers those?) had a kiddie pool tied on top. Except it wasn't tied very securely as at one point I see it move from shaking in the wind to flying free right at us. My host didn't flinch, as if this was a regular occurrence in Boston traffic. A flick of the steering wheel and he negotiated around the small plastic pool, weaving out of the lane and back again. He didn't even break stride in whatever story he was telling at the time.

At his house, he treated me to leftovers; introduced me to his small dog who seemed to like having another friend in the house; and made me feel at home. We logged into the systems that night to watch performance on some batch processing. I don't recall what baud modem he had, but it was fast enough to paint the screen of a 3270 emulator, and to allow us to see how things were running. We didn't pull an all-nighter as we knew there would be more to do with the problem, and status for managers in the morning.

The next day, we went back to work gathering more data to help with a better fix. I would return home and consult with others. The ultimate solution was to introduce the RECORDMDC option for minidisk cache. It worked like a charm. I learned a lot through that crit sit (critical situation) about teamwork, our customers, and making the best of the moment. If you're out there Dincer, thanks for being a great host. It set the bar for how z/VM should treat its guests: don't starve them, keep them informed, protect them, and don't push them to exhaustion.

The "M" is for Machines

Hardware was king when I started with IBM, or at least that's what I remember being told by the hardware engineers. At the time, software and hardware performance at the Glendale Lab was made up of several first-line departments that met at a third-line level. VM/SP Development was a separate organization. Part of this was done so that the development owners and the development QA wasn't under the same management. There were also a lot more people involved in the product at the time. The hardware and software performance groups had a good relationship. I learned a lot from my peers in hardware. Often hanging out or doing lunch with them. Marty Curtin was one of them. He had been in software and then crossed over to processor performance. Years later he'd be doing software again, for one of the other System z operating systems. I think it's called z/OS. Marty continues to be a friend.

One of the Endicott processors in the 1980s was the 4381. It was a very good VM box. I don't think it was a coincident that both were influenced heavily by Endicott. People who worked, played, and ate together were able to bring some good synergy to the hardware and software. The 4381 was a very good seller; it set high expectations and met them.

Over the years, the IBM annual report has shifted from focusing on hardware to more software and even more services. However, hardware is still king as it's the start of the IBM stack. Upon the hardware foundation, we add hypervisor, operating systems, and middleware. Hardware, particularly System z just lends itself better to be qualified with the word 'legendary': Legendary Availability, Legendary Security, Legendary Innovation, etc.. The innovation continues to amaze me, such as RAIM (redundant array of independent memory) introduced in the z196 processor family.

I was visiting a customer once and discussing a possible system design. In this case, I was a little cautious about setting expectations. Someone had quoted 40 years MTBF (mean time between failures) stat. I commented, "That is for the hardware, remember software is written by mortals". One of the customers smiled and asked, "Oh, so the processor designers aren't mortal, they're gods?". That made me smile and I quipped something about visiting Mount Olympus/Poughkeepsie.

As I look back, perhaps Atlas is the System z hardware team, they hold everything on their shoulders, tirelessly.

The "B" is for Business

In college, I took one economics class, but those concepts never really took root in my brain garden. I left college with only a vague idea of how computer programming was tied to business. It was pretty much comp sci degrees were getting job offers and that was good enough for me.

When I joined IBM, we still had a 'full employment' policy. So the concept of the business climate impacting my ability to work for IBM hadn't been invented yet. IBM prided itself on its ability to protect its employees and the loyalty that generated. In 1988, D. Quinn Mills wrote The IBM Lesson: The Profitable Art of Full Employment. It tells of various events in IBM history where the corporation retrained people and kept them. I still have my copy and thumbing through it, I think some people who didn't live through that era may believe it's science fiction. While tempting to dwell on full employment longer, I'll move on to my real topic.

Somewhere along the way, I started to realize that behind the technology and my day to day work evaluating performance, there were people striving to figure out how to make the most money out of all of this. I learned there was a difference between marketing and sales. I learned that as great as the software was, it didn't sell itself.

I think my eyes were opened after having seen real customers and even more so, IBM account teams. The people that wore white shirts every single day. Over a couple decades I would meet people who would make me realize that, like it or not, every single IBMer was in marketing whenever they were exposed to customers or the outside world. Some of these people included: Reed Mullen, Jim Porell, Len Santalucia, and Barton Robinson.

Let me share a story about Barton which help made me realize that we're in a business. We were doing a conference in Europe where they had a speaker orientation meeting the night before the conference started. Both Barton and I were speakers at this event. Part of the package they gave us speakers was a list of registered attendees. At one point, I looked over to see Barton making notations on the attendee list.  He noticed my puzzled look and explained:

"These companies here", he said indicating one mark, "they are already my customers. And these others will be my customers at the end of the conference."

 I looked around at some of my peers, we weren't looking at that list. We were looking at the list of restaurants. I realized I had a lot to learn about business.

I started thinking more about how do my customers actually use VM? How do they make money or lower expenses or become more productive using it? My view of my job and VM started to change as I looked for business value in what we did, and not just technological value. Those that work with me during plan build stages have perhaps heard my rules of five:

• Name five customers who are asking for this new function, or
• Name five customers who would benefit from this new function, or
• Name five customers who we can articulate the value of this new function.

A number of years ago, I added a tag line to some of my emails for z/VM: Making systems practical and profitable for customers through virtualization and its exploitation, because the "B" stands for business.

The "I" is for International

The world has become smaller in 40 years, but I am still amazed at how I have been able to interact world-wide. How wide is the International VM influence in my life?

• Earlier this year, I commented on the birthday wishes posted on my Facebook page from so many countries. Friends mostly made through working with VM. 
• The stats on this blog currently show readers from 13 different countries and every continent except Antarctica.
• I have made live presentations in 13 countries; and webcasts to many more.
• My record for a single day is individual phone calls with customers and IBM account teams in 8 different time zones.
• I have been yelled at in 6 different languages, 7 if you count American English and British English as different languages. :)
• I've lost count of number of countries from which there are people I have discussed VM while having a beer or coffee.

I can't say enough about my international customers and fellow IBMers, how gracious they are and what wonderful hosts they were. They forgive my inability to speak more than one language; and some would assert that I don't even speak that one particularly well. One of my first international discussions was done over email (VM notes at the time). I had reviewed some VMMAP data from a customer in Europe. After I reviewed the data, I could see the disk activity was very high and delays were forming there. However, instead of saying that, I used a phrase that did not translate very well:

It looks like the DASD got pounded.

The IBMer in Europe replied with something along the lines of:

Bill, the DASD subsystem is in good order. Nothing has hit it, there is no physical damage that I can see.

Those that know me, know I tend to use a lot of slang and colloquialisms. I know I am difficult to follow at times, but I appreciate the patience people have with me as a speaker.

Through the friendships that have developed around the world, I've become more aware of world events. I'm able to empathize with those facing hardships, whether they be earthquakes in Japan or fires in Australia. And I've been able to celebrate the victories such as World Cups, Olympic Games, or other fun events.

I recently was on the road working with a customer with an IBM team that had people who had boarded planes from four different countries to get there to work with a local IBM team that hailed from three other countries. It truly was an International Team. As this isn't unusual for me, I need to stop and think every now and then about how special it is to be part of something like this. So many people, different in many ways, brought together by a product and a love for that product.