Plus One to Grow On

In case you weren't counting, yesterday's entry was the 40th. But I thought I'd add one more to wrap up things. When I started this series of writings, I was very afraid that I would run out of topics or the time required would be too much, and I would fail. I'm glad to say, we did it. I greatly appreciate the support from various people.


Since many of the entries were written late at night, there were even greater chances of errors in grammar and spelling than my day time writing. Thanks very much to those that provided corrections, in particular: Perry Ruiter, Mary Ellen Carollo, and Phil Smith.

Thanks also to the various people that provided topic ideas or allowed me to think out loud. Including the noon time walking group: Kevin Adams, Alan Altmark, John Franciscovich, Mark Lorenc, and Bill Stephens.

Thanks to those who networked the topics, sharing with friends and colleagues, in particular: Gabe Goldberg.

Thanks to those of you that left comments, that made it much more enjoyable for me.

Most importantly, thanks to those of you that read and gave me feedback to let me know that it was worthwhile. Many of your positive comments touched me, and encouraged me when I needed it the most.

I learned a lot in this journey. I have a much greater appreciation for professional writers, especially those that write on a daily basis. There are things I wish I had done differently. I really wish I had timed things out better and had created a more detailed outline of all the topics on day one.

I know all of you reading this have your own stories to tell, and I encourage you to go tell them and share them.

I am also fascinated at how various topics ending up creating new threads and connections:

• I was contacted by a brand new System z person in Belgium working on z/OS who came upon the blog.
• I exchanged emails with an author whose book was sitting on the 'please read me' pile on my desk, after he was forwarded one of the entries on problem determination.
• Reconnecting with some of the people discussed in the topics

I am also able to get some statistics on readers of the blog. It appears we ended with readers from almost 30 countries. In the Olympic spirit, here are the top ten countries. I had fun watching as Latvia came on strong during later half, but was edged out by France in the past couple of days:

1. United States
2. Russia
3. Canada
4. United Kingdom
5. Germany
6. Netherlands
7. Japan
8. Australia
9. Brazil
10. France

So what's next? Talk to me in five years. Though my idea for the 45th is a series of interviews with others.


Thanks again everyone!!

An Impossible History

Every five years, I seem to hear the same question, "How many more years can the success continue?". It's probably not a bad question to ask. But before raising the query out loud, perhaps we should consider why the question is being asked. Are we just being like the news and dwelling on the downside of life? That's unfortunate if we are. On the other hand, not knowing what lays ahead in the future, we should enjoy today. Doing whatever we can to improve life for one another.

I recall in the 1990s asking some people, "If this is the last year I work in VM/ESA, what's the most important thing I should accomplish?". Now that did fuel some rumors that I was leaving, but that wasn't my intent. I was looking for help in setting priorities and knowing that what I did was important. That year we fixed a number of problems in the VM state sampling that was causing problems in performance analysis and keeping customers from getting the most out of their VM systems. It became the 'thing' I had to do.

Someone in the organization came to me not long ago. They were a little overwhelmed at the news of cuts and pressure to do more, and the uncertainty of the future. Empathizing with them, I shared the following quote that I had heard recently, which was one of the driving forces behind me starting this blog:

"I am only one, but I am one. I cannot do everything, but I can do something. And I will not let what I cannot do interfere with what I can do." - Edward Everett Hale

At SHARE last week, I got to meet Dave Tuttle, part of the original VM/370 team. He had a number of stories to tell. The one quote I wrote down was, "It was mostly a matter of doing the impossible". The VM product has a solid foundation in its design points which have survived decades. When you add talented people to the proven design principles, you get a combination that succeeds. When those people are fueled by passion for the product and compassion for others, you have an unstoppable force. I still believe we have those people. When I see younger people inside and outside of IBM working with the product that are cut from that same cloth, I know this impossible history has a possible future.

Influential People: Reed Mullen

I've written on several influential people in previous posts. The actual list of people who have impacted and influenced me is rather long, especially over the course of my career. Most of the previous ones served mostly as mentors. Reed has been a role model and given me words of wisdom. However, his role in my life and career, and the z/VM product has been more than that.

Reed Mullen, Sir Reed the Optimistic, became a Knight of VM in 1997. He was the senior planner for most of VM/ESA and for some of the z/VM lineage as well. Senior Planner was the title, but he was more than that. His vision, creativeness, and articulation are the yardstick by which others are measured today.

He is also the only IBMer to have his own bobble-head figure:

And what discussion about Reed would be complete without the infamous cheese-butt story. The following picture circulated at SHARE, showing Reed wearing not a cheese-head, but a cheese-butt. Reed is a serious Packers fan. I believe this picture was just prior to the Packer's 1997 attempt to repeat as Super Bowl Champs.

Five or six years ago, I picked up the book "Vital Friends: People You Can't Afford to Live Without", by Tom Rath. It discusses the importance of friendship, particularly in the workplace. The book discusses eight vital roles that friends play. It also has a little web based application to help you determine which roles your friends play. Reed's roles came out to be: collaborator, mind opener, and navigator.

As a collaborator, Reed and I do have several common interests and we relate to one another well. It has allowed us to pool our resources and talents at times. In March of 2005, Reed and I did a series of CMG regional conferences that hosted "Virtualization" days. Reed and I represented z/VM and then they'd also have speakers from VMware and Microsoft. I would cover the technical basics and Reed the business aspects. We kicked butt.

As a mind opener, Reed really did help me see the business side of things. He also was a help in tempering my passion when it got the best of me. Lastly, he was able to point me to the upside of most situations.

The third role was navigator. This didn't mean telling me which direction to go. Often it was more about helping me talk through the options I faced. The process of working through the pros and cons allowed me to come to a decision in which I had confidence. Reed as a navigator is ironic in one perspective. When traveling by car, Reed hates to be in the passenger seat, especially if the driver goes slow.

Which reminds me of one of my favorite Reed stories. I believe we were coming back from the CMG regional meeting in Washington DC, in late March of 2005. Though it may have been a Hillgang meeting. We had driven (ok, I drove for about 15 minutes and then Reed wanted to drive).  On the way home, we stopped for dinner in Frackville, PA, on Route 81. Shortly after getting back on the highway we saw some flurries. Until then, the roads had been clear. Within the next couple miles as we climbed in elevation, the snow was more significant and several inches could be seen on the shoulders. Traffic slowed. And then in a section of the divided highway cut through some hills, traffic stopped. We surmised there was an accident. After several minutes of standing still, Reed called home to tell his wife that he'd be late. It was probably around 7 or 8pm at that time.

After an hour or so of not moving, Reed would call his wife to let her know that he probably wouldn't be home tonight, that once we got moving, we'd find a hotel. At this point, our laptop batteries were close to 0%. Another hour went by with us moving about, ... well not moving at all. We took inventory. Neither of us had any food or anything to drink. For a brief period, I thought Reed was looking at me like in the cartoons, where I had transformed into a big turkey drumstick.

During inventory, Reed pointed out that we didn't have much fuel left. We decided to turn off the engine. While snow is a good insulator, the metal and glass rental car was a better conductor, aided by the winds cutting through the pass. As the optimist, Reed would point how great it would feel when we turned the engine back on and heat worked its magic. We would do that several times in the hours of standing still that followed. We would doze off periodically. I tried to keep one eye open, remembering that drumstick image.

Around 2am, we'd hear someone tapping on windshields telling people to wake up and move. Apparently a big part of the delay was truckers who had gone to sleep and were harder to wake up back in the sleeper cabs. As we started moving, we could see several snowmen built along side the road. Based on them, we believe the back up was well over a mile. We would end up driving straight home, after we stopped to fill the tank.

There are few people with which I'd rather be stranded than Reed. Though, I now always make sure I have a full tank of gas, something to eat, and water when traveling that stretch of Route 81. We love to tell that story now. I can't hear "Frackville" without thinking of my friend Reed.

Reed was a leader in VM, not because of a title of Senior Planner, but because he lead and people followed. His ability to communicate surpassed only by his character and integrity.

Good Night, Good Knights

Last evening at the 40th Anniversary/Birthday party for VM at SHARE, we honored sixteen outstanding people and welcomed them into the Order of the Knights of VM. I'll let the LVM SHARE officers post the list of new Knights. One can see the previous knights here.

Some of you know and understand the Knights of VM, others of you may be scratching your head and wondering if I hit my head. The Knights are people who have contributed to the VM product and community over the years. They are selected by the Community usually every five years in conjunction with a major VM anniversary. Each knight or dame is given a title specific to their contribution or character.

I have few things framed in my office, but the certificate showing me as a companion in the Order of the Knights of VM is one of them. And if I could only have one thing on my wall, it would be that frame. Is there any greater thrill than to be honored by one's peers? At the same time, I am humbled, feeling as though my name doesn't belong on the same list as so many outstanding individuals.

Last night, it was precious to see the faces of individuals as their names were called. There are no fancy acceptance speeches. That's not the way of a Knight. There was applause. There were smiles at some of cuter titles bestowed on our knights and dames. There was admiration in the eyes of others, knowing how these people touched the product and touched the people involved. That is the way of a Knight.

Before I had become a knight, I recall joking with my manager at the time. I asked him if I rode a horse to work, where could I keep it. He told me I didn't have a need for a horse. So you can imagine my excitement when becoming a knight. I thought for sure I now had a legitimate reason to own a horse. I mean, what's a knight without a horse? I lost the argument. But there are times when I'll gallop in the halls of IBM Endicott. You know, that one sided skip we all did as kids pretending we were riding a horse. It always makes me feel better. You should try it.

One of the previous knights, Nick Gimbrone, Sir Nick the Insightful, made yet another insightful point during the discussion for this year's selection of knights. He reminded us how the knights of old had support, their squires and shield bearers. Just another wonderful picture of the VM Community.

Congratulations to the Knights and Dames of the Class of 2012! Thank you for serving with honor and diligence.

Sir Bit, The Performer

A Rainout and A Rap

Over the years, I've been able to have some fun while at SHARE. In particular, I've been able to attend various sporting events with my SHARE friends. Most recently, I attended the ACC men's basketball tournament final in Atlanta, Georgia. We watched Florida State upset North Carolina. I recall seeing a couple NHL hockey games, including Tampa Bay Lightning when they were still playing in old Expo Hall at the Florida State Fair grounds back in 1992. Baseball games included Boston's Fenway, Baltimore's Camden Yards, Toronto's Skydome, San Francisco's AT&T park, Atlanta's Turner Field, and Chicago's Wrigley Field.

I remember one game at Wrigley. After about two innings, some rain started falling and they pulled out the tarps. We were seated in an area with an overhang, so the rain didn't bother me at all. The Cubs never got the game in, but that didn't bother us. We sat there and chatted, had a few beers, and dined on Chicago dogs and pizza. That's one of my favorite non-work SHARE memories. Actually, we probably talked a little VM performance as well, so there was some work. :-)

There was also at least one minor league game, watching the Nashville Sounds play. That was early in my SHARE journey. That game would plant the seed for the new nickname Rapmaster Bill. It started when leaving the ballpark when I stopped to buy a ball cap. I made the mistake of putting the ball cap on backwards. Jeff Wilk, one of the VMPR ribbon wearers at the time, got this big smile and goes, "What's with the hat? Are you like a rapper or something?" Then the smile got even bigger as he proclaimed, "That's it, we're gonna call you Rapmaster". There were various stories told through out the week about what happened at the ballpark. Someone had also collected all the souvenir beer cups from that game to create a large stack of them known as the "torch". The story would change as to how many the new IBM Rep (me) had contributed to the stack.

Craig Welch was the IBM Rep from the Storage Device group at the time. He egged me on and convinced me that we should have some fun and do a rap. Silly me, I sad yes. So for the closing session, Craig and I came running in wearing our suits, well at least the tops of the suits. We also wore shorts and high top sneakers, and baseball caps on backwards. And we did a little rap that went like:

We're the VM rappers that's who we be          
Rapmaster Bill and Craig MC              
and we're here to tell you two fast stories
                          
First is a tale we hope you like         
so here's the Rapmaster on the mic         
                                
I went to the park to see a Sounds game.          
And it was there that I got my name.                                
Rain came down  I bought a hat                 
Had 2 beers  and that was that!
                                
But Roger, Carol, Jeff, and Neal
embellished the story; And that's the deal
                                
 Now sit right back and you'll hear a tale         
 from Craig MC who just can't fail
                                
Many SHAREs ago a cookbook was tried            
soon we produced the Perf Info Guide
                                
Now when you name something, be careful, you dig? 
because what we ended up callin' its the PIG        
                                
so now that the PIG's the latest SHARE rage           
Carol Mac and Jeff, come up to the stage

I thought we were going to give some of the more dignified IBMers heart attacks. Apparently our act was new, both in dress code and enthusiasm. In ways, I think it helped me feel more approachable to my customers.

Over the years, Craig and I would do a number of Raps. Craig's role would change and he stopped making it to SHARE. I did a few solo raps, but it wasn't the same. The Rapmaster has since retired. My fifteen minutes are up. But it was a lot of fun.

Is There a Doctor in the House?

One of the fun memories and experiences at SHARE was the VM Performance Project Performance Clinics. Each SHARE, we would have a 6pm session where people could bring in their VMMAP, VMPRF, Performance Toolkit, or ESAMAP listings (performance reports) for review. It was a fun and well done event. A few of us more experienced people (Barton, Marty, myself, and others) would play the role of the doctors. The 'patients' would come to the clinic and one of the other VMPR project members would do the triage, and direct people to the doctor that might specialize in their environment or performance tools. We would then go over the reports with them.

There were some patients who had real problems. Their systems were hurting in one way or another. In some cases, they were pretty sure what the problem was and just wanted some confirmation. In other cases, they didn't have any complaints, but were there mostly as a check-up. Occasionally one of us doctors would spot something abnormal. We'd go over what we saw with the customer. So in ways, it was like a teaching hospital. In fact, there were times other people came to the clinic and told us "I'm just here to observe.", and they would, with the patient's approval, sit in on the discussion. There were also times when I'd call in one of the the other doctors for consult. "Hey, have you ever seen pending time this high on an ESCON channel?" or "Do you remember the APAR where RACF wasn't releasing storage (memory) associated with virtual machines as they logged off?"

A couple of the patients would be repeat attenders. I don't recall any real hypochondriacs, but just people who liked the idea of someone looking over their systems. Often smaller shops where they didn't have other people to bounce ideas off. The most satisfying were the cases where the previous SHARE we had spotted a problem and prescribed a solution; and the customer came back to show us they had corrected the situation and everything now looked good.

Some of you don't believe that I'm actually a shy person, but I really can be. But what a wonderful ice breaker the performance clinic was. There's nothing like sitting down with someone and looking through performance data on their system to get to know one another. It gave us a topic of common interest. I would learn more about what people do with their systems, and they would sometimes learn more about VM performance.

We don't run the performance clinic anymore. The world is better connected so I can get listings at any time. From time to time, I will still look over performance data at SHARE. These days, with wireless and laptops, listings aren't required. You simply bring up your system and show it in real time.

At a couple of IBM Technical Conferences, we used the idea slightly differently. For those that registered early for the conference, they'd be eligible for a drawing for a free performance review by either myself or Barton Robinson of Velocity. We'd each get data from a different customer, analyze it, and put together a short presentation on the data and our findings as part of a session at the conference. It was a close to a shootout as we got.

At one of the SHAREs, Gretchen Thiele of 3M at the time, honored us doctors by presenting us with real stethoscopes. It was a prized possession on one of my bookshelves for many years, and a great conversation starter. Though some of my non-IT friends didn't understand the fun in looking at rows and rows of numbers. I would later donate the stethoscope to a friend who was in medical missions in Guatemala. But I still have this button to remind me of the Performance Clinic.

 

My First SHARE

Here I am at SHARE in Anaheim, California. I believe this is my 47th SHARE meeting. Some other speeds and feeds:

• 178 = Number of times I've presented at SHARE
• 23 = Number of cities visited
• 16 = Number of States or Provinces visited with SHARE
• 16 = Number of VM releases I've spoken about at a SHARE
• 8 = Number of times I've been to SHARE in Anaheim (highest of the cities)
• 6 = Highest number of sessions I've presented in a single SHARE metting
• 6  = Number of  Chocolate Pig-outs attended
• 2 = Number of countries I've been to for SHARE (Not counting Guide-SHARE Europe which would add two more)
• 155,240 = Number of miles traveled for SHARE.

For me it all began in August of 1990 in New Orleans. That was my first SHARE. I went there for just one day of the conference to present "Introduction to VM/SP Performance". SHARE attendance was much larger in those days. They needed a half dozen or more hotels to hold all the attendees at the time.

I got in fairly late the night before. Registration was closed. My session was also the first session slot the next morning at the same time registration opened. I was going to break a rule by showing up to a session without a badge. My first SHARE. The climate in New Orleans is a tad different than Endicott, NY, especially in August. Also, keep in mind that these were the days where IBMers wore suits all the time in public.

I remember walking into the room for my session, brow wet with sweat and my shirt soaked. The combination of heat, humidity, nerves, fear of no badge, and stage fright all rolled together. I must have been quite the sight.I have no idea why I didn't throw up or just turn around and leave. I was greeted by one of the nicest people I've ever met at SHARE, Carol MacNaughton, of Bell Northern Research at the time. Like a life guard saving a floundering swimmer, she introduced herself and explained that she would be chairing my session and was there to help. It got better after that.

I can't find that presentation and I don't remember much of what I said. I do recall making my first joke. I told the audience that on the flight down to New Orleans, I sat next to a twelve year old who traveled frequently. During the pre-flight safety demonstration, he turned to me and said "why do they go through this every time? We all know it."  With that, I smiled and put up the official IBM Disclaimer chart. (I put it up on the overhead projector, we used transparencies in those days). I got a couple laughs or at least smiles and I made it through the presentation.

I was able to attend a few other sessions that day, including a requirements session. I was amazed at how much went on at a conference. Even in one day, I saw there was something special about this group of people, including that I had much to learn from them. (And I still do.) I had survived and I was hooked. I'd skip the interim SHARE after that, but go to San Francisco in 1991 as part of the transition SHARE with Paul Van Leer.


The rest as they say is history. I can't thank the customers and vendors enough, especially Carol, for their kindness in welcoming me. I'm looking forward to a great week here at SHARE. If you're here, please be sure to see Jim Elliott's session: 45 years of Mainframe Virtualization on Wednesday at 4:30, followed by the VM Birthday Party at 6:00.

With a Little Help From My Friends

International Business Machines Corporation. Big Blue. Revenue of almost $107 Billion in 2011. Infinite resources? Well, not quite. Occassionally we get a problem where we need a little more help. That's ok. We've got extra help. Our customers.

I'm not sure which is the better analogy, the twelth man from football or the sixth man from basketball. For the non-sports fans, the former refers to the impact of the home crowd and the latter refers to the first player off the bench. In either case, both can be the difference between a win  and a loss. Our customers are like that, and I'm not talking about the fact that they give us money. (I really appreciate that fact, but it's not where I want to focus.)

I wrote earlier that learning how customers use our systems makes me a better software engineer. But there are other ways that customers help. Two of the big ones are: determining best practices and problem determination.

While we at IBM attempt to provide solutions with clearly articulated best practices, we occassionally fall short, and some of that comes from the increased complexity of end-to-end solutions. I'd like to know what the average number of different vendors is for a given solution, or the number of possible combinations of vendor pieces in the stack. It's difficult for any one vendor to evaluate all those possibilities, or even a couple vendors in partnership. Enter the customer. When you share your experiences with us, we're better able to see what works best.

In fact, often when I provide the IBM best practices, I'll get the questions, "So how many customers are running that way? Can we talk to one of them?" It's part of the reason why we're so persistent about asking people to be references. By taking the experiences, good and bad, from various customers, I've been able to help guide new customers better. By knowing these customers well, I also know their skill levels, organization structure, and to some degree IT investment strategy. When talking to a customer, I sometimes feel their patience start to slip away as I'm asking various questions about their environment or the politics in their shop. I know they just want me to answer the question du jour of "What's the best practice for fill in the blank?". Well, what I'm doing is trying to match the profile of the customer asking the question with profiles of other customers.

I've also called on customers to help with problem determination. Have you ever felt a bump on one side of your head and immediately check the other side to see if it's normal? It can be effective. I do it with systems too, compare good with bad data. However, there are times when a customer has a problem and I don't have a "good" system like that. Or they are running vendor software where I have no experience. No problem. I'll think of two or three customers running that and ask them for some data, or ask if they have seen symptoms. There have also been times where I don't know of a customer running that way, but will post to IBMVM listserv and ask for a sample of data for a given environment. I've never been disappointed with help provided.

There is a tradition on many sports teams of retiring the jersey number for an exceptional player as a way to honor them. The Seattle Seahawks did it right. They retired number 12 to honor their fans, the crowd, the twelfth man.

Happy 40th VM

August 2, 1972 was the announcement of VM/370. Now, some will say it's more than 40 years if you go back to CP-67, or even further back to CP-40. While the earliest versions of VM were known to customers, it wasn't until VM/370 that a formal offering was available. Tradition has used that as the starting point. The question for me is whether this is an anniversary or a birthday.

Part of me wants it to be a birthday, because you hear about birthday cake more than you do anniversary cake. I like cake. I like the idea of the years prior to 1972 as the time IBM was with child. The gestation period was longer than 9 months, and even exceeded the 22 to 25 months gestation of an African Elephant. That's not necessarily a comment on the size of IBM at the time.

Part of me wants it to be an anniversary, because every five years is when the big celebrations occur. Birthdays make me think about getting older or aging. You can use cute expressions like 40 is the new 25, but it's still older than 35. Anniversaries on the other hand are more about recognizing the longevity, endurance, and growth. I also think of an anniversary more as recognizing an event. Often an event that brought together a group of people, and in some cases a commitment or bond. There are elements of that aspect that seem to fit VM.

Birthdays also seem to have an end, which we don't like to think about. But the anniversary of the founding of a town or a country doesn't seem to end. I wasn't involved with VM for the 5th or 10th anniversaries, and my involvement was still new for the 15th. It's also unlikely that I'll still be writing blogs for the 55th. Others will pick up the banner and the cause, keeping tradition alive.

In both cases, recognizing the event is a chance to tell stories, share memories, honor the pioneers and champions, and celebrate. All of which the VM community does well.

Community: To Everything there is a Season

When I started working with customers, I remember one of the more cynical IBMers warning me to be careful that I didn't fall into the Stockholm Syndrome. At first it didn't register, because I confused it with the Helsinki Formula which was a new breakthrough in treatment for thinning hair at the time. After I got over the idea of being self-conscious, I realized he was really talking about hostages becoming attached or feeling positive about their kidnappers. I still found it odd because I hadn't been threatened or tied up by customers. I will acknowledge that I have become attached to customers, partly in the simple partnership of doing business and in other ways by learning they are human, real people just like myself.

I have celebrated various events with customers. The VMSHARE conferencing discussed yesterday had a couple of forums, called MEMOs,  related to this: ITSABOY and ITSAGIRL. At one SHARE I even wrote a rap (oh, I let that history out) for Anne Marie Marcoux's first child Annick:

I was oh so very blue
But then I heard a baby's due
That really made me smile
For more than just a while
Cuz babies are so cool
They never call me fool
Their hair is very short
A look I do support
They may get in predicaments
But never write requirements
But the very biggest thrill
is when they call me Uncle Rapmaster Bill

Today we celebrate more grandchildren than children, but the new generation is still working at it.

It seemed the VM Community had a knack for finding reasons to celebrate. Marriages were announced in MEMO ITSATHEM. People leaving one job, joining another, earning a degree; all would be celebrated.  At SHARE there is a tradition of honoring someone who has been serving as a rep or volunteer for a long time when they step down, or step up to a new position. The traditional gift is a teddy bear. My joining the IBM Quarter Century Club was sweetened by reading congratulatory letters from many of you.

Perhaps a more telling aspect of the Community is not the happy times, but the sad times. These have included the down times of business, with people looking for work. And various times where the VM Community faced powers within IBM that didn't see the value they saw in it. While those were rally points in the VM history, there have also been more serious moments. The concern shown the VM lab in Endicott during the floods of  2006 and 2011 warm my heart. I've been even more proud watching the group rally around those who lose a loved one. Thomas Fuller wrote, "No man can be happy without a friend, nor be sure of his friend until he is unhappy." This is why I know the VM Community is full of friends.

Shrinking the World and Compressing Time

Yesterday I wrote about the VM Community. The backbone of that community has been electronic communication. Now, the fortunate folks out there who never knew a world before the internet, blogs, RSS, etc. may be thinking 'boooring'. I'd ask them to think about how long this has been around and the ground breaking that was involved.

The first electronic conferencing or forums that I was aware of was VMSHARE. David N. Smith of TYMSHARE introduced it at a SHARE conference in 1976. VMSHARE was one of the earliest electronic conferences. Dave wrote the software for this and convinced his management to supply the system and network. One of the driving forces behind it was the desire for greater community and discussing ideas in between conferences. Later that decade, in 1979, VMSHARE was extended to Europe. You can still view the archives.

Here is my first entry to the VMSAHRE world, the NEWGUY FORUM:

 I just got lost for 20 minutes reading some of the old appends. :-) The SHARE community would give me a few new nicknames.

Over time, various electronic conferences and bulletin boards would become available as networks increased and expanded. Internally at IBM, conferencing or "forums" would become common place and was based on powerful software running on VM. This would be used for all sorts of topics and communication within IBM. One of the forums that I recall with a heavy heart, was one that was spontaneously created on January 28, 1986 to share information on the space shuttle Challenger disaster. At that time, IBM Federal Systems was still part of IBM and had a role in the space shuttle program.

Another software product that would appear was LISTSERV from L-Soft. Under Eric Thomas' leadership and vision, it would become a formal product in 1994. While today it runs on a variety of platforms, it started and continues to run on VM. The mailing lists managed by LISTSERV would replace VMSHARE with various lists supporting TCP/IP, Pipelines, VM, and of course Linux on System z. VMESA-L was a popular list during the hay days of VM/ESA. To avoid future name changes, the list changed to IBMVM after z/VM came along. If you aren't on these lists, check out the VM Home Page for information on them.

Perhaps I'm not objective enough, but one of the things I've noticed, and others confirm, is that the VM related discussion groups tend to be friendlier than your typical newsgroups. Flaming appends are very rare, as is trolling. The group as a whole does a great job of policing itself. Newcomers are made to feel welcome and encouraged to join the discussions. The few times I can recall that a moderator got involved in the past few years was to remind us the discussion was getting a bit silly and off topic.

The IBMVM list is like a lifeline for novices and veterans alike. VM is used around the world. So there is always someone awake who can help me or you.

.BB

The VM Community

The VM Community encompasses a large group of diverse people: IBMers, Vendors, Customers, and Consultants. Within each of those groups, are more subgroups, developers, support, sales, etc. all bound together by the common technology known as VM.

As I mentioned earlier, my view of this community's scope grew over the years. It's amazing to look at how various people and groups have contributed so much to VM and the use of it. This involved contributing code and features, helping one another use the product, and influencing IBM.

Some relative newcomers to VM were surprised lately when I spoke of the OCO (Object Code Only) wars. They were even more surprised when I started describing how much our customers modified and enhanced the product with their own code. And I really floored them when I told them that a number still use modifications. Beyond the modifications are hundreds of user written programs, scripts, tools, white papers, redbooks, and presentations.

While some shops only have a single system programmer, I point out that they're never alone. You have this community backing you. And it's a community with  credentials. We tend to measure experience in decades rather than years. That's not to say there aren't people that are new to VM. We have a number of them as well, and it's a nice blend. As a new system programmer, there will seldom be a time where you'll face a problem that hasn't been faced before. And you'll be able to find someone to help as well.

One of the things that impresses me is that this VM Community crosses borders, parties, and other lines that tend to divide. I'll see system programmers from two different competing corporations exchange ideas. I'll see people agreeing with one another on an electronic forum where I know they voted for different candidates. It's just one of the refreshing things about the VM Community. Of course there is good nature ribbing and joking at times. That's part of being in a family, or so my mother told me, the youngest of four children.

That's not to say there aren't moments of tension. Particularly between IBM and its customers. But like a good community, we work it out and adapt. I mentioned in an earlier post that Paul Van Leer taught me to represent IBM to the customer and the customer to IBM. So there have been a few times where I felt tension from both sides in carrying out that mission statement. I'm thankful those moments have been few.

There are a number of definitions for community. The following one is from the The American Heritage® Science Dictionary. I probably could have opened with this definition. Actually, I could have opened and closed with it.

Community: A group of organisms or populations living and interacting with one another in a particular environment. The organisms in a community affect each other's abundance, distribution, and evolutionary adaptation. Depending on how broadly one views the interaction between organisms, a community can be small and local, as in a pond or tree, or regional or global as in a biome.

Influential People: Joe Tingley

I had various people showing me the ropes when I started with IBM: Greg Gasper, Eric Strom, Richard Shultz, and others. But I want to talk about Joe Tingley, as he had a significant impact on me, in multiple ways. The project where I learned the most with Joe was the introduction of AVS into, I believe VM/SP 6. AVS (APPC/VM VTAM Support) allowed programs utilizing APPC/VM for communication to work across VTAM, an SNA based network. At the time, IP was out there, but in business SNA was king. (Have I reached my quota of acronyms for the day?)

As this was new function, it was more difficult than just doing a comparison measurement. You never know how many measurements one will have to make to validate the line item. But it's usually safe to say that the most difficult measurement is the first one. That's an area where Joe always thrived, the difficult measurements and configurations. It was a challenge he fed off. So the first thing I learned from Joe was to not shy away from challenges.

As I mentioned, sometimes new code doesn't work. While Joe wasn't a programmer, he had a good vision for problem determination. How do you narrow down on the problem. He would encourage me to try to get down to as small a window on the problem: line of code, module, driver change, etc.. This became instilled in me and was beneficial, both for working on customer problems as well as new code. Developers always like you better when you can say "I think the problem is related to this segment of code in HCPxxx" rather than "it don't work good."

Often when working with Joe, I would hear "come on Wild Bill, we need to talk to fill in the blank." (Perhaps we'll explain that nickname another time.) Joe knew a lot of people and we'd go to them directly. I learned the importance of dialogue because often in these problems an email wasn't efficient. Email, in Joe's book, was for giving managers status. This also shortened the path in getting an answer to a problem that was blocking progress. It also helped me as a new person, because people got to meet me. That way I wasn't just a name on the copylist of an email.

The gaining efficiency by using the right communication method was indicative of Joe making the best use of all resources. He was very good at, let's call it scrounging or scavenging machine time, disk space, and other resources. He'd be very creative in that manner. Related to this was his work ethic. If there was 20 minutes left in a machine shot, some people would say, well that's not enough to do a whole lot, I'll try again another day. Joe would use that 20 minutes, and more if the next shift didn't show up on time. He worked hard, and played hard.

Joe would leave VM performance as we were scaled down, but eventually would come back to VM Development Lab doing both system test and performance. Again looking for a challenge, he took on the various cryptographic tests. He didn't call me Wild Bill as much but he still continued to encourage me. Joe would continue to work while fighting cancer. And he fought with an attitude that inspired many for several years.

In May of this year, Joe passed away. I consider him a hero, who is missed by family, friends, and co-workers. Thanks Joe for all you taught me.

Notebook Number 1

Oh boy, are you folks in trouble now. I'm moving offices in the near future and in process of sorting through things. I found my stack of notebooks from over my career. The first thing I noticed is that I think my handwriting was much neater in 1985.

So here's the entry from May 12, 1986, I took the liberty of spelling out a few things that would have been in an odd shorthand:

  Building 8 Raised Floor 4361-5 / 8 Meg
  (let Eric know when finished w/4361-5)
  1. Change uCode and IML
  2. Restore System VMSRES --> 240  VMPK01 --> 241
      VME R2 w/Fortran VMPK02 --> 330  VMPK03 --> 331
  3. RUNFOP FSMIXFOP 20 20 1 182
         061158-065658  Looks Good
  4. Install Read Chain modules, Regen CMS & CP
  5. RUNFOP FSMIXFOP 20 20 1 182
         075653-084153     sleepers
  6. RUNFOP FSMIXFOP 20 20 1 182
        092419-100919  sleepers

This was a good memory for me. As I got a chance to help evaluate some new algorithms on chaining of reads. We might as well go right through this. Building 8 was one of the Glendale Lab buildings. The processor had 8 megabytes of real memory, and was one of the larger configurations at the time. I believe in that time frame, we did benchmarks with the minimum memory requirement of 768KB.

Doing performance runs usually involved dedicated hardware and that meant getting machine shots. In those days, we submitted requests like the week before and someone built a schedule for the following week. This meant planning was important, and there were often deals being made or people looking for spare time. So the second line was a reminder that I was to let Eric know that the machine was free when I finished my run. I don't recall if I had borrowed time from him or he was looking for spare time. Yes, this is the same Eric that would later be my manager.

Step 1 involved changing the microcode. That particular raised floor was shared with Glendale Engineering Processor Lab. Those guys would play with all sorts of things. So to avoid a surprise, we'd often put our own microcode disks in and re-IML (initial machine load or initial microcode load). I remember those were big floppy disks, maybe 8", I'm not sure.

Disk was also at a premium, so we'd often have to restore systems. That's what I was doing in my step 2. What I'd be doing is first a base measurement and then install the new prototype and do a couple measurements. Then we compare the differences. The "RUNFOP" was an exec that did the setup and ran a workload using a tool called FSID (Full Screen Internal Driver). It was a cool tool. It was modifications to CP that allowed it to create simulated virtual machines with hooks in modules like DMKGRF. I believe Galen Miller was the originator. There would later be VM/XA and VM/ESA versions of FSID. For many years, I owned the VM/ESA version. It helped me learn a fair amount about the system as it required mods to 3270 handling as well as the dispatcher and scheduler.

 The parameters meant we'd run 20 userids simulating a CMS workload. So 20 userids in 8MB. The "sleepers" comment was that occasionally some of the simulated userids got out of sync and appeared to sleep or hang for some period of time. It could also mean that there was a problem with the system or the script of commands being executed.

The other thing I noticed from the notes is that the "061158-065658" were timestamps of the measurement. For me to have the measurement period start at 06:11, meant I must have gotten up pretty early. Just the price of working on a great system. :-)

Mail Call

I remember in college checking the mail. We had those little boxes at the mail station in the student union, you know the kind with the little window. I remember getting excited when I you could see some of the light from the other side was blocked, meaning there was some mail in there. It was even better when it was a package slip!! When you had a care package from home, everyone became your best friend.

With my new job at IBM in 1985 was I still happy to see mail in my apartment mailbox. At work we also had 'mail folders' out near the secretarial area. I probably checked that at least twice a day. In those days we still got a lot of paper memos, many with 'buck slips' attached. Those little slips with everyone's name on it and you crossed off your name and bucked it on to the next. Who had someone in their department that was infamous for having the buck slip items get stuck in their mail folder?

Then there was, be still my heart, notes delivered via CMS NOTE command. I could write to people in my own department or around the world via NOTE. The VM systems were interconnected and magic would get them from here to there and back again. We also had tools such as NAMES and RDRLIST to handle contacts and look at the notes others sent you. This was really fun. I mean, this really helped facilitate business.

Due to a tragic error (i.e. user error) some of my earliest mail is no longer available. This is the oldest one in my NOTEBOOK files and triggered a lot of memories:

Later I would be introduced to PROFS (Professional Office System) which would evolve into IBM OfficeVision which ran on a few IBM platforms, but ran best on VM (OV/VM). OV not only included mail and contacts, but also calendar and document facilitation. Oh, and those calendars and documents could be shared and had built in management functions. Hmm, perhaps Lotus and others didn't invent collaboration software after all!

The dependency of corporations on OV/VM was measured by how they reacted when there was a problem. I would see how the "Profs" system being down would go from "the what?" to "How can we get anything done with this running slowly?!!"  Corporations became more and more dependent on this collaboration software. The buck slip started to fade away. The marketing sound bite used in that era was "9 million people log onto OV/VM each day". By today's standards, that may not seem like a lot, but in 1991 that was real penetration in the market place. I believe the largest single VM system in the OV/VM usage was around 20,000 seats. Amazing for that era, and still respectable by today's standards.

The OV/VM team when I worked with them was based out of an IBM lab near Dallas. They were a good team and worked well with the VM team. They cared about performance, and not just speeds and feeds. I remember working with them to implement data collection in the VM monitor data stream.

Besides OV/VM there were a number of other mail related VM tools. Internally within IBM, many people used Mailbox. Outside of IBM, the most well known one would be MAILBOOK or the predecessor, RiceMail. Additionally, z/VM supports SMTP and a very impressive implementation of IMAP.

Over the years, mail and electronic mail became less fun. It became part of "work". The ever increasing snowball rolling down the hill after me. I started capturing the data for the graph shown below a number of years ago for a productivity study of sorts. Can you pick out the months where we had a major deliverable or a critical customer situation? I shouldn't complain about my email load, as I know others get far more than I do.

Email History

I recall a fair amount of gnashing of teeth as we migrated off of OV/VM onto the newly acquired IBM Lotus Notes. The stability of it did improved significantly when it was moved to run on Linux on z/VM. Hooray! My mail is back on z/VM. The numbers above don't include the messages I get from IBMVM and LINUX-390 listserver lists. Those still go to my z/VM userid reader and I don't always consider them work.

Be Our Guest

It's time to talk a little about the guests that have run on VM. I mean what is a host without guests? When I started with VM, it was during the time that the product was more CMS focused, but even then guests were a part of the ecosystem. Earlier blogs mentioned z/VM as a guest, so I won't talk about that too much today.

The first guest I was exposed to was VSE/Systems Package, or simply VSE/SP. Like VM, VSE has a rich history and dates back a few years earlier than VM. VSE/SP was an evolution from DOS/VSE. The product as I knew it was more job or transaction oriented than the interactive CMS. This was real business software. Its use by customers as a guest of VM was very extensive, and there were a lot of programs focused on improving the synergy. I recall a lot of effort on CICS/VM, which was meant to extend the CICS (Customer Information Control System) transaction server into the VM environment as well. Beyond that, there was a lot of work to improve the synergy of VSE and VM. The ability to have VSE and VM handshake on page faults for example. VSE was, and is, an excellent guest.
The VM and VSE interaction was an exciting time. Not only was I exposed to new and interesting technology, but also new people and the Boeblingen Lab. We continue to work with some of those people still on VSE and others on Linux. In particular, I remember Dr. Wolfgang Kraemer, the VSE Performance Leader. I really liked Dr. Kraemer as he was really good at what he did and fun. I still get an involuntary smile whenever I think of him calling me "Billy Boy" with his German accent.

The next guest I would hear about was MVS. It would be described to me as a larger cousin of VSE, offering more function and more scaling. It would later become a component of OS/390, which would evolve into z/OS. Since the introduction of logical partitions to System z, it's become rare for production z/OS workloads to run on VM. However, it is on VM where development of the z/OS system and many of the related software occurs both inside and outside of IBM. The z/OS family has never embraced handshaking technology to the degree VSE did. Part of that is tied to wanting to run the same on VM as it did on bare metal or logical partitions. The height of our involvement with MVS guests may have been in support of Y2K testing. Many customers ramped up their MVS guests, including virtual sysplexes during that time frame. As z/OS Development and z/VM Development often face similar challenges in implementation of new hardware, I've gotten to work with a number of them over the years. Their attention to detail and RAS are impressive.

The 1980s and 1990s also had other guests that I recall. MUSIC (Multi-User System for Interactive Computing aka McGill University System for Interactive Computing) was developed by our friends at McGill University. It was leading edge software that allowed students and professors to create and run their own programs interactively. Another unique guest was MUMPS (Massachusetts General  Hospital Utility Multi-Programming System) . I never ran it myself, but I recall customers who did. Some may classify it as a language environment rather than an operating system, I don't know enough to argue those points. It was used predominantly in the healthcare industry. And then there was also AIX/370, a port of the Unix oriented AIX that today runs on System p.

I must not forget to mention TPF (Transaction Processing Facility) of which the current family is z/TPF. This guest also has a long history going back to the Airlines Control Program (ACP). It's all about fast and efficient transactions. TPF is known for high volume reservation and authorization processing in the travel and finance industries. Like MVS, it is very rare for production TPF workloads to run on z/VM, but there is a lot of testing done on VM. A fascinating aspect of TPF on VM are the vendor additions or extensions to VM, most notably VPARS and VTAPE from VSSI. Which make TPF an even more effective guest.

And then there is Linux. In the late 1990s, I would hear of different discussions of porting it to System z. Then quickly the discussions, turned to rumors, and then reality. Of all the guests, Linux has been the largest game changer. As many of my friends were telling me I should look for another job, that VM was a dead end, there was new life breathed into the VM product. Linux also has the cutest mascot of the guests as well, which doesn't hurt. Linux is more like VSE than some of the other guests. Many of the former VSE team is involved with it in Boeblingen. It's used heavily in production environments. And it has great handshaking with VM.

I joke with a friend that a bed and breakfast might be my next career path. Perhaps the idea intrigues me since I've really enjoyed getting to know the guests staying at Hotel VM.

Pesky End Users, Not

In an earlier blog, there was a comment from James Vincent about being just a 'pesky end user', and I said I'd get back to that. The IBM heritage through the Watson family was customer oriented and embodied by the second of the three original IBM Basic Beliefs:

A desire to have the best customer service of any company in the world.

For the first year or two of my career, customers were like antelope. I had read about them and seen them on television, but never in person. I wouldn't see one until the announce day for VM/SP 5. At that time, it was customary to send programmers from the labs out to branch offices for announce day. I was one of the lucky ones selected, and was sent to Nashville for three days of meetings with customers and IBMers in the field. The customers looked a lot like me, only a little more comfortable in suits. On the last day, a customer jokingly said, "you must be from a development lab, you're wearing a blue shirt." At the time, I was still paying off college loans, so my wardrobe was limited. All the IBMers from the branch office would have fresh white shirts each and every day, so I stood out on that last day. I really should look for my notes from that trip, as I can't remember which customers I visited. I just remember it opened my eyes.

SHARE and other conferences would continue to make an impact on me. Meeting customers and hearing how they actually used our product put a face on both the customer and the business. It was no longer just about implementing a design, supporting an architecture, or adding a new feature. It was about giving our customers what they needed to be productive and successful.

Last year we arranged a couple meetings where customers spoke to the z/VM Development Lab. At first I wasn't sure how many on our team would be able to take time to attend. I was very pleased to see a great turnout for each event. I was even more encouraged to hear some of the feedback as my co-workers got a glimpse of what our customers' lives are like.

One of the most frequent comments was about the awe of realizing the importance of z/VM in a customer enterprise. The idea that millions of transactions, millions of dollars were dependent on z/VM gave them a sense of pride. Hearing of critical monitoring systems running on z/VM gave them a sense of responsibility bordering on duty. While discussing a customer usage of z/VM with one young programmer, he admitted that it made him a little nervous to think of how much people were relying on the code he wrote. I was encouraged, as I knew he got it.

I also heard comments of respect for how much a z/VM system programmer has to know besides z/VM, the various products and systems that make up an overall application solution and business process. The names of some of the products were new to us, let alone how they worked.  We live in our well defined world of z/VM, but system programmers have to journey through different lands of software and vendors.

z/VM system programmers are able to juggle multiple applications and various types of workloads. Their own customers on these z/VM systems aren't simulated workloads, but often people who get vocal and angry when things do not work. They have naming conventions and schedules and change control systems to understand. Most work for corporations in the business of making money. The systems that support or generate revenue are kept running in harmony by these skilled and diligent people.

It's also fun to see how customers use or extend the system in ways we didn't expect. The degree of integration they create is amazing. The example that popped into my head just now is the creative way the NAMES file construct has been used over the years. I've seen customers use it as a mini database, tie it into a disaster recovery solution, or provide a problem tracking system based partly on it. There are also modifications to various components of z/VM, tons of REXX and Pipelines based tools, and exploitation of various user exits.

After hearing and seeing how our customers use z/VM to make the world run, I don't think of them as 'pesky end users'. I think of them more as musicians who take the instrument we craft, and create rich tones from it. It is then that I'm in the audience, listening with pride.

Influential People: Wes Ernsberger

I've worked with a large number of people over the years in the core VM performance team, but I believe Wes Ernsberger is one of the folks that is on top in the longevity list. Wes already had several years in performance when I joined IBM in 1985. That was in the peak of resources, when there were multiple departments committed to VM performance. The headcount for performance decreased significantly after the consolidation of VM from Kingston and Endicott to just Endicott. There would be more reductions during the gold rush also known as the client/server era. The 1990s would see an end to the IBM full employment practice. Reductions and re-balancing of resources in the mid to late 1990s would show the VM performance team drop to a six-pack, and then three: myself, Bill Guzior, and Wes Ernsberger. Fortunately Linux arrived and the team began to grow again, moving towards much healthier numbers.

The era of three was a very tightly-knit team. I remember we had a rule that both Wes and I weren't permitted to vacation at the same time, just so we could ensure coverage for problems. Wes and I would do the bulk of the design evaluation and data analysis. Bill G would keep the systems running and do the bulk of the measurements. VM wasn't strategic at the time, which had downsides. But we were also very free to lead ourselves, determine priorities, and do the most important things. I really enjoyed those years. Wes and Bill are great teammates.

Wes retired a number of years ago now. The first month he was gone, it was very common to hear "I miss Wes." in the hallways. Years later, I still find myself saying it on at least a monthly basis.

Occasionally there is someone in the Development Lab or else where that does something for performance. If it is truly heroic, we bestow upon them official VM Performance Hero Badges. I believe the last set we gave out was for the team that worked through the enhancements to Limit Shares (new SET SRM LIMITHARD settings). When Wes was leaving, I commented that he was beyond being a hero. He was a superhero. The kind that saves the day time and time again, but takes no credit and appears to be the guy next door.

Wes taught me several things, a couple through direct conversation, but mostly through example and being a role model.

A criticism of Wes when it came time to give out awards was often that he hadn't put in as much overtime as others that were up for an award. This was wrong for a couple reasons. Wes was incredible at projecting schedules. We would get a line item to evaluate. He'd look through the basics and start building a plan. He was thorough in this, thinking through the dependencies and factoring in contingencies. Time and time again he'd be very accurate. While others had to deal with the stress of churn in their schedules, Wes was calm and on track. As a result, he very seldom had to work huge amounts of overtime to make up for not planning well.

Related to his use of time, was Wes' work ethic. I have never worked with anyone with as high a true productivity rate. I don't recall Wes ever spending more than 10 or 15 minutes on a distraction such as hanging out at the coffee machine, talking about the football or baseball game from the night before, or just goofing off. It wasn't that Wes was anti-social, he was just focused on work while he was there. Five days a week, one of the first in the office. And then most often you could find him there for a couple hours on Saturday morning.

The idea of planning went beyond just schedules. I'm told that back when runs had to be done by turning over jobs to operators to run or run yourself during a weekly run window, Wes really made sure he had done everything right ahead of time to ensure he got the most out of that run window. Nothing would be worse than turning in a job and getting back the results to find there was a stupid mistake that ruined the run, meaning, you had to wait for the next window to re-run it.

Wes taught me a lot about design analysis. It was fun. Working with design early on to understand the goals and start to create building blocks to understand the potential performance, or lack thereof for a new line item. Through Wes I learned how to break things down into the basic components and build simple spreadsheet models of things. These were great tools for setting and refining expectations.

He really was a great teacher. When the performance team started growing in size again. Wes did a lot of the mentoring and teaching of new people on the team. Patient. He was so patient with everyone. He even built a plan prior to retirement about who would get each piece of his work. He walked everyone through the delegated tasks. I've never seen work transfer done so effectively. I can't thank him enough for that.

Wes had many other fine traits, but I'll end with one I admired the most: he was open to criticism or suggestions. For many of the years that I led the VM Performance Evaluation team, I would write up feedback on each of the team members. I'd provide that feedback to the member and their manager usually just prior to rankings, or "contribution assessment sessions". I felt it was a privilege and an obligation to do this for my team. I remember after doing this one year, Wes stopped over to my office. He thanked me for the positive words, and then asked, "What can I do better? I like knowing I'm doing good things, but knowing what I'm doing wrong is more valuable."   And I knew he was serious. Likewise, he would let me know where I could improve as well. From that, we were a better team. VM was a better product. Earth was a safer planet for computing.  And isn't that what being a superhero is all about?

Correcting Mestakes

Oh, did I misspell mistake? That reminds me. As a relatively new hire, I was looking at some of the workloads we used to evaluate CMS performance. While looking over the commands, I saw one named XXXXX. I scratched my head as I wasn't sure what this command did. I looked around and couldn't find anything on this mysterious command. I finally spoke with one of the more senior members who was an expert in CMS performance, Gary Hine.

Gary smiled, "Ohhh, that's one of the more common commands people use." I looked back at him the same way I used to look back at my older brothers when I thought they were spinning a tale, and asked what it did. Still smiling Gary replied, "Nothing". He then went on to explain that one of the more common commands was a typo which was an invalid command. They had picked this up at some point when studying real users. (Though I know my rate of typos is proportional to how many people are watching me type). I found it interesting that we actually measured the performance of mistakes. It made sense. The quicker it would execute, the quicker the user would realize they made a mistake and correct it.

In System z, it's one of the things we think about a lot (perhaps still not enough, but a lot). That is, what happens when there is a mistake? How obvious is it that there was a problem, and more importantly how obvious is the method to correct things? A few years ago Richard Lewis and Chuck Morse started doing hands-on Labs at SHARE and other conferences. I started attending those sessions to help, but also to learn. I got a whole new perspective seeing a student who made a mistake updating CP Config and trying to sort out what they did wrong. Oh, you typed PAG001 instead of PG0001. The hard part in writing documentation is often clearing your mind so you only know what the reader will know before they start to read the documentation. Yes, not so easy.

The other area where I remember us doing significant performance work in the early days that surprised me was HELP. These days HELP isn't used that often relative to the number of transactions running through a z/VM system. In the past though, it was used quite heavily and we made various improvements over the years. I'm biased, but I still think it's easier in some regards than some other help systems.

For example, the software for this blog. I want to find out... well heck. I can't find any help on this gui based blog editor. So there goes my example. But we've all been there. Lets look at Thunderbird email client instead:

1. Click help 
2. Click help contents
3. Wait for browser to start
4. In search menu, type in 'help set font'
5. Scratch head
6. In search menu, type in 'set font'
7. Look at dozen options provided
8. Pick one
9. Back to thunderbird
10. Click Tools
11. Click Options
12. Click Composition
13. Click General
14. Set fonts
15. See if that did what I wanted

Seems like a lot of clicking.

z/VM HELP has short comings, but it's fairly concise and gives me much of what I need. And the number of steps in navigating has never been so many that I forgot what I needed before I found it.

While assisting in one of the labs I mentioned earlier, I saw a student make a mistake and get an error message. I showed them that they could type HELP message_number and presto. More smiles.

Knowing we all make mistakes, learning from them, and helping others avoid or correct mistakes is something VM has been doing for a long time.

Speaking of help. A shout out to the various people that let me know about typos and other corrections. It is appreciated.

It Depends; and with Linux It Depends Even More

Does anyone know who David Bradley is? He's the one of the designers on the original IBM PC who designed Control+Alt+Delete to trigger a soft reboot. His line that captured his 15 minutes of fame was:

"I may have invented Control-Alt-Delete, but Bill Gates made it famous."

I didn't invent "It Depends". I actually first heard it from Marty Zimelis when he was with Amdahl doing education. I believe it was rule number one on his list of Performance Analyst Rules. (I need to find and post the other ones). But I know some people think I own the royalties on it. When Linux came on the scene, I did add the "with Linux it depends even more."

Now, I know some people consider "It Depends" to be a cop out or a way to avoid answering a question. I don't mean for it to be that. It's really just a preamble for me which translates into:

Be careful, there is not necessarily a single or unqualified answer to your question. I am going to ask you to think about some things and understand the environment and factors that play in that environment. I will be very sad if you just take one aspect of what I say and run off and misuse that information. If my response involves a range of values, you need to know why the answer may lean towards one or the other end of that range.

As you can imagine, it's much easier to say "it depends" than the above. Some of you have caught on and after hearing "it depends", ask what are the dependencies?

At some point in my career, I was faced with the reality that there are a set of people out there that actually listen to what I say. I became more aware of the idea that what I said had weight. I began to be more cautious about giving absolutes, and quicker to point out whether what I was sharing was conjecture, provable, or proven fact. Being clear on the difference of those has become very important to me in problem determination.

I've been in situations with major problems. Then Jimmy will say something in the hallway such as,  "I wonder if it's the network?".  Some listener will repeat it as "Jimmy thinks the problem is the network.", which becomes "Jimmy identified the problem as the network."  Heavy sigh.

I often hear that z/VM or System z is too complex or difficult. And I often think, compared to what? I've never met anyone of average intelligence that had trouble navigating around the CMS environment and editing in a week or two; writing useful REXX scripts in a month; understanding the fundamentals of z/VM system programming in a few months. I hear the 3270 is too difficult. Seriously? It's the same keyboard as every other system, just less mousing. I lose this argument a lot with some people. Oh well.

(I'd really like to see a study on the ratio of time spent typing vs. clicking for the average z/VM system programmer, Linux Admin, and Windows Server Administrator. My conjecture is that the ratios would be closer than some people believe.)

Please don't get me wrong. I'm all for simplification. However, there's a difference between putting a GUI frontend on SET SRM values and simplification. What we want to avoid is eye candy for sake of eye candy; and making it simpler to screw up the system.

I always had trouble putting my finger on why some people seem to have so much trouble with Linux on z/VM. Until one day, Steve Gracin from IBM ATS gave me the answer. "It doesn't matter what system or interface you're using, you have to know what you're doing." And in some cases, we're just not willing to learn that. (See early blog entries about the value of mentors and community). And in other cases, we're not willing to admit we don't understand. Most systems that do significant things are complex and therefore, it's our jobs to understand aspects of the system.

So back to "it depends". Often, that's the start of thinking about what factors come into play. What objectives are there? The beauty of Linux is that it is so flexible; the challenge of Linux is it is so flexible. I work with customers running 256MB virtual machines and virtual machines over 100GBs. What is right for one, may be wrong for the other. And someone needs to think about the difference. Understanding the dependencies of ones system is a key aspect to knowing what you're doing. And if you want a job where thinking isn't required, I suggest systems programming, on any platform, is not for you.

Customers, you should know you're not the only ones I give the "it depends" line. Inside, I do the same thing in planning and design meetings. A new solution or line item is proposed and I know it will have different degrees of benefit for different customers. The answer to "what is the benefit to the customer?" is "it depends". Now, what are those factors. Understanding those, helps us to understand the scope of benefit to customers and the return on the risk. Some of our less successful ventures have been where we built everything and then asked the question.

So I hope you all have a better appreciation for what's behind "it depends".

How a Cat Works

"If you try and take a cat apart to see how it works, the first thing you have on your hands is a non-working cat.” - Douglas Adams

Yesterday, I discussed my love for second-level systems, and how the use of them makes us understand z/VM and gives us a place to experiment and learn. Earlier in this blog, I mentioned the aspect of architecture purity. I'd like to take a step down into an aspect of z/VM that not everyone has the need or time to examine. But it's powerful stuff that allows you to see inside the cat without taking it apart.

DISPLAY, TRACE, and STORE. These three CP commands are the keys to magical places. I didn't learn these reading a book, but again by seeing them used early in my job.

I seem to recall seeing DISPLAY used for both virtual and real to check eye catchers in modules. In Performance we'd often get new code to test and actually looking at it in memory was best way to make sure it was what we thought it was suppose to be.

I remember for some VTAM performance measurements, we'd want to disable some of the timer loops in the VTAM code. I remember being shown how to run a trace to see the addresses of where the code that triggered the loop existed, and thinking that's pretty cool. But then my mentor went on to disable the loop by storing branch into that code. This to me was like milking a cow while it was moving. And a whole new perspective opened up to me.

I would smile as I asked a developer how a particular diagnose or instruction like STSI works. Some, instead of getting out the full Principles of Operation, would just grab a virtual machine, drop into CP, store the opcode in memory perhaps change a register, and begin execution at that instruction. Geeky and yet fashionable.

In the late 1990s and early 2000s it seemed everyone working on VM had been working long enough that everyone knew these things. Linux changed that. A couple years ago a Linux developer from one of the distributions found himself working on System z. At a SHARE conference, he asked me if there were any utilities in z/VM to help with problem determination. He worked on an area of the kernel where the problems were often at a point where the kernel wasn't fully up or that tracing from Linux just wasn't feasible. So at a break between sessions, we sat down and I just started showing him some of the things you can do with Display, Trace, and Store. His eyes lit up. I had just armed him with various methods to solve a myriad of possible problems.

There are many examples. One last one I'll mention was working on the raised floor with Geoff Blandy. He took it a step too far when he started modifying the CP that we were running on at the time, with Store Host. Now I was milking the cow while riding on it; and I really didn't think that was particularly safe. At first it was just one or two instructions. This will be quicker than rebuilding he assured me. I think when we were done, he had put in a few dozen instructions this way while I kept an eye on RTM to see if anything was running. Somehow it all worked; or at least didn't crash. Of course I wasn't thinking of the Display Host when at the end he asked, "Ok, just take what we did and modify the source".

This approach is a bit unorthodox, but then so was Geoff in many ways. I did learn from him, and I do have my share of EXECs that modify code or constants in z/VM. All thanks to the keys of Display, Trace, and Store. A powerful aspect of the product that over the years has contributed not just to z/VM, but z/OS, VSE, Linux on System z, actually every OS that ever ran on it.

Spending Time in the Fort

Growing up we all had moments where we pretended to be a fireman, or policeman, or the Lone Ranger, or Captain Kirk, or a System Programmer with our own z196. They tell me playing is one of the ways we learn. I've found it interesting over the years to watch how different people learn.

I believe VM helps us learn how VM works, as well as System z in general. One of my early assignments was to build my own second-level system. And it's still something I recommend to our new hires no matter which discipline (development, system test, performance).  You learn a lot about how the system is 'put together' in the process. Once you have your second level system, you have a new proving ground or playground.

Most of us in the Development Lab these days are still somewhat focused or specialized. A second level system allows us to explore other aspects. For example, we most often think about a z/VM system that is up and running, and these days they just run and run. So we don't often see initialization or shutdown. Both are important to understand and our second-level systems give us a chance to see and experience that. I've heard that some customers schedule shutdown/IPLs just so there operators get a chance to practice, particularly IPLing the system. Lots of things can be learned or practiced by doing this on second-level system.

Once I had my second-level system, I was able to try almost any command, especially the harmful ones. Seeing is believing. Does cold start really erase all the user spool files? Well, by golly, it does. Good thing I did that second-level. Occasionally, I need to give a customer a procedure to follow. While our documentation is good and I've been around a long time, I still often test it out second level. This way I can match the customer level better, avoid customizations that may be on my production first level, and validate things closer. One time last year when I skipped this, I missed an important aspect of the procedure related to privilege classes.

Beyond this, I found my second-level system gave me a reason to build automation or customization around it. It's interesting to wander around the Lab and see that a lot of people have written their own tooling for their second level systems. Some of it is incredibly elaborate. Using temporary disks or virtual disks in storage for their page space or perhaps also spool space. Various programs to generate loads or start of one-off applications. Just amazing the creativity that occurs in that space.

For those that love VM, their second-level system is a place of fun. At times it makes me think of the fort we built in the neighbors barn. The hours of creativity and fun spent in that fort, and in my second-level system. Spending time in both of them has made me late for dinner more than once.

Influential People: Eric Strom

I first met Eric as I was being introduced to folks in the VM Performance departments (yes departments - at the time there were two in Endicott and multiple ones in Kingston). He immediately busted on me because I went to Pitt and he was a Penn State grad, the two being major rivals at the time. I learned a lot from Eric about the discipline of measurements and analysis. A few years later Eric would move into management and I would be working for him. At the risk of giving him a big head, I consider him to be the most influential manager in my career.

Eric believed in me more than I believed in myself. He was the manager that decided I needed customer exposure and put me in to be a rep to SHARE, which would change me significantly. He's the model of the manager who challenged his people, but didn't set them up for failure. Years later we would joke about me going to SHARE and being so green. I asked him if he had realized how I was not ready to fill the shoes of Paul Van Leer. Eric said he knew it, but he also knew that I didn't realize what I was getting into yet, so it would be okay.

Eric would continue to push me, give me opportunities, but also be in my corner when needed. He helped me improve my communication, introduced me to other technical leaders, and when necessary, tell me to suck it up.

After managing VM Performance, I believe his next job was in the crit sit office (critical customer situations). We would occasionally work together on situations; and he would continue to encourage me. He would move around over the next ten or fifteen years, back into management, back out. Eventually he would find himself in performance again, only this time for z/OS.

I also credit Eric with being the key to IBM embracing the Internet. Now you may be saying "Whoa Bill, now that's a stretch!". Just give me a minute to explain. In the late 1980s and early 1990s there was a very bright young man named David Grossman working in VM Performance. Coming out of Michigan, he was a Unix-weenie at heart, and a very self-motivated, hard worker. He would tell Eric that while he enjoyed the performance work, his heart wasn't really in it. Eric already knew that, but also knew that most managers would kill to have someone so talented. What happened was Dave gave Eric a number of quality years in VM, and then Eric worked hard to find him a great job as one of the IBM liaisons at Cornell University in a type of super computer or theory lab.

Hold on, I'm coming to the part about the internet. Dave was working away at Cornell in 1994 during the winter Olympics (Dave was also a great athlete, often riding bike between Endicott and Ithaca for fun after work). Dave discovered that a web site run by another company was taking the IBM Olympic data and serving it up on their own web site. That launched him into action to get IBM executive attention and ultimately IBM into the e-game. I can't do the story justice, but there was a excellent Harvard Business Review article that describes it all. 

All of that as a result of mutual trust and loyalty between a manager and their employee.

Some of you might recognize the name Strom, as in Earl Strom, long time NBA official. In 1995, Eric gave the Hall of Fame acceptance speech for his father, who had passed away the year before. Eric recently found that someone put a copy of it on YouTube; and I watched it the other night. Eric spoke of his father's love of the game and his respect for the players and others involved in the game.

While I'm a horrible basketball player, I'd like to think Eric coached me some as a software engineer.
Love and respect in one's work is possible for NBA officials, and software engineers.

Love and respect. Trust and Loyalty.

A Boy and his Toys

I always have a few toys in the office. I've been reminded you have to get older, but you don't have to grow up. I like to have fun. But that's not the only reason I keep some of these in my office. You see, I work with some brilliant people, the kind of people that have incredible memories and amazing cognitive skills. Their minds work faster than mine. So when I have a complex topic to discuss with them, I ask them to stop over. And then I make sure some of the toys are out in plain sight. This has worked well over the years for several people. Damian Osisek is my favorite target in this respect. He'll start working on the maze or a puzzle while I quiz him about some aspect of architecture, cache congruency classes, super scalar processing, or SIE implementation details. I liken this to knee-capping a processor to lower the capacity rating. The amazing thing is, Damian will solve puzzles during the discussion. While I own some of the puzzles, I can't solve them even when focused solely on them. I'm fortunate to work with people that need to be knee-capped. Oh, that sounds bad, but you know what I mean.

I appreciate vendor give-aways at conferences and trade-shows, some of my better toys. You might be able to pick out the Sterling Software Yo-Yo in the picture above. Carol Everitt recently found it and gave it to me. It used to light up. We're working on restoring that aspect of it. :)

I also have had some LEGOs in my office from time to time. (No surprise to some of  you.) These have come in handy as well. I once used them to illustrate a point back when there was some discussion about how to rank or appraise individuals, and the whole idea of measuring contribution. I handed a single LEGO brick to the manager and suggested they build something. They caught on quickly that you can't build anything too interesting with a single brick. But a set (team) of bricks is what allows greater things to be built. Last I looked, the manager still had the LEGO on their desk.

The Cornfield Workload

Nostalgia time again. How many remember the old orange-covered Technical Bulletins that the System Centers used to produce? (How many actually remember when we had regional Area System Centers?). Back in those days, the VM Performance report was a collaboration of the VM Performance Evaluation team in the Development Lab and one of the System Centers. We'd pull together the bulk of the data and information and then turn it over for additional writing and editing and formalizing to one or more people in the centers.

Seeing my name on one of the inside pages listing the 'contributors' was awesome. And Mom got a copy that year for Christmas. Having a last name that starts close to the beginning of the alphabet has pros and cons. On one hand, I was one of the first names on the list. On the other hand, when people were looking for someone to help answer questions, I was one of the first names on the list. I recall getting an email shortly after we published a performance report for one of the last VM/SP releases. Someone in the field had read the report and wanted more information on the CORNFIELD benchmark. Yes, the CORNFIELD benchmark. Don't worry, I had not heard of it either. I wrote back asking which CORNFIELD benchmark? To which they replied, "The one you mention in the VM/SP Performance Report".

Up to that point, I had thought I wasn't new to performance anymore and knew pretty much what all we did. But suddenly there is a whole benchmark that I had never even heard about and which apparently was in our Performance Report. I started asking around. I got some ideas, but they were all stretches. There was a workload we ran for VSE called PACE COB. Someone suggested maybe they think COB as in corn on the cob? (I actually don't recall what the PACE COB meant. I think it used POWER and I remember we eventually replaced it with the DYNAPACE workload). Then someone discussed maybe it was part of the CMS interactive workloads we ran to load test VM with a simulation of multiple CMS users doing program development. One of the simulations included using XEDIT to create a FORTRAN programs, compile them, and run them. Get ready for this. One of the FORTRAN programs was one that computed the coverage when spreading manure on fields. Maybe cornfields?!!

I went back to my new friend in the field with my two new possibilities. Which were both wrong. To make things easier (and perhaps because he was starting to think he should have started at the other end of the alphabet), he pointed me to the exact page in the report where we talk about CORNFIELD. The page corresponded to a section where we reported on CMS performance, actually providing pathlength and page reference data on various CMS commands. I started reading through the table of commands: SET, ACCESS, COMPARE, CORNFIELD, ... 

CORNFIELD?!!! Did you figure it out yet? This was in early days, at least for me, of proof or spell checking. The software decided COPYFILE should have been CORNFIELD. I laugh about it now, but at the time, many of us looked at each other asking how did we miss that? It was a good lesson learned.

Thinking back on this story, I also reflect fondly on the desire of those involved to report accurately and completely on the performance of a new release. We took pride in it. We worked well with other teams since in those days not everyone could 'publish' information. It was an expense so you wanted to get it right. These days the requirements to publish information are down to just 'can you operate a kyebored?'. Or maybe there aren't any requirements.

The LEGO of System z Software (aka Pipelines)

As I look back over the things that impacted my productivity as a VM performance analyst and tester, I find myself coming back to CMS Pipelines. Read Melinda Varian's paper on the history of z/VM for more details on how IBM's John Hartmann came up with the idea, and how it found its way into the product.

I had heard tell of IBMers using Pipelines in late 1980s, but it was always bit-jockey types and I had assumed it was the same ones who got giddy over the idea of using a plugboard to program or using dials on control panels to set IPL addresses. In other words, people who were either way smarter than me or loved this stuff way more than me (or perhaps both). I didn't really start to embrace Pipelines until it was incorporated into VM/ESA V1R1.1 (CMS 8).

My love for Pipes grew exponential in that first year. At first, my reaction was, "This is interesting". Then I started to see what others were doing with it, and my reaction moved towards, "This is cool". Finally, as I began to use it for heavy lifting in my analysis work, "This is awesome".

I still remember trying to do some analysis to reduce some of VM/ESA's master processor usage in early 1990s, this was the time frame where larger n-ways were starting to appear on the horizon. The use of OV/VM was becoming heavier as well, and made many requests of CP that were master only related. It was an important area to improve. I started to think that to understand some of master only processing I would need to mine more data out of the CP trace table. I could gather large amounts of data through TRSAVE, but then what? This wasn't like dump analysis where the answer is typically within the last 200 trace entries. This was a place where Pipes became my friend as I would massage the data in various ways. Continuing to narrow in on the culprits.

This Pipes-aided analysis led to two changes in VM/ESA V1R2.1 that significantly reduced the master processor usage. That led to one of my first formal IBM awards. I'd like to thank the academy, my parents, and Pipes; without their support I would not have reached the solution as easily.

Pipes would come to the rescue at various times after that. I would write a little utility to read the CP trace table on a running systems and produce some stats that was useful for performance analysis in some extreme cases. The MONVIEW and Friends utilities now shipped as samples in z/VM are Pipes-based. Some of which actually involved user-written stages. At one point John even provided a Pipelines stage to directly read *Monitor data, appropriately named STARMON. Now I was giddy!

I never rose to Master Plumber status in the fraternity of Pipelines coders, but I loved them just the same. I admit playing with LEGO bricks also. The beauty is anyone can build what their mind sees. The child that creates a LEGO model car made up of 9 bricks and the man-child that builds a 10,000 brick replica of the United States capital building each go one brick at a time. And there would be much more stress if the bricks hadn't been designed to work together as they do. The possibilities are amazing. Just five 8-studded bricks of the same color can be put together in over 10 million configurations. Perhaps that's why no two Pipes programs are exactly alike.

I also just noticed. Both CMS Pipelines and LEGO bricks originated in Denmark. Keep plumbing and keep building.