From: glen mccready To: Dead Beef <0xdeadbeef@substance.abuse.blackdown.org> Date: Fri, 28 Jul 1995 20:13:04 -0400



---------- Forwarded message ----------
Date: Fri, 28 Jul 1995 12:05:02 -0400
From: Keith Bostic <bostic@bsdi.com>
To: /dev/null@python.bostic.com
Subject: VAXen, my children, just don't belong some places.


VAXen, my children, just don't belong some places. In my business, I am
frequently called by small sites and startups having VAX problems. So when
a friend of mine in an Extremely Large Financial Institution (ELFI) called
me one day to ask for help, I was intrigued because this outfit is a
really major VAX user - they have several large herds of VAXen - and
plenty of sharp VAXherds to take care of them.

So I went to see what sort of an ELFI mess they had gotten into.  It seems
they had shoved a small 750 with two RA60's running a single application,
PC style, into a data center with two IBM 3090's and just about all the
rest of the disk drives in the world. The computer room was so big it had
three street addresses. The operators had only IBM experience and, to
quote my friend,  they were having "a little trouble adjusting to the
VAX", were a bit hostile towards it and probably needed some help with
system management. Hmmm, Hostility... Sigh.

Well, I thought it was pretty ridiculous for an outfit with all that VAX
muscle elsewhere to isolate a dinky old 750 in their Big Blue Country,
and said so bluntly. But my friend patiently explained that although
small, it was an "extremely sensitive and confidential application."  It
seems that the 750 had originally been properly clustered with the rest
of a herd and in the care of one of their best VAXherds.  But the trouble
started when the Chief User went to visit his computer and its VAXherd.

He came away visibly disturbed and immediately complained to the ELFI's
Director of Data Processing that, "There are some very strange people in
there with the computers." Now since this user person was the Comptroller
of this Extremely Large Financial Institution, the 750 had been promptly
hustled over to the IBM data center which the Comptroller said, "was a
more suitable place."  The people there wore shirts and ties and didn't
wear head bands or cowboy hats.

So my friend introduced me to the Comptroller, who turned out to be five
feet tall, 85 and a former gnome of Zurich.  He had a young apprentice
gnome who was about 65. The two gnomes interviewed me in whispers for
about an hour before they decided my modes of dress and speech were
suitable for managing their system and I got the assignment.

There was some confusion, understandably, when I explained that I would
immediately establish a procedure for nightly backups. The senior gnome
seemed to think I was going to put the computer in reverse, but the
apprentice's son had an IBM PC and he quickly whispered that "backup"
meant making a copy of a program borrowed from a friend and why was I
doing that? Sigh.

I was shortly introduced to the manager of the IBM data center, who
greeted me with joy and anything but hostility. And the operators really
weren't hostile - it just seemed that way.  It's like the driver of a Mack
18 wheeler, with a condo behind the cab, who was doing 75 when he ran over
a moped doing it's best to get away at 45.  He explained sadly, "I really
warn't mad at mopeds but to keep from runnin' over that'n, I'da had to
slow down or change lanes!"

Now the only operation they had figured out how to do on the 750 was
reboot it.  This was their universal cure for any and all problems.  After
all it works on a PC, why not a VAX?  Was there a difference?  Sigh.

But I smiled and said, "No sweat, I'll train you.  The first command you
learn is HELP" and proceeded to type it in on the console terminal.  So
the data center manager, the shift supervisor and the eight day operators
watched the LA100 buzz out the usual introductory text.  When it finished
they turned to me with expectant faces and I said in an avuncular manner,
"This is your most important command!"

The shift supervisor stepped forward and studied the text for about a
minute. He then turned with a very puzzled expression on his face and
asked, "What do you use it for?" Sigh.

Well, I tried everything.  I trained and I put the doc set on shelves by
the 750 and I wrote a special 40 page doc set and then a four page doc
set. I designed all kinds of command files to make complex operations into
simple foreign commands and I taped a list of these simplified commands
to the top of the VAX. The most successful move was adding my home phone
number.

The cheat sheets taped on the top of the CPU cabinet needed continual
maintenance, however. It seems the VAX was in the quietest part of the
data center, over behind the scratch tape racks. The operators ate lunch
on the CPU cabinet and the sheets quickly became coated with pizza
drippings, etc.

But still the most used solution to hangups was a reboot and I gradually
got things organized so that during the day when the gnomes were using
the system, the operators didn't have to touch it. This smoothed things
out a lot.

Meanwhile, the data center was getting new TV security cameras, a halon
gas fire extinguisher system and an immortal power source. The data center
manager apologized because the VAX had not been foreseen in the plan and
so could not be connected to immortal power.  The VAX and I felt a little
rejected but I made sure that booting on power recovery was working right.
At least it would get going again quickly when power came back.

Anyway, as a consolation prize, the data center manager said he would have
one of the security cameras adjusted to cover the VAX.  I thought to
myself, "Great, now we can have 24 hour video tapes of the operators
eating Chinese takeout on the CPU." I resolved to get a piece of plastic
to cover the cheat sheets.

One day, the apprentice gnome called to whisper that the senior was going
to give an extremely important demonstration. Now I must explain that what
the 750 was really doing was holding our National Debt.  The Reagan
administration had decided to privatize it and had quietly put it out for
bid. My Extreme Large Financial Institution had won the bid for it and
was, as ELFI's are wont to do, making an absolute bundle on the float.

On Monday the Comptroller was going to demonstrate to the board of
directors how he could move a trillion dollars from Switzerland to the
Bahamas.  The apprentice whispered, "Would you please look in on our
computer? I'm sure everything will be fine, sir, but we will feel better
if you are present.  I'm sure you understand?"  I did.

Monday morning, I got there about five hours before the scheduled demo to
check things over. Everything was cool. I was chatting with the shift
supervisor and about to go upstairs to the Comptroller's office.  Suddenly
there was a power failure.

The emergency lighting came on and the immortal power system took over
the load of the IBM 3090's.  They continued smoothly, but of course the
VAX, still on city power, died. Everyone smiled and the dead 750 was no
big deal because it was 7 AM and gnomes don't work before 10 AM. I began
worrying about whether I could beg some immortal power from the data
center manager in case this was a long outage.

Immortal power in this system comes from storage batteries for the first
five minutes of an outage.  Promptly at one minute into the outage we hear
the gas turbine powered generator in the sub-basement under us
automatically start up getting ready to take the load on the fifth minute.
We all beam at each other.

At two minutes into the outage we hear the whine of the backup gas turbine
generator starting. The 3090's and all those disk drives are doing just
fine.  Business as usual. The VAX is dead as a door nail but what the
hell.

At precisely five minutes into the outage, just as the gas turbine is
taking the load, city power comes back on and the immortal power source
commits suicide.  Actually it was a double murder and suicide because it
took both 3090's with it.

So now the whole data center was dead, sort of.  The fire alarm system
had it's own battery backup and was still alive. The lead acid storage
batteries of the immortal power system had been discharging at a furious
rate keeping all those big blue boxes running and there was a significant
amount of sulfuric acid vapor. Nothing actually caught fire but the smoke
detectors were convinced it had.

The fire alarm klaxon went off and the siren warning of imminent halon
gas release was screaming.  We started to panic but the data center
manager shouted over the din, "Don't worry, the halon system failed its
acceptance test last week. It's disabled and nothing will happen."

He was half right, the primary halon system indeed failed to discharge.
But the secondary halon system observed that the primary had conked and
instantly did its duty, which was to deal with Dire Disasters.  It had
twice the capacity and six times the discharge rate.

Now the ear splitting gas discharge under the raised floor was so massive
and fast, it blew about half of the floor tiles up out of their framework.
It came up through the floor into a communications rack and blew the cover
panels off, decking an operator. Looking out across that vast computer
room, we could see the air shimmering as the halon mixed with it.

We stampeded for exits to the dying whine of 175 IBM disks.  As I was
escaping I glanced back at the VAX, on city power, and noticed the usual
flickering of the unit select light on its system disk indicating it was
happily rebooting.

Twelve firemen with air tanks and axes invaded. There were frantic phone
calls to the local IBM Field Service office because both the live and
backup 3090's were down. About twenty minutes later, seventeen IBM CEs
arrived with dozens of boxes and, so help me, a barrel. It seems they knew
what to expect when an immortal power source commits murder.

In the midst of absolute pandemonium, I crept off to the gnome office and
logged on. After extensive checking it was clear that everything was just
fine with the VAX and I began to calm down. I called the data center
manager's office to tell him the good news. His secretary answered with,
"He isn't expected to be available for some time.  May I take a message?"
I left a slightly smug note to the effect that, unlike some other
computers, the VAX was intact and functioning normally.

Several hours later, the gnome was whispering his way into a demonstration
of how to flick a trillion dollars from country 2 to country 5.  He was
just coming to the tricky part, where the money had been withdrawn from
Switzerland but not yet deposited in the Bahamas.  He was proceeding very
slowly and the directors were spellbound. I decided I had better check up
on the data center.

Most of the floor tiles were back in place. IBM had resurrected one of
the 3090's and was running tests.  What looked like a bucket brigade was
working on the other one. The communication rack was still naked and a
fireman was standing guard over the immortal power corpse. Life was
returning to normal, but the Big Blue Country crew was still pretty shaky.

Smiling proudly, I headed back toward the triumphant VAX behind the tape
racks where one of the operators was eating a plump jelly bun on the 750
CPU. He saw me coming, turned pale and screamed to the shift supervisor,
"Oh my God, we forgot about the VAX!" Then, before I could open my mouth,
he rebooted it.  It was Monday, 19-Oct-1987.  VAXen, my children, just
don't belong some places.
		-- Mike O'Brien, obrien@aero.org