Mainframe Talent: An Endangered Species

Big iron still gets big use, but lack of personnel is a growing problem.

It's estimated that 60% of corporate America's data sits in mainframes. IBM (NYSE: IBM), the premier mainframe vendor, says its big iron business grew 25% last year. And for the first time in more than two decades, mainframe pricing went up. All of these are signs that mainframes are still vital IT systems.Nonetheless, General Mills Inc. is shutting down its Amdahl mainframe. The $13 billion maker of such foods as Cheerios, Betty Crocker cake mixes, and Yoplait yogurt is getting rid of the mainframe because it can't find replacements for its retiring mainframe experts, and it doesn't want to pay the high prices mainframe consultants command.

It's clear that there's a human-resources problem when it comes to the mainframe. According to studies done by Meta Group, 60% of the people in data centers housing mainframes are 50 or older; 10% are 60 or older. When comparing IT staffers who work on Unix and Windows servers with those who can work on mainframes, a striking age gap emerges: for example, 22% of Unix and Windows staffers are around the age of 30, compared with just 5% of mainframe workers who are around that age. Only 8% of Unix-and Windows-trained IT employees are 50 or older, according to the studies. Meta Group analyst Rob Schaefer says the mainframe isn't as sexy a system as Windows, Unix-based Web servers, or Linux. "For our mainframe customers, attrition is disproportionate compared to the other platforms," he says, "and no new blood is coming in."

Half the mainframe talent has retired from Mission Linen Supply, Szerwo says.
However, not everyone will abandon mainframes in the foreseeable future. That's because of the huge investments companies have already made in mainframe applications and the millions of dollars it can take to migrate those applications to other platforms or host them at outsourcing shops.

There continue to be efforts designed to help businesses that are keeping their mainframes. For one, the mainframe is becoming more automated, so it requires fewer IT specialists to run and maintain it. Now, says Gartner industry analyst John Phelps, many companies are maintaining their mainframe environments, rather than developing new applications for them, and that typically requires about one-third the number of people. Some colleges are teaching mainframe skills and even getting funding from mainframe vendors such as IBM.

Also, tools are available to help ease integration between mainframe data, typically residing in DB2 and Cobol applications, and client-server and Web applications. Xbridge's Host Data Connect software, for example, can convert Cobol data into a Windows format.

The number of people with mainframe skills for such things as the MVS operating system, Cobol, and the Customer Information Control System started to erode during the '80s, says Ron Hankison, president and CEO of Xbridge. "By the end of that decade, only a few people knew what was going on with MVS, and eyes started to glaze over at the mention of Cobol, CICS, and the mainframe's transaction monitor," Hankison says. Still, he says there are 5 million to 9 million new lines of Cobol code written each year. "It's almost impossible to get rid of the mainframe," he says.

IBM also is trying to ease mainframe pressures, Gartner's Phelps says. It's providing grants to 40 universities and colleges, including Colorado State University, Marist College, and the University of Georgia, so they'll teach mainframe-oriented skills as part of their curriculum. It wouldn't disclose the value of the various grants. Not surprisingly, most of the IBM-funded education centers on Linux on the mainframe. IBM has ambitious plans under way to develop a Linux-only mainframe. The plans are to merge Linux with the hardware remote-access service functionality of the mainframe. Analysts say that a Linux mainframe lacks the heart of an IBM mainframe, the z/OS operating system, but it has the identical high-end hardware capabilities.

A wonderful product called Image Focus for z/OS

IMAGE Focus is a software product from NewEra Software, Inc. It uses proprietary technology to track changes to operating system parameters, and provides the ability to do a ’Virtual IPL’ of a z/OS image to ensure that a real IPL will not encounter invalid parameters.

Image Focus runs as a started task under z/OS. It works by providing a series of “inspectors”. These inspectors - OS-Inspector, Sysplex-Inspector, JES-Inspector and VTAM-Inspector - perform the actual checking of each component.

IMAGE Focus is composed of three components, each of which may be optionally installed and operated independently of the others. These components are:

Recovery : The IMAGE Focus recovery started task is designed to maintain its own independent communications subsystem and provides ISPF application support to a single locally-attached non-SNA 3270 console. This provides the ability to logon, check, and fix problems when VTAM, JES, or TSO are unavailable at IPL time. This potentially avoids the need to perform unnecessary IPLs following the discovery of a problem; that is, an installation may “forward” fix problems found early in the IPL process.

Multi-user : When installed as a VTAM application to support multiple simultaneous users, IMAGE Focus maintains a multi-user started task. This provides the ability for users to logon using their external security product controlled userids and perform checking, browsing, and editing of system parameters.

Background : The IMAGE Focus background started task reports IPL changes that would result in future IPL

failures to a designated user or group through the TSO Broadcast Facility or via E-mail. These notices are sent at intervals controlled by IMAGE Focus, or optionally by the installation’s job scheduling package.

The checking process starts with the LOADxx member whose value is entered by the user, or as a variable if run in batch. It checks that IPL text exists on the IPL volume and that the SYS1.NUCLEUS data set can be opened. It processes each PARMLIB and PROCLIB member for syntactical correctness and related data sets for referential integrity and attribute characteristics. This process may be run in the foreground by a user, in batch, or on a continuous basis. During this continuous inspection, the inspector will determine if there are any problems with system definitions, warn of potential failures, and compile reports on suspect components. These reports identify errors in critical data sets, document their locations, and catalog the data sets that may prevent a successful future IPL.

This checking extends to JES, VTAM and TCP/IP and the product provides the ability to create installation-specific checking routines over and above those supplied.
The product provides a “what if” option that includes support for new releases of the operating system. This allows the user to plug existing IPL parameters into the next version of z/OS and check that they are still viable.

Key Features

The Image FOCUS Core, IFOCORE, includes the following:

• Operating System Inspection - Supporting all releases of MVS and z/OS through V1R8, evaluates operating system configuration components for weaknesses that will cause IPL failure.

• Dynamic Element Inspection - Pinpoints changes in the LnkLst, LpaLst and ApfLst that have not been updated in the system configuration.

• System Component Inspection - Allows for the inspection of individual system components. Within the Operating System, for example, individual z/OS Parmlib Members maybe inspected.

• New z/OS Release Analysis - Isolates areas in your current operating system configuration that will need to change in order to become functional when used with a new release of z/OS.

• Controlled Image Monitoring - Fully automates the interval inspection of Controlled Images and the reporting of results.

• Blueprinting & Change Detection - Blueprints the z/OS configuration for Controlled Images, detecting and reporting changes

• Subsystem Recovery - Provides access to ISPF system repair functions when TSO is not available. This Image FOCUS Core function is often called 'NOTSO".

z/OS Catalog fault finding and fixing

This page describes how to use raw IDCAMS to investigate and fix catalog problems. It is usually easier to use a maintenance product for this. The shortcuts below will take you to specific points on the page.

Listing catalog information

The ICF catalog structure is complex, so if you have a problem, the first thing to work out is exactly which component is broken. Typically you'll be passed a problem ticket from someone who cannot access his data. Start by querying the catalog status for an affected file, by using the listcat command. The easiest way is to pick up a file list from ISPF option 3.4, then type the following line command against a file.

LISTCAT ENT(/) ALL

Here's a tip which you probably know already. You'll find yourself using the listcat command all the time. Shortcut it by sticking the following clist into a command library in your logon procedure. Type TSO LISTA to find an appropriate library. Add the clist as member name LC.

PROC 1 MEMBER
LISTC ENT(&MEMBER) ALL
EXIT CODE(0)

Now when you want to list the catalog entry for a dataset, all you need to do is type LC against it. The LISTCAT output tells you; which catalog the dataset entry is in, which volume the catalog thinks the dataset is on, and SMS class information. You get a lot more detail for VSAM datasets.

Checking the physical condition of a Catalog

If you are getting a lot of problems in the same catalog, then it is possible you have a physical error on the catalog dataset (it is just a VSAM KSDS). Physical errors often cause the catalog backups to fail too. If you use LISTCAT ENT(/) ALL against a catalog, it will tell you which master catalog it is in, which volume it is on, and a list of all the aliases which point to it. To see the full VSAM information, you need to use listcat ent(/) catalog(/) all

To check the physical status of the catalog, run the following IDCAMS statements in batch

EXAMINE NAME( -
CATALOG.ICFUSER.VOL001 - ) -
INDEXTEST DATATEST -
ERRORLIMIT(50)

and hopefully you see the output

IDC01700I INDEXTEST BEGINS
IDC01724I INDEXTEST COMPLETE - NO ERRORS DETECTED
IDC01701I DATATEST BEGINS
IDC01709I DATATEST COMPLETE - NO ERRORS DETECTED

If the examine reports problems, then you might have to recover the catalog, as described on the next page. However, before you start to recover, try an IDCAMS verify first, as it might just be that the file was not closed properly. The syntax is

VERIFY DATASET(catalog.name)

However, if you have a catalog maintenance product onsite, it will have functions to allow you to fix physical catalog errors. The maintenance product section has more details.

Checking the logical Catalog condition

The following IDCAMS command will check that the dataset entries in a catalog are complete. For example, it will check that VSAM truenames are associated with a cluster record.

DIAGNOSE ICFCATALOG -
INDATASET(CATALOG.ICFUSER.VOL001) -
LIST

and you should see
IDC01360I THE FOLLOWING ENTRIES HAD NO ERRORS:
followed by a list of all the entries

If your diagnose runs clean, but you still get catalog errors, try refreshing the catalog buffers using the console command

F CATALOG,CLOSE(catalog.name)


The following command will check out the logical structure of a VVDS

//DD1 DD DISP=SHR,DSN=SYS1.VVDS.VVOL001
//SYSIN DD *
DIAGNOSE VVDS -
INFILE(DD1) -
LIST

All the previous two commands do is check out that a single catalog entity is correct. What you need, is to ensure all the catalog components are consistent.

//DD1 DD DISP=SHR,DSN=SYS1.VVDS.VVOL001
//SYSIN DD *
DIAGNOSE ICFCATALOG -
INDATASET(Catalog.name) -
COMPAREDD(DD1)

This will check that the entries in a catalog are consistent with those in one VVDS. It does not check out the entries in all the other VVDS datasets. To find out which VVDSs are referenced by a catalog, use the command

listcat level(sys1.vvds) catalog(CATALOG.ICFUSER.TSG)

Two errors are possible. If the VVDS contains datasets, but there are no catalog entries for them, create them by using

DEFINE CLUSTER (NAME(SYS1.VVDS.Vvolser) VOLUME(volser)-
NONINDEXED RECATALOG) -
CATALOG(catalog.name)

If the Catalog contains entries, but they are not in the VVDS and don't physically exist, use

DELETE SYS1.VVDS.Vvolser NOSCRATCH CATALOG(catalog.name)

The next command will validate a VVDS against a catalog

//DD1 DD DISP=SHR,DSN=SYS1.VVDS.VVOL001
//SYSIN DD *
DIAGNOSE VVDS -
INFILE(DD1) -

COMPAREDS(Catalog.name)

If you end up with a missing VSAM Volume Record (VVR) for one component of a VSAM file, then do a delete / noscratch at the cluster level first to clean things up. That leaves all the components uncatalogued. Then do a define without the recatalog parameter, specifying the correct volumes and allocated space for the file you are fixing. So in the IDCAMS statement below, change the dataset name, volumes and space as appropriate.

DEFINE CLUSTER -
(NAME(OPROD.TEST)) -
DATA (NAME(OPROD.TEST.DATA) VOL(V3003F) TRK(3,1) ) -
INDEX (NAME(OPROD.TEST.INDEX) VOL(V30038) TRK(1,1) )

IBM removed support for VSAM parameters IMBED and REPLICATE some time ago. They must be removed from all catalogs before you install z/OS 1.8 or they will cause abends.

The number of processing strings is quite critical to catalog performance as they control the number of possible parallel read requests. The default value is 2, but this is too low. Consider changing your strings 5 or 7, depending on how busy it is. You will also need the correct number of data and index buffers. 5 data buffers and 7 index buffers are about right for 5 strings. If you define more buffers than this you are probably just wasting storage.

Extended Catalog Sharing can improve performance between LPARs as this puts the VVDS information into the coupling facility. This was problematical at first but seems to be stable now.

You may also get response time problems in a shared environment if you are converting hardware reserves to dataset enqueues with CA-MIM. Consider using GRS in a STAR configuration instead.

z/OS VSAM Recovery

VSAM files are typically used by on-line transaction based applications. VSAM files used by a DBMS can use the DBMS forward recovery process. VSAM files driven by CICS need traditional backup methods that involve bringing the systems down at some point overnight, then copying the data off to tape. If the file gets deleted or corrupted, then all transactions made since the backup are lost. This is unacceptable to businesses today, so VSAM updates are often recorded in transaction logs, so the file can be rolled forward to the point of failure. Recovery usually involves a restore from the latest backup, then a rollforward through the logs. These processes are discussed below.

Basic Recovery from backup

DFDSS recovery

Any successful depends on a good backup first. With DFDSS, it is probably best to specify the parameters SELECTMulti(first) and Sphere to get good VSAM backups. Selectmulti(first) means that if this is the first portion of a multi-volume dataset, then DFDSS will go out and find all the other portions and dump them together. An alternative is Selectmulti(any), which means that if the volume contains any portion of a multi-volume file, DFDSS will find and backup the complete file. This can be expensive if you have a lot of multi-volume files spread over lots of disks, as you will end up taking multiple backups of each file. Sphere means dump all portions of a VSAM file, including alternate indexes and paths.

If you do the backup correctly, you should be able to just do a standard dataset restore specifying the VSAM cluster name, as long as you also specify SPHERE to get all the associated AIX clusters and paths back too. If the cluster was dumped without SPHERE, then you need to restore any Alternate Indexes as seperate clusters, and build the paths with IDCAMS commands.

FDRABR recovery

FDRABR will handle single volume VSAM files easily. you simply select the file you want to restore using the base cluster name. FDRABR will then restore all parts of the file, including alternative indexes, but FDRABR will not restore paths. You need to do that yourself using IDCAMS. The command is

DEFINE PATH ( NAME(aix.cluster.name.PATH) )

Multi-volume restores are a bit harder. If you have multi-volume datasets, its your responsibility to make sure that all the volumes are dumped at the same time, so all the parts are consistent. If the backup is ok, then a restore should be as simple as

RESTORE TYPE=ABR,DYNTAPE,DSNENQ=USE

SELECT DSN=cluster.name

FDRABR should then go away, work out which volume backups it needs, dynamically mount the tapes, dynamically mount the output volumes, and restore the cluster. You need the same number of output volumes as the original file, with sufficient space on each. FDRABR restores each part of the file as a physical sequential, with a temporary volser ####Vx. Once it restores the last part, it catalogs the file up correctly as VSAM, with the correct volsers.

If the restore fails, then the 'VSAM special considerations' section of the FDR manual will assist.

VSAM Forward Recovery

If you record all updates to a VSAM file in a seperate log, and timestamp those updates, then you can use a technique called 'Forward Recovery' to get the file back to any point in time. CICS, the IBM transaction monitor, records transaction logs. If you use raw VSAM and want to recover to a point in time, you may need a third party product to provide logging facilities. The basic requirements for VSAM forward recovery are -

A Recovery Point, which is a time at which you took a full backup of all your VSAM datasets using one of the techniques above. This provides you with a backup copy of the whole file, which you should take at regular intervals, daily or possibly weekly.

A Journal, which is a chronologically ordered list of all the changes made to datasets since a Recovery Point. The Journals (or Logs) will only contain updates made to the file. You can write CICS journals to disk or tape, and write to a single file, or flip between files. Single files are easier to maintain, but will fill up quickly if there is a lot of update activity. Journal management is used to keep track of multiple files, and present them to the recovery program in the right order.

Forward recovery is a two stage process. First the dataset is backed out to the position it was in at the recovery point by restoring backup copies. Secondly all dataset changes that have been journalled since the recovery point are applied.

Fixing VSAM file components

To delete a VSAM file, you just delete the cluster entry, and all the components are deleted with it. You can either do this with an IDCAMS DELETE job, or simply type DEL against the VSAM cluster name on a 3.4 listing (this is explained in the Z/OS utility section)

If you have VSAM components without a cluster, then you can catalog them up using an IDCAMS RECATALOG command as below. You need to know the volumes the the uncatalogued components are held on

DEFINE CLUSTER -
(NAME(PIN.MVSOLZAX.ZMG) -
RECATALOG) -
DATA (NAME(PIN.MVSOLZAX.ZMG.DATA) VOLUMES(DV000E) ) - INDEX (NAME(PIN.MVSOLZAX.ZMG.INDEX) VOLUMES(DV003B) )

Alternatively you can delete the uncatalogued components using IDCAMS statements like

//DD1 DD VOL=SER=DV000E,DISP=SHR,UNIT=DISK
//DD2 DD VOL=SER=DV003B,DISP=SHR,UNIT=DISK
//SYSIN DD *
DELETE PIN.MVSOLZAX.ZMG.DATA - VVR FILE(DD1) -
CATALOG(CATALOG.ICFUSER.USER4)
DELETE PIN.MVSOLZAX.ZMG.INDEX - VVR FILE(DD2) -
CATALOG(CATALOG.ICFUSER.USER4)

The TRUENAME is a Catalog record that relates a VSAM component to its cluster. Sometimes, usually after a failed attempt to delete a VSAM file, you can be left with an orphaned truename record in the catalog. If this happens you cannot recreate the file, you will get duplicate name failures if you try. The answer is to use IDCAMS DELETE with the TRUENAME parameter as below.

DELETE PIN.MVSOLZAX.ZMG.DATA - TRUENAME

z/OS VSAM Striping

VSAM striping can be used to improve the performance of datasets that cannot cope with the IO rate from a single volume. In general, VSAM datasets with a high random access pattern benefited from being spread over several volumes. Aa an example, I once had a problem DB2 database that was very heavily accessed and really suffered from poor performance. This was fixed by making it go multi-volume. However, this is not the same as Striping. Striping means that each control interval in the file is written to another disk, so the dataset is spread evenly over several volumes. True striping is considered to improve sequential performance. RAID disks stripe datasets over physical volumes at hardware level, but this still appears as one logical volume to z/OS, as z/Os will still schedule one IO at a time to it. PAV aliases fixed that problem as they permit multiple concurrent IOs to one logical volume, so maybe striping is not as useful as it was. Be aware that if you use PAV, VSAM striping may make performance worse. However, if you want striping, this is how you do it.

Striping will only work for VSAM files that are allocated in Extended Format. To do this, the dataset must be SMS managed and be allocated with a dataclass that has a 'Data Set Name Type' of 'EXT'. Striping is not supported for Alternate Indexes or VSAM files using RLS. You specify that you want striping on a Storage Class. There are two different ways to do this, using Sustained Data Rate (SDR) or Guaranteed Space with an SDR greater than 0.

SDR: within the storage class definition set the 'Sustained Data Rate' parameter to a value that should be a multiple of four. The system will then divide that number by four to determine the number of stripes to use. For example, if you set SDR to 16, the system allocates 4 stripes (assuming you use 3390 format disks).

Guaranteed Space: If you set SDR > 0 and set the Guaranteed Space parameter to 'Y' then the system will use as many stripes as the number of volumes or units that you specify in your dataset allocation. VOLUMES(* * * *) or UNIT=(3390,4) should both allocate four stripes. Note that SMS may allocate fewer stripes if it does not have enough volumes with the right characteristics to support your request.

If you define a storageclass called 'STRIPED', then to allocated a striped file, you either code STORCLAS(STRIPED) in your IDCAMS allocation, or pick a dataset pattern and put a clause in your STORCLAS ACS routine so datasets matching that pattern get the STRIPED storage class. To convert a file to striped, rename the old file and all its components, re-allocate the original file with a STRIPED storage class, then IDCAMS REPRO the data from the old file to the new one.

Bangalore : The rising divorce rate in the IT sector

The Information Technology boom in Bangalore is not unknown. However, everything comes with a price. Statistics reveal that in 2006 alone, 1,246 cases of divorce pertaining to those in the IT sector have landed in the matrimonial courts in Bangalore. Financial freedom, lack of time at home, erratic working hours, work pressure, financial security and stress are being seen as the main reasons for this fiasco. The worrying factor is that the number of divorce cases pertaining to those in the IT sector has seen a steady rise since 2003. In 2003, the number of cases from the IT sector was 283 while in 2004 it went upto 526. Statistics available show that in 2005 the figure went up to 946 and in 2006 the figure was 1,246.

The year 2007 has not been too kind. The statistics available till June 2007 state that the number of divorce cases from the IT sector is 828 already. Experts state that the figure is likely to increase by the end of the year. What is more shocking is that divorce cases from the IT sector seem to be contributing to the number in a big way.