Enable JavaScript.

DDM cover of News 34-38 magazine

1982-1992: DDM, Distributed Data Management Architecture

The year is 2014, many years after finally retiring from the work-a-day world. I have hesitated to write about my work on IBM's Distributed Data Management Architecture these past 22+ years because it was such an intense experience that lasted for so many years. Now, after enough time for reflection, I can make sense of this experience.

The first version of this memoir contained a lot of technical details about DDM Architecture: what were its goals, how did it evolve over time, what IBM products were developed, who were the key players, and so forth. I had to do a bit of research, some of which involved finding references to other IBM software architecture efforts. I found that Wikipedia was a useful source, at least at a high level. I also noticed that there was no Wikipedia article on DDM, and that was intolerable to me, given the amount of effort the DDM architecture and product development teams had expended on it. So, I took my memoir, reworked it to make it less personal, and greatly extended it as a full-blown article. (See IBM Distributed Data Management Architecture.) This memoir is now primarily a personal account of my work on DDM architecture. It also links to a variety of other DDM memoirs.

If you've read about the

, you know that DDM architecture was an orphan that no one wanted, and that I was in need of a job after my aborted move to IBM Research. (See .) John Bondy rescued both of us by adopting us into a department he created for a variety of such orphans. I was very glad to have both a new IBM home and effective control over a project I could develop to the best of my abilities.

During the first year, I had two key tasks; the first was to sell the idea of DDM architecture to someone in IBM who would commit to developing DDM products. John had me flying all over the country pitching DDM to the various IBM system houses, to executives, to research groups and even to university computer science programs. There was solid agreement that something like DDM was needed, but no one would commit to implementing it; the timing just wasn't right. Actually, we did have one client , but what they really wanted was not what DDM was all about. (See

.)

My second task that year was to design a general framework in which a variety of distributed data services could be included. At this point, I didn't know what specific services would be wanted, but I did know that the abstract approach of OSI FTAM, on which the Golden Gate version of DDM was based, was not acceptable. I settled on an object-oriented design that incorporated ideas from the Smalltalk programming language (See

.) and the layered object structures of the System/38 (See ), along with ideas from emergence theory, and with general principles of systems architecture firmly in mind. The documentation we produced about DDM architecture was itself object-oriented. (See .)

The first level, or version, of DDM architecture supported the record-oriented file systems of the System/36, the System/38 and CICS VSAM. Ken Lawrence from the System/36 had convinced me that the straightforward System/36 file system would be a good starting point, which it proved to be when it was easily enhanced to meet System/38 requirements. Rick Sanders was a crucial colleague in this effort; he handled communication, security and concurrency issues as I worked on file and access method designs.

DDM level 1 was published in 1986. Ken Lawrence received an Outstanding Technical Achievement Award, Rick Sanders received an Outstanding Contribution Award, and I received a coveted Outstanding Innovation Award, all with nice bonuses.

The second level of DDM added support for the hierarchical directories and stream-oriented files typical of PC/DOS and Unix systems. Jan Fisher and Sunil Gaitonde had joined the architecture team and did most of the design work in these areas. DDM level 2 was published in 1988.

Meanwhile, Sanders and I participated in the design of IBM's Distributed Relational Database Architecture (DRDA.) (See

.) We designed and documented the additional DDM objects, protocols and messages required by DRDA. But was DRDA just a subset of DDM or was DDM just a component architecture of DRDA? This question was never resolved and it resulted in some frayed tempers. (See )

DDM level 3 was published in 1990, at the same time as DRDA was announced. Rick received a second Outstanding Achievement Award and I received a second Outstanding Innovation Award.

My primary involvement in DDM level 4 was in the definition of a Data Description and Conversion (DD&C) addition to DDM for the Distributed File Management (DFM) mainframe project. (See

.) As part of this effort, I defined a new programming language, A Data Language (ADL), for describing and converting data. An ADL compiler was developed and used by the Distributed File Management (DFM) mainframe project.

Other DFM requirements were handled by Jan Fisher and Sunil Gaitonde. DDM level 3 was published in 1990.

Level 5 consisted of DDM enhancements in support of DRDA and PC/DOS libraries. These, too, were handled by Jan Fisher and Sunil Gaitonde. DDM level 5 was never formally published.

By that time, I had left the DDM architecture team. Throughout my years of working on various aspects of DDM Architecture, I viewed my role as being purely technical and paid little attention to internal IBM politics. This attitude generally worked well, but it worked against me in a very personal way.

I began work on DDM in 1982 as an Advisory Programmer, and in 1990 at the end of my work on DDM I was still at the same level. This was in spite of having received two Outstanding Innovation Awards, published three major articles in the IBM Systems Journal, received two US patents, and level one "walking on water" appraisals five years in a row. To put it mildly, I believed I deserved a promotion to Senior Programmer. My manager agreed but he was not able to convince Rochester lab managers who preferred promoting their own employees; at least, that was the excuse I was given. Finally, I went to see Jim Flynn, a third level manager in the lab, who was sympathetic. He investigated my situation and interviewed many of the people I had worked with on DDM. A week later, he said that I would get my promotion, but only if I left the DDM group and returned to work in the Rochester lab.

Great, I was pretty tired of DDM anyway. The only problem was that there really wasn't work available for another Senior Programmer in the lab. I was assigned to an Advanced Architecture department (essentially a holding pen for excess senior technical employees) and spent the next year struggling to find work appropriate to my talents and abilities. Frustrated, I took an opportunity for early retirement from IBM. And so ended my years of work on DDM and my years as an IBM employee. But hey, I left as a Senior Programmer!

In subsequent years of working as a consultant, I found that this mattered very little to anyone outside of IBM.

1982-1992: DDM, l'architecture pour la gestion de donées distribué

Désolé, pas encore traduit.



IBM Outstanding Innovation Award for Distributed Data Management Architecture

IBM Outstanding Innovation Award for SAA Distributed Relational Database Architecture

IBM Systems Journal, Distributed Files for SAA

IBM Systems Journal, Inside IBM's  Distributed Data Management Architecture

IBM Systems Journal, Data Description and ConLevel Architecture