Enable JavaScript.

1982-1983: DDM & Bulk Data Transfer


My back was against the chalk board of the conference room in IBM's Kingston, NY programming center. Cathy Sewell, Lynn Brooks, Lou Romano and Jim Ashfield were closing in on me. They were doing their best to convince me that my job was to design

(DDM) messages for bulk file transfer the way they wanted it designed, and that my views about what constituted good vs. bad architecture really didn't matter. The argument had been going on for days, and we'd regressed from attempting to understand each other's points of view to the current shouting match. It was becoming clear to me that they didn't want what I had to offer, and I didn't want to provide them what they wanted. Finally, my breaking point was reached. I put up my hands to get them to back away from me.

"This is getting us nowhere." I said. "I'm leaving. If you decide you want the kind of design I've been talking about, give me a call. Otherwise, you're on your own." And at that, I walked out on my only client.

I can understand why they accused me of being arrogant, but my position was not based on arrogance. It was based on a deeply held view of what a software architecture should be. And if sticking to your beliefs is arrogance, then I'm guilty as charged. Howard Roark strikes again! But I didn't dynamite anything.

The idea behind the Bulk Data Transfer, or BDT, project was to make it easy to transfer large files between computer systems through the use of store-and-forward communications facilities. Because I was the technical owner of IBM's DDM architecture, I had been drawn into this effort, along with Jim Ashfield who was representing the Systems Network Architecture Distribution Services (SNADS) architecture.

Quite elegant, SNADS had been designed as part of IBM's office automation strategy at a time when communications links between computers were scarce, expensive, slow, and needed to be carefully managed. Using SNADS protocols, a document could be asynchronously sent to a list of recipients geographically distributed throughout a network. Copies of a document flowed from system to system as communications lines became available; hence the store and forward aspect. The Bulk Data Transfer (BDT) project wanted to use SNADS to efficiently transmit large files to a distribution list, but they also needed a way to encode the files for transfer. (At that time, files of a million bytes were considered large.)

The IBM office products encoded documents according to IBM's Mixed Object Document Content Architecture (MODCA), which carefully specified how each part of a document was to be encoded and how those parts were to relate to each other. The result was that any system could create a document data stream and transmit it, and any system could receive a document data stream and decode it. The BDT project wanted a similar data stream architecture for encoding files.

Writing specifications for encoding a file in terms of its attributes and records was not very difficult, but this wasn't the issue that torpedoed the use of DDM file encodings by BDT. The problems that arose had to do with allowing the sender of a file to specify what to do with the file at the receiving system under a variety of conditions. For example, what if the named file already exists? Should the transmitted file replace it, or be appended to it, or be used to update specific records? The BDT project wanted to be able to support all of these possibilities. Fine, but how should these options be specified in the data stream?

Lynn, Lou, and Jim wanted me to add more attributes to the file being transmitted to say how it should be handled. Further, they also wanted all of the attributes to be rigidly mapped to specific bit sequences, in traditional control-block style. These bit sequences would then be interpreted by the BDT program on the receiving system. I argued that this was not the way to go because these options were not properties of a file per se, because encoding them as bit sequences was inherently inflexible, and because the range of file services provided by DDM architecture would not be able to grow over time.

Instead, I argued for a data stream similar to the normal structure of assembled programs, pointing to the IBM System/360 Principles of Operation as a model. In this approach, the data stream would effectively be a program, a series of encoded instructions that specified what to do to their operands. Among these operands would be encoded files. The instructions would be sequentially interpreted by the receiving BDT server to produce whatever effects the sender desired. For example, there would be an instruction to determine if a named file already existed, and there would be branching instructions to allow the sender to program what was to be done under each condition: copy, append, replace, update, etc.

We had reached this impasse after a series of meetings in Rochester, Minnesota, Kingston, New York, Raleigh, North Carolina, and Sindelfingen, Germany. After all that effort I really did not want to throw in the towel, but it was time to back away and talk things over with my manager, John Bondy. I returned to Rochester that evening and the next morning, he was waiting for me when I arrived at my office.

"What did you do to those people?" John asked. "They want me to fire you." He had spent an hour on the phone with Matt Helms, Lynn's manager, the previous afternoon, after I had left Kingston.

I carefully walked him through the whole situation. "What it comes down to, John, is that we can give them what they want, but it'll never grow into the full-blown DDM architecture that we both know is needed. And instead of doing that, I believe we're better off walking away from them and waiting for a client who really wants what DDM is all about."

John thought hard about this. We had spent over a year trying to sell DDM to various IBM products, and BDT had been our only nibble. And worse, if we walked away from BDT, we'd have a reputation of being hard to work with that might be fatal. But if we gave them what they wanted, DDM would be stuck with unfixable design problems.

"We're going back to Kingston." John said, eventually. "Both of us this time. I'll listen to both sides and we'll come to a decision. And if we can't find a compromise, we'll walk away from BDT and hope for the best."

A week later, John and I were back in that same conference room in Kingston. The discussion had gone on for over an hour. I wasn't by myself, this time, but there were far more BDT project people in the room, so the ratio had gotten worse. This time, John and I were both standing with our backs to the chalk board and everyone was red in the face. At one point, John and Matt were nose to nose, shouting at each other with clenched fists. No compromise was found. We walked.


2012: Epilog


This episode played over and over in my memory for many years. Surely, it must have been possible to resolve this to everyone's satisfaction. But it wasn't until thirty years later, way too late to be of any value, that a solution bubbled up from my subconscious mind.

A year after the confrontation in Kingston, the DDM architecture I wanted had been implemented by IBM's System/36 and IBM's System/38. The System/38 programmers liked the file models that we had designed and the simple, flexible way I had structured the data stream as nested objects. However, they also wanted to be able to communicate commands and data that were unique to the System/38, things that would not be of interest to other systems and therefore not appropriate for inclusion in a generic architecture like DDM.

Initially, this felt like the BDT problem all over again, but this time DDM architecture was sufficiently developed that a solution was easily found. We added a protocol that enabled the communicating systems to know who they were talking to; for example, a System/38 talking with a System/36, or a System/38 talking with another System/38. In the latter case, they could define whatever extensions to DDM they wanted. This was a simple and effective solution that kept both systems working with us.

With twenty-twenty hindsight, the same could have been done with BDT. They could have used DDM data stream structures to encode the files to be transferred and done whatever they wanted in their own extensions to the architecture. An opportunity had been lost and a client turned away, but the DDM architecture John Bondy and I had envisioned was realized and implemented in numerous IBM products.

1982: Le transfer en gros de données


Désolé, pas encore traduit.