hn-classics/_stories/2010/13252608.md

created_at	title	url	author	points	story_text	comment_text	num_comments	story_id	story_title	story_url	parent_id	created_at_i	_tags	objectID	year
2016-12-25T02:41:26.000Z	Reclaiming Software Engineering (2010)	http://www.zerobanana.com/essays/reclaiming-software-engineering	akkartik	62			14					1482633686	story author_akkartik story_13252608	13252608	2010

When I use a word, it means just what I choose it to mean—neither more nor less.

It is the fate of every vocation to be misunderstood by outsiders, and engineering is by no means an exception: ‘Most people think engineers do things in a very mechanistic way. … They don’t seem to have any idea of the creativity and ingenuity in engineering.’2 To the public at large the engineering method is just following an impenetrable set of scientifically predetermined instructions, the engineer a ‘soulless apparatchik’3 —the sort of person who might regard the employee-feeding machine from Chaplin’s Modern Times as a splendid labour-saving device. It is as if engineers had created such machines in their own image.

Software engineering has an even bigger perception problem, though: it is widely misunderstood even by insiders. Civil engineers who fail to recognise their work in the typical engineering mythology may be certain that, by definition, it is the public perception at fault. What is engineering if not what civil engineers do? By contrast, without some traditional engineering experience for guidance a software developer in the same predicament cannot rule out the thought that software development is somehow unlike engineering—or, worse, that we are doing it improperly.

Certainly there are those even within the engineering profession who do subscribe to the outsider view. Software developers are engaged in the exact same debate about the nature of our work as engineers in the more traditional fields are about theirs. The difference is that while software developers disagree about how software development ought to work, most accept without question the mechanistic theory of how engineering works—the engineering method as ‘adherence to a specified morphology’4 . The morphology with which we are all familiar is known as the Waterfall Model,5 and the idea that all ‘real’ engineers follow it is practically an article of faith.6 It would come as a shock to most in the software community to discover that this is not really the way engineering works. Proponents of a mechanistic kind of ‘software engineering’ have been allowed to define the term in agreement with their own prejudices.

At the root of the unwillingness of the software community to accept software development as a form of engineering lies a single misconception, inexplicably shared across ideological boundaries. To see what it is, we must first define a little terminology.

The technical term for the output of an engineering process is an ‘artefact’. The artefact is only nominally what customers pay for. The actual product is some experience that occurs between the artefact and the user.7 This is what Pirsig was referring to when he wrote that ‘Quality is not a thing. It is an event.’8 A bridge is an artefact but it is not the product. The product is not getting wet.

A design document—for example, a set of engineering drawings—describes how to construct one or more artefacts, but is not itself an artefact. A building is an artefact; the product it provides, or one of them, is (again) not getting wet.9 You can get some of the effect by holding a set of working drawings above your head, but beyond a certain point—probably a stiff breeze, anything more than a light drizzle or when your arm gets tired, whichever comes first—you are bound to want the actual building.

Our problem arises when people mistake software source code for an artefact. Source code is not an artefact because the user does not interact with it; there is no product experience created in this way. Source code is a design document because it describes how to construct the true artefact, which is the machine-executable code.10 Source code is the map, not the territory.

This point is critical to the understanding of software development because, as Jack Reeves explained in 1992, it identifies our work as designing rather than constructing (or, for that matter, manufacturing) software.11 Construction is an activity performed by compilers and the associated infrastructure around them—not for nothing do we call this the ‘build system’. We are software engineers, not software construction workers.

We should not see engineering as a metaphor for software development. Each branch of engineering has its own set of heuristics, moulded in part by the nature of the materials with which it works, and software engineering is naturally no exception. Chief among its unique features is that software construction is both fast and cheap—often taking on the order of seconds to minutes, generally with zero marginal cost. Most engineers do not have the luxury of such a short feedback loop between their designs and the finished artefacts. It would be inappropriate for software engineering to simply adopt a morphology from another branch of engineering, even if such a thing existed. The common thread uniting those branches is not some shared morphology but the fact that none is capable of being reduced to such a morphology.

Those who see software development as mechanistic in nature invariably also mistake it for a construction activity. Usually these people will say ‘Software engineering is not yet a true engineering discipline’,12 but that they hope it will eventually become one.13, 14 The obstacle is always a lack of ‘rigour’, the solution more ‘discipline’.15 Apparently the best way to instil this discipline may to threaten the practitioner with imprisonment.16

The most prominent example of this school of thought is the unfortunately-named Software Engineering Institute at Carnegie Mellon University. It is funded by the United States Department of Defence, whose bizarre procurement policies treat designs and artefacts identically. Unsurprisingly, the SEI tends to label ideas that fit its patron’s worldview as ‘software engineering’, even when they may be the opposite of established engineering practice.

One form this takes is the fetishisation of reliability over all other properties of a system. To the mechanist, the fact that software sometimes crashes is damning evidence that software is not yet an engineering discipline. In part this is coloured by an unjustifiably narrow view of the range of work in which engineers are engaged. Pencils and paperclips break all the time.17 Even in areas where safety of life is an issue—popularly assumed to be the sole province of engineers—failures still occur absent any negligence on the part of engineers.18 To assume that software is unique among humanity’s creations in its tendency to fail is to ignore both the evidence and the more prosaic explanation: that the cost to society of ensuring it never can usually outweighs the cost of taking the risk.

The temptation is strong to declare that this reliability-obsessed process is not engineering at all, but it must be resisted. I do not claim that all software is engineered, at least in the modern sense. Yet so long as we are developing software within a social context, where the finished artefact has users other than its designers, we can scarcely escape the conclusion that what we are doing is engineering. Still, I would certainly argue that methodologies that disregard the expenditure of resources are the antithesis of good engineering.

If I wished to be kind to the SEI, I would note only that the context in which they operate is not shared by most—or even many—software engineers, and that they could do us all a favour by choosing a name that does not imply that they represent the only way of doing software engineering. If I were feeling less charitable, I might add that even in their chosen context their methods have proven conspicuously unsuccessful.

On the other side of the fence, many of those who reject the mechanistic theory of software development nevertheless mistake the source code for an artefact just the same. At the nexus of these two ideas lies the concept of software development as ‘craft’. Recognition that all software development is unpredictable design work and belief that software is constructed by humans can be reconciled only in a medieval model19 of software development, in which the design and construction of artefacts are not separate activities:

The technology of the middle ages was the technology of the artisan. …

In the Renaissance

components, the engineer and the worker.20

Seen in this light, the epitome of the software craftsman is to be found in the story of Mel, the ‘Real Programmer’, who famously programs directly in machine code. Mel’s legendary technical skills have made him a hero to many, yet he seems completely isolated from the economic consequences of his work.21 It exists in a vacuum, outside of any social context.

Something is always lost when the engineer is separated from the worker, but such transitions occur because these losses are outweighed by the gains. The combined efforts of all the master craftsmen in Florence might never have been sufficient to complete the cathedral dome had Filippo Brunelleschi—or some other engineer—not devised a way to build it.22 In software such a split was already in evidence by the 1960s, simply because the methodology Mel used to craft his blackjack program imposed a very low upper bound on software’s transformative power. In the intervening decades the workers—‘programmers’, in the contemporary sense of the term—have been automated out of existence. The revolution will not be televised; it already happened and you missed it.

To regress to a guild of medieval Real Programmers does not strike me as a goal to which we should aspire. We need not do so if we recognise that source code is not an artefact and accept our role as software designers and not software builders. Engineers, freed from the need to personally construct their artefacts, take no less care in and derive no less pleasure from their work than did the craftsmen who preceded them.

In all our discourse about software engineering, we have failed to challenge the assumption that it could only be brought about through the application of ‘rigour’, ‘discipline’ and government-mandated participation in a monoculture that treats every line of code as if lives depended on it, on pain of imprisonment, regardless of context. This vision may be to some a shining beacon of hope, to others a nightmare that makes Kafka look like Disney. But the essence of engineering is not to be found in these things. To be an engineer is to strive for change in a world where resources are limited.

We are the heirs of an engineering tradition as old as civilisation, but Eden is burning. The name of our profession itself—the very core of our identity—has been co-opted by some of the most dysfunctional organisations in society and redefined in ways that conflict with our values. In order to claim our birthright we must first recapture what it means to be a software engineer.