527 lines
30 KiB
Markdown
527 lines
30 KiB
Markdown
---
|
||
created_at: '2015-05-19T12:28:32.000Z'
|
||
title: 'E. W. Dijkstra Archive: “Why is software so expensive?” (2009)'
|
||
url: https://www.cs.utexas.edu/users/EWD/transcriptions/EWD06xx/EWD648.html
|
||
author: tomaskazemekas
|
||
points: 95
|
||
story_text:
|
||
comment_text:
|
||
num_comments: 63
|
||
story_id:
|
||
story_title:
|
||
story_url:
|
||
parent_id:
|
||
created_at_i: 1432038512
|
||
_tags:
|
||
- story
|
||
- author_tomaskazemekas
|
||
- story_9569521
|
||
objectID: '9569521'
|
||
year: 2009
|
||
|
||
---
|
||
"Why is software so expensive?" An explanation to the hardware designer.
|
||
|
||
Recently I received an invitation from a sizeable (and growing) hardware
|
||
company. For many years its traditional product line has been
|
||
high-quality analog equipment; in a more recent past, however, digital
|
||
components are beginning to play a more important role. The company's
|
||
corporate management was aware of more or less unavoidably entering the
|
||
(for the company unfamiliar) field of software, was aware of the
|
||
existence of its many pitfalls without having a clear understanding of
|
||
them, and I was invited to explain to the company's corporate management
|
||
what the design of software is all about, why it is so expensive, etc.
|
||
|
||
Having many other obligations, I don't know yet whether I shall be able
|
||
to accept the invitation, but, independent of that, the challenge
|
||
absolutely delights me. Not only have I programmed for more than 25
|
||
years, but right from the beginning up till this very day I have done so
|
||
in, over periods even close, cooperation with hardware designers,
|
||
machine developers, prototype testers, etc. I think that I know the
|
||
average digital hardware designer and his problems well enough to
|
||
understand why he does not understand why designing software is so
|
||
difficult. To explain the difficulty of software design to him is hard
|
||
enough —almost as hard as explaining it to a pure mathematician— , to
|
||
explain it to a group of designers with their background and
|
||
professional pride in high quality analog equipment adds definitely a
|
||
distinctive flavour to the challenge\! Observing myself thinking about
|
||
how to meet it and realizing that, even if I accept the invitation, my
|
||
host will not have the exclusive rights of my explanation, I decided to
|
||
take pen and paper. Hence this text.
|
||
|
||
\* \* \*
|
||
|
||
To the economic question "Why is software so expensive?" the equally
|
||
economic answer could be "Because it is tried with cheap labour." Why is
|
||
it tried that way? Because its intrinsic difficulties are widely and
|
||
grossly underestimated. So let us concentrate on "Why is software design
|
||
so difficult?". One of the morals of my answer will be that with
|
||
inadequately educated personnel it will be impossible, with adequately
|
||
educated software designers it might be possible, but certainly remain
|
||
difficult. I would like to stress, right at the start, that current
|
||
problems in software design can only partly be explained by identified
|
||
lack of competence of the programmers involved. I would like to do so
|
||
right at the start, because that explanation, although not uncommon, is
|
||
too facile.
|
||
|
||
It is understandable: it must be very frustrating for a hardware manager
|
||
to produce what he rightly considers as a reliable machine with a
|
||
splendid cost/performance ratio, and to observe thereafter that, by the
|
||
time the customer receives the total system, that system is bug-ridden
|
||
and its performance has dropped below the designer's worst dream. And
|
||
besides having to swallow that the software guys have ruined his
|
||
product, he is expected to accept that while he works more and more
|
||
efficiently every year, the software group is honoured for its
|
||
incompetence by yearly increasing budgets. Without further explanations
|
||
from our side we programmers should forgive him his occasional
|
||
bitterness, for by accusing us of incompetence, he sins in ignorance....
|
||
And as long as we haven't been able to explain the nature of our
|
||
problems clearly, we cannot blame him for that ignorance\!
|
||
|
||
\* \* \*
|
||
|
||
A comparison between the hardware world and the software world seems a
|
||
good introduction for the hardware designer to the problems of his
|
||
software colleague.
|
||
|
||
The hardware designer has to simulate a discrete machine by essentially
|
||
analog means. As a result the hardware designer has to think about
|
||
delays, slopes of signals, fan-in and fan-out, skew clocks, heat
|
||
dissipation, cooling and power supply, and all the other problems of
|
||
technology and manufacturing. Building essentially from analog
|
||
components implies that "tolerances" are a very essential aspect of his
|
||
component specifications; his quality control is essentially of a
|
||
statistical nature, and, when all is said and done, quality assurance is
|
||
essentially a probabilistic statement. The fact that with current
|
||
quality standards the probability of correct operation is very, very
|
||
high should not seduce us to forget its probabilistic nature: very high
|
||
probability should not be confused with certainty (in the mathematical
|
||
sense) and it is therefore entirely appropriate that no piece of
|
||
equipment is delivered without being exercised by test programs. As
|
||
technology is more and more pushed to its limits —and it is so all the
|
||
time— and tolerances become narrower and narrower, the control of these
|
||
tolerances becomes a major concern for the hardware builders. Compared
|
||
to the hardware designer who constantly struggles with an unruly nature,
|
||
the software designer lives in heaven, for he builds his artefacts from
|
||
zeros and ones alone. A zero is a zero and a one is a one: there is no
|
||
fuzziness about his building blocks and the whole engineering notion of
|
||
something being "within tolerance" is just not applicable there. In this
|
||
sense the programmer works indeed in a heavenly environment. The
|
||
hypothetical one hundred percent circuit designer who equates the
|
||
problems of design and building with the problems of keeping the
|
||
tolerance under control must be blind for the programming problems: once
|
||
he has simulated the discrete machine correctly, all the really hard
|
||
problems have been solved, haven't they?
|
||
|
||
To explain to the hardware world why programming yet presents problems,
|
||
we must draw attention to a few other differences. In very general terms
|
||
we can view "design" as bridging a gap, as composing an artefact of
|
||
given components; as long as "the target artefact" and "the source
|
||
components" don't change, we can reuse the old design. The fact that we
|
||
need to design continuously is because they do change. Here, however,
|
||
hardware and software designers have been faced with very different,
|
||
almost opposite types of variation, change and diversity.
|
||
|
||
For the hardware designer the greatest variation has been in "the source
|
||
components": as long as machines have been designed he has had to catch
|
||
up with new technologies, he has never had the time to become fully
|
||
familiar with his source material because before he had reached that
|
||
stage, new components, new technologies had appeared on the scene.
|
||
Compared to the drastic variation in his "source components", his
|
||
"target artefact" has almost remained constant: all the time he has
|
||
redesigned and redesigned the same few machines.
|
||
|
||
For the programmer the variation and diversity is just at the other end:
|
||
the hardware designer's target is the programmer's starting point. The
|
||
programmer's "source components" have been remarkably stable —in the
|
||
eyes of some: even depressingly so\!— : FORTRAN and COBOL, still very
|
||
much en vogue, are more than a quarter of a century old\! The
|
||
programmers find the diversity at the other side of the gap to be
|
||
bridged: he is faced with a collection of "target artefacts" of great
|
||
diversity. Of very great diversity even; of an essentially very great
|
||
diversity even, because here we find reflected that today's equipment,
|
||
indeed, deserves the name "general purpose".
|
||
|
||
During the last decade software designers have carried on an almost
|
||
religious debate on "bottom-up" versus "top-down" design. It used to be
|
||
"bottom-up": I think that now the "top-down" religion has the majority
|
||
as its adherents. If we accept the sound principle that, when faced with
|
||
a many-sided problem, we should explore the area of our greatest
|
||
uncertainty first (because the solution of familiar problems can be
|
||
postponed with less risk), we can interpret the conversion of the
|
||
programming community from "bottom-up" to "top-down" as a slow
|
||
recognition of the circumstance that the programmer's greatest diversity
|
||
is at the other side of the gap.
|
||
|
||
Besides being at the other side of the gap to be bridged, the variation
|
||
and diversity the programmer is faced with is more open-ended. For the
|
||
understanding of his source components the hardware designer has as a
|
||
last resort always physics and electronics to fall back upon: for the
|
||
understanding of his target problem and the design of algorithms solving
|
||
it the software designer finds the appropriate theory more often lacking
|
||
than not. How crippling the absence of an adequate theory can be has,
|
||
however, only been discovered slowly.
|
||
|
||
With the first machine applications, which were scientific/technical,
|
||
there were no such difficulties: the problem to be solved was
|
||
scientifically perfectly understood and the numerical mathematics was
|
||
available to provide the algorithms and their justification. The
|
||
additional coding to be done, such as for the conversion between decimal
|
||
and binary number systems and for program loaders, was so trivial that
|
||
common sense sufficed.
|
||
|
||
Since then we have seen, again and again, that for lack of appropriate
|
||
theory, problems were tackled with common sense, while common sense
|
||
turned out to be insufficient. The first compilers were made in the
|
||
fifties without any decent theory for language definition, for parsing,
|
||
etc., and they were full of bugs. Parsing theory and the like came
|
||
later. The first operating systems were made without proper
|
||
understanding of synchronization, of deadlock, danger of starvation,
|
||
etc., and they suffered from the defects that in hindsight were
|
||
predictable. The indispensable theory, again, came later.
|
||
|
||
That people have to discover by trying that for some problems common
|
||
sense alone is not a sufficient mental tool, is understandable. The
|
||
problem is that by the time the necessary theory has been developed, the
|
||
pre-scientific, intuitive approach has already established itself and,
|
||
in spite of its patent insufficiency, is harder to eradicate than one
|
||
would like to think. Here I must place a critical comment on a
|
||
management practice that is not uncommon among computer manufacturers,
|
||
viz. to choose as project manager someone with practical experience from
|
||
an earlier, similar project: if the earlier project had been tackled by
|
||
pre-scientific techniques, this is then likely to happen to the new
|
||
project as well, even if the relevant theory is in the meantime
|
||
available.
|
||
|
||
A second consequence of this state of affairs is that one of the most
|
||
vital abilities of a software designer faced with a new task is the
|
||
ability to judge whether existing theory and common sense will suffice,
|
||
or whether a new intellectual discipline of some sort needs to be
|
||
developed first. In the latter case it is essential not to embark upon
|
||
coding before that necessary piece of theory is there. Think first\! I
|
||
shall return to this topic later, in view of its management
|
||
consequences.
|
||
|
||
\* \* \*
|
||
|
||
Let me now try to give you, by analogy and example, some feeling for the
|
||
kind of thinking required.
|
||
|
||
Since IBM stole the term "structured programming" I don't use it anymore
|
||
myself, but I lectured on the subject in the late sixties at MIT. A key
|
||
point of my message was that (large) programs were objects without any
|
||
precedent in our cultural history, and that the most closely analogous
|
||
object I could think of was a mathematical theory. And I have
|
||
illustrated this with the analogy between a lemma and a subroutine: the
|
||
lemma is proved independently of how it is going to be used and is used
|
||
independently of how it has been proved; similarly a subroutine is
|
||
implemented independently of how it is going to be used and is used
|
||
independently of how it has been implemented. Both were examples of
|
||
"Divide and Rule": the mathematical argument is parcelled out in
|
||
theorems and lemmata, the program is similarly divided up in processes,
|
||
subroutines, clusters etc.
|
||
|
||
In the meantime I know that the analogy extends to the ways in which
|
||
mathematical theories and programs are developed. By word of mouth I
|
||
heard recently that Dana S. Scott described the design of a mathematical
|
||
theory as an experimental science, experimental in the sense that
|
||
adequacy and utility of new notations and concepts were determined
|
||
experimentally, to wit: by trying to use them. This, now, is very
|
||
similar to the way a design team tries to cope with the conceptual
|
||
challenges it faces.
|
||
|
||
When the design is complete one must be able to talk meaningfully about
|
||
it, but the final design may very well be something of a structure never
|
||
talked about before. So the design team must invent its own language to
|
||
talk about it, it must discover the illuminating concepts and invent
|
||
good names for them. But it cannot wait to do so until the design is
|
||
complete, for it needs the language in the act of designing\! It is the
|
||
old problem of the hen and the egg. I know of only one way of escaping
|
||
from that infinite regress: invent the language that you seem to need,
|
||
somewhat loosely wherever you aren't quite sure, and test its adequacy
|
||
by trying to use it, for from their usage the new words will get their
|
||
meaning.
|
||
|
||
Let me give you one example. in the first half of the sixties I designed
|
||
as a part of a multiprogramming system a subsystem whose function it was
|
||
to abstract from the difference between primary and secondary store: the
|
||
unit in which information was to be shuffled between storage level was
|
||
called "a page". When we studied our first design, it turned out that we
|
||
could regard that only as a first approximation, because efficiency
|
||
considerations forced us to give a subset of the pages in primary store
|
||
a special status. We called them "holy pages", the idea being that, the
|
||
presence of a holy page in primary store being guaranteed, access to
|
||
them could be speeded up. Was this a good idea? We had to define "holy
|
||
pages" in such a way that we could prove that their number would be
|
||
bounded. Eventually we came up with a very precise definition which
|
||
pages would be holy that satisfied all our logic and efficiency
|
||
requirements, but all during these discussions the notion "holy" only
|
||
slowly developed into something precise and useful. Originally, for
|
||
instance, I remember that "holiness" was a boolean attribute: a page was
|
||
holy or not. Eventually pages turned out to have a "holiness counter",
|
||
and the original boolean attribute became the question whether the
|
||
holiness counter was positive or not.
|
||
|
||
If during these discussions a stranger would have entered our room and
|
||
would have listened to us for fifteen minutes, he would have made the
|
||
remark "I don't believe that you know what you are talking about." Our
|
||
answer would have been "Yes, you are right, and that is exactly why we
|
||
are talking: we are trying to discover about precisely what we should be
|
||
talking."
|
||
|
||
I have described this scene at some length because I remember it so well
|
||
and because I believe it to be quite typical. Eventually you come up
|
||
with a very formal and well-defined product, but this eventual birth is
|
||
preceded by a period of gestation during which new ideas are tried and
|
||
discarded or developed. That is the only way I know of in which the mind
|
||
can cope with such conceptual problems. From experience I have learned
|
||
that in that period of gestation, when a new jargon has to be created,
|
||
an excellent mastery of their native tongue is an absolute requirement
|
||
for all participants. A programmer that talks sloppily is just a
|
||
disaster. Excellent mastery of his native tongue is my first selection
|
||
criterion for a prospective programmer; good taste in mathematics is the
|
||
second important criterion. (As luck will have it, they often go hand in
|
||
hand.)
|
||
|
||
I had a third reason for describing the birth of the notion "holy" at
|
||
some length. A few years ago I learned that it is not just a
|
||
romantization, not just a sweet memory from a project we all liked: our
|
||
experience was at the heart of the matter. I learned so when I wished to
|
||
give, by way of exercise for myself, the complete formal development of
|
||
a recursive parser for a simple programming language, defined in terms
|
||
of some five or six syntactic categories. The only way in which I could
|
||
get the formal treatment right was by the introduction of new syntactic
|
||
categories\! Those new syntactic categories characterized character
|
||
sequences which were meaningless in the original programming language to
|
||
be parsed, but indispensable for the understanding and justification of
|
||
the parsing algorithm under design. My formal exercise was very
|
||
illuminating, not because it had resulted in a nice parser, but because
|
||
in a nice, formal nutshell it illustrated the need for the kind of
|
||
invention software development requires: the new syntactic categories
|
||
were exemplary for the concepts that have to be invented all the way
|
||
long, concepts that are meaningless with respect to the original problem
|
||
statement, but indispensable for the understanding of the solution.
|
||
|
||
\* \* \*
|
||
|
||
I hope that the above gives you some feeling for the programmer's task.
|
||
When dealing with the problems of software design, I must also devote a
|
||
word or two to the phenomenon of bad software manager. It is
|
||
regrettable, but bad software managers do exist and, although bad, they
|
||
have enough power to ruin a project. I have lectured all over the world
|
||
to programmers working in all sorts of organizations, and the
|
||
overwhelming impression I got from the discussions is that the bad
|
||
software manager is an almost ubiquitous phenomenon: one of the most
|
||
common reactions from the audience in the discussion after a lecture is
|
||
"What a pity that our manager isn't here\! We cannot explain it to him,
|
||
but from you he would perhaps have accepted it. We would love to work in
|
||
the way you have described, but our manager, who doesn't understand,
|
||
won't let us." I have encountered this reaction so often that I can only
|
||
conclude that, on the average, the situation is really bad. (I had my
|
||
worst experience in a bank, with some government organizations as good
|
||
seconds.)
|
||
|
||
In connection with bad managers I have often described my experience as
|
||
a lecturer at IBM, because it was so illuminating. Just before I came,
|
||
the interior decorator had redone the auditorium, and in doing so he had
|
||
replaced the old-fashioned blackboard by screen and overhead projector.
|
||
As a result I had to perform in a dimly lighted room with my sunglasses
|
||
on in order not to get completely blinded. I could just see the people
|
||
in the front rows.
|
||
|
||
That lecture was one of the most terrible experiences in my life. With a
|
||
few well-chosen examples I illustrated the problem solving techniques I
|
||
could formulate at that time, showed the designer's freedom on the one
|
||
hand, and the formal discipline needed to control it on the other. But
|
||
the visible audience was absolutely unresponsive: I felt as if I were
|
||
addressing an audience of puppets made from chewing gum. It was for me
|
||
sheer torture, but I knew that it was a good lecture and with a dogged
|
||
determination I carried my performance through until the bitter end.
|
||
|
||
When I had finished and the lights were turned up I was surprised by a
|
||
shattering applause... from the back rows that had been invisible\! It
|
||
then turned out that I had had a very mixed audience, delighted
|
||
programmers in the back rows and in front rows their managers who were
|
||
extremely annoyed at my performance: by openly displaying the amount of
|
||
"invention" involved, I had presented the programming task as even more
|
||
"unmanageable" than they already feared. From their point of view I had
|
||
done a very poor job. It was at that occasion that I formulated for
|
||
myself the conclusion that poor software managers see programming
|
||
primarily as a management problem because they don't know how to manage
|
||
it.
|
||
|
||
These problems are less prevalent in those organizations —I know a few
|
||
software houses— where the management consists of competent, experienced
|
||
programmers (rather than a banker with colonial experience, but still
|
||
too young to retire). One of the problems caused by the
|
||
non-understanding software manager is that he thinks that his
|
||
subordinates have to produce code: they have to solve problems, and in
|
||
order to do so, they have to use code. To this very day we have
|
||
organizations that measure "programmer productivity" by the "number of
|
||
lines of code produced per month"; this number can, indeed, be counted,
|
||
but they are booking it on the wrong side of the ledger, for we should
|
||
talk about "the number of lines of code spent".
|
||
|
||
The actual coding requires a great care and a non-failing talent for
|
||
accuracy; it is labour-intensive and should therefore be postponed until
|
||
you are as sure as sure can be that the program you are about to code
|
||
is, indeed, the program you are aiming for. I know of one —very
|
||
successful— software firm in which it is a rule of the house that for a
|
||
one-year project coding is not allowed to start before the ninth month\!
|
||
In this organization they know that the eventual code is no more than
|
||
the deposit of your understanding. When I told its director that my main
|
||
concern in teaching students computing science was to train to think
|
||
first and not to rush into coding, he just said "If you succeed in doing
|
||
so, you are worth your weight in gold." (I am not very heavy).
|
||
|
||
But apparently, many managers create havoc by discouraging thinking and
|
||
urging their subordinates to "produce" code. Later they complain that 80
|
||
percent of their labour force is tied up with "program maintenance", and
|
||
blame software technology for that sorry state of affairs, instead of
|
||
themselves. So much for the poor software manager. (All this is
|
||
well-known, but occasionally needs to be said again.)
|
||
|
||
\* \* \*
|
||
|
||
Another profound difference between the hardware and the software worlds
|
||
is presented by the different roles of testing.
|
||
|
||
When, 25 years ago, a logic designer had cooked up a circuit, his next
|
||
acts were to build and to try it, and if it did not work he would probe
|
||
a few signals with his scope and adjust a capacitor. And when it worked
|
||
he would subject the voltages from the power supply to 10 percent
|
||
variations, adjust, etc., until he had a circuit that worked correctly
|
||
over the whole range of conditions he was aiming at. He made a product
|
||
of which he could "see that it worked over the whole range". Of course
|
||
he did not try it for "all" points of the range, but that wasn't
|
||
necessary, for very general continuity considerations made it evident
|
||
that it was sufficient to test the circuit under a very limited number
|
||
of conditions, together "covering" the whole range.
|
||
|
||
This iterative design process of trial and error has been taken so much
|
||
for granted that it has also been adopted under circumstances in which
|
||
the continuity assumption that justifies the whole procedure, is not
|
||
valid. In the case of an artefact with a discrete "performance space"
|
||
such a program, the assumption of continuity is not valid, and as a
|
||
result the iterative design process of trial and error is therefore
|
||
fundamentally inadequate. The good software designer knows this; he
|
||
knows that from the observation that in the cases tried his program
|
||
produced the correct result he is not allowed to extrapolate that his
|
||
program is OK; therefore he tries to prove mathematically that his
|
||
program meets the requirements.
|
||
|
||
The mere suggestion of an existence of an environment in which the
|
||
traditional design process of trial and error is inadequate and where,
|
||
therefore, mathematical proof is required, is unpalatable for those for
|
||
whom mathematical proofs are beyond their mental grasp. As a result, the
|
||
suggestion has encountered a considerable resistance, even among
|
||
programmers who should know better. It is not to be wondered at that in
|
||
the hardware world the recognition of the potential inadequacy of the
|
||
testing procedure is still very rare.
|
||
|
||
Some hardware designers are beginning to worry, but usually not because
|
||
they consider the fundamental inadequacy of the testing approach, but
|
||
only because the "adjustment" has become so expensive since the advent
|
||
of LSI-technology. But even without that financial aspect they should
|
||
already worry, because in the meantime a sizeable fraction of their
|
||
design activity does take place in a discrete environment.
|
||
|
||
Recently I heard a story about a machine —not a machine design by
|
||
Burroughs, I am happy to add— . It was a microprogrammed multiprocessor
|
||
installation that had been speeded up by the addition of a slave store,
|
||
but its designers had done this addition badly: when the two processors
|
||
operated simultaneously on the same word, the machine with the slave
|
||
store reacted differently from the version without it. After a few
|
||
months of operation a system breakdown was traced back to this very
|
||
design error. By testing you just cannot hope to catch such an error
|
||
that becomes apparent by coincidence. Clearly that machine had been
|
||
designed by people who hadn't the foggiest notion about programming. A
|
||
single competent programmer on that design crew would have prevented
|
||
that blunder: as soon as you complicate the design of a multiprocessor
|
||
installation by the introduction of a slave store, the obligation to
|
||
prove —instead of just believing without convincing evidence— that after
|
||
the introduction of the slave store the machine still meets its original
|
||
functional specification is obvious to a competent programmer. (Such a
|
||
proof doesn't seem to present any fundamental or practical difficulties
|
||
either.) To convince hardware designers of the fact that they have moved
|
||
into an environment in which their conventional experimental techniques
|
||
for design and quality control are no longer adequate is one of the
|
||
major educational challenges in the field.
|
||
|
||
I called it "major" because, as long as it isn't met, hardware designers
|
||
won't understand what a software designer is responsible for. In the
|
||
traditional engineering tradition, the completed design is the
|
||
designer's complete product: you build an artefact and, lo and behold,
|
||
it works\! If you don't believe it, just try it and you will see that
|
||
"it works". In the case of an artefact with a discrete performance
|
||
space, the only appropriate reaction to the observation that it has
|
||
"worked" in the case tried, is: "So what?". The only convincing evidence
|
||
that such a device with a discrete performance space meets its
|
||
requirements includes a mathematical proof. It is a severe mistake to
|
||
think that the programmer's products are the programs he writes; the
|
||
programmer has to produce trustworthy solutions, and he has to produce
|
||
and present them in the form of convincing arguments. Those arguments
|
||
constitute the hard core of his product and the written program text is
|
||
only the accompanying material to which his arguments are applicable.
|
||
|
||
\* \* \*
|
||
|
||
Many software projects carried out in the past have been overly complex
|
||
and, consequently, full of bugs and patches. Mainly the following two
|
||
circumstances have been responsible for this:
|
||
|
||
1. dramatical increase of processor speeds and memory sizes, which made
|
||
it seem as if the sky were the limit; only after the creation of a
|
||
number of disastrously complicated systems it dawned upon us, that
|
||
our limited thinking ability was the bottleneck;
|
||
|
||
2. a world that in its desire to apply those wonderful new machines
|
||
became over-ambitious; many programmers have yielded to the pressure
|
||
to stretch their available programming technology beyond its limits;
|
||
this was not a very scientific behaviour, but perhaps stepping
|
||
beyond the limit was necessary for discovering that limit's
|
||
position.
|
||
|
||
In retrospect we can add two other reasons: for lack of experience
|
||
programmers did not know how harmful complexity is, and secondly they
|
||
did not know either, how much complexity can usually be avoided if you
|
||
give your mind to it. Perhaps it would have helped if the analogy
|
||
between a software design and a mathematical theory had been widely
|
||
recognized earlier, because everyone knows that even for a single
|
||
theorem the first proof discovered is seldom the best one: later proofs
|
||
are often orders of magnitude simpler.
|
||
|
||
When C. A. R. Hoare writes —as he did early this year— "...the threshold
|
||
for my tolerance of complexity is much lower than it used to be" he
|
||
reflects a dual development: a greater awareness of the dangers of
|
||
complexity, but also a raised standard of elegance. The awareness of the
|
||
dangers of complexity made greater simplicity a laudable goal, but at
|
||
first it was entirely an open question whether that goal could be
|
||
reached. Some problems may defy elegant solutions, but there seems
|
||
overwhelming evidence that much of what has been done in programming
|
||
(and in computing science in general) can be simplified drastically.
|
||
Numerous are the stories of the 30-line solutions concocted by a
|
||
so-called professional programmer —or even a teacher of programming\!—
|
||
that could be reduced to a program of 4 or 5 lines.)
|
||
|
||
To educate a generation of programmers with a much lower threshold for
|
||
their tolerance of complexity and to teach them how to search for the
|
||
truly simple solution is the second major intellectual challenge in our
|
||
field. This is technically hard, for you have to instill some of the
|
||
manipulative ability and a lot of the good taste of the mathematician.
|
||
It is psychologically hard in an environment that confuses between love
|
||
of perfection and claim of perfection and, by blaming you for the first,
|
||
accuses you of the latter.
|
||
|
||
How do we convince people that in programming simplicity and clarity —in
|
||
short: what mathematicians call "elegance"— are not a dispensable
|
||
luxury, but a crucial matter that decides between success and failure? I
|
||
expect help from economic considerations. Contrary to the situation with
|
||
hardware, where an increase in reliability has usually to be paid for by
|
||
a higher price, in the case of software the unreliability is the
|
||
greatest cost factor. It may sound paradoxical, but a reliable (and
|
||
therefore simple) program is much cheaper to develop and use than a
|
||
(complicated and therefore) unreliable one. This "paradox" should make
|
||
us very hesitant to attach too much weight to a possible analogy between
|
||
software design and more traditional engineering disciplines.
|