Monday, August 20, 2007

A short one

After auditing some of my coworkers' code today, I would like to paraphrase Felix von Leitner:

Program(mer)s get worse much faster than the computers get better.

Sunday, July 29, 2007

Progress v8: Progress is.

[This is my eighth, and final, article on progress. If you want to read the whole story, here is my first article]

This time I'm going to start with a little story, and the conclusion of this part is probably not what you would have expected, either, but that's what life is: a constant surprise package waiting to be opened.

I came to the demo party in Denmark, The Party 97, expecting to see cool things. One of the first cool things that we saw, after all the two thousand PCs and other computers were turned on simultaneously, was a power outage that took both the shiny Christmas decorations outside, the PCs inside, as well as the domestic power supply of half Aars. When the lights came back up and the demos actually started running, the Party Motto displayed by the demos, that originally had been "Batteries not included", was quickly hacked to display "Power not included" by almost every crew.

When I came to Aars I certainly didn't expect that the winner would be Second Reality, just like at Assembly '93 in Finland. It was the same demo, with music, the speech samples, the 3d part... everything. Except, the disk had to be turned over in the middle. Because this time, it wasn't running on a 80486 with 50 MHz, it was running on the Commodore 64, with an 800 KHz, 8-bit CPU, a machine that should, by all standards of measurement, be about 200 times slower than the machine the demo was originally written for.

People's jaws were hanging wide open.

But not all people's. If you read the reviews, you will see that the "old school" C64 crowd wasn't even that impressed. All of the effects have been done on the C64 BETTER at some point. It was the PC crowd that peed their pants for excitement.


And how can the C64 people with their tiny lame machines that can't even multiply in hardware outdo a PC?

Because they had 10 years more practice: the C64 was built in 1982. Read: 10 years of programming practice beats 6 years of hardware evolution.

Does that mean that the all that PC business had been superfluous, and the C64 architecture should have been kept forever? No new standards, no improvements, no changes...?

No, of course not. We could not do the things we do with computers now, if we would have been stuck with a system that doesn't really allow modularity. If the CPUs wouldn't have made the steps to 32 and now 64 bit, if memory sizes wouldn't have grown over the years, if the CPUs didn't have virtual memory management, modern genetics and Google would have been impossible. And by now, PCs are incorporating the design elements that made C64 and AMIGA technically superior back into the design: sound and graphic chips that do their own jobs with little CPU help, shared memory to avoid copying where memory is a constraint, better standards, etc, etc.

But the actual problem that this C64 demo has shown us in 1997 is by now all-pervasive in the entire software industry: nobody, but really nobody has more than a few years of practice using the new systems and frameworks, because the technologies become obsolete within less than two years from the moment of their creation, and everybody is forced to move on.

Remember the times where such a thing as "craftsmen" existed? People like smiths, or even, not quite so long ago, car mechanics, who spent their lifetimes learning how to do their job well. When knowledge evolved, it took generations. The next generation could learn the old things, invent new things, and pass both on to the next.

Today, we cannot afford to. If we only learn one trade, like one programming language, we will be obsolete within five years, max. If you can learn quickly, you won. If you learn slowly but well... well, you're screwed. By the time you have learned your one thing, it is already obsolete. Pass knowledge to your kids? You're kidding! Even the stuff they learn in school is already outdated. They only learn it so they know the theory. How to do things... well, they will have to learn that on the job.

This is, in my opinion, why there is so much bad code out there. Programs that don't work, because their creators have never learned to do their job well, instead having to spend the time to learn yet another new technology that will be obsolete in a year. This is why everybody and their grandfather nowadays think that they can program, because "X is a new technology, so I had as much time to learn it as everybody else out there." People who had never heard of "Product X" or anything similar two years ago, and when they heard of it, didn't understand it properly, are now sold as the top master level consultants for "Product X" by CSC.

What it really tells us, is, there is only that much speed at which progress can take place. There needs to be a balance between invention and expertise. Innovation cannot take place if you don't have any experience with the thing you are trying to innovate. Otherwise you will end up re-inventing the wheel again, and again, and again instead -- because you didn't have enough time to go out there and find out that wheels exist. Or because you are not the first victim of the cycle, and there are so many different wheels to choose from, that it is easier to just invent your own than to go through all the existing specifications.

You see, in the end, progress is its own worst enemy.

Friday, June 22, 2007

Progress v7: So old is good... or is it?

[OK, it has been two weeks again, and this is my seventh article on progress. I have been a little busy, and the progress stuff is really starting to get complicated at this point. But you can always start with my first article ;-]

So in the last episode we have seen that standards are good for you. In fact, the only reason why our technology works, for example, why you are able to read this, is because standards exist.

This is also how science is made: using common standards (of measurement), one theory can be based on another. So the new researchers don't have to go and re-invent all the theories before they finally can start producing original work¹.

So old is good, right? We can build on a foundation happily ever after?

Except, of course, when it is flawed. Then you can either stop when you recognize that your house is going to fall, or build all the way up until it falls apart. The same good practice that allows us to finally build high also allows us to build on errors of others.

Take the science of Astronomy for example. For hundreds of years it could provide us with no insights more useful then astrology, because it was based on the flawed premise -- the Ptolemaic Universe. It was only after Copernicus and Galilei that we started to be able to look into the actual workings of the universe, because they found the fundamental flaw in the theory of the world to that point: the Earth was not the center of the universe.

If not for the Copernicus, would Galileo have ended up observing the laws of motion? Would Newton have formulated the theory of gravitation based on that?

On the other hand... if not for Aristotle and Ptolemy, would somebody have discovered the laws two thousand years earlier?

We'll never know of course. What we do know, is, that about 2250 years ago, a man called Aristachus of Samos had already developed a very modern view of the solar system. It's just, the geocentric model of Aristotle was viewed as a much simpler explanation for why there is no parallax -- why stars don't visibly "wobble" when the earth move around the sun. In Aristotle's model the earth simply didn't move -- and the calculations for the positions of the planets produced results with accurate enough to look right. And thou shalt never change a running system...

This is one example that shows that with too little information, Occam's Razor can be applied both ways.

It also shows, that based on the same data, you can develop one perfectly working model that is also perfectly useless as anything but a working model (some of the modern unification theories come to mind), and another working model from which you can infer additional information about the subject -- extended use, so to say.²

And also, often enough, you don't see which is which until waaaay later.

So, competing theories, competing libraries, competing standards, are good... if and only if there is suspicion that the current system they are competing with is really inadequate and it hampers development.

But the thing is... Even if software people finally agree on standards, even if people actually cooperate instead of developing against each other, even if we all build on the same foundation... we are still shooting at a moving target.

How so? Well... see you in the next episode of the progress wars!

¹:This is why, when somebody says, "I am just as entitled to an opinion about meteorology as scientist X" he is talking BS -- unless, of course, he actually spent as much time studying the theories behind meteorology as scientist X, and has no bias.
A bias might be his conviction, or it might be his boss telling him that whatever the logical answer would be, the answer that will keep his job safe, is the one that his boss wants to hear. Unfortunately, the laws of physics are not political, only the people studying them are. But this is a different topic.

²: Like, for example, an open-source software library: there, I can learn from something other people already invented and thought through before me. If I see enough of that, I can also pick the components that I think are right, and discard the components that I think are wrong. In other words, I can learn.
With closet, err, closed-source code, this is impossible. You can only build on top of it, and hope that the developer really knew what he was doing and made no mistakes that will affect what you are doing.
Kind of like building on ground where you can't check whether it actually is a swamp. In fact, it is illegal for you to check whether it is a swamp, and you have to build on that ground because "everybody else does it". And where the holy creator of the ground thought that investing 20% of the necessary work is good enough.

Friday, June 1, 2007

Charity for the Rich (a tangent)

Let's leave progress aside, for a moment, and look at a very strange development:
Year after year, developers all over the world donate their expensive time to rich corporations like Microsoft or Linden Labs for free... and for no added value to themselves or their products. Not only that, but the time and money also just get thrown away by the corporations they donate it to! Why? Let's see.

Let's take Microsoft as the first example.

Every time you design a web site, you can do it one of two ways: standard-compliant, or Microsoft. These two are mutually exclusive, since Microsoft decided to take the otherwise standard tags, CSS rules etc, and render them differently from everybody else. There are scores of websites dedicated to how to work around "misapplications of standards" (which are, in my opinions, bugs and not a separate class of problem.)

Every web designer who wants his website to be viewed both by IE users, and by everybody else, basically has to develop his web site twice, or at least spend a lot of time learning the incompatibilities of IE to other browsers, and IE7 to IE6 to IE5 and so on, and ways to make his web site work with every one of those more-or-less properly -- since all of those browsers have different sets of problems. Most people I know who stopped doing web design, have stopped for this exact reason.

Why does this problem exist? Because Microsoft considers it more important to fix some bugs then others (as can be read frequently on Chris Wilson's blog), completely disregarding the fact that fixing some bugs will either make IE-recognizing code fail, or the browser will be recognized as IE but will not render the web page as expected, resulting in the necessity to add yet another version of the same web page, to cater for yet another partially correct version of IE. (See also, IE8 announced. Opt in? Does this mean, I have to go back to all my pages and add new code that will cause IE8 to render it the same as Firefox already does it?)

To sum it up: because Microsoft doesn't want to invest money in standard compliance, every single web developer has to invest money out of his own pocket to fix Microsoft's problems.

WOW! This is a perfect example of capitalism: extracting surplus value from somebody who isn't even your employee!

Except... the amount of money wasted worldwide in this operation is -- even by a conservative estimate -- hundreds of times larger than the amount of money Microsoft would have to invest to fix the problem. (Just compare the number of website developers worldwide that have to cope with the problem to the amount of developers working on IE.)

And with this, we arrive at Linden Labs.

There has been a lot of attention to Second Life, the Linden Labs' 3D environment, lately. As I noted earlier on a different blog (I will re-post it here eventually), Second Life is neither unique, nor technologically more advanced than similar solutions (e.g. There.com). What makes them unique is that creating new content is really cheap: the environment has a (crappy) built-in object editor, and uploading new textures costs only L$10 -- about 3 cents. They also allow you to sell stuff, and earn L$, which can be easily converted into real (US) Dollars, at a price.

Many a developer starts, starry-eyed, to develop new content for the system, thinking of what great things you could potentially do in a 3D environment. Some of them do manage to create decent things (and some of them, even, to sell them), before they ultimately collide with the technological barriers of the Linden Scripting Language and Second Life physics. Which, to put it gently, suck ocean liners through cocktail straws. For example, until March 2007 you could not create animals that actually moved their legs, without writing a distributed swarm-like system of scripts -- one for every joint. And if you actually manage to write something that doesn't collapse under the weight of its own code, you would discover that the physics system has bugs that prevent the "animal" from working properly anyway.

But I digress.

The really interesting fact about Second Life is that most of the content sucks is created by it's users, whereas most of the money is made by Linden Labs, since you pay them
  • every time you upload anything
  • every time you change dollars into L$
  • every time you change L$ into dollars
  • every month -- if you "own" a piece of virtual property
But not only that -- the content of the 3D world is, in large part, what attracts the users to the world in the first place. So, by making new content for Second Life, you make their world more attractive for new users -- who come, and pay their money to Linden Labs. And LL didn't even have to pay anybody for it. Linden Labs, too, is extracting surplus value from people who aren't even Linden Labs employees.

In fact, now that they open-sourced the Second Life client, they don't even have to maintain their 3D client themselves, as hundreds of enthusiasts are fixing their code for them too -- donating their time to a company that is bathing in money. Money enough, in fact, so that they can just buy additional servers, and not worry about the fact that the server code is as inefficient as it is... But that is a subject of a different discussion.

And you know what? They waste most of the users' efforts, too.

Almost every Wednesday, the entire Second Life network goes down for maintenance. When it comes back, often enough, many objects that worked perfectly before, don't any more. Because, for example, a five-year-old bug that everybody had to work around, and which had been incorporated into every moving object as a feature, got "fixed". Which means, just as with every version of IE (which, luckily for us, doesn't come around quite that often), every programmer has to check every script after every update. Not many do, which means that a significant percentage of scripts, objects and weapons in Second Life are broken at any given time.

The interesting difference between Linden Labs and Microsoft is: Microsoft is just a wannabe monopolist of the Web, while Linden Labs has absolute control over Second Life. Linden Scripting Language is their own invention, used nowhere but in Second Life -- so if they choose to say "everybody uses this, we should not fix it", nobody would complain, because unlike the case of Microsoft and the web standards, it would be a fact, with "everybody" actually being 100% of Second Life developers who use the given function, since there is no alternative.

So, what do we learn from these examples?

If we donate our time and money to corporations that don't actually need it, they will not say "thank you" -- they will, in fact, not even think about it, respect it, or pay any attention to it.

So if you are a developer... just don't do it. If you want to donate your time and your knowledge to something, choose an Open Source project. At least in this case, the user community will profit, instead of it being a complete waste of your time.

Progress v6: is the age of standards the worst enemy of Progress?

[This is my sixth article on Progress. I'm sorry for the long period of silence on that matter, the real life had caught up with me, but there is also good news: this is not, as planned, the last article on progress. There is at least one more, or more likely, two more, to come.]

In the previous five articles I made the impression, that contrary to the public opinion, to achieve actual progress, older is better. And in the cases I mentioned, it is. But let's look at an extreme example of this, the by now fairly well known story that "space shuttle engine's width is two horse asses plus X".

While the latest conclusion is, that, indeed, this is an urban legend, and the space shuttle might not have any relation to a horse's ass whatsoever, the basic story (bar the newer, Usenet-based addition with the space shuttle), that the width of the railroad track is based on the dimensions of two horse rears, same as the chariot width in the old Rome is affirmed in the same article:
[...]the dimension common to both was that of a cart axle pulled by two horses in harness (about 1.4m or 4ft 8in). This determined both the Roman gauge and Stephenson's, which derived from the horsedrawn wagon ways of South Northumberland and County Durham coalfields.[...]
[from: Crow, James. Housesteads. London: B.T. Batsford, 1995. ISBN 0-7134-6085-7 (pp. 33-34).]
The same article also says:
At the time of the Civil War, even though nearly all of the Confederacy's railroad equipment had come from the North or from Britain (of the 470 locomotives built in the U.S. in 1860, for example, only 19 were manufactured in the South), 113 different railroad companies in the Confederacy operated on three different gauges of track. This lack of standardization was, as historian James McPherson points out, one of the many reasons the Union was able to finally vanquish the Confederacy militarily:
The Confederate government was never able to coax the fragmented, run-down, multi-gauged network of southern railroads into the same degree of efficiency exhibited by northern roads. This contrast illustrated another dimension of Union logistical superiority that helped the North eventually to prevail.
[from: McPherson, James. Battle Cry of Freedom. New York: Oxford Univ. Press, 1988. ISBN 0-19-503863-0 (pp. 318-319, 514-515)]

So what does this tell us? First of all, that the conclusion that
In other words, there was nothing inevitable about a railroad gauge supposedly traceable to the size of wheel ruts in Imperial Rome. Had the Civil War taken a different course, the eventual standard railroad gauge used throughout North America might well have been different than the current one.

is both somewhat true,... and, to a degree, a circular argument: "The war was won partly because of the railroad gauge, but had the war taken a different course, the railroad gauge would have been different." (If the outcome of the war wouldn't have depended on the railroad gauge, that is?)

The second, and most important, conclusion is, in a collaborative environment, it is better to use well-established standards then not to use them: having a standard-sized railroad might have helped the South to win the war, by allowing the trains to pass through an unified railroad system instead of having to unload and re-load trains (or change the gauge on the wagons, as they do it when trains transit from e.g. Russia into western Europe and vice-versa).

And in modern times, there is even more to say for keeping of the standard: even though the width of the railroad track might not be entirely optimal for modern trains, there is an enormous amount of existing equipment based on the standard, which would ALL have to be changed if somebody decided to switch standards.

Or in other words, there is no reason not to use a standard just because it is 2000 years old, but there is a lot of reason to re-use existing equipment. And it doesn't preclude us in the least from making progress, (for example, in this particular case, from developing high-speed trains like ICE3 and TGV)

Which, incidentally, is what I have been saying for the last five articles. So no, the age of standards doesn't impede progress to any significant degree, if the standards make sense. This is an important distinction. The history of science would be much more straightforward if it wasn't. How so?... Come back next week(or next month?), and see for yourself, in... Progress wars! ;-)

Sunday, May 13, 2007

Progress v5: Is greed the worst enemy of progress?

[This is my fifth article on progress. Believe it or not, more is coming]


In the previous article we've seen that marketing is not a friend of progress, because it binds existing know-how into doing useless things. But why do companies see fit to create hype? Why on earth would two big corporations go to the length to create programming languages and to create a hype around them? And why would entrepreneurs generate Buzzwords like WEB2.0 and sell their business plans based on those?

Let's take those examples one by one.

The companies' explanation for the programming language hype was:
"because it is cross-platform" and "because it is simple and powerful".

Simple? How long does it take a newbie to get the signature of "public static void main(String[] args)" right? And to remember to wrap it in a "public class Foo"? (remember, there was no eclipse or netbeans when "java" started, no auto-completion etc.) A Foo that is also in it's own, properly named file? And that he has to rename the file to Foo.java (from foo.java or FOO.JAVA) when he copies the file to any real file system?

Now why didn't they, for example, take a sub-set of C++ and modified it to produce byte code similar to java's? It's not that hard to change the assembler part of an existing machinery. Or a language similar to Pascal or Delphi, if they wanted a "stricter" language? Why invent a language with a longer (and therefore slower to type ==> more typo-prone) syntax that had, at the beginning, any number of shortcomings to compensate for the benefit of the garbage collector? (Oh, by the way, pretty much all scripting languages where you can dynamically create objects and reference them have one, even Visual Basic.)

I'd say, it suspiciously looks like the real reasons were
- because they wanted to OWN a programming language (tm).
- because they didn't want new software that would compile with existing compilers out there, but wanted to bind people to their platform.
- because if you own a programming language, you can sell all kinds of things associated with it:
- lessons
- books
- software services
- additional software libraries (..."enterprise editions"...)
- Also, at the beginning you are the only one who can. Since you invented it, and you own it, and you don't publish the specs until way later when other people start to write books and libraries for your language.
- At which point you simply create a new version that has new features that are not backwards-compatible to the old one, and start selling new books and libraries for that one.
- rinse, repeat, ad nauseum.

And how do you bait people into it?
- promise "cross-platform" ("compile once, run anywhere"), since this is a known problem in the software industry.
- promise "less error-prone" and "faster software development" through "garbage collection" and "type safety".
(The solution of using a sub-set of C++ where you make certain operators illegal and that compiles into a byte code would, by the way, do the very same thing, but allow you to compile the stuff with existing compilers, and allow existing C++ programmers with >10-15 years of experience to program on the new platform with no adjustment time. And this would be bad... how?).

Summary: the motivation is to create a new, previously non-existing market. To earn money. In other words, greed.

And since Microsoft EXCELs at greed, they simply had to follow that trail with .NET. Promising basically the same things.

And of course, the motivation to put IT-Buzzwords (WEB2.0, AJAX, XML, ...) on a business plan is exactly the same: attract investors => get money. And of course the entrepreneur would do that. Since the investor wouldn't budge unless he SEES a buzzword! Or how else should he, who has no idea of the trade, recognize what's worth investing into?

Take AJAX, for example. It looks very much like the word has been invented/hyped solely for that reason -- since people have been using the actual technologies behind it for years (a little google search will help), and also, in many configurations used today technically it's not really AJAX because it transfers the data as YAML or javascript/JSON -- because with many types of data XML garbage-to-data ratio is a little high. Like, 2:1 or worse. Even java2 object serialization has a compacter format and is faster to parse... But that's a different topic.)

So what does all of that tells us? Similar to the points I've discussed in PrOgReSs V3:

- If somebody invents a new toolbox and convinces people that "his is better" while it is actually not, or
- if people are forced to re-start their projects using buzzword technologies instead of working technologies because otherwise they don't get funded,

it takes people with know-how working on exiting projects, and binds them learning the new stuff that doesn't let them accomplish whatever they were working on any faster. Which means, now they have to add the learning curve of the new stuff to the time their projects need to get done -- with no additional benefit.

And greed (or some would say, capitalism) is responsible for both of the phenomena.

So is greed the worst enemy of progress?

Well, that could very well be. But there is a little more to that story. It has to do with human life span, experience, time, and of course, progress. Tune in for the next episode of the progress wars... ;-) probably next week.