Beta Software: Beta the devil you know
...than the software you don?t. Guy Clapperton extols the virtues of the people who lay down their lives, and their computers, to beta testing
It?s happened again. Your worst nightmare. No, not the one with the six-foot tall blonde pilchard who walks into your living room and says: ?Hello, I?m Joanna Lumley? then plants a great big kiss straight on your lips (oh, you get that one too?), but the Beta Tester Bug Nightmare. It?s a nasty one; this little bloke with glasses and spots rushes into your office and shouts ?yer bastard yer bastard yer?ve scuppered me business? and calmly explains that when you asked him to beta test Whifferdill Seven, the shapechanging accounts package, he started running all of his accounts on it and they?ve died horribly as a result.
Then you wake up and he?s still standing there. Only he?s not there in person, there?s a letter or email complaining that your beta software is riddled with holes, and people?s businesses have actually been damaged as a result.
Such a fate has befallen the likes of Microsoft and Netscape over the past couple of weeks; there are bugs in beta versions of both of their browsers and they?ve done a neat impersonation of certain brown substances hitting a rapid air distributor accordingly.
If you are using Internet Explorer (IE) 4, for example, and you visit a URL using streaming technology ? the sort that enables the display of partly-received pages while the rest are loading ? and SSL (secure socket layer) technology, then wahey, it?ll crash. You might of course prefer to use Netscape Navigator, but because of the implementation of Java, people are now able to send malicious applets down the wires which will wipe your hard disk when you open them.
Cute, you might think, but new technology needs to be tested. And for that, you need beta sites. The snag is how to build them up.
Colin Messitt, proprietor of shareware developer Oakley Data Systems, recruits people who have already shown interest in the products he develops. Once their interest and capacity has been established, he opens up a closed conference on a system such as Cix, preferring to steer clear of the internet. ?You get too many airheads out there,? he says.
Not that airheads need be such a disaster. Consultant Julian Voelcker doesn?t think so (let?s establish that quickly before any eager lawyers who happen to be reading this start going into overdrive), but he concedes that he simply hasn?t the time that some of the beta testers sitting in their spare rooms all day playing with software ? teenagers recruited through the magazines, often ? can spare.
Voelcker came through a more circuitous route, starting off by being approached to look into the then-imaginatively-entitled Chicago, since ploddingly renamed Windows 95. ?I was disappointed about a number of things in the beta version not getting into the final release, so I vented my anger by writing to Mr B Gates at Microsoft.?
Expecting nothing, he was surprised when the personal reply arrived, although not particularly stunned to note that it didn?t go into much detail on his issues. The upshot was that he was conscripted into something called Club Internet Explorer. ?The idea is we run around saying nice things about Microsoft products,? he says.
Microsoft and the other companies for which Voelcker now beta tests have set up newsgroups for testers to compare notes with each other. Messitt, although eschewing the internet model, is pleased to encourage testers to talk to each other in closed groups on the more parochial Cix system.
?It?s useful if, as well as talking to us, they can talk to each other. One of them may say ?this would be ideal for me if it did such and such?, and another might say ?and it?d need to be blue?, then explain why.?
The objectives in getting hold of beta testers can be many. They are looking for two things, basically, bugs and possible improvements. ?Personally I?m always looking for things I can suggest that might be put into the end product,? says Voelcker. ?You?re never going to find as many bugs as the 14 and 15-year-old kids who play with it in their bedrooms and sit there breaking it all day.?
He reports his progress on every package he evaluates, but considers that he is simply filfilling his obligation, no more.
Messitt definitely wants to hear from anyone who presses a key and finds the whole system collapses, or that it throws out unlooked-for results. ?We are also interested in the overall design and whether it matches the way the user will want to use it,? he says.
It is at this stage of a well-crafted featurette that you the reader expect a turning point of sorts ? something lucid, eloquent and informative to cap it all off. OK, here goes: beta test software is excellent stuff as long as you use it as something to play with, but there are people who do more than that. They run their business on it ? no, really. And frankly, if you use beta test software to run your business you are daft.
Voelcker freely admits that he expects to be ?bitten on the backside? at some stage by his habit of using beta systems for mission-critical stuff, something he does ?all the time?, and doesn?t have the decency to be even slightly embarrassed about. ?Touch wood, I?ve never lost any data through using beta software,? he says.
He had Windows 95 up and running as his sole operating system for a full year before it went public, and has no regrets doing so. He confesses, though, that he thinks he has been very lucky.
Messitt?s attitude is more conservative, in that he expresses the usual caveats about mission-critical stuff not being suitable for potentially flaky software. ?It depends on what it is,? he says. ?If it was a graphics drawing package I wouldn?t mind so much, I?d just switch back to Photoshop or whatever. If I had my accounts on it I?d be more wary.?
Sensible enough, but he ruins the effect by confessing that he now has IE 4 on his main machine rather than one of the sacrificial lambs normally used for beta stuff. PC Dealer tells him about the bug that makes disks crash ? it?s interesting, you?d never have thought you could actually listen to someone?s hair going grey over the telephone but in this case that?s what happens. ?Well, if my disk crashed in an hour I?d be a little upset, but not devastated.?
Quite rightly, he points out that full product can also throw out unexpected bugs, or ?issues? as certain software firms now call them. And the customers will have paid for these.
The key, it seems, is to treat beta software as what it is ? a dry run, something that isn?t even at the final alpha stage yet. Now try a little trick. Log on to any of the major IT press sources on the Web, and if they have a search facility ? most do ? enter the term ?beta?. You will find manufacturing, accounting, lots of functions that are not merely mission-critical, but which could be described as defining a company?s mission in themselves, being run on beta software.
Bewilderingly, companies that indulge in this potentially self-destructive policy are not merely honest about what they are doing, they are positively proud of it. ?Ho yez, we?re making all of our new fleet of cars using a system that might not work, but at least we?re not paying for it, and our database is still in beta too. But we?re sure it?s all right,? they might as well say. Bonkers, the lot of them.
But for developers, this deranged behaviour is in their interests. It?s easy to suggest people with a single PC should use their old one if testing beta systems, isolating anything that might have even a vague stab at corrupting precious data. But if your application is anything larger ? say a full-blown relational database or an enterprise networking system ? there is less chance of finding some 15-year-old kid who has enough boxes in their spare room to do a decent test of it.
But the fact is beta software is not finished. Beta software is designed to irritate reviewers when the manufacturer sends something out that it admits won?t work, but asks the evaluator to test it and say something intelligent about it.
Beta releases are things that might work if you?re lucky, but they are unregistered and the chances are you won?t get support for them after the beta programme is over. If you?re a developer, you might have the unreasonable idea that people ought to pay for a full product if they actually want to make a profit in business through using it (although you might also consider that offering people who give you feedback a free upgrade is no more than courteous, and is cheaper than paying a consultant).
But let?s leave the support issue to one side for a moment ? ask yourself how much of the above can?t be said of actual released software. The chances are, the answer is not a lot.
To kick this little tirade off, the dreaded ?M? word was dropped, you?ll recall, with a mention of some ferocious bogeymaking bugs in a beta version of Internet Explorer 4. The thing is, that wasn?t the whole story. You see, the same bug, at least the one that scuppers your disk if you combine streaming technology with SSL, hits the full-release version of IE 3 equally hard. The Java difficulties specified elsewhere for both Netscape and Microsoft are also out there in real world, real-time applications. So what is the difference?
The difference, basically, is that beta tests (controversial bit coming up here) don?t actually work terribly well. Granted, you get your feedback and detect some of the bugs, but you can bet anything you like that a whole set of them will still be there once you unleash the little darling on an unsuspecting public.
You?ll remember the furore about Windows 3 when it was first released, and the absence of any furore about 3.11 ? this being because Windows 3.11 was basically a bug fix. This can only mean the beta test didn?t quite throw out all the results it was hoped it would.
Messitt, a firm advocate of using beta testers, is the first to admit the idea can be flawed. Buggy software does indeed get out in the market. ?When I think of the one or two bugs that have got out in our products, they are often found by people in the wide world who?d do things sophisticated and like-minded people wouldn?t dream of,? he comments.
And it?s not just the fruitcakes who will find out, say, that pouring washing up liquid into the back of your computer makes it blow up in spite of there being nothing in the manual about it. In the week this feature was being written, academics noted three separate bugs in the two major internet browsers. The testers just hadn?t noticed.
And who says the unpaid tester should have all that responsibility anyway? You could almost argue that using the beta brigade is a wonderful means of obtaining free consultancy. If they?re happy enough with the software and a T-shirt ? a T-shirt, for pity?s sake ? it can?t cost as much as paying a consultant.
Once again, Messitt agrees up to a point, but stresses that if someone volunteers to beta test the chances are they?re doing so because they are interested and they?ll actually enjoy the experience.
It?s not just a leisure pursuit either, as Voelcker clarifies. As a consultant, if he can go to a customer with a problem and honestly claim he knows their package inside and out already, he?s at a distinct advantage. ?And you get a lot of free software,? he adds, no doubt sporting a corporate logo about his person even as he speaks. ?There are always goodies from the computer firms.? What, a headband as well as a T-shirt? Possibly.
Nevertheless, and in spite of his early experience of debating products with Micro-soft, Voelcker?s real satisfaction comes when some, if not all, of his suggestions are incorporated in the final released version of a product. ?Any suggestions and technical points you make are always taken notice of,? he says.
Cynical longer-term readers of and contributors to PC Dealer might scoff a little at anything that sounds half so altruistic ? no one does anything for free, we might think, especially not a consultant.
Except they do if they are beta testers. There is never any payment, and if the software is in frequent use there is often a comparatively high risk. At the end of the day, the software we eventually use would be poorer without them.