Helping to dig for data

CRM and data warehousing have failed to deliver, so enterprises will have to use VARs to help them claw back the cash they spent on database technology. Nick Booth reports.

It is said that nothing really changes in the computer industry apart from the names. For example, look how databases have evolved.

In the 1980s we stopped calling them plain old databases and started referring to them as management information systems, even though they were essentially the same silos of customer records.

Then, in the wake of the backlash against all of that 80s flashiness, the same process was given a more blue-collar name: data warehousing.

Better still, you were able to examine the records by employing 'data mining' tools, as if there had really been a revolution and the horny-handed sons of toil were now delivering on simple, honest objectives.

The marketing ploy must have worked too, because enterprises poured billions of dollars into creating data warehouses in the belief that at last they really would have a decisive lead over their competitors. Did it work?

Nobody really knows because, like many of the promised benefits of IT, it is difficult to gauge its success.

There are growing suspicions that many of the technologies supposed to 'empower' companies with competitive intelligence - customer relationship management (CRM) and data warehousing, for example - have failed.

You can usually tell when the game's up for a technology, according to management consultant Simon Clark at Katapult-IT. It gets reinvented as something else.

"It's telling that these days no one talks about data warehouses. The new buzzwords are business intelligence," he said. "But essentially it has the same goals as CRM and data warehousing."

The problem is that many projects got out of hand and now need to be rethought. Data warehouses just became data dumping grounds, according to Clark, but CRM systems have done much more damage.

"Companies talked about having a relationship with their customers, but it was pretty one-sided," he explained.

"All they did was keep enough information on their customers to enable them to keep bombarding them with unsolicited phone calls and junk mail, trying to upsell them stuff they hadn't thought about buying.

"If CRM describes a relationship, it's where one party has to get a restraining order against the other. They should call it customer stalking software."

According to analyst Gartner, data warehousing projects have a massive failure rate, with more than 60 per cent of projects missing targets.

Despite this, Martin Butler, chief analyst at Butler Group, insisted that data warehousing cannot be abandoned.

"Most enterprises have sunk millions into data warehousing and not seen anything for it. But that doesn't mean they'll abandon the work," he said.

"They can't afford to. To get the return on investment, they will have to evolve to a new model of integrated business intelligence."

The news that even more money is needed to complete the project would be heart-sinking for most IT directors. Or at least it would be if they were spending their own money.

Channel opportunities
On the other hand, this is great news for the channel, because VARs and systems integrators will be called upon to do the tweaking, data cleaning and integration projects that will finally make databases deliver some useful information in a timely manner.

But why have data warehousing and CRM, to name but two concepts, been so disastrous? According to Butler, there are three reasons why projects usually fail.

First, they are usually far more expensive than originally mooted. The second failing is that the project usually goes on far longer than originally imagined. Then, as the last part of a triple whammy, the results are usually disappointing.

Cast your mind back to the early days of data warehousing. Companies were endlessly being told that if they used data mining tools on data warehouses their marketing departments would find relationships between the different reasons that a sale was made.

Having identified these, they could then put the elements in place to improve the chances of more sales being made.

The only example of successful analysis of this type was the story of the relationship between nappies and beer.

Apparently, someone analysed some sales data and found that beer sales on Friday afternoon peaked at the same time as nappy sales.

Further investigation (of sales receipts, presumably) showed that men in their 30s would stop off at a convenience store to buy nappies, and would often buy a pack of beer.

By identifying this trend, the company capitalised on it. They subsequently moved beer next to nappies on all their supermarket shelves, which brought about a much greater increase in impulse-buying of beer.

This was the only example of a successful data warehousing analysis that any marketing manager ever quoted. After a while, people began to wonder why nobody else could boast of such a marketing coup.

Many sceptics believe that the nappy/beer joint marketing success story was a work of fiction anyway.

The lack of genuine success stories to report on tells its own story about data warehousing and CRM, however.

Why didn't they work? "There was no lack of causal relationships between different fields of data, but often they were of no significance," said Butler.

"Marketing analysts would find that, say, barbecue equipment sold really well on days when the Stock Exchange was taking a hammering.

"From this they would conclude that there was some link between the two. But their marketing efforts would come to nothing."

Poor data analysis
But it wasn't the data that was at fault, it was the failure of corporations to analyse it properly. They only ever went a fraction of the way. There are three steps to business intelligence, argues Butler.

Stage one is finding relationships between different elements in data, which is what data warehousing practitioners might have achieved with their data mining tools.

But it doesn't end there. Next you have to test whether the relationships are significant, spurious or just mere coincidence. And finally, you must decide what you will do to act on that information.

There is a long way to go and a lot more money to be spent getting there, according to Butler. It must be a road worth travelling because Microsoft has thrown its hat into the ring and is about to announce its own brand of reporting tools.

Of course, they will be called something completely different, but when Microsoft enters a market and claims to have invented it, that's usually a pretty solid endorsement of the work that has gone on so far. Meanwhile, data mining has some ground to make up.

"I met someone recently who had spent three years using data mining tools to establish some patterns in the horse racing business," explained Butler.

"After three years of cross-referencing certain types of horse with certain types of conditions, and playing around with other variables, he had to conclude that there were no net results and no improvement in understanding.

"When you consider that most businesses are infinitely more complex than horse racing, you can understand why there's a lot to be done."

The age-old problem is that IT people in companies know how to use data mining tools but don't understand the data because they don't get involved in the main business.

On the other hand, the people who make the decisions that run the business don't have the understanding of analysis tools. Surely VARs have a role to play in bridging that gap?

Ian Black, director of corporate communications at Autonomy, agreed that a lot of work needs to be done to make information systems deliver something useful.

"The problem is that 80 per cent of information is unstructured, but 70 per cent of the value of information is locked in the unstructured data. There still needs to be a huge amount of manual effort to make information meaningful," he said.

One of the big mistakes Black identifies, which VARs could possibly rectify, is in the implementation of systems.

"Often, people aren't part of the process of a deployment. We forget that people remain people; they will never be standardised and will continue to be incompatible with many things," he said.

"We ignore how people learn and behave at our peril. If a deployment is imposed on them, they won't accept it as well as if they have a say in its installation."

The good news is that business intelligence is suddenly fashionable, because the spotlight has fallen on it as the present incarnation of database technology.

Market researcher IDC has indicated that the UK business intelligence market is growing at 60 per cent year on year.

Mark Ellis, vice president of Computer Associates' EMEA division, said: "The market is being driven by factors that people don't talk about, such as risk aversion.

"There's all kinds of legislation being passed that encourages firms to get a better understanding of their clients."

There is an argument that the real work that needs to be done is in fine-tuning the way distributed databases work.

Incompatible systems
Not every company has a data warehouse, and more frequently you find companies with a hotchpotch of different systems containing information about customers.

Often this network contains systems that are barely compatible, as a result of a company's rapid growth by acquisition.

The upshot is that extracting information is as slow and painful as extracting teeth, and can lead to inaccuracy. This costs firms millions of pounds a day, according to identity management consultancy White Obsidian.

"Companies with distributed databases discover that the flip side of this arrangement is that their information strategy is all over the place," explained Richard Burke, White Obsidian's co-founder.

"This causes them massive losses as opportunities are missed, and even day-to-day management of existing relationships gets more laborious."

Identity management takes the approach that the cost of managing 'digital identities' (i.e. database records) can be rationalised by reducing the complexity, pain and overheads.

The key to this is the deployment of directory technology, since understanding how directories work can be put to use in cleaning data, identifying and reducing duplicates, and ensuring that communication between databases is maximised.

The efficiency this brings about saves millions, argued Burke. Badly matched data is expensive then. If you know Microsoft's Active Directory from Novell's NDS, you could make a fortune.

A properly set up directory can manage the interfaces between different systems, and optimise the management, improve communication and ensure consistency of information.

Skills shortages
Sadly there is a serious shortage of people who have a wide enough range of understanding to cater for all the work that is out there.

"There's a real shortage of people who understand directories, which is a pity because we are in desperate need of more experts to help us with the projects we're being asked to embark on," said Burke.

"We're trying to recruit people all the time, but the quality and breadth of knowledge isn't there. Within five minutes of an interview we can usually tell that the candidate is bluffing."

Like data warehousing, the directory solution providers are going to suffer a backlash from IT directors.

According to independent sources, the misplaced loyalty that ICL/Fujitsu showed to Microsoft has not helped the cause of directory experts.

"ICL had a strong incentive to push Microsoft solutions, with the result that Active Directory was sold into areas where Novell's NDS would have done the job better," said one source.

"Now they've been forced down the Microsoft route, some companies are regretting the money they've spent which may have to be written off."

What is required are neutral VARs/system integrators that have good knowledge of the strengths and weaknesses of the main directory offerings - from Sun, Microsoft, Novell and IBM - and no compunction to push one over the other.

One company is alleged to have spent £20m over 12 months on a Microsoft project that was never completed, while an integrator promised to do it for £3m in four months. The problem is that Microsoft is adept at convincing people that they should standardise on everything Microsoft supplies.

Meanwhile, Novell claims that it is fighting back. "We need to change perception," said Steve Brown, Novell UK's managing director.

"We're the recognised leader in directories, and we're gradually convincing the big five systems integrators that they should come back to working with us."

The market for business intelligence will be one of the great growth areas in IT, according to Butler. And, as he points out, you can never over-estimate the lack of intelligence in this industry.

CONTACTS:

White Obsidian (020) 8213 5191
www.white-obsidian.com

Novell (01344) 724 000
www.novell.com

Computer Associates (01753) 577 733
www.ca.com

Copper Eye (01225) 745 408
www.coppereye.co.uk

Autonomy (01223) 448 000
www.autonomy.com

Butler Group (01482) 586 149
www.butlergroup.com

Responsetek (020) 7397 8580
www.responsetek.com

Katapult-IT (01932) 227 982
www.katapult-it.com