A balanced approach to measuring the intranet

Ever since intranets arrived on the corporate radar, understanding return on investment (ROI) and establishing metrics has been a challenge. The problem is that many intranets are dumping grounds for information. Balanced against them are those intranets that make a very real contribution to the business. But even in cases where cost savings aren’t the key driver for an intranet, with the right techniques cost savings can be realized.

The difficulty of ROI
Calculations of ROI in intranet implementations are particularly difficult for three reasons:

1. There may be no baseline data prior to use of the intranet, making “before” and “after” comparisons impossible. But this weakness can be addressed by using the Balanced Business Scorecard approach to measure against a set of evolving targets (See Figure 1).
2. There are too many other independent variables – isolating cause and effect is a challenge.
3. Some benefits are intangible and hard to quantify.

These challenges should not be an excuse to ignore ROI. There is a case for pragmatism, which says as long as the return identified meets the corporate standard for investment performance (and any well-conceived, CEO-driven, well-managed implementation should comfortably exceed this), then the whole exercise does not have to evaluate exhaustively each and every financial impact.

Figure 1: The generic Balanced Scorecard

Setting appropriate targets
Before the start of a project – an intranet included – the organization should set out a target, or what it wants to achieve. The best way to define that target is through key performance indicators. These should not be too much of a mystery, as the typical organization has around 50. Not all of them, however, will be relevant to an intranet project.

When you’ve defined the target, the next thing to define is “where the organization is now.” Often, of course, there will be a gap, and closing that gap is the journey the organization must make if it is to reap the benefits of the project.

The proof of the value of the intranet will reach optimum levels when the gap has been closed. So how do you understand where you want to go? Employee questionnaires are often a good place to start.

 

What to measure?
Over 500 years after Pacioli first documented the original metrics of double-entry bookkeeping, Kaplan and Norton enhanced these key mechanisms of management control to embrace forward-looking and broader measures using their concept of the Balanced Business Scorecard.

They defined a set of measures distributed into four categories. The categories range from those, typically the financial indicators, which take considerable time to impact the bottom line – the lagging indicators – to those which bring more immediate results – the leading indicators.

The scorecard is described as balanced because this method reconciles traditional historical measurements of corporate performance with a wider range of holistic measures (e.g. customer retention), the effectiveness of key business processes and the pace of organizational learning and cultural change inherent in a shift of the corporate mindset.

Setting up the collection and collation of internal data to feed a Balanced Business Scorecard is not a trivial task. It’s crucial that this element of the intranet strategy is adequately funded and enjoys board level sponsorship and visibility. Nor is this a task which has much value unless it’s part of a sustained program over many years.

And even that task can be dwarfed by the level of cultural change needed to make the numbers on the scorecard move in a positive direction. But it’s a crucial step if organizations are to realize lasting benefits from those investments which often flounder from lack of drive and uncertain direction.

 

Measuring for learning
The Philips Electronics Balanced Business Scorecard implementation, called “Business Excellence through Speed and Teamwork” (BEST) has set target setting and delivery measurement as core requirements for a CRM project. It’s a methodology well-suited to providing the management cockpit which is so essential for complex implementations such as major intranets.

Organizations like Philips get a further return on investment from the formal process of documenting and learning from their experience: measurement means recording; recording leads to action. The lessons of how changes and improvements were achieved (or not) and what corrective actions were applied and with what success are a vital asset in building the corporate experience.

Thus, measurement is also a significant step towards becoming a learning organization. In that respect, we come full circle where the meaning of “value” is concerned. Whether the value to be measured is about efficiency and effectiveness, ROI or corporate knowledge and intellectual capital, measurement is likely to contribute to the corporate good.

 

What makes a valid question?
Where measuring the effectiveness or value of an intranet is concerned, there are new online questionnaires that can be used to gauge the corporate pulse. They can help to close the gap between where the organization was in the pre-intranet phase, and what it hopes to achieve with the intranet. Questions can be designed to see what users think of it and how they use it, with the results to be measured against the original objectives.

People often write survey questions without an understanding of what makes a question valid. Questions such as, “Do you like the intranet? – Please rate on a scale of one to five” or “Has the intranet made your life easier?” are useful at the end of a survey to elicit qualitative proof and stories to show the boss that people are happier.

But relying on them solely will skew your results. Instead, task-based questions should be used, relating the rate of intranet use to the particular project being done.

It’s also useful to consider a couple of questions that judge the validity of people’s responses. For example, list five functions and ask which functions the employees use. Then go to the intranet logs and see if it’s true.

If it isn’t, organizations can take two approaches. One organization we worked with found out that its employees had ordered a lot of functionality from the IT department that they subsequently never used, which the logs proved. Managers then sent around an e-mail criticizing employees for the discrepancy.

A more positive approach is to ask why employees may have thought they needed the functionality but then didn’t access it. It could be that they weren’t rewarded adequately with the content they needed, and there should be further investigation.
Benefits that have “humanistic” aims – such as those involved in making staff feel more valued and satisfied in a knowledge acquiring and sharing culture – are by their nature fuzzier, but can be measured after users participate in the online (intranet-based) questionnaires.

 

Cost savings or revenue boosts
If ROI is the big concern, the subject can be looked at from two viewpoints: cost savings (or avoidance) and/or increases in revenue due to the intranet leading to improved performance. These measurements would not necessarily involve staff in online questionnaires, but they would require some proof from the financial department or accountants.

It may be easier to address cost savings, which come not only from not spending on document design, print and distribution, but also from improvements in the way that staff communicate and collaborate. An online questionnaire could quickly gauge the mood of staff, including management, on the latter point – i.e. “does the intranet enable us to communicate and collaborate more cost effectively” (or indeed, “are we sinking beneath waves of e-mail that are impeding communication”).

Online questionnaires need not involve all staff. Randomly selected focus groups could give a snapshot representative response to questions, which can be fine-tuned to drill deeper into what users think of any aspect of the intranet and affiliated knowledge systems or the corporate knowledge strategy.