LLM Book Knowledgebase

How to Measure Anything: Finding the Value of Intangibles in Business

32 min read

How to Measure Anything: Finding the Value of Intangibles in Business

Metadata

Highlights & Notes

correct a costly myth that permeates many organizations today: that certain things can’t be measured.

When you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind; it may be the beginning of knowledge, but you have scarcely in your thoughts advanced to the state of science. —Lord Kelvin (1824–1907),

The “flexibility” to create new products The value of information The risk of bankruptcy Management effectiveness The forecasted revenues of a new product The public health impact of a new government environmental policy The productivity of research The chance of a given political party winning the White House The risk of failure of an information technology (IT) project Quality of customer interactions Public image The risk of famine in developing countries

Why do we care about measurements at all? There are just three reasons. The first reason—and the focus of this book—is that we should care about a measurement because it informs key decisions. Second, a measurement might also be taken because it has its own market value (e.g., results of a consumer survey) and could be sold to other parties for a profit. Third, perhaps a measurement is simply meant to entertain or satisfy a curiosity (e.g., academic research about the evolution of clay pottery).

Measurement is about supporting decisions, and there are even “micro-decisions” to be made within measurements themselves.

If the outcome of a decision in question is highly uncertain and has significant consequences, then measurements that reduce uncertainty about it have a high value.

Applied Information Economics: A Universal Approach to Measurement Define the decision. Determine what you know now. Compute the value of additional information. (If none, go to step 5.) Measure where information value is high. (Return to steps 2 and 3 until further measurement is not needed.) Make a decision and act on it. (Return to step 1 and repeat as each action creates new decisions.)

measure what matters, make better decisions.

Success is a function of persistence and doggedness and the willingness to work hard for twenty-two minutes to make sense of something that most people would give up on after thirty seconds. —Malcolm Gladwell, Outliers: The Story of Success

you should start thinking about measurements as a multistep chain of thought. Inferences can be made from highly indirect observations.

This approach to solving a Fermi question is known as a Fermi decomposition or Fermi solution. This method helped to estimate the uncertain quantity but also gave the estimator a basis for seeing where uncertainty about the quantity came from. Was the big uncertainty about the share of households that had tuned pianos, how often a piano needed to be tuned, how many pianos a tuner can tune in a day, or something else? The biggest source of uncertainty would point toward a measurement that would reduce the uncertainty the most.

The concept of measurement as “uncertainty reduction” and not necessarily the elimination of uncertainty is a central theme of this book.

Measurement: A quantitatively expressed reduction of uncertainty based on one or more observations.

For all practical decision-making purposes, we need to treat measurement as observations that quantitatively reduce uncertainty. A mere reduction, not necessarily elimination, of uncertainty will suffice for a measurement.

Shannon proposed a mathematical definition of information as the amount of uncertainty reduction in a signal, which he discussed in terms of the “entropy” removed by a signal.

the key lesson is that measurements are more than you knew before about something that matters.

A problem well stated is a problem half solved. —Charles Kettering (1876–1958), American inventor, holder of 300 patents, including electrical ignition for automobiles There is no greater impediment to the advancement of knowledge than the ambiguity of words. —Thomas Reid (1710–1769), Scottish philosopher

Once managers figure out what they mean and why it matters, the issue in question starts to look a lot more measurable.

The clarification chain is just a short series of connections that should bring us from thinking of something as an intangible to thinking of it as tangible.

Clarification Chain If it matters at all, it is detectable/observable. If it is detectable, it can be detected as an amount (or range of possible amounts). If it can be detected as a range of possible amounts, it can be measured.

Figure out what you mean and you are halfway to measuring it.

Most of the apparently difficult measurements, however, involve indirect deductions and inferences.

sometimes even small samples can tell you something that improves the odds of making a better bet in real decisions.

Rule of Five There is a 93.75% chance that the median of a population is between the smallest and largest values in any random sample of five from that population.

The Single Sample Majority Rule (i.e., The Urn of Mystery Rule) Given maximum uncertainty about a population proportion—such that you believe the proportion could be anything between 0% and 100% with all values being equally likely—there is a 75% chance that a single randomly selected sample is from the majority of the population.

“If you don’t know what to measure, measure anyway. You’ll learn what to measure.”

The only valid reason to say that a measurement shouldn’t be made is that the cost of the measurement exceeds its benefits.

it has no value. This is a slight oversimplification, but what makes a measurement of high value is a lot of uncertainty combined with a high cost of being wrong.

If you are betting a lot of money on the outcome of a variable that has a lot of uncertainty, then even a marginal reduction in your uncertainty has a computable monetary value.

“Is there any measurement method at all that can reduce uncertainty enough to justify the cost of the measurement?”

It is impossible to find any domain in which humans clearly outperformed crude extrapolation algorithms, less still sophisticated statistical ones.

The fact is that the preference for ignorance over even marginal reductions in ignorance is never the moral high ground. If decisions are made under a self-imposed state of higher uncertainty, policy makers (or even businesses like, say, airplane manufacturers) are betting on our lives with a higher chance of erroneous allocation of limited resources. In measurement, as in many other human endeavors, ignorance is not only wasteful but can also be dangerous. Ignorance is never better than knowledge.

Four Useful Measurement Assumptions It’s been measured before. You have far more data than you think. You need far less data than you think. Useful, new observations are more accessible than you think.

when you know almost nothing, almost anything will tell you something.

The problem was not a lack of data but the existence of so much data that wasn’t in a structured, easily analyzed database.

when you know next to nothing, you don’t need much additional data to tell you something you didn’t know before.

“satisfaction index.” The Cleveland Orchestra was just a bit more resourceful with the data available: It started counting the number of standing ovations. While there is no obvious difference among performances that differ by a couple of standing ovations, if we see a significant increase over several performances with a new conductor, then we can draw some useful conclusions about that new conductor. It was a measurement in every sense, a lot less effort than a survey, and—some would say—more meaningful. (I can’t disagree.)

Above all else, the intuitive experimenter, as we showed the origin of the word “experiment” denotes, makes an attempt. It’s a habit. Unless you believe you already know in advance the precise outcome of an attempted observation—of any kind—then that observation tells you something you didn’t already know. Make a few more observations, and you know even more. Think of measurement as iterative. Start measuring it. You can always adjust the method based on initial findings.

Prior to making a measurement, we need to answer the following: What is the decision this measurement is supposed to support? What is the definition of the thing being measured in terms of observable consequences and how, exactly, does this thing matter to the decision being asked (i.e., how do we compute outcomes based on the value of this variable)? How much do you know about it now (i.e., what is your current level of uncertainty)? How does uncertainty about this variable create risk for the decision (e.g., is there a “threshold” value above which one action is preferred and below which another is preferred)? What is the value of additional information?

The first two questions define what this measurement means within the framework of the decisions which depend on it. In the context of management, if a measurement matters at all, it is because it must have some conceivable effect on decisions and behavior. If managers can’t identify a decision that could be affected by a proposed measurement and how it could change those decisions, then the measurement simply has no value.

Information has value because it reduces risk in decisions.

I’ve heard managers say that they need to measure, say, project performance in order to track project progress. Of course, this is a circular statement. The question they need to ask is what might they change about the project if they knew more about its progress? Could they cancel it? Would they accelerate its implementation? Would they reallocate funds within the project?

An unidentified decision is no better than having no decision in mind at all.

It is also routinely a wasted resource. The data on the dashboard was usually not selected with specific decisions in mind based on specific conditions for action. It was often merely hoped that when the right conditions arose in the data, the manager would recognize a need to act and already know what action is required in sufficient detail that they could react without any delay. It is usually not, for example, worked out in advance that when revenue at a particular store or region drops by 10% compared to seasonally and economically adjusted sales targets, that a predefined project will be executed to correct it. Nor was it worked out in advance that the 10% change was the point that justified that particular action.

There have to be two or more choices and the best choice is not certain.

A decision has potentially negative consequences if it turns out you took the wrong position.

Finally, a decision has a decision maker. Difficulty defining a decision sometimes comes down to simply identifying whose decision it is.

Decision Model Estimated Costs of Action X. Estimated Benefits of Action X. If Benefits of Action X exceed Costs of Action X, execute Action X.

Uncertainty: The lack of complete certainty, that is, the existence of more than one possibility. The “true” outcome/state/result/value is not known. Measurement of Uncertainty: A set of probabilities assigned to a set of possibilities. For example: “There is a 60% chance this market will more than double in five years, a 30% chance it will grow at a slower rate, and a 10% chance the market will shrink in the same period.” Risk: A state of uncertainty where some of the possibilities involve a loss, catastrophe, or other undesirable outcome. Measurement of Risk: A set of possibilities each with quantified probabilities and quantified losses. For example: “We believe there is a 40% chance the proposed oil well will be dry with a loss of $12 million in exploratory drilling costs.”

instrument is calibrated to ensure it gives proper readings. Calibrated probability assessments are the key to measuring your current state of uncertainty about anything.

The most important questions of life are indeed, for the most part, really only problems of probability. —Pierre Simon Laplace, Théorie Analytique des Probabilités, 1812

In statistics, a range that has a particular chance of containing the correct answer is called a confidence interval (CI).

90% CI is a range that has a 90% chance of containing the correct answer. For example, you can’t know for certain exactly how many of your current prospects will turn into customers in the next quarter, but you think that probably no less than three prospects and probably no more than seven prospects will sign contracts. If you are 90% sure the actual number will fall between three and seven, then we can say you have a 90% CI of three to seven.

assessing uncertainty is a general skill that can be taught with a measurable improvement.

You can also force your natural anchoring tendency to work the other way. Instead of starting with a point estimate and then making it into a range, start with an absurdly wide range and then start eliminating the values you know to be extremely unlikely. If you have no idea how much a new plastic injection molding factory will cost, start with a range of 10 billion and start making it narrower.

Repetition and feedback. Take several tests in succession, assessing how well you did after each one and attempting to improve your performance in the next one. Equivalent bets. For each estimate, use the equivalent bet to test if that range or probability really reflects your uncertainty. Consider potential problems. Think of at least two reasons why you should doubt your assessment. Avoid anchoring. Think of range questions as two separate binary questions of the form “Are you 95% certain that the true value is over/under (pick one) the lower/upper (pick one) bound?” Reverse the anchoring effect. Start with extremely wide ranges and narrow them with the “absurdity test” as you eliminate highly unlikely values.

[I]n the conception we follow and sustain here, only subjective probabilities exist—i.e., the degree of belief in the occurrence of an event attributed by a given person at a given instant and with a given set of information.18

It is better to be approximately right than to be precisely wrong. —Warren Buffett

Remember, if a measurement matters to you at all, it is because it must inform some decision that is uncertain and has negative consequences if it turns out wrong.

Risk Paradox If an organization uses quantitative risk analysis at all, it is usually for routine operational decisions. The largest, most risky decisions get the least amount of proper risk analysis.

there are really only three basic reasons why information ever has value to a business: Information reduces uncertainty about decisions that have economic consequences. Information affects the behavior of others, which has economic consequences. Information sometimes has its own market value.

Value of Information Expected Value of Information (EVI) = Reduction in expected opportunity loss (EOL) i.e., EVI = EOLBefore Info - EOLAfter Info Where EOL = chance of being wrong × cost of being wrong Expected Value of Perfect Information (EVPI) = EOLBefore Info (EOLAfter is zero if information is perfect)

But for now, just know that each additional decrease in uncertainty usually takes more effort than previous decreases in uncertainty.

Knowing something about the monetary value and cost of the information in a measurement puts a new light on what is “measurable.” If someone says a measurement would be too expensive, we have to ask, “Compared to what?” If a measurement that would just reduce uncertainty by half costs 5,000,000, then the measurement certainly is not “too expensive.” Indeed, it would be a bargain. But if the information value is zero, then any measurement is too expensive. Some measurements might have marginal information values—say, a few thousand dollars; not enough to justify some formal effort at measurement but a bit too much just to ignore. For those measurements, I try to think of approaches that can quickly reduce a little uncertainty—say, finding a related study or making a few phone calls to a few more experts.

Myth: When you have a lot of uncertainty, you need a lot of data to tell you something useful. Fact: If you have a lot of uncertainty now, you don’t need much data to reduce uncertainty significantly. When you have a lot of certainty already, then you need a lot of data to reduce uncertainty significantly. In other words—if you know almost nothing, almost anything will tell you something.

The relatively high return on small uncertainty reductions is magnified further when we consider time-sensitive decisions. For example, if you are considering major investments in real estate when the market is low, the opportunity may itself evaporate if you take too long to make a decision. Reducing uncertainty doesn’t just take money, it takes time. This may lead to the type of decision error Howard Raiffa, one of the pioneers of decision theory, referred to as “solving the right problem too late.”

Once again, chasing perfect certainty or surrendering to unaided expert opinion are both costly mistakes.

The Measurement Inversion In a decision model with a large number of uncertain variables, the economic value of measuring a variable is usually inversely proportional to how much measurement attention it typically gets.

If the organization is used to using, say, surveys to measure things, uncertainties that are best measured with other methods probably don’t get measured as often. If the organization is good at measuring things based on data-mining methods, it will tend to measure only variables that lend themselves to that approach.

It seems that to have a truly profound revelation, you almost always have to look at something other than what you have been looking at in the past. Being able to compute the value of information has caused organizations to look at completely different things—and doing so has frequently resulted in a surprise that changed the direction of a major decision.

Understanding how to measure uncertainty is key to measuring risk. Understanding risk in a quantitative sense is key to understanding how to compute the value of information. Understanding the value of information tells us what to measure and about how much effort we should put into measuring it.

First, we know that the early part of any measurement usually is the high-value part. Don’t attempt a massive study to measure something if you have a lot of uncertainty about it now. Define it, assess your current uncertainty, and determine how much it matters to the decision by computing its information value. Then, if it’s a high value measurement, measure a little bit, remove some uncertainty, and evaluate what you have learned. Were you surprised? Is further measurement still necessary? Did what you learn in the beginning of the measurement give you some ideas about how to change the method? Iterative measurement gives you the most flexibility and the best bang for the buck.

Lessons from Computing the Value of Information Value of Measurement Matters. If you don’t compute the value of measurements, you are probably measuring the wrong things, the wrong way. Be Iterative. The highest-value measurement is the beginning of the measurement, so do it in bits and take stock after each iteration.

Nor were they, like some managers today, dismissive of instruments because they had limitations and errors of their own. Of course they have errors. The question is “Compared to what?” Compared to the unaided human? Compared to no attempt at measurement at all? Keep the purpose of measurement in mind: uncertainty reduction, not necessarily uncertainty elimination.

Decomposition involves figuring out how to compute something very uncertain from other things that are a lot less uncertain or at least easier to measure.

Decompose It Many measurements start by decomposing an uncertain variable into constituent parts to identify directly observable things that are easier to measure.

Decomposition effect: The phenomenon that the decomposition itself sometimes turns out to provide such a sufficient reduction in uncertainty that further uncertainty reduction through new observations are not required.

about 25% of the high-value measurements were addressed with decomposition alone.

decomposition alone is a big help especially when quantities are highly uncertain. Of course, these are exactly the quantities that will tend to have higher information values in decisions.

One example is how Amazon.com provides free gift wrapping in order to help track which books are purchased as gifts. At one point Amazon was not tracking the number of items sold as gifts; the company added the gift-wrapping feature to be able to track it. Another example is how consumers are given coupons so retailers can see, among other things, what newspapers their customers read. Inexpensive personal sensors and apps for smart devices are available for many types of measurement about human activity.

Some Basic Methods of Observation Follow its trail like a clever detective. Do forensic analysis of data you already have. Use direct observation. Start looking, counting, and/or sampling if possible. If it hasn’t left any trail so far, add a “tracer” to it so it starts leaving a trail. If you can’t follow a trail at all, create the conditions to observe it (an experiment).

As a ballpark estimate, I shoot for spending approximately 10% of the EVPI on a measurement and, depending on the circumstances, sometimes even as low as 2%.

The first 100 samples reduce uncertainty much more than the second 100. In fact, even the first 10 samples tell you a lot more than the next 10. The initial state of uncertainty tells you a lot about how to measure it.

All measurements have error. As with all problems, the solution starts with the recognition that we have the problem—which allows us to develop strategies to compensate, at least partially. Those who tend to be easily thwarted by measurement challenges, however, often assume that the existence of any error means that a measurement is impossible.

“Precision” refers to the reproducibility and conformity of measurements, while “accuracy” refers to how close a measurement is to its “true” value.

Quick Glossary of Error Systemic error/bias: An inherent tendency of a measurement process to favor a particular outcome; a consistent bias. Random error: An error that is not predictable for individual observations; not consistent or dependent on known variables (although such errors follow the rules of probability in large groups). Accuracy: A characteristic of a measurement having a low systemic error—that is, not consistently over- or underestimating a value. In some fields this is used synonymously with “validity.” Precision: A characteristic of a measurement having a low random error; highly consistent results even if they are far from the true value. In some fields of research, the terms “reliability” and “consistency” will be used in the same way.

The simplest solution is to keep observations a secret from those being observed.

It is the mark of an educated mind to rest satisfied with the degree of precision which the nature of the subject admits and not to seek exactness where only an approximation is possible. —Aristotle (384 b.c.–322 b.c.)

There are a number of reasons, however, where it is impractical to test, track, weigh, or even count every item in a population. But we can still reduce uncertainty by looking at just some items from a population. Anything short of a complete census of the population is a sample.

When you have a lot of uncertainty, a few samples greatly reduce it, especially with relatively homogeneous populations. In some cases, calibrated estimators were able to reduce uncertainty even with only one sample—which is impossible with the traditional statistics we just discussed. Calibrated estimators are mostly rational yet conservative. Doing more math reduces uncertainty even further.

The idea of “statistically significant” is often completely misremembered as some fixed minimum sample size or is invoked informally (i.e., without any calculation) as an objection to a measurement. Even if the math for statistical significance is remembered and done correctly, the results are often misinterpreted. Even if the math is right and interpreted correctly, the mathematically precise meaning of statistical significance is not really what a decision maker wanted to know in the first place.

whether a finding is statistically significant is not the same thing as whether your current state of uncertainty is less than it was before or what the economic value of that uncertainty reduction would be.

In other words, statistical significance is not about whether the measurement was informative or economically justified. Small samples may easily fit those requirements—the requirements the decision maker cares about.

We saw that reducing uncertainty about a big risky bet can be worth a lot even, in some cases, where uncertainty is reduced only slightly. This is what matters to the decision maker.

Basically, the recatch method is merely two independent sampling methods where we compare the overlap between the two samples to estimate the size of the population.

For example, if you wanted to see the share of time that employees spend in a given activity, you randomly sample people through the day to see what they were doing at that moment. If you find that in 12 instances out of 100 random samples, people were on a conference call, you can conclude they spend about 12% of the time on conference calls (90% CI is 8% to 18%).

Remember, usually you want to measure something because it supports some decision. And these decisions tend to have thresholds where one action is required if a value is above it, and another is required if the value is below it. But most statistical methods aren’t about asking the most relevant question: “When is X enough to warrant a different course of action?”

The word “experiment” could be broadly used to mean any phenomena deliberately created for the purpose of observation. You “experiment” when you run a security test to see if and how quickly a threat is responded to. But usually a key feature of the controlled experiment is that you account for possible errors.

If we change a feature on a product and want to determine how much this affects customer satisfaction, we might need an experiment. Customer satisfaction and, consequently, repeat business might change for lots of reasons. But if we want to see if this new feature is cost justified, we need to measure its impact apart from anything else. By comparing customers who have bought products with this new feature to customers who did not, we should be better able to isolate the effects of the new feature alone.

Correlation between two sets of data is expressed as a number between +1 and -1. A correlation of 1 means the two variables move in perfect harmony: As one increases, so does the other by a perfectly predictable amount. A correlation of -1 also indicates two closely related variables, but as one increases, the other decreases in lockstep. A correlation of 0 means they have nothing to do with each other.

Perhaps the biggest misconception some managers may run into is the belief that correlation proves causation. The fact that one variable is correlated to another does not necessarily mean that one variable causes the other. If church donations and liquor sales are correlated, it is not because of some collusion between clergy and the liquor industry. It is because both are affected by how well the economy is doing.

A number of other authors to this day debate that even when done properly, statistical significance is unnecessary4 and that its use has been a costly error in public policy.

When presented new information, we have no other option than to relate it to what we already know—there is no blank space in our minds within which new information can be stored so as not to “contaminate” it with existing information. —Clifford Konold, Scientific Reasoning Research Institute, University of Massachusetts

A Prior-Knowledge Paradox All conventional statistics assume (a) the observer had no prior information about the range of possible values for the subject of the observation, and (b) the observer does have prior knowledge that the distribution of the population is not one of the “inconvenient” ones. The first above assumption is almost never true in the real world and the second is not true more often than we might think.

Bayesian statistics deals with the issue of how we update prior knowledge with new information.

The objective of a Bayesian inversion is the same as applied math in general: to give you a path from things that seem easier to quantify to something you believe to be harder to quantify.

Instinctive Bayesian Approach Start with your calibrated estimate. Gather additional information (polling, reading other studies, etc.). Update your calibrated estimate subjectively, without doing any additional math.

Ignoring the prior probabilities when interpreting new information is a common error.

Anything you need to quantify can be measured in some way that is superior to not measuring it at all. —Gilb’s Law

I’m calling this method of updating prior knowledge based on dissimilar but somewhat related examples the “heterogeneous benchmark” method. When people feel they have no idea what a quantity might be, just knowing a context of scale, even for unlike items, can be a huge help. If you need to estimate the size of the market for your product in a new city, it helps to know what the size of the market is in other cities. It even helps just to know the relative sizes of the economies of the different cities. This is an example of the “you have more data than you think” rule from Chapter 3. We don’t need perfectly homogeneous examples to compare things to in order to reduce some uncertainty about a problem.

Getting a Sense of Scale Heterogeneous benchmark: A method where calibrated estimators are given other quantities as benchmarks to estimate an uncertain quantity, even when those quantities seem only remotely related. Example: Estimating the sales of a new product by knowing the sales of other products or similar products by competitors.

Again, with examples like brand damage, uncertainty is so high that almost any data informing a sense of scale is a reduction in uncertainty—therefore, a measurement.

Benchmarks are a practical way to bring some sense of scale to the problem whenever uncertainty is so high that it seems unmanageable.

Bayesian methods can be used for controlled experiments and regression models. They can be used in catch-recatch, spot sampling, clustered sampling, or any other sampling method you like.

Given a particular observation, it may seem more obvious to frame a measurement by asking the question, “What can I conclude from this observation?” or, in probabilistic terms, “What is the probability X is true, given my observation?” But Bayes showed us that we could, instead, start with the question, “What is the probability of this observation if X were true?” The second form of the question is useful because the answer is often more straightforward and it leads to the answer to the first question. It also forces us to think about the probability of different observations given a particular hypothesis and what that means for interpreting an observation.

“Absence of evidence is not evidence of absence”

The Likert scale. Respondents are asked to choose where they fall on a range of possible feelings about a thing, generally in the form of “strongly dislike,” “dislike,” “strongly like,” “strongly disagree,” and “strongly agree.” Multiple choice. Respondents are asked to pick from mutually exclusive sets, such as “Republican, Democrat, Independent, other.” Rank order. Respondents are asked to rank order several items. Example: “Rank the following eight activities from least preferred (1) to most preferred (8).” Open ended. Respondents are asked to simply write out a response in any way they like. Example: “Was there anything you were dissatisfied with about our customer service?”

Here are five simple strategies for avoiding response bias: Keep the question precise and short. Wordy questions are more likely to confuse. Avoid loaded terms. A “loaded term” is a word with a positive or negative connotation, which the survey designer may not even be aware of, that affects answers. Asking people if they support the “liberal” policies of a particular politician is an example of a question with a loaded term. (It’s also a good example of a highly imprecise question if it mentions no specific policies.) Avoid leading questions. A “leading question” is worded in such a way that it tells the respondent which particular answer is expected. Example: “Should the underpaid, overworked sanitation workers of Cleveland get pay raises?” Sometimes leading questions are not deliberate. Like loaded terms, the easiest safeguard against unintended leading questions is having a second or third person look the questions over. The use of intentional leading questions leads me to wonder why anyone is even taking the survey. If they know what answer they want, what “uncertainty reduction” are they expecting from a survey? Avoid compound questions. Example: “Do you prefer the seat, steering wheel, and controls of car A or car B?” The respondent doesn’t know which question to answer. Break the question into multiple questions. Reverse questions to avoid response set bias. A “response set bias” is the tendency of respondents to answer questions (i.e., scales) in a particular direction regardless of content. If you have a series of scales that ask for responses ranging from 1 to 5, make sure 5 is not always the “positive” response (or vice versa). You want to encourage respondents to read and respond to each question and not fall into a pattern of just checking every box in one column.

Two good indicators of revealed preferences are things people tend to value a lot: time and money. If you look at how they spend their time and how they spend their money, you can infer quite a lot about their real preferences.

It is not too bold a statement to say that a software development project is one of the riskiest investments a business makes. For example, the chance of a large software project being canceled increases with project duration. In the 1990s, those projects that exceeded two years of elapsed calendar time in development had a default rate that exceeded the worst rated junk bonds (something over 25%).

Is a programmer who gets 99% of assignments done on time and 95% error free better than one who gets only 92% done on time but with a 99% error-free rate? Is total product quality higher if the defect rate is 15% lower but customer returns are 10% higher? Is “strategic alignment” higher if the profit went up by 10% but the “total quality index” went down by 5%?

The value of management must show up in the financial performance of the firm.

Billy Bean, the manager of the Oakland A’s baseball team, decided to throw out traditional measures of performance for baseball players. The most important offensive measure of a player was simply the chance of not getting an out. Likewise, defensive measures were a sort of “out production.” Each of these contributed to the ultimate measure, which was the contribution a player made to the chance of the team winning a game relative to his salary. At a team level, this converts into a simple cost per win. By 2002, the Oakland A’s were spending only 3 million per win.

But what does performance mean if not a quantifiable contribution to the ultimate goals of the organization?

Predictive Analytics: The Power to Predict Who Will Click, Buy, Lie, or Die describes both existing and emerging analytical tools for fully exploiting massive data.

Anything currently estimated using expensive survey methods can be researched in different ways by any Internet-literate college student.

Or, instead of mining the web for information with screen-scapers and mashups, you could use the web to facilitate direct surveys of clients, employees, and others. Key Survey is one such web-based survey firm (www.keysurvey.com). These firms offer a variety of statistical analysis capabilities; some have an “intelligent” or adaptive survey approach where the survey dynamically asks different questions depending on how respondents answer earlier questions. Although these capabilities can be very valuable, many clients of web-based survey services find that the cost reduction alone is reason enough to use these methods of measurement.

Consider these statistics. It used to cost Farm Journal, a client of Key Survey, an average of 5 per respondent for a 40- to 50-question survey of farmers. Now, using Key Survey, it costs Farm Journal 25 cents per survey, and it is able to survey half a million people.

Consensus Point www.consensuspoint.com A service for businesses that want to set up prediction markets for internal use. Developed by some of the same people who created Foresight Exchange, the business has a lot of flexibility in how to set up and create reward systems for good forecasters, including monetary incentives. Foresight Exchange www.ideosphere.com A free website available to the public and one of the earliest experiments on the concept of prediction markets. All bets are “play money.” Claims are proposed by the public and reviewed by volunteers. It is an active market with a large number of players, and a good way to get introduced to prediction markets. NewsFutures www.newsfutures.com A direct competitor for Consensus Point, it offers businesses services to set up prediction markets.

in two real-life projects. Define the decision and the variables that matter to it (see Chapter 4). Model the current state of uncertainty about those variables (see Chapters 5 and 6). Compute the value of additional measurements (see Chapter 7). Measure the high-value uncertainties in a way that is economically justified (see Chapters 8 through 13). Make a risk/return decision after the economically justified amount of uncertainty is reduced. (See the risk/return decision described in Chapters 6 and 11.) Return to step 1 for the next decision.

No value question will ever be asked that doesn’t ultimately imply alternatives. If you have the right alternatives defined and the true decision defined, the value question will be much more obvious.

Backlinks