Category: Data

If you read much of the recent flurry of writing about Pay for Success, you will notice a regular pattern where authors acknowledge that widespread implementation will require “better data” and then quickly change the subject. Surely better data is on the way. We live in an age where it is easy to take this kind of inexorable progress for granted, but given the level of enthusiasm for Pay for Success, it is worth considering what it will realistically cost to get good enough data.

Certainly the whole potential of Pay for Success rests on data. In order to offer strong financial incentives for success, a government agency must be able to know that their private partner has succeeded. And measuring the “success” of a social program is notoriously hard. We all know it when we see it, but it is not simple to write out a clear and unchanging definition for any given program. A youth employment program cannot simply be judged by the number of youth who get jobs—we need to say something about the quality of those jobs, the level of challenge facing the youth who enter the program, the local economy’s strength, etc.

This is an example of what economists call information asymmetry. George Akerlof, who won the Nobel Prize for his work on information asymmetry, wrote a paper in 1970 about the market for used cars. Some used cars are in great shape and others are what Dr. Akerlof called “lemons:” they look fine but have been poorly maintained or have other hidden problems. Sellers know which kind of car they have, but buyers cannot immediately tell which is which. Sellers of above-average cars generally have to settle for average price, and buyers have to risk paying average price for a below-average car. A key point is that buyers can partially overcome this asymmetry by investing in information about a potential car; they can hire a mechanic to examine it. But there is also a limit—a simple inspection might weed out the worst cars, but the difference in value between an average and an above-average car may not be enough to justify a more complete inspection.

Information asymmetry has historically been one reason that we have created nonprofit organizations. Take childcare: A childcare provider knows whether they are providing quality care or not, but it is difficult for parents to tell the difference. It would be very easy for an unscrupulous operator to boost profits by cutting important corners. It is not that they have more incentive to cut corners than someone who makes toothpaste, but because the parents who pay are not the day-to-day users of the service, it is easier to hide the cost-cutting. Organizing a child care center as a not-for-profit organization does not overcome the information asymmetry, but it does accommodate it by reassuring parents that at least the center does not have any incentive to provide low-quality care.

Like parents, philanthropic donors are not present daily to see whether an organization is doing everything it can to make the most difference. Instead, they have to settle for knowing that the groups to which they give are trying to make a difference and do not have a profit motive to cut corners. The downside of this approach is that donors, like used car buyers, may sometimes have to accept average performance.

Rather than accommodating information asymmetry, Pay for Success tries to overcome it. This is like pushing water uphill—it can be done, but you have to invest energy to do it. The very idea of Pay for Success requires a significant investment in information. In place of a government agency directly funding a social service agency and accepting average performance, a social impact bond (SIB) requires several layers of intermediaries and generally two levels of professional evaluation: an evaluator who works directly with the program to measure impact and an independent assessor who reviews the data on behalf of the government agency.

McKinsey & Company developed a proforma to analyze the financial benefits of a hypothetical SIB focused on juvenile justice [1]. They found that even if an SIB-backed intervention produced significant savings for government agencies, the SIB structure was far more costly than directly funding the same services. In their model, a $14.4 million direct investment in preventive services would save the government $14.4 million in corrections costs over a period of about eight years. A successful SIB that funded the same $14.4 million program would incur an additional $5.7 million in research and administrative costs, success fees, and investor profits, and McKinsey & Company estimates that it would therefore take 12 rather than eight years before the public savings justified the increased cost.

This extra cost sets the bar pretty high for the performance gains that the SIB must deliver. Information technology improvements will continue to make investment to overcome information asymmetry practical in more and more situations, but when the cost of collecting data is taken into account, the social problems that lend themselves to an SIB will be harder to find than they would be if perfect information were free. Once we have found them all, there will still be many important social problems that are worthy of public investment.

If we want to confront some of our most complex social challenges, we have to come to terms with the reality that a significant level of information asymmetry is a fact of life and we cannot wish it away by calling for better data. For some social problems, sizable investment in information may make it practical to offer financial incentives to the best-performing programs. For the rest, we do not have to give up on using data to drive improved performance, but sometimes it might be more cost-effective to focus on raising the performance of the average program instead of providing financial incentives for above-average performance.