The economist is often likened to the astronomer: one who is powerless to influence the forces they study and resigned to formulate their ideas based on observation alone. An ideal approach for economists would be to hold everything in a market constant, change just one variable of interest and then observe the result. Doing so is the only true proof of causality, but economists are generally considered powerless to do this. The closest they can come is when a ‘natural experiment’ is formed by chance, but these are never precise enough to truly isolate the effect of one change.
However, astronomy is in one sense not as observational as it first seems; it is built on principles of particle physics, which have been well tested in the laboratory. Is there a similar role for experimentation as a foundation for economic theory? The rise of experimental economics says so.
A brief history
In the 1950s, peripheral economic journals became scattered with papers describing findings from controlled laboratory experiments that had more in common with psychology, but whose results were being used to appraise economic theory. As this type of research became increasingly prevalent, it produced more and more surprising results that often directly contradicted well-established theory, and were thus taken with more than a pinch of salt. However, the exponential rise of the field became a trend in the following decades. Nowadays, a large proportion of published economics papers report or refer to experimental findings, and experimentation is generally accepted as a valuable source of information in the discipline. In 2002, Vernon Smith and Daniel Kahneman won the Nobel Prize “for having established laboratory experiments as a tool in empirical economic analysis”, which is perhaps the clearest sign of the field’s coming of age.
Experimental economics’ primary medium is the laboratory experiment. These have become more sophisticated over time, and nowadays usually involve subjects using computers to complete tasks and interact with other subjects. The most unique feature of economic experiments is that subjects are typically paid according to the payoffs they earn within the experiment, so that they are incentivised to act as they would in a freely-occurring situation.
What has it taught us?
Much early experimental work focused on the idea of creating experiments that replicate real-world markets, with buyers, sellers and transactions. Vernon Smith was the pioneer of this use of experiments; using the principle of ‘induced values’ to gain control over subjects’ preferences, he found that the prices and quantities converged on the levels that a model of perfect competition would predict, despite almost all of its assumptions not holding. Because of the accuracy with which in-experiment markets can predict outcomes of those in the real world, experiments are often used as a ‘wind tunnel’ for new market designs. They have particularly been used in this sense to test various auction mechanisms, the results of which have often fed into the design of government auctions for telecommunications licences, emissions permits and more.
The most common use of experiments in economics is to test pre-existing theories, similarly to how Smith’s experiments tested models of competitive equilibrium. Not all results have been as complementary to the theory as his, however. Results from individual decision making experiments can reject many of the assumptions made by standard decision and preference models, insofar as they are accurate. For example, ‘preference reversal’ is a puzzling yet robust finding that, when given a choice between two gambles, subjects very often choose the safer gamble, but place a higher value on the riskier one when asked to value the same gambles separately. Depending on interpretation, this example challenges standard assumptions of the procedure-invariance of preferences, and of their transitivity and consistency. Similarly challenging are ‘dictator’ and ‘ultimatum’ games, where subjects are often found to freely give money away to other players even when there is no game-theoretic explanation of why they might do so. Results from these types of experiment contest the assumption that preferences are solely self-regarding, and hint at the importance of concepts like altruism.
Lab experiments are also a natural place to examine strategic interactions between multiple players. A particular focus, perhaps because of its clear policy implications, has been on public goods and testing the standard economic theory that nobody would voluntarily contribute to their provision, because of the ‘free-rider’ problem. Experiments have revealed, however, that concepts of altruism and reciprocity can sustain high levels of contribution. More broadly, the lab is often used to test how outcomes of strategic interactions compare to game theoretic predictions, with results suggesting that equilibrium concepts do not fully capture the outcomes of real-world interactions.
These kind of observations are of great relevance outside of the lab. For example, in analysing the effect of a policy change, welfare economics makes many of the assumptions that experimental results challenge. Knowledge of observed biases, furthered through experimentation, can and should be used to predict how individuals and groups will actually be affected by a policy change. Use of behaviour observed in experiments has also recently been used to argue for certain political approaches; in ‘Nudge’, Richard Thaler and Cass Sunstein advocate ‘libertarian paternalism’ – methods of influencing decisions without removing choice – which has its foundations in behavioural observations in the lab, and that has gained traction with David Cameron and Barack Obama.
Are experiments realistic?
As a new and fast-rising field, experimental economics has faced a great deal of scrutiny, and is no stranger to criticism. Perhaps the most frequent critique is that economic experiments are simply ‘unrealistic’. It is natural to wonder whether observation of students sitting at computers playing abstract games can really make predictions about what might happen in a complex, dynamic marketplace. However, experiments are by their very nature abstractions of reality, so to analyse this criticism it is important to look at specific realism concerns and whether they cause the lab to be truly unrealistic, and thus unrepresentative, or whether they are just evidence of necessary simplification.
For some, the relatively modest payoffs typically used cause the lack of realism, if they are not high enough to encourage subjects to think through their behaviour as thoroughly as they would otherwise. This is not the reason why outcome-dependent incentives are used in experimental economics, however. Instead, they are used to create the some constraints and forces that subjects would feel in real world situations, and to ensure all subjects put the same value on outcomes. In this sense, experiments are designed to create situations that are no less realistic than the natural scenarios they model, only simpler. Even if increasing the stakes was argued to make an experiment more realistic, extensive testing has shown that doing so, sometimes hugely, does not drastically alter behaviour.
Experiments could also be said to be unrealistic because they typically only involve students, whose decisions may not be representative of the wider population, limiting the predictive power of experiments. However, in theory-testing, this should not matter, because the predictions of economic models rarely depend on demographic factors. In others, such as decision-making experiments, this argument holds more weight. However, the only inherent reason why lab experiments rely on students is convenience; there is no reason why a wider subject pool could not be used, although this is often more easily achieved through field testing.
Field testing is perhaps the most obvious reaction to criticisms of experimental realism. This approach, championed primarily by John A. List, applies a lab-like methodology to naturally-occurring environments. Field experiments inherently sacrifice control to gain greater realism, but, as previously argued, there is little to suggest that lab experiments inherently lack realism. Considering that a lab experiment could offer incentives just as large as in the field, and use subject pools that are just as diverse, it is unclear what true benefit field testing brings. As James Heckman and Armin Falk argue in an article for Science, the real quest should not be for an abstract concept of ‘realism’ but for the best way to isolate the causal effect and most effectively control for the others.
The future
Experimental economics is becoming an integral part of research in a range of areas within economics, and rightly so. As it becomes more widely accepted, it is important to think of it as less of a niche sub-field and more of a tool that, when applied appropriately and carefully, can shed light on the gulf between economic theory and empirical reality in ways that were previously impossible. Whilst studying how individuals and groups should act has been historically integral to economics, it would be foolish to suggest that a study of how they actually act is not just as important.
There are, however, some unresolved criticisms of the field. Inherent to experimentation are questions of whether issues like self-selection bias and the Hawthorne effect jeopardise results. Neither is generally considered the major problem that they can be in other disciplines, and in some cases they can even be leveraged for the experimenter’s means, but despite this they are persistent issues that any experimenter must consider.
There is also much criticism that is less credible. Catherine Eckel and Herbert Gintis argue, in a paper tellingly entitled “Blaming the Messenger”, that some criticism is symptom of little more than a lack of willingness to accept the often stark implications that experimental findings have for traditional economic theory. That this is the case is unsurprising given the meteoric rise of the field within a discipline that is more familiar with steady development. To me, the view that experimental settings are unrealistic fits all too easily into this category of insincere criticism.
It is often forgotten that experimental results are not inherently contrarian; experiments tend to give theory its ‘best shot’ and sometimes produce results showing that a theory has greater predictive power than was expected. Regardless, it is in economists’ best interests to remain open-minded about developments in experimentation and to focus on a dialogue about genuine concerns with experimental methodology. In this way, problems can be addressed and methods refined so that experimentation can be used in as many areas of economics as it has the true potential to benefit. Experimental economics can become a significant part of the future of economics – as long as economists let it.
Written for Nottingham Economic Review
and crossposted at http://neronline.co.uk/?p=693.