Collective Awareness

Conversation : TECHNOLOGY

Collective Awareness

A Dialogue Between J. Doyne Farmer, Don Ross [10.3.18]


Don Ross

Doyne Farmer

THE REALITY CLUB [New]

Don Ross responds to Doyne Farmer: Despite this healthy state of knowledge about material investment, production, and consumption, we will have more economic crises in the future. In particular, we’ll have a next crisis, on a global scale. It will likely come, again, from financial markets. I’m not persuaded by Farmer’s suggestion that we might get a better handle on this source of risk by running inductions on masses of information about corporate resource allocations. These will be affected, massively, by global financial dynamics, but will likely have little systematic influence on them, even if it is some event in the old-fashioned economy that turns out to furnish a trigger for financial drama. [...]

DON ROSS is professor and head of the School of Sociology, Philosophy, Criminology, Government, and Politics at University College Cork in Ireland; professor of economics at the University of Cape Town, South Africa; and program director for Methodology at the Center for Economic Analysis of Risk at the J. Mack Robinson College of Business, Georgia State University, Atlanta. Don Ross's Edge Bio Page

COLLECTIVE AWARENESS

Economic failures cause us serious problems. We need to build simulations of the economy at a much more fine-grained level that take advantage of all the data that computer technologies and the Internet provide us with. We need new technologies of economic prediction that take advantage of the tools we have in the 21st century.  

J. DOYNE FARMER is director of the Complexity Economics Programme at the Institute for New Economic Thinking at the Oxford Martin School, professor in the Mathematical Institute at the University of Oxford, and an external professor at the Santa Fe Institute. He was a co-founder of Prediction Company, a quantitative automated trading firm that was sold to the United Bank of Switzerland in 2006. J. Doyne Farmer's Edge Bio Page

~ ~ ~ ~

I'm thinking about collective awareness, which I think of as the models we use to collectively process information about the world, to understand the world and ourselves. It's worth distinguishing our collective awareness at three levels. The first level is our models of the environment, the second level is our models of how we affect the environment, and the third level is our models of how we think about our collective effect on ourselves.

Understanding the environment is something we've been doing better and better for many centuries now. Celestial mechanics allows us to understand the solar system. It means that if we spot an asteroid, we can calculate its trajectory and determine whether it's going to hit the Earth, and if it is, send a rocket to it and deflect it.

Another example of collective awareness at level one is weather prediction. It's an amazing success story. Since 1980, weather prediction has steadily improved, so that every ten years the accuracy of weather prediction gets better by a day, meaning that if this continues, ten years from now the accuracy for a two-day weather forecast will be the same as that of a one-day weather forecast now. This means that the accuracy of weather prediction has gotten dramatically better. We spend $5 billon a year to make weather predictions and we get $30 billion a year back in terms of economic benefit.

The best example of collective consciousness at level two is climate change. Climate change is in the news, it's controversial, etc., but most scientists believe that the models of climate change are telling us something that we need to pay serious attention to. The mere fact that we're even thinking about it is remarkable, because climate change is something whose real effects are going to be felt fifty to 100 years from now. We're making a strong prediction about what we're doing to the Earth and what's going to happen. It's not surprising that there's some controversy about exactly what the outcome is, but we intelligent people know it's really serious. We are going to be increasingly redirecting our efforts to deal with it through time.

The hardest problem is collective awareness at level three—understanding our own effects on ourselves. This is because we're complicated animals. The social sciences try to solve this problem, but they have not been successful in the dramatic way that the physical and natural sciences have. This doesn’t mean the job is impossible, however.

Climate prediction had the big advantage that it could piggyback on weather prediction. As weather predictions got more accurate, climate models automatically got more accurate, too. There is a way in which climate prediction is actually easier than weather prediction. You don't try to say what's going to happen three days in the future, you try to say what's going to happen, on average, if things change. If we pump 100 parts per million more CO2 into the atmosphere, how much is that going to warm things up? A climate model is just a simulation of the weather for a long time, but under conditions that are different from those now. You inject some greenhouse gases into the atmosphere, you simulate the world, and you measure the average temperature and the variability of the weather in your simulation.

Climate predictions get a huge benefit from all the effort that's gone into weather prediction. I've been trying to get a good number on how much we've invested in weather prediction, but it is certainly $100 billion dollars or more. Probably more. It's probably closer to $1 trillion that we've invested since 1950, when we did the first numerical weather predictions. It sounds like a lot of money, but the benefits are enormous.

I've been thinking about how we can make better economic models, because a lot of the problems we're having in the world right now are at least in part caused by economics and the interaction of economics with mass sociology. Our cultural institutions are lagging technological change, and having a difficult time keeping pace with them. The economy plays a central role. Since the '70s, the median wage in the US has been close to flat. At the same time, the rich have been getting richer at a rate of two or three percent per year. A lot of the factors that are driving the problems we're having involve the interaction of the economy with everything else. We need to pursue some radically different approaches to making economic models.

It's interesting to reflect on the way we do economic modeling now. How do those models work? What are the basic ideas they're built on? We got an unfortunate taste of the ways in which they don't work in 2006, when some prescient economists at the New York Fed asked FRB/US, the leading econometric model, "What happens if housing prices drop by twenty percent?" This was 2006—their intuition was right on target—over the next two years, housing prices dropped by almost thirty percent. FRB/US said there'd be a little bit of discomfort in the economy, but not much. The answer FRB/US gave them was off by a factor of twenty. It made such bad forecasts because the model didn’t have the key elements that caused the crisis to happen.

Since then, economists have focused a lot of effort on adding these key elements, for example, by coupling financial markets to the macroeconomy. FRB/US didn’t model the banking system, and couldn’t even think about the possibility that banks might default. Issues like that are now in those models. The models have gotten better. But there is still a good chance that when we have the next crisis, we'll get similarly bad answers. The question is, how can we do better?

The first thing one has to say is that it's a hard problem. Economics is a lot harder than physics because people can think. If you make a prediction about the future of the economy, people may respond to your prediction, automatically invalidating it by behaving in a way that creates a different future. Making predictions about economics is a lot harder than using physics to predict the behavior of the natural world.

Fortunately, the most interesting things we want to do aren't to predict what GDP is going to do next month, but to make predictions about what happens if we tinker with the system. If we change the rules so that, say, people can't use as much leverage, or if we put interest rates at level X instead of level Y, what happens to the world? These are conditional forecasts, in contrast to predicting tomorrow's weather, which is an unconditional forecast. It's more like climate prediction. It’s an easier problem in some ways and harder in others because it is necessary to simulate a hypothetical world and take into account how people will behave in that hypothetical world. If you have a system like the economy that depends on thinking people, you have to have a good model for how they think and how they're going to respond to the changes you're making.

~ ~ ~ ~

When I was a graduate student, Norman Packard and I decided to take on the problem of beating roulette. We ended up building what turned out to be the first wearable digital computer. We were the first people to take a computer into a casino and successfully predict the outcome of roulette and make a profit. We were preceded by Claude Shannon and Ed Thorpe who built an analog computer that allowed them to predict roulette in Shannon’s basement, but they never successfully took it into the casino. My roulette experience changed the rest of my life because it set me on a career path in which I became an expert on prediction. This never would have occurred to me before that.

If a system is chaotic it means that prediction is harder than it is for a system that isn’t chaotic. But nonetheless, perhaps by better understanding chaotic systems and acknowledging that the chaos is there, we can predict them better. When I was studying chaos, I thought that perhaps there was a way to take advantage of chaos. Drawing on my roulette experience, John Sidorowich and I came up with an algorithm for building a nonlinear model of a time series, for making better short-term predictions of low‑dimensional chaotic behavior. This is just like weather: Chaos still overwhelms the predictions in the long term, but prediction in the short term is possible. In some cases we could beat standard models pretty well using our method.

We applied this to turbulent fluid flows, ice ages, and sunspots. Some clown in the audience would always say, "Have you tried applying this to the stock market?" I got tired of hearing this question. I was approaching ten years at Los Alamos, where they give you a little Nambé-ware nut dish to commemorate ten years of service. That freaked me out. I figured that if I kept hanging around, I'd be there at twenty years and thirty years. So I left just before they gave me the nut dish.

Norman and I started a company called Prediction Company, which predicted the stock market. After many years of hard work, we built a system that made reliable predictions of certain aspects of the stock market. We were betting not on the big movements, but on the little ripples. We could predict the idiosyncratic movements of stocks about a month in advance. The predictions were far from perfect, but we made lots of independent bets and it made a steady stream of profits. It's been hugely elaborated on since then, but it's still being traded and still making money. But making money isn't my goal in life, so after eight years I quit and went back to doing basic research. I went to the Santa Fe Institute, where I decided to put my complex systems background together with my domain knowledge about financial markets and try to create better theories for what makes the financial system and the economy tick. That's what I've been doing for the last fifteen or twenty years.

My models make alternative assumptions to those that are made in standard economic models. The biggest assumption is equilibrium. A standard economic model assumes that people have a utility function, which measures what they want. Each person—each agent—maximizes their utility. Each agent also has a way of forming expectations about the world, which they use to maximize their utility. The most powerful model of the world is rational expectations. A rational agent has a good model of the world and understands everybody else's model of the world as well. A rational agent can think about all these, know what other people are going to do, and optimize utility accordingly.

Equilibrium means that outcomes will match expectations. This is true in a statistical sense. That doesn't mean we're right every time, but it means that we are right on average. If people are rational, if they believe that GDP is going to go up by two percent on average, then GDP goes up by two percent on average. Of course this is just on average—it might go down in a given year.

Starting in the '80s, with more and more effort over the last twenty-five or thirty years, economists have shown that real people aren’t rational, and they have been trying to tinker with their models to take this into account. There were always some economists who said people aren't rational, but the mainstream view has been that maybe people aren't rational, but let's see how far we can get with rational models, and then treat the deviations from rationality as needed. Rational models are well defined, they're clear, and they can be solved. Otherwise, it is too easy to get lost, because as soon as you don't let people be rational, you have to figure out how they do think, and the way real people think is complicated. It’s easy to get lost in a wilderness of bounded rationality, as it's called. There're too many different ways for people to be non-rational.

Daniel Kahneman is one of the behavioral economists who has studied ways in which people are not rational. In my opinion, he didn't go nearly far enough. Kahneman says that people don't use utility, and instead use an alternative called prospect theory. But prospect theory is pretty close to utility, and it's still a poor model of what motivates people.

Utility is about goals. It means maximizing something, like the logarithm of wealth, or consumption through time, appropriately discounted to include not just your consumption today, but your consumption tomorrow, which isn't quite as important as your consumption today. Prospect theory makes utility a little bit more complicated because it treats your losses differently than your gains. Both of them are reasonable starting points; I'm not convinced by either one. Utility gives a useful way to put somebody's goals into a model and take into account the fact that we have goals. It's a very reasonable starting point, but we need to go beyond it.

The way economists have been going beyond it is to add what are called "frictions" to their models. Frictions are essentially constraints on perfect rationality. For example, in an idealized model, wages would always adjust so that supply equals demand, or in economics jargon, so that the labor market clears. But the real world doesn’t work that way. If you're running a company, it's not possible to constantly adjust the wages of your employees. It’s particularly hard to lower their wages. You can’t just go in and say, "The labor market has gotten tighter, so I'm going to lower your wage." So, to make things more realistic, economists have added a constraint that says wages are sticky. This is called a friction. In the revised model the rational agent knows this, and takes it into account in making decisions.

Macro models have developed over the years by adding more and more "frictions." This involves adding constraints to idealized models in which you typically have representative agents or maybe a distribution of agents, and in which each agent who might represent a household reasons about their consumption over their lifetime, makes a bunch of planning decisions, and then updates those as new information is received about what's going on in the economy. Most of us don't function that way.

We need to seriously re-examine the whole program. Economic failures cause us serious problems. We need to build simulations of the economy at a much more fine-grained level that take advantage of all the data that computer technologies and the Internet provide us with. We need new technologies of economic prediction that take advantage of the tools we have in the 21st century.  

Places like the US Federal Reserve Bank make predictions using a system that has been developed over the last eighty years or so. This line of effort goes back to the middle of the 20th century, when people realized that we needed to keep track of the economy. They began to gather data and set up a procedure for having firms fill out surveys, for having the census take data, for collecting a lot of data on economic activity and processing that data. This system is called “national accounting,” and it produces numbers like GDP, unemployment, and so on. The numbers arrive at a very slow timescale. Some of the numbers come out once a quarter, some of the numbers come out once a year. The numbers are typically lagged because it takes a lot of time to process the data, and the numbers are often revised as much as a year or two later. That system has been built to work in tandem with the models that have been built, which also process very aggregated, high-level summaries of what the economy is doing. The data is old fashioned and the models are old fashioned.

It's a 20th-century technology that's been refined in the 21st century. It's very useful, and it represents a high level of achievement, but it is now outdated. The Internet and computers have changed things. With the Internet, we can gather rich, detailed data about what the economy is doing at the level of individuals. We don't have to rely on surveys; we can just grab the data. Furthermore, with modern computer technology we could simulate what 300 million agents are doing, simulate the economy at the level of the individuals. We can simulate what every company is doing and what every bank is doing in the United States. The model we could build could be much, much better than what we have now. This is an achievable goal.

But we're not doing that, nothing close to that. We could achieve what I just said with a technological system that’s simpler than Google search. But we’re not doing that. We need to do it. We need to start creating a new technology for economic prediction that runs side-by-side with the old one, that makes its predictions in a very different way. This could give us a lot more guidance about where we're going and help keep the economic shit from hitting the fan as often as it does.

~ ~ ~ ~  

I'm at the Institute for New Economic Thinking at the Oxford Martin School where I’m the director of Complexity Economics. I'm also a professor in the Mathematics Department at the University of Oxford. I have a group with about ten graduate students and five postdocs. We're doing research to build models of the economy. We're building models at several different levels.

One effort is to build models of the financial system. Our models use snapshots of the portfolios of banks and other large financial institutions at different points in time to understand systemic risk. We model what the banks are doing, how they affect each other, and how shocks propagate around the financial system. We look at things like how stable the financial system is.

Another project is on simulating housing markets. We have worked with the Bank of England to analyze policies for regulating housing markets. For example, about a year or a year and a half ago, the Bank of England instituted a policy that banks had to make eighty-five percent of their loans to people whose loan-to-income ratio is below 3.5. They did this to ensure stability in housing prices, to damp a possible bubble. We did a simulation of housing markets and saw that this policy worked pretty well.

We are now working on understanding regional differences in housing prices. Can we understand quantitatively why prices in London are so much higher than in other parts of the UK? Can we make a map of how housing prices change around the UK and what causes this so that the policymakers can think about the consequences of, for example, putting in a new rail line? Housing prices play an important role in the economy. Should we be encouraging new housing developments, and if so, where, and at which part of the housing spectrum? We can simulate housing markets to help regulators answer these kinds of questions.

We've also been thinking about the insurance business. There is a new directive called Solvency II, which is a regulation stating how much capital insurance companies need to hold in reserve and what kinds of models they're allowed to use to estimate their risk. We have concerns that the restrictions on the models they're allowed to use may be dangerous because they force all the insurance companies to do more or less the same thing, to act as a herd, making the system fragile. Currently seventy-five percent of catastrophe insurance is based on the same model. A better understanding of the benefits of model diversity could allow us to improve on Solvency II, and might prevent a collapse of the insurance business.

We've also been looking at technological change. We gather data on the cost and performance of technologies through time to better understand technological change. The answers are surprising. Out of the 200 or so technologies that we've looked at, about fifty of them obey a version of Moore's law. The cost of these technologies drops exponentially in time, at different rates for different technologies. As we all know, computer prices drop really fast. The cost of transistors drops at forty percent per year, as does that of many other computer components. The price of gene sequencing has dropped even faster. Other technologies, like solar photovoltaics, have been dropping at  a rate of ten percent per year. Since the first photocell went into the Vanguard satellite in 1956, the cost of a solar cells has dropped by more than a factor of 3000.

In contrast, if you look at the price of coal, the price fluctuates, but over the long term, once you adjust for inflation, it's been roughly constant for 150 years. The price of oil has been roughly constant for more than 100 years. It fluctuates up and down—we know it's been from more than $100 a barrel down to $20 or $30 a barrel—but there's no overall trend like there is for technologies like solar cells that have improved over time.

We've been trying to understand why some technologies improve so much faster than others. We've also been trying to better understand the patterns in technological improvement and how we can use them. In particular, Francois Lafond and I have developed a method for forecasting future technology prices. It's very simple: We just model Moore's law as a random walk with drift. Our model is probabilistic. We don't make misleading statements like, "Solar energy will be a factor of four cheaper by 2030." We make probabilistic statements like, "Most likely, cost for solar energy will drop by a factor of four by 2030, but there's a five percent chance it won't drop in cost at all." We've used our collection of historical records of technological change to test the accuracy of our model, and it does very well. We make forecasts, test them, and compare the accuracy to that predicted by the model. We don't just make a forecast, we say how good that forecast is, and we say what the range of possibilities is and what their probabilities are.

We've been using this to think about investment in climate change. How much do we have to invest and which technologies should we be investing in to get to zero carbon energy emissions as soon as possible? The answers we're finding look good. Unless we are unlucky, our results indicate that we're going to get there quickly. Solar energy will probably contribute at least twenty or thirty percent of our energy within about ten years. Again, there are some big error bars around these numbers, but this is the most likely outcome.

To evaluate the real cost of something, it is important to look not just at the value now, but also the value in future. This is done by taking an average over present and expected future costs and benefits to compute what is called net present value. What is the real cost of making the transition to climate change? Work by Rupert Way and I suggest that because energy is going to get cheaper than it is now due to solar energy and wind, the net present value is actually negative, meaning we're going to benefit. Climate change is not a net cost, it's a net gain. We win.

I get pursued by hedge funds that want me to consult for them. I've done some consulting, for example, for a hedge fund that was thinking about technology investments, and we advised them about likely rate of progress for a technology like robotics versus a technology like solar cells. But making money is not my primary interest now.

I left Prediction Company after being there for eight years. We were finally doing well, making steady profits, and I was at a fork in the road. My intention had been to stay for five years. I'd already made more money than I had intended to make. I thought, how do I want to feel on my deathbed? Do I want to say I made a lot of money, or do I want to say I got to pursue my love of science and I maybe did something good for the world? I chose the latter course.

I've been spending the last ten years using the expertise I gained from predicting roulette and predicting the stock market and trying to apply it to do something useful for humanity. Unfortunately, I have to say, it's much easier to get funded to beat the stock market than it is to help the world.

Because of that, I am thinking about starting another company. I'm being driven to do that for two reasons. One is that it may be easier to fund the things I want to do via private enterprise. The models I want to build require a lot of resources. I want to build a system that can make better predictions about the economy. I want to build something that all the central banks will want to use because it will give them a superior view into the future. To use a caricature of brain anatomy, at Prediction Company we built a market cerebellum. We looked at past data and searched for consistent patterns and made bets on them. We had no idea why those patterns happened. But what I want to do now is build a cerebrum. I want to build a system that decomposes the market into its parts, that models the causal mechanisms, and that allows us to think about those important "what if" questions: What if we change the rules? Can we find better outcomes? We've been chipping away at these problems in my group at Oxford.

I love being an academic. It's wonderful because I get to work with the smartest people in the world. But it's hard to do a focused effort. To do what I want to do now, I feel like I need a team. I need to do what we did at Prediction Company. We hired thirty people, put them all in the same building, and we said, "This is the problem. We're going to work on this problem full time until we crack it."

I've decided to start another company. This time, the company is going to be much more public-focused. One of the problems with Prediction Company is that we had to keep what we were doing a secret. We had to do that out of respect for our investors. My new company is going to be much more open. Of course we will try to make money, that’s what investors need to be happy, but we will also work to create an open source toolkit that everybody can use to make better models of the economy. We'll make some key add-ons to make a profit, but I want to have the open source toolkit in the center so that we can lift the whole technology of economic prediction up from where it is now.

~ ~ ~ ~  

I came to Oxford for several reasons. I got offered a job there, it seemed like a good opportunity, and I got to do what I wanted to do. I like working with smart people and I wanted to be in Europe. I aspire to be a member of the European Union, a European Union citizen. As it works out, I will probably get my British passport about two or three months before Britain leaves the European Union, which is a disappointment. Fotini Markopoulou, my wife, is Greek, and she wanted to be back in Europe. It was an adventure to come here. The complexity economics community that I'm a part of is much more centered in Europe than it is in America. The American economics establishment is very conformist; the European establishment is a bit more open.

The big picture in the present world has me very worried. Populism seems to be taking over. This is at least in part caused by the wrong kind of economics. We need new ideas about economics because we need to change the way we run our economies. Eric Beinhocker, Fotini Markopoulou, Steen Rasmussen, are running a workshop at the Santa Fe Institute to think about how we can create a new set of economic principles and connect them to politics. We want to do what Friedman and Hayek and others did when they developed neoliberalism. They developed a new set of economic ideas and convinced politicians like Thatcher and Reagan to follow them.

We don’t agree with neoliberalism—we think we need new ideas. We need to provide a better intellectual basis. We need a new kind of economics, one that fuses ideas from political science and sociology and elsewhere to give a more coherent scheme for how we go forward in the world that can dramatically accelerate our collective awareness at level III.

Reality Club Discussion

don_ross's picture
Don Ross
Head, School of Sociology, Philosophy, Criminology, Government, and Politics, University College Cork, Ireland; Professor of Economics, University of Cape Town, South Africa; Program Director, Methodology, Georgia State University, Atlanta

Economic Theory Versus Big Data

In "Collective Awareness" J. Doyne Farmer relates highlights from his inspirational career as a data gatherer, simulator, and forecaster. There are several nice take-aways from his account. It’s informative and fun to have quantitative estimates put on the value of improvement in weather forecasting over the past half-century. And I’m happy for readers to hear the truth that innovative thinking about economics finds more encouragement here in supposedly sclerotic Europe than it currently does in America.

Every scientist should love new data and love new ways of gathering new kinds of data even more. The so-called “big” data generated by and from the Internet that Farmer celebrates is of unquestioned value to scientists and policy makers, and in future will yield streams of benefits no one has yet thought of.

I want to raise a caution flag, however, about three questionable ideas that Farmer’s discussion promotes. One is about the nature of inference: that large quantities of directly observed data are generally more informative than smaller sets of deliberately cultivated data. The second is philosophical: that the best way to understand a complicated system is always, or even usually, to focus on details about its individual components. The third, and the one that exercises my greatest concern here, is that some of the worst current features of our political and social life can partly be blamed on an old-fashioned way of doing economics that should be replaced. I think that displacement of economic theory by simulations based on crunching big data would invite disaster for human welfare.

Big data, as the phrase is typically used, doesn’t just refer to having relatively many observations. It usually also denotes data that are generated naturally, without the deliberate manipulation of a motivated searcher and harvester. It is no accident that people started enthusing about big data after the Internet came along. As Farmer says, online activity by billions of users spontaneously creates a record of fantastically many demographic and behavioral correlations. Isn’t it obviously a gain if we can use all of these as a basis for trying to infer general social and economic facts and causal relationships, rather than just those we hope will turn out to be important on the basis of prior theory? After all, as everyone knows, theories are never perfect and often they prove to be outright wrong. Might big data not be the ticket to seeing around and through our blind spots?

Correlations, in themselves, aren’t always informative, and they tend to get less informative rather than more informative when you have lots of them populating a system that’s dense with causal relationships (like an economy). Let’s consider a much-debated example.

In top American universities, being female is correlated with getting higher grades. The same thing is true in American high schools. Yet men, just one year after graduation from top universities, so before differential effects of becoming fathers rather than mothers can be having much influence, statistically earn higher salaries than women. An obvious causal generalization about effects of sexism seems to be directly implied by these correlations.

We should be a bit cautious, though, because we know that the part of the population that goes to top universities is laboriously selected from the population that goes to high school. Also, in high school students have only very limited degrees of choice about which subjects to study, whereas in university, at least past the first couple of years, they can specialize in anything from musical composition to biochemistry. Gender isn’t a randomly distributed variable where choice of university subjects is concerned. Finally, in some subjects, like physics, it’s harder to earn top grades than in than others. So, we have two selection effects operating: selection into top universities from the group that completes high school, and selection into university subjects that aren’t randomly distributed with respect to grades.

From the simple correlation between being female and getting higher grades one might naturally infer that, on average, young women (or perhaps just women, period?) have higher cognitive ability than young men (men?). But we’ve pointed out that the overall correlation comes from two different populations, one in high school and one in universities. Might girls do better than boys in high school because the boys in that population are more likely to be competing with one another to resist authority and socialization? Might some of that difference be washed out, or even reversed, by the selection into top universities? (It could be reversed if the boys who make it to university are selected to be more self-disciplined than the girls, because they are the boys who had to overcome peer effects of a kind the girls were less likely to encounter.) Then what if young men, whose egos have been better nourished by a sexist culture, are more confident in their abilities than young women, and consequently are more likely to choose more difficult subjects in university? If these hypotheses, each related to one of the two selection effects, were true, then the inference that young women have higher cognitive ability than young men might be completely invalid; or it might be true, but of only minor causal significance. Note that on the second hypothesis, sexism might still be an important part of the problem but producing its effects in a different way from what the simple inference suggests.

This example is not made up. The correlations just described have been carefully studied by economists who are interested in trying to determine how much of the male salary advantage in the US professional workplace is accounted for by intentional or systemic sexism. This is obviously relevant, and importantly so, to a range of policy choices. There is as yet no consensus on the central relationship of interest. But a few things are clear. We will not be able to estimate causal effects of sexism on the gender wage gap just by looking at post-educational or at pre-educational correlations, especially in the whole population. And the path to progress does not consist in piling on ever more observed correlations. Their multiplicity is the source of the inference problem, not the way to its solution.

Why have economists been the main investigators of this problem? It isn’t just because, or even mainly because, one of the crucial indicator variables, “salary," is “economic” in the everyday sense of the word. The explanation is rather that because almost all of the systems that interest economists involve multifarious selection effects, economists have developed special expertise and statistical technologies—“econometrics”—for quantitatively estimating these kinds of effects. Econometrics is theory. It is not theory in the sense of grand hypotheses about how economies work or how societies are structured. It is theory about general relationships between properties of correlational data in social structures (properties like variance, skew, and standard error structure) and the kinds of causal processes that generate such data.

Here is the main takeaway point from all of this. The analyst can most accurately apply econometric theory about sample selection effects to the extent that she knows a lot about the processes that generated her data. She might try to gain this knowledge in various ways, which are not mutually exclusive. She might use other theory, in which she has empirical reason to be confident, about the causes of some variable values, and use that knowledge as “bootstrapping” leverage to statistically restrict hypotheses about the causes of some other relevant variable values. (This is why econometric knowledge is more specific than general knowledge of statistics.) Or she might run an experiment in which she controls a data generating process that she has good empirical reason to believe is a reliable model for the data generating process she’s studying. Or she might, using a method Farmer emphasizes, try to simulate the data generating process in a computer model.

What is not compatible with any of these methods or will at the very least make their application more difficult, is having to take account of data that were generated by very complex medleys of different processes—like billions of people using the Internet in multifarious and complicated ways—about which there is no known structural model with known parameters restricting effect sizes. The more data of this kind there are—so, the “bigger” the data, in the current meaning of the phrase—the greater the barrier to understanding the causal relationships they support.

In the example of correlations between gender, educational performance, and wages, I noted that one of the variables of interest is “cognitive ability." This illustrates another problem: it’s a “variable” that isn’t itself defined by a generally accepted theory of measurement. Is cognitive ability a single dimension, or is it composed out of elements that are themselves structurally related to one another? Is it expressed in all of a person’s behavior, or only in behavior that exceeds some threshold of inferential or memory-management difficulty? If we’re not quite clear on what “cognitive ability” is, how can we know what it might be correlated with, especially when we compare it across different populations and different data-generating processes? Cognitive ability is a structured variable, and structured variables are trouble for scientific inference unless we have accurate models of their structures that we can use to design measurement strategies.

Scientists faced with structured variables often try to deal with them by analyzing them into their contributing elements. This method worked particularly well in the first domains in which science was historically successful, astronomy and physical mechanics. In these systems, effects of component sub-systems can often be separately measured or estimated and “added up” to generate overall values for the structured variables that remain stable across contexts.

Farmer suggests that use of big data will help us to do better macroeconomics because it will allow us access to “much more fine-grained data” that will let us “simulate what every company is doing and what every bank is doing in the United States." He thus suggests that we will gain better estimations of the structured variables that matter for macroeconomic policy—rates and changes in productivity, price variability, total output, labor-force participation, etc.—by analyzing them into their directly observed components. He says that this will be an improvement on the current reliance on designed surveys.

I think that Farmer is quite wrong about this.

Part of the problem with his inference is that the big data on which he looks forward to relying will be generated by a vast range of largely invisible processes with frequently incomparable dimensions, and different degrees of dimensionality. Surveys, by contrast, are designed data-harvesting instruments. The designers are “survey econometricians," experts in the art of fishing for types of values that allow the processes that generated them to be inferred, and to contribute in pre-understood ways to the structural equations and sample selection controls that will be used to model them. Surveys are deliberately designed to tame the effects of multiply entangled correlations. Big data, by contrast, just are a salad of such correlations—that’s what’s “big” about them.

There is an additional source of difficulty with Farmer’s suggestion, a difficulty related to the general limitations of achieving understanding of systems by analysis. The analytical method that worked so well in early astronomy and mechanics began to yield problems as soon as scientists started applying it to biological systems. The variables of interest in such systems are typically not compositional but functional. Cognitive ability, as featured in our example, is defined partly by reference to effects that are distinguished in turn by reference to normative values. A person is relatively cognitively able if she can think well, but that varies not only with raw thinking equipment but with task context. It seems likely that differences in people’s neural processing architecture and speed statistically contribute to differences in cognitive abilities. But there is plenty of evidence, much of it furnished by economists, suggesting that environmental factors in childhood, including purely social influences, play at least as large a role in generating the streams of data over a person’s life that are accounted for in attributing relative cognitive ability to her. This is a typical feature of biological (including psychological and social) structured variables.

In the nineteenth century, scientists were so flummoxed by their inability to analytically decompose structured biological variables that many posited a special non-physical “life force” as a transmitter of various behavioral and developmental differences among species and organisms. Of course, biologists, medical researchers, and psychologists gradually learned how to think ecologically instead of (only) analytically. But this learning evidently comes hard. In the second half of the twentieth century, it took cognitive scientists a few decades to generally recognize that many aspects of intelligence, learning, emotional response, and even memory are processed by parts of external environments with which organisms are dynamically coupled, and effective problem solving doesn’t have to—indeed, could not possibly be—done entirely “inside” their brains and nervous systems.

Farmer is a leading complex systems analyst, so he would be the first to emphasize that the kind of decompositional analysis familiar from mechanical domains has only very limited reach across the range of scientific research areas. Yet his remarks echo a widespread view among macroeconomists that we can understand structured macroeconomic variables well only by drilling into the properties and dispositions of the individual people whose activities “add up to” the economy. Since the 1970s, mainstream opinion has held that macroeconomic models must have “microfoundations."

A non-economist might think that macroeconomics should have microfoundations just because economies are made up of individual people, households, and firms. That is, attention to microfoundations might just reflect a general commitment to explaining and modeling systems by analyzing them. This isn’t correct, however, and Farmer’s discussion shows that he knows why it isn’t. Economists worry about implementing policies designed to change macroeconomic relationships—say, between prevailing interest rates and aggregate savings—that would require that people remain unaware of the policy maker’s efforts, and incapable of adjusting their behavior to these efforts. For example, if the monetary policy authority (in the US, the Federal Reserve Bank, or “the Fed") lowers interest rates because it wants Americans to spend more, Americans might all realize that this is what it’s trying to do, and that this will imply higher prices in the near future. They might then respond not by reducing their savings but by politically coordinating to demand higher wages. The Fed’s action might then yield only a small change in the consumption to savings ratio and a large, problematic, increase in wage and price inflation. This is the sort of problem reflected in the title of Farmer’s article, “Collective Awareness."

The collective awareness challenge is real (sometimes—it has historically been very easy to induce Americans to reduce their savings, much harder to get them to increase savings). It is reflected in commitment by economists that what Xavier Ragot has called “sensible” microfoundations: economists shouldn’t recommend policies that, to work, require many people or companies to do things that aren’t in their interests, or to ignore information they can get without much effort or expense.

However, a major strand of macroeconomic theory, introduced by Paul Krugman to a wide readership of non-economists under the label “freshwater macroeconomics” (because it has been developed and promoted at universities located around the Great Lakes), defends a more radical interpretation of the collective awareness problem. According to this theory, monetary policy cannot have any significant effects on the macroeconomy at all. In my opinion, as in Krugman’s, freshwater macroeconomics has been clearly refuted by empirical facts. Statistically minded readers can find the basis of this refutation summarized, in an engaging and accessible way, in Paul Romer’s readily accessible 2016 article “The Trouble With Macroeconomics." Romer explains the continuing survival of this theoretical movement in terms of social psychology and deference to revered academic leaders. I have argued in some of my work, that we can identify its core economic mistake as ignoring the fact that what matters to macroeconomic effects is not just what will happen sooner or later, but what will happen sooner versus what will happen later.

Farmer might say at this point that it is an advantage of computer simulations of the economy over static equilibrium models that in simulations events do happen in definite orders, and with definite lags; sooner and later are clearly distinguished. However, it is generally much harder than Farmer suggests to build computer simulations in which these temporal relationships mirror the ones you should expect in the world unless the simulations are based on accurate structural theories.

Economic theorists do rely on simulations. My issue of contention with Farmer concerns the nature of the theory on which the most useful simulations are based. A good deal of confusion has resulted from the fact that the modeling technology inspired by freshwater macroeconomic theory, known as “DSGE” (for Dynamic Stochastic General Equilibrium), is the core of the technology used by practical policy makers at the Fed, in other countries' central banks, and in private financial institutions. If freshwater macroeconomic theory is empirically misguided, must the modeling and simulation approach that it historically inspired not be equally suspect?

The answer is “no”—and this shouldn’t be surprising on reflection. If the economists at the Fed didn’t believe that monetary policy can affect the real economy, they would have no reason to bother doing their jobs. I assure the reader that neither the Fed nor any other politically independent national central bank is full of cynics who secretly regard their mission as pointless. The DSGE models they use are evolved from the so-called “Real Business Cycle” foundations that Krugman and Romer rightly attack, but this evolution has occurred under the intense pressure of and feedback from real-time policy experience. The Fed’s working models don’t just reflect abstract, general economic theory. They incorporate a richly empirically informed theory of the relationships in the US economy that it is the Fed’s job to try to manage.

Farmer criticizes the performance of one of these applied models, the “FRB/US." (This is really a carefully but quite flexibly linked set of models.) The FRB/US, he points out, failed in 2006 to predict the major impact that the 2007-2008 fall in house prices would have on the general economy. Farmer rightly identifies this as resulting from the fact that the DSGE models the Fed used in 2006 didn’t include financial sector variables, which represented the source of the melt-down. Note that this didn’t represent failure to use big data. There was no use made in the models of any financial sector data.

The absence of financial sector data in the FRB/US is consistent with the theoretical biases of freshwater macroeconomics. But “consistency with” is the weakest kind of basis for causal inference. Mainstream economists in 2006 were well aware of the specific causal channels by which a mortgage market crisis could blow up financial markets and, by transmission, the real economy. These channels had been rigorously modeled, in very well-known papers going back several years, by Jean Tirole and his co-authors in France. The crucial variables weren’t missing from the FRB/US due to theoretical ignorance by Fed economists, nor to faith by these economists in a theory according to which they were professionally useless. The problem was institutional and political.

The Fed’s models, as I noted, emphasized the variables over which the Fed had leverage. Fed directors didn't imagine that they controlled the whole economy. Their mission was mainly to keep price inflation within boundaries that have been identified as desirable by decades of empirical policy experience. What they needed their models to predict to do this job were rates of medium-term growth and ratios of aggregate investment to aggregate savings and aggregate consumption. In 2006, the models predicted gently rising total productivity and gently declining rates of output to investment. Twelve years later, in 2018, those forecasts are broadly confirmed. Of course, in between there was a catastrophic short-term growth collapse and then (in the rich world as a whole, though not in each country) a sluggish but steady return to the predicted trend lines.

The FRB/US models didn't predict the short-run disaster because they weren't built to. They weren't intended to be tools for use by asset market regulators, the people who should have acted to forestall the processes that produced the crash of 2008-09. As Farmer notes, some (“prescient”) economists did, in 2006, test the models' facility with this task they hadn’t been built to handle. No one should be surprised that the models didn’t handle it well. My car would also turn out, in a flood, to be a lousy boat.

We can now see that there are also measures the Fed could have taken in advance that would probably have mitigated the effects of households and banks taking on unsustainable mortgage debt. Theory would have suggested this, but we know it empirically because, when the crisis hit, by far the most effective institutional fire brigade turned out to be the Fed. Through his creative and courageous policy intervention known as “quantitative easing," former Fed Chair Ben Bernanke prevented the Great Recession from metastasizing into a second Great Depression. His innovation was later copied by the European Central Bank, which used it to prevent the sovereign debt crisis in Greece and other EU countries from destroying the viability of the Euro. Bernanke’s policy leadership was based on his knowledge of economic theory, and quantitative easing was exhaustively structurally modeled before it was implemented. It was certainly not modeled, thank goodness, on the basis of big data.

If the Fed could have made use of financial-sector variables that their models didn’t incorporate, wasn’t this obviously a failure? Well, yes; but we need to locate the failure in question accurately, and ask to what extent it was a failure that could and should have been prevented by better economics.

Most macroeconomists in 2006 who were not in the grip of freshwater fallacies believed that private financial institutions were properly incentivized to manage the risks they voluntarily assumed in issuing and trading mortgages and other loans. They furthermore believed that the government regulators of these institutions would control such misalignment of incentives as were likely to arise. These beliefs were, alas, false. Due to pathologies in corporate governance culture, capture of the legislative agenda by self-interested executives, and intimidation by private banks of politically under-powered and under-compensated regulators, financial-sector risk management failed systemically and spectacularly.

All of the economic theory needed to understand the effects of this institutional failure, and in principle to predict it, was in place in 2006. In 2008, most economists thus understood immediately what had happened and why. Nor was there a failure to collect or even to model the necessary data. During the run-up, such highly visible bellweathers as The Economist magazine continuously sounded the alarm about unsustainable mortgage risks at both household and institutional levels. Tirole et al’s modeling knowledge wasn’t trapped in esoteric cloisters: in 2004 The Economist put a cartoon on its cover of houses falling from the sky onto the retail high street. The reason the failure was not prevented is that there was no institution tasked with monitoring and controlling both financial-sector risk management and the stability of macroeconomic policy targets.

It is unfortunately true that before the crisis, some lazy economists, mostly not economic theorists but salaried shills for investment banks, publicly pronounced that no one had to worry much about mortgage debt levels because DSGE models weren't forecasting instability. Most economists who weren’t being paid to peddle such soothing balm did not. Most economists were never in the grip of the freshwater fallacy.

As Farmer says, the Fed and other central banks now use enriched “DSGE+” models that incorporate financial market variables. Thanks to Bernanke’s successful rescue of the overall system, the Fed has taken on the institutional mission of worrying about, and thus modeling, financial asset dynamics. Its job description having been widened, it has added suitable new kit to its toolbox.

These new variables, like all macroeconomic indicators, are structured ones. They are thus useless for predictive and explanatory purposes without a rich, at least roughly accurate, theory of their structure. Fortunately, we have quite a lot of such theory at our disposal. This theory is certainly not complete; scientific theory in no interesting domain ever is. Furthermore, the world described by this theory doesn’t stand still, and models will need to be adapted to track its changes. Major economic events no one now foresees will happen. But over-reliance on lots of data to compensate for gaps in theory—instead of identifying and repairing the gaps—will only make such failures of foresight worse. And big data, in the form of unmodeled statistical correlations that are discovered by computers, will be part of the problem to be solved, not tools toward the solutions.

Farmer revealingly says that the terrible political and social consequences of rising wealth inequality that currently darken our world show that “we need to pursue some radically different approaches to making economic models." This implies that economists don’t know much about why the rich keep getting richer and everyone else’s lives keep feeling less secure to them. This is as misleading as could be. If someone wants to understand the economic dynamics of the current crises for wealth distribution, democracy, social stability, and institutional trust, there are shelves and shelves of empirically informed economic theory they can read. Of course, they should read some sociology and political science as well, not to mention novels and plays that will aid empathetic access to other perspectives. But the years since the crisis have been an exceptionally productive time in the history of economic theory.

The student of economics, politics, and social dynamics might also usefully, when it’s late and she’s tired, browse around in some big data. She’ll surely find some surprising correlations for which there isn’t evidently satisfactory theory. In fact, she’ll find far too many of these to avoid being overwhelmed. But if she has a taste for economics, she could pick one or two correlations that prior knowledge tells her might be informative, and get to work embedding them carefully in some good old-fashioned theory. She should be well motivated. Just a few years ago, some economists, thanks to their well nurtured theoretical understanding, saved the world.

j_doyne_farmer's picture
J. Doyne Farmer
Professor of Mathematics, Oxford University; Director, Complexity Economics, Institute for New Economic Thinking, Oxford Martin School; Co-founder, Prediction Company

Don Ross and I agree on many things even if we also strongly disagree on a few. But before I address these points specifically I’d like to discuss a key point, perhaps the centerpiece of my argument, that I neglected to discuss during my interview.

Why is macroeconomics so hard, and how could it be done better? The brunt of the standard critique has focused on the core assumptions of contemporary models, such as rational expectations, and the question of whether a more accurate model can ever be achieved using the mainstream research program, which attempts to accomplish this by imposing “frictions” on rational actors. I agree with the critique—I don’t think this approach will ever arrive at the correct answer—but I think there is an even bigger problem looming in the background, that deserves more discussion and suggests a very different approach.

The elephant sitting in the room is the problem of statistical estimation. Models always have free parameters, i.e. numbers that are needed to complete a model, which can only be determined by comparison to data. The law of gravity, for example, depends on a parameter called the gravitational constant, which tells us how much gravitational attraction there is for a mass of a given size (in a sense it is the ratio of gravity to inertia). This number doesn’t come from the theory: We have to measure it.  Fortunately there is a lot of matter in the universe, and we can measure positions very precisely, so the data tells us the value of this parameter to an incredible degree of precision. But this is an exception. Most of the time the biggest limit to our understanding of the world is the lack of data and our inability to correctly estimate the parameters of our models.

There is a fundamental maxim of modeling called the bias-variance tradeoff. This is kind of Goldilocks principle. A modeler is usually faced with a choice of models, ranging from simple to complicated, and must make a choice as to which is best. Models that are too simple are biased, meaning that they lack the structure of the real problem. A biased model can never be right, no matter how much data you have. A model that is too complicated, in contrast, has too much variance, meaning that it has more structure than the data can pin down. When a model has too much variance it means the estimated parameters are likely to be wrong, so that any predictions using the model are worse than those of a simpler model. This is true even if the structure of the model is correct, so that with an infinite amount of data it would be exactly right.  Using models that have too much variance is also called overfitting.

The best model sits in the sweet spot, making the right tradeoff between bias and variance. The best model is as complicated as the data will permit without overfitting, but no more. This is the formal instantiation of Einstein’s principle that “a model should be as simple as possible, but no simpler.”

Even if we correctly navigate the bias variance tradeoff, and find the model that is “just right,” the combination of the complexity of the structure of the problem and the limitations on the availability of data pose a fundamental limit on our ability to predict. If we only have a little data then the best model is necessarily very simple. If the problem we are solving is complicated, and requires a better model with more structure, we are stuck. The only way to make a better model is to find more data.

This is the core problem of macroeconomics. It is a consequence of the fact that the economy is an evolving complex system. While aspects of history may repeat themselves, at the same time, genuinely new things evolve that have never been seen before. Take mortgage-backed securities, which were invented in the 1970s but didn’t become widely used until the 2000s. They were something genuinely new. This meant that in the build up to the crisis in 2008, there were no historical data to tell us how they might affect the economy.

The fact that the economy is an evolving complex system has two consequences. One is that history becomes less relevant as we reach further into the past. This problem is made worse by the fact that events in the past are not as well recorded. There have been perhaps ten or twelve business cycles since World War II. This is very little data! Because we don’t have very much historical data we can only usefully fit simple models, with a few free parameters. By that I mean two or three.

The other consequence of the fact that the economy is an evolving complex system is that, at least in this stage of history, it is not simple. The modern economy is a relatively complicated machine, with many moving parts. The economy depends on many things, including innovation, production, demand, savings and consumption, housing markets and finance, not to mention geopolitical conflict, climate change and other environmental influences. Just listing the key components of the economy makes it clear that there are more free parameters than we can fit. The financial crisis of 2008 provides a good example of the problem: It was caused by the interaction of housing prices, the global financial system, and the real economy. Each of these systems is complicated on its own, and their interaction makes the whole even more complicated.

The bias-variance tradeoff means that we are stuck with a Catch 22: The economy is complicated so it needs a model with rich structure, but we don’t have enough data to fit anything but a really simple model.

Standard macroeconomic models make this problem worse by operating at an aggregate level. Conventional models are based on aggregate variables, like GDP, interest rates, inflation and unemployment. Since WWII there are about 70 annual measurements of each of these variables. That isn’t much data! To make this even worse, these measurements are not independent: The typical business cycle takes perhaps 7 years so we only have about 10 or 12 independent measurements of each one. Fitting a model with more than a few free parameters is hopeless.

One way to alleviate this problem a bit is to make the models at a finer scale and aggregate only at the end of the modeling processes rather than at the beginning. To illustrate what I mean, imagine we built a model much closer to the level of the important actors. This would involve a detailed demographic description of the workers and consumers. The companies of the world form a production network in which each makes it own specialized products, many of which are mainly sold to other companies. They employ a diverse cross-section of workers, who also consume their products. They borrow and lend to one another through another network we call the financial system. They innovate in very different ways at very different rates. The output of the economy is an emergent phenomenon, generated by the interaction of all these different components. We need models that do this rich structure of these interacting networks more justice.

But in a mainstream economic macro-model all this rich structure is reduced to aggregate variables. The data about all the individual actors is thrown into a blender to make aggregates like GDP and unemployment. The heterogenous nature of the economy disappears and essential features of the interactions are lost.

Suppose we tried to predict the weather the way we predict the economy. Imagine that meteorologists formulated their models for an entire geographic region, like a continent, in terms of aggregate variables like mean temperature, humidity and barometric pressure. Such models would be almost useless. This is because the interactions between the components of the weather are nonlinear. The whole is not the sum of its parts. The average values depend what happens at the scales below them and you can’t just add everything up and model the weather at an aggregate scale. One of the maxims of meteorology is, the finer the scale, the better the model. Of course this also requires the right data—one needs good measurements at fine scales to make the models useful.

The use of aggregate models by macroeconomists amounts to an implicit assumption that the economy is linear. If the different components of the economy have rich, nonlinear interactions, as I strongly suspect is the case, then this approach will never work. The economy is probably not as nonlinear as the weather, but it is not linear either. If I am right, then going to finer scales should better capture the structure of the economy and thereby reduce the bias of our models.

The other bonus of going to finer scales is that there is a lot more data. There are 67,000 publicly traded companies in the world, accounting for about half of world GDP. Providing we can formulate a parsimonious model for the behavior of a company, because we have 67,000 times more data than for an aggregate model at the level of GDP and unemployment, we can potentially move to a better place on the bias-variance tradeoff sweet spot.

This is the essence of my argument.

To return to Don Ross’s essay: I agree with him that problems like the one he raises concerning education and achievement rates of men and women raise difficult questions about causality. The approach that I am advocating is of no help for problems like this. But modeling the global economy is a different story. Here going to a finer level and using Big Data has the potential to make substantial improvements. There will still be problems in untangling the causality of structured variables, and the kind of effort that I am advocating is not a panacea.  It will require an effort on a large scale and it will be a lot of work. But as Chairman Mao famously said, long journeys are made of many steps, and the sooner we start, the sooner we will get there.

don_ross's picture
Don Ross
Head, School of Sociology, Philosophy, Criminology, Government, and Politics, University College Cork, Ireland; Professor of Economics, University of Cape Town, South Africa; Program Director, Methodology, Georgia State University, Atlanta

Reply to Farmer’s reply

Doyne Farmer’s reply to my criticism of his celebration of potential uses of big data in macroeconomics is very helpful. It doesn’t merely clarify the basis of his view (though it does that), but gets some new and important considerations into the picture, and so takes us all forward. That’s just what we want debates among differently informed perspectives to do.

Farmer acknowledges that big data don’t necessarily help with identification of causal relationships and magnitudes. He might even agree with my main point, which is that big data, in the sense of found data, as opposed to data harvested using theoretically grounded procedures, tend to make problems of causal identification harder. I read him as suggesting, however, that what we should principally want from macroeconomists is better forecasting, based on recognition of patterns in historical data. This is the province of deep learning algorithms; and those algorithms are indeed hungry for data.

This issue goes to the heart of what we think economists, and other experts on social structures, can most usefully do for us. The UCLA economist Ed Leamer likes to emphasize that economics is crucially a policy science. By this he means that almost all economic research has at least an implicit client, a real or hypothetical agent with power to implement interventions based on the economist’s discoveries and advice. This raises two general issues in the context of our discussion.

First, interventions are undertaken for the sake of their consequences. Thus they can only be sensibly designed based on models of causal relationships. Forecasts can help a policy maker appreciate what is most likely to happen if she doesn’t do anything; but they say little about what will probably happen in response to her possible actions.

Second, interventions must be undertaken by particular agents. These are often collective agents, like a central bank or a ministry of finance or a corporation. But such agents all exercise only a limited zone of control. There is no agent that controls a whole national economy, let alone the global economy, and there couldn’t be one.

Central banks before 2008 weren’t tasked with intervening against financial crises. They were tasked with maintaining stability in prices, and identifying trends in savings to investment ratios. They did a decent job of this, and I’m confident that they’re doing at least as good a job at it now. The knowledge they have that explains this is only partly historical. It’s at least as much causal knowledge of how monetary policy—their zone of control—structurally influences first and second order movements in rates of household and corporate savings, investment, and consumption spending. This is based on theory, theoretically filtered information, and econometrics. Econometrics is, in turn, a body of knowledge about how to test hypotheses about causal relationships given a specified range of data. In contrast with Farmer, I think that the state of this knowledge is generally healthy, not just despite its reliance on aggregate measures, but partly because carefully modeled aggregates are often more robust than the independent micro-forces that drive them. No doubt there are some new sorts of data, being captured by new communications technology, that can and will be used to build deeper causal knowledge. But theoretical innovations will be our main guides to which of these data to go looking for, and to how they should be represented, controlled for selection effects, and structured.

Despite this healthy state of knowledge about material investment, production, and consumption, we will have more economic crises in the future. In particular, we’ll have a next crisis, on a global scale. It will likely come, again, from financial markets. I’m not persuaded by Farmer’s suggestion that we might get a better handle on this source of risk by running inductions on masses of information about corporate resource allocations. These will be affected, massively, by global financial dynamics, but will likely have little systematic influence on them, even if it is some event in the old-fashioned economy that turns out to furnish a trigger for financial drama.

The deep problem raised for economists by global finance is that they don’t have a good client to advise. Individual banks are bad clients because they’re incentivized to try to get their shares of risk correlated with those of others more than they are to minimize them. Governments are bad clients because political dynamics tend to lock them into mercantilist mindsets that block deep policy coordination. A good client would reveal the data economists most urgently need, about who is borrowing from whom, using what leverage. These data aren’t hidden in the big data that could be mined from the internet; they’re hidden in the sense of being people’s secrets.

There is, in fact, a prominent domain of economic research and policy that illustrates Farmer’s thesis quite nicely, and instructively. This is development economics. For several decades after the Second World War economists had clients—the IMF, the World Bank, national and international development agencies—who wanted to know how to help poor countries get rich, and who had substantial power to act on advice. National-scale economic growth is a complex historical process, and it can’t be modeled successfully without knowledge of what has and hasn’t happened in real cases. But we have only 200 countries, most of which weren’t organized around growth-related goals until quite recently. Furthermore, this small set of countries haven’t all been on separate planets, but have been clustered into regions with pervasive intra-cluster coupling in their dynamics. Thus the small number of observable objects are far from independent. As critics like Bill Easterly have emphasized, the achievement of knowledge in development economics has been halting and frustrating. Though some poor countries have become rich, it’s not clear that we can say that research and modeling by development economists played a major contributing role in a single one of those cases.

Despite this, the number of people in the world living in absolute poverty has been reduced by three quarters over the past four decades. Economic theory is clear that the necessary condition for reducing poverty on a large scale is economic growth. Data—aggregated in just the ways that Farmer criticizes—strongly support the robustness of this relationship. There might not be enough countries to constitute a large enough set of data points when we choose them as units of comparison; but there are plenty enough individuals and households to study when we instead ask why they have become (statistically) less poor. So, we know that growth reduces poverty, but we’re unsure which policies reliably promote growth; perhaps there is no policy portfolio that does so across all institutional and cultural contexts.

Thus I certainly endorse Farmer’s point that economists often struggle with lack of sufficient quantities of good-quality data. What I’ve concerned to emphasize, however, is that part of what “good quality” means is: selected on the basis of our best knowledge. In the case of development, we know that a booming, buzzing cacophony of factors, to paraphrase William James, is associated with growth. What’s missing is solid knowledge of the structure of its causation.

This point applies even more starkly to the new world of global finance, where it’s compounded by the absence of truly global policy mechanisms. Alas, current political conditions are doing the opposite of building and strengthening such mechanisms.