RICHARD THALER: It is the case, as I said before, that the kids that were better, turned out to be much better adults.

KAHNEMAN: Right. They had a better future.

THALER: I don't think we know that much about how you teach those skills, do we?

KAHNEMAN: Well, again, you would look at cultures, because I'm convinced that is not going to be the same in China and here. Clearly some cultures know how to do this better than others, but what it is I don't know, and I'm not sure that you could teach it locally. It would have to be something broader.

MULLAINATHAN: One of the things that we talked about vis-a-vis working out is something I 'd love to know if there is evidence on, which is I find working out four times a week is much harder than seven times a week. It is not even a habit issue. There's no choice left. Its no longer even a temptation. It's not even in my choice set.

PARKER: Isn't one of the answers to this social scrutiny because it multiplies the pain, multiplies the consequences. Its like if it's a cultural thing, and doing something that is considered wrong the consequences are now massively multiplied and the consequences don't go away when you leave the immediate situation. I've heard that hypothesized as one of the primary differences between Chinese culture and American culture.

TREISMAN: Big brother is watching you.

THALER: I don't know how much evidence there is to it, but it seems intuitive that one of the things that good parenting involves is giving kids some practice in self control so they learn how to use that muscle.

KAHNEMAN: A lot of practice! There are things, there are rewards that you will gain from self-control now. That happens routinely.

THALER: But that will involve giving some freedom, right? There are two ways to err, I mean there are millions ways to err as a parent. But, you can think of two extremes. You could be so lax that the kids always get what they want right away. Or you could be so strict that they never have any discretion. The right answer is somewhere in the middle.

MULLAINATHAN: Like binge drinking in US colleges versus the lack of it in Europe. These guys have been told no alcohol, and then they get to college and boom! I get to do all of this stuff. There were these kids I knew whose parents didn't let them watch television. And then they get to college and they have absolutely no way to control themselves.

ROMER: The model that you create for people matters a lot as well. It's just role modeling, and then maybe influencing the people they hang around with.

MULLAINATHAN: But what's interesting about this is we're conjecturing. This is not a place where we have lots of evidence. Is it just that its hard to run these experiments when we're looking for long term changes?

KAHNEMAN: It's an experiment in character formation.

BEZOS: But wait a second, if you know it's stable in time from age four, because you did the big lollipop small lollipop experiment for 40 years. It seems like you should be able to study what happens in the first four or five years of life.

KAHNEMAN: It doesn't predict just the child's income it predicts the parents income.

THALER: Although the remarkable thing is that those videotapes I'm showing, which are ancient, were at the Stamford Daycare Center. These are the kids of Stamford graduate students and young faculty so it's a very truncated example. Walter then moved to Columbia and the local sample has become enriched. But even within that very truncated sample, it had predictive power.

MULLAINATHAN: There are all of these Headstart experiments that are going on which are done by a population that is very much focused on teaching, such as how do we teach math; why can't we take some of those and add modules, why can't we add self-control training modules to Headstart experiments? They're there. There's an entire field that does these things.

KAHNEMAN: You would see that, without social support, this thing would just go, it would just vanish.

TREISMAN: That's what teachers try and do, they try and teach them all kinds of things.

MULLAINATHAN: I'm not saying we're not doing it, I'm saying we're studying it more systematically. And it's possible one lesson we learn is that none of the things that we think work in school, can work. It's true for example, even in Headstart in the education domain. One thing they are finding is that the average effects are tiny and they are mediated by stuff at home. But then the next generation of stuff takes that into account. The next generation of stuff is not just teaching the child, but doing interventions as part of Headstart that brings in the mother and the father and involves them. Are there cheap and cost effective ways of involving them? All I am saying is that maybe there is a tool out there for studying this that's outside of the usual lab that we tend to operate in.

PARKER: If what you're trying to do is simulate some of the things that work in say China or other Asian nations then maybe what those nations are doing is not teaching self-control but institutionalizing punishment, or creating greater social scrutiny for doing things that are wrong, and maybe it has nothing to do with the kind of self control training you can get in a Headstart program.

MULLAINATHAN: Good point, but even that we might be able to do in a Headstart program where instead of teaching self control lets suppose we set up social norms amongst the children and it maybe that your sense of perceived punishment is set pretty early. It's like the eyes on the wall. Most punishments are relatively perceived punishment, what will our friends think about it.

HILLIS: Do you think there is a general skill of self-control or perhaps a particular skill of delayed gratification and a payoff? Perhaps you need to be taught each of these kinds of self-control.

THALER: In fact, we know the answer to that, and you're right, there is no general—and it's funny—it's the same guy. The great contribution to psychology by Walter Mischel, the guy who did those videotapes, is to show that there is no such thing as a stable personality trait.

KAHNEMAN: Stable yes, general no.

THALER: Right. One person is honest in some domains but not in others; they have great self-control regarding their money, but then they're fat.

KAHNEMAN: Lots of people are hard working and have very little control over their diet.

THALER Right. Sendhil works out every day and then spends all his time on websites. Right? Yes, we know the answer to that.

ROMER: For what it's worth, the comparable conversation in South Korea or in Singapore would not be how do we give our kids more self control, but how do we make kids that are as creative like American kids. There are several dimensions here that we've got to think about.

THALER: All right, so it's open discussion.

I want to make a quick comment. Jeff and I were talking earlier about what do we hear about Kindle prices. And Jeff made the comment that no one will pay more for the Kindle version than they would pay for the hardback, or the written version. Jeff, what I thought you might find interesting, it's related to but Sendhil was talking about yesterday. My impression is that when the paperback version comes out, the Kindle price goes down. Right? This is just like one of Sendhil's experiments yesterday. I've got my Kindle and let's say I would rather read it on my Kindle than as a book. And now we've changed some irrelevance rice but I can now buy a paperback. I would rather have them on my Kindle but this came from a reference point that is going to affect my willingness to pay for the Kindle version.

BEZOS: There is a shift in time. the paperback comes out after the hardback. And so it's another reason why the price should go down at about that time. The issuance of the paperback, it's basically an indicator that there's enough time has passed for the price to go down.  

MULLAINATHAN: But the flip side of this issue, when there's a book that's only available in paperback and selling at $9.95, it seems like it's to your advantage to introduce the hardback that no one's going to buy and put it on the site, because when the Kindle price is $11, well, it's more than the paperback, but it's less than the hardback.

BEZOS: Are you looking for a job? Let's talk after.

ROMER: But that one's subtle. That could be nothing to do with choice sets. It's a good that would compete, a new good that competes on price and it's partly its price discrimination. It could be that this has nothing to do with choice set effects.

KAMANGAR: An earlier premise that we've assumed in all these discussions is that there are a few anonymous, expert designers of your choice sets, who construct your choices for important decision areas like health care.  The Web has changed this model, I think.  Now you have information about the actions of your friends and other web identities, so you're able to see the decisions of a large number of choice architects, which may then affect the hundreds of decisions you may make over the course of a week about questions like what to buy, what to wear, what to do, and where to go.

THALER: Because of Twitter?

KAMANGAR: In the model we were discussing earlier, there's one person who more or less decides my Medicare choice set.  They are trying to put themselves into my shoes, but there are hundreds of other decisions I make in the course of my week that this person has no influence over.  Now, with the Web, I can see what article my friend read or what book my friend purchased.

BEZOS: You can choose your choice architect.

KAMANGAR: Exactly.

MULLAINATHAN: A concrete example of this is Chow Hound versus Zagat's. Used to be I go to Zagat's to tell me if a restaurant is good. There are these five guys at Chow Hound who I trust. What I want to know is what is their rating specifically.

It would be interesting if there were software you could create to facilitate that for things beyond restaurants. Imagine if there was something like a Facebook application for basic choices, financial choices, where there was some easy way to structure the problem of financial choices so that the information could be passed through much more effectively.

KAMANGAR: Another example of the potential that comes with this change is that today my investment decisions are somewhat based on what I know about whom else is investing in a company. But it's hard to get that information. It would be useful to have transparency about a company's investors. For example, if I knew Jeff had invested in a particular retail operation that would be enough information for me to want to invest. Of course, Jeff and others will in some cases want to protect the privacy of their choices, so this identity information today is obscured through legal entities. But for an increasing set of decisions, the Web will allow people to share their choices, and bring to life new choice architects.

MULLAINATHAN: It would also be interesting to see if there was a way to ask questions like what percentage stocks and bonds, to even facilitate more. The problem with what you just described is that it promotes emulation behavior, which has a downside in that maybe Jeff is investing in that retail company as a hedge but that doesn't make sense for you to do as a hedge. It would be interesting to provide not just data for people to emulate each other but information on why something is being done and some way to transmit specific advice.

THALER: I want to go to forms and they'll say other people like you ...

KAHNEMAN: These are people who have your tastes are now shopping for this.

THALER: The difference between the Amazon model and the NetFlix model is the NetFlix model they ask you a bunch of questions about movies you've seen. You guys don't do that. Yours is based on what you bought.

BEZOS: There are two different kinds of recommendations. There are personal ones based on your purchase history and your rating history. And then there are ones that are just people who bought this product.

THALER: Right. The two models are a bit different. One is finding people like you. This model is: people who bought this book also bought that book.

ROMER: We know that systems have a kind of indeterminacy in that if some perturbation gets one object to be purchased more often, then other people purchase it. You can run experiments, if you take a population of people and they'll all end up with one thing that everybody is recommending because everybody else is buying it. Or in a different population, you get a very different outcome.

THALER: Does everybody know this experiment? They created eight different worlds with young people downloading music. And they gave them a bunch of music, a list of songs, and you could listen to the clips of the songs or listen to the whole song and then download it if you wanted to. There was one world where you got no feedback about what other people were doing. And then there were several other worlds where you got feedback on what other people in your group were doing. What happened was that consensus formed strongly within each group, but the groups were wildly different. That's the small perturbation point.

ROMER: If all that music is equivalently good it doesn't matter, but you've got to worry about equilibrium where everybody converges on a mortgage or a financial institution. That's not operable.

MULLAINATHAN: The key is to transmit not just data to emulate but information. Just data on what's been done is what generates emulation. If you start to transmit what you liked then so a bunch of people tried A they liked it that's why we're pooling on it so that's why its important that we are clear about what we are transmitting Information on just my purchases, just that, is not nearly as useful as that's why I thought relative to what Amazon does now or what Netflix does now it seems like in Salar's comment you are emphasizing people.

KAMANGAR: You pick somebody who is not only an expert at that subject but someone who you want to emulate when it comes to that subject.

THALER: Somebody who can emulate you.

The other point that's related to what we've been talking about and its vaguely related to what we were talking about in the second session yesterday about machine-readable information, is the question of how the information age is going to change things. For example, Zagat's has changed things, for better or for worse. And we can start to think now pro-actively ...how we want to think about this stuff.

KAHNEMAN: There is a concrete way for Amazon to engage the user. At the moment, there is no incentive for me to rate the books. I get nothing for it. Some people just like to start off, if you don't. But suppose we're talking films and not books, suppose that as an incentive to rate the films that I had seen that I am going to see the ratings of people whose ratings are similar to mine. That would produce an incentive for me to rate some films, I hated this one, I like this AND you would get the recommendation.

THALER: Well, that is the Netflix model.

PARKER: There are studies on something very similar to what you're proposing and we were using the hypothesis that most people don't like rating things but just about everybody enjoys voting and there's a subtle differences between rating or reviewing and voting. And some of that has to do with immediate gratification.

The widget on content news sites that has the highest click-through of pretty much any widget that appears whether its clicking-through to see a full article or clicking to another page, is polls, because you don't get to see the results of the poll until you click, and there's a cost benefit there.

The cost of clicking and saying "yes, I agree", or "no I don't agree", is so low, even though the reward of getting to see what everyone thought is very low, even the most jaded web user who knows that this data is totally unscientific and not that meaningful still wants to know and the cost is so low that they do it. We found that across a variety of platforms, polls, where there is a hidden answer, do extremely well, but that rankings, for example you have to give five stars, do terribly, like less than 30 percent of the people will rank them, but more than 50 percent of people will vote.

LECLERC: Vote as in yes or no.

PARKER: Yes or no, in order to see concealed results.

THALER: It can be a yes or no, or it could be "who's the best quarterback of all time, pick one of these five."

KAHNEMAN: I have proposed that as a model to Gallup, that is, to set up panels of experts on particular problems where the reward would be exactly that. That is you express your opinion on a problem and what you get is what other experts in the same field are thinking about that problem. That would be very valuable.

PARKER: There's another dimension to this also, which is that reviews work slightly better than rankings. At first blush, this seems wrong, because it’s so easy to rank something—the cost is incredibly low. But the thing is, the benefit is even lower, since people know that ranking doesn't get you anything—there's almost no benefit at all.

The only people who do rankings are people who are dismissively derided as self-indulgent sociopathic types who want their ideas to be registered with this kind of collective decision-making process, so that maybe they can influence other people down the line. But the amount of influence that you have is difficult to measure because you don't know how many other people have ranked so its not clear to you that your five star ranking is going to influence anyone.

The review on the other hand is what you say on screen and you have the sense that you're weighing in and perhaps that's more meaningful but in both of those cases sort of normal people, the majority of people, won't do it because they don't, they're doing it for strangers, putting it out there like a blogger that doesn't have a significant readership, on a blog post, plus you have to have a lot of time on your hands to do it.

There's all these reasons why reviewing is slightly better than rating, both of them suffer from the same the problem which is you don't know your audience, you can't see them, it doesn't make you famous, maybe it influences somebody's buying decision who you'll never see.

MULLAINATHAN: There's another advantage of polls over ratings. I tried to use that Netflix feature of rating movies. Ratings are in that dead zone. Here is what I mean: is this a 2 or a 3 or a 3 1/2. I want to give it a 3 ½ but I don't know…It's painful. Good / bad is easy to do. And what else is easy to do is to give qualified responses, to say I like this but the ending was crappy, does that make it a 2 or a 3. I don't know. Rating systems have enough dimensionality that you have to choose but not enough that you can genuinely express.

PARKER: Right. No one knows what a scale of one to five is. Everybody's idea of a scale of one to five is different, of a one to ten is different. You don't even know what yours is, so you certainly don't know how other people will use the scale. There are two ways you increase the number of people reviewing.

One is to create a community of people who are all active ly reviewing things and who all get to know each other through that process. That's the Yelp solution. They created a community where their core users contribute a lot of valuable content that is consumed by a large passive audience. It started as a virtual community but evolved into one where people meet online and later hang out in person. The best reviewers are recognized by the community for the quantity and quality of their reviews.

Then there's the other way, the Flixster on Facebook approach, where you're writing reviews for your actual friends so you get social feedback in the form of comments or “thank yous” and you're seeing other people’s reviews and providing feedback to them too, so it seems like a meaningful and purposeful kind of interaction. It’s like writing email to your friends, only in some ways better because your friends provide the impetus to write, but there’s additional value to persistence; that is, putting your thoughts out there on the Internet for posterity and potentially influencing a group of people beyond your immediate friends.

DYSON: I am going to change the subject. We're trying to sum up and I am interested in the specific hard problems that we came up with. I am going to throw out the ones that I noticed.

ROMER: Decision drinks.

PARKER: I drink half a red bull before I make decision. First I think about it a little bit sober, and then I drink half a red bull and think about it a little bit differently.

THALER: One of my goals coming here was to get all of you techno-wizards to start thinking about the ways in which you could help alter society for the better and the machine-readable data idea is an idea that I am pushing but it's also like a metaphor for other ways in which technology can change things.

The government is bad about this. If you look at essentially any government website, its hopeless, and the question of well, can the private sector do something. And if so what? I think is an interesting question. Any private sector website beats any public sector website and that's not so surprising there is a reason why you pay more for Fed Ex than the Post Office.

MULLAINATHAN: My ulterior motive in coming was that it seems that philanthropic behavior doesn’t recognize yet the benefits of R&D. Those of you in this room and in technical fields have seen the enormous impact of focused R&D efforts that combine pure research with development, building concrete solutions to important problems.

The research and development cycle has been powerful in those fields. What I wanted to give, in some small way, was the sense that R&D in the poverty space has a similarly huge potential to have enormous impact. For some reason in the poverty space people get too focused on well, there's this NGO that has a potential answer.

Let’s search for the innovation in the world. A great complement to this is centralized R&D. Not just doing R ("Research") but doing R plus D ("Development") can have big returns, creating potentially very impactful programs as well as basic insights about human behavior and social phenomena such as poverty.

KAHNEMAN: What we're saying is that there is a technology emerging from behavioral economics. It's not only an abstract thing. You can do things with it. We are just at the beginning. I thought that the input of psychology into behavioral economics was done. But hearing Sendhil was very encouraging because there was a lot of new psychology there. That conversation is continuing and it looks to me as if that conversation is going to go forward. It's pretty intuitive, based on research, good theory, and important.

BROCKMAN: Psychology is considered a science. Can the same be said for behavioral economics? Is it scientific, or is it seat of the pants?

KAHNEMAN: Its clearly based on data. It's data rich.

BROCKMAN: Where's the intersection? When does psychology become economics?

KAHNEMAN: You saw the perfect example in Sendhil's work, He collaborates with the psychologists and they give similar talks and I am sure they share slides a lot. But a lot of the time you could not tell that Sendhil is not a psychologist, because he is thinking with both sets of tools.

THALER: Behavioral economics and good psychology, there's a lot of art. There is science and there are well-crafted experiments, but thinking about what the right experiment to run, was art and, there are 80 gazillion experiments, which ones are relevant to getting people to plant the right seed. That's a problem that Sendhil and I have been talking about for, well, since he was born. You're now seeing the results of 15 years of conversations. And there wasn't a scientific way of answering that question.

MULLAINATHAN: Part of the wedge is that the name, although it is descriptive from an academic point of view, is not descriptive from a product point of view for an outsider. A lot of what makes behavioral economics interesting is psychology, it is about what happens inside the mind. These phenomena are taking things that are happening inside the mind and interfacing them with things happening in the world, the environment, and getting feedback or getting interesting responses from that.

We happen to call the word economics. But it's not economics. You could be talking about crime, you could be talking about many things, in the social domain, the entire spectrum of human behavior. Anyone who is interested in the broader world should be interested in something we currently call "behavioral economics".