Philip Tetlock: I thought it might be a good idea before we go into the quality of questions that are input into tournaments to say a few more words about the nature of the Good Judgment Project and what it meant to win the forecasting tournament, how it won the forecasting tournament, and what inferences you might or might not want to draw from victory in forecasting tournaments in general.
If you were to turn to slide twenty-eight in the book, it raises the question, how much can the Good Judgment Project improve foresight?
On the weekend of July 30th, Edge convened one of its "Master Classes." In the past, these classes have featured short courses taught by people such as psychologist and Nobel Laureate Daniel Kahneman ("A Short Course in Thinking About Thinking"); behavioral economists Richard Thaler and Sendhil Mullainathan, again with Kahneman ("A Short Course in Behavioral Economics"); and genomic researchers George Church and J. Craig Venter ("A Short Course on Synthetic Genomics").
This year, the psychologist and social scientist Philip E. Tetlock presented the findings based on his work on forecasting as part of the Good Judgment Project. In 1984, Tetlock began holding "forecasting tournaments" in which selected candidates were asked questions about the course of events: In the wake of a natural disaster, what policies will be changed in the United States? When will North Korea test nuclear weapons? Candidates examine the questions in teams. They are not necessarily experts, but attentive, shrewd citizens.
Steven Pinker, who has written about Tetlock's work on Superforecasting, noted that "Tetlock is one of the very, very best minds in the social sciences today. He has come up with one brilliant idea after another, and superforecasting is no exception. Everyone agrees that the way to know if an idea is right is to see whether it accurately predicts the future. But which ideas, which methods, which people have an actual, provable track record of non-obvious predictions vindicated by the course of events? The answers will surprise you, and have radical implications for politics, policy, journalism, education, and even epistemology—how we can best gain knowledge about the world we live in."
Among Tetlock's "students" at the Edge weekend were many intellectual heavyweights including political scientist and National Medal of Science winner Robert Axelrod; psychologist, Nobel Laureate, and recipient of the 2013 Presidential Medal of Freedom Daniel Kahneman; the political scientist and Director of Stanford’s CASBS Margaret Levi; Google Senior Vice President Salar Kamangar; psychologist and National Medal of Science winner Anne Treisman; Roboticist Rodney Brooks, former head of MIT's Computer Science Lab; W. Daniel Hillis, pioneer in massively parallel computation; medical inventor Dean Kamen; and Peter Lee, Corporate Vice President, Microsoft Research, overseeing MSR NExT.
Over the weekend in Napa, Tetlock held five classes, which are being presented by Edge in their entirety (8.5 hours of video and audio) along with accompanying transcripts (61,000 words). Commenting on the event, one of the participants wrote:
"The interesting thing is that this is not about a latest trend that might scale in one or two years, but about real change that might take a decade or two. Also, these masterclasses are not only much more profound than any of the conferences popularizing contemporary intellectualism. The possibility to spend that much time with the clairvoyants in a setting like this also gives you a sense of community so much greater than any of the advertised."
PHILIP E. TETLOCK, Political and Social Scientist, is the Annenberg University Professor at the University of Pennsylvania, with appointments in Wharton, psychology and political science. He is co-leader of the Good Judgment Project, a multi-year forecasting study, the author of Expert Political Judgment and (with Aaron Belkin) Counterfactual Thought Experiments in World Politics, and co-author (with Dan Gardner) of Superforecasting: The Art & Science of Prediction (forthcoming, US, Crown, September 29th; UK, Random House, September 24th). Further reading on Edge: "How To Win At Forecasting: A Conversation with Philip Tetlock" (December 6, 2012). Philip Tetlock's Edge Bio Page.
Available for pre-ordering
Available for pre-ordering
Robert Axelrod, Political Scientist; Walgreen Professor for the Study of Human Understanding, U. Michigan; Author, The Evolution of Cooperation; Member, National Academy of Sciences; Recipient, the National Medal of Science; Stewart Brand, Founder, The Whole Earth Catalog; Co-Founder, The Well; Co-Founder, The Long Now Foundation; Author, Whole Earth Discipline; John Brockman, Editor, Edge; Author, The Third Culture; Rodney Brooks, Panasonic Professor of Robotics (emeritus), MIT; Founder, Chmn/CTO, Rethink Robotics; Author, Flesh and Machines; Brian Christian, Philosopher, Computer Scientist, Poet; Author, The Most Human Human; Wael Ghonim, Pro-democracy leader of the Tarir Square demonstrations in Egypt; Anonymous administrator of the Facebook page, "We are all Khaled Saeed"; W. Daniel Hillis, Physicist; Computer Scientist; Chairman, Applied Minds; Author, The Pattern on the Stone; Jennifer Jacquet, Assistant Professor of Environmental Studies, NYU; Author, Is Shame Necessary?; Daniel Kahneman, Professor Emeritus of Psychology, Princeton; Author, Thinking, Fast and Slow; Winner of the 2013 Presidential Medal of Freedom; Recipient of the 2002 Nobel Prize in Economic Sciences; Salar Kamangar, Senior Vice President, Google; Fmr head of YouTube; Dean Kamen, Inventor and Entrepreneur, DEKA Research; Andrian Kreye, Feuilleton Editor, Sueddeutsche Zeitung, Munich; Peter Lee, Corp. VP, Microsoft Research; Former Founder / Director, DARPA's technology office; Former Head, Carnegie Mellon Computer Science Department & CMU's Vice Provost for Research; Margaret Levi, Political Scientist, Director, Center For Advanced Study in Behavioral Sciences (CASBS), Stanford University; Barbara Mellers, Psychologist; George Heyman University Professor at UPennsylvania; Past President, Society of Judgment and Decision Making; Ludwig Siegele, Technology Editor, The Economist; Rory Sutherland, Executive Creative Director and Vice-Chairman, OgilvyOne London; Vice-Chairman, Ogilvy & Mather UK; Columnist, The Spectator; Philip Tetlock, Political and Social Scientist; Annenberg University Professor at UPenn; Author, Expert Political Judgment; and (with Dan Gardner) Superforecasting (forthcoming); Anne Treisman, James S. McDonnell Distinguished University Professor Emeritus of Psychology at Princeton; Recipient, National Medal of Science; D.A.Wallach, Recording Artist; Songwriter; Artist in Residence, Spotify; Hi-Tech Investor
"La Miravalle" at Spring Mountain Vineyard
In the circle of clairvoyants: At a vineyard north of San Francisco, Philip Tetlock of the University of Pennsylvania (left) presented his findings. Initially skeptical was Nobel Laureate Kahneman (third from left). Photo: John Brockman / edge.org
It is as though high status pundits have learned a valuable survival skill, and that survival skill is they've mastered the art of appearing to go out on a limb without actually going out on a limb. They say dramatic things but there are vague verbiage quantifiers connected to the dramatic things. It sounds as though they're saying something very compelling and riveting. There's a scenario that's been conjured up in your mind of something either very good or very bad. It's vivid, easily imaginable.
It turns out, on close inspection they're not really saying that's going to happen. They're not specifying the conditions, or a time frame, or likelihood, so there's no way of assessing accuracy. You could say these pundits are just doing what a rational pundit would do because they know that they live in a somewhat stochastic world. They know that it's a world that frequently is going to throw off surprises at them, so to maintain their credibility with their community of co-believers they need to be vague. It's an essential survival skill. There is some considerable truth to that, and forecasting tournaments are a very different way of proceeding. Forecasting tournaments require people to attach explicit probabilities to well-defined outcomes in well-defined time frames so you can keep score.
Tournaments have a scientific value. They help us test a lot of psychological hypotheses about the drivers of accuracy, they help us test statistical ideas; there are a lot of ideas we can test in tournaments. Tournaments have a value inside organizations and businesses. A more accurate probability helps to price options better on Wall Street, so they have value.
I wanted to focus more on what I see as the wider societal value of tournaments and the potential value of tournaments in depolarizing unnecessarily polarizing policy debates. In short, making us more civilized. ...
There is well-developed research literature on how to measure accuracy. There is not such well-developed research literature on how to measure the quality of questions. The quality of questions is going to be absolutely crucial if we want tournaments to be able to play a role in tipping the scales of plausibility in important debates, and if you want tournaments to play a role in incentivizing people to behave more reasonably in debates.
CLASS III — Counterfactual History: The Elusive Control Groups in Policy Debates
(Publication date: September 1)
CLASS IV — Counterfactuals and the Making of (Better) Superforecasters
(Publication date: September 8)
For the psychology professor Philip Tetlock, the hunt for Osama Bin Laden is a classic example of the insufficiency of secret-service agencies. When Barack Obama gave the green light for that operation four years ago, he knew he was making one of the most difficult decisions in his life—one that would not only mean life or death for those involved, but also sway the course of history and help determine his legacy. The prognoses offered by the secret-service agencies were inconclusive: some put the likelihood for success at 40%, others at 80%. In the movie based on this operation, Zero Dark Thirty, the CIA agent Maya insists she is 100% certain of success. In reality, Obama determined the chances stood at fifty-fifty and gave the green light against the advice of his secretary of defense.
In Tetlock's view, such imprecisions present an unacceptable risk. Forecasts alleging complete certainty are, of course, unscientific. But Tetlock argues that a historic decision must not be based on imprecise reports. While Obama may have enjoyed luck on a historic scale, with his special task force finding Bin Laden and killing him, Tetlock insists that the work of secret-service agencies must change—fundamentally.
Since the eighties Tetlock has worked on precisely this endeavor. For four years now he has pursued research at the University of Pennsylvania at the behest of the Intelligence Advanced Research Projects Activity (IARPA), which the NSA and the CIA, together with fourteen other American secret-service agencies, established in 2006, in order to develop new methods for secret-service work in the post-9/11 era. Among IARPA’s divisions are the Office for Anticipating Surprise, the Office of Smart Collection, and the Office of Incisive Analysis.
Psychologists' "forecasting tournaments" capture the interest of the NSA and the CIA
This past weekend Tetlock met with twenty scientists and engineers on a vineyard north of San Francisco. Two European journalists were invited; otherwise, the meeting was closed to the public. Tetlock wanted to discuss the results of his Good Judgment Project, which he has worked on for 24 years. The scientists discuss the project under ideal circumstances: sheltered from the summer heat in the cool living room of a stately Victorian house. With palms in the garden, a front porch and wainscoting, the house exudes colonial splendor. The air is redolent with the rose beds in front of the windows and the precious woods of the furniture. The host is John Brockman of Edge Foundation, Inc. (http://edge.org), which is the best network for such debates in the country. That explains the presence of such intellectual heavyweights as the Nobel Laureate in Economics Daniel Kahneman, the political scientist and National Medal of Science winner Robert Axelrod, the political scientist Margaret Levi, and Google Vice President Salar Kamangar. It isn’t easy to hold one’s own in such a group. Kahneman in particular, the cleverest of them all, is skeptical.
Tetlock begins by recounting the history of the Good Judgment Project. In 1984 he began holding "forecasting tournaments" in which selected candidates are asked questions about the course of events. In the wake of a natural disaster, what policies will be changed in the United States? When will North Korea test nuclear weapons? Candidates examine the questions in teams. They are not necessarily experts, but attentive, shrewd citizens. One of the best forecasters so far is Bill Flack, a former official of the U.S. Department of Agriculture from Nebraska.
This can't be the end of human evolution. We have to go someplace else.
It's quite remarkable. It's moved people off of personal computers. Microsoft's business, while it's a huge monopoly, has stopped growing. There was this platform change. I'm fascinated to see what the next platform is going to be. It's totally up in the air, and I think that some form of augmented reality is possible and real. Is it going to be a science-fiction utopia or a science-fiction nightmare? It's going to be a little bit of both.
JOHN MARKOFF is a Pulitzer Prize-winning journalist who covers science and technology for The New York Times. His most recent book is the forthcoming Machines of Loving Grace: The Quest for Common Ground Between Humans and Robots. John Markoff's Edge Bio Page
THE NEXT WAVE
I'm in an interesting place in my career, and it's an interesting time in Silicon Valley. I grew up in Silicon Valley, but it's something I've been reporting about since 1977, which is this Moore's Law acceleration. Over the last five years, another layer has been added to the Moore's Law discussion, with Kurzweil and people like him arguing that we're on the brink of self-aware machines. Just recently, Gates and Musk and Hawking have all been saying that this is an existential threat to humankind. I simply don't see it. If you begin to pick it apart, their argument and the fundamental argument of Silicon Valley, it's all about this exponential acceleration that comes out of the semiconductor industry. I suddenly discovered it was over.
Now, it may not be over forever, but it's clearly paused. All the things that have been driving everything that I do, the kinds of technology that have emerged out of here that have changed the world, have ridden on the fact that the cost of computing doesn't just fall, it falls at an accelerating rate. And guess what? In the last two years, the price of each transistor has stopped falling. That's a profound moment.
We know there's a law of nature, the second law of thermodynamics, that says that disorderliness grows with time. Is there another law of nature that governs the complexity of what happens? That talks about multiple layers of the structures and how they interact with each other? Embarrassingly enough, we don't even know how to define this problem yet. We don't know the right quantitative description for complexity. This is very early days. This is Copernicus, not even Kepler, much less Galileo or Newton. This is guessing at the ways to think about these problems.
SEAN CARROLL is a research professor at Caltech and the author of The Particle at the End of the Universe, which won the 2013 Royal Society Winton Prize, and From Eternity to Here: The Quest for the Ultimate Theory of Time. He has recently been awarded a Guggenheim Fellowship, the Gemant Award from the American Institute of Physics, and the Emperor Has No Clothes Award from the Freedom From Religion Foundation. Sean Carroll's Edge Bio Page
LAYERS OF REALITY
I've always studied the laws of physics. I've always been curious about how the universe works, where it comes from, what are the rules that govern the behavior of the universe at the deepest level, so I do physics for a living. I study cosmology and the Big Bang and what happened before the Big Bang, if anything. It's a system of things that hooks up in very complicated ways to our human scale lives. There's the natural world that scientists study, and we human beings are part of the natural world.
In modern science, and I include the humanities here, science in a German sense of science—rigorous scholarship across all domains—in modern science we've gotten used to the idea that science doesn't offer meaning in the way that institutional religions did in the past. I'm increasingly thinking that this idea that modernity puts us in a world without meaning—philosophers have banged on about this for a century-and-a-half—may be completely wrong. We may be living in an intellectual building site, where a new story is being constructed. It's vastly more powerful than the previous stories because it's the first one that is global. It's not anchored in a particular culture or a particular society. This is an origin story that works for humans in Beijing as well as in Buenos Aires.
It's a global origin story, and it sums over vastly more information than any early origin story. This is very, very powerful stuff. It's full of meaning. We're now at the point where, across so many domains, the amount of information, of good, rigorous ideas, is so rich that we can tease out that story.
DAVID CHRISTIAN is Professor of History, Macquarie University, Sydney; Author, Maps of Time: An Introduction to Big History. David Christian's Edge Bio Page
WE NEED A MODERN ORIGIN STORY: A BIG HISTORY
I'm a Russian historian, and I love teaching Russian history. I taught it during the Cold War when it seemed exceptionally significant. Teaching it in Australia, where I was, was a bit like talking about the dark side. I felt my students needed to know about that world.
I'm not Russian, but I was teaching Russian history and eventually I realized I was giving the subliminal message that humans are divided, at a fundamental level, into competing tribes. Having lived through the Cuban Missile Crisis, I remember it vividly. I was a schoolboy in England where this tribalism threatened to blow us all up. That was a very vivid experience for me. I thought, for historians to keep teaching this subliminal message—that we're divided by tribes—is not a good thing.
JB: It all started with a young scientist named Laurie Santos at a conference that I ran saying, “How do we get rid of some of these ideas that are just standing in front of us? Just blocking everybody?”
LP: What are the ideas that blind us now do you think? And blind us into confusion, and argument, and that kind of controversy?
JB: Name a field. ... It comes down to: is science advertising or is it argument?
LP: Your favorite. Which would be one of yours?
JB: Daniel Kahneman has studied human rationality and found out that characteristics we thought we had as humans aren’t necessarily the case. We are not Homo Economicus, we’re not the rational human beings we thought we were. A lot of what we do is pre-conscious and without acknowledgement.
LP: Now that’s interesting. So that’s a big idea about who we are and how we control our lives with rationality and free will. Another idea was the idea of love as well. This is one that attracts criticism from one of your contributors. So tell us a bit more about that. ...
The reasons why I'm engaged in trying to lower the existential risks has to do with the fact that I'm a convinced consequentialist. We have to take responsibility for modeling the consequences of our actions, and then pick the actions that yield the best outcomes. Moreover, when you start thinking about—in the pallet of actions that you have—what are the things that you should pay special attention to, one argument that can be made is that you should pay attention to areas where you expect your marginal impact to be the highest. There are clearly very important issues about inequality in the world, or global warming, but I couldn't make a significant difference in these areas.
JAAN TALLINN is a co-founder of The Centre for the Study of Existential Risk at University of Cambridge, UK as well as The Future of Life Institute in Cambridge, MA. He is also a founding engineer of Kazaa and Skype. Jaan Tallinn's Edge Bio Page
by Max Tegmark
I find Jaan Tallinn remarkable in more ways than one. His rags-to-riches entrepreneur story is inspiring in its own right, starting behind the Iron Curtain and ending up connecting the world with Skype. How many times have you skyped? How many people do you know who created a new verb?
Most successful entrepreneurs I know went on to become serial entrepreneurs. In contrast, Jaan chose a different path: he asked himself how he could leverage his success to do as much good as possible in the world, developed a plan, and dedicated his life to it. His ambition makes even the goals of Skype seem modest: reduce existential risk, i.e., the risk that we humans do something as stupid as go extinct due to poor planning.
Already after a few short years, Jaan’s impact is remarkable. He is a key supporter of a global network of non-profit existential risk organizations including The Future of Humanity Institute, The Machine Intelligence Research Institute, The Global Catastrophic Risk Institute, The Centre for the Study of Existential Risk at University of Cambridge, and The Future of Life Institute, the last two of which he co-founded.
I’ve had the pleasure to work with him on The Future of Life Institute from day one, and if you’ve heard of our recent conference, open letter and well-funded research program on keeping artificial intelligence beneficial, then I’d like to make clear that none of this would have happened if it weren’t for Jaan’s support. If you’re an intelligent life form reading this text millions of years from now and marvel at how life is flourishing, you may owe your existence to Jaan.
MAX TEGMARK is a Physicist, MIT; Researcher, Precision Cosmology; Founder, Future of Life Institute; Author, Our Mathematical Universe. Max Tegmark's Edge Bio Page
I split my activity between various organizations. I don't have one big umbrella organization that I represent. I use various commercial organizations and investment companies such as Metaplanet Holdings, which is my primary investment vehicle,to invest in various startups, including artificial intelligence companies. Then I have one nonprofit foundation called Solenum Foundation that I use to support various so-called existential risk organizations around the world.
"To accomplish the extraordinary, you must seek extraordinary people."
A new generation of artists, writing genomes as fluently as Blake and Byron wrote verses, might create an abundance of new flowers and fruit and trees and birds to enrich the ecology of our planet. Most of these artists would be amateurs, but they would be in close touch with science, like the poets of the earlier Age of Wonder. The new Age of Wonder might bring together wealthy entrepreneurs ... and a worldwide community of gardeners and farmers and breeders, working together to make the planet beautiful as well as fertile, hospitable to hummingbirds as well as to humans. —Freeman Dyson
In his 2009 talk at the Bristol Festival of Ideas, Freeman Dyson pointed out that we are entering a new Age of Wonder, which is dominated by computational biology. The leaders of the new Age of Wonder, Dyson noted, include "biology wizards" Kary Mullis, Craig Venter, medical engineer Dean Kamen, and "computer wizards" Larry Page, Sergey Brin, and Charles Simonyi, and John Brockman and Katinka Matson, the cofounders of Edge, the nexus of this intellectual activity.
Every year since 1999, we have hosted The Edge Annual Dinner (sometimes referred to as “The Billionaires' Dinner”). Guests have included the leading third culture intellectuals of our time, dining and conversing with the founders of Amazon, AOL, eBay, Facebook, Google, Microsoft, PayPal, Space X, Skype, and Twitter. It is a remarkable gathering of outstanding minds—the people who are rewriting our global culture.
Through such gatherings, its online publications, master classes, and seminars, Edge, operating under the umbrella of the non-profit 501 (c) (3) Edge Foundation, Inc., promotes interactions between the third culture intellectuals and technology pioneers of the post-industrial, digital age, the "worldwide community of gardeners and farmers and breeders" referred to by Dyson as the leaders of the "Age of Wonder”. Edge members share the boundaries of their knowledge and experience with each other and respond to challenges, comments, criticisms, and insights. The constant shifting of metaphors, the intensity with which we advance our ideas to each other—this is what intellectuals do. Edge draws attention to the larger context of intellectual life.
An indication of Edge's role in contemporary culture can be measured, in part, by its Google PageRank of "8", which places it in the same category as The Economist, Financial Times, Le Monde, La Repubblica, Science, Süddeutsche Zeitung, Wall Street Journal, and Washington Post. Its influence is evident from the attention paid by the global media:
"The world's smartest website … a salon for the world's finest minds." Guardian—"the fabulous Edge symposium" New York Times—"A lavish cerebral feast", Atlantic—"Not just wonderful, but plausible", Wall St. Journal "Fabulous", Independent—" Thrilling", FAZ—"The brightest minds", Vanity Fair, "The intellectual elite", Guardian—"Intellectual skyrockets of stunning brilliance", Arts & Letter Daily—"Terrific, thought provoking", Guardian— "An intellectual treasure trove, San Francisco Chronicle—"Thrilling colloquium", Telegraph—"Fantastically stimulating", BBC Radio 4—"Astounding reading", Boston Globe—"Where the age of biology began", Süddeutsche Zeitung—"Splendidly enlightened", Independent—The world’s best brains", Times— "Brilliant... a eureka moment at the edge of knowledge", Sunday Times—"Fascinating and provocative", Guardian—"Uplifting ...enthralling", Daily Mail—"Breathtaking in scope", New Scientist—"Exhilarating, hilarious, and chilling", Evening Standard—"Today's visions of science tomorrow", New York Times
Edge is different from The Invisible College (1646), The Club (1764), The Cambridge Apostles (1820), The Bloomsbury Group (1905), or The Algonquin Roundtable (1919), but it offers the same quality of intellectual adventure.
Perhaps the closest resemblance is to the early nineteenth century Lunar Society of Birmingham (1765), an informal dinner club and learned society of leading cultural figures of the new industrial age—James Watt, Joseph Priestly, Benjamin Franklin, and the two grandfathers of Charles Darwin, Erasmus Darwin and Josiah Wedgwood. The Society met each month near the full moon. They referred to themselves as "lunaticks". In a similar fashion, Edge attempts to inspire conversations exploring the themes of the post-industrial age.
In this regard, Edge is not just a group of people. I see it as the constant shifting of metaphors, the advancement of ideas, the agreement on, and the invention of, reality.
Vancouver, March 18, 2015