It is a bit embarrassing to admit a preoccupation with this gigantic old question, but it is human, I suppose. Tackling it straight on seems to be an exercise in hubris, but if you stick to science, you soon realize that we are still struggling to figure out what the question is. It helps, I think, to distinguish four separate questions.
The first question is why capacities for suffering exist at all. Why do organisms care if they are injured? Why do they try so hard to avoid dying? Why do they fight just to have sex? Why do we experience a certain kind of pain just from being ignored? Such motives, behaviors, and experiences are made possible by brain mechanisms shaped by natural selection. While many individual experiences of suffering arise because something has gone wrong, either in person's life or brain, the capacities for suffering and pleasure exist because they are useful, at least for the genes that make them possible. This is terribly sobering. Many people still confuse the question of why capacities for suffering exist, with the very different question of what causes suffering in individual instances. I have called this the "clinician's fallacy" because doctors and therapists so often treat defenses as if they were diseases. Eventually the distinction will become clear.
The second question is why we so often continue to do things that make us miserable. Why do we pursue goals we can't reach given that this causes so much unhappiness? Why we can't take Buddha's advice and transcend our desires? The answer is that people who have given up difficult goals have had fewer children. These goals are not just wealth, power and sex. Trying without success to protect and help one's children causes intense suffering and everyone recognizes why we can't give up this goal. The evolutionary origins of our motives do not make us helpless puppets but they can help us to understand why controlling our desires is difficult.
The third question is why we treat others the way we do. It is silly to say that people are innately generous or selfish, but the fact of poverty is universal. I spent this week on call where the truth hits you in the face; for all the riches of our society, millions of people have no job, no money, few friends and not even a warm place to sleep. Politicians enact policies that make it even easier for the rich to keep their riches. This is nothing new, but neither is it unalterable. Any improvement, however, needs to start from the realities of human nature.
The fourth question is very different. The other three ask why people are mostly the same, but this one instead asks why people are different. The explanations for differences in suffering include differences in genes, experiences, personalities, and social settings. Most of our efforts to understand suffering have been here. This research provides genuine knowledge, but only part of a complete answer.
Big problems often motivate proposals for grand quick solutions that give rise to horrendous unanticipated consequences. A gradual deepening of our evolutionary understanding of ourselves offers more modest but surer hope. Many misgivings about evolutionary approaches to human behavior come from a simple misconception. Natural selection explains how the competitive struggles of life shaped us, but this does not mean that life is only a struggle nor does not mean that life cannot be made better. Quite the contrary. If we want to prevent social catastrophes and gradually improve our world, we had better start with a real understanding of why we are the way we are. Negative psychology tells us why some people are unhappy and how bad this is for them. In another corner, positive psychology tells us why some people are happier than others and how good this is for them. What we need now is "diagonal psychology" that investigates the costs of experiencing positive emotions when they are not warranted, and the benefits of capacities for suffering. This will offer a real foundation for understanding why the world is so full of suffering.