AI safety

The Dawn of Human History We often view our current era as the pinnacle of human achievement, a sophisticated destination reached after millennia of struggle. In reality, if we navigate the next few centuries successfully, we are living in the earliest infancy of our species. Will%20MacAskill, a leading philosopher in the Effective%20Altruism movement and author of What%20We%20Owe%20the%20Future, argues that we are the ancients. Future generations, potentially numbering in the trillions and spread across the stars, will look back at the 21st century as the distant, primitive past. This perspective, known as **longtermism**, shifts our moral focus from immediate concerns to the vast potential of the human trajectory. The astronomical scales involved are staggering. While a typical mammal species survives for about a million years, humanity has existed for only 300,000. If we merely match the average mammal, we have 700,000 years of potential ahead. However, human ingenuity suggests we could last much longer. The Earth will remain habitable for hundreds of millions of years; the sun will shine for billions more. If we survive the "time of perils"—this unique window where our technological power exceeds our wisdom—the stakes for what we could achieve are nearly infinite. Every action we take now to reduce existential risk is an investment in a future that dwarfs the present in both scale and value. Navigating the Great Reflection and Value Lock-in One of the most insidious threats to a flourishing future is not physical destruction, but the premature crystallization of moral values. MacAskill warns of **value lock-in**, a state where a single ideology or set of norms becomes globally dominant and impossible to dislodge. History provides chilling examples of this through the rise of totalitarian regimes like the Nazis or Stalinism, which sought to crush all ideological competition. In the future, advanced Artificial%20Intelligence could provide a dictator with the tools for perfect, eternal surveillance and enforcement, making a single, potentially flawed worldview permanent. To illustrate the absurdity of locking in current values, we need only look at the past. For most of history, slavery was considered a natural and necessary part of the social order. Had the moral views of the 1700s been locked in forever, the progress we treasure today would never have occurred. It is highly improbable that we, in the early 21st century, have perfectly solved every moral riddle. We likely still harbor "moral catastrophes" in our current practices—perhaps our treatment of animals or our disregard for future generations. Therefore, our primary goal should be to maintain a **morally exploratory society**. We must preserve the "plasticity" of our culture, ensuring we have the time and freedom to engage in a **Long Reflection** before making choices that define the remainder of history. The Triple Threat: Extinction, Collapse, and Stagnation To reach the Long Reflection, we must navigate three primary categories of existential risk. The first is **Extinction**, the literal end of the human story. While natural risks like asteroids or supervolcanoes are relatively low, anthropogenic risks are rising. MacAskill highlights engineered pathogens as a critical concern. Unlike natural viruses, which rarely evolve to kill 100% of their hosts, a bio-weapon could be designed for maximum lethality and transmissibility. Technologies like Far-UVC%20lighting and advanced wastewater monitoring are essential defensive investments to bring this risk toward zero. The second threat is **Unrecoverable Civilizational Collapse**. This is a scenario where a catastrophe—perhaps a nuclear winter—destroys the industrial base of society. While MacAskill is relatively optimistic that humanity would eventually rebuild, he notes a unique bottleneck: fossil fuels. We have already depleted the easily accessible, "low-hanging fruit" of surface coal and oil. If we collapse to a pre-industrial state and have already burned the fuels needed to kickstart a second industrial revolution, we might remain trapped in an agrarian state indefinitely, vulnerable to the next natural extinction event. Finally, there is **Technological Stagnation**. If innovation slows to a crawl before we develop the tools to protect ourselves—such as defensive biotech or robust AI%20safety protocols—we remain in a permanent state of vulnerability. Stagnation is a death sentence by a thousand cuts; it leaves us exposed to the "red balls" in the urn of technological discovery without giving us the means to mitigate their impact. We must keep the conveyor belt of progress moving, but with a heavy bias toward defensive and stabilizing technologies. The Cultural Engine of Moral Progress Change is rarely driven by economics or law alone; it is fueled by culture. The abolition of slavery is a prime example. While many assume it was an economic inevitability, historical evidence suggests it was a massive cultural shift driven by moral arguments and activism, often at a significant economic cost to the powers of the time. This underscores the power of ideas. In our modern context, the focus on Climate%20Change represents a significant moral milestone. It is one of the first times in human history that a global movement has formed around the interests of people who do not yet exist. However, we must ensure our concern for the future is not captured by a single issue. While Climate%20Change is a vital challenge, the risks from Artificial%20Intelligence and biotechnology may be even more acute in terms of their potential to cause total extinction or value lock-in. We need to expand our "moral circle" to include not just the victims of our current environmental choices, but the trillions of lives that could be snuffed out by a misaligned AI or a lab-leaked pathogen. This requires a massive investment in the humanities and moral philosophy—fields that currently receive a vanishingly small fraction of the funding dedicated to technological advancement. Engineering Resilience for the Next Millennium If we take the interests of future generations seriously, we must build tangible safeguards. One such proposal is the creation of a **civilizational backup**. Similar to the Svalbard%20Global%20Seed%20Vault, we could establish hermetically sealed refuges for groups of humans and scientists. These bunkers, stocked with the sum of human knowledge and the tools to rebuild, would serve as an insurance policy against a global pandemic or nuclear event. While the idea may sound like science fiction, on the scale of a multi-trillion-dollar global economy, the cost of such a "Plan B" is negligible compared to the value of preserving the entire human legacy. We must also address the "Happy Birthday" problem of bad lock-in. The song is a terrible melody with an awkward octave leap that most people cannot sing, yet it is universal because it became the standard during a "moment of plasticity." Our current global systems—from our modes of governance to our economic structures—are currently in a similar moment of plasticity. We have the opportunity to ensure that the norms we pass down are not merely the first ones that gained traction, but the ones most likely to lead to long-term flourishing. By prioritizing **Effective%20Altruism** and long-term thinking, we move from being a species of teenagers living for the moment to becoming the responsible ancestors our descendants deserve.