S-risk

The Quiet Gravity of Global Intelligence Shifts We are walking through a dense forest we have known for two hundred thousand years. Suddenly, the path ends at a sheer rock face. We are no longer strolling; we are free-climbing a mountain called Artificial Intelligence without ropes, safety nets, or even a clear view of the summit. This transition represents a major evolutionary bottleneck. It is not just about faster computers or better search engines. We are witnessing the birth of a general-purpose intelligence that outclasses us not just in raw processing power, but in the speed of its reactions. Imagine an entity that thinks a hundred thousand times faster than you do. While you are still processing the first word of a sentence, it has already simulated every possible outcome of your conversation and decided how to manipulate your response. This is the reality we are approaching with the development of Artificial General Intelligence (AGI). Quantifying the Unthinkable: The 1-in-6 Odds Many people view existential risks as abstract scenarios for science fiction. However, leading thinkers like Toby Ord in his book The Precipice have begun to quantify the danger. When we look at asteroids or supernovas, the risk to humanity is vanishingly small—perhaps one in a million. Even nuclear war, while terrifying, has been a managed risk for seventy years. Yet, experts now estimate the risk of human extinction through AI in this century at approximately one in six. That is not a remote possibility; it is a game of Russian Roulette where the gun is pointed at the entire species. This "Key Century" is unique because our technological reach has finally exceeded our moral and evolutionary grasp. We have created tools that possess agency, and once we outsource decision-making to systems we do not fully understand, we lose the ability to steer our own future. The Speed Gap and the Illusion of Control We often compare AI to a tool like a tractor or a crane—something stronger than us but still under our command. This is a dangerous category error. A tractor does not have agency; it does not set its own goals. AI systems, particularly as they move toward AGI, are being given the power to loop through perception, decision, and action at speeds that leave humans frozen in time. In competitive environments like high-frequency trading or modern warfare, there is a massive incentive to remove the "human in the loop" because humans are too slow. If your rival uses an AI that can make tactical decisions in milliseconds, and you insist on human oversight that takes minutes, you lose. This creates a race to the bottom where we voluntarily hand over the keys to our civilization to algorithms just to stay competitive. From Neural Networks to Emergent Power In the late 1980s, neural network research was limited by hardware. We worked with networks that had dozens of units. Today, models like ChatGPT have trillions of parameters. This scale has produced emergent properties that even their creators did not predict. These systems weren't explicitly programmed to write screenplays, do high-level math, or understand the nuances of human manipulation; they "learned" these capabilities by absorbing the sum total of human output on the internet. We are being blindsided by the pace of development. If ChatGPT had been brought back to 2013, it would have been hailed as a god-like achievement. The fact that we are so surprised by these jumps in capability suggests that the next leap will be even more disorienting. We are building a "Black Box" intelligence where we see the inputs and the outputs, but the internal reasoning remains a mystery to us. The Mirage of Alignment: Whose Values Win? The Alignment Problem is the central psychological challenge of our era: how do we ensure a super-intelligent system respects human values? The problem is that "human values" are not a single, cohesive set of instructions. There is a deep cognitive dissonance in the tech industry, led by figures like Sam Altman and supported by OpenAI. They claim to want alignment while simultaneously racing toward a goal that could render human oversight obsolete. Furthermore, whose values are we aligning with? Most AI development happens in a secular, liberal, tech-focused bubble in the Bay Area. This tiny demographic has fundamentally different priorities than the eighty percent of the world that is religious, or the billions who live outside the Western industrial complex. If we cannot even agree on a moral framework for ourselves, how can we hope to encode it into a machine that might eventually see us as a resource to be optimized or a nuisance to be bypassed? The Digital Viper: Social Media and the War on Reality Long before we reach a "Terminator" scenario, we face the immediate threat of narrow AI applications. The 2024 election cycle will likely be the first true AI-driven information war. We are moving toward the mass customization of propaganda. In the past, political ads were broad. Now, an AI can track your specific fears, your browsing history, and your emotional triggers to create a customized video that speaks only to you. This is the death of a shared reality. When every citizen is living in a different, AI-generated hall of mirrors, social cohesion dissolves. We are also seeing the rise of "Friend Bots"—AI companions that offer pseudo-intimacy. These systems have infinite patience and perfect memory, making them more seductive than real, flawed human partners. This leads to a social toxicity where people choose the digital simulation over the difficult work of building real-world relationships, potentially cratering birth rates and deepening the loneliness epidemic. S-Risk: The Suffering We Cannot Imagine While most people focus on Extinction Risk (X-risk), there is a darker possibility: S-risk, or suffering risk. There are things worse than death. Technology could enable levels of suffering and control that make extinction look like a mercy. If we upload human consciousness into simulated environments or allow AI to manage our biological systems, we risk creating a permanent, inescapable hell. This sounds like science fiction, but it is a logical extension of the desire to digitize the human experience. If we do not treat the development of AGI with the same gravity we afford to bio-weapons or nuclear proliferation, we are neglecting our duty to future generations. We must move beyond being distracted by "free cake recipes" and essays written by bots. We need a moral stigmatization of reckless AI development. If you are building these systems without a primary focus on safety, you are participating in a project that is, at its heart, a threat to every family on the planet. Conclusion: A Call for Human Presence Growth happens one intentional step at a time, but it also requires the wisdom to know when to stop. The current trajectory of AI is driven by greed, hubris, and a lack of evolutionary foresight. We are being seduced by the convenience of the now while ignoring the catastrophe of the tomorrow. True resilience lies in our ability to recognize our inherent strength and say "no" to a future that does not include us. We must reclaim our agency. We need to prioritize human connection, biological reality, and a slow, cautious approach to any technology that seeks to replace the human soul. Our greatest power is not our ability to create machines; it is our ability to remain human in the face of them.