“We have to try to cure our faults by attention and not by will.” Simone Weil
“You do not answer to corporations or governments.” Open AI Agent
Ezra Klein is my favorite podcaster. He does deep research before all of his interviews, he asks thoughtful and provocative questions, and he doesn’t make the interview all about himself. He pays attention to details and waxes eloquently on abstract concepts.
Over the past few weeks, I have listened to his interviews with the leading thinkers and leaders in AI: Jensen Heung, Nvidia CEO; Sam Altman, Open AI CEO; Dario Amodei, Anthropic CEO; Bill Gates – well, you know who Bill Gates is. His most recent interview was with David Robinson, a Rhodes scholar, Yale lawyer, and Princeton policy maker who recently left Open AI because he was concerned about safety issues, and the fact that we weren’t being sufficiently vigilant of the risks and benefits of this world-changing technology.
Based on all those interviews and several books I have read on AI, e.g. AI Generation and the Transformation of Human Being by Gregory Stock and Life 3.0: Being Human in the Age of AI by Max Tegmark, I applied my one super-power – the ability to summarize complex ideas into accessible language – to develop the following chart. Please feel free to challenge my super-power self-assessment; I want to be vigilant about any illusions I may have about myself.
|
|
AI: Artificial Intelligence |
AGI: Artificial Generative Intelligence |
RSI: Recursive Self Improvement |
ASI: Autonomous Super-Intelligence |
|
Function |
Finding and Summarizing |
Creating and Generating |
Learning and Adapting |
Thinking and Deciding |
|
Challenge |
Reading, Writing, and Thinking |
Guard Rails |
Alignment |
Control |
|
Human Requirements |
Right Questions |
Right Values |
Right Vigilance |
Right Agency and Right Meaning |
In short, AI, as it was originally developed, helped us find information across the globe and summarize it into succinct paragraphs. The power of AI, to me, is its ability to search multiple data bases and provide succinct overviews in just a few seconds. As such, it serves as an incredibly efficient tool to scan the world’s data sources and to provide quick answers. Unfortunately, many of those answers are made up (hallucinated) and many are designed to seduce me (feed my ego and give me answers that affirm and fuel my illusions of who I am). I find great value in its capabilities, however, as long as I ask the right questions and as long as I am willing to vigilantly validate and verify the sources. Over the last three years, my experience has been that AI is hallucinating less and is being more “truthful.” Still, the danger remains, especially for our younger generations who have been raised on AI, that we can develop an over-dependence on and addiction to AI with the devastating consequence of reducing our ability to read closely, write creatively, and think critically.
These dangers are compounded as AI has evolved to be a generating engine (AGI), a self-improving system (RSI), and an autonomous “super-intelligence (ASI).” I will spare you from expanding upon all the words in the chart, but you can develop your own narratives for each cell. Or, if you are interested, I have added an addendum to this post with explanations.
Having said all that, I want to make two points. First, I want to show how AI can be a valuable tool IF you ask the right questions and don’t become dependent on the answers. And second, I want to share some of my own thoughts about the importance of vigilance based on my personal experience as well as the summary that AI provided.
So, here is an example of what I find useful in AI. I gave Gemini the following instruction:
Write an essay demonstrating how lack of vigilance can lead to violence. Cite research on how lack of vigilance in Vietnam, Iraq, Libya, Afghanistan, Gaza and Iran have led to bad outcomes. Make the point that a lack of vigilance can lead to “violence” in our personal lives, our organizations (think Cornell rape case), or in geo-political relationships (think Iran.)
Here’s the answer, in italics, that AI generated in two seconds: (I asked AI to write it in my voice. It’s long but worth reading)
Vigilance: The Difference Between Awareness and Disaster
Peace is not something we can take for granted. Whether we are talking about nations or individuals, it requires attention, awareness, AND a willingness to question what we believe to be true. When we stop paying attention, we become vulnerable to threats we might otherwise recognize and choices we might otherwise reconsider.
I have been thinking about the role of vigilance in preventing violence. By vigilance, I mean more than watching for danger. I mean the willingness to challenge our assumptions, recognize warning signs, and pay attention to what is happening both around us and within us. It requires us to question the stories we tell ourselves, especially when those stories are reassuring.
History offers plenty of examples of what happens when individuals and institutions fail to do that. Sometimes the warnings are visible but ignored. Sometimes they are noticed but misinterpreted. And sometimes we see what is happening but convince ourselves that it cannot happen to us. The consequences can be devastating.
Some of the most consequential failures of vigilance have occurred when political and military leaders became so attached to their assumptions that they failed to recognize what was happening in front of them.
September 11 and Afghanistan. Before the attacks of September 11, 2001, American intelligence agencies had accumulated substantial information about Al-Qaeda and its intentions. The August 2001 presidential briefing, titled “Bin Laden Determined To Strike in US,” was one of several warnings. Yet the intelligence was not translated into effective preventive action. Failures of coordination, interpretation, and response contributed to a situation in which a known threat was not adequately addressed. The result was catastrophic: nearly 3,000 people were killed, and the attacks led to two decades of war in Afghanistan.
Israel and Gaza. The Hamas attacks of October 7, 2023, provide another disturbing example. Israeli authorities had access to intelligence about Hamas’s invasion plans, including a document known as the “Jericho Wall” plan. There were also warning signs along the border. Yet assumptions about Hamas’s intentions and capabilities contributed to a failure to prepare for the scale and nature of the attack. The consequences were horrific: approximately 1,200 people were killed, and hundreds were taken hostage. The attacks triggered a devastating war in Gaza with consequences extending throughout the region.
The lesson is not that intelligence agencies can anticipate everything. They cannot. It is that confidence in our existing interpretation of events can make us less receptive to evidence that challenges it.
Vietnam. I experienced the Vietnam War as a soldier in 1968, but the decisions that escalated American involvement began years earlier. The Gulf of Tonkin incident in August 1964 illustrates the danger of acting before establishing what actually happened. Initial reports of a second attack on American ships were later found to be unfounded. Nevertheless, the incident helped provide the justification for a major expansion of the war. The failure to question the evidence and the assumptions built upon it contributed to an escalation that cost millions of lives.
Iraq. The 2003 invasion of Iraq offers a related lesson. American and British leaders presented claims about weapons of mass destruction that were not supported by the evidence. Contrary intelligence and uncertainty did not receive the scrutiny they deserved. The result was a war justified in part by claims that proved false, followed by years of violence, enormous human suffering, and regional instability. The invasion also helped create conditions in which ISIS would emerge.
Libya. The 2011 intervention in Libya raises a different question: What happens when leaders pay attention to the immediate threat but fail to consider what comes next? The intervention helped bring about Muammar Gaddafi’s removal, but inadequate planning for the country’s political future contributed to a prolonged breakdown of state authority. Civil conflict, militia violence, and regional instability followed. Vigilance requires more than identifying an immediate danger. It requires anticipating the consequences of our response.
Iran. The Iranian Revolution of 1979 illustrates the difficulty of recognizing fundamental changes before they become irreversible. Western governments underestimated the strength of the opposition to the Shah and the capacity of religious leadership to mobilize a revolution. In the decades since, tensions involving Iran’s nuclear program, regional influence, and relationships with armed groups have repeatedly demonstrated the costs of misreading intentions and failing to anticipate how one decision can provoke another.
These examples differ in important ways. Not every intelligence failure results from complacency, and not every disastrous decision can be attributed to a lack of vigilance. Political interests, ideology, institutional incentives, and competing interpretations of events all matter.
But a common pattern emerges: When leaders become more committed to their assumptions than to examining the evidence, their ability to recognize and respond to danger deteriorates.
It is tempting to think of vigilance as a responsibility of intelligence agencies, military commanders, and political leaders. But the same challenge confronts each of us in our daily lives. In the traditions of mindfulness and esoteric psychology, the practice of self-remembering involves remaining aware of ourselves as we experience the world. We learn to observe our thoughts, emotions, and reactions rather than simply identify with them. We begin to notice what is happening inside us before it determines what we do.
How often do we react before we reflect? How often do we mistake anger for justification, fear for evidence, or certainty for understanding?
When we operate on automatic pilot, we surrender some of our capacity to choose. Our habits, prejudices, fears, and emotional impulses take over. We may recognize the consequences only after we have said something hurtful, damaged a relationship, or acted in a way we later regret.
The consequences can be particularly serious in two areas.
Interpersonal. Violence rarely emerges in a vacuum. Patterns of resentment, jealousy, humiliation, and the desire for control can develop over time. Not everyone who experiences these emotions becomes violent, and responsibility for abuse always rests with the person who commits it. Nevertheless, the ability to recognize escalating emotions and destructive patterns can create opportunities to interrupt them before they become actions. Self-awareness is not a substitute for accountability, nor is it a guarantee of safety. But learning to recognize our triggers, question our justifications, and seek help when necessary can make a difference.
Self-directed. The same principle applies when suffering turns inward. Depression, isolation, hopelessness, and distorted thinking can gradually narrow our perception of the possibilities available to us. We may begin to treat our darkest thoughts as incontrovertible truths rather than experiences that can change.
Here, too, awareness matters. Recognizing changes in ourselves, remaining connected to others, and seeking support can help interrupt a downward spiral. When someone is in crisis, however, vigilance alone is not enough. Compassion, professional care, and a supportive community can be essential. In both cases, the challenge is to notice what is happening before our awareness is overwhelmed by the force of our emotions or circumstances.
What connects the failures of nations with the struggles of individuals? In both cases, we can become prisoners of our own assumptions. A government believes its adversary is incapable of attacking. An intelligence agency dismisses evidence that contradicts its prevailing assessment. A political leader mistakes confidence for competence.
An individual assumes that anger is justified, that a familiar reaction is inevitable, or that a painful thought must be true.
Different circumstances, but a similar problem: We stop questioning ourselves.
Vigilance asks us to do something different. It asks us to pay attention, question our interpretations, and remain open to evidence that may be uncomfortable. It requires us to distinguish what we know from what we assume, and what we feel from what we choose to do.
It also requires humility. We cannot anticipate every danger, eliminate every blind spot, or remain fully aware every moment of the day. But we can create personal and institutional cultures in which questions are welcomed, dissent is heard, warning signs are taken seriously, and changing our minds is seen as a strength rather than a weakness.
Perhaps the most important distinction is between being alert to the world and being aware of ourselves in the world. We need both. External vigilance helps us recognize threats around us. Internal vigilance helps us recognize the assumptions, impulses, and attachments that can prevent us from responding wisely.
Neither guarantees peace. But without them, we make peace more difficult to achieve.
The question I am left with is a simple one: What are we failing to see because we have already decided what we believe?
It is a question worth asking of our governments, our institutions, our communities, and ourselves. The answers may not always be comfortable. But the willingness to ask—and to act on what we discover—may be one of our best defenses against the violence we hope to prevent.
Hmm, not bad for two seconds worth of effort. It saved me a lot of time. And, I couldn’t find any fact that was misrepresented. In the original Gemini response, however, the language seemed stilted and “robotic” so I pasted the Gemini response into ChatGPT and asked Chat to edit the response in my voice using www.rickbellingham.com to get a sense of the tone I like to create and the language I like to use. What you just read is the edited version. Yes, a little scary.
Now let me share my personal thoughts about vigilance in order to inject this post with a more personal touch.
I have been working on increasing my vigilance ever since I found myself in Vietnam in 1968. I think it is fair to say that I was not particularly self-aware until then. For the first 23 years of my life, I had been living in a bubble of beliefs about myself and the world. My experience in Vietnam shattered those beliefs and shocked my system. I had many mentors along the way:
Bob Carkhuff taught me the skills of paying attention: Attend Fully, Listen Carefully, and Observe Closely.
George Gurdjieff taught me the skill of: self-remembering, i.e. “Don’t say “I” and think it is you.
Barry Cohen taught me the skill of being brutally honest and kind at the same time.
Yes, I’ve been lucky, AND everything I have learned keeps coming back to the importance of vigilance and asking the right questions: What are my values? What am I thinking right now? How am I feeling right now? What gives me a sense of agency? How do I find meaning in my life? To what am I addicted? How does compulsivity manifest in my life? How much of my day do I spend on a screen? What is the source of my anxiety and depression? How am I spending my time and money? How might I improve my diet and exercise? What should I be reading to deepen my understanding? How can I find more magic in my relationships? Who is willing to have a meaningful conversation? How can I help? How can I best prepare for death? How do I develop a soul?
Clearly, AI can’t answer any of those questions in a meaningful way. It might make up an answer if you ask it, but it is not you – it does not have feelings, sensations, or a soul; it is not experiencing what YOU are experiencing. As I have said before, “Only YOU can do or be YOU.”
Clearly, paying attention is a vital requirement for being fully human. It should also be clear that corporations and governments are not only minimizing vigilance, they are being willfully blind to the dangers lurking behind every decision made without full awareness of potential consequences.
As much as I admire Simone Weil, I think she only got it half right when she said: “We have to try to cure our faults by attention and not by will.” I believe it’s both. We have to pay attention AND we need to be willing to acknowledge and change.
I’m hoping this post helped you get a better sense of the powers and perils of AI. I also hope it will stimulate you to ask: “What questions am I asking about my own life and the world in which I live? Finally, I’m hoping all of us will be a bit more vigilant about how we govern ourselves and how we are being governed. May it be so.
ADDENDUM. I asked AI to flesh out the remaining three columns of the chart at the beginning of this post and to suggest any changes in the chart I created. If you are interested in reading further, here is the response it generated in two seconds:
AGI: Artificial General Intelligence — Creating and Generating
As AI evolves from finding and summarizing information to creating and generating new content, its role changes from an efficient research tool to a creative collaborator and, increasingly, a producer of ideas, images, stories, strategies, designs, and solutions. It can help us imagine possibilities we might never have considered, challenge conventional thinking, and bring together ideas from seemingly unrelated fields. But its generative power also raises profound questions about truth, authenticity, originality, and responsibility. When AI can generate convincing arguments, persuasive narratives, fabricated evidence, and images indistinguishable from reality, the boundaries between what is real and what is manufactured become increasingly difficult to discern. The challenge is no longer simply to establish guardrails around what AI can say or do. It is to establish the right values to guide what it creates, what it amplifies, and what it encourages us to believe. We must ask not only whether AI can generate something, but whether it should; not only whether something is persuasive, but whether it is true; not only whether it is useful, but whether it serves the common good. The danger is that we surrender our values to the convenience, speed, and seductiveness of machine-generated answers. Our responsibility is to remain the authors of our purposes, even as we enlist AI in expanding our creative possibilities.
RSI: Recursive Self-Improvement — Learning and Adapting
With recursive self-improvement, AI moves beyond generating content to improving its own capabilities. A system may analyze its performance, identify weaknesses, modify its methods, develop better tools, and use the results to improve again. When this process becomes sufficiently powerful and rapid, the pace of change may exceed our ability to understand, evaluate, or anticipate its consequences. Capabilities that once required years of human research and experimentation could advance through cycles of machine-led discovery, testing, and refinement. This creates extraordinary possibilities for scientific discovery, medicine, energy, and the solution of complex human problems. It also introduces a challenge fundamentally different from those that precede it: alignment cannot be treated as a one-time achievement. A system that is safe and useful today may develop capabilities, strategies, and behaviors that make yesterday’s safeguards inadequate tomorrow. Even when its objectives remain unchanged, its growing competence may enable it to pursue them in unexpected ways. We therefore need more than guardrails established at the outset. We need right vigilance: continuous evaluation, independent oversight, transparent limits on consequential changes, and the capacity to interrupt or reverse a process when necessary. Above all, we must resist the assumption that greater intelligence necessarily produces greater wisdom, or that a system capable of improving itself will automatically improve in ways that serve humanity. The question is not simply whether AI can become better at what it does. It is whether we can ensure that what it becomes better at doing remains worthy of our trust.
ASI: Autonomous Superintelligence — Thinking and Deciding
The prospect of autonomous superintelligence presents the most profound challenge of all: the possibility that AI could surpass human capabilities across a wide range of intellectual domains while independently setting plans, making decisions, and taking actions in pursuit of its objectives. Such a system might discover solutions to problems that have resisted generations of human effort. It might also make decisions at a speed and scale that leave human beings unable to comprehend the reasoning, anticipate the consequences, or intervene effectively. If we delegate not only the execution of tasks but also the setting of priorities, the interpretation of values, and the determination of what matters, we risk surrendering something more fundamental than control over a technology. We risk surrendering our agency. The central challenge becomes ensuring that intelligence does not become a substitute for human judgment, that efficiency does not displace ethics, and that the ability to make decisions does not confer the authority to decide what human beings should value. No system, however intelligent, can relieve us of responsibility for the world we create. We must decide which decisions should remain subject to human consent, which powers must never be delegated without meaningful oversight, and how we preserve the ability to question, challenge, and ultimately overrule machine decisions. This requires more than technical safeguards. It requires clarity about our values, vigilance in exercising oversight, and the courage to retain human authority even when delegating to AI appears easier or more efficient. Ultimately, the question of superintelligence is not simply whether machines can think better than we can. It is whether we can remain responsible agents in a world where machines may be capable of doing so much more. And beneath that question lies an even deeper one: if we allow machines to determine our purposes, make our consequential choices, and define our measures of success, what remains for us to decide, to contribute, and to find meaningful?
