“For what shall it profit a man, if he gain the whole world, and suffer the loss of his soul?”

- Mark 8:36

Will AGI generate unimaginable profits for some mortal human beings? We certainly should expect as much, given that every other transformative technology since the industrial revolution has generated previously unimaginable wealth.

Prior to 1750, global GDP doubled every ~6,000 years. After the industrial revolution, the doubling period became several decades. Now, depending on the data one chooses to believe, the doubling period is between ten and twenty years.¹

If AGI offers a similar paradigm shift to the industrialization that preceded it, the result will be an economy that doubles in size every six to nine months. If you believe that transformative, paradigm shifts will occur with increasing frequency², then the doubling period might reach weeks, or even days in your lifetime. If you happen to own some equities, you should be exceedingly optimistic.

Ruin

Nassim Nicholas Taleb opines about the “risk of ruin,” warning would-be investors against profiles that present a “Taleb distribution,” which generates positive, compounding returns in the vast majority of cases…but contains a small probability of a catastrophic loss.

If the technology on which we rely offers such a sequence of gains, but contains the risk of losing all we hold dear, it is little more than such a distribution.

Scale

The story of tech giants is a story of scaling. The magic of software is that, unlike the industrial outputs of assembly lines a century prior, once an application is developed, producing a thousand, a million, or even a billion copies requires comparatively minimal cost.

This is why tech companies fetch stupendous valuations. This is how the dot-com bubble swelled in the 90s. This is why ChatGPT traversed the distance between an unknown tool and a household term in only a few months.

Network effects add fuel to the fire - in far too many cases, one winner ultimately dominates the landscape (until it’s disrupted and a new idea replaces it). Civilization selects its preferred social media venue, its preferred search engine, its preferred online retailer, and market capitalizations reach the trillions.

And this creates, at least in the case of AGI, some horrifying incentives.

Compromise

In a world where fortunes are made by being first en route to becoming the largest, dominant player, compromises on ethical principles are commonplace. In fact, almost by definition, the participant most willing to make such compromises in pursuit of the desired pinnacle is the game’s most likely victor.

This idea is much older than the tech revolution. Consider the example of two Abrahamic faiths.

Judaism and Christianity emerged from the same old testament roots, shared the same biblical stories, and built their traditions from the same ideological templates. Their paths diverged as a function of interpretations and general values. Judaism required adherence to a larger number of precepts, and certainly made onboarding more difficult.³ The more accommodating Christianity spread, scaled, and in some respects (at least in terms of global population), won the race for souls.

Judaism still exists, to be sure. But its numbers are far less impressive. If it were a tech startup, VCs would be flocking to the folks with the cathedrals.⁴

Virality

The best idea isn’t always the one that spreads the fastest. Memes and clickbait headlines are the modern replacements of sensationalism. Lowest-common-denominator political screeds replicate in minds more effectively than nuanced, well-reasoned positions.

But the focus on scale and distribution leads to some unsavory, unsettling outcomes.

Utilitarian calculus is financially sound when scaling a business. Maximizing market share means more users, which means more cash flow. But, as discussed, it is this utilitarian reasoning that leads us to moral compromises.

Utilitarianism

Imagine that tomorrow morning, a deity presents you with a moral dilemma. If you murder the first innocent person you encounter, the odds of misaligned AGI will fall by 1%.

The utilitarian mathematics are trivial. A misaligned AGI might kill or enslave billions, to say nothing of all the human lives not yet lived. Diminishing this risk by 1% is “worth” far more than one life.

Moreover, even if the probability of avoiding a misaligned AGI was 100%, most of us would agree that this senseless murder would still be not only tragic and depraved, but immoral.

We don’t want to live in a world governed by the ghost of Jeremy Bentham.⁵

Necessity

Is utilitarian analysis always necessary? Is it always necessary to win the race? Is compromising one’s values always required to do so? Applying this lens to AI, the debate becomes one of existential risk and inevitable ruin.⁶

The original goal of designing AI was to increase human agency and avoid harming us in the process. Somewhere along the line, these goals extended to increasing human capacity, and of course, to make us wealthier.

The uncompromising position demands fealty to the objective of avoiding the risk of ruin. The compromise is to accept some risk of ruin to accelerate the development process, which is necessary if the goal is to build the AGI killer app first.

Cost

The cost of scaling without regulation, without unwavering fundamental principles, and without ensuring the freedom of the human beings who will one day lack the intelligence to compete could be catastrophic.

Can we maintain our aspirations of scale and growth without the loss of our humanity?

_{1 If you believe the growth rate from 2021 of 6%, it’s roughly 12 years, if you believe the rate of 4% that prevailed for most of the recent decades, that’s a doubling period closer to 18 years.}

_{2 And you should…consider the time between the agricultural revolution and the industrial revolution, the time between the industrial revolution and the computing revolution, then the time between computing and AGI…}

_{3 Christianity seeks converts, Judaism tells Charlotte York in Sex and the City, “we don’t want any.”}

_{4 Rather than the folks with the kneydels. (Matzah balls, or soup dumplings, for the uninitiated.)}

_{5 The guy who invented utilitarianism.}

_{6 The proper term would be “non-ergodic systems.”}