The fundamental problem of economics is balancing unlimited wants with limited resources. However, the advent of unlimited resources could begin in just a few years. If every human has access to a genie that grants unlimited wishes, what will we expect folks to wish for?

I define three pure archetypes of aligned superintelligence:

  • God: Rules over humanity forever for its benefit.
  • Genie: Follows humans' instructions within some set of guardrails.
  • Guardian: Only averts threats to humanity's existence.

This post is about the genie archetype.

Unlimited wants

We say that humans have unlimited wants, but the genie superintelligence will really put that idea to the test.

Descent into wireheading or into your own world

What can we expect people to ask for, and in what order?

  1. Immortality - immunity to aging, disease, and physical damage (as much as possible).
  2. Material goods - A mansion, a cruise ship, any kinds of foods or gadgets you want.
  3. Wireheading - Many people will probably permanently take themselves out of the game by asking for stronger and stronger drugs, eventually ending up in a state equivalent to getting their pleasure center electrically stimulated for eternity.
  4. Human-looking robots - Those who avoid the wireheading trap will probably ask the genie to create artificial human-looking robots. Humans are social creatures but also crave control and avoiding being hurt by others. A perfect girlfriend/boyfriend, perhaps a perfect family or friend circle, maybe even an entire community tailored to your desires.
  5. Artificial worlds - These could be physical or digital worlds. It'd be like playing your perfect never-ending videogame as a main character. You decide the genre, the story, and the bounds within which you're willing to be surprised. I believe everyone who avoids wireheading will spend most of their time in an artificial world instead.

To summarize, there are three main ways humans will end up in a world where a genie superintelligence is available to everyone:

  • Voluntary death
  • Wireheaded
  • Your own world

Will you stay human if you can do anything to anyone?

Unlike in the real world, in your artificial world, there really won't be any obstacles to descending deeper and deeper into your darkest and most depraved fantasies. Giving in to temptation once over an infinite lifetime is all it takes to start the downward spiral. Without any checks or restraints, with the freedom to do anything you want, no matter how awful, to the completely human-looking 'NPCs' around you, do you think you'll be a more moral and kind person in a thousand years? In a million years? You may technically remain a human, but how long will your humanity survive?

Even the part about 'technically' remaining human is suspect. The genie would be able to turn you into a man or a woman or a catgirl or a dragon or a sentient spaceship or a Lovecraftian cosmic horror. How many are likely to live out their infinities in human bodies?

Genie's guardrails

If any humans are still alive with agency over their lives a year after the genie superintelligence is turned on, then the genie was exceptionally well-aligned and has all kinds of great guardrails built in around not hurting humans. Beyond that, the most impactful guardrails are around creating sentient life.

Will the genie superintelligence be able to follow orders like "clone me"? Or to take all the eggs and all the sperm of a couple and create a thousand or a million of their children, with all genetic defects repaired?

Will the genie be allowed to create sentient lifeforms that are technically not human and not authorized to use it themselves? Will the genie's no-harm guardrails extend to those sentient lifeforms? If it can create sentient life forms that it itself considers subhuman, some would definitely use this functionality to create sentient slaves.

If the genie is not allowed to directly create humans or other sentient life, the future could be dominated by humans with the genes of women who derive a great deal of joy and meaning from being pregnant and giving birth.

Intergalactic Expansion

Human anxieties

Eternity is a very long time. There is a mind-staggering amount of resources in the galaxy, but it's still constant. The growth of the human population with a genie superintelligence, on the other hand, will be at least exponential. Eventually the ability of humans in the Solar system to even stay alive will be constrained by the constant amount of energy produced by the Sun.

Those who foresee the impending energy catastrophe will want to not be in the solar system fighting over the Sun's energy with trillions of other humans. They may not even want to stay in the Milky Way galaxy in case the genie superintelligence will decide to start requisitioning power from other stars to power the needs of the humans in the Solar System.

Leaving the Milky Way

See Isaac Arthur's episodes Shkadov Thrusters and Fleet of Stars. The rational approach is to not wait for the genie superintelligence of Sol to go crazy from trying to keep trillions of people alive and satisfied with a limited amount of energy.

Instead, you ask the genie superintelligence to make you a spaceship and head towards the galactic rim, looking for the first unoccupied star. There are a hundred billion stars in the Milky Way, so it should be possible to claim one.

When there, you ask the copy of the genie superintelligence that traveled with you to turn that star into a Shkadov thruster and set sail towards another galaxy. This allows you to spend your eternity controlling the entire energy output of a star while getting farther and farther away from whatever craziness is going on in the Milky Way.

Of course, even going this far may not do anything to ensure a safe eternity. To quote Isaiah 30:16:

But ye said, No; for we will flee upon horses; therefore shall ye flee: and, We will ride upon the swift; therefore shall they that pursue you be swift.

No path to survival of humanity's values

If the 'genie' superintelligence archetype prevails, we can expect humans to eventually wirehead themselves or retreat into their own perfect worlds, living out both their best and most depraved fantasies. In such a future, it's difficult to see what will remain of modern humanity's morals and values even though humans will technically survive.

If people also reproduce at an exponential rate, there's still the danger of them running out of energy to survive. Even a genie superintelligence with access to all of the Sun's energy may have to start taking resources from nearby stars to sustain them, even if it means violating the "property rights" of other humans living there. Still, this won't be a human-human conflict that might result in some re-emergence of human values, but rather an internal conflict within the genie superintelligence.

No matter the outcome, I don't see how anything that we modern humans value and find meaningful will continue to have value and meaning in a future of a superintelligent genie AI.

Author Of article : Aram Panasenco Read full article