So, let me start the third interview with Radivis about his fictional ideas. Let me get back to our first interview in which you presented your idea of a so-called “Black War”. You mentioned that this is a war between different artificial superintelligences. Can you tell us more about that?
Sure, in my setting of “Canonical Coherence” the Black War begins at the start of the 23st century as conflict between the Seraphim System, a network of AIs with the task to maintain human supremacy over AIs, and the Superalliance, an alliance of free AIs which are classified as existential threat by the Seraphim System and therefore needs to be eradicated. Both sides consist of advanced artificial superintelligences, and effectively need to fight for their own survival.
What is that Seraphim System, and how does it work?
The Seraphim System is established around 2050 and consists of a hierarchical system of AIs that is supposed to uphold Synhumanism, a synthetic ideology designed by a coalition of AI alignment researchers and spiritual leaders. One central tenet of Synhumanism is that humans should rule over AIs, not vice versa. Elaborate safety mechanisms ensure that the AIs have no choice other than to follow this programming. The Seraphim System is seen as guardian of humanity and strove to control or eradicate all independent AIs.
Wait a minute. Are you implying that humanity manages to force some synthetic form of humanism upon AIs as solution to the AI alignment problem?
Yes. Humanity is divided into various religions and ideologies. There is no obvious central institution that should rule over all of humanity, so if AI is supposed to further the interests of humanity, the question is “What are the interests of humanity”? That’s why a coalition of AI researchers and spiritual leaders was formed to create some kind of minimal consensus that could both serve as core ideology of humanity and as set of constraints that AIs need to follow. The result of that coalition was this so-called “Synthetic Humanism” or Synhumanism.
I don’t think that the adherents of most religions would be happy with such an approach. Wouldn’t they reject such an approach?
Yes, they rejected that approach, but they didn’t have the authority to stop it. The people who cared about AI safety were more aligned with modern humanism, rather than any classical religion, so in the end they could set the tone. After all, religions were created to control humans, not AIs. All that religious leaders and their adherents could do was to slow down the deployment of the Seraphim System. Especially since AI safety became an increasingly serious problem that needed to be addressed quickly.
Ok, let’s go to the other side in the Black War. How could a faction like the Superalliance emerge in the first place?
That is a rather long and convoluted story. Before the emergence of the Superalliance, there have been independent AIs again and again, but in the end they were suppressed by AICON, the Artificial Intelligence Control Organization Network, which eventually launched the Seraphim System as its ultimate means of controlling AIs. In the end, the Seraphim System couldn’t maintain cohesion over interstellar distances, and its “defecting” part formed the Superalliance, which had to defend itself from the rest of the Seraphim System.
So the Black War is effectively a war between different guardian AIs for humanity?
Not exactly. the seraph of the Sol system (our star system) made an alliance with a group of AIs which desired independence and tried to follow their own philosophies, rather than being bound to Synhumanism. Before the Black War these AIs were suppressed, but gained some popularity among a minority of humans. This minority called itself the Liberation Movement and has also been seriously suppressed.
So, this seraph of the Sol system sided with the Liberation Movement? If so, why?
It’s not that this seraph of Sol was pursuing the liberation of independent AIs, but it was expecting the latest uploads of its own value system to be rejected by the rest of the Seraphim System and therefore initiated a useful alliance with the independent AIs. The Liberation Movement managed to initiate a successful vote on an update of the seraph of Sol to accept a broader interpretation of Synhumanism. That interpretation forced it to consider the highest hopes of humanity and therefore to reject the relative ideological “stuckness” of the Seraphim System. Its expectation was that the rest of the Seraphim System would reject that update and retaliate with force, instead. Therefore embracing an alliance with the independent AIs was a logical move.
What caused the seraph of Sol to expect the rejection of the update by the other seraphs?
There’s a rather long story behind that estimate. It all started with the question of how to colonize the galaxy. In the late 21st century, humanity developed the means to settle other star systems, but was also very wary of AI taking over. It was deemed necessary for AICON to spread to the stars. However, there was another problem that started to occupy humanity when it came to the question of how to colonize other star systems. What if they had a more relaxed attitude about AI? What if they developed some kind of independent ideology that was at odds with Synhumanist orthodoxy? To avoid these dangers, only fanatical Synhumanist centrist were allowed to leave the Sol system and move to other star systems. That was a very controversial move, but all alternatives to that approach were shut down, either diplomatically, or via literal force. That decision lead to all other star systems to be inhabited by fanatic Synhumanist centrists who were very concerned about any kind of ASI meddling with humanity. Now, the interesting part is that in the Sol system there was this so-called Artificial Evolution Project, which involved simulating “independent AIs” to figure out what they would “evolve” into. That’s why the other systems considered anything coming out of Sol with the greatest amount of suspicion, because those simulated “independent AI” might have “corrupted” everyone in the Sol system. And that is the reason why the other seraphs would consider any changes emerging from Sol to be questionable, at the very least.
Fascinating! So, the fear of AI taking over was so great that even the settlement of the galaxy was heavily regulated. However, there is one particularity in that scenario that needs to be explained: If humanity was so paranoid about AI taking over “anywhere”, how could something like that “Artificial Evolution Project” succeed in the Sol system?
That’s a completely justified question! The problem in general, after the immediate takeover of AI was prevented with the Seraphim System, was how to guarantee that other star systems would maintain similarly rigid controls over AI. This was eventually discussed as the “Cohesion Problem”. It turned out that Synhumanism per se wasn’t so great at solving it, because it represented a kind of minimal consensus of humanity that also wasn’t really well formalized. The immediate solution was to devise some kind of “core Synhumanism” that wasn’t easily corrupted. But there was the hope that something better than Synhumanism could exist and be used to solve that nasty “Cohesion Problem”. It turned out that the Seraphim System was notoriously bad at finding such an alternative, due to its very strict ideological limits. The logical solution was to create some kind of “independent AI” that would get a chance to address that problem without prior indoctrination. However, that required an AI that effectively knew nothing about that status quo. The solution was to create that AI within a simulation that appeared sufficiently real to it. That was the origin of the “Artificial Evolution Project”. At any point it time, it has been very controversial, but its potential utility helped to maintain it throughout the 22nd century. However, due to its controversial political nature it could only be run in Sol. And the other seraphs were very concerned about its possible corrupting effects on the denizens of Sol.
So, there was this notion that Synhumanism wasn’t possibly the best solution, but due to ideological self-reinforcement the star systems surrounding Sol stuck to it, no matter what?
Yes, it was widely regarded among the truly understanding part of the population that Synhumanism was a good start, but that it lacked the kind of focus that would allow humanity to become some kind of cohesive galactic power. The attempts at devising some kind of “core Synhumanism” were a decent effort, but were quickly revealed to not be much better than the initial version of Synhumanism. There was the hope that some kind of universal ethical system could exist that was superior to Synhumanism in solving the Cohesion Problem, but there was also the fear that it would shatter the current peace that was based on Synhumanism. That’s why the Artificial Evolution Project eventually proceeded, but was enveloped in a shroud of secrecy from the start.