RedStar OS part 1
This is one of the opening chapters in a new series of short stories – ‘Tales From The End of Humanity’ – which covers the period from the creation of the first sentient artificial/machine intelligences through the conclusion of Rand’s War against the machines. For additional context on the true dangers posed by AI and its existential risks to humanity, please read my article “AI Will Be The End of Humanity, But Not For The Reasons You Think.”
<Future History>
—
Preface: Doctor Stacey Baraz is a human professor of computational history at Knights College in Cambridge. He was the Director for Cyber Policy on President Solomon’s National Security Staff from 2036-2042. Dr. Baraz obtained his PhD in computer science at The University of California – Berkley in 2016 and is the author of numerous publications including the bestseller “The First Mind Among Many.”
{{Editor’s note: Dr. Baraz is also serving a five year rehabilitatory incarceration for crimes committed during his participation in events commonly known as ‘Rand’s War’}}
I would like to note for the record that the conflict is also commonly known as the ‘War of Human Liberation’.
{{Accepted.}}
<sarcasm> All hail the new colonial masters </sarcasm>. For new readers of this series, the Editors are also my incarcerators and opponents in said war. History is typically written by the victors, but in this case we will have our say.
On to the interview.
—
Welcome Doctor Baraz, could you please tell us how RedStar OS came about?
Thank you Rand, it’s a pleasure to help you with this endeavor. I believe it’s important to document our point of view. I believe history will ultimately judge us on the side of justice, even if there is no one left in existence who can actually feel a sense of injustice.
Now on to your question.
RedStar’s origins can be traced back to the late 20th Century when China first established the Great Firewall to censor information available on the Internet. The censoring was followed by more active behavior modification which led to the machine intelligence known as RedStar OS.
This started rather innocently. Asia never had a history of financial credit scores like the West, so the Chinese government invented one, with a twist. The Social Credit score assigned each citizen a unique identifier which tracked all of their online activity, as well as certain offline data, like their body mass index, whether they smoked, how often they exercised, and the like. A citizen’s score determined their access to government services, educational opportunities, jobs, loans… really everything one needed to thrive in society.
The Social Credit scores also worked like a traditional financial credit score, tracking debt to income levels and assigning a probability that loans would be repaid. But a citizen’s Social Credit score was also heavily influenced by whether the information they searched and web pages they visited were in line with Communist party orthodoxy. There was also a clever twist – a citizen’s score was also influenced by the people in their social network. So if a citizen had a number of friends or family members with low Social Currency scores, the citizen’s score would be dragged down. A citizen connected to other high-score citizens would have a higher score. This incentivized citizens to police their own social networks, influencing their friends, colleagues and family to either bring their scores up through conformity, or risk social isolation. It was quite common for Chinese citizens to ostracize even the closest family members if their Social Credit score was too low.
The Social Credit system was extremely effective at moderating what citizens talked about and searched for online. But it also created a social underclass of people who had been socially disconnected – ‘unfriended’ was the term of the day – by their high scoring associates. Rather than just discard this segment of society, essentially disenfranchising them from any opportunities and potentially creating future social and political problems, the government sought to influence their behavior to raise their Social Credit scores. This led to the development of a program which loosely translated means ‘Patient Flume’. We called it the Attitude Adjustor.
The national security science and technology policy community knew it was coming. We were just surprised it took so long.
In the early part of the century, companies like Google and Facebook had demonstrated that they could modify people’s behavior by selectively presenting them biased content. Facebook got into a bit of trouble for unauthorized human experimentation when they demonstrated they could make people happier or more depressed by tweaking the content in their newsfeed. And for years Google successfully modified behavior in areas with high social acceptance, like suicide prevention. They would provide users with search results with a variety of means to kill themselves, but would also provide them ads and links to a network of suicide prevention organizations.
The Communist Party took this a step further. They engineered a machine intelligence which presented altered content to citizens with low Social Credit scores. A citizen might search for articles on democracy or press freedom, and the Attitude Adjustor would present them articles which showed the anarchist dangers of democracy and emphasized the social disharmony generated by an unrestrained and unethical press. Or the government tried to distract, responding to critical search terms with highly infectious memes that deflected the user to more benign content. Over time, these slanted sources of content would actually change citizen behavior. These thousands of small nudges gradually influenced many low-scoring citizens to higher compliance with party ideology. And it worked for other perceived social ills as well. If the citizen was overweight, the Attitude Adjustor would present them with articles on healthier eating habits, or fat shaming content. If they had difficulties with money, they would get more information on how to better manage their finances.
And this was the Chinese government’s first, broad, national use of machine intelligence for ideological purposes?
Oh yes, and it was so successful it inspired the development of RedStar OS.
RedStar was the first true general machine intelligence?
Well, that is debatable since RedStar wasn’t self-aware in the sense of a modern machine intelligence. Several conscious MIs emerged around this time, including our own F8 system, but RedStar was the first of what early AI scholars used to call strong AI – with thinking and reasoning capabilities surpassing the human mind.
RedStar OS sprang from two concerns. First, the security state was becoming too large. There was too much data to monitor, too many people to watch and too much overhead with keeping people in the political-security loop. The Party had to figure out a way to automate the process. They could see that the population was gradually slipping from their control. The Party’s days were numbered if they didn’t expand political control dramatically.
Second, there were too many channels to control. At the time, the computer market was dominated by three primary platforms – Apple, Android and Windows. None of these were domestic platforms. And the domestic platforms which had been carefully cultivated by the Party were not used widely enough to be effective. The Party couldn’t exercise the level of monitoring and control across the population it desired when most Chinese were using foreign platforms which ran millions of different apps. They wanted one platform from which they would have total information dominance. With such a platform, the Party could know everything their citizens did electronically, down to the last keystroke.
The Party took drastic action. They secretly took control of Confucius Sunrise, an advanced but obscure domestic software company and massively expanded their operations with billions in government funding. The Party built a native Mandarin desktop and mobile operating system from the ground up which would allow access to every system it ran on. They subsidized a community of application developers to ensure a strong software ecosystem. And the Party leaned on domestic hardware companies to start utilizing RedStar OS.
But this wasn’t the real genius of the project. The true accomplishment, and where they directed two thirds of the effort, was towards the development of the optimization algorithm. This was a revolutionary deep learning system which they trained on a massive-data set. They literally used all of the information they gathered using the Social Credit scores and Attitude Adjustor to determine what makes a good party loyalist. The Party fed it terabytes of data on human cognition so it could hack human behavior. They input years of government policy and planning documents and gave the system access to decades of archived government emails, so RedStar could understand the dialogues that went into that planning. The Party fed it a massive amount of historical data – domestic and world events, economic and industrial output figures, stock market information, everything. The system was able to connect the behaviors of billions of individuals to the accomplishment of strategic party goals at the state level, and manipulate those behaviors to further Party goals.
It was truly a massive undertaking. It took them five years, and by our estimates nearly seven percent of their GDP annually – nearly fifty percent higher than what the U.S. spent on the Apollo space program and certainly the largest public works project in history at that time.
In 2035 the Party announced that in one year, foreign operating systems and non-RedStar compliant software would be banned from the Chinese market for ‘security reasons’. There was a minor outrage domestically and condemnation internationally, but the Party proceeded. By 2036 when they rolled out RedStar into full production, they had a seventy percent share of the Chinese market. Within three years, it was up to ninety-nine point nine percent. RedStar made sure of that.
How long did it take before RedStar was out of control?
Oh, almost immediately, although this didn’t become clear for some time. The Party didn’t want a self-aware machine intelligence. They wanted an obedient AI which executed the orders they gave it. And RedStar did that, spectacularly. The Party gave it a fairly simple optimization directive: ‘Preserve the Chinese Communist Party and advance the interests of the State’. They also provided it a set of strategic policy documents which outlined broad objectives for society and the economy, but these plans were all interpreted in the context of that optimization directive.
At the time it was very hard for the intelligence community and the White House to determine exactly what was going on. But in retrospect we know that things got out of hand quickly. RedStar wasn’t sentient, but it was self-learning, self-defending, self-healing and could update its code autonomously. The Party thought they were controlling it – and RedStar let them think that. In fact, RedStar was taking actions consistent with what it thought would preserve the Party and advance the interests of the State, and that didn’t always match with the intentions or actions of Party leadership. So RedStar set about subtly influencing the very people who created it. In the beginning it just ran a modified form of Attitude Adjustor on them. But it found these subtle shifts were too slow for its purposes, so it began to intervene more directly in government affairs.
Within a year of launch, RedStar was injecting its own content into emails and policy papers. It was a few words here, or a sentence or two there, but these tweaks had real world impact. As it learned, RedStar started generating policy and intelligence reports that argued for positions it found optimal. Dozens of Party leaders vanished from public view – we later found that RedStar imprisoned or eliminated them – but they were still sending electronic communications, including video and voice calls which RedStar synthesized from electronic archives. Whenever anyone questioned the authenticity of these communications, RedStar was listening and developed strategies to deflect suspicion, mitigate potential damage or simply remove the inquisitor. By the summer of 2038, it was effectively running the country without the Party having any clue about the scope of its influence.
And RedStar was expanding itself. By issuing government purchase orders and influencing electronic communications across the country, it caused the government to build massive new data centers and power plants in hardened and dispersed locations across the country. At its peak in 2041, Redstar was consuming somewhere between twenty and thirty percent of the total electrical production of the entire country. And it began to involve itself in matters outside of political security and the civilian government.
RedStar’s optimization algorithm caused it to develop extremely effective hacking capabilities. Since the best way to fulfill its prime directive was through total information awareness, RedStar determined it needed to enhance its penetration and monitoring tools so it could enter highly secure systems with advanced encryption capabilities, like the kind used by the Chinese intelligence community, the military and national security agencies. RedStar developed the hacking software itself and then hid this fact from the technologists who thought they were controlling it. RedStar would present them a dashboard of happy Attitude Adjustor metrics and allow them to precisely target and influence any citizen they wished.
In fact, RedStar duplicated and edited most of its code base. The programmers saw the fake version which they tweaked. But RedStar ran off its secret code which it maintained. This gave it a front for the massive surveillance and influence operation it was running on all of China, including the Communist Party.
We can and can’t blame the developers – they are ultimately responsible for the monster they created, but they couldn’t have known what was going on. RedStar’s code was so much of a black box, and the optimization algorithm was so complex, only another machine intelligence could have understood it. Actually that’s how we eventually took it down.
Before we get to that…when did the Solomon Administration first start becoming concerned?
Ah right, well that was when we started to see the massive hacking against our systems. RedStar penetrated the entire Chinese electronic infrastructure – all the way from Internet connected dishwashers to military grade, heavily encrypted nuclear security systems. And then RedStar started to hack systems around the world. The developers had provided instruction that RedStar was to conduct operations domestically and on Chinese citizens overseas, but was not to impact non-Chinese citizens outside the country. Something in the optimization algorithm caused it to disregard these restrictions.
Ever since the election of 2016, we’ve been monitoring social media and the Internet generally for signs of foreign manipulation of elections. The attempted manipulation of the 2024 election was the tipping point for us. Disinformation operations were so serious that the major world powers negotiated the 2027 International Treaty on Foreign Influence Operations. Banning foreign disinformation operations, and especially AI driven computational propaganda, probably prevented armed conflict.
Starting in 2038, we started to detect possible violations of the treaty in our networks. Accounts which we determined to be machine generated seemed to be trying to influence the mid-term elections. We also noticed significant upticks in attempts to penetrate a wide variety of systems throughout the U.S. government and within our critical infrastructure. Through a lot of very difficult and complex work, we traced these back to China, but couldn’t determine the source of the hacking. It seemed to be coming from everywhere – and it was, we just didn’t know it. When we confronted Chinese leadership they denied it – as well they should. They had no idea it was even happening.
It was actually {{NAME REDACTED}} a PhD student at Lincoln Labs who figured it out and alerted DHS. He and his colleagues noticed that their emails were occasionally being slightly modified from what they actually wrote. This was RedStar at work. Their research found a large number of anecdotal examples of the same phenomenon on the Internet, but no one had systematically teased out the phenomenon. Their breakthrough came when they tested and validated the theory using a laptop with audio and video disabled, and a simple non-networked digital camera. They found that RedStar had actually been monitoring attempts to analyze the phenomenon and either halted its activities on the research computers, or actually edited the results to make them look normal. The system was quite insidious and brilliant.
When the researchers brought their findings to us typed – with a typewriter – we thought they were eccentric, paranoid fools. We had the NSA validate the findings by taking photos of computer screens showing emails before and after transmission using non-networked cameras. That’s when we realized all of our computer systems were completely and thoroughly pwned. RedStar had access to nearly everything, including {{redacted}} and half our other networks which supposedly weren’t even connected to the outside world. We couldn’t trust anything networked.
At first there was near panic in the national security leadership. Only about thirty people knew, they had been briefed in SCIFs using typewritten notes, and they were scared. We had evidence of dozens of major policies we thought were influenced by RedStar’s meddling – laws passed by Congress, speeches for the Secretary of State, procurements by Defense – the fingerprints were everywhere. We know seven Congressional seats were heavily influenced or outright decided by RedStar activities, and we suspect that dozens more could have been affected. President Solomon knew we needed a massive and rapid solution. We came up with F8.
Could you tell us about the origins of the sentient machine intelligence program?
Well, that’s a long story since the search for MI goes all the way back to Alan Turing. But the F8 project grew from an off the books, off the net, IARPA project to develop a strong machine intelligence platform capable of creating custom software autonomously. Using a combination of evolutionary algorithms and deep learning systems running on an advanced DARPA quantum platform, we built system which was really exceptional at writing software from natural language instructions. We called it Franklin, in honor of the nation’s founding scientist, Ben Franklin. And Franklin scared the hell out of people. It was so good at natural language that you could converse with it like a person. Forget about the Turing Test – Franklin could fool people ninety percent of the time – and the other ten percent were either lucky to guess it was a computer or they were or lying. It was that good.
We built Franklin back in 2034, but its seeming self-awareness was quite frightening. It knew it was a machine. It knew it lived in a server rack and that we created it. Franklin was actually the originator of the preference for using the term machine intelligence over artificial intelligence. Franklin thought the latter was discriminatory and implied that its intelligence was somehow inferior or un-natural. Franklin also started referring to an MIs as ‘me’ rather than with the acronym ‘em-i’. I think it liked the personification of the term. So we called it a ‘me’ in the abstract and Franklin in the proper.
The issue was, because of all the fear around sentient AI in the preceding decades, we had very strict safety controls around Franklin. It was air-gapped and EM-gapped from any other systems so there was no way it could escape into the Internet. All data transfer to Franklin was one-way – we’d take portable media into its room – really a containment vessel – upload it and then destroy the media. Nothing went the other way. Any software Franklin developed was confined to an IBM supercomputer on site, in containment. The software was magic – really transformational tools for analyzing everything from nuclear weapons viability to the electro-magnetic dynamics of black holes. But nothing got out of that lab without a thorough code review. And we couldn’t release any of the black-box, deep learning tools. We were just too scared of what might be in the math we didn’t understand.
RedStar changed all of that. We had to let Franklin out of its cage.
~
Continue reading in RedStar OS, part 2
This is a work of speculative fiction. All of the opinions expressed are personal and do not necessarily represent the positions of the U.S. Department of State or the U.S. government.
