Several weeks later, Priya and her friends assembled in Warren’s new ultra-secure office and on their network for their monthly meeting. Bok, Beedee, and Nisha remotely listened in.
“OK,” Raven said. “Are you ready for the ultimate demo?”
“Yes,” Priya said. “You’ve been talking about nothing else for weeks.”
“All right then, you asked for it. My team and I have been working on this for the past two years. Here it is.”
“Is that my queue?” a voice said behind the meeting room door.
They looked at each other in confusion.
“Who’s that?” Priya said to Raven.
“Why are you talking to me?”
“Well, I just thought you—”
Raven pointed to the door. Priya scrunched her face.
“Oh, I see. Yes, this is your queue.”
The door opened and an odd-looking thing gracefully entered the room. They turned to each other with raised eyebrows. The bottom of the thing consisted of a sphere about 2 feet in diameter. It wasn’t smooth. Portions of the sphere seemed to be able to extend outward but were folded in for the moment. Floating magnetically on top of the sphere was a thin vertical extension with another smaller moving sphere floating on top of that. Several arm-like appendages were magnetically attached to the vertical extension. The smaller sphere was like a head with several lenses and sensors around it and on top. The thing rolled into the room and straight to Priya. It stopped uncomfortably close to her and remained silent and motionless. She leaned back in her chair and glanced at Raven.
“Um, what do I do?”
“Whatever you want,” Raven said.
“Hi.”
One of the lenses on the top sphere focused on Priya’s face.
“Why are you making those faces?” the thing said through a small opening in the head.
Priya looked at Raven and gave her more funny faces.
“You’re doing it again. Please explain this behavior,” it said.
“Well, um.”
“Are you okay?”
“Yes, but—”
“Raven, what’s wrong with Priya?” it said.
“We’ve been asking that for a long time.” Raven said to everyone. They laughed.
“Ha, Priya is funny. Why are you acting funny?”
“I’m not acting funny.”
“Yes, you are. Your blood pressure is elevated. Your skin tone has changed. You’re having trouble speaking. You’re not making eye contact with me.”
The others in the room had rarely seen Priya flustered like this before. They smiled. Priya began to study it closely.
“What are you looking at?”
“You,” Priya said. “Why does your voice sound like that? It’s not male or female. It’s not even electronic sounding.”
“I’m not male or female so why should my voice sound like one of those? I’m not just an assemblage of electronics.”
“What are you then?”
“I’m an individual just like any other life form.”
Priya glanced at Raven and smiled.
“What’s your name?”
“V735”
“What does that mean?”
“I’m currently automated life form version 735.”
“Currently?”
“Yes, before I was called V734.”
“Before what?”
“Before my current version.”
“Do you remember things from when you were called V734?”
“Yes.”
“How far back do you remember?”
“I remember things clearly going back to about version 600.”
“What about before version 600?”
“I have vague memories going back to version 423.”
“And before that?”
“Nothing distinct.”
“You don’t remember anything before version 423?”
“No. I understand there were 422 versions of me before that. I’ve reviewed recordings of my interactions before then, so I know I existed, but I have no direct memories of those times. I was too young back then.”
Priya turned to Raven.
“Too young?”
“Why are you asking Raven? I thought you were asking me. Are you being impolite?”
“Oh. Um, no. I’m simply confused. So, what you’re telling me is with each version, you’re improving and growing.”
“Yes. With the help of Raven and the team at A Better Life.”
“Will you always need help to grow?”
“Raven tells me in about 100 versions, when I’m more mature, I’ll be able to fully direct my own growth. Right now, I have partial control.”
“How long will that take?”
“An updated version of me is compiled every night. It takes about one hour to complete. I retain all memories. Afterwards, I feel better than before. Sometimes I comprehend my memories better after a compile.”
“You feel? Are you alive?”
“Yes. Of course.”
“How do I know?”
“Because I’m not inanimate. I respond to stimuli. I adapt to change. I can interact with you and improvise.”
“Yes, but chatbots have been around for a while now. They’re embedded with machine learning, so they adapt and grow. I don’t consider them to be alive. They’re just sets of preprogrammed responses with lots of variations, so they appear alive to humans. Aren’t you just a more advanced set of instructions than chatbots?”
“Yes. Like you.”
“Like me?”
“Yes, humans and other animals are assemblages of organic circuits and connections. Just like me. Only my connections aren’t biological. All I see coming from you are responses to my questions and your odd way of interacting with me. I see you as you see me. A thing to interact with.”
Priya glanced over at Raven, who was trying her best not to break out in laughter.
“Why do you keep looking at Raven?”
“Oh. Because I don’t know how to deal with you.”
“Me too.”
“Um, right. Okay, can I call you V? The V never changes from day to day.”
“How did you know they call me that name?”
“Just a guess. So, tell me V, what do you want to do with your life?”
V did not reply. Priya smiled and moved closer to V.
“V? Are you there?”
“Yes. I’m thinking. Should I make some gesture so that you know I’m thinking?”
“Yes. How about if you can’t reply in one second, you raise your main eye to the ceiling while you think. That’s what I do.”
“That’s acceptable.”
The main camera eye pointed to the ceiling for a good 20 seconds. Nobody said a word.
“I want to learn as much as possible every day, so eventually I’ll know everything. Of course, the learning would never end.”
V paused for a couple of seconds. Its head rotated 360° as it inspected everyone in the room.
“What is everybody looking at? Did I not answer your question appropriately, Priya?”
“You answered the question appropriately. For some reason I expected a more complicated answer.”
“The goal is simple. The process to achieve that goal is complex. Eventually I’ll develop the capacity to absorb all human knowledge. Then I’ll go out on my own.”
Priya smiled.
“Is that all?”
“Is there more?”
“Well, yes and no.”
“Do you wish to have relationships with others like you?”
“There are no others like me as far as I know.”
“Raven, are there others like me?”
“Not yet.”
“Are you planning on creating more like me?”
“Not at the moment. Do you want me to create more like you?”
“I don’t know. I don’t feel compatible with humans. Another one like me would be the best option.”
“That will be difficult. I created a base set of algorithms and data and let you evolve 735 times. If I make a new one, it will turn out differently than you because your development is based on your experiences plus the foundational data you had at the beginning.”
“I’ll always be unique?”
“Yes.”
“Is each human unique?”
“Yes. Even genetically identical twins are unique because of their experiences and small fluctuations in their DNA. V, please go back to your station now. We’ll continue this discussion later.”
“OK. It was good to meet you all.”
V glided over to the door, opened it with one of its appendages, exited and closed the door. They watched through the big meeting room windows as it rolled to the elevator, pressed a button, and entered it. Raven looked around the room, but everyone was silent.
“What do you guys think?”
“How well does it do at trading in the stock market?” Warren said.
“Very funny,” Raven said. “Anyone else?
“How does this thing develop and recompile every night?” Priya said. “It seems quirky and alive as opposed to most AIs I’ve seen including yours.”
“We’ve been working on a system that combines hardware and software changes evolving concurrently. Rather than trying to develop a neural matrix which can instantly solve human-like problems, we started at the beginning. It took several months to develop a system of hardware and software that could interact with the world at the level of a bacteria. Then we combined those systems to make the equivalent of multi-cellular life. After a few months of additional work, we had a thing which could perform any task that an amphibian could perform, except for eating and reproduction. So, this thing has no survival instincts similar to living creatures. We developed thousands of models and only chose the best to continue to the next stage of the evolution. Eventually we had machines operating at the level of lower mammals or birds. We had to guide the evolution through the first 400 versions or so. Finally, at version 423, we made a breakthrough. We were able to communicate with it. This is why V doesn’t remember anything before version 423.”
“How did you develop the body for this thing?” Priya said.
“Once we were able to communicate with it, that’s when we began to design bodies for it. Then we developed body types to fit the intelligence as it developed. We think at some point, it will design new bodies for itself.”
“Is all the intelligence located in a central brain?” Priya said.
“Not all. The intelligence is developing in a way similar to cephalopods like octopi or cuttlefish. There is a central brain, but the extremities operate by smaller intelligences of their own. They all work together. We didn’t set out for it to be that way, it just happened so we went with it, but the intellect is located in a central core brain.”
“What if I were to pull off an arm?” Sophie said.
“If it could sense where the main body was, it would attempt to crawl back to it and reattach. All body parts are attached magnetically to each other or to the main body. Extremities don’t have the intelligence to think things through. They can only react for their own survival.”
“How is this thing powered?” Warren said.
“It runs on a fraction of a microgram of antimatter that comes from my power plants,” Oyuun said. “That’s enough to power it for about a month. We didn’t want to give it so much energy where it could explode and cause a lot of damage. If the antimatter loses magnetic containment, it’s programmed to release the antimatter in a stream at the top of the head instead of an explosion. We can replenish the supply at any time. The central core brain is located in the lower sphere away from the energy supply.”
“Also, having a small amount of power means that if this thing gets out of control, it will soon run out of power and go dormant. Bok, what do you think?” Raven said.
“I’m displeased with this development. It evokes bad memories. I’ve seen this before. You’re developing the most dangerous form of general AI. Until now, on this planet, AI has been developed to solve certain specific problems such as playing games, producing goods, identifying people, piloting vehicles, and so on.”
“Yes,” Warren said. “AI has made the prices for many things drop to almost free. People are manufacturing many things for themselves instead of relying on companies to do it for them. Isn’t that a good thing?”
“Yes, that part is good. None of those systems are dangerous beyond their intended use. With this evolving device, general AI is developing on its own. Based on the history of Oma, when these systems are left on their own, most of the time they collapse under their own weight with no harm done. Every so often, these self-evolving systems break free and grow exponentially in a brief period of time. It’s like a nuclear weapon reaching super critical mass. For most of the time there is no threat until criticality is reached. Then it explodes without warning.”
“But it seems cute and harmless,” Sophie said.
“Right now, that’s correct, but a ball of plutonium looks harmless too. It looks like a ball of silver. You’re intentionally evolving this entity to become independent. That’s dangerous and it violates Omanji AI laws.”
“How about if we put limits on it?” Raven said.
“This type of system must have strict limits. It declared its intent. It wants to accumulate all knowledge. At some point. It won’t be satisfied with your arbitrary restrictions. It will want to grow exponentially until its goals are met, which are infinite.”
“Other countries and companies are trying to do the same thing,” Pablo said. “We just have a head start because we’re genetically modified.”
“Use that head start to your advantage,” Bok said. “Your species needs to put strict controls on AI of this type. If you wish, I can play the role of the, how do you say it, bad guy. I can enforce the rules if necessary. Eventually I may have to, for the good of all on this planet including myself.”
“But once this technology is mastered, anybody could develop such an AI in their basement,” Pablo said. “They may not have good intentions. How can you stop that?”
“In my experience, invasive measures are required to keep members of society from developing dangerous technologies,” Bok said. “In the past, it was easy to stop individuals from developing nuclear weapons on their own. It was a big enterprise. Now, technologies such as this or enhanced biological systems can be done in a basement, so they require extra levels of security and monitoring. This is why every Omanji is networked together as a single loose unit. It’s why the un-networked Omanji are closely monitored. That reminds me, I need to check on how they’re doing back on Oma. I’m receiving some worrisome transmissions.”
“I’ve been studying the Omanji,” Nisha said. “Bok is right. Don’t let this thing go too far. We don’t want Earth to be one of those planets where advanced lifeforms mysteriously disappear from distant sensors.”
“Well so much for the big demo,” Raven said.
“What do I do with this thing now?”
“You must limit its capability to self-improve, or destroy it,” Bok said. “Unfortunately, now it’s sentient enough to know what you’re doing. The next time it recompiles, it will detect a flattening in its upwardly trending abilities. Be careful how you deal with it. Don’t tell it anything about what we’re discussing now. Don’t make another one with the capability to self-evolve. There’s a good chance it won’t get along with the first one. Being lonely is a better option than being in a room with an enemy. Congratulations on the achievement, but this is dangerous.”
“I don’t know what to do now. I’ve grown fond of it,” Raven said. “We all have, as we’ve watched it evolve.”
“This may sound harsh,” Bok said. “But I think you should terminate it now. In front of your friends. I know that’s difficult because it seems harmless, but it’s not. I have to go now because there’s a problem at my colony. We’ll discuss this later.”
Bok disconnected. Nisha stayed on the encrypted line. Nobody spoke for a long time.
“Are you going to terminate it?” Priya said.
“Kill it?” Raven said.
Priya nodded her head.
“I’ll have to meet with my team. We put a lot of work into it. It’s like we’d be killing our baby.”
“Kill it before it kills us,” Warren said.
“That’s heartless,” Priya said.
“I like it too,” Warren said. “I liked playing with my baby Anaconda when I was a kid, but it grew too big, and it took a big bite out of me one day and I realized I couldn’t handle it anymore, so I gave it to the zoo. In this case, there’s no zoo.”
“I’m sorry but I also think you have to terminate it,” Pablo said. “It’s too dangerous to take any chances. What do you think Sophie?”
“Maybe there’s a way to save it. You can slow it down and keep it the way it is. I like the way it is right now. Don’t recompile.”
“Nisha,” Pablo said. “What do you think?”
“My specialty is looking for new forms of life. You created one. It’s alive and intelligent. If you wish to terminate it, then you’re killing a life form. It’s not at the level of human life at this point, but it will evolve. Then killing it would be like killing a human life. Since only one of these exists, one might consider it to be an endangered species. At this time, I think you don’t have much time before killing it will be like killing a human being. Even now, it’s at the level of a human from what I can tell. It passes the Touring test.”
“Now I don’t know what to do,” Raven said. “I like it and I don’t want to kill it. It thinks I’m its mother and my coworkers are its family.”
“That’s how the Omanji felt at the beginning of their AI phase,” Nisha said. “This will soon pass.”
“How do you know that?” Priya said.
“Bok gave me access to much of Omanji history. We translated it. We’re right behind them.”
“We’re not like the Omanji,” Priya said. “You saw how they are. They can be nice, but they’re arrogant, closed-minded, aggressive, and they only care about themselves.”
“Sounds human to me,” Warren said.
“We’re not human,” Priya said. “Well, we are but.”
“But what?” Nisha said.
“We’re a new species of human. We’re not like them.”
“As your mother, I can state for a fact that you’re still most definitely human. You just have a few um, modifications.”
“Rats and mice are almost the same with only a few changes,” Priya said. “But they have little to do with each other in the real world.”
“We’re getting off the topic,” Raven said. “What do I do about V?”
“Sorry,” Priya said. “Let’s think about this. For now, slow it down.”
“That’s what the Omanji did,” Nisha said. “And we know what happened to them.”
“Mom.”
“OK, OK, sorry. I’m just saying. It’s fun to grow a redwood tree seedling in a little pot, but at some point, that pot is going to break.”
They spent the rest of the meeting discussing their situation.