They all stood to greet the newcomers. There were about a dozen or fifteen of them there. Obviously most of them already knew Wahram and Genette, and Swan was introduced to more people than she could remember.
When the introductions were done and they were all seated on the floor, Wahram turned to Swan. “Swan, we would like to have a conversation with you that does not include Pauline. We hope you will again agree to turn her off.”
Swan hesitated, but something in Wahram’s face-some strangled inarticulate entreaty, as one might see on the face of Toad when trying to convince Rat and Mole to join him in something he thought terribly important- caused her to say, “Yes, of course. Turn everything off, Pauline.” And after she heard the click Pauline used to announce her sleep, Swan pushed the button behind her ear for good measure.
“She’s off,” Swan said. She turned off Pauline all the time, but she didn’t like other people asking her to do it.
Inspector Genette jumped up and stood on the table before her; they were almost exactly at eye level. “We’re also wondering if we can make a check to be sure that Pauline is completely inactivated. Sometimes the human host cannot be sure. You’ll notice that I left Passepartout back in the city, for instance.”
“It could be recording you from a distance, no?” she said.
Genette looked doubtful. “I don’t think so, but it’s precisely to forestall any such eavesdropping that we are inside this confidential space. We have black-boxed ourselves. But we’d like to make sure the interior is clear by running some tests on you.”
“All right,” Swan said, as huffy as Pauline would have been. “Check her out, but I’m sure she’s asleep.”
“Sleeping people still hear things. We want her off. And indeed, may I recommend to you the advantages of keeping your qube separated from your own body.”
“Impolite people have often suggested that,” Swan replied.
The tests of Pauline’s level of activity were run by way of wands placed against her neck; then Swan was asked to briefly don a flexible mesh cap.
“All right,” Wahram said when one of his colleagues nodded confirmation. “We are alone here now, and this conversation is not being recorded. We must all agree to keep what is said here secret. Will you do that?” he said to Swan.
“I will,” Swan said.
“Good. Alex started these meetings, along with Jean here. She felt there were problems developing that should be discussed outside the realm of the system’s artificial intelligences. And one problem was a new kind of qube appearing on the scene. Inspector?”
Inspector Genette said to Swan, “You recall those supposed people on the Inner Mongolia? In a way they passed the Turing test, or the Swan test, I guess you could call it, in that you thought they were people putting on an act. People do that sometimes, and in many ways it’s a likelier explanation than the existence of a completely achieved humanoid.”
“I still think they were people,” Swan said. “Do you know any different?”
“Yes. Those were three of the humanoid qubes that we have discovered. There are about four hundred of them. Most of them act very much like people, and keep a low profile. A few act very strangely. The three you met were some of the strange ones. Another one made that break-in attempt at Wang’s station on Io. We recovered the remains of it from the lava, and the quantum dot framework was still detectable.”
Swan shook her head. “Those three I met seemed a little too foolish to be machines, if you see what I mean.”
“Maybe you’re just used to Pauline,” the inspector suggested.
Swan said, “But she’s often foolish. Nothing new there. Although I admit, she surprises me quite a lot. More than most people surprise me.”
“You are always saying otherwise to her,” Wahram noted with a curious glance.
“Yes. I like to tease her.”
Genette nodded. “But you programmed Pauline to have a bold character-a conversationalist, directed to respond to things at a slant. There’s some recursive programming in her that gives more weight to associative and metaphorical thinking than to logical if-thens.”
“Well, but that’s only part of it. Deduction is logical, supposedly, and she has a strong deduction program. But deduction turns out to be almost as metaphorical as free association. In the end it’s wild what she will say.”
Wahram said to the entire group, “The question of programming lies at the heart of today’s meeting. There’s clear evidence to suggest that some qubes are actively self-programming, in particular the ones involved with assembling these humanoids with qubes for brains. We don’t know that any humans asked them to do that, and we don’t know why they’re doing it. So-the first questions concern what they are, and who’s making them. We know that they can’t communicate internally with each other because of the decoherence issues. They’re not some kind of entangled group mind, in other words. But they can communicate just like we do, by talking with each other, using all the ways we ourselves communicate. But in their case, when they employ quantum encryption, it’s not possible to break their codes. Robin here”-who was the person on the other side of Wahram, and who nodded at Swan-“has been coordinating the recording of their conversations over radio and the cloud, and even in some direct vocal communications. While we can’t crack their codes, we can see that they’re talking.”
Swan said, “But go back a bit-how could they self-program? I’ve heard that recursive self-programming does nothing but speed up operations they already know.”
“Well, but if they were instructed to try to make something, for instance, then it might lead to some odd results. Pushing at ways to make something work could have initiated other ideas in them. It might be much like the way they play a game of chess. They’re given a task, which is to win, and they’re told to figure out ways to do it, and then, in their usual testing of all possible options, they might have had certain unexpected successes at modeling effective courses of action to get what they want. That wouldn’t be exactly a higher-order process, but it still might do the job, and lead to new algorithms. And that could then feed back into trying more things. At some point in trying to self-program for more effectiveness, they might have stumbled into consciousness, or something like it. Or the process might just have resulted in some strange new behaviors, even destructive behaviors. This, anyway, is the theory we’ve been pursuing.”
“Do the original qube programmers think this kind of process could go very far? I mean, wouldn’t the qubes still be stuck in algorithms?”
“As it turns out, the programmers who first built quantum computers used differing structures, and they ended up creating several different internal operating architectures. So really there are different kinds of qubes, each kind with different forms of cognition-different protocols, algorithms, neural networks. They have brain imitations of various sorts-aspects of what you might call self-awareness, and many other features of consciousness. They’re not simply one design, and in terms of their mentation, they may have started speciating.”
Inspector Genette took over: “We’re seeing clear signs of self-programming in the qubes. Where that may have led is hard to say. But we’re worried, because they don’t have the brain architecture and chemistry that makes us think the way we do. We think very emotionally. Our emotions are crucial to decision making, long-term thinking, memory creation-our overall sense of meaning. Without these abilities we wouldn’t be human. We wouldn’t be able to function as individuals in groups. And yet the qubes don’t have emotions, but are instead thinking by way of different architectures, protocols, physical methods. Thus they have mentalities that are not at all human, even if they are in some sense conscious. And we can’t even be sure that they resemble each other in the ways they’ve emerged into this new state. We don’t know if they think in math or in logic terms, or in a language like English or Chinese. Or if different qubes aren’t different in that way too.”
Swan nodded as she thought it over. If the silly girls had been qubes-the lawn bowler also-that was rather amazing, just in terms of morphology. As to mentation, none of this particularly surprised her. “I talk to Pauline about these issues all the time,” she told them. “But what’s clear to me from those conversations is how crippled the qubes are by these mental absences you speak of. Maybe it is the lack of emotions. There’s so much they can’t do.”
“So it has seemed,” Wahram said after a silence. “But now it looks like they may be generating goals for themselves. Maybe there are some pseudo-emotions there; we don’t know. Probably they still aren’t very wise- more like crickets than dogs. But, you know-we don’t know how our own minds work, in terms of creating the higher levels of consciousness. Since we can’t get inside the qubes to see what’s happening in them, we’re even
