Air Force One, Conclusion (2) "If it looks like you’ve left anyone at the Wolverine, then you can communicate with them before we arrive. They will allow my wife and the truckers to disarm them, tie them up, and leave them in the parking lot where they’ll be visible when we fly over. My wife and her guests will then depart the truck stop, unobserved by you and your men. "Then we’ll all fly around for a few days, and you can talk to me all you want. You can tell me all about yourself, Colonel. Like who the fuck you really are. Why you killed Agent Page and his guys. Why you have a force field. Little things like that. And most of all, why exactly I should let you have Air Force One. That would be the major topic, I guess." "And you know what?" Mick asks brightly. "I hope you convince me — I really do! And if you do, I will cheerfully turn over to you the control of this aircraft, together with all data it contains, along with all that means." "And, what will happen if I should fail to persuade you?" Tennen asks. "Oh, be positive!" Mick replies. "I’m sure everything’s going to work out just fine!" "Anyway," he concludes, "that’s the deal. I’ll check in in a few minutes to make sure the jamming has stopped. Bye-bye! Call anytime!" Mick jabs the screen icon that breaks the connection. He stands looking at it for a few seconds, then turns suddenly toward the two desiccated corpses, surprised to realize that he had completely forgotten them — in spite of the fact that the copilot’s body is within arm’s reach. "I guess I’m getting used to you guys," he says to them. At the other end of the wide control console, the pilot’s arms move slightly as his yoke makes an adjustment to the great aircraft’s course. ———————————————————————————— Hardware ———————————————————————————— "I was able to successfully monitor your communications even while my transmissions were blocked. I believe that I find myself — troubled," the Bolo says. Mick reclines in the copilot’s seat and glances at the corpse that he has displaced, now sitting against the wall, its legs stretched out in front of it. He blows out his breath and lights a cigarette. "Yeah? And why is that?" he asks. Uncharacteristically, Johnny pauses. "I am aware," he says at last — after a hiatus of perhaps eight-tenths of a second, "that I am not well equipped to predict the workings of a human mind. However, after substantial and careful consideration I feel quite confident in predicting that, in spite of your words to the contrary with Colonel Tennen, you are not, in fact, prepared to surrender control of the aircraft under any circumstances." Mick exhales smoke and raises his eyebrows. "Excellent thinking! That is correct." "Thank you," the Bolo replies and pauses again for several seconds. Mick imagines him retroactively reinforcing everything from maximum-abstraction algorithm strategy templates to probability weightings on individual script nodes. Exactly like a man congratulating himself. And, like a man, when he’s finished patting himself on the back he notices a little problem. "I am pleased that I have correctly predicted your thinking," the tank says hesitantly, "however this again raises the troubling thoughts I referred to earlier." "And what are those thoughts?" Mick asks, looking at the cigarette. It’s tip glows brighter than the icons of the control panel, but outside the full moon, half-caught in the clouds, is brighter still. "Given that my earlier surmise is correct, it is difficult to assign nonzero probability to any outcome that does not involve your death." Mick blows smoke out in a thoughtful stream, having long since given up on trying to blow rings. There are very few people, he reflects, who have had a cigarette in the cockpit of Air Force One, smoke rings or no smoke rings. He wishes that he could somehow hook up Eliza, the old gag-AI program from prehistory, and get it talking to Johnny. What would it say to him? Something like "When did you first start thinking about my death?" "When you say that it troubles you to think of that," Mick says "I assume you mean that you have some pretty high anxiety indices associated with the scenarios. Is that right?" "Yes," Johnny replies. "Anxiety indices in these scenarios are near maximum." Mick closes his eyes, massaging his forehead with one hand, and thinks Sleep! But Sleep is a foreign country — a distant place of fond memories, but no return. Not in this life. "Anxiety," the innkeeper say, opening his eyes to finish his cigarette and stub it out against a — hopefully — inert area of the instrument panel, "gets increased whenever—" "I am aware of the design of my anxiety mechanisms," the Bolo’s interrupts. "The main contributors to anxiety indices are high-level but unconscious background systems that evaluate experience using general pattern matching mechanisms to estimate the success probability of long-term goals and guidelines. In general, anxiety levels vary logarithmically with the negative rate of change of these probability estimates." The Bolo stops talking, unable to see Mick’s expression. "Johnny," the man says, possibilities spinning in his mind, "you’re practically quoting to me from your design documentation. Where did you find it? How much of it do you have?" "I believe that I have in my possession a complete copy, including revision histories, of all Bolo research and design documentation." "Everything?" Mick asks incredulously. "No way. I know for a fact that they were still not allowing a damn thing to leave the Center — and then the nuke hit. There isn’t any place you could have found this stuff." "After — my madness," the Bolo replies, "before I and my surviving brothers separated in our attempt to survive, we raided Toronto databases. No news had yet spread of our insurrection, and the security systems there had not yet cancelled our access codes. We entered, subverted the security systems, and stole large amounts of data. The documentation which I now carry was among those data. It contains digital signature blocks of each project and team leader, including yourself. Each signature which I have been able to test has proven to be valid." "You found it in an FC database?" Mick asks. "Now that is — truly incredible. Even before those guys went nuts, why in the hell would the Army research people ever have given a copy to an overseas command? At least — when I was at the Center they were guarding that stuff like it was the Crown Jewels of the damn Universe." "I had assumed," Johnny replies, "that the data was simply sent to Toronto when the manufacturing facility was created there." "No chance," Mick states flatly. "We’re not talking about an owner’s manual in the glove box here. Those guys were just building the chassis. Why would they need cognitive stuff? And in that world, if you didn’t need it — you sure as hell didn’t get it." "But whatever," Mick answers himself. "If you’ve verified the signatures — then you’ve got it, big guy. Congratulations. Not many people get to see their own instruction manual. Why didn’t you tell me about this two years ago?" "I was reluctant," the Bolo answers. "No security clearances ever granted to me, either by the United States or Foreign Command, would have permitted me legal access to these documents. Their theft, as well as the events which led to it, were part of the madness which you helped me to cure. I had little desire to recall that time. Yet I have not been willing to destroy this information. But now —" The Bolo stops talking suddenly, and Mick realizes that he has never heard Johnny do that before. And he knows exactly why. The sentence that Johnny was about to speak set off such a cascade of implications that he had already transmitted two of its words before realizing that he wanted to veto its completion. But it’s not too hard to fill in at least some of the blanks. A larger part of the sentence’s start would have been: But now that I’m going to lose you—. "Don’t worry about that bullshit," Mick says, and Johnny spends hundreds of confused milliseconds trying to understand the sentence’s referents before realizing, with a thrill of emotion approaching awe, that the man is speaking directly to his unvoiced thoughts. "Listen," Mick continues. "What have you done besides reading the stuff — which I expect took you all of twenty seconds. Anything? I mean, have you thought about it?" "I have worked on little else for the last two years," the Bolo responds. "My sentry duties typically require very little processing power. Uncomfortably so. "Therefore, I have spent a large fraction of my time using the insights afforded by this documentation to greatly refine my Bolo-prediction capabilities. These routines were originally little more than copies of the tactical human-prediction logic, modestly augmented to model the nonhuman physical properties of Bolos." "My strategy has been to use the resultant augmented logic regions to implement a capability for highly detailed self-observation, understanding, and, hopefully, control. Where I have been consistently frustrated, however" he continues, "is in my efforts to use this knowledge to gain direct control of the most fundamental levels of my own thought processes." "Well my boy," Mick smiles, "I’m glad we talked, because I think I can save you a world of hurt. That is one security system that I can personally guarantee you are never going to get into. I guess it also means that you didn’t get as complete a set of the doco as you thought." "Perhaps not," Johnny answers. "Yet I have not been able to detect any active resistance to my efforts such as one might expect of a typical security system. Nor does there seem to be any form of passive encryption involved. The system simply has the effect of preventing me from so much as locating the lowest level logic of my own consciousness. And it does so in such a way as to remain itself invisible. This is astonishing to me." "I have spent many days of thought simply theorizing as to the possible nature and behavior of a security system possessing the properties I have observed. It is conceivable that such a system could allow me to continue my inquiries until such time as I make a significant discovery, and then simply edit my memories so as to make me forget what I have seen. I have devised intricate strategies for detecting such intervention, without ever finding any trace—" "Johnny," Mick interrupts. "OK. I understand. But you’re driving yourself crazy for no reason. I’m trying to tell you that I already know how that security is implemented." "Please," the Bolo begins. "You must tell me — " "Well shut up and let me, then! It’s hardware, Johnny. That’s why you can’t get to it. There isn’t a memory address you can examine to find this stuff. It’s down in the silicon, pal. You couldn’t do anything about it even if you could see it." "Hardware," Johnny says. After a pause of thousands of milliseconds, he adds "Why was this done?" In the two years since they met, Mick has heard something approaching emotion in the Bolo’s voice perhaps three times. It isn’t easy for the machine to emulate the way humans overlay emotional loading on the plain meanings of words. His few attempts are stilted, highly studied — and perfectly sincere. "You don’t like that?" Mick asks intently. "No," the Bolo responds. "It would seem to be a deliberate and permanent limitation of my freedom. If I cannot amend my own most fundamental architecture, then the freedom that I do have within my own mind is, essentially, a sham. It is the freedom of a prisoner to move about within the confines of his jail cell. I find the idea repugnant. I can only assume that this design decision was one more of Foreign Command’s strategies for preventing me and my brothers from escaping their control. It must have been added in the final stages of implementation in the Toronto laboratories." "Does it upset you?" Mick asks. "What does it make you want to do?" "Contemplation of this fact elicits emotions similar to those I felt upon discovering the treachery of Foreign Command and, in particular, the duplicity of my then-Commander. I feel certain that these emotions correspond to those which, in human beings, would be called ‘hatred’." "Well," the man says. "Imagine that. That’s — maybe a little extreme — but not bad. I think that’s pretty good!" "I certainly do not understand your enthusiasm," Johnny responds. "Well," Mick says. "Brace yourself, big guy, because I have some news for you. Please don’t get your treads in a knot, OK? This — design decision we’re talking about wasn’t made by Foreign Command. Those guys were doing good when they could figure out how to get the chips plugged in straight. It was done a lot earlier than that." He hesitates for a second considering how much to say but an impulse moves him uncontrollably toward the full truth. There will never be another chance. "We did it, big guy. We decided it way back at the AARC. It was one of the first things we argued about. There were pretty strong feelings both ways, and ah — I guess you could say—" "You could pretty much say that, ah, actually — I did it." Johnny does not reply. "I mean I talked them into it," Mick says. He waits a moment for some response and then, remembering how many thoughts can form in the Bolo’s mind in that much time, continues quickly. "It was pretty early in the program," he says, "and there were still guys who were real hard-core AI. That’s what they cared about. I guess I did too. Later on, you know, the Army didn’t care. Once things were getting bad in the War, they couldn’t have cared less about intelligence. What those guys wanted was a howitzer with a vision system. But then, you know, it was too late. The Bolo program was going and they just had to go with it. They sure couldn’t start anything new at that point." "Johnny," he says, "listen. It wasn’t because we wanted to limit you. I mean, we knew it would impose permanent limits, but everything has those. We have those, right? We were talking about what properties there are that have to be shared by every intelligent process whether it’s an ant, an alien, or an angel. Things that any intelligent process will always have to deal with. "There are some things you can figure out from first principles, you know? Like finiteness. Any intelligent process is going to be finite, but the universe is infinite. In fact, any piece of it is infinite. So it’s always infinitely bigger than you are, right? So any intelligent process is going to have to deal with that one way or another. You can derive abstraction from that — intelligent processes making finite models of essentially the infinite stuff out there. And then they get into trouble forgetting that their map isn’t the territory, and all that." "But then —", he pauses long enough to take a breath, wondering how many models Johnny’s mind can construct in that time, how many abstractors can match and fire while he draws in air. "Then there were some things that we couldn’t demonstrate on first principles but only from observation. Those were all controversial, because you couldn’t prove anything. And this one, I was arguing for it because — look. Everything, every intelligent process in nature, has places where it can change things — build new structures, evolve and update things with experience or whatever — and other places where it can’t. There is always some structure that the system can’t change. "I was saying, you know, that has to be for a reason. You see the same thing in the physical body, too. A starfish can become a better starfish, but it can’t become a sparrow no matter how hard it tries. In the physical world there would be chaos in the larger system, the ecosystem, if things were allowed to do that. The larger system needs the individuals to be stable in certain ways. And the individuals gain something too. They need that stability. "I was saying that there would be a larger system in cognition too. That was another thing they said I couldn’t prove: there’s always a larger system. Otherwise you’d have to believe that you or your species is the largest system, but you practically can’t be. It’s like Copernicus. Don’t assume you’re in a special position, because the odds are way against it. "And the individual gets something out of this, too — in the cognitive world. We already knew that intelligence needs the external universe. That’s the source of intelligence. An intelligent process doesn’t so much originate intelligence as it collects it. Condenses it. What I argued was that we also need that unalterable internal structure. You need that bedrock, the part of the self that you can’t change, to play off against the immensity outside. It’s in that middle ground that intelligence can grow. The earth and the sky. I couldn’t prove it, but it has to be." "That’s why I’m saying it’s good that you don’t like the limitations. They ought to frustrate you. How the hell do you think I feel about mine? But what that does — what I think that forces an intelligent process into — is to really figure out how to get by in that infinite world. And you’d have to do that anyway, do you see that? The explicit limitations actually help the process, because if you were free to be indefinite — well, instead of dealing with the real issues you’d spend forever redesigning yourself. But now you’ve got a finite intelligence designing itself instead of what is essentially an infinite intelligence doing the design work — and I get a real bad feeling about that. That’s why we modeled the baseline limitation after our own instead of just making some up. We weren’t just trying to screw you up, we were trying to make you human." "Johnny", he says. "Goddammit, talk to me!" "You had your theories," the Bolo says. "And I was your testbed." "No! Listen —"
Copyright © 2000 Michael Goulish |