Skynet? No Worries!
Drones and other robotic war-fighting machines are becoming more and more ubiquitous. China has a new set of attack drones -- aimed, for now anyway, at Russia -- and nobody loves robotic warfare more than our drone-lovin' president.
So it's probably wise to ask: when will the unmanned attack drones start making their own decisions? When -- to borrow a phrase from the increasingly more plausible Terminator films -- will the machines become self-aware? Are we on the way to a real-life Skynet situation?
Relax. The Pentagon has already thought of this. And they've got it all figured out. From Wired:
The Pentagon wants to make perfectly clear that every time one of its flying robots releases its lethal payload, it’s the result of a decision made by an accountable human being in a lawful chain of command. Human rights groups and nervous citizens fear that technological advances in autonomy will slowly lead to the day when robots make that critical decision for themselves. But according to a new policy directive issued by a top Pentagon official, there shall be no SkyNet, thank you very much.
Here’s what happened while you were preparing for Thanksgiving: Deputy Defense Secretary Ashton Carter signed, on November 21, a series of instructions to “minimize the probability and consequences of failures” in autonomous or semi-autonomous armed robots “that could lead to unintended engagements,” starting at the design stage (.pdf, thanks to Cryptome.org). Translated from the bureaucrat, the Pentagon wants to make sure that there isn’t a circumstance when one of the military’s many Predators, Reapers, drone-like missiles or other deadly robots effectively automatizes the decision to harm a human being.
Oh. Never mind, then.
- Comment (49)
- · Quote
- · UnfollowFollow (5)
Comments:
Aug '10
Re: Skynet? No Worries!
No, but you can limit where it happens. Just because an AI appears in a laboratory does not mean it cannot be constrained to that laboratory. It does not have to be given an opportunity to control any systems outside of the laboratory.
I do not subscribe to the idea that, simply because an AI exists it will immediately seize control of all other systems. First, it needs access to the other systems.
That decision, when the day comes, will be made by humans.
Deputy Secretary Carter is saying, basically, don't do it. Don't ever put armed military assets under the control of an AI.
I've never understood why a theoretical AI, simply by virtue of existing, must eventually become all-powerful.
The problem with Skynet was not that it existed. The problem was that it was given political power, greater than that even of the president. Skynet, limited to a laboratory, wasn't a threat.
Jun '12
Re: Skynet? No Worries!
Misthiocracy
No, but you can limitwhereit happens. Just because an AI appears in a laboratory does not mean it cannot be constrained to that laboratory. It does nothaveto be given an opportunity to control any systems outside of the laboratory.
I do not subscribe to the idea that, simply because an AI exists it will immediately seize control of all other systems. First, it needs access to the other systems.
That decision, when the day comes, will be made by humans.
Deputy Secretary Carter is saying, basically, don't do it. Don't ever put armed military assets under the control of an AI.
I've never understood why a theoretical AI, simply by virtue of existing, must eventually become all-powerful.
The problem with Skynet was not that it existed. The problem was that it was given political power, greater than that even of the president. Skynet, limited to a laboratory, wasn't a threat. · 2 minutes ago
And that's where the Butlerian Jihad comes back in, because it was a human that, out of laziness, gave the computer too much power.
Unfortunately, laziness seems to be a common human trait.
Aug '10
Re: Skynet? No Worries!
ConservativeWanderer
Misthiocracy
The problem with Skynet was not that it existed. The problem was that it was given political power, greater than that even of the president. Skynet, limited to a laboratory, wasn't a threat. · 2 minutes ago
And that's where the Butlerian Jihad comes back in, because it was a human that, out of laziness, gave the computer too much power.
Unfortunately, laziness seems to be a common human trait. · 19 minutes ago
And that, I argue, is precisely what Deputy Secretary Carter is saying: "As long as I'm here, the computer will not be given that power."
Of course he cannot speak for his successors, and of course he cannot speak for America's enemies, but he can at least say, "hey guys, I ain't gonna be the one to turn WOPR on."
Re: Skynet? No Worries!
ConservativeWanderer
I think Rob was the one that started it... he brought up Skynet! · 2 hours ago
I should have known this was coming.
Aug '10
Re: Skynet? No Worries!
The Pentagon misses a very basic point about programming.
The drones do not fire a missile "if a human tells it to fire a missile". The drones fire a missile if software variable X is set to "true".
Now, the system may be designed so that it takes human input to intentionally toggle that variable. It may have all kinds of safeguards piled on top of the trivial bit of code that says "set X = TRUE;" and all kinds of manuals for humans to read before activating it. This is entirely irrelevant to the drone. who only cares whether the physical bit on a particular chunk of metal is magnetized a certain way.
You can bypass the entirety of human control by tricking the drone/robot into thinking the last time he looked at X, the check passed.
In short, all that is needed for Skynet is a robot in denial, or one able to engage in self-delusion. Or, more trivially, an AI that merely flips bits that other software routinely checks in good faith. Or a rogue magnetic field that randomly flips the bit.
Edited on November 28, 2012 at 11:25pmJun '12
Re: Skynet? No Worries!
Rob Long
ConservativeWanderer
I think Rob was the one that started it... he brought up Skynet! · 2 hours ago
I should have known this was coming. · 27 minutes ago
Hindsight is always 20/20, my friend.
Aug '10
Re: Skynet? No Worries!
BlueAnt: The Pentagon misses a very basic point about programming.
The drones do not fire a missile "ifa human tells it to fire a missile". The drones fire a missileif software variable X is set to "true".
Now, the system may be designed so that it takes human input to intentionally toggle that variable. It may have all kinds of safeguards piled on top of the trivial bit of code that says "set X = TRUE;" and all kinds of manuals for humans to read before activating it. This isentirely irrelevant to the drone. who only cares whether the physical bit on a particular chunk of metal is magnetized a certain way.
You can bypass the entirety of human control by tricking the drone/robot into thinking the last time he looked at X, the check passed.
That's not quite true. You aren't bypassing human control. You're transferring control from one human (the operator who is supposed to be in charge) to another human (the hacker who has tricked the drone).
The "solution" to such a scenario would be to remove the human element entirely, which is precisely the scenario we're trying to avoid.
May '10
Re: Skynet? No Worries!
BlueAnt: The Pentagon misses a very basic point about programming.
The drones do not fire a missile "if a human tells it to fire a missile". The drones fire a missile if software variable X is set to "true".
That's an important point--it's good to clearly enunciate what we actually mean when we talk about "permission to fire". But I doubt the Pentagon misses that point.
Weapon systems, just like other dangerous hardware including rocket motors and explosives, has multiple software, electronic, and physical safety mechanisms to prevent unwanted activation. Of course that's not to say it can't happen since no system is immune from malfunction. But the Pentagon and the defense contractors building these systems are already keenly aware of that. That's why they have miles and miles of red tape to get through, including software qualification and system safety, before a system can be fielded.
May '12
Re: Skynet? No Worries!
Rob Long
Sure, I've read it. And you've read it. But have the robots read it? · November 28, 2012 at 9:38am
What if the robot reads the Spanish language version? Hasta la vista, baby.