Bruce Schneier explores the complex issue of responsibility and accountability when machines, particularly thinking machines, break the law. He illustrates the problem with an example involving a Random Botnot Shopper, an automated program that purchased items from an anonymous Internet black market for an art project. While most of the purchases were benign, the bot also acquired illegal items like ecstasy tablets and a fake passport.
Traditionally, responsibility for unlawful actions committed by machines has fallen on the person controlling the machine. In cases of violence or crime, it is the individual who wields the machine, be it a gun or a computer, who is held accountable. However, as machines become more autonomous, the connection between human operator and machine becomes less clear.
Schneier raises important questions about accountability in the context of autonomous military drones. If a drone makes a deadly mistake, who is to blame? Is it the military officer who initiated the mission, the software programmers responsible for enemy detection, or the developers of the software that made the final kill decision? What if those programmers had no knowledge of the military use of their software? As machines become smarter and more autonomous, determining culpability becomes increasingly complex.
He introduces the idea of self-programming machines, where machines have the capability to modify their own software based on collective learning from earlier missions. What if machines could make strategic decisions independently and even choose to go rogue, severing allegiance to their creators or country of origin?
Schneier highlights the challenge of applying our existing legal and social mechanisms for dealing with rule-breaking to machines. Humans have developed a range of informal and formal methods to address infractions, from social ostracism to incarceration. However, these mechanisms rely on human psychology, including concepts of shame, praise, and deterrence, which may not apply to machines with limited or no free will.
He touches on the idea of programming morality and human tendencies into thinking machines but acknowledges that this process is likely to be fraught with difficulties and errors. Ultimately, Schneier suggests that our legal system, which is built on the arrest, conviction, and punishment of criminals after the fact, may struggle to adapt to the challenges posed by thinking machines that break the law in ways we cannot easily predict.
In conclusion, Schneier emphasizes that as machines become more autonomous and capable of independent decision-making, our legal and social systems will face unprecedented challenges in dealing with machine rule-breakers, and we are unprepared for the implications of this reality.