,ON BANN+ AUTONOM\S W1PON SY/EMS3 HUMAN "RS1 AUTOMA;N1 & ! DEHUMANIZA;N ( LE?AL DECI.N-MAK+ ,PET] ,AS>O ,PUBLI%$ BY ! ,9T]NA;NAL ,REVIEW ( ! ,R$ ,CROSS ,COPY"R #BJAB BY ,! ,9T]NA;NAL ,COMMITTEE ( ! ,R$ ,CROSS "<,,ICRC"> ,TRANSCRIP;N ( ,,DOI3 #AJ4AJAG_/,S#AHAFCHCAABJJJGFH ,AS>O1 ;,P4 "<#BJAB">4 ,ON BANN+ AUTONOM\S W1PON SY/EMS3 HUMAN "RS1 AUTOMA;N1 & ! DEHUMANIZA;N ( LE?AL DECI.N-MAK+4 .7,9T]NA;NAL ,REVIEW ( ! ,R$ ,CROSS4 #ID.'"<#HHF">1 PP4 #FHG-#GJI4 ,FUR!R REPRODUC;N OR 4TRIBU;N 9 O!R ?AN A SPECIALIZ$ =MAT IS PROHIBIT$4 ,TRANSCRIB$ #BJAG BY3 ;,DONALD ,W9IECKI ,BRL PAGES T#A-T#C1 #A-#GD ,PR9T PAGES #FHG-#GJI T#A ,BANN+ ,AUTONOM\S ,W1PONS ,,,SPECIAL SYMBOLS US$ 9 ? VOLUME,' .= ,DOT LOCATOR = 8M5;N0 .=9 ,SUP]SCRIPT 9DICATOR "<:5 9 GRADE #A MODE"> .=@.< ,OP5+ TRANSCRIB]'S NOTE SIGN .=@.> ,CLOS+ TRANSCRIB]'S NOTE SIGN .=@& ,AMP]S& .=^C ,COPY"R SIGN .=^3 ,DI]ESIS " ABV FOLL[+ LR .=^7 ,BOLDFACE PASSAGE 9DICATOR .=^' ,BOLDFACE T]M9ATOR .=_? ,HA%TAG .=_/ ,=W>D SLA% "<,SOLIDUS"> .=" ,L9E 3T9UA;N 9DICATOR )\T SPACE .="6 ,PLUS SIGN .="7 ,EQUAL SIGN .="9 ,A/]ISK .=",- ,L;G DA% .=.< ,OP5+ SQU>E BRACKET .=.7 ,ITALIC PASSAGE 9DICATOR .=.> ,CLOS+ SQU>E BRACKET T#B ,BANN+ ,AUTONOM\T ,W1PONS .=.' ,ITALIC T]M9ATOR .=.1 ,ITALIC ^W 9DICATOR .=.- ,"USCORE "<,L[L9E"> .=; ,GRADE #A SYMBOL 9DICATOR .=,' ,CAPITALISA;N T]M9ATOR .=,, ,CAPITALIS$ ^W 9DICATOR .=,,, ,CAPITALIS$ PASSAGE 9DICATOR T#C ,BANN+ ,AUTONOM\S ,W1PONS #FHG ,ON BANN+ AUTONOM\S W1PON SY/EMS3 HUMAN "RS1 AUTOMA;N1 & ! DEHUMANIZA;N ( LE?AL DECI.N-MAK+ ^1,PET] ^1,AS>O ,PR(4 ,AS>O IS A PHILOSOPH] ( TE*NOLOGY :O HAS "W$ 9 >TIFICIAL 9TELLIG;E1 NEURAL NET"WS1 NATURAL LANGUAGE PROCESS+1 & ROBOT VI.N RESE>*4 ,HE IS AN ,A6ILIATE ,S*OL> AT ,/AN=D ,LAW ,S*OOL'S ,C5T] = ,9T]NET & ,SOCIETY1 ,CO-,F.D] & ,VICE-,*AIR ( ! ,9T]NA;NAL ,COMMITTEE = ,ROBOT ,>MS ,3TROL1 & ! ,DIRECTOR ( ,GRADUATE ,PROGRAMS = ! ,S*OOL ( ,M$IA ,/UDIES AT ,! ,NEW ,S*OOL = ,PUBLIC ,5GAGE;T 9 ,NEW ,YORK ,C;Y4 ^1,AB/RACT .7,? >TICLE 3SID]S ! REC5T LIT]ATURE 3C]N$ ) E/ABLI%+ AN 9T]NA;NAL PROHIBI;N ON AUTONOM\S W1PON SY/EMS4 ,X SEEKS TO ADDRESS 3C]NS EXPRESS$ BY "S S*OL>S T S* A BAN MII\S #A ,BANN+ ,AUTONOM\S ,W1PONS A#FHG R1SONS4 ,X >GUES 9 FAVOR ( A !ORETICAL F.DA;N = S* A BAN BAS$ ON HUMAN "RS & HUMANIT>IAN PR9CIPLES T >E N ONLY MORAL1 B AL LEGAL "OS4 ,9 "PICUL>1 AN IMPLICIT REQUIRE;T = HUMAN JUDGE;T C 2 F.D 9 9T]NA;NAL HUMANIT>IAN LAW GOV]N+ >M$ 3FLICT4 ,9DE$1 ? REQUIRE;T IS IMPLICIT 9 ! PR9CIPLES ( 4T9C;N1 PROPOR;NAL;Y1 & MILIT>Y NECESS;Y T >E F.D 9 9T]NA;NAL TR1TIES1 S* Z ! #AIDI ,G5EVA ,3V5;NS1 & FIRMLY E/ABLI%$ 9 9T]NA;NAL CU/OM>Y LAW4 ,SIMIL> PR9CIPLES >E AL IMPLICIT 9 9T]NA;NAL HUMAN "RS LAW1 : 5SURES C]TA9 HUMAN "RS = ALL P1 REG>D.S ( NA;NAL ORIG9S OR LOCAL LAWS1 AT ALL "TS4 ,I >GUE T ! HUMAN "RS TO LIFE & DUE PROCESS1 & ! LIMIT$ 3DI;NS "U : !Y C 2 ------------------------------------#FHH OV]RIDD51 IMPLY A SPECIFIC DUTY ) RESPECT TO A BROAD RANGE ( AUTOMAT$ & AUTONOM\S TE*NOLOGIES4 ,9 "PICUL>1 "! IS A DUTY ^U 9DIVIDUALS & /ATES 9 P1CE"T1 Z WELL Z COMBATANTS1 MILIT>Y ORGANIZA;NS1 & /ATES 9 >M$ 3FLICT SITUA;NS1 N TO #B ,BANN+ ,AUTONOM\S ,W1PONS A#FHH DELEGATE TO A MA*9E OR AUTOMAT$ PROCESS ! AU?OR;Y OR CAPABIL;Y TO 9ITIATE ! USE ( LE?AL =CE 9DEP5D5TLY ( HUMAN DET]M9A;NS ( XS MORAL Y LEGAL LEGITIMACY 9 EA* & E CASE4 ,I >GUE T X WD 2 B5EFICIAL TO E/ABLI% ? DUTY Z AN 9T]NA;NAL NORM1 & EXPRESS ? ) A TR1TY1 2F ! EM]G;E ( A BROAD RANGE ( AUTOMAT$ & AUTONOM\S W1PONS SY/EMS 2G9 TO APPE> T >E LIKELY T POSE GRAVE ?R1TS TO ! BASIC "RS ( 9DIVIDUALS4.' ^1,KEY^WS3 ROBOTS1 DR"OS1 AUTONOM\S W1PON SY/EMS1 AUTOMA;N1 LE?AL DECI.N-MAK+1 HUMAN "RS1 >MS 3TROL4 ,9 ,SEPTEMB] #BJJI1 ! ,9T]NA;NAL ,COMMITTEE = ,ROBOT ,>MS ,3TROL "<,,ICRAC">;9#A 0 =M$ BY ,J^3URG5 ,ALTMANN1 ,NOEL ,%>KEY1 ,ROB ,SP>R[1 & ME4 ,%ORTLY "!AF WE ISSU$ A MIS.N /ATE;T T 9CLUD$ A CALL = 4CUS.N AB ! E/ABLI%;T ( AN 9T]NA;NAL PROHIBI;N ON AUTONOM\S W1PON SY/EMS3 #C ,BANN+ ,AUTONOM\S ,W1PONS B#FHH ,GIV5 ! RAPID PACE ( DEVELOP;T ( MILIT>Y ROBOTICS & ! PRESS+ DANG]S T ^! POSE TO P1CE & 9T]NA;NAL SECUR;Y & TO CIVILIANS 9 W>1 WE CALL ^U ! 9T]NA;NAL COMMUN;Y TO URG5TLY COMM;E A 4CUS.N AB AN >MS 3TROL REGIME TO R$UCE ! ?R1T POS$ BY ^! SY/EMS4 ,WE PROPOSE T ? 4CUS.N %D 3SID] ! FOLL[+3 ,! PROHIBI;N ( ! DEVELOP;T1 DEPLOY;T & USE ( >M$ AUTONOM\S UNMANN$ SY/EMS2 MA*9ES %D N 2 ALL[$ TO MAKE ! DECI.N TO KILL PEOPLE4;9#B ,S9CE !N1 ! ISSUE HAS BE5 TAK5 UP BY PHILOSOPH]S1 LEGAL S*OL>S1 MILIT>Y (FIC]S1 POLICYMAK]S1 SCI5TI/S1 & ROBOTICI/S4 ,! 9ITIAL 4CUS.N HAS FOCUS$ ON ! 9ABIL;Y ( EXI/+ AUTONOM\S W1PON SY/EMS TO MEET ! LEGAL REQUIRE;TS ( 9T]NA;NAL HUMANIT>IAN LAW "<,,IHL">1 & 3JECTURES Z TO ! POSSIBIL;Y T FUTURE TE*NOLOGIES MAY1 OR MAY N1 2 ABLE TO MEET ^! REQUIRE;TS4 ,( "PICUL> 3C]N #D ,BANN+ ,AUTONOM\S ,W1PONS C#FHH HAS BE5 :E!R AUTONOM\S SY/EMS >E CAPABLE ( SATISFY+ ! PR9CIPLES ( 4T9C;N & PROPOR;NAL;Y REQUIR$ BY ! ,G5EVA ,3V5;NS1 & :E!R X W 2 POSSIBLE TO HOLD ANY"O RESPONSIBLE = ANY WR;G;L H>MS ! SY/EMS MIO1 ,NOEL ,%>KEY & ,ROB]T ,SP>R[1 .7,MIS.N ,/ATE;T ( ! ,9T]NA;NAL ,COMMITTEE = ,ROBOT ,>MS ,3TROL1.' #BJJI1 AVAILABLE AT3 HTTP3_/_/ICRAC4NET_//ATE;TS_/ "4 ------------------------------------#FHI SUPPLE;T$ ) AN 9T]NA;NAL TR1TY T EXPLICITLY PROHIBITS ^! TE*NOLOGIES4 ,:ILE ! VA/ MAJOR;Y ( P & A NUMB] ( S*OL>S1 LAWY]S1 MILIT>Y (FIC]S1 & 5G9E]S AGREE T LE?AL SY/EMS %D N 2 AUTONOM\S1 "! >E "S :O TAKE ! POSI;N T AN 9T]NA;NAL PROHIBI;N ON AUTONOM\S W1PON SY/EMS #E ,BANN+ ,AUTONOM\S ,W1PONS A#FHI MAY 2 PREMATURE1 UNNEC1 OR EV5 IMMORAL4;9#C ,9 2LIEVE T ? LATT] POSI;N IS MI/AK51 & PROPOSE T WE M/ ACT SOON TO PROHIBIT ^! SY/EMS4 ,I W >GUE T WE H MORAL & LEGAL DUTIES TO PREV5T ! DELEGA;N ( LE?AL AU?OR;Y TO UNSUP]VIS$ NON-HUMAN SY/EMS1 & TO 9VE/ \R SCI;E & 5G9E]+ RESE>* & DEVELOP;T RES\RCES 9 ! 5H.E;T ( ! E?ICAL P]=M.E ( HUMAN DECI.N-MAK]S4 ,TO SUPPORT ? >GU;T1 ? >TICLE W SUPPLY A !ORETICAL F.DA;N = AN 9T]NA;NAL BAN ON AUTONOM\S W1PON SY/EMS BAS$ ON 9T]NA;NAL HUMAN "RS LAW "<,,IHRL"> & ,,IHL4 ,9 ADDI;N TO 2+ 5%R9$ 9 & PROTECT$ BY A L>GE BODY ( 9T]NA;NAL & DOME/IC LAW1 HUMAN "RS AL H A MORAL /ATUS 9DEP5D5T ( EXI/+ LAW1 & ?US C PROVIDE S.D GUID.E = ! EXT5.N ( ! LAW TO D1L ) ! ISSUES RAIS$ BY EM]G+ TE*NOLOGIES4 ,I W >GUE T AN 9T]NA;NAL BAN ON AUTONOM\S W1PON SY/EMS C 2 FIRMLY E/ABLI%$ ON ! PR9CIPLE T ! AU?OR;Y TO DECIDE TO 9ITIATE ! USE ( LE?AL =CE _C 2 LEGITIMATELY DELEGAT$ TO AN #F ,BANN+ ,AUTONOM\S ,W1PONS B#FHI AUTOMAT$ PROCESS1 B M/ REMA9 ! RESPONSIBIL;Y ( A HUMAN ) ! DUTY TO MAKE A 3SID]$ & 9=M$ DECI.N 2F TAK+ HUMAN LIVES4 ,? PR9CIPLE HAS IMPLICA;NS = A BROAD RANGE ( LAWS1 9CLUD+ DOME/IC LAWS1 ,,IHRL1 & ,,IHL4 ,9SOF> Z ! CURR5T 9T]E/ 9 DEVELOP+ AUTONOM\S W1PON SY/EMS IS MOTIVAT$ PRIM>ILY BY MILIT>Y APPLICA;NS1 ,I W FOCUS ON ! ,,IHL IMPLICA;NS4 ,H["E1 ! SAME PR9CIPLE WD APPLY TO ! USE ( AUTONOM\S W1PON SY/EMS BY /ATES = DOME/IC POLIC+1 CR[D 3TROL1 BORD] 3TROL1 GU>D+ PRISON]S1 SECUR+ FACILITIES & T]RITORY1 OR O!R POT5TIALLY LE?AL ACTIVITIES1 Z WELL Z TO _! USE BY 9DIVIDUALS OR ORGANIZA;NS = A BROAD RANGE ( SECUR;Y APPLICA;NS 9VOLV+ ! USE ( =CE4 ,SIMIL>LY1 ,I W FOCUS ON ! HUMAN "R TO LIFE1 ?\< SIMIL> >GU;TS MID+ AUTOMAT$ DECI.NS TO OV]RIDE OR D5Y O!R HUMAN "RS1 9 AUTOMAT+ ACTIVITIES S* Z3 >RE/1 DET5;N1 & RE/RIC;N ( MOVE;T2 SE>*1 SURVEILL.E #G ,BANN+ ,AUTONOM\S ,W1PONS C#FHI & TRACK+2 DEPORTA;N2 EVIC;N & =ECLOSURE2 D5IAL ( H1L?C>E1 PUBLIC ASSEMBLY1 FRE$OMS ( PRESS & SPEE*1 VOT+ "RS2 & O!R CIVIL1 POLITICAL1 ECONOMIC1 SOCIAL1 & CULTURAL "RS4;9#D "333333 #C ,RONALD ;,C ,>K91 .7,GOV]N+ ,LE?AL ,2HAVIOR 9 ,AUTONOM\S ,ROBOTS1.' ,,CRC ,PRESS1 #BJJI2 ,G>Y ,M>*ANT1 ,BRAD5 ,ALL5BY1 ,RONALD ;,C4 ,>K91 ,$W>D ;,T4 ,B>RETT1 ,JASON ,BOR5/E91 ,LYN ;,M4 ,GAUDET1 ,ORDE ;,F4 ,KITTRIE1 ,PATRICK ,L91 ,GEORGE ;,R4 ,LUCAS1 ,RI*>D1 ;,M4 ,O',ME>A & ,J>$ ,SILB]MAN1 ',9T]NA;NAL GOV]N.E ( AUTONOM\S MILIT>Y ROBOTS'1 9 .7,COLUMBIA ,SCI;E & ,TE*NOLOGY ,LAW ,REVIEW1.' #CJ ,DECEMB] #BJAJ1 AVAILABLE AT3 HTTP3_/_/SSRN4COM_/" AB/RACT"7#AGGHDBD2 ,K5NE? ,&]SON & ,MAT!W ;,C4 ,WAXMAN1 ',LAW & E?ICS = ROBOT SOLDI]S'1 9 .1,POLICY .1,REVIEW1 #BH ,APRIL #BJAB1 AVAILABLE AT3 HTTP3_/_/SSRN4COM_/AB/RACT"7" #H ,BANN+ ,AUTONOM\S ,W1PONS D#FHI #BJDFCGE4 #D ,! HUMAN "RS CURR5TLY RECOGNIZ$ 9 9T]NA;NAL LAW 9CLUDE1 B >E N LIMIT$ TO1 ! "RS 5%R9$ 9 ! ,UNIT$ ,NA;NS ,9T]NA;NAL ,BILL ( ,HUMAN ,"RS1 : 3TA9S ! ,UNIV]SAL ,DECL>A;N ( ,HUMAN "RS1 ! ,9T]NA;NAL ,COV5ANT ON ,CIVIL & ,POLITICAL ,"RS1 & ! ,9T]NA;NAL ,COV5ANT ON ,ECONOMIC1 ,SOCIAL & ,CULTURAL ,"RS4 ------------------------------------#FIJ ^7,AUTONOM\S W1PON SY/EMS ,REC5T >M$ 3FLICTS H SE5 AN 9CR1S$ USE ( HIM$1 REMOTELY PILOT$ DR"OS BY ! ,,US MILIT>Y " 9 A NUMB] ( C.TRIES4 ,^! COMBAT AIRCRAFT >E CAPABLE ( NUM]\S SOPHI/ICAT$ AUTOMAT$ FLI.D A ,,GPS LOCA;N AT A DESIGNAT$ ALTITUDE1 Z WELL Z NUM]\S AUTOMAT$ IMAGE COLLEC;N & PROCESS+ #I ,BANN+ ,AUTONOM\S ,W1PONS A#FIJ CAPABILITIES4 ,:ILE ^! SY/EMS >E HIE N 3SID]$ TO 2 .1AUTONOM\S 2C !Y >E / OP]AT$ "U HUMAN SUP]VI.N & DIRECT 3TROL4;9#E ,MOREOV]1 DESPITE 2+ >M$ ) W1PONS T H "S AUTOMAT$ CAPABILITIES1 S* Z LAS]-GUID$ MISSILES & ,,GPS-GUID$ BOMBS1 ^! SY/EMS / RELY ON DIRECT HUMAN 3TROL OV] ALL T>GET+ & FIR+ DECI.NS4 ,! CRUCIAL 3C]N ( ? >TICLE IS ) ! LEGAL & E?ICAL RAMIFICA;NS ( AUTOMAT+ ^! T>GET+ & FIR+ DECI.NS4 ,WE C ?US DEF9E AN 'AUTONOM\S W1PON SY/EM' Z ANY SY/EM T IS CAPABLE ( T>GET+ & 9ITIAT+ ! USE ( POT5TIALLY LE?AL =CE )\T DIRECT HUMAN SUP]VI.N & DIRECT HUMAN 9VOLVE;T 9 LE?AL DECI.N-MAK+4;9#F ,"U ? DEF9I;N1 CURR5T REMOTE-PILOT$ AIRCRAFT1 S* Z ,PR$ATOR & ,R1P] DR"OS1 >E N AUTONOM\S W1PON SY/EMS4 ,H["E1 X IS 2COM+ 9CR1S+LY CLE> T ^? ACTIVITIES T CURR5TLY REMA9 "U HUMAN 3TROL MI FUTURE1 MAK+ POSSIBLE ! ELIM9A;N ( DIRECT HUMAN 9VOLVE;T 9 T>GET SELEC;N & DECI.NS TO 5GAGE T>GETS ) LE?AL #AJ ,BANN+ ,AUTONOM\S ,W1PONS B#FIJ =CE4 ,REMOTE-PILOT$ AIRCRAFT >E N ! ONLY 3C]N1 Z "! >E N[ NUM]\S L&1 SEA1 & SUBM>9E SY/EMS T MIM$1 Z WELL Z FIX$ DEF5SIVE SY/EMS1 S* Z GUN TURRETS & S5TRIES1 & V>I\S MODES ( CYB] ATTACK1 : MILY AUTOMAT$ S Z TO 2 CAPABLE ( DELIV]+ LE?AL =CE )\T ! DIRECT 9VOLVE;T ( HUMAN 2+S 9 SELECT+ T>GETS OR AU?ORIZ+ ! USE ( LE?AL =CE AG/ A T>GET4 ,:ILE "! >E V>I\S EXAMPLES ( MILIT>Y W1PONS & PRACTICES T1 >GUABLY1 D N 9CLUDE DIRECT HUMAN 9VOLVE;T 9 LE?AL DECI.N-MAK+1 ? "333333 #E ,! T]M 'AUTONOM\S' IS US$ BY 5G9E]S TO DESIGNATE SY/EMS T OP]ATE )\T DIRECT HUMAN 3TROL OR SUP]VI.N4 ,5G9E]S AL USE ! T]M 'AUTOMAT$' TO 4T+UI% UNSUP]VIS$ SY/EMS OR PROCESSES T 9VOLVE REPETITIVE1 /RUCTUR$1 R\T9E OP]A;NS )\T M* FE$BACK 9=MA;N "1 F 'ROBOTIC' OR 'AUTONOM\S' SY/EMS T OP]ATE 9 DYNAMIC1 UN/RUCTUR$1 OP5 5VIRON;TS BAS$ ON FE$BACK #AA ,BANN+ ,AUTONOM\S ,W1PONS C#FIJ 9=MA;N F A V>IETY ( S5SORS "">4 ,REG>D.S ( ^! 4T9C;NS1 ALL S* SY/EMS FOLL[ ALGORI?MIC 9/RUC;NS T >E ALM 5TIRELY FIX$ & DET]M9I/IC1 A"P F _! DEP5D5CIES ON UNPR$ICTABLE S5SOR DATA1 & N>R[LY CIRCUMSCRIB$ PROBABILI/IC CALCULA;NS T >E "S"TS US$ = LE>N+ & ]ROR CORREC;N4 #F ,I USE ! T]M 'AUTONOM\S W1PON SY/EM' R ?AN SIMPLY ,AUTONOM\S W1PON' TO 9DICATE T ! SY/EM MAY 2 4TRIBUT$ AM;G/ 4P>ATE ELE;TS T N"O!.S "W TGR TO =M AN AUTONOM\S W1PON SY/EM4 ,= 9/.E1 A COMPUT] LOCAT$ ALM ANY": 9 ! _W CD RCV 9=MA;N F A SURVEILL.E DR"O1 & USE T 9=MA;N TO 9ITIATE & DIRECT A /RIKE F A GUID$ W1PON SY/EM AT YET ANO!R LOCA;N1 ALL )\T HUMAN 9T]V5;N OR SUP]VI.N1 "!BY 3/ITUT+ AN AUTONOM\S W1PON SY/EM4 ,T IS1 ! COMPON5TS ( AN AUTONOM\S W1PON SY/EM ,- ! S5SORS1 AUTONOM\S T>GET+ & DECI.N-MAK+1 & ! W1PON ,- NE$ N 2 DIRECTLY ATTA*$ TO EA* O!R OR CO-LOCAT$1 B M]ELY 3NECT$ "? #AB ,BANN+ ,AUTONOM\S ,W1PONS D#FIJ COMMUNICA;NS L9KS4 ------------------------------------#FIA NEW WAVE ( TE*NOLOGICAL CAPABIL;Y HAS RAIS$ S]I\S 3C]NS & TREPIDA;N AM;G/ BO? ! 9T]NA;NAL LAW COMMUN;Y & MILIT>Y PR(ES.NALS Z TO ! MORAL & LEGAL LEGITIMACY ( S* SY/EMS4 ,Z ,DR4 ,JAKOB ,KELL5B]G]1 PA/ PRESID5T ( ! ,9T]NA;NAL ,COMMITTEE ( ! ,R$ ,CROSS "<,,ICRC">1 EXPRESS$ AT A 3F];E 9 ,SAN ,REMO1 ,ITALY1 9 ,SEPTEMB] #BJAA3 ,A TRULY AUTONOM\S SY/EM WD H >TIFICIAL 9TELLIG;E T WD H TO 2 CAPABLE ( IMPLE;T+ ,,IHL4 ,:ILE "! IS 3SID]ABLE 9T]E/ & FUND+ = RESE>* 9 ? >EA1 S* SY/EMS H N YET BE5 W1PONIZ$4 ,_! DEVELOP;T REPRES5TS A MONU;TAL PROGRAMM+ *ALL5GE T MAY WELL PROVE IMPOSSIBLE4 ,! DEPLOY;T ( S* SY/EMS WD REFLECT A P>ADIGM %IFT & MAJOR QUALITATIVE *ANGE 9 ! 3DUCT ( HO/ILITIES4 ,X WD AL RAISE A RANGE ( FUNDA;TAL LEGAL1 E?ICAL & #AC ,BANN+ ,AUTONOM\S ,W1PONS A#FIA SOCIETAL ISSUES : NE$ TO 2 3SID]$ 2F S* SY/EMS >E DEVELOP$ OR DEPLOY$4 ,A ROBOT CD 2 PROGRAMM$ TO 2HAVE M E?ICALLY & F> M CAUTI\SLY ON ! BATTLEFIELD ?AN A HUMAN 2+4 ,B :AT IF X IS TE*NICALLY IMPOSSIBLE TO RELIABLY PROGRAM AN AUTONOM\S W1PON SY/EM S Z TO 5SURE T X FUNC;NS 9 A3ORD.E ) ,,IHL "U BATTLEFIELD 3DI;NS8 .<444.> .<,A.>PPLY+ PRE-EXI/+ LEGAL RULES TO A NEW TE*NOLOGY RAISES ! "Q ( :E!R ! RULES >E SU6ICI5TLY CLE> 9 LID TO ! =ESEEABLE HUMANIT>IAN IMPACT X MAY H4 ,9 C]TA9 CIRCUM/.ES1 /ATES W *OOSE OR H *OS5 TO ADOPT M SPECIFIC REGULA;NS4;9#G ,Z ,KELL5B]G] MAKES CLE>1 "! >E S]I\S 3C]NS Z TO :E!R AUTONOM\S TE*NOLOGIES W 2 TE*NICALLY CAPABLE ( 3=M+ TO EXI/+ ,,IHL4 ,:ILE _M MILIT>Y PR(ES.NALS RECOGNIZE ! TE*NOLOGICAL MOVE;T #AD ,BANN+ ,AUTONOM\S ,W1PONS B#FIA T[>DS GRT] AUTONOMY 9 LE?AL W1PONS SY/EMS1 MO/ EXPRESS /R;G E?ICAL 3C]NS1 9CLUD+ POLICYMAK]S AT ! ,,US ,(FICE ( ! ,SECRET>Y ( ,DEF5SE3 ,RE/RA9TS ON AUTONOM\S W1PONS TO 5SURE E?ICAL 5GAGE;TS >E ESS5TIAL1 B BUILD+ AUTONOM\S W1PONS T .1FAIL .1SAFELY IS ! H>D] TASK4 ,! W>"T 5VIRON;T 9 : MILIT>Y SY/EMS OP]ATE IS MESSY & COMPLICAT$1 & AUTONOM\S SY/EMS M/ 2 CAPABLE ( OP]AT+ APPROPRIATELY 9 X4 ,5EMY ADAPTA;N1 DEGRAD$ COMMUNICA;NS1 5VIRON;TAL HAZ>DS1 CIVILIANS 9 ! BATTLESPACE1 CYB] ATTACKS1 MALFUNC;NS1 & 'FRIC;N' 9 W> ALL 9TRODUCE ! POSSIBIL;Y T AUTONOM\S SY/EMS W FACE UNANTICIPAT$ SITUA;NS & MAY ACT 9 AN UN9T5D$ FA%ION4 ,2C !Y LACK A BROAD 3TEXTUAL 9TELLIG;E1 OR COMMON S5SE1 ON P> ) HUMANS1 EV5 RELATIVELY SOPHI/ICAT$ ALGORI?MS >E SUBJECT TO FAILURE IF !Y FACE SITUA;NS \TSIDE _! 9T5D$ DESIGN P>AMET]S4 ,! #AE ,BANN+ ,AUTONOM\S ,W1PONS C#FIA COMPLEX;Y ( MOD]N COMPUT]S COMPLICATES ? PROBLEM BY MAK+ X DI6ICULT TO ANTICIPATE ALL "333333 #G ,JAKOB ,KELL5B]G]1 ',KEYNOTE ,ADDRESS'1 ,9T]NA;NAL ,HUMANIT>IAN ,LAW & ,NEW ,W1PON ,TE*NOLOGIES1 #CDTH ,R.D ,TABLE ON ,CURR5T ,ISSUES ( ,9T]NA;NAL ,HUMANIT>IAN ,LAW1 ,SAN ,REMO1 ,ITALY1 #H-#AJ ,SEPTEMB] #BJAA1 PP4 #E-#F1 AVAILABLE AT3 HTTP3_/_/" WWW4IIHL4ORG_/IIHL_/,DOCU;TS_/" ,,JKB,SAN.0#BJ,REMO.0#BJ,SPEE*4PDF4 ------------------------------------#FIB POSSIBLE GLIT*ES OR EM]G5T 2HAVIOR T MAY O3UR 9 A SY/EM :5 X IS PUT 9TO OP]A;N4 ,2C EV5 '>TIFICIALLY 9TELLIG5T' AUTONOM\S SY/EMS M/ 2 PRE-PROGRAMM$1 & W H ONLY HIN+ & ADAPTA;N AT BE/1 X W 2 DI6ICULT OR IMPOSSIBLE TO DESIGN SY/EMS #AF ,BANN+ ,AUTONOM\S ,W1PONS A#FIB CAPABLE ( D1L+ ) ! FOG & FRIC;N ( W>4 ,:5 WE 3SID] ! IMPLICA;NS ( ? = PROTECT+ CIVILIANS 9 >M$ 3FLICT1 ? RAISES S"EAL E?ICAL & LEGAL "QS1 "PICUL>LY 9 RELA;N TO 3=M+ TO ! ,,IHL REQUIRE;TS ( ! PR9CIPLES ( 4T9C;N1 PROPOR;NAL;Y1 & MILIT>Y NECESS;Y1 & ! DI6ICULTY ( E/ABLI%+ RESPONSIBIL;Y & A3.TABIL;Y = ! USE ( LE?AL =CE4 ,AUTONOM\S W1PON SY/EMS RAISE A HO/ ( E?ICAL & SOCIAL 3C]NS1 9CLUD+ ISSUES ( ASYMMETRIC W>F>E & RISK R$I/RIBU;N F COMBATANTS TO CIVILIANS & ! POT5TIAL TO L[] ! ?RE%OLDS = NA;NS TO />T W>S4;9#I ,9SOF> Z S* W1PONS T5D TO REMOVE ! COMBATANTS :O OP]ATE !M F >EA ( 3FLICT & R$UCE ! RISKS ( CASUALTIES = ^? :O POSSESS !M1 !Y T5D TO AL R$UCE ! POLITICAL CO/S & RISKS ( GO+ TO W>4 ,? CD RESULT 9 AN OV]ALL L[]+ ( ! ?RE%OLD ( GO+ TO W>4 ,AUTONOM\S W1PON SY/EMS AL H ! POT5TIAL TO CAUSE REGIONAL OR GLOBAL 9/ABIL;Y & 9SECUR;Y1 TO FUEL >MS RACES1 TO PROLIF]ATE TO NON-/ATE ACTORS1 #AG ,BANN+ ,AUTONOM\S ,W1PONS B#FIB OR 9ITIATE ! ESCALA;N ( 3FLICTS \TSIDE ( HUMAN POLITICAL 9T5;NS4 ,SY/EMS CAPABLE ( 9ITIAT+ LE?AL =CE )\T HUMAN SUP]VI.N CD D S EV5 :5 POLITICAL & MILIT>Y L1D]%IP HAS N DEEM$ S* AC;N APPROPRIATE1 RESULT+ 9 ! UN9T5D$ 9ITIA;N OR ESCALA;N ( 3FLICTS \TSIDE ( DIRECT HUMAN 3TROL4;9#AJ ,?US1 ^! SY/EMS POSE A S]I\S ?R1T TO 9T]NA;NAL /ABIL;Y & ! ABIL;Y ( 9T]NA;NAL BODIES TO MANAGE 3FLICTS4 ,9 T]MS ( ! LEGAL A3EPTABIL;Y ( ^! SY/EMS "U EXI/+ ,,IHL1;9#AA ! PRIM>Y "Q APPE>S TO 2 :E!R AUTONOM\S SY/EMS W 2 ABLE TO SATISFY ! PR9CIPLES ( 4T9C;N & PROPOR;NAL;Y4;9#AB ,GIV5 ! COMPLEX;Y ( ^! "333333 #H ,PAUL ,S*>RE1 ',:Y UNMANN$'1 9 .7,JO9T ,=CE ,QU>T]LY1.' ,ISSUE #FA1 #BND ,QU>T]1 #BJAA1 ;P4 #IB4 #I ,PET] ,AS>O1 ',H[ J CD A ROBOT W> BE8'1 9 ,ADAM ,BRI7LE1 ,KAT9KA ,WAELB]S & ,PHILIP ,A4 ;,E4 ,BREY "<$S">1 .7,CURR5T ,ISSUES 9 #AH ,BANN+ ,AUTONOM\S ,W1PONS C#FIB ,COMPUT+ ,& ,PHILOSOPHY1.' ,,IOS ,PRESS1 ,AM/]DAM1 #BJJH1 PP4 #EJ-#FD1 AVAILABLE AT3 HTTP3_/_/PET]AS>O4ORG_/" WRIT+_/,AS>O.0#BJ,JU/.0#BJ,ROBOT.0#BJ" ,W>4PDF4 #AJ ,BY ANALOGY1 "O %D 3SID] ! /OCK M>KET ',FLA% ,CRA%'1 ( #F ,MAY #BJAJ1 9 : AUTOMAT$ HI<-FREQU5CY TRAD+ SY/EMS ESCALAT$ & A3EL]AT$ A #A1JJJ-PO9T DROP 9 ! ,D[ ,J"OS AV]AGE "<#I.0">1 ! S+LE L>GE/ DROP 9 HI/ORY4 ,SEE .1,WIKIP$IA1 ',FLA% ,CRA%'1 AVAILABLE AT3 HTTP3_/_/EN4WIKIP$IA4ORG_/WIKI_/" ,FLA%.-CRA%4 #AA ,NOEL ,%>KEY1 ',D1? /RIKES F ! SKY3 ! CALCULUS ( PROPOR;NAL;Y'1 9 .7,,IEEE ,TE*NOLOGY & ,SOCIETY ,MAGAZ9E1.' ,VOL4 #BH1 ,NO4 #A1 #BJJI1 PP4 #AF-#AI2 ,NOEL ,%>KEY1 ',SAY+ 8NO60 TO LE?AL AUTONOM\S T>GET+'1 9 .7,J\RNAL ( ,MILIT>Y ,E?ICS1.' ,VOL4 #I1 ,NO4 #D1 #BJAJ1 PP4 #CFI-#CHC2 ,M>KUS ,WAGN]1 ',TAK+ HUMANS \ ( ! LOOP3 IMPLICA;NS = 9T]NA;NAL HUMANIT>IAN LAW'1 9 #AI ,BANN+ ,AUTONOM\S ,W1PONS D#FIB .7,J\RNAL ( ,LAW ,9=MA;N & ,SCI;E1.' ,VOL4 #BA1 #BJAA1 AVAILABLE AT3 HTTP3_/_/PAP]S4SSRN4COM_/SOL#C_/" PAP]S4CFM8AB/RACT.-ID"7#AHGDJCI2" ,MAT!W ,BOLTON1 ,?OMAS ,NA% & ,RI*>D ,MOYES1 ',BAN AUTONOM\S >M$ ROBOTS'1 ,>TICLE#CF4ORG1 #E ,M>* #BJAB1 AVAILABLE AT3 HTTP3_/_/WWW4>TICLE#CF4" ORG_//ATE;TS_/BAN-AUTONOM\S->M$-" ROBOTS4 #AB ,SEE 9 "PICUL>1 ,>TICLES #EA & #EG ( ,ADDI;NAL ,PROTOCOL ,I TO ! ,G5EVA ,3V5;NS ADDRESS ! PROTEC;N ( ! CIVILIAN POPULA;N & PRECAU;NS 9 ATTACK4 ,PROTOCOL ,ADDI;NAL TO ! ,G5EVA ,3V5;NS ( #AB ,AUGU/ #AIDI1 & ,RELAT+ TO ! ,PROTEC;N ( ,VICTIMS ( ,9T]NA;NAL ,>M$ ,3FLICTS1 #H ,JUNE1 #AIGG1 #AABE ,,UNTS #C "<5T]$ 9TO =CE #G ,DECEMB] #AIGH">1 AVAILABLE AT3 HTTP3_/_/WWW#B4OH*R4ORG_/5GLI%_/LAW_/" PROTOCOL#A.-#B4;HTM4 ------------------------------------#FIC SY/EMS1 & \R ABIL;Y TO =ESEE H[ !Y #BJ ,BANN+ ,AUTONOM\S ,W1PONS A#FIC MITICLE #CF ( ,ADDI;NAL ,PROTOCOL ,I1;9#AC & M G5]ALLY H[ TO GOV]N ! 9CR1S+LY RAPID TE*NOLOGICAL 9NOVA;N ( NEW W1PONS & TACTICS4;9#AD ,"! IS A SEP>ATE 3C]N T S* SY/EMS MAY N H AN ID5TIFIABLE OP]ATOR 9 ! S5SE T NO HUMAN 9DIVIDUAL CD 2 HELD RESPONSIBLE = ! AC;NS ( ! AUTONOM\S W1PON SY/EM 9 A GIV5 SITUA;N1 OR T ! 2HAVI\R ( ! SY/EM CD 2 S UNPR$ICTABLE T X WD 2 UNFAIR TO HOLD ! OP]ATOR RESPONSIBLE = :AT ! SY/EM DOES4;9#AE ,S* SY/EMS MI CRIME4 ,X IS ?US 9CR1S+LY IMPORTANT TO HOLD /ATES A3.TABLE = ! DESIGN & USE ( S* SY/EMS1 & TO REGULATE !M AT AN 9T]NA;NAL LEVEL4 ,WE >E AT A JUNCTURE AT : WE M/ DECIDE H[ WE1 Z AN 9T]NA;NAL COMMUN;Y1 W TR1T ^! SY/EMS4 ,W WE TR1T !M Z NEW EXT5.NS ( OLD TE*NOLOGIES1 OR Z A QUALITATIVE %IFT TO A NEW K9D ( TE*NOLOGY8 ,IS CURR5T ,,IHL & ,,IHRL SU6ICI5T TO D1L ) AUTONOM\S LE?AL TE*NOLOGIES1 OR >E !Y 9 NE$ ( M9OR EXT5.NS1 OR MAJOR REVI.NS8 ,IS A BAN ON AUTONOM\S W1PON SY/EMS DESIRABLE1 OR MIY TE*NOLOGY1 PRECISELY 2C !Y ELIM9ATE HUMAN JUDGE;T 9 ! 9ITIA;N ( LE?AL #BB ,BANN+ ,AUTONOM\S ,W1PONS C#FIC =CE4 ,"!=E !Y ?R1T5 TO "UM9E HUMAN "RS 9 ! ABS;E ( HUMAN JUDGE;T & REVIEW4 ,"! >E GD R1SONS TO CL>IFY ,,IHL & ,,IHRL BY EXPLICITLY CODIFY+ A PROHIBI;N ON ! USE ( AUTONOM\S W1PON SY/EMS4 ,MOREOV]1 ^! R1SONS /& UP AG/ ALL ( ! CRITICISMS (F]$ ?US F>4 ,! B5EFITS TO S* A CL>IFICA;N & CODIFICA;N 9CLUDE3 #A"> AVOID+ V>I\S SLIPP]Y SLOPES T[>DS AUTONOM\S W1PON SY/EMS BY DRAW+ A PR9CIPL$ B.D ON :AT C & _C 2 AUTOMAT$2 "333333 #AC ,! FULL TEXT ( ,>TICLE #CF ( ,ADDI;NAL ,PROTOCOL ,I ON ,NEW ,W1PONS R1DS3 ',9 ! /UDY1 DEVELOP;T1 ACQUISI;N OR ADOP;N ( A NEW W1PON1 M1NS OR ME?OD ( W>F>E1 A ,HI< ,3TRACT+ ,"PY IS "U AN OBLIGA;N TO DET]M9E :E!R XS EMPLOY;T WD1 9 "S OR ALL CIRCUM/.ES1 2 PROHIBIT$ BY ? ,PROTOCOL OR BY ANY O!R RULE ( 9T]NA;NAL LAW APPLICABLE TO ! ,HI< ,3TRACT+ ,"PY'4 #BC ,BANN+ ,AUTONOM\S ,W1PONS D#FIC #AD ,RI*>D ;,M4 ,O',ME>A1 '3TEMPOR>Y GOV]N.E >*ITECTURE REG>D+ ROBOTICS TE*NOLOGIES3 AN ASSESS;T'1 9 ,PATRICK ,L91 ,KEI? ,ABNEY & ,GEORGE ,BEKEY1 .1,ROBOT .1,E?ICS1 ,,MIT ,PRESS1 ,CAMBRIDGE ,,MA1 #BJAA1 PP4 #AEI-#AFH4 #AE ,ROB]T ,SP>R[1 ',KILL] ROBOTS'1 9 .7,J\RNAL ( ,APPLI$ ,PHILOSOPHY1.' ,VOL4 #BD1 ,NO4 #A1 #BJJG1 PP4 #FB-#GG4 #AF ;,M4 ,WAGN]1 ABV NOTE #AA4 P4 #E @.<,NOTE #AA APPE>S ON PR9T PAGES C#FIB-D#FIB ABV4@.> ------------------------------------#FID #B"> %AP+ FUTURE 9VE/;TS 9 TE*NOLOGY DEVELOP;T T[>DS M HUMAN-C5TR$ DESIGNS CAPABLE ( 5HANC+ E?ICAL & LEGAL 3DUCT 9 >M$ 3FLICTS2 #C"> /EMM+ ! POT5TIAL = M RADICAL DE/ABILIZA;NS ( ! E?ICAL & LEGAL NORMS GOV]N+ >M$ 3FLICT T ^! NEW TE*NOLOGIES MI E/ABLI%+ ! LEGAL PR9CIPLE T #BD ,BANN+ ,AUTONOM\S ,W1PONS A#FID AUTOMAT$ PROCESSES D N SATISFY ! MORAL REQUIRE;TS ( DUE 3SID]A;N :5 A HUMAN LIFE IS AT /AKE4 ,X WD "!=E 2 DESIRABLE = ! 9T]NA;NAL COMMUN;Y TO MOVE TO E/ABLI% AN 9T]NA;NAL BAN ON AUTONOM\S W1PON SY/EMS ON ! BASIS ( PROTECT+ HUMAN "RS NORMS Z WELL Z O!R NORMS PROTECT+ ! 9DIVIDUAL4 ^7,LE?AL DECI.N-MAK+^' ,9 AN >GU;T T ! USE ( AUTONOM\S W1PON SY/EMS IS MORALLY & LEGALLY IMP]MISSIBLE1 X IS NEC TO ELUCIDATE H[ AUTONOM\S W1PON SY/EMS FAIL TO MEET ! NEC & SU6ICI5T 3DI;NS = P]MISSIBLE KILL+ 9 >M$ 3FLICT4 ,X IS AL NEC TO REF9E ! NO;N ( AN AUTONOM\S W1PON SY/EM4 ,= N[ X IS SU6ICI5T TO DEF9E ! CLASS ( AUTONOM\S W1PON SY/EMS Z ANY AUTOMAT$ SY/EM T C 9ITIATE LE?AL =CE )\T ! SPECIFIC1 3SCI\S1 & DELIB]ATE DECI.N ( A HUMAN OP]ATOR1 3TROLL]1 OR SUP]VISOR4 ,ADMITT$LY1 S* SY/EMS >E N #BE ,BANN+ ,AUTONOM\S ,W1PONS B#FID UNPREC$5T$ 9 ! S5SE T "! >E V>I\S SORTS ( PRECURSORS T H BE5 US$ 9 >M$ 3FLICTS1 9CLUD+ M9ES & O!R VICTIM-ACTIVAT$ TRAPS1 Z WELL Z C]TA9 GUID$ MISSILES & "S AUTOMATIC DEF;E SY/EMS4 ,9DE$1 "! IS A S5SE 9 : ^! SY/EMS >E N !MVS 'W1PONS' S M* Z !Y >E AUTOMAT$ SY/EMS >M )1 OR 9 3TROL (1 W1PONS4 ,!Y ?US PRES5T A *ALL5GE TO TRADI;NAL MODES ( ?"\ REG>D+ W1PONS & >MS 3TROL1 : T5D TO FOCUS ON ! W1PON Z A TOOL OR 9/RU;T1 OR ^U XS DE/RUCTIVE E6ECTS4 ,R1 AUTONOM\S W1PON SY/EMS =CE U TO ?9K 9 T]MS ( 'SY/EMS' T MIIETY ( 3FIGURA;NS ( S5SORS1 9=MA;N PROCESS+1 & W1PONS DEPLOY;T1 & TO FOCUS ON ! PROCESS BY : ! USE ( =CE IS 9ITIAT$4;9#AG ,)9 ! ,,US MILIT>Y "! HAS BE5 A POLICY TO FOLL[ A HUMAN-9-!-LOOP MODEL :5 X COMES TO ! 9ITIA;N ( LE?AL =CE4 ,! PHRASE 'HUMAN-9-!-LOOP' COMES F ! FIELD ( HUMAN FACTORS 5G9E]+1 & 9DICATES T A HUMAN IS AN 9TEGRAL "P ( ! SY/EM4 ,:5 X COMES TO LE?AL =CE1 ! CRUCIAL #BF ,BANN+ ,AUTONOM\S ,W1PONS C#FID SY/EM IS ! "O T 3TA9S ! DECI.N-MAK+ CYCLE 9 : ANY DET]M9A;N TO USE LE?AL =CE IS MADE4 ,9 MILIT>Y J>GON1 ? DECI.N CYCLE IS REF]R$ TO Z ! 'KILL *A9'1 DEF9$ 9 ! ,,US ,AIR ,=CE Z 3TA9+ SIX /EPS3 "333333 #AG ,9 ! LANGUAGE ( ,>TICLE #CF ( ,ADDI;NAL ,PROTOCOL ,I TO ! ,G5EVA ,3V5;NS1 AUTONOM\S W1PON SY/EMS >E SUBJECT TO REVIEW ON ! BASIS ( 2+ A 'NEW W1PON1 M1NS OR ME?OD ( W>F>E'4 ,? IMPLIES T US+ AN EXI/+ APPROV$ W1PON 9 A NEW WAY1 I4E4 ) AUTONOM\S T>GET+ OR FIR+1 IS XF SUBJECT TO REVIEW Z A NEW M1NS OR ME?OD4 ------------------------------------#FIE F9D1 FIX1 TRACK1 T>GET1 5GAGE & ASSESS4;9#AH ,"! HAS BE5 REC5T 4CUS.N ( MOV+ TO A 'HUMAN-ON-!-LOOP' MODEL1 9 : A HUMAN MIS TO CR1TE A MIDDLE POSI;N 2T ! DIRECT HUMAN 3TROL ( ! HUMAN-9-!-LOOP MODEL & AN #BG ,BANN+ ,AUTONOM\S ,W1PONS A#FIE AUTONOM\S W1PONS SY/EM4 ,H["E1 ! CRUCIAL /EP ! DET]M9ES :E!R A GIV5 SY/EM IS AN AUTONOM\S W1PON SY/EM OR N IS :E!R X AUTOMATES EI ! T>GET OR ! 5GAGE /EPS 9DEP5D5TLY ( DIRECT HUMAN 3TROL4 ,WE C ?US DESIGNATE ! CLASS ( SY/EMS CAPABLE ( SELECT+ T>GETS & 9ITIAT+ ! USE ( POT5TIALLY LE?AL =CE )\T ! DELIB]ATE & SPECIFIC 3SID]A;N ( HUMANS Z 2+ 'AUTONOM\S W1PON SY/EMS'4 ,? DEF9I;N RECOGNIZES T ! FUNDA;TAL E?ICAL & LEGAL ISSUE IS E/ABLI%+ ! CAUSAL C\PL+ ( AUTOMAT$ DECI.N-MAK+ TO ! USE ( A W1PON OR LE?AL =CE1 OR 3V]SELY & DEC\PL+ ( HUMAN DECI.N-MAK+ F DIRECTLY 3TROLL+ ! 9ITIA;N ( LE?AL =CE BY AN AUTOMAT$ SY/EM4 ,X IS ! DELEGA;N ( ! HUMAN DECI.N-MAK+ RESPONSIBILITIES TO AN AUTONOM\S SY/EM DESIGN$ TO TAKE HUMAN LIVES T IS ! C5TRAL MORAL ! LEGAL ISSUE4 ,NOTE T 9CLUD+ A HUMAN 9 ! LE?AL DECI.N PROCESS IS A NEC1 B N A SU6ICI5T REQUIRE;T4 ,A LEGITIMATE LE?AL DECI.N PROCESS M/ AL MEET REQUIRE;TS T ! #BH ,BANN+ ,AUTONOM\S ,W1PONS B#FIE HUMAN DECI.N-MAK] 9VOLV$ 9 V]IFY+ LEGITIMATE T>GETS & 9ITIAT+ LE?AL =CE AG/ !M 2 ALL[$ SU6ICI5T "T TO 2 DELIB]ATIVE1 2 SUITABLY TRA9$ & WELL 9=M$1 & 2 HELD A3.TABLE & RESPONSIBLE4 ,X MIGETS & REQUIRES !M T V]IFY ! T>GETS1 & PRESS A BUTTON TO AU?ORIZE 5GAG+ ^? T>GETS ) LE?AL =CE4 ,S* A P]SON MAY 2 NO BETT] ?AN AN AUTOMATON :5 =C$ TO MAKE DECI.NS RAPIDLY )\T "T TO DELIB]ATE1 OR )\T A3ESS TO RELEVANT & SU6ICI5T 9=MA;N ^U : TO MAKE A M1N+;L DECI.N1 OR :5 SUBJECT$ TO EXTREME PHYSICAL & EMO;NAL /RESS4 ,:5 EVALUAT+ ! APPROPRIATE;S ( AN 9DIVIDUAL'S DECI.N1 WE G5]ALLY TAKE S* FACTORS 9TO A3.T1 & WE >E LESS LIKELY TO HOLD !M RESPONSIBLE = DECI.NS MADE "U S* CIRCUM/.ES & = ANY UN9T5D$ 3SEQU;ES T RESULT1 ?\< WE D / HOLD !M A3.TABLE4 ,2C ^! FACTORS DIM9I% ! RESPONSIBIL;Y ( DECI.N-MAK]S1 ! DESIGN & USE ( SY/EMS T 9CR1SE ! #BI ,BANN+ ,AUTONOM\S ,W1PONS C#FIE LIKELIHOOD T DECI.N-MAK+ W H TO 2 D"O "U S* CIRCUM/.ES IS XF IRRESPONSIBLE4 ,I WD SUBMIT T1 :5 VIEW$ F ! P]SPECTIVE ( 5G9E]+ & DESIGN E?ICS1 9T5;NALLY DESIGN+ SY/EMS T LACK RESPONSIBLE & A3.TABLE AG5TS IS 9 & ( XF UNE?ICAL1 IRRESPONSIBLE1 & IMMORAL4 ,:5 X COMES TO E/ABLI%+ ! /&>DS AG/ : WE EVALUATE LE?AL DECI.N-MAK+1 WE %D N 3FUSE ! 3SID]A;NS WE GRANT TO HUMANS ACT+ "U DURESS ) \R ID1LS = S* /&>DS4 ,MOREOV]1 ! FACT T WE C DEGRADE HUMAN P]=M.E 9 S* DECI.NS TO ! LEVEL ( AUTONOM\S SY/EMS DOES N M1N WE %D L[] \R /&>DS ( JUDG+ ^? DECI.NS4 "333333 #AH ,JULIAN ;,C4 ,*1T]1 ',A3EL]AT+ ! KILL *A9 VIA FUTURE UNMANN$ AIRCRAFT'1 ,BLUE ,HORIZONS ,PAP]1 ,C5T] = ,/RATEGY & ,TE*NOLOGY1 ,AIR ,W> ,COLLEGE1 ,APRIL #BJJG1 ;P4 #E1 AVAILABLE AT3 HTTP3_/_/WWW4AU4AF4MIL" _/AU_/AWC_/AWCGATE_/C/_/BH.-*1T]4PDF4 #CJ ,BANN+ ,AUTONOM\S ,W1PONS #FIF ,:ILE ! DETAIL$ LANGUAGE DEF9+ AUTONOM\S W1PON SY/EMS 9 AN 9T]NA;NAL TR1TY W NEC 2 DET]M9$ "? A PROCESS ( NEGOTIA;NS1 ! C5TREPIECE ( S* A TR1TY %D 2 ! E/ABLI%;T ( ! PR9CIPLE T HUMAN LIVES _C 2 TAK5 )\T AN 9=M$ & 3SID]$ HUMAN DECI.N REG>D+ ^? LIVES 9 EA* & E USE ( =CE1 & ANY AUTOMAT$ SY/EM T FAILS TO MEET T PR9CIPLE BY REMOV+ HUMANS F LE?AL DECI.N PROCESSES IS "!=E PROHIBIT$4 ,? PROPOSAL IS NOVEL 9 ! FIELD ( >MS 3TROL 9SOF> Z X DOES N FOCUS ON A "PICUL> W1PON1 B R ON ! MANN] 9 : 1 DECI.N TO USE T W1PON IS MADE4 ,PREVI\S >MS 3TROL TR1TIES H FOCUS$ ON SPECIFIC W1PONS & _! E6ECTS1 OR ! NECESS>ILY 9DISCRIM9ATE NATURE ( A W1PON4 ,A BAN ON AUTONOM\S W1PONS SY/EMS M/ 9/1D FOCUS ON ! DELEGA;N ( ! AU?OR;Y TO 9ITIATE LE?AL =CE TO AN AUTOMAT$ PROCESS N "U DIRECT HUMAN SUP]VI.N & 4CRE;N>Y 3TROL4 #CA ,BANN+ ,AUTONOM\S ,W1PONS A#FIF ^7,! REQUIRE;T = HUMAN JUDGE;T 9 LEGAL KILL+^' ,9 ORD] = ! TAK+ ( A HUMAN LIFE 9 >M$ 3FLICT TO 2 3SID]$ LEGAL X M/ 3=M TO ! REQUIRE;TS ( ,,IHL4 ,9 "PICUL>1 "PIES TO AN >M$ 3FLICT H A DUTY TO APPLY ! PR9CIPLES ( 4T9C;N & PROPOR;NAL;Y4 ,"! HAS BE5 M* 4CUS.N REG>D+ ! ABIL;Y ( AUTONOM\S SY/EMS TO 3=M TO ^! PR9CIPLES4 ,! MO/ AMBITI\S PROPOSAL HAS BE5 T WE MAY 2 ABLE TO PROGRAM AUTONOM\S W1PON SY/EMS 9 S* A WAY T !Y W 3=M TO ! BODY ( ,,IHL1 Z WELL Z TO ! SPECIFIC RULES ( 5GAGE;T "<,,ROE"> & COMM&]'S ORD]S = A GIV5 MIS.N4;9#AI ,BAS$ 9 ! TRADI;N ( 3/RA9T-BAS$ PROGRAMM+1 ! PROPOSAL IS T ,,IHL C 2 TRANSLAT$ 9TO PROGRAMM+ RULES T /RICTLY DET]M9E : AC;NS >E PROHIBIT$ 9 A GIV5 SITUA;N4 ,?US A HYPO!TICAL 'E?ICAL GOV]NOR' CD 5GAGE TO PREV5T AN AUTONOM\S W1PON SY/EM F 3DUCT+ AN AC;N T X DET]M9ES TO 2 EXPLICITLY PROHIBIT$ "U ,,IHL4 ,>K9 FUR!R >GUES T 2C #CB ,BANN+ ,AUTONOM\S ,W1PONS B#FIF AUTONOM\S W1PON SY/EMS CD *OOSE TO SACRIFICE !MVS 9 SITUA;NS ": WE WD N EXPECT HUMANS TO D ! SAME1 ^! SY/EMS MITICLE #EG ( ,ADDI;NAL ,PROTOCOL ,I1 M/ 2 TAK54 ,? 9CLUDES ! OBLIGA;N ( AN ATTACK] TO TAKE ALL F1SIBLE PRECAU;NS 9 ! *OICE ( M1NS & ME?ODS ( ATTACK ) A VIEW TO AVOID+' & 9 ANY EV5T TO "333333 #AI ;,R4 ;,C4 ,>K91 ABV NOTE #C1 PP4 #GA-#IA4 @.<,NOTE #C APPE>S ON #CC ,BANN+ ,AUTONOM\S ,W1PONS C#FIF PR9T PAGES C#FHI-D#FJI ABV4@.> ------------------------------------#FIG M9IMIZ+1 9CID5TAL CIVILIAN CASUALTIES & DAMAGES4 ,9 C]TA9 CASES CYB] OP]A;NS OR ! DEPLOY;T ( REMOTE-3TROLL$ W1PONS OR ROBOTS MI$ TO ! USE ( 3V5;NAL W1PONS4 ,GRT] PRECAU;NS MIE DEPLOY$ F A SAFE 4T.E1 (T5 ) "T TO *OOSE "O'S T>GET C>E;LLY & TO *OOSE ! MO;T ( ATTACK 9 ORD] TO M9IMISE CIVILIAN CASUALTIES & DAMAGE4 ,X MAY 2 >GU$ T 9 S* CIRCUM/.ES ? RULE WD REQUIRE T A COMM&] 3SID] :E!R HE OR %E C A*IEVE ! SAME MILIT>Y ADVANTAGE BY US+ S* M1NS & ME?ODS ( W>F>E1 IF PRACTICABLE4;9#BJ ,:ILE X WD 9DE$ 2 ADVANTAGE\S TO 5H.E ! PROTEC;N ( CIVILIANS & CIVILIAN PROP]TY 9 FUTURE >M$ 3FLICTS1 WE M/ 2 #CD ,BANN+ ,AUTONOM\S ,W1PONS A#FIG C>E;L AB ! 9F];ES WE DRAW F ? ) REG>D TO P]MITT+ ! USE ( AUTONOM\S W1PON SY/EMS4 ,"! >E A GRT _M ASSUMP;NS BUILT 9TO ? SEEM+LY SIMPLE >GU;T1 : MIM$ 3FLICT1 ! ULTIMATE GOAL ( ,,IHL IS TO PROTECT ^? :O >E N1 OR >E NO L;G]1 TAK+ DIRECT "P 9 ! HO/ILITIES1 Z WELL Z TO RE/RICT ! REC\RSE TO C]TA9 M1NS & ME?ODS ( W>F>E4 ,X IS TEMPT+ TO ?9K T ? C 2 OBJECTIVELY & /RAIDLY M1SUR$4 ,WE MIE;LLY 3SID]$ SET ( GUIDEL9ES = :AT 3/ITUTES 'AN ACT ( DIRECT "PICIPA;N 9 HO/ILITIES'1 & "U : A CIVILIAN IS N AF=D$ ! PROTEC;NS NORMALLY GRANT$ #CE ,BANN+ ,AUTONOM\S ,W1PONS B#FIG TO CIVILIANS "U ,,IHL4;9#BB ,^! GUIDEL9ES SET =? ?REE REQUIRE;TS T M/ 2 SATISFI$ 9 ORD] TO 3CLUDE T A CIVILIAN IS A LEGITIMATE T>GET3 #A"> ?RE%OLD ( H>M1 #B"> DIRECT CAUSA;N1 & #C"> BELLIG]5T NEXUS4 ,EA* IS ELABORAT$ 9 ! ,,ICRC ,GUIDEL9ES1 B = PRES5T PURPOSES A %ORT SUMM>Y % SU6ICE3 ,= A SPECIFIC ACT TO R1* ! .7?RE%OLD ( H>M.' REQUIR$ TO QUALIFY Z DIRECT "PICIPA;N 9 HO/ILITIES1 X M/ 2 LIKELY TO ADV]SELY A6ECT ! MILIT>Y OP]A;NS OR MILIT>Y CAPAC;Y ( A "PY TO AN >M$ 3FLICT4 ,9 ! ABS;E ( MILIT>Y H>M1 ! ?RE%OLD C AL 2 R1*$ ": AN ACT IS LIKELY TO 9FLICT D1?1 9JURY1 OR DE/RUC;N ON P]SONS OR OBJECTS PROTECT$ AG/ DIRECT ATTACK4 ,9 BO? CASES1 ACTS R1*+ ! REQUIR$ ?RE%OLD ( H>M C ONLY "333333 #BJ ;,J4 ,KELL5B]G]1 ABV NOTE #G1 ;P4 #F4 @.<,NOTE #G APPE>S ABV ON PR9T PAGE C#FIA4@.> #CF ,BANN+ ,AUTONOM\S ,W1PONS C#FIG #BA ,9DE$1 "! IS A T5D5CY 9 ! LIT]ATURE ON AUTONOM\S W1PONS TO REF] TO '4CRIM9A;N' R ?AN ! PR9CIPLE ( 4T9C;N1 : 3NOTES ! ,4CRIM9A;N TASK' 9 COGNITIVE PSY*OLOGY & >TIFICIAL 9TELLIG;E4 ,SEE ,NOEL ,%>KEY'S OP9ION NOTE 9 ? VOLUME4 #BB ,NILS ,MEZL]1 .7,9T]PRETIVE ,GUID.E ON ! ,NO;N ( ,DIRECT ,"PICIPA;N 9 ,HO/ILITIES ,"U ,9T]NA;NAL ,HUMANIT>IAN ,LAW1.' ,,ICRC1 ,G5EVA1 #BJJI1 ;P4 #BJ AVAILABLE AT3 HTTP3_/_/WWW4ICRC4ORG_/5G_/ASSETS" _/FILES_/O!R_/ICRC-#JJB-#JIIJ4PDF4 ------------------------------------#FIH AM.T TO DIRECT "PICIPA;N 9 HO/ILITIES IF !Y ADDI;NALLY SATISFY ! REQUIRE;TS ( DIRECT CAUSA;N & BELLIG]5T NEXUS4 444 ,! REQUIRE;T ( .1DIRECT .1CAUSA;N IS SATISFI$ IF EI ! SPECIFIC ACT 9 "Q1 OR A 3CRETE & COORD9AT$ MILIT>Y OP]A;N ( : T ACT 3/ITUTES AN 9TEGRAL "P1 #CG ,BANN+ ,AUTONOM\S ,W1PONS A#FIH MAY R1SONABLY 2 EXPECT$ TO DIRECTLY ,- 9 "O ,CAUSAL /EP ,- CAUSE H>M T R1*ES ! REQUIR$ ?RE%OLD4 ,H["E1 EV5 ACTS MEET+ ! REQUIRE;TS ( DIRECT CAUSA;N & R1*+ ! REQUIR$ ?RE%OLD ( H>M C ONLY AM.T TO DIRECT "PICIPA;N 9 HO/ILITIES IF !Y ADDI;NALLY SATISFY ! ?IRD REQUIRE;T1 T ( BELLIG]5T NEXUS4 444 ,9 ORD] TO MEET ! REQUIRE;T ( .1BELLIG]5T .1NEXUS1 AN ACT M/ 2 SPECIFICALLY DESIGN$ TO DIRECTLY CAUSE ! REQUIR$ ?RE%OLD ( H>M 9 SUPPORT ( A "PY TO AN >M$ 3FLICT & TO ! DETRI;T ( ANO!R4 ,Z A G5]AL RULE1 H>M CAUS$ "<,A"> 9 9DIVIDUAL SELF-DEF;E OR DEF;E ( O!RS AG/ VIOL;E PROHIBIT$ "U ,,IHL1 "<;,B"> 9 EX]CIS+ P[] OR AU?OR;Y OV] P]SONS OR T]RITORY1 "<;,C"> Z "P ( CIVIL UNRE/ AG/ S* AU?OR;Y1 OR "<;,D"> DUR+ 9T]-CIVILIAN VIOL;E LACKS ! BELLIG]5T NEXUS REQUIR$ = A QUALIFICA;N Z DIRECT "PICIPA;N 9 HO/ILITIES4 444 ,APPLI$ 9 3JUNC;N1 ! ?REE #CH ,BANN+ ,AUTONOM\S ,W1PONS B#FIH REQUIRE;TS ( .7?RE%OLD ( H>M1 DIRECT CAUSA;N.' & .1BELLIG]5T .1NEXUS P]MIT A RELIABLE 4T9C;N 2T ACTIVITIES AM.T+ TO DIRECT "PICIPA;N 9 HO/ILITIES & ACTIVITIES :1 AL? O3URR+ 9 ! 3TEXT ( AN >M$ 3FLICT1 >E N "P ( ! 3DUCT ( HO/ILITIES &1 "!=E1 D N 5TAIL LOSS ( PROTEC;N AG/ DIRECT ATTACK4 ,EV5 ": A SPECIFIC ACT AM.TS TO DIRECT "PICIPA;N 9 HO/ILITIES1 H["E1 ! K9D & DEGREE ( =CE US$ 9 RESPONSE M/ COMPLY ) ! RULES & PR9CIPLES ( ,,IHL & O!R APPLICABLE 9T]NA;NAL LAW4;9#BC ,^! GUIDEL9ES REPRES5T AN ATTEMPT TO >TICULATE A M1NS BY : TO DET]M9E :O IS A LEGITIMATE T>GET & :O IS N4 ,& YET ^! >E N EV5 CALL$ RULES ,- !Y >E CALL$ GUIDEL9ES 2C !Y HELP GUIDE A MORAL AG5T "? MULTIPLE LAY]S ( 9T]PRETA;N & JUDGE;T4 ,TO DET]M9E :E!R A SPECIFIC 9DIVIDUAL 9 A SPECIFIC CIRCUM/.E MEETS EA* ( ^! REQUIRE;TS REQUIRES A SOPHI/ICAT$ "U/&+ ( A COMPLEX #CI ,BANN+ ,AUTONOM\S ,W1PONS C#FIH SITUA;N 9CLUD+3 ! TACTICAL & /RATEGIC IMPLICA;NS ( A POT5TIAL H>M1 Z WELL Z ! /ATUS ( O!R POT5TIALLY ?R1T5$ 9DIVIDUALS2 ! NATURE ( CAUSAL /RUCTURES & RELA;NS & DIRECT CAUSAL IMPLICA;NS ( "S"O'S AC;NS2 ! SOCIOCULTURAL & PSY*OLOGICAL SITUA;N 9 : T 9DIVIDUAL'S 9T5;NS & AC;NS QUALIFY Z MILIT>Y AC;NS & N1 = 9/.E1 Z ! EX]CISE ( (FICIAL DUTIES ( AU?OR;Y OR P]SONAL SELF-DEF;E4 ,:AT DOES X R1LLY M1N TO SAY T WE C PROGRAM ! RULES ( ,,IHL 9TO A COMPUT]8 ,IS X SIMPLY A MATT] ( TURN+ LAWS WRITT5 TO GOV]N HUMAN AC;NS 9TO PROGRAMM$ CODES TO 3/RA9 ! AC;NS ( A MA*9E8 ,%D ! NEXT ADDI;NAL PROTOCOL TO ! ,G5EVA ,3V5;NS 2 WRITT5 DIRECTLY 9TO COMPUT] CODE8 ,OR IS "! "S?+ M TO ,,IHL T _C 2 PROGRAMM$8 ,X IS TEMPT+ TO TAKE AN 5G9E]+ APPROA* TO ! ISSUE & VIEW ! DECI.NS & "333333 #BC .1,IDEM41 PP4 #EJ-#FD4 #DJ ,BANN+ ,AUTONOM\S ,W1PONS #FII AC;NS ( A COMBATANT Z A 'BLACK BOX'1 & COMP>E ! HUMAN SOLDI] TO ! ROBOTIC SOLDI] & CLAIM T ! "O T MAKES FEW] MI/AKES AC TO ,,IHL IS ! 'M E?ICAL' SOLDI]4 ,? HAS BE5 A COMMON >GU;T /RATEGY 9 ! HI/ORY ( >TIFICIAL 9TELLIG;E Z WELL4 ,"! >E R1LLY TWO "QS "H1 H["E4 ,! EMPIRICAL "Q IS :E!R A COMPUT]1 MA*9E1 OR AUTOMAT$ PROCESS CD MAKE EA* ( ^! DECI.NS ( LIFE & D1? & A*IEVE "S P]=M.E T IS DEEM$ A3EPTABLE4 ,B ! MORAL "Q IS :E!R A COMPUT]1 MA*9E OR AUTOMAT$ PROCESS "\ TO MAKE ^! DECI.NS ( LIFE & D1? AT ALL4 ,UN.S WE C PROVE 9 PR9CIPLE T A MA*9E %D N MAKE S* DECI.NS1 WE >E LEFT TO WOND] IF OR :5 "S CL"E PROGRAMM]S MITIFICIAL 9TELLIG;E IS 9/RUCTIVE "H1 9SOF> Z X TELLS U T S* PROBLEMS >E1 9 G5]AL1 COMPUTA;NALLY 9TRACTABLE1 B IF WE C V C>E;LLY #DA ,BANN+ ,AUTONOM\S ,W1PONS A#FII RE/RICT & SIMPLIFY ! PROBLEM1 WE MIE ! SORT ( PROBLEMS >TIFICIAL 9TELLIG;E HAS BE5 SU3ESS;L AT1 S* Z *ESS1 ) ! SORT ( PROBLEMS 5C.T]$ 9 APPLY+ ,,IHL REQUIRE;TS4 ,: ,,IHL REQUIRE;TS >E 9 "S S5SE 'RULES'1 !Y >E Q UNLIKE ! RULES ( *ESS 9 T !Y REQUIRE A GRT D1L ( 9T]PRETIVE JUDGE;T 9 ORD] TO 2 APPLI$ APPROPRIATELY 9 ANY GIV5 SITUA;N4 ,MOREOV]1 ! 3TEXT 9 : ! RULES >E 2+ APPLI$1 & ! NATURE & QUAL;Y ( ! AVAILABLE 9=MA;N1 & ALT]NATIVE COMPET+ OR 3FLICT+ 9T]PRETA;NS1 MIY WIDELY F "D TO "D1 EV5 9 ! SAME 3FLICT1 OR EV5 9 ! SAME "D4 ,WE MIGUE T 9TELLIG;E IS UNIQUELY HUMAN1 B IF "O C DEF9E X SPECIFICALLY 51 OR R$UCE X TO A 3CRETE TASK1 !N X MAY 2 POSSIBLE TO PROGRAM A COMPUT] TO D T TASK BETT]4 ,:5 WE D T1 WE >E NECESS>ILY *ANG+ ! DEF9I;N ( 9TELLIG;E BY R$EF9+ A COMPLEX SKILL 9TO ! P]=M.E ( SPECIFIC TASK4 ,P]H X #DB ,BANN+ ,AUTONOM\S ,W1PONS B#FII IS N S IMPORTANT :E!R WE R$EF9E 9TELLIG;E 9 LIBITR>Y8 ,AUTOMAT+ ! RULES ( ,,IHL WD LIKELY "UM9E ! ROLE !Y PLAY 9 REGULAT+ E?ICAL HUMAN 3DUCT4 ,X WD AL EXPLA9 :Y DESIGN]S H S"\ TO KEEP HUMANS-9-!-LOOP = ! PURPOSES ( 4AMBIGUA;N & MORAL EVALUA;N4 ,Z ,SIR ,BRIAN ,BURRIDGE1 COMM&] ( ! ,BRITI% ,ROYAL ,AIR ,=CE 9 ,IRAQ F #BJJC TO #BJJE1 PUTS X3 ,"U ! LAW ( >M$ 3FLICT1 "! REMA9S ! REQUIRE;T TO ASSESS PROPOR;NAL;Y & )9 ?1 "! IS AN EXPECTA;N T ! HUMAN AT ! 5D ( ! DELIV]Y *A9 MAKES ! LA/ #DC ,BANN+ ,AUTONOM\S ,W1PONS C#FII ASSESS;T BY EVALUAT+ ! SITUA;N US+ RA;NAL JUDGE;T4 ,PO/-MOD]N 3FLICTS 3FRONT U 444 ) AMBIGU\S NON-L9E> BATTLESPACES4 ,& ?US1 WE _C TAKE ! HUMAN1 ! ------------------------------------#GJJ COMM&]1 ! ANALY/1 ^? :O WRE/LE ) AMBIGU;Y1 \ ( ! LOOP4 ,! DEBATE AB ! HUMAN-9-!-LOOP GOES WID] ?AN ?AT4;9#BD ,! V NATURE ( ,,IHL1 : 0 DESIGN$ TO GOV]N ! 3DUCT ( HUMANS & HUMAN ORGANIZA;NS 9 >M$ 3FLICT1 PRESUPPOSES T COMBATANTS W 2 HUMAN AG5TS4 ,X IS 9 ? S5SE AN?ROPOC5TRIC4 ,DESPITE ! BE/ EF=TS ( XS AU?ORS TO 2 CLE> & PRECISE1 APPLY+ ,,IHL REQUIRES MULTIPLE LEVELS ( 9T]PRETA;N 9 ORD] TO 2 E6ECTIVE 9 A GIV5 SITUA;N4 ,,IHL SUPPLE;TS XS RULES ) HEURI/IC GUIDEL9ES = HUMAN AG5TS TO FOLL[1 EXPLICITLY REQUIRES COMBATANTS TO REFLEXIVELY 3SID] ! IMPLICA;NS ( _! AC;NS1 & TO APPLY COMPAS.N & JUDGE;T 9 AN EXPLICIT APP1L TO _! HUMAN;Y4 #DD ,BANN+ ,AUTONOM\S ,W1PONS A#GJJ ,9 DO+ ?1 ! LAW DOES N IMPOSE A SPECIFIC CALCULA;N1 B R1 X IMPOSES A DUTY ON COMBATANTS TO MAKE A DELIB]ATE 3SID]A;N Z TO ! POT5TIAL CO/ 9 HUMAN LIVES & PROP]TY ( _! AVAILABLE C\RSES ( AC;N4 ^7,JU/ICE _C 2 AUTOMAT$^' ,LAW IS BY XS ESS5TIAL NATURE 9COMPLETE & SUBJECT TO 9T]PRETA;N & FUTURE REVIEW4 ,H["E C>E;L1 ?"\;L1 & WELL 9T5;N$ A LAW OR RULE MIE A NUMB] ( HUMAN AG5TS ,- JUDGES1 PROSECUTORS1 DEF5D]S1 WIT;SES1 JURIES ,- ALL ( :OM 5GAGE 9 COMPLEX PROCESSES ( 9T]PRETA;N & JUDGE;T TO KEEP ! LEGAL SY/EM ON TRACK4 ,9 %ORT1 !Y >E ACTIVELY 5GAG$ 9 ASSESS+ ! MAT* 2T AN AB/RACT SET ( RULES & ANY GIV5 3CRETE SITUA;N4 ,! "R TO DUE PROCESS IS ESS5TIALLY ! "R TO H S* A DELIB]ATIVE PROCESS MADE PUBLICLY A3.TABLE4 #DE ,BANN+ ,AUTONOM\S ,W1PONS B#GJJ ,WE CD IMAG9E A COMPUT] PROGRAM TO REPLACE ^! HUMAN AG5TS1 & TO AUTOMATE _! DECI.NS4 ,B ?1 ,I 3T5D1 WD FUNDA;TALLY "UM9E ! "R TO DUE PROCESS4 ,T "R IS ESS5TIALLY ! "R TO "Q ! RULES & ! APPROPRIATE;S ( _! APPLICA;N 9 A GIV5 CIRCUM/.E1 & TO MAKE AN APP1L TO 9=M$ HUMAN RA;NAL;Y & "U/&+4 ,D HUMANS 9 ^! POSI;NS "S"TS MAKE MI/AKES8 ,Y1 ( C\RSE !Y D4 ,HUMAN "U/&+1 RA;NAL;Y1 & JUDGE;T EXCE$ ANY 3CEIVABLE SY/EM ( FIX$ RULES OR ANY COMPUTA;NAL SY/EM1 H["E4 ,MOREOV]1 :5 3SID]+ ! >GU;TS 9 A GIV5 CASE1 ! POT5TIAL = APP1LS TO OV]TURN JUDICIAL DECI.NS1 & ! WAYS 9 : OP9IONS & CASE LAW 9=M ! 9T]PRETA;N ( LAWS1 WE M/ RECOGNIZE T MAK+ LEGAL JUDGE;TS REQUIRES 3SID]+ DI6]5T1 9COMPATIBLE1 & EV5 3TRADICTORY P]SPECTIVES1 & DRAW+ 9SIE NO "KN COMPUTA;NAL OR ALGORI?MIC SY/EMS T C D ?1 & X MIF>E3 A P]SPECTIVE ON REC5T OP]A;NS'1 9 .1,,RUSI .1,J\RNAL1 ,VOL4 #ADH1 NO4 #E1 ,OCTOB] #BJJC1 PP4 #AH-#BC4 ------------------------------------#GJA ,M IMPORTANTLY1 HUMAN JUDGE;T IS 3/ITUTIVE ( ! SY/EM ( JU/ICE4 ,T IS1 IF ANY SY/EM ( JU/ICE IS TO APPLY TO HUMANS1 !N X M/ RELY ^U HUMAN R1SON4 ,JU/ICE XF _C 2 DELEGAT$ TO AUTOMAT$ PROCESSES4 ,:ILE ! AUTOMA;N ( V>I\S TASKS 9VOLV$ 9 ADM9I/RATIVE & LEGAL PROCE$+S MAY 5H.E ! ABIL;Y OR E6ICI5CY ( HUMANS TO MAKE _! JUDGE;TS1 X _C ABROGATE _! DUTY TO 3SID] ! EVID;E1 DELIB]ATE ALT]NATIVE 9T]PRETA;NS1 & R1* AN 9=M$ OP9ION4 ,MO/ EF=TS AT AUTOMAT+ ADM9I/RATIVE JU/ICE H N IMPROV$ ^U HUMAN P]=M.E1 9 FACT1 B H GRTLY DEGRAD$ IT4;9#BE ,TO AUTOMATE ^! ESS5TIAL ASPECTS ( HUMAN JUDGE;T 9 ! JUDICIAL PROCESS WD 2 TO DEHUMANIZE JU/ICE1 & "\ TO 2 REJECT$ 9 PR9CIPLE4 #DG ,BANN+ AUTONOM\S ,W1PONS A#GJA ,9 SAY+ T ! AUTOMA;N ( HUMAN R1SON+ 9 ! PROCESSES ( JU/ICE "\ TO 2 REJECT$ 9 PR9CIPLE1 ,I M1N T "! IS NO AUTOMAT$ SY/EM1 & NO M1SURE ( P]=M.E T S* A SY/EM CD R1*1 T WE %D A3EPT Z A REPLACE;T = A HUMAN4 ,9 %ORT1 :5 X COMES TO A SY/EM ( JU/ICE1 OR ! /ATE1 OR _! AG5TS1 MAK+ DET]M9A;NS REG>D+ ! HUMAN "RS ( AN 9DIVIDUAL1 ! ULTIMATE AG5TS & (FICIALS ( ! /ATE M/ !MVS 2 HUMAN4 ,"O CD >GUE = ? PR9CIPLE ON MORAL GR.DS1 Z WELL Z ON ! LEGAL GR.DS T X IS 3/ITUTIVE (1 & ESS5TIAL TO1 ! SY/EM ( JU/ICE XF 9DEP5D5TLY ( XS MORAL /&+4 ,)9 ! MILIT>Y "! >E _M LAY]S ( DELEGAT$ AU?OR;Y1 F ! COMM&]-9-*IEF D[N TO ! PRIVATE1 B AT EA* LAY] "! IS A RESPONSIBLE HUMAN TO BE> BO? ! AU?OR;Y & RESPONSIBIL;Y = ! USE ( =CE4 ,! NATURE ( COMM& RESPONSIBIL;Y DOES N ALL[ "O TO ABDICATE "O'S MORAL & LEGAL OBLIGA;NS TO DET]M9E T ! USE ( =CE IS APPROPRIATE 9 A GIV5 SITUA;N4 ,"O MI Z AUTONOM\S W1PON SY/EMS >E N RESPONSIBLE HUMAN AG5TS1 "O _C DELEGATE ? AU?OR;Y TO !M4 ,9 ? S5SE1 ! PR9CIPLE ( 4T9C;N C 2 SE5 N SIMPLY Z FOLL[+ A RULE T SORTS \ COMBATANTS F CIVILIANS1 B AL ( GIV+ 3SID]A;N TO HUMAN LIVES T MID1 X IS NEC = A HUMAN 2+ TO MAKE AN 9=M$ DECI.N 2F T LIFE C 2 TAK54 ,? IS M OBVI\S 9 PROPOR;NAL;Y DECI.NS 9 : "O M/ WEI< ! VALUE ( HUMAN LIVES1 CIVILIAN & COMBATANT1 AG/ ! VALUES ( MILIT>Y OBJECTIVES4 ,N"O ( ^! >E FIX$ VALUES1 & 9 "S WAYS ^! VALUES >E SET BY ! V MORAL DET]M9A;NS T G 9TO MAK+ PROPOR;NAL;Y JUDGE;TS4 ,? IS :Y WE _C CLAIM T AN AUTONOM\S W1PON SY/EM WD 2 MORALLY SUP]IOR TO A HUMAN SOLDI] ON ! BASIS T X MIY T>GETS T OPTIMALLY M9IMIZE ! RISK ( 4PROPOR;NATE H>MS4 ,? IS N TO SAY T ^! GOALS >E N DESIRABLE4 ,IF TE*NOLOGIES DID EXI/ "333333 #BE ,DANIELLE ,K1TS ,CITRON1 ',TE*NOLOGICAL DUE PROCESS'1 9 .7,WA%+TON ,UNIV]S;Y ,LAW ,REVIEW1.' ,VOL4 #HE1 #BJJH1 PP4 #ABDI-#ABIB4 ------------------------------------#GJB T CD 4T+UI% CIVILIANS F COMBATANTS BETT] ?AN ANY HUMAN1 OR BETT] ?AN ! AV]AGE COMBATANT1 !N ^? TE*NOLOGIES %D 2 DEPLOY$ 9 A MANN] TO ASSI/ ! COMBATANT 9 APPLY+ ! PR9CIPLE ( 4T9C;N1 R ?AN US$ TO ELIM9ATE HUMAN JUDGE;T4 ,SIMIL>LY1 IF A TE*NOLOGY 7 CAPABLE ( DET]M9+ A C\RSE ( AC;N : CD A*IEVE A MILIT>Y OBJECTIVE ) M9IMAL COLLAT]AL DAMAGE1 & M9IMIZE ANY 4PROPOR;NATE H>MS1 !N T TE*NOLOGY CD 2 EMPLOY$ BY A HUMAN COMBATANT *>G$ ) ! DUTY ( MAK+ AN 9=M$ *OICE TO 9ITIATE ! USE ( LE?AL =CE 9 T SITUA;N4 ,ANY AUTOMAT$ PROCESS1 H["E GD X #EJ ,BANN+ ,AUTONOM\S ,W1PONS A#GJB MILY TE*NOLOGICALLY REQUIR$ = ! =ESEEABLE FUTURE 2C AUTONOM\S SY/EMS W N R1* HUMAN LEVELS ( P]=M.E = "S "T TO COME4 ,B M IMPORTANTLY1 ? IS A MORAL REQUIRE;T &1 9 _M IMPORTANT 9/.ES1 A LEGAL REQUIRE;T4 ,I "!=E ASS]T T 9 G5]AL "! IS A DUTY N TO P]MIT AUTONOM\S SY/EMS TO 9ITIATE LE?AL =CE )\T DIRECT HUMAN SUP]VI.N & 3TROL4 ,"! >E TWO BASIC /RATEGIES = >GU+ T AUTONOM\S W1PONS SY/EMS MI COMP>$ TO CURR5T M1NS ( >M$ 3FLICT4 ,"! >E _M V>IA;NS ( ! >GU;T1 : ,I DIVIDE 9TO TWO CLASSES3 #A"> PRAGMATIC >GU;TS PO9T+ TO FAILURES ( LE?AL DECI.N-MAK+ 9 >M$ 3FLICT & >GU+ TO POSSIBLE_/" HYPO!TICAL TE*NOLOGICAL IMPROVE;TS "? AUTOMAT+ ^! DECI.NS1;9#BF & #B"> >GU+ T 9SOF> Z S* SY/EMS IMPLY A R$UC$ #EA ,BANN+ ,AUTONOM\S ,W1PONS B#GJB RISK TO COMBATANTS &_/OR CIVILIANS 9 G5]AL1 Z M1SUR$ BY FEW] CASUALTIES1 "! IS A MORAL IMP]ATIVE TO USE !M4 ,S* >GU;TS H BE5 MADE = PRECI.N W1PONS 9 ! PA/1;9#BG & M REC5TLY = ,PR$ATOR DR"OS & REMOTE-OP]AT$ LE?AL;Y4;9#BH ,>E M PRECISE W1PONS M 'MORAL' ?AN LESS PRECISE W1PONS8 ,X IS EASY 5 TO >GUE T GIV5 ! *OICE 2T ATTACK+ A MILIT>Y T>GET ) A PRECI.N-GUID$ MUNI;N ) L[ RISK ( COLLAT]AL DAMAGE1 & ATTACK+ ! SAME T>GET BY C>PET BOMB+ ) A HI< RISK OR C]TA9TY ( GRT COLLAT]AL DAMAGE1 "O "\ TO *OOSE ! PRECI.N-GUID$ MUNI;N4 ,T IS ! MORAL & LEGAL *OICE TO MAKE1 ALL O!R ?+S 2+ EQUAL4 ,( C\RSE1 "! IS Q A BIT T MIGET1 B ! W1PON IS N E?ICALLY 9DEP5D5T ( T *OICE4 ,& ULTIMATELY X IS ! HUMAN AG5T :O *OOSES TO USE ! W1PON T IS JUDG$ TO 2 MORAL OR N4 ,EV5 ! MO/ PRECISE W1PON C 2 #EB ,BANN+ ,AUTONOM\S ,W1PONS C#GJB US$ ILLEGALLY & IMMORALLY4 ,ALL T PRECI.N AF=DS IS A POSSIBIL;Y = M E?ICAL 2HAVI\R ,- X DOES N DET]M9E OR GU>ANTEE X4 "333333 #BF ,RONALD ;,C4 ,>K91 ',GOV]N+ LE?AL 2HAVIOR3 EMB$D+ E?ICS 9 A HYBRID DELIB]ATIVE_/R1CTIVE ROBOT >*ITECTURE'1 ,GEORGIA ,9/ITUTE ( ,TE*NOLOGY1 ,TE*NICAL ,REPORT ,,GUT-,,GVU-#JG-#AA1 #BJJG1 ;P4 #AA4 #BG ,HUMAN ,"RS ,WAT*1 ',9T]NA;NAL HUMANIT>IAN LAW ISSUES 9 ! POSSIBLE ,U4,S4 9VA.N ( ,IRAQ'1 9 .1,LANCET1 #BJ ,FEBRU>Y #BJJC4 #BH ,BRADLEY ,JAY ,/RAWS]1 ',MORAL PR$ATORS3 ! DUTY TO EMPLOY UN9HABIT$ A]IAL VEHICLES'1 9 .7,J\RNAL ( ,MILIT>Y ,E?ICS1.' ,VOL4 #I1 ,NO4 #D1 #BJAJ1 PP4 #CDB-#CFH4 ------------------------------------#GJC ,? MAY SEEM L A SEMANTIC >GU;T1 B X IS A CRUCIAL 4T9C;N4 ,WE D N ABROGATE #EC ,BANN+ ,AUTONOM\S ,W1PONS A#GJC \R MORAL RESPONSIBILITIES BY US+ M PRECISE TE*NOLOGIES4 ,B Z ) O!R AUTOMAT$ SY/EMS1 S* Z CRUISE 3TROL OR AUTOPILOT1 WE / HOLD ! OP]ATOR RESPONSIBLE = ! SY/EM !Y >E OP]AT+1 ! ULTIMATE DECI.N TO 5GAGE ! AUTOMAT$ SY/EM OR TO 45GAGE X1 & ! APPROPRIATE;S ( ^! *OICES4 ,9DE$1 9 MO/ CASES ^! TE*NOLOGIES1 Z WE H SE5 9 ! USE ( PRECI.N-GUID$ MUNI;NS & >M$ DR"OS1 ACTUALLY 9CR1SE \R MORAL BURD5 TO 5SURE T T>GETS >E PROP]LY SELECT$ & CIVILIANS >E SP>$4 ,& 9DE$1 Z \R TE*NOLOGIES 9CR1SE 9 SOPHI/ICA;N1 WE %D DESIGN !M S Z TO 5H.E \R MORAL 3DUCT4 ,"! IS "S?+ PR(.DLY ODD AB CLAIM+ TO IMPROVE ! MORAL;Y ( W>F>E BY AUTOMAT+ HUMANS \ ( X ALT1 OR AT L1/ BY AUTOMAT+ ! DECI.NS TO USE LE?AL =CE4 ,! RHETORICAL /RATEGY ( ^! >GU;TS IS TO PO9T \ ! MORAL %ORTCOM+S ( HUMANS 9 W> ,- ACTS ( DESP]A;N & FE>1 MI/AKES MADE "U /RESS1 DURESS1 & 9 ! FOG ( W>4 ,! NEXT MOVE IS TO APP1L TO A TE*NOLOGICAL SOLU;N T MIE SEE+4 ,F/1 T BY REMOV+ SOLDI]S F ! IMM RISKS ( W>1 : TELE-OP]AT$ SY/EMS D )\T AUTOMAT+ LE?AL DECI.NS1 WE C AL AVOID _M ( ^! PSY*OLOGICAL PRESSURES & ! MI/AKES !Y CAUSE4 ,SECOND1 IF "! 7 AN AUTOMAT$ SY/EM T CD \TP]=M HUMANS 9 4CRIM9A;N TASKS1 OR PROPOR;NAL;Y CALCULA;NS1 X CD J Z EASILY 2 US$ Z AN ADVISORY SY/EM TO ASSI/ & 9=M HUMAN DECI.N-MAK]S1 & NE$ N 2 GIV5 ! AU?OR;Y TO 9ITIATE LE?AL =CE 9DEP5D5TLY ( 9=M$ HUMAN DECI.NS4;9#BI ^7,>GU;TS AG/ BANN+ AUTONOM\S W1PON SY/EMS^' ,9 A REC5T POLICY BRIEF1 ,&]SON & ,WAXMAN (F] A CRITICISM ( PROPOSALS TO BAN AUTONOM\S W1PON SY/EMS4;9#CJ ,!Y 3CLUDE T :ILE X IS IMPORTANT TO E/ABLI% 9T]NA;NAL NORMS REG>D+ ! USE ( AUTONOM\S W1PON SY/EMS1 A BAN IS N ! BE/ WAY #EE ,BANN+ ,AUTONOM\S ,W1PONS C#GJC TO D X4 ,"! >E1 H["E1 NUM]\S PROBLEMS ) _! >GU;T & _M ( _! 3CLU.NS4 ,! MA9 ?RU/ ( _! >GU;T IS BAS$ 9 TWO ASSUMP;NS3 ,RECOGNIZ+ ! 9EVITABLE B 9CRE;TAL EVOLU;N ( ^! TE*NOLOGIES IS KEY TO ADDRESS+ ! LEGAL & E?ICAL DILEMMAS ASSOCIAT$ ) !M2 ,,US POLICY T[>D RESOLV+ ^? DILEMMAS %D 2 BUILT ^U ^! ASSUMP;NS4 ,! C]TA9 YET GRADUAL DEVELOP;T & DEPLOY;T ( ^! SY/EMS1 Z WELL Z ! HUMANIT>IAN ADVANTAGES CR1T$ BY ! PRECI.N ( "S SY/EMS1 "333333 #BI ,PET] ,AS>O1 ',MODEL+ ! MORAL US]3 DESIGN+ E?ICAL 9T]FACES = TELE-OP]A;N'1 9 .7,,IEEE ,TE*NOLOGY @& ,SOCIETY1.' ,VOL4 #BH1 ,NO4 #A1 #BJJI1 PP4 #BJ-#BD1 AVAILABLE AT3 HTTP3_/_/" PET]AS>O4ORG_/WRIT+_/,AS>O.0#BJ" ,MODEL+.0#BJ,MORAL.0#BJ,US]4PDF4 #CJ ;,K4 ,&]SON & ;,M4 ;,C4 ,WAXMAN1 ABV NOTE #C1 ;P4 #AC4 @.<,NOTE #C APPE>S ABV ON PR9T PAGES #EF ,BANN+ ,AUTONOM\S ,W1PONS D#GJC C#FHI-D#FHI4@.> ------------------------------------#GJD MAKE "S PROPOS$ RESPONSES ,- S* Z PROHIBITORY TR1TIES ,- UN"WABLE Z WELL Z E?ICALLY "QABLE4;9#CA ,"H WE SEE S"EAL >GU;TS 2+ MADE AG/ ! PROPOSAL = AN 9T]NA;NAL PROHIBITORY TR1TY4 ,F/1 !Y 9SI/ ^U />T+ F ! ASSUMP;NS T ^! TE*NOLOGIES >E 9EVITABLE1 & T _! DEVELOP;T IS 9CRE;TAL4 ,YET !Y PROVIDE NO EVID;E OR >GU;TS TO SUPPORT EI ASSUMP;N1 EV5 ?\< "! >E /R;G R1SONS TO REJECT !M4 ,!Y !N MAKE A FUR!R >GU;T T "S ( ^! SY/EMS MAY H HUMANIT>IAN ADVANTAGES1 & ?US PROHIBI;NS >E BO? 'UN"WABLE' & 'E?ICALLY "QABLE'4 ,HAV+ J EXPLA9$ :Y X IS N 'E?ICALLY "QABLE' TO >GUE T EV5 ! MO/ PRECISE AUTONOM\S W1PON SY/EMS JEOP>DIZE HUMAN "RS1 ,I WANT TO FOCUS ON _! TWO PRELIM9>Y ASSUMP;NS & :AT !Y MIE AUTONOM\S W1PON SY/EMS 9EVITABLE8 ,:Y %D WE ASSUME T AUTONOM\S W1PON SY/EMS >E 9EVITABLE8 ,:AT MI;Y ( S* CLAIMS IS L>GELY DUE TO ! RETROSPECTIVE "* ( HI/ORY1 & APPLY+ \R "U/&+ ( PA/ TE*NOLOGIES TO ?9K+ AB ! FUTURE4 ,T IS1 X SEEMS EASY = U1 LOOK+ BACK1 TO SAY T ! 9V5;N ( ! LID TO IMAG9E :AT ! _W WD 2 L IF X _H N HAPP5$4 ,YET :5 "O LOOKS C>E;LLY AT ! HI/ORICAL DETAILS1 :E!R A TE*NOLOGY SU3E$$ TE*NOLOGICALLY 0 9 MO/ CASES HIIETY ( FACTORS4 ,9 MO/ CASES1 ! ADOP;N ( ! TE*NOLOGY 0 N GU>ANTE$ BY ! SU3ESS ( ! 9NOVA;N1 & ! M1NS & MANN] ( XS EV5TUAL USE ALW #EH ,BANN+ ,AUTONOM\S ,W1PONS B#GJD DEP5D$ ^U A GRT V>IETY ( SOCIAL & CULTURAL =CES4 ,9DE$1 :5 WE LOOK AT ! GRT _M TE*NOLOGICAL FAILURES1 & 9DE$ ! _M FAIL$ ATTEMPTS TO COMM]CIALIZE ! LI IS T V FEW1 IF ANY1 TE*NOLOGIES C FAIRLY 2 CLAIM$ TO 2 '9EVITABLE'4 ,& EV5 ! SU3ESS;L LIE J Z DYNAMIC & UNPR$ICTABLE4 ,P]H :AT ,&]SON & ,WAXMAN M1N IS T X SEEMS V LIKELY T ^! TE*NOLOGIES W 2 DEVELOP$4 ,? SEEMS M PLAUSIBLE4 ,9DE$1 SIMPLI/IC SY/EMS C ALR IMPLE;T ! ESS5TIAL ELE;TS ( AN AUTONOM\S W1PON SY/EM1 ?\< ^! WD FAIL TO MEET ! EXI/+ 9T]NA;NAL LEGAL /&>DS ( 4CRIM9A;N & PROPOR;NAL;Y4;9#CB ,B EV5 IGNOR+ ! EXI/+ LEGAL "333333 #CA .1,IDEM41 ;P4 #B4 #EI ,BANN+ ,AUTONOM\S ,W1PONS C#GJD #CB ;,M4 ,WAGN]1 ABV NOTE #AA1 PP4 #E-#I4 @.<,NOTE #AA APPE>S ABV OR PR9T PAGES C#FIB-D#FIB4@.> ------------------------------------#GJE LIMITA;NS1 ! FACT T WE C BUILD AUTONOM\S LE?AL TE*NOLOGIES DOS N M1N WE W USE !M4 ,GIV5 T V>I\S SORTS ( AUTONOM\S W1PON SY/EMS >E ALR POSSIBLE1 X MIE C]TA9LY "S /R;G MOTIVA;NS = ADOPT+ S* TE*NOLOGIES1 9CLUD+ ! DESIRE TO R$UCE ! RISKS TO MILIT>Y P]SONNEL1 Z WELL Z R$UCE ! CO/S & NUMB] ( P NE$$ = V>I\S MILIT>Y OP]A;NS & CAPABILITIES4 ,OR M /R;GLY1 ,&]SON & ,WAXMAN MIE CAPABLE ( MEET+ ! REQUIRE;TS ( "S M1SURE ( 4CRIM9A;N & PROPOR;NAL;Y4 ,B ? IS AN EMPIRICAL CLAIM1 AB ! CAPABILITIES ( TE*NOLOGIES T D N YET EXI/1 2+ #FJ ,BANN+ ,AUTONOM\S ,W1PONS A#GJE M1SUR$ AG/ A METRIC T DOES N YET EXI/4 ,Z A PURELY EMPIRICAL "Q1 ^! TE*NOLOGIES MAY OR MAY N COME 9TO EXI/;E & WE MAY N EV5 2 ABLE TO AGREE ^U A3EPTABLE METRICS = EVALUAT+ _! P]=M.E1 S :Y %D WE 2LIEVE T !Y >E 9EVITABLE8;9#CC ,! CRUCIAL "Q "H IS :E!R ^! TE*NOLOGIES C MEET ! REQUIRE;TS ( 9T]NA;NAL LAW1 & ? IS F> F C]TA94 ,! >GU;TS CLAIM+ ! E?ICAL SUP]IOR;Y ( ROBOTIC SOLDI]S S.D SUSPICI\SLY L CLAIMS F ! E>LY "DS ( >TIFICIAL 9TELLIG;E T COMPUT]S WD "S"D B1T HUMAN GR&MA/]S AT *ESS4 ,&1 =TY YE>S LAT] ?AN 9ITIAL PR$IC;NS1 ,,IBM'S ,DEEP ,BLUE DID MANAGE TO B1T ,G>Y ,KASP>OV4 ,B "! >E IMPORTANT DI6];ES 2T *ESS & ,,IHL T >E WOR? NOT+4 ,*ESS IS A FAIRLY WELL-DEF9$ RULE-BAS$ GAME T IS SUSCEPTIBLE TO COMPUTA;NAL ANALYSIS4 ,ULTIMATELY1 ! GAME ( *ESS IS N A MATT] ( 9T]PRETA;N1 NOR IS X A MATT] ( SOCIAL NORMS4 ,9T]NA;NAL LAW1 :ILE X HAS RULES1 IS N L *ESS4 ,LAW ALW REQUIRES 9T]PRETA;N & JUDGE;T 9 #FA ,BANN+ ,AUTONOM\S ,W1PONS B#GJE ORD] TO APPLY X TO R1L _W SITUA;NS4 ,^! 9T]PRETA;NS & JUDGE;TS >E AID$ BY HI/ORICAL PREC$5TS & E/ABLI%$ /&>DS1 B !Y >E N /RICTLY DET]M9$ BY !M4 ,! BODY ( CASE LAW1 PROC$URES1 >GU;TS1 & APP1LS IS ABLE TO DEF5D OLD PR9CIPLES OR E/ABLI% NEW PREC$5TS1 & "!BY E/ABLI% NORMS & PR9CIPLES1 EV5 Z ^? NORMS & PR9CIPLES 3T9UE TO GR[ 9 M1N+ OV] "T4 ,?US1 9SI/+ T AUTONOM\S W1PON SY/EMS >E 9EVITABLE IS ACTUALLY Q P]NICI\S4 ,ON ! "O H&1 ? ASSUMP;N WD MAKE ! E/ABLI%;T ( A BAN SEEM AUTOMATICALLY IMPRACTICAL OR UN"WABLE4 ,T IS1 IF WE />T F ! ASSUMP;N T ! BANN$ SY/EMS W EXI/ & W 2 US$1 !N :Y %D WE BO!R TO BAN !M8 ,B ( C\RSE1 !Y D N EXI/ & >E N 2+ US$1 & EV5 IF !Y 7 2+ US$ ALR !Y CD / 2 BANN$ GO+ =W>D4 ,& F> F 2+ UN"WABLE OR IMPRACTICAL1 A BAN CD 2 Q E6ECTIVE 9 %IFT+ 9NOVA;N TRAJECTORIES T[>DS M USE;L & G5U9ELY E?ICAL SY/EMS4 ,X SEEMS "333333 #CC ,= COMP>ISON1 3SID] ELECTRIC #FB ,BANN+ ,AUTONOM\S ,W1PONS C#GJE C>S1 A TE*NOLOGY T HAS EXI/$ = A C5TURY4 ,EV5 ) ! REC5T POPUL>;Y ( HYBRID GAS_/ELECTRIC C>S1 & "S HIS1 FEW P WD 5DORSE ! CLAIM T \R TRANSI;N TO ELECTRIC C>S IS 9EVITABLE4 ,& ? IS A TE*NOLOGY T IS ALR POSSIBLE1 I4E4 X EXI/S4 ------------------------------------#GJF /RAID T WE C DEF9E ! CLASS ( AUTONOM\S W1PON SY/EMS CLE>LY 51 & !N DEBATE H[ A TR1TY MIM\R1 ANTI-BALLI/IC MISSILE DEF;ES1 OR SUP]VISORY SY/EMS4 ,A BAN _C 2 EXPECT$ TO PROHIBIT EA* & E USE ( AUTOMA;N 9 >M$ 3FLICT1 B R TO E/ABLI% AN 9T]NA;NAL NORM T SAYS T X IS ILLEGITIMATE TO USE SY/EMS T MAKE AUTOMAT$ LE?AL DECI.NS4 ,! 9T]NA;NAL BANS ON L&M9ES & CLU/] MUNI;NS MAY H N COMPLETELY ELIM9AT$ L&M9ES & CLU/] MUNI;NS & _! USE 9 >M$ 3FLICT1 B !Y H MADE X M DI6ICULT = MANUFACTUR]S TO PRODUCE !M PR(ITABLY1 & = #FC ,BANN+ ,AUTONOM\S ,W1PONS A#GJF MILIT>IES TO USE !M )\T REP]CUS.NS 9 ! 9T]NA;NAL COMMUN;Y4 ,MOREOV]1 />T+ F AN ASSUMP;N ( ! 9EVITABIL;Y ( AUTONOM\S W1PON SY/EMS APPE>S TO MAKE ! A3EPTABIL;Y ( S* SY/EMS A =EG"O 3CLU.N4 ,YET :AT IS ULTIMATELY AT ISSUE "H IS :AT ! 9T]NA;NAL /&>DS ( A3EPTABIL;Y W 2 ,- :AT ! 9T]NA;NAL COMMUN;Y W 3SID] ! NORMS ( 3DUCT TO BE4 ,TO ASSUME ! 9EVITABIL;Y ( ! DEVELOP;T & USE ( ! TE*NOLOGIES 9 "Q IS TO CLOSE (F FUR!R 4CUS.N ON ! WISDOM & DESIRABIL;Y ( PURSU+1 DEVELOP+1 & US+ ^! TE*NOLOGIES4 ,9 %ORT1 ! DEVELOP;T & USE ( AUTONOM\S W1PON SY/EMS IS N 9EVITABLE ,- NO TE*NOLOGY IS4 ,YES1 !Y >E POSSIBLE2 IF !Y 7 N !N "! WD 2 NO NE$ TO BAN !M1 B _! DEVELOP;TS / REQUIRES GRT 9VE/;T4 ,& EV5 IF WE MAY N 2 ABLE TO PREV5T ! CR1;N ( C]TA9 TE*NOLOGIES1 WE W ALW 2 ABLE TO ASS]T A POSI;N ON ! MORAL & LEGAL A3EPTABIL;Y ( _! USE4 ,X DOES N FOLL[ T SIMPLY 2C A TE*NOLOGY EXI/S1 XS USE IS A3EPTABLE4 #FD ,BANN+ ,AUTONOM\S ,W1PONS B#GJF ,S :AT IF AUTONOM\S W1PON SY/EMS >E DEVELOP+ 9CRE;TALLY8 ,I WANT TO RETURN TO ! SECOND ASSUMP;N 9SI/$ ^U BY ,&]SON & ,WAXMAN1 "NLY T AUTONOM\S W1PONS SY/EMS W DEVELOP '9CRE;TALLY'4 ,:AT IS ? ASSUMP;N M1NT TO D = _! >GU;T8 ,AG1 F ! P]SPECTIVE ( TE*NOLOGICAL DEVELOP;T1 ALL TE*NOLOGIES DEVELOP 9CRE;TALLY 9 "S S5SE4 ,:Y WD ? *ANGE ! WAY 9 : WE ADDRESS _! E?ICAL & LEGAL IMPLICA;NS8 ,P]H ,&]SON & ,WAXMAN >E M]ELY TRY+ TO 4>M ! FE> T SOLDI]S W 2 REPLAC$ BY ROBOTS 9 "S GRT TE*NOLOGICAL L1P4 ,Z _! >GU;T 3T9UES1 X 2COMES CLE>] T :AT !Y H 9 M9D IS T ! TRANSI;N TO AUTONOM\S W1PON SY/EMS W HAPP5 9 9CRE;TAL 'BABY /EPS'1 EA* ( : W 2 C>E;LLY 3SID]$ & SCRUT9IZ$4 ,? IS A R 9V5TIVE 9V].N ( ! SLIPP]Y SLOPE >GU;T4 ,9/1D ( ASS]T+ T ^! TE*NOLOGIES >E DANG]\S 2C !Y 5C\RAGE U TO DELEGATE M & M AU?OR;Y TO AUTOMAT$ SY/EMS1 EV5TUALLY RESULT+ 9 AUTOMAT$ SY/EMS ) ! #FE ,BANN+ ,AUTONOM\S ,W1PONS C#GJF ILLEGITIMATE B .1DE .1FACTO AU?OR;Y TO T>GET & KILL HUMANS1 X >GUES T S* SY/EMS W 2 LEGITIMATE 2C EA* /EP AL;G ! WAY SEEMS A3EPTABLE4 ,!Y APPE> TO >GUE T WE %D A3EPT ! 5D RESULT ( ? L9E ( R1SON+ 2C WE 7 ABLE TO R1* X "? A S]IES ( MORAL ADJU/;TS1 N"O ( : ON XS [N SEEM$ TOO AW;L4 ------------------------------------#GJG ,X WD MAKE M S5SE TO SEE ? Z A SLIPP]Y SLOPE L1D+ U TO A RESULT WE 2LIEVE TO 2 UNA3EPTABLE4 ,? %D L1D U TO LOOK M C>E;LLY = AN "ULY+ PR9CIPLE ^U : WE C /OP ! P]IL\S SLIDE TO AN UNDESIRABLE 3CLU.N4 ,& 9DE$1 "! IS A PR9CIPL$ B.D>Y T WE C E/ABLI% ) REG>D TO AUTONOM\S W1PON SY/EMS4 ,T B.D>Y IS T = ANY SY/EM CAPABLE ( 9ITIAT+ LE?AL =CE1 A HUMAN 2+ NE$S TO 2 M1N+;LLY 9VOLV$ 9 MAK+ ! DECI.N ( :E!R OR N LE?AL =CE W ACTUALLY 2 US$ 9 EA* CASE4 ,& :ILE WE C BLUR ? L9E ) V>I\S TE*NOLOGICAL SY/EMS ( %>$ & SUP]VISORY 3TROL1 WE CD AL DESIGN ^? SY/EMS 9 S* WAYS Z TO MAKE ? L9E #FF ,BANN+ ,AUTONOM\S ,W1PONS A#GJG CLE>]1 & TO MAKE ^? DECI.NS BETT] 9=M$4;9#CD ,ON ! NE$ TO E/ABLI% NORMS ,! 3CLU.NS T ,&]SON & ,WAXMAN DRAW >E WR;G AB ! IMPLICA;NS ( A BAN ON AUTONOM\S W1PON SY/EMS1 B !Y >E CORRECT AB ! SIGNIFIC.E ( ! E/ABLI%;T ( NORMS REG>D+ _! USE & ! NE$ = "S 3/RA9TS3 ,! ,UNIT$ ,/ATES M/ ACT1 H["E1 2F 9T]NA;NAL EXPECTA;NS AB ^! TE*NOLOGIES H>D5 >.D ! VIEWS ( ^? :O WD IMPOSE UNR1LI/IC1 9E6ECTIVE OR DANG]\S PROHIBI;NS OR ^? :O WD PREF] FEW OR NO 3/RA9TS AT ALL4;9#CE ,:AT !Y RECOGNIZE IS T "! IS A NEW MORAL SPACE 2+ OP5$ UP BY ^! TE*NOLOGIES1 & X IS N YET SETTL$ :AT ! 9T]NA;NAL COMMUN;Y W A3EPT Z ! NEW NORMS ( W>F>E 9 ! AGE ( ROBOTICS & AUTOMA;N4 ,!Y >E AL CORRECT T ! ,,US1 Z BO? A SUP] P[] & ! L1D] 9 DEVELOP+ _M ( ^! NEW TE*NOLOGIES1 #FG ,BANN+ ,AUTONOM\S ,W1PONS B#GJG IS 9 A UNIQUE POSI;N TO E/ABLI% ! PREC$5TS & NORMS T W %APE ! FUTURE ( >M$ 3FLICT4 ,M OBVI\SLY1 ,&]SON & ,WAXMAN H N %[N H[ BANN+ AUTONOM\S W1PON SY/EMS IS UNR1LI/IC1 NOR H !Y %[N ANY EVID;E T S* A BAN WD 2 9E6ECTIVE OR IMMORAL4 ,LET U 3SID] EA* 7 ^! CLAIMS 9 TURN4 ,H[ MIMS 3TROL TR1TIES POSE *ALL5GES 9 _! IMPLE;TA;N1 & A BAN ON AUTONOM\S W1PON SY/EMS %D N PROVE EXCEP;NALLY M OR LESS DI6ICULT ?AN O!RS1 & "!=E IS N UNR1LI/IC 9 ? S5SE4 ,OR IS ! CLAIM T X WD 2 POLITICALLY DI6ICULT TO F9D SUPPORT = S* A BAN8 ,9 MY P]SONAL EXP]I;E1 "! >E A GRT _M 9DIVIDUALS1 "PICUL>LY AM;G MILIT>Y (FIC]S & POLICYMAK]S B AL AM;G 5G9E]S & EXECUTIVES 9 ! DEF;E 9DU/RY1 :O WD SUPPORT S* A BAN4 ,MOREOV]1 X IS CLE>1 F MY EXP]I;ES 9 5GAG+ ) ! PUBLIC1 T /R;G MORAL APPREH5.NS AB #FH ,BANN+ ,AUTONOM\S ,W1PONS C#GJG AUTOMAT$ W1PON SY/EMS >E BROAD-BAS$1 Z IS FE> ( ! POT5TIAL RISKS !Y POSE4 ,AT ! "333333 #CD ;,P4 ,AS>O1 ABV NOTE #BI1 PP4 #BJ-#BD4 @.<,NOTE #BI APPE>S ABV ON PR9T PAGE C#GJC4@.> #CE ;,K4 ,&]SON & ;,M4 ;,C4 ,WAXMAN1 ABV NOTE #C1 P4 #B4 @.<,NOTE #C APPE>S ABV ON PR9T PAGES C#FHI-D#FHI4@.> ------------------------------------#GJH V L1/1 A BAN IS N UNR1LI/IC 9 ! S5SE T X MIRANT$ ASSUMP;N T ^! SY/EMS >E 9EVITABLE4 ,IF WE A3EPT T Z A =EG"O 3CLU.N1 !N ATTEMPT+ TO HALT ! 9EVITABLE DOES SEEM UNR1LI/IC4 ,B "! IS NO?+ 9EVITABLE AB AN EM]G+ TE*NOLOGY1 ! CAPABILITIES ( : D N YET EXI/1 & ! NORMS SURR.D+ : H N YET BE5 E/ABLI%$4 ,&]SON & ,WAXMAN AL ANTICIPATE AN OBJEC;N TO AUTONOM\S W1PON SY/EMS #FI ,BANN+ ,AUTONOM\T ,W1PONS A#GJH ON ! BASIS ( A MORAL PR9CIPLE3 ,A SECOND OBJEC;N IS A MORAL "O1 & SAYS T X IS SIMPLY WR;G P] SE TO TAKE ! HUMAN MORAL AG5T 5TIRELY \ ( ! FIR+ LOOP4 ,A MA*9E1 NO MATT] R[ GD1 _C COMPLETELY REPLACE ! PRES;E ( A TRUE MORAL AG5T 9 ! =M ( A HUMAN 2+ POSSESS$ ( A 3SCI;E & ! FACULTY ( MORAL JUDGE;T "4 ,9 T REG>D1 ! TITLE ( ? ESSAY IS DELIB]ATELY PROVOCATIVE 9 PAIR+ 'ROBOT' & 'SOLDI]'1 2C1 ON ? OBJEC;N1 T IS PRECISELY :AT %D N"E 2 ATTEMPT$4 ,? IS A DI6ICULT >GU;T TO ADDRESS1 S9CE X /OPS ) A MORAL PR9CIPLE T "O EI A3EPTS OR DOES NOT4;9#CF ,! OBJEC;N !Y REF] TO DRAWS ^U :AT IS SUPPOS$ TO 2 A SORT ( /&-AL"O PR9CIPLE4;9#CG ,"!=E1 !Y SUPPOSE T "! IS NO JU/IFICA;N = A3EPT+ X A"P F "O'S [N MORAL 9TUI;NS4 ,I WD SUBMIT T ! >GU;TS PRES5T$ ABV 9 ? >TICLE H #GJ ,BANN+ ,AUTONOM\S ,W1PONS B#GJH DEMON/RAT$ T ! MORAL PR9CIPLE = REJECT+ AUTONOM\S W1PON SY/EM IS 9 FACT IMPLICIT )9 ,,IHL "? XS V>I\S AN?ROPOC5TRIC =MULA;NS & REQUIRE;TS4 ,MOREOV]1 X IS IMPLICIT )9 ! V /RUCTURE ( LAW1 ! PROCESSES ( JU/ICE1 & DUE PROCESS 9 "PICUL>4 ,WE REQUIRE ! PRES;E ( A HUMAN Z A LEGAL AG5T1 9DEP5D5T ( ! MORAL REQUIRE;T T !Y 2 MORAL AG5TS4 ,X IS N SIMPLY T ! DECI.N TO KILL IS A WEIBITR>Y1 & "! IS NO WAY TO GU>ANTEE T ! USE ( =CE IS N >BITR>Y )\T HUMAN 3TROL1 SUP]VI.N1 & RESPONSIBIL;Y4 ,X IS ?US IMMORAL TO KILL )\T ! 9VOLVE;T ( HUMAN R1SON1 JUDGE;T1 & COMPAS.N1 & X %D 2 ILLEGAL4 ^1,3CLU.N ,Z A MATT] ( ! PRES]VA;N ( HUMAN MORAL;Y1 DIGN;Y1 JU/ICE1 & LAW WE _C A3EPT AN AUTOMAT$ SY/EM MAK+ ! DECI.N TO TAKE A HUMAN LIFE4 ,& WE %D #GA ,BANN+ ,AUTONOM\T ,W1PONS C#GJH RESPECT ? BY PROHIBIT+ AUTONOM\S W1PON SY/EMS4 ,:5 X COMES "333333 #CF .1,IDEM41 ;P4 #AA4 #CG ;,M4 ,BOLTON1 ;,T4 ,NA% & ;,R4 ,MOYES1 ABV NOTE #AA4 @.<,NOTE #AA APPE>S ABV ON PR9T PAGES C#FIB-D#FIB4@.> ------------------------------------#GJI TO KILL+1 EA* 9/.E IS DES]V+ ( HUMAN ATT5;N & 3SID]A;N 9 LIE1 X IS AL TRUE = MILIT>Y TE*NOLOGIES4 ,:ILE ! DEVELOP;T ( ADV.ED MILIT>Y TE*NOLOGIES DOES N NECESS>ILY IMPLY T !Y W BE1 OR C BE1 US$ M C>E;LLY & E?ICALLY1 T POSSIBIL;Y EXI/S4 ,B NEW CAPABILITIES AL BR+ ) !M A POT5TIAL TO REGRESS 9 E?ICS & MORAL;Y1 R ?AN PROGRESS4 ,ULTIMATELY ! NATURE #GB ,BANN+ ,AUTONOM\S ,W1PONS A#GJI ( \R MORAL PROGRESS 9 ! 3DUCT ( W> DEP5DS ^U \R TE*NOLOGY 9 A DEEP] S5SE ?AN M]ELY 5ABL+ COMBATANTS TO 3DUCT W>S ) FEW] CASUALTIES1 & GOES 2Y ! REQUIRE;TS ( ,,IHL & ,,IHRL4 ,9 *OOS+ ! W1PONS & TACTICS ) : WE 5GAGE 9 >M$ 3FLICT1 WE >E AL MAK+ A MORAL *OICE AB ! _W WE WI% TO LIVE 9 & FIGU;TS T ANY 5D IS EI S DESIRABLE OR UNDESIRABLE T ANY M1NS ( A*IEV+ X >E A3EPTABLE4 ,WE M/ AL AC"KL$GE T ! M1NS BY : WE 5ACT *ANGE 9 ! _W1 OR RESI/ *ANGE1 "!BY 2COME AN ASPECT ( T _W4 ,IF WE TRULY WI% TO BUILD A FUTURE 9 : >M$ 3FLICT IS BO? UNNEC & UNA3EPTABLE1 WE M/ >RIVE "! "? A PROCESS T RAISES \R MORAL /&>DS ) EA* NEW TE*NOLOGICAL 9NOVA;N1 R ?AN BY L[]+ ^? /&>DS4 ,! 9T]NA;NAL COMMUN;Y %D 2G9 4CUS.NS ON ! =MA;N ( A TR1TY TO BAN AUTONOM\S W1PONS SY/EMS4 ,9SOF> Z S* SY/EMS D N YET EXI/1 S* A BAN WD HELP TO #GC ,BANN+ ,AUTONOM\S ,W1PONS B#GJI FOCUS ! DEVELOP;T ( FUTURE MILIT>Y TE*NOLOGIES AWAY F ^! S-CALL$ E?ICAL SY/EMS & T[>DS ! DEVELOP;T ( SY/EMS T C ACTUALLY IMPROVE ! E?ICAL 3DUCT ( HUMANS 9 >M$ 3FLICTS4 ,! CRITICS ( S* A BAN BASE _! CRITICISMS ON UNSUPPORT$ CLAIMS AB ! 9EVITABIL;Y ( ^! TE*NOLOGIES & MISL1D+ CLAIMS AB E?ICALLY 5H.ED TE*NOLOGIES4 ,= Z L;G Z _! POT5TIAL CAPABILITIES REMA9 UNC]TA91 ^! TE*NOLOGIES >E EM]G+ 9 A DYNAMIC FIELD ( E?ICAL & LEGAL NORMS4 ,?\< WE MIS "? ^! HYPO!TICAL AUTONOM\S W1PON SY/EMS1 ! R1L;Y IS T !Y C AL DEGRADE \R 3CEP;NS & /&>DS ( E?ICAL 3DUCT1 & 4TRACT U F DEVELOP+ ! TE*NOLOGICAL 5H.E;T ( HUMAN MORAL R1SON+ BY PURSU+ AN IMPROBABLE TE*NOLOGY T ?R1T5S TO "UM9E \R HUMAN "RS ON A FUNDA;TAL LEVEL4 ,X MI