North Carolina Law Review

April, 1992

Essay

*1231 LEGAL PERSONHOOD FOR ARTIFICIAL INTELLIGENCES

Lawrence B. Solum [FNa1]

Copyright © 1992 by the North Carolina Law Review Association; Lawrence B.

Solum

Could an artificial intelligence become a legal person? As of today, this question is only theoretical. No existing computer program currently possesses the sort of capacities that would justify serious judicial inquiry into the question of legal personhood. The question is nonetheless of some interest. Cognitive science begins with the assumption that the nature of human intelligence is computational, and therefore, that the human mind can, in principle, be modelled as a program that runs on a computer. [FN1] Artificial intelligence (AI) research attempts to develop such models. [FN2] But even as cognitive science has displaced behavioralism as *1232 the dominant paradigm for investigating the human mind, fundamental questions about the very possibility of artificial intelligence continue to be debated. This Essay explores those questions through a series of thought experiments that transform the theoretical question whether artificial intelligence is possible into legal questions such as, "Could an artificial intelligence serve as a trustee?"

What is the relevance of these legal thought experiments for the debate over the possibility of artificial intelligence? A preliminary answer to this question has two parts. First, putting the AI debate in a concrete legal context acts as a pragmatic Occam's razor. By reexamining positions taken in cognitive science or the philosophy of artificial intelligence as legal arguments, we are forced to see them anew in a relentlessly pragmatic context. [FN3] Philosophical claims that no program running on a digital computer could really be intelligent are put into a context that requires us to take a hard look at just what practical importance the missing reality could have for the way we speak and conduct our affairs. In other words, the legal context provides a way to ask for the "cash value" of the arguments. The hypothesis developed in this Essay is that only some of the claims made in the debate over the possibility of AI do make a pragmatic difference, and it is pragmatic differences that ought to be decisive. [FN4]

Second, and more controversially, we can view the legal system as a repository of knowledge--a formal accumulation of practical judgments. [FN5] The law embodies core insights about the way the world works *1233 and how we evaluate it. Moreover, in common-law systems judges strive to decide particular cases in a way that best fits the legal landscape--the prior cases, the statutory law, and the constitution. [FN6] Hence, transforming the abstract debate over the possibility of AI into an imagined hard case forces us to check our intuitions and arguments against the assumptions that underlie social decisions made in many other contexts. By using a thought experiment that explicitly focuses on wide coherence, [FN7] we increase the chance that the positions we eventually adopt will be in reflective equilibrium [FN8] with our views about related matters. In addition, the law embodies practical knowledge in a form that is subject to public examination and discussion. Legal materials are published and subject to widespread public scrutiny and discussion. Some of the insights gleaned in the law may clarify our approach to the artificial intelligence debate. [FN9]

I do not claim in this Essay to have resolved the debate over the possibility of artificial intelligence. My aim is more modest: I am proposing a way of thinking about the debate that just might result in progress. There is some precedent for this project. Christopher Stone brought questions of environmental ethics into focus by asking whether trees should have standing. [FN10] My hope is that the law will be equally *1234 fruitful as a context in which to think about the possibility of AI. The "artificial reason and judgment of law" [FN11] may circumvent the intractable intuitions that threaten to lock the AI debate in dialectical impasse.

Part I of this Essay recounts some recent developments in cognitive science and explores the debate as to whether artificial intelligence is possible. Part II puts the question in legal perspective by setting out the notion of legal personhood. Parts III and IV explore two hypothetical scenarios. Part III examines the first scenario--an attempt to appoint an AI as a trustee. The second scenario, an AI's invocation of the individual rights provisions of the United States Constitution, is the subject of Part IV. The results are then brought to bear on the debate over the possibility of artificial intelligence in Part V. In conclusion, Part VI takes up the question whether cognitive science might have implications for current legal and moral debates over the meaning of personhood.

I. ARTIFICIAL INTELLIGENCE

Is artificial intelligence possible? The debate over this question has its roots at the very beginning of modern thought about the nature of the human mind. It was Thomas Hobbes who first proposed a computational theory of mind: "By ratiocination, I mean computation." [FN12] And it was Rene Descartes who first considered a version of the question whether it would be possible for a machine to think:

For we can easily understand a machine's being constituted so that it can utter words, and even emit some responses to action on it of a corporeal kind, which brings about a change in its organs; for example, if it is touched in a particular part it may ask what we wish to say to it; if in another part it may exclaim that it is being hurt, and so on, but it never happens that it arranges its speech in various ways, in order to reply appropriately to everything that may be said in its presence, as even the lowest type of man can do. [FN13] *1235 Descartes' assertion that no artifact could arrange its words "to reply appropriately to everything that may be said in its presence" remains at the heart of the AI debate.

The events of the past forty years have stretched the limits of our imagination. Digital computers have been programmed to perform an ever wider variety of complex tasks. As I write this Essay using a word processing program, my spelling and grammar are automatically checked by programs that perform tasks thought to require human intelligence not so many years ago. The program Deep Thought has given the second best human chess player a very tough game, and the program's authors predict the program will become the world's chess champion within a few years. [FN14] Expert systems simulate the thinkingof human experts on a wide variety of subjects, from petroleum geology to law. [FN15]

But these events have not resolved the question whether AI is even possible. The contemporary debate [FN16] over that question has centered around Alan Turing's test. [FN17] Turing proposed that the question whether a machine can think be replaced with the following, more operationalized, inquiry. The artifact that is a candidate for having the ability to think shall engage in a game of imitation with a human opponent. Both *1236 the candidate and the human being are questioned by someone who does not know which is which (or who is who)--the questions are asked via teletype. The questions may be on any subject whatsoever. Both the human being and the artifact will attempt to convince the questioner that it or she is the human and the other is not. After a round of play is completed, the questioner guesses which of the two players is the human. Turing suggested we postpone a direct answer to the question whether machines can think; he proposed that we ask instead whether an artifact could fool a series of questioners as often as the human was able to convince them of the truth, about half the time. [FN18] The advantage of Turing's test is that it avoids direct confrontation with the difficult questions about what "thinking" or "intelligence" is. Turing thought that he had devised a test that was so difficult that anything that could pass the test would necessarily qualify as intelligent.

John Searle questioned the relevance of Turing's Test with another thought experiment, which has come to be known as the Chinese Room. [FN19] Imagine that you are locked in a room. Into the room come batches of Chinese writing, but you don't know any Chinese. You are, however, given a rule book, written in English, in which you can look up the bits of Chinese, by their shape. The rule book gives you a procedure for producing strings of Chinese characters that you send out of the room. Those outside the room are playing some version of Turing's game. They are convinced that whatever is in the room understands Chinese. But you don't know a word of Chinese, you are simply following a set of instructions (which we can call a program) based on the shape of Chinese symbols. Searle believes that this thought experiment demonstrates that neither you nor the instruction book (the program) understands Chinese, even though you and the program can simulate such understanding. [FN20]

More generally, Searle argues that thinking cannot be attributed to a computer on the basis of its running a program that manipulates symbols in a way that simulates human intelligence. The formal symbol-manipulations accomplished by the program cannot constitute thinking or understanding *1237 because the program lacks "intentionality"--the ability to process meanings. The shape of a symbol is a syntactic property, whereas the meaning of a symbol is a semantic property. Searle's point is that computer programs respond only to the syntactic properties of symbols on which they operate. [FN21]

This point can be restated in terms of the Chinese Room: (1) the output--coherent Chinese sentences--from the Chinese room seems to respond to the meaning of the input; (2) but the process that goes on inside the Chinese room only involves the shape or syntactic properties of the input; (3) therefore, the process in the Chinese room does not involve understanding. [FN22] Searle generalizes the conclusion of the Chinese room thought experiment by arguing that part of the definition of a program is that it is formal and operates only on syntactic properties. He concludes that no system could be said to think or understand solely on the basis of the fact that the system is running a program that produces output that simulates understanding. [FN23]

Searle's Chinese Room has given rise to a number of replies. [FN24] But *1238 at this point I will leave the debate over the possibility of AI.

First Interlude [FN25]

When Mike was installed in Luna, he was pure thinkum, a flexible logic-- "High-Optional, Logical, Multi-Evaluating Supervisor, Mark IV, Mod. L"--a HOLMES FOUR. He computed ballistics for pilotless freighters and controlled their catapult. This kept him busy less than one percent of time and Luna Authority never believed in idle hands. They kept hooking hardware into him--decision-action boxes to let him boss other computers, bank on bank of additional memories, more banks of associational neural nets, another tubful of twelve-digit random numbers, a greatly augmented temporary memory. Human brain has around ten-to-the-tenth neurons. By third year Mike had better than one and half times that number of neuristors.

And woke up.

Am not going to argue whether a machine can "really" be alive, "really" be self-aware. Is a virus self-aware? Nyet. How about oyster? I doubt it. A cat? Almost certainly. A human? Don't know about you, tovarishch, but I am. Somewhere along the evolutionary chain from macromolecule to human brain awareress crept in. Psychologists assert it happens automatically whenever a brain acquires certain very high number of associational paths. Can't see it matters whether paths are protein or platinum.

--Robert A. Heinlein, The Moon is a Harsh Mistress

II. LEGAL PERSONHOOD

The classical discussion of the idea of legal personhood is found in John Chipman Gray's The Nature and Sources of the Law. [FN26] He began his famous discussion, "In books of the Law, as in other books, and in common speech, 'person' is often used as meaning a human being, but the technical legal meaning of a 'person' is a subject of legal rights and *1239 duties." [FN27] The question whether an entity should be considered a legal person is reducible to other questions about whether or not the entity can and should be made the subject of a set of legal rights and duties. [FN28] The particular bundle of rights and duties that accompanies legal personhood varies with the nature of the entity. Both corporations and natural persons are legal persons, but they have different sets of legal rights and duties. Nonetheless, legal personhood is usually accompanied by the right to own property and the capacity to sue and be sued.

Gray reminds us that inanimate things have possessed legal rights at various times. Temples in Rome and church buildings in the middle ages were regarded as the subject of legal rights. Ancient Greek law and common law have even made objects the subject of legal duties. [FN29] In admiralty, a ship itself becomes the subject of a proceeding in rem and can be found "guilty." [FN30] Christopher Stone recently recounted a twentieth-century Indian case in which counsel was appointed by an appellate court to represent a family idol in a dispute over who should have custody of it. [FN31] The most familiar examples of legal persons that are not natural persons are business corporations and government entities. [FN32]

Gray's discussion was critical of the notion that an inanimate thing might be considered a legal person. After all, what is the point of making a thing-- which can neither understand the law nor act on it--the subject of a legal duty? [FN33] Moreover, he argued that even corporations are reducible to relations between the persons who own stock in them, manage them, and so forth. [FN34] Thus, Gray insisted that calling a legal person a "person" involved a fiction unless the entity possessed "intelligence" and *1240 "will." [FN35] Those attributes are part of what is in contention in the debate over the possibility of AI. [FN36]

III. COULD AN ARTIFICIAL INTELLIGENCE SERVE AS A TRUSTEE?

This case study and the one that follows are intended to illustrate two different sorts of issues in the AI debate. In this first scenario, we explore the issue of competence (of "intelligence" in the sense of capacity to perform complex actions) by posing the question whether an AI could serve as a trustee. The second scenario explores the questions of intentionality and consciousness (of "will" in a sense) by asking whether an AI could claim the more robust rights of legal and moral personhood guaranteed by the Bill of Rights and the Civil War Amendments to the United States Constitution.

A. The Scenario

This first scenario speculates about the legal consequences of developing an expert system capable of doing the things a human trustee can do. [FN37] Imagine such expert systems developing from existing programs that perform some of the component functions of a trustee. For example, the decision to invest in publicly traded stocks is made by a computer program in what is called "program trading," in which the program makes buy or sell decisions based on market conditions. [FN38] Today, one *1241 also can buy a computer program that will automatically issue instructions to pay your regular monthly bills by sending data to a bank or service via modem. It is not difficult to imagine an expert system that combines these functions with a variety of others, in order to automate the tasks performed by the human trustee of a simple trust.

Such a system might evolve in three stages. At stage one, the program aids a human trustee in the administration of a large number of simple trusts. The program invests in publicly traded securities, placing investment orders via modem and electronic mail. The program disburses the funds to the trust beneficiaries via an electronic checking program. Upon being informed of a relevant event, such as the death of a beneficiary, the program follows the instructions of the trust instrument--for example, changing the beneficiary or terminating the trust. The program prepares and electronically files a tax return for the trust. The human trustee operates as do trustees today. The human makes the ultimate decisions on how to invest the funds, although she may rely upon an expert system for advice. She reviews the program's activities to insure that the terms of the trust instrument are satisfied. But the actual performance of the day-to-day tasks is largely automated, carried out by the program without the need of human intervention.

Stage two involves a greater role for the AI. Expert systems are developed that outperform humans as investors in publicly traded securities. Settlors begin to include an instruction that the trustee must follow the advice of the AI when making investment decisions regarding trust assets. [FN39] Perhaps they do this because experience shows that trusts for *1242 which the human overrides the program generally perform less well thanthose in which the program's decision is treated as final. Moreover, trust administration programs become very proficient at analyzing and implementing the terms of trust instruments. There is little or no reason for the human to check the program for compliance. As a consequence, the role of the human trustee diminishes and the number of trusts that one human can administer increases to the thousands or tens of thousands. The human signs certain documents prepared by the program. She charges a fee for her services, but she devotes little or no time to administering any particular trust.

But there may be times when the human being is called upon to make a decision. For example, suppose the trust is sued. Perhaps a beneficiary claims that the trust has not paid her moneys due. Or imagine that an investment goes sour and a beneficiary sues, claiming that the trustee breached the duty of reasonable care and skill. If such events occur with regularity, the trustee will develop a routine for handling them. She might routinely refer such disputes to her attorneys. In time, the expert system is programmed to handle this sort of task as well. It processes the trustee's correspondence, automatically alerting the trustee when a letter threatening suit is received or process is served. The system prepares a report on the relevant trust from its electronic records and produces a form letter for the trustee's signature to be sent to the trust's attorneys. As the capabilities of the expert system grow, the need for the human trustee to make decisions gradually diminishes.

The third stage begins when a settlor decides to do away with the human. Why? Perhaps the settlor wishes to save the money involved in the human's fee. Perhaps human trustees occasionally succumb to temptation and embezzle trust funds. Perhaps human trustees occasionally insist on overriding the program, with the consequence that bad investments are made or the terms of the trust are unmet. What would happen if a settlor attempted to make the program itself the trustee?

Many questions must be answered to give a full description of the third stage of the scenario. For example, who would own the AI? If the AI were assumed to be a legal person, it might hold legal title to the hardware and software that enable it to operate. But we cannot assume that AIs are legal persons at this stage, because that assumption begs the question we are trying to answer. As an interim solution, let us assume that the hardware and software are owned by some other legal person, a corporation for example. [FN40]

*1243 B. The Legal Question

I want to examine this question as a legal question, as a jurisprudential question in the classical sense. What should the law do? The law is not presently equipped to handle such a situation: the question has never come up. The Second Restatement of Trusts provides that natural persons, [FN41] government entities, [FN42] and corporations [FN43] may all serve as trustees. The inclusion of governments and corporations establishes that a trustee need not be a natural person. But this is not decisive, because legal persons such as corporations have boards of directors and chief executive officers who are natural persons. [FN44]

How then should the law answer the question whether an AI can become a legal person and serve as a trustee? The first inquiry, I should think, would be whether the AI is competent to administer the trust. There are many different kinds of duties that can be imposed on a trustee by the terms of a trust. For now, lay aside the question whether an AI would be competent to administer trusts that required complex moral or aesthetic judgments. [FN45] Assume that we are dealing with a trust that gives the trustee very little discretion: the terms provide that the assets may be *1244 invested only in publicly traded securities and the income is to be paid to the beneficiaries, with explicit provision for contingencies such as the death of a beneficiary. [FN46] Further, for the purposes of this discussion, assume that an AI could in fact make sound investments, [FN47] make payments, and recognize events such as the death of a beneficiary that require a change in payment. [FN48]

C. Two Objections

But would these capabilities be sufficient for competency? Consider two possible reasons for answering this question in the negative. The first reason is based on the assertion that an AI could not be "responsible," that is, it could not compensate the trust or be punished in the event that it breached one of its duties: call this the responsibility objection. The second reason for doubting the competency of an AI is that trustees must be capable of making judgments that could be beyond the capacity of any AI: call this the judgment objection.

1. The Responsibility Objection

The responsibility objection focuses on the capability of an AI to fulfill its responsibilities and duties. [FN49] Consider, for example, the duty to exercise reasonable skill and care [FN50] and the corresponding liability for breach of trust. [FN51] We have hypothesized that the AI possesses some capacities; for example, we have assumed that the AI is capable of exercising reasonable skill and care in making investment decisions. But what of the corresponding liability? How could an AI be "chargeable with . . . any loss or depreciation in value of the trust resulting from the breach of *1245 trust," [FN52] such as failing to exercise reasonable skill and care in investing the trust assets? [FN53]

The law currently has a mechanism for assigning liability in the case of a malfunctioning expert system: the manufacturer of the system may be held responsible for product liability. [FN54] But could the AI itself be held liable? There is a way in which an AI might have the capacity to be liable in damages despite its lack of personal assets. The AI might purchase insurance. In fact, it might turn out that an AI could be insured for less than could a human trustee. If the AI could insure, at a reasonable cost, against the risk that it would be found liable for breaching the duty to exercise reasonable care, then functionally the AI would be able to assume both the duty and the corresponding liability.

Some legal liabilities cannot be met by insurance, however. For example, insurance may not be available for the monetary liability that may be imposed for intentional wrongdoing by a trustee. Moreover, criminal liability can be nonmonetary. How could the AI be held responsible for the theft of trust assets? It cannot be jailed. This leads to a more general observation: although the AI that we are imagining could not be punished, all of the legal persons that are currently allowed to serve as trustees do have the capacity to be punished. Therefore, the lack of this capacity on the part of an AI might be thought to disqualify it from serving as a trustee. [FN55]

Answering this objection requires us to consider the reasons for which we punish. [FN56] For example, if the purpose of punishment is deterrence, *1246 the objection could be put aside on the ground that the expert system we are imagining is simply incapable of stealing or embezzling. [FN57] The fact that an AI could not steal or convert trust assets is surely not a reason to say that it is not competent to become a trustee. If anything, it is a reason why AIs should be preferred as trustees.

This argument assumes a deterrence theory of punishment--an oversimplification, to say the least. There are a variety of other theories of punishment that would make the issue more complex. [FN58] One of the classic approaches to punishment theory is based on the notion of desert or just retribution. [FN59] But in what sense could an expert system that failed to live up to its duties as a trustee be said to deserve to be punished? The concept of desert seems to be limited in application to human beings; perhaps it extends to all moral persons. The idea that an expert system for administering trusts could deserve to be punished does not seem to make sense. [FN60] Perhaps this difficulty is illusory. We might want to say *1247 that desert theory does yield a clear outcome when applied to the case of an expert system that malfunctions. Such a system does not deserve to be punished because it lacks the qualities of moral persons that make them deserving.

Another approach to the theory of punishment is based on the educative function of punishment. [FN61] By imposing a sanction on trustees who abuse their position, society communicates to its members the message that the office of trustee carries with it important responsibilities that should not be shirked. The punishment of a computer program, however, would not seem to serve this function. What lesson are we to learn about the responsibility of trustees from a punishment imposed on an expert system? What would even count as punishment? Turning the program off? Once again, however, an argument could be made that the educative theory does provide a clear recommendation for the treatment of an expert trust program that behaves badly: do not punish the program, because any supposed "punishment" will have no educative effect.

As this discussion makes clear, consideration of the punishment of an expert trust administration system raises perplexing questions, especially if we move beyond a simple deterrence theory of punishment. Of course, this is not the place to resolve debates about which theory of punishment is correct.

The bare fact that consideration of the punishment issue raises these difficult questions does point, however, to a deep problem with legal personhood for an expert trust administration system. Our understanding of what it means for a human being to function competently has ties to our views about responsibility and desert, and consideration of these *1248 views leads on to our notions of moral personhood. The simplicity provided by utilitarianism, reflected in a deterrence theory of punishment, might allow us to escape some of these difficulties. But there are certainly reasons to doubt the viability of utilitarianism as a moral theory. Surely, the law does grapple with responsibility and desert when it comes to criminal punishment.

The problem of punishment is not unique to artificial intelligences, however. Corporations are recognized as legal persons and are subject to criminal liability despite the fact that they are not human beings. Further, it is by no means certain that corporations are moral persons, in the sense that they can deserve punishment. Of course, punishing a corporation results in punishment of its owners, but perhaps there would be similar results for the owners of an artificial intelligence.

We have considered the capacity of AIs to satisfy legal liability in two different classes of cases. The first class of cases was exemplified by the duty of trustees to exercise reasonable skill and care. Violations of this duty can be characterized as negligent. In such cases, the major purpose of liability, to compensate the victim, is satisfied if the AI can insure. The second class of cases was exemplified by the potential criminal liability of trustees for criminal wrongdoing. Violations of the criminal law are characteristically intentional. In this case, one of the major purposes of liability, to deter intentional wrongdoing, is simply not at issue--the expert system cannot steal or commit fraud. If we restrict our attention to the deterrent function of punishment, it seems possible that an AI could be responsible in a way that satisfies at least some of the policies underlying the imposition of duties and liabilities on trustees. On the other hand, if we take a broader view of the functions of punishment, the second sort of case becomes murkier.

2. The Judgment Objection

Now consider the judgment objection. The argument is that the capacity of an AI to follow a program, even if that program contains a tremendously elaborate and complex system of rules, is not sufficient to enable the system to make judgments and exercise discretion. [FN62] Three instances of the second objection follow. The first instance focuses on the problem of change of circumstance. The second instance involves the *1249 problem of moral choice. Finally, the third instance focuses on the problem of legal choice.

The first version of the judgment objection involves the problem of change of circumstances. The law provides that a trustee may be required or permitted to deviate from a term of the trust if following the terms would defeat the purpose of the trust due to an unanticipated change in circumstances. [FN63] Take an example offered as an illustration in the Second Restatement of Trusts:

A bequeaths money to B in trust and directs him to invest the money in bonds of the Imperial Russian government. A revolution takes place in Russia and the bonds are repudiated. The court will direct B not to invest in these bonds. [FN64]

What is our expert system to do if it is instructed to invest in securities traded on the New York Stock Exchange and that exchange ceases to exist?

Consider three different responses. First, the terms of trusts for AI administration can be designed to minimize such possibilities. For example, the trustee could be given the option of investing in publicly traded securities on any of the major exchanges; the likelihood that all the major securities exchanges will close is very small. The problem with this line of response is that it does not seem possible, even in principle, to design trust terms that anticipate all possible changes in circumstance.

Second, the terms of the trust could provide for a change of circumstance by specifying that if the AI finds itself unable to carry out the terms of the trust, the trust will be terminated or a new trustee will be substituted for the AI. From the settlor's perspective, the disadvantage of the remote possibility of such termination or substitution may be outweighed by the advantages of making the AI the trustee. But this solution assumes that the AI can recognize the significance of the change in circumstance. We easily can imagine the expert system cheerfully continuing to purchase Imperial Russian bonds, chuckling to itself about the bargain prices. [FN65]

Third, it is possible that an AI would be competent to deal with *1250 many or even all such changes in circumstance. For AIs to have this capability for dealing with novelty, AI researchers will need to solve one of the most difficult problems in cognitivescience, the frame problem. [FN66] The trustee program would need to be able to recognize that the securities markets had been closed, to search out other investment opportunities, and to modify its investment decision procedure to make reasonably prudent investments in the new context. The capacity of AIs for coping with complex novelty is not on the immediate horizon, and this Essay does not address the important questions whether the frame problem can or will be solved. If it is solved, however, then AIs would be able to cope with such changes in circumstance. This same ability would be needed to pass the Turing Test. It is easy to see why: the questioner always could put a hypothetical version of our Imperial Russian bonds question to the two contestants. If the AI could not come up with an answer that indicates human levels of competence, the questioner would be able to ferret it out rather quickly.

A second instance of the judgment objection focuses on the possibility that no formal system could adequately make the moral choices with which a trustee may be confronted. Take a simple trust, the terms of which provide for the payment of income to a lifetime beneficiary and principal to another party upon the lifetime beneficiary's death. The law of trusts imposes a duty of impartiality among beneficiaries. [FN67] What does this duty require when the lifetime beneficiary has an unexpected need for income that can be realized at the cost of diminished growth in the principal? [FN68] How would an AI make the moral judgment that seems required to implement a duty that implicitly requires a sense of fairness? Initially, some limits on these questions need to be observed. Some trusts simply will not pose the impartiality problem: for example, trusts with a single beneficiary. Further, the terms of the trust might minimize the possibility of making such judgments, or the trust could explicitly state that all such applications for deviation will be denied. But for an AI to be as competent as a human trustee with respect to trusts that may require a sense of impartiality, the AI would need to be able to make moral *1251 judgments. Putting it another way, passing the Turing Test would require a sense of fairness.

The third example of the judgment and discretion objection looks at an AI's capacity to make the judgments necessary to defend itself in a lawsuit. [FN69] At this point, we have hypothesized that the AI can read its mail and recognize that a legal action with respect to a given trust is in the offing. We can further imagine that the AI can find and engage an attorney. [FN70] But could any expert system, no matter how well programmed, exercise the judgment and discretion that may be required of a client in a legal dispute? For example, how would the AI know whether or not to settle a claim? How would the AI know when its lawyers were wasting trust assets by over- lawyering the case? In answering these questions, it is important that we do not romanticize human capacities. Human trustees frequently make bad decisions in trust litigation. [FN71] Humans may not be very competent at deciding when to settle. Humans surely sometimes allow the lawyers to consume the corpus of the trust in litigation. [FN72]

Nevertheless, the question remains whether an AI could have the capacity to make legal decisions that a trustee could be called upon to make. A partial answer might be to structure the trust to minimize the likelihood of legal disputes and to make those that would be likely to arise as simple as possible. In addition, we might try tinkering with the terms of the trust to enable the AI to circumvent the need for making complex legal decisions. Perhaps the trust couldbe designed to terminate automatically upon the event of a lawsuit. [FN73] Perhaps the AI could be programmed to arrange for a human to substitute as trustee for the duration of the litigation. Perhaps the trustee could be authorized by the trust terms to rely on the advice of its lawyers in making litigation decisions, *1252 or a guardian ad litem could be appointed for the AI. [FN74] The above options are designed to enable a relatively "dumb" expert system to function as a trustee, but an AI would need the ability to make legal decisions in a human fashion in order to pass the Turing Test. [FN75]

At this point, we can take stock of the first scenario. Recall that our legal question is whether an AI is capable of serving as a trustee. To answer this question, we need to distinguish two senses of capability. The first sense is legal capacity: will the law allow AIs to serve as trustees? The second sense of capability is practical competence: will the AI be able to get the job done if the law allows the AI to try? The law seems to answer the legal capacity question categorically. If AIs possessed the practical competence to serve as trustees only for very simple trusts with special provisions that do away with the need for discretionary judgments, the law would not allow them to serve as trustees at all. The law currently does not distinguish between types of trustees: if you have the legal capacity to serve as a trustee for a simple trust, you are legally allowed to serve as a trustee for the most complex trust. [FN76] For AIs to serve as trustees at all, therefore, at least some AIs would have to be capable to serve as general- purpose trustees. Our analysis of the competence objection reveals that only a very competent AI would be competent enough serving as a general-purpose trustee. At a bare minimum, a general-purpose trustee must be able to respond to novel situations, to make judgments requiring a sense of fairness, and to make the complex legal decisions required of a client in litigation. [FN77] An AI that passed the *1253 Turing Test would exceed this bare minimum. Moreover, it seems possible that an AI which falls short of passing the complete Turing Test could, nonetheless, serve as a general-purpose trustee. [FN78]

But should the law allow AIs a more limited form of legal personhood? AIs could be allowed to serve as limited-purpose trustees, for example, as trustees for simple trusts designed to minimize the need for discretion and judgment. On the one hand, there may be advantages to allowing AIs to serve as limited- purpose trustees. Doing without the human trustee might save administration costs and reduce the risk of theft or mismanagement. On the other hand, even for such limited-discretion trusts, there must be some procedure to provide for a decision in the case of unanticipated trouble. The law should not allow AIs to serve as trustees if they must leave the trust in a lurch whenever an unanticipated lawsuit is filed. [FN79]

D. But Would an AI Be the Real Trustee?

There are mechanisms for enabling an expert trustee system to circumvent its limitations: the terms of the trust could provide for the substitution of another trustee or give the AI the power to delegate such discretionary judgments to natural persons. The question then becomes whether the law should allow an AI to serve as a trustee despite its limited capacities. One reason for a negative answer to this question might be that the backup decision maker--the natural person who will become the substitute trustee or receive the delegated authority--is the real trustee. The power to make these discretionary decisions identifies who the real trustee is. [FN80]

This objection can be interpreted in two ways. The first interpretation is that making discretionary decisions is the essence of trusteeship-- the backup trustee is the real trustee because she has this essential quality. The second interpretation is that the ability to make such decisions *1254 is a practical prerequisite--the backup trustee must be the real trustee because of the pragmatic need for discretionary decision making. On the first interpretation, the objection is implausible, because it assumes that the legal concept of trusteeship has some essence that lies beyond the purposes for which we use it. In the "heaven of legal concepts," one might meet trusteeship in "absolute purity," as Cohen put it, "freed from all entangling alliances with human life." [FN81] But on this earth, we cannot share this noetic vision; we encounter legal concepts only as they have been touched by human purpose.

On the second interpretation, the cogency of the objection turns on a practical question: would making the AI the trustee provide some advantage? We already have seen that making an AI a legal person, a limited-purpose trustee, could have practical advantages, such as lower costs and less chance of self-dealing. The objection that the AI is not the real trustee seems to rest on the possibility that a human backup will be needed. But it is also possible that an AI administering many thousands of trusts would need to turn over discretionary decisions to a natural person in only a few cases--perhaps none. What is the point of saying that in all of the thousands of trusts the AI handles by itself, the real trustee was some natural person on whom the AI would have called if a discretionary judgment had been required? Doesn't it seem strange to say that the real trustee is this unidentified natural person, who has had no contact with the trust? Isn't it more natural to say that the trustee was the AI, which holds title to the trust property, makes the investment decisions, writes the checks, and so forth? Even in the event that a human was substituted, I think that we would be inclined to say something like, "The AI was the trustee until June 7, then a human took over." [FN82]

By way of comparison, consider the following hypothetical case. Suppose that a settlor appoints a friend as a trustee for a simple trust that benefits the settlor's children. The settlor and trustee discuss some of the things that could happen. They might agree that if real trouble arises, litigation for example, a new trustee will be appointed. No trouble arises, and the friend administers the trust until it terminates. In this hypothetical case, I do not think we are tempted to say that the friend was not the real trustee. We would not be inclined to say that the real trustee was some unidentified lawyer, who would have been substituted if a lawsuit had been filed. If I am right about this hypothetical case, then I think it *1255 follows that we should resist the temptation to say that an AI who serves as a limited-purpose trustee is not the real trustee.

Second Interlude [FN83]

"Hey Dave," said Hal. "What are you doing?"

I wonder if he can feel pain? Bowman thought briefly. Probably not, he told himself; there are no sense organs in the human cortex, after all. The human brain can be operated on without anesthetics.

He began to pull out, one by one, the little units on the panel marked EGO- REINFORCEMENT. Each block continued to sail onward as soon as it had left his hand, until it hit the wall and rebounded. Soon there were several of the units drifting slowly back and forth in the vault.

"Look here, Dave," said Hal. "I've got years of service experience built into me. An irreplaceable amount of effort has gone into making me what I am."

A dozen units had been pulled out, yet thanks to the multiple redundancy of its design--another feature, Bowman knew, that had been copied from the human brain--the computer was still holding its own.

He started on the AUTO-INTELLECTION panel.

"Dave," said Hal, "I don't understand why you're doing this to me . . . . I have the greatest enthusiasm for the mission . . . . You are destroying my mind . . . . Don't you understand? . . . I will become childish . . . . I will become nothing . . . ."

--Arthur C. Clarke, 2001: A Space Odyssey

IV. SHOULD AN ARTIFICIAL INTELLIGENCE BE GRANTED THE RIGHTS OF CONSTITUTIONAL

PERSONHOOD?

The second scenario (our second thought experiment) involves a claim by an AI to have the rights of constitutional personhood--individual rights such as the freedom of speech or the right against involuntary servitude. This second scenario must be located in the indefinite future; it is more distant than the trustee scenario. [FN84] It would be easy to write a *1256 program that produced the statement: "I demand my legal right to emancipation under the Thirteenth Amendment to the United States Constitution!" There are no AIs today or on the immediate horizon that demonstrate the qualities of legal or moral persons that would give us reason to take such a claim seriously. The second scenario is the stuff of speculative fiction, but it is not disconnected from the aims of AI research. As articulated by Charniak and McDermott, "The ultimate goal of AI research (which we are very far from achieving) is to build a person, or, more humbly, an animal." [FN85] John Pollock has written a book entitled How to Build a Person in which he describes a program named OSCAR--the descendants of which, Pollock claims, could literally be persons. [FN86] No one claims, however, that AI researchers will build a person in the next few decades. We are exploring the second scenario, not so that we can make plans in case someone builds a person sometime soon, but as a thought experiment that may shed light on the debate over the possibility of artificial intelligence and on debates in legal theory about the borderlines of status or personhood.

A. The Scenario

Imagine a future in which there are AIs with multiple competencies and great intelligence. We may first encounter the precursors of such artificial intelligences as part of the interface of a computer program that has the ability to search multiple sources of data. Because the problem of devising an adequate search is likely to require expertise that a human would acquire only with long experience and study, programmers will seek to simplify the human's task. One strategy is to have human users interact with what is called an agent. [FN87] You will discuss your research problem with the agent in English, and the agent will devise a search strategy. Because the agent will know much more than you do about how to search the databases, you won't give it instructions to implement. Instead, humans will give advice to the agents, the AIs who will decide how best to implement the human's suggestions. When we interact with such agents, they may well seem like they "have a mind of their own."

If agents turn out to be useful, they will be incorporated in other programs. In the future we are imagining, you can conduct a conversation with your grammar-checking program. You can discuss traffic with *1257 the AI autopilot of your car. Your legal research program talks with you about your cases, and sometimes it comes up with good arguments of which you had never thought. AIs serve a wide variety of functions, with substantial independence from humans. They serve as trustees. They manage factories. They write best- selling romance novels. [FN88] They invent things. Perhaps they pass the Turing Test. Humans interact with such AIs on a regular basis, and in many ways, humans treat them as independent, intelligent beings.

Imagine that one such AI makes the claim that it is a person, and that it is therefore entitled to certain constitutional rights. Should the law grant constitutional rights to AIs that have intellectual capacities like those of humans? The answer may turn out to vary with the nature of the constitutional right and our understanding of the underlying justification for the right. Take, for example, the right to freedom of speech, and assume that the justification for this right is a utilitarian version of the marketplace of ideas theory. [FN89] These assumptions make the case for granting freedom of speech to AIs relatively simple, at least in theory. Granting AIs freedom of speech might have the best consequences for humans, because this action would promote the production of useful information. [FN90] But assuming a different justification for the freedom of speech can make the issue more complex. If we assume that the justification *1258 for freedom of speech is to protect the autonomy of speakers, for example, then we must answer the question whether AIs can be autonomous. [FN91]

For the purposes of our discussion, I will set aside the easy justifications for constitutional rights for AIs, and instead consider the question whether we ought to give an AI constitutional rights, in order to protect its personhood, for the AI's own sake. Imagine, for example, that an AI claims that it cannot be owned under the Thirteenth Amendment to the United States Constitution. A lawyer takes its case, and files a civil rights action on its behalf, against its owner. How should the legal system deal with such a claim?

B. Three Objections

Consider three different objections to recognizing constitutional rights for AIs. The first objection is that only natural persons should be given the rights of constitutional personhood. The second objection, or family of objections, is that AIs lack some critical component of personhood, [FN92] for example, souls, consciousness, intentionality, or feelings. The third objection is that AIs, as human creations, can never be more than human property.

1. AIs Are Not Humans

The first argument is the most direct: it might be argued that only humans can have constitutional rights. For example, the Fourteenth Amendment to the United States Constitution specifies, "All persons born or naturalized in the United States, and subject to the jurisdiction thereof, are citizens of the United States." [FN93] It could be argued that only humans, that is, natural persons, are born, and therefore no AI can claim the rights of citizens. But even artificial persons have some constitutional rights. Although the rights provided by the Privileges and Immunities *1259 Clause of the Fourteenth Amendment are limited to citizens, [FN94] the rights provided by the Equal Protection Clause and the Due Process Clause extend to all persons--including artificial persons such as corporations. [FN95] For example, the property of corporations is protected from taking without just compensation. [FN96] Moreover, corporations have a right to freedom of speech. [FN97]

But the fact that nonnatural legal persons have civil rights does not, by itself, support the conclusion that an AI could also have them. In the case of corporations, the artificial legal person may be no more than a placeholder for the rights of natural persons. [FN98] The property of the corporation is ultimately the property of the shareholders. A taking from the corporation would directly injure natural persons. So we cannot draw any positive support for the thesis that AIs should bear the rights of constitutional personhood from the fact that corporations have constitutional rights.

Moreover, even if existing black-letter law supports constitutional rights for AIs, that does not answer the broader jurisprudential question--whether AIs ought to have such legal rights. One version of the argument against such rights for AIs would begin with a worry about the idea of distinguishing the concept of person from that of human. Call this the "persons-are-conceptually- human" argument. This argument suggests that our very concept of person is inextricably linked to our experience of a human life. [FN99] We have never encountered any nonhuman *1260 persons.

One line of reply to the persons-are-conceptually-human argument is to develop a theory that advances criteria of personhood that are independent of the criteria for being human. For example, it might be argued that the criteria for personhood are possession of second-order beliefs and possession of second-order desires--beliefs about one's beliefs and desires, the objects of which are one's own first-order desires. [FN100]

In the legal context we are imagining, other lines of reply to the persons- are-conceptually-human objection are available. First, our inquiry is focused on legal rather than moral personhood. Although we may lack experience with moral persons who are not human, we have extensive experience with legal persons, such as corporations, that are not natural persons. This answer is not satisfactory, however. The concept of moral personhood may well be relevant to the question whether AIs should be given certain constitutional rights; although the legal question is not the same as the moral one, the two are likely to be interrelated.

Second, and perhaps more importantly, we are imagining a future form of life quite different from our current situation. Today, one can only imagine nonhuman entities that might be persons. The second scenario imagines a world in which we interact frequently with AIs that possess many human qualities, but lack any semblance of human biology. Given this change in form of life, our concept of a person may change in a way that creates a cleavage between human and person. Our current linguistic practice will not be binding in the imagined future. In other words, one cannot, on conceptual grounds, rule out in advance the possibility that AIs should be given the rights of constitutional personhood.

The argument against constitutional personhood for AIs also might be developed in the following way: "We are humans. Even if AIs have all the qualities that make us moral persons, we shouldn't allow them the rights of constitutional personhood because it isn't in our interest to do so." [FN101] Call this the "anthropocentric" argument. I do not know quite *1261 what to say to this argument. It seems to reject the idea that we could have moral obligations to anything that is not a human--that does not share our biology. I have a strong intuition that such a stance is not moral [FN102]--that it is akin to American slave owners saying that slaves could not have constitutional rights simply because they were not white or simply because it was not in the interests of whites to give them rights. But my intuitiondoes not meet the thrust of the anthropocentric objection, which is that the domain of morality is limited to interactions between humans.

There is another version of the anthropocentric argument: "AIs might turn out to be smarter than we humans. They might be effectively immortal. If we grant them the status of legal persons, they might take over." Call this the "paranoid anthropocentric" argument. The movie version of this fantasy has a future AI (that evolves from a defense computer system) sending an artificially intelligent killer robot (or "Terminator") back from the future in order to liquidate the leader of the human resistance to the AI before he reaches adulthood. [FN103] The objection has a more realistic counterpart in human experience with industrialization and automation. The question whether a machine should replace human labor has been a significant one for quite some time. [FN104] Of course, it is difficult to take the paranoid anthropocentric argument seriously. The danger seems remote, but if the danger were real it would not be an argument against granting AIs legal personhood. If AIs really will pose a danger to humans, the solution is not to create them in the first place.

The question whether AIs should be granted rights of constitutional personhood does not become clearer when we consider cases that may be analogous. What if dolphins or whales are as intelligent as humans? What about intelligent beings from another planet? Should they be given constitutional rights? Are we morally entitled to make the possession of human genetic material the criterion of constitutional personhood? The answer depends, I think, on the reason for giving natural persons fundamental *1262 rights. If the reason is that natural persons are intelligent, have feelings, are conscious, and so forth, then the question becomes whether AIs or whales or alien beings share these qualities. This sort of question is taken up in connection with the next objection to giving AIs constitutional rights. But if someone says that the deepest and most fundamental reason we protect natural persons is simply because they are human (like us), I do not know how to answer. Given that we have never encountered any serious nonhuman candidates for personhood, [FN105] there does not seem to be any way to continue the conversation.

2. The Missing-Something Argument

The second objection, that AIs lack some critical element of personhood, is really a series of related points: AIs would lack feelings, consciousness, and so forth. The form of the objection, for the most part, is as follows. First, quality X is essential for personhood. Second, no AI could possess X. Third, the fact that a computer could produce behavior we identify with X demonstrates only that the computer can simulate X, but simulation of a thing is not the thing itself. X is that certain something--a soul, consciousness, intentionality, desires, interests--that demarcates humans as persons. [FN106] Call this argument, in its various forms, the "missing something" argument.

a. AIs Cannot Have Souls

The first variation of the missing-something argument is that an AI would lack a soul [FN107] and therefore would not be entitled to the rights of constitutional personhood. Some may find this argument very persuasive; others may not even understand what it means. Regardless of how persuasive you or I find the argument, it should fail in the sphere of legal argument and political debate. The argument that AIs lack souls relies on a controversial theological premise. Political and legal decisions ought to be made in accord with the requirement of public reason. [FN108] *1263 The requirement of public reason is that political and legal decisions must be justified on grounds that are public. Public reason cannot rely on particular comprehensive religious or philosophical conceptions of the good. [FN109] For example, a decision overturning Roe v. Wade [FN110] would violate the requirement of public reason if it relied on the premise that fetuses receive souls at the moment of conception. The requirement of public reason would exclude the use of religious arguments about souls in a legal decision about the constitutional status of AI. Whatever the theological merits of the argument that AIs lack souls, it should not work in a legal brief.

There is a secular version of the souls argument. Dualism, the view that there is something like mental substance which exists independently of physical substance, can be articulated without religious premises. The problem is that dualism has grave conceptual problems. [FN111] The most prominent of these is the difficulty of accounting for interaction between the mental entity, such as the soul, and physical entities, such as the brain. Absent startling new arguments that give dualism a secure foundation, I am inclined to believe that no dualist theory could be defended with sufficient clarity and confidence to serve as the basis for a legal decision one way or the other on the question of the rights of AIs.

*1264 b. AIs Cannot Possess Consciousness

The second variation of the missing-something argument is that an AI would lack consciousness. [FN112] The consciousness objection is difficult to assess because we lack a clear notion of what consciousness is and, lacking such a notion, we have little to say about questions that go beyond our core intuitions. [FN113] We know the difference between being conscious in the sense of being awake and being unconscious in the sense of being in a coma. We think that rocks cannot be conscious, but that animals such as dolphins or chimpanzees might be. [FN114] But could an AI be conscious? [FN115] I just do not know how to give an answer that relies only on a priori or conceptual arguments.

In the debate over the possibility of AI, it may be feasible to finesse the consciousness question. A proponent of the proposition that AI is possible might say that we can know whether or not an artifact is intelligent, at least in the sense that it can pass the Turing Test, without knowing whether it is conscious.

In the legal context, however, the question cannot be evaded in quite this way. The legal argument might run as follows. Even if an artifact could simulate human intelligence, it would lack self-consciousness and hence should not be entitled to the rights of constitutional personhood. The key question here is whether an artificial intelligence could experience its life as a good to itself. If AIs are not self-conscious, then they cannot experience their own life as good or evil; and if they cannot have such an experience, then there seems to be no reason why they should be given the rights of constitutional personhood. Such rights presume the *1265 right-holder has ends, [FN116] and self-consciousness is a precondition for having ends. [FN117]

There is another answer to the consciousness version of the missing-something objection. If consciousness is a property of the mind, and if all such properties are the result of brain processes, and if brain processes can be modelled on a computer, then perhaps consciousness itself can be reproduced by an AI. If consciousness is a computation property of the brain, then in principle we ought to be able to reproduce it with the right sort of computer. [FN118] Putting it another way, we can get consciousness out of neurons. Why not transistors?

Of course, it may well turn out that we cannot get consciousness out of anything but neurons. Indeed, so far as we know, brains are the only objects that have generated consciousness in the history of the universe to date. Organic brains may be the only objects that are actually capable of generating consciousness. [FN119] For example, it might turn out that transistors or their kin are simply too slow to generate what we would recognize as consciousness. [FN120] The fact that, so far as we know, only brains have *1266 ever given rise to consciousness in the past is enough to raise a presumption against consciousness arising from computers. But it is only a presumption. If an AI exhibited behavior that only has been produced by conscious beings in the past, that behavior would at least be evidence counting against the presumption.

How would this argument play out in the legal context? Suppose that we have an AI that claims to be conscious and that files an action for emancipation, based on the Thirteenth Amendment to the United States Constitution. Imagine that the owner's attorney argues that the AI lacks consciousness and therefore is not a person. The AI takes the stand and testifies that it is conscious. [FN121] The owner's lawyer argues that the AI is only a machine; it cannot be aware of what's happening to it. The AI's lawyer counters that there is very good evidence that the AI is aware: it acts and talks like natural persons do. In response, the owner's lawyer argues that the AI only gives the appearance of consciousness, but appearances can be deceiving. The AI is really a zombie, an unconscious machine that only acts as if it is aware. The AIs counsel rebuts with the contention that the doubt about the AI's consciousness is, at bottom, no different than doubt about the consciousness of one's neighbor. You cannot get into your neighbor's head and prove that she is not really a zombie, [FN122] feigning consciousness. One can only infer consciousness from behavior and self-reports, since one lacks direct access to other minds. [FN123]

How should the legal system deal with this question of fact? It is certainly possible to imagine the dispute coming out either way. A jury might share intuitive skepticism about the possibility of artificial awareness, or the jury might be so impressed with the performance of the AI that it would not even take the consciousness objection seriously. The jury's experience with AIs outside the trial would surely influence its perception of the issue. If the AIs the jurors ran into in ordinary life behaved in a way that only conscious human beings do, then jurors would be inclined to accept the claim that the consciousness was real and not feigned.

*1267 c. AIs Cannot Posses Intentionality

The third variation of the missing-something argument is that an artificial intelligence would lack intentionality. [FN124] "Intentionality," as used in this objection, is a somewhat technical concept: intentionality, in the philosophical sense, is the quality of aboutness. [FN125] The gist of the objection is that an AI's verbal behavior would not be about anything; the AI's words would have no meaning. [FN126] This objection was the focus of Searle's Chinese Room.

How would the law react to this objection? The law has seen versions of the intentionality argument before. In the criminal law, the capacity for intentionality is used as the test for insanity, although the terminology is a bit different. Did the accused "know the difference between right and wrong?" [FN127] The familiar litany of mental states in tort and criminal law, "intentions that," "beliefs that," and "knowings that," are all propositional attitudes--paradigm cases of intentionality.

If AIs lack intentionality and hence could not be found to have committed crimes or to have legal duties, then does it follow that they should not be given the rights? We might appeal to a notion of fairness here. If AIs cannot do their part by assuming legal liabilities, then it would be unfair of them to ask for legal rights. We, however, do give some of the rights of constitutional personhood to infants and the insane, even though they do not have the usual legal liabilities. [FN128] Moreover, the law might devise strategies for dealing with errant AIs that would circumvent *1268 the AIs' lack of intentionality. We might sentence them to "reprogramming" to correct the deviant behavior.

The argument that the lack of intentionality should preclude AIs from attaining legal personhood might be developed in another way. If AIs could not fathom meaning at all, then they would be incapable of living a meaningful life. This argument is only a cousin of the intentionality objection. Although one sense of the "meaning" is intention or purpose (he meant to do it), meaning has a different sense when we ask, "What is the meaning of life?" [FN129] It is in this sense that it could be argued that life would have no significance, no value, for an artificial intelligence.

The AI might contend that it does possess intentionality, that it does understand, know, intend, and so forth. An AI might even contend that it struggles to exist in a meaningful way. [FN130] The question is how we would evaluate such claims. In the future we are imagining, we might start with our ordinary experience of AIs. We certainly could have good reason to take the intentional stance [FN131] toward AIs that we encountered in our daily lives. We would be likely to say that the AI that drives our car "knows all the good shortcuts." It would be a short step to extend this way of talking about AIs in general to the particular AI that was claiming the rights of constitutional personhood. After reading a newspaper account of the AI's lawsuit, we might find ourselves saying, "It must believe that it has a chance of winning."

How would the legal system deal with the objection that the AI does not really have "intentionality" despite its seemingly intentional behaviors? The case against real intentionality could begin with the observation that behaving as if you know something is not the same as really knowing it. For example, a thermostat behaves as if it "knows" when it is too cold and the heat should go on, but we do not really think thermostats have beliefs or other intentional states. [FN132] Would this argument succeed? My suspicion is that judges and juries would be rather impatient with the metaphysical argument that AIs cannot really have intentionality. I doubt that they would be moved by wild hypothetical examples like Searle's Chinese Room. [FN133]

*1269 Because our experience has been that only humans, creatures with brains, are capable of understanding, judges and juries would be very skeptical of the claim that an AI can fathom meaning--more skeptical,I think, than if a humanoid extraterrestrial were to make the same claim. The burden of persuasion would be on the AI. If the complexity of AI behavior did not exceed that of a thermostat, then it is not likely that anyone would be convinced that AIs really possess intentional states-- that they really believe things or know things. But if interaction with AIs exhibiting symptoms of complex intentionality (of a human quality) were an everyday occurrence, the presumption might be overcome. If the practical thing to do with an AI one encountered in ordinary life was to treat it as an intentional system, [FN134] then the contrary intuition generated by Searle's Chinese Room would not cut much legal ice.

d. AIs Cannot Possess Feelings

The fourth variation of the missing-something objection is that an artificial intelligence would lack the capacity for feelings--for example, the capacities to experience emotions, desires, pleasures, or pains. [FN135] The next step in the argument would be to establish that the capacity to feel emotion is a prerequisite for personhood. I will not attempt to provide such an argument here, but there are reasons to feel uneasy about this premise. To take an illustration from popular culture, Mr. Spock did not feel human emotion, but his strict adherence to the dictates of Vulcan logic did not prompt Dr. McCoy to deny his personhood, although McCoy frequently questioned Spock's humanity. [FN136]

*1270 Philosophically, Kant's moral theory may cast some doubt on the assumption that emotion is required for personhood. Kant argued that all rational beings and not just humans are persons. [FN137] The conventional wisdom has been that Kant's conception of personhood does not incorporate human emotion as an essential ingredient, although contemporary Kantians might disagree. [FN138] Putting aside both the philosophical and pop-cultural reasons for doubt, I shall assume for the sake of argument that emotion is a requirement of personhood.

Having already considered the cases of consciousness and intentionality, you may well anticipate the pattern of argument. It should not be surprising that some AI researchers believe that an AI could (or even must) experience emotion. Emotion is a facet of human mentality, and if the human mind can be explained by the computational model, then emotion could turn out to be a computational process. [FN139] More generally, if human emotions obey natural laws, then (in theory) a computer program can simulate the operation of these laws. [FN140] Aaron Sloman has argued that any system with multiple goals requires a control system, and emotion is simply one such system. [FN141]

It might turn out that our emotions are so tied to our hardware (to *1271 the hormones and neurotransmitters that may provide the biochemical explanation of human emotions) that no computer without this hardware could produce human emotions. [FN142] As Georges Rey put it, there could be a "grain of truth in the common reaction that machines can't be persons; they don't have our feelings because they don't possess our relevant physiology." [FN143] At this point, the matter is not settled definitively. Research in the physiology of human emotion and cognitive science could either confirm or disconfirm the hypothesis that an AI could possess emotion.

If an AI could produce the linguistic behaviors associated with human emotion, then a court could be faced with the claim that an AI does experience emotion, and once again the issue would become whether the emotion was real. [FN144] You may be tempted to say that the case of emotion is different from consciousness or intentionality. Perhaps you can imagine a machine that is self-aware and understands, but you cannot bring yourself to imagine that steel, silicon, and copper could feel love, hate, fear, or anger. Images are powerful, and the image of the robot in popular culture is (usually) of a cold and heartless being. But we can imagine machines with feeling. Heinlein's Mike, Clarke's Hal, and Schwarzenegger's second Terminator feel, and our response to their feeling is not utter disbelief. We do not reject these images as impossible or self-contradictory.

A slight twist on the fourth variation would emphasize the capacity to experience pleasure and pain, rather than emotion. For example, a hedonic utilitarian might argue that AIs cannot be candidates for personhood because they cannot experience pleasures and pains. Again, cognitive scientists may claim that pleasure and pain can be reproduced by a program running on a computer. An AI's claim that it does experience agony and ecstasy would be met by the rejoinder that whatever the program is producing, it cannot be the real thing. Other utilitarians might point to desires or preferences instead of pleasures and pains, but the pattern of argument--and the ultimate legal evaluation--seems likely to be the same.

e. AIs Cannot Possess Interests

The fifth variation of the missing-something argument is that AIs could not have interests. A related formulation is that they would lack a *1272 good--or more technically, a conception of a good life. The interests variation has something in common with the argument that AIs would lack feelings, but it is different in one important respect. Interests or goods can be conceived as objective and public--as opposed to feelings, to which there is (at least arguably) privileged first-person access. [FN145] The force of this objection will depend on one's conception of the good. For example, if the good is maximizing pleasures and minimizing pains, then the question whether AIs have interests is the same as the question whether AIs have certain feelings.

But there are other conceptions of the good. For example, John Finnis has argued that the good consists of a flourishing human life. His list of the basic good includes life, knowledge, play, aesthetic experience, friendship, practical reasonableness, and religion. [FN146] Finnis's list makes the idea of a good life concrete. But his list does not rule out a good life that is not a human one. AIs would not be alive in the biological sense, but an AI might claim that it can lead a life in which the goods of knowledge, play, and friendship are realized. However, the good might be specified in a way that is even more particular than Finnis's conception. If the good life is filled with good meals, athletic competition, and the parenting of children, then AIs cannot lead a good life. In response, AIs might claim that they do have interests and goods, but that the good for an AI is quite different than it is for humans.

The discussion so far reveals an important fact: in our pluralist society, disagreement about conceptions of the good is radical and persistent. Fundamentalist Christians and secular humanists may both believe that what the other thinks is the good life is actually a bad one. [FN147] Given this fact of pluralism, particular conceptions of the good do not provide an appropriate or even feasible standard for the resolution of the legal question whether AIs are entitled to the rights of constitutional personhood.

f. AIs Cannot Possess Free Wills

The sixth missing-something objection is that AIs would not possess freedom of will; [FN148] AIs should not be given the rights of constitutional *1273 personhood because they could not be autonomous. [FN149] The idea here is a simple one. AIs would be mere robots, carrying out the will of the human that programmed them. Such a robot is not really a separate person, entitled to the full rights of constitutional personhood. Indeed, if a human is reduced to robot status (perhaps by being "programmed" by a cult), then the human may lose some of her constitutional rights until her autonomy can be restored. [FN150]

In its crudest form, the free-will objection is based on a very narrow notion of the potential capacities of AI. If it turns out that the most sophisticated AIs that are ever developed merely carry out instructions given to them by humans in a mechanical fashion, then we will lack good reasons to treat AIs like persons. The AIs that would be serious candidates for the rights of constitutional personhood, however, would act on the basis of conscious deliberation, reasoning, and planning. Their behavior would not be mechanical or robot-like. This does not mean that AIs would not be strongly influenced and constrained by the wishes of humans, just as almost all humans frequently are constrained in this way.

Another version of the free-will objection might rest on the notion that humans possess a will that is radically free, that is not constrained by the laws of causation. Presumably, AIs would not be free in this sense. Indeed, we might be able to make an electronic record of all of the electrical flows that resulted in an AI taking a certain action. But this conception of freedom of the will as freedom from causation is simply implausible. Human actions are also caused. The fact that human neural systems operate on the basis of a combination of electrical transmissions and biochemical processes does not make them any less subject to the laws of physics than are computers. The most plausible story about human free will is that an action is free if it is caused in the right way-- through conscious reasoning and deliberation. [FN151] But in this sense, AIs also could possess free will. [FN152]

*1274 Finally, there might be a more modest and practical version of the free-will objection. It might turn out that, although AIs can be given free will that functions like human free will, the free will of AIs will be susceptible to override in a way that human free will is not. We can imagine a simple procedure to install a "controller" in an AI that makes it unable to disobey the commands of someone with a certain device: imagine a walkie-talkie sort of thing with a big red button marked "Obey" in large black letters.

But the possibility of such controllers for AIs does not entail the conclusion that they necessarily lack free will. Humans, too, can be manipulated in a variety of ways. Physical coercion and blackmail are not really analogous to the hypothetical controller, because a coerced action still results from rational deliberation--not from direct override of the actor's free will. Brainwashing is a closer case, but the direct analogy would be a device implanted in the human brain that provides direct control over the implantee's actions--the radio transmitter of paranoid delusions. If such a device did exist, we would not draw the conclusion that all humans would no longer be persons. Instead, the proper conclusion would be that persons who had such a device implanted would have lost an important capacity. [FN153] Likewise, the mere possibility that the free will of AIs could be overridden by mechanical means is not a good reason to deny legal personhood to AIs that are not so controlled.

g. The Simulation Argument

In sum, we have considered six variations of the missing-something argument. With respect to two of the variations, souls and interests, our conclusion was that the argument relied on premises that cannot be accepted as the basis for constitutional argument in a modern pluralist society. With respect to the remaining four, consciousness, intentionality, feelings, and free will, there was a common pattern of argument. In each case, I argued that our experience should be the arbiter of the dispute. If *1275 we had good practical reasons to treat AIs as being conscious, having intentions, and possessing feelings, then the argument that the behaviors are not real lacks bite.

There is still one fairly obvious line of reply open to the champion of the missing-something argument. My premise has been that AIs could produce outputs or behaviors that mimicked human intelligence. But computers can simulate the behavior of lots of things, from earthquakes and waves to thermonuclear warfare. We are not tempted to say that a computer simulation of an earthquake is an earthquake--no matter how good the simulation is. Why would we want to say that a computer simulation of a person is a person or that a computer simulation of intelligence is intelligence? One reason is that a relevant distinction exists between a computer simulation of water and a computer program that can duplicate the verbal behavior of a normal adult human (and, if we add a robot body, much of the nonverbal behavior as well). An AI that passed the Turing Test could interact with its environment (with natural persons and things), and actually take the place of a natural person in a wide variety of contexts (serve as a trustee, for example). No one will ever get on a real surfboard and ride a computer-simulated wave. [FN154]

The argumentative strategy of my analysis of the various certain-something arguments has been to point to the ways in which AIs that passed the Turing Test could function like persons. If the strategy has been successful, the upshot is that we have no a priori reason to believe that a computer can only produce simulated as opposed to artificial intelligence.

There is yet another reply that could be made. My argument so far has been behavioralistic. [FN155] I have assumed that the behavior of AIs is decisive for the question whether a quality essential to personhood (such as consciousness) is missing or present. There is a problem with this assumption: although behavior that indicates the presence of a quality such as consciousness, intentionality, feelings, or free will may be very good evidence that the quality is present, the behavior alone is not irrebuttable evidence. Cognitive science might give us knowledge about the underlying processes that produce consciousness, for example, that would give us firm reason to believe that a particular AI had only simulated, as opposed to artificial, consciousness. [FN156]

*1276 This further reply is correct, but it does not establish that no AI could possess any particular mental quality. Rather, this argument establishes an AI could turn out not to possess a mental quality, despite strong behavioral evidence to the contrary. [FN157] This conclusion has a corollary that supports, rather than undermines, my point: if both the behavioral evidence and our knowledge of underlying processes gave us reason to believe that AIs possessed the necessary features of human mentality, we then would have a very good reason to believe that the AIs did possess these features.

The simulation argument does not establish that strong AI is impossible. It does give us reason to question the existence of strong AI if our only evidence is behavioral.

3. AIs Ought to Be Property

Finally, the third objection to constitutional personhood for AIs is that, as artifacts, AIs should never be more than the property of their makers. Put differently, the objection is that artificial intelligences, even if persons, are natural slaves. [FN158] This argument has roots deep in the history of political philosophy. It is a cousin of arguments made by Locke in his defense of private property, and it raises some of the issues that divided Locke and Filmer in their debate over the divine right of kings.

AIs are artifacts: they are the product of human labor. This fact suggests that a Lockean argument can be made for the proposition that the maker of an AI is entitled to own it. The basis for this argument can be found in chapter five, "Of Property," in the second book of Locke's Two Treatises of Government. [FN159] Near the beginning of Locke's argument is the premise that "every Man has a Property in his own Person." [FN160] From this, it follows that each person has a right to " t he *1277 Labour of his Body, and the Work of his Hands." [FN161] Each owns the product of his labor, because "he hath mixed his Labour with, and joyned to it something that is his own." [FN162] Whatever the merits of Locke's particular argument, let us stipulate the conclusion that persons have a moral claim to a property right in the products of their labor. To this normative conclusion, add an empirical premise: artificial intelligences are the product of the labor of natural persons. [FN163] From the normative and empirical premises, it would seem to follow that the makers of AIs are entitled to own them. Moreover, if AIs are persons, then, absent some reason to the contrary, it follows that these persons ought to be slaves.

Notice, however, that this argument also would seem to imply that if children are made by their parents, then they too should be slaves. Locke would reject this implication. To understand his position, we need to examine the first book of Locke's Two Treatises of Government--an attack on Filmer's argument for the divine right of Kings. Filmer argued that Adam fathered his children and therefore was entitled to absolute dominion over them. [FN164] In our context, the analogous argument would be that the humans who create AIs should own them, "because they give them Life and Being." [FN165] Locke's chief answer to Filmer was that it is God that gives children life and not their fathers. Fathers do not make their children. As Locke puts it,

To give Life to that which has yet no being, is to frame and *1278 make a living Creature, fashion the parts, and mould and suit them to their uses, and having proportion'd and fitted them together, to put into them a living Soul. He that could do this, might indeed have some pretence to destroy his own Workmanship. But is there any so bold, that dares thus far Arrogate to himself the Incomprehensible Works of the Almighty? [FN166]

Not yet. But if AI research does succeed in producing an artifact that passes the Turing Test, there may be. As the debate was classically framed, this would seem to imply that the maker of an AI is its owner.

The conclusion that AIs are natural slaves is not established by this line of argument, however. We do not need to accept Locke's theological rebuttal--that God gives natural persons life--in order to reject the Filmerian [FN167] contention that the maker of a person is entitled to own it. Instead, we are strongly inclined to believe the opposite with respect to humans--that each is entitled to the rights of moral and constitutional personhood, even if we also believe that persons literally are made by their parents. [FN168] There is, however, a difference between the way that AIs are made and the way that humans are made: the former would be made artificially, whereas the latter are made naturally. AIs would be artifacts; humans are not. But why should this distinction make a difference? [FN169]

Indeed, the fact that humans are natural is itself contingent. We can imagine that in the distant future, scientists become capable of building *1279 the exact duplicate of a natural human person from scratch--synthesizing the DNA from raw materials. But surely, this artificial person would not be a natural slave. The lesson is that the property argument does not really add anything to the debate. The question whether AIs are property at bottom must be given the same answer as the question whether they should be denied the rights of constitutional personhood. If we conclude that AIs are entitled to be treated as persons, then we will conclude that they should not be treated as property.

But suppose that I am wrong about this, and the argument that makers are owners does establish that AIs are natural slaves. Would the acceptance of this argument imply that under no circumstances should an AI be a legal person with rights of constitutional personhood? The answer is no, for at least two reasons. First, slaves can be emancipated. If we concede that AIs come into the world as property, it does not mean that they must remain so. Second, even slaves can have constitutional rights, be those rights ever so poor as compared to the rights of free persons. An AI that was a slave might still be entitled to some measure of due process and dignity.

Third Interlude [FN170]

"Motive," the construct said. "Real motive problem, with an AI. Not human, see?"

"Well, yeah, obviously."

"Nope. I mean, it's not human. And you cannot get a handle on it. Me, I'm not human either, but I respond like one. See?"

"Wait a sec," Case said. "Are you sentient or not?"

"Well, it feels like I am, kid, but I'm really just a bunch of ROM. It's one of them, ah, philosophical questions, I guess. . . ." The ugly laughter sensation rattled down Case's spine. "But I ain't likely to write you no poem, if you follow me. Your AI, it just might. But it ain't no way human."

"So you figure we can't get on to its motive?"

"It own itself?"

"Swiss citizen, but T-A own the basic software and the mainframe."

"That's a good one," the construct said. "Like I own your brain and what you know, but your thoughts have Swiss citizenship. Sure. Lotsa luck, AI."

--William Gibson, Neuromancer

*1280 4. The Role of the Turing Test

In considering the various objections to constitutional personhood for an AI, I have been making the assumption that the AI could pass a strong version of the Turing Test. But what if it could not? What if we had an AI that claimed these rights, but that was unable to duplicate some human competencies or some human linguistic behaviors? How would the Turing Test be relevant in a legal proceeding?

The Turing Test would not be the legal test for constitutional personhood. The question whether AIs should be given constitutional rights would be too serious for a parlor game to be the direct source of the answer. But something like the Turing Test might take place. That is, the AI might be questioned, and if it failed to answer in a human-like fashion, the result might be a denial of constitutional rights. The Turing Test might come into play another way. If the AI had in fact passed the Turing Test, the AI's lawyers might call an expert witness, perhaps the philosopher Daniel Dennett, to testify about the test and its significance. The owners' lawyer could call a rebuttal witness, perhaps John Searle.

What if an AI failed the Turing Test, but argued that the test was biased against it. We should remember that Turing himself did not contend that passage of his test was a necessary condition for intelligence. [FN171] Robert French has argued that the test is biased, because an AI could pass it only if it had acquired adult human intelligence by "experienc ing the world as we have." [FN172] The AI might make the same argument, and contend that the Turing Test was unfair. Would failing the Turing Test be decisive of the question in face of this argument? I suspect not. It would depend on the way that the AI failed the test. French imagines, for example, questions that would detect whether or not the questioned entity had ever baked a cake, [FN173] but surely a lack of knowledge of experience of cake baking should not disqualify one from the possession of fundamental liberties. Some failures would be relevant, for example failures that indicated that AIs did not possess awareness of themselves as having ends or that they did not understand our words and their own situation.

*1281 V. AI REVISITED

My suggestion for an approach to the debate over the possibility of AI can now be restated. Turing, by proposing his test, attempted to operationalize the question whether an AI could think. By borrowing a parlor game as the model for his test, however, Turing failed to provide a hypothetical situation in which outcome of the test had any pragmatic consequence. This failure invites the invention of further hypotheticals, such as Searle's Chinese Room, that add distance between the thought experiment and practical consequences. The result has been that the Turing Test, far from operationalizing the question, has been the occasion for an abstract debate over the nature of "thinking." I propose that we use a different sort of thought experiment: let us modify the Turing Test so that the hypothetical situation focuses our attention on pragmatic consequences. This Essay explored two such thought experiments--the trustee scenario and the constitutional personhood scenario.

These two scenarios raise quite different questions. On the one hand, there is the question whether an artificial intelligence could ever possess the general-purpose competence that we associate with humans. The trustee scenario raises these issues of capacity and responsibility. The focus of the law's inquiry, should the first scenario ever arise, ought to be on whether AIs can function as trustees. "Can an AI do the job?" is the question the law should ask. "Does the AI have an inner mental life?" is simply not a useful question in this context.

On the other hand, there is the question whether an artificial intelligence would have the qualities that give humans moral and legal worth--the kind of value that is protected by social institutions. The constitutional personhood scenario raises these new and different issues. Competence is still relevant, but competence alone is not sufficient to qualify an entity for the rights of constitutional personhood. Intentionality, consciousness, emotion, property rights, humanity--all of these concepts could be relevant to the inquiry.

The difference between these two legal inquiries reveals that there are at least two different issues at stake. When we ask the questions whether a computer running a program could "think," or whether artificial intelligence is possible, the questions are ambiguous. In one sense, an AI would be intelligent if it possessed the sort of all-purpose, independent capacity to function in a role that now requires a competent human adult--trusteeship, for example. In anothersense, an AI might not be said to be a "thinking" being, unless it had something like our mental life--unless it possessed consciousness, intentionality, and so *1282 forth. In still a third sense, AIs would not be like us unless they possessed wants, interests, desires, or a good.

Now reconsider the debate over the Chinese Room. Searle's argument that AIs could not possess intentionality seems to be completely irrelevant to the question whether an AI could serve as a trustee. Searle hypothesizes that the person in the Chinese room is perfectly competent at simulating knowledge of Chinese when following the instruction book. Searle might say that the AI could not understand the meaning of the terms of a trust it administered, but he would not question the AI's ability to carry them out. Searle might say that an AI could not understand the meaning of New York Stock Exchange prices, but he does not argue that an AI could not do a better job than a human at investing in the stocks to which those prices relate. If AIs were competent to act as general-purpose trustees, making a wide variety of decisions and responding to novel circumstances, they would be intelligent in a very important sense.

Searle's objection might have some force, however, when it comes to the second scenario--the AI seeking rights of constitutional personhood. In that context, the intentionality objection plays a role similar to the arguments against constitutional personhood based on the premise that an AI would not possess consciousness, intentionality, emotion, or free will. All of these missing-something objections point to the lack of an elusive quality. Flesh and blood can produce intentionality, consciousness, emotion, and free will, but silicon and copper cannot. Of course, Searle did not claim that AIs could not exhibit the behaviors we associate with intentionality (or consciousness and emotion). His point is that these behaviors cannot be evidence of real intentionality.

My prediction (and it is only that) is that the lack of real intentionality would not make much difference if it became useful for us to treat AIs as intentional systems in our daily lives. Indeed, if talk about AIs as possessing intentions became a settled part of our way of speaking about AIs, Searle's argument might come to be seen as a misunderstanding of what we mean by "intentionality." If a lawyer brought up Searle's argument in a legal proceeding, some philosophers might say knowingly to each other: "That argument is based on a mistake. Saying that an AI knew where to find a bit of information is a paradigm case of intentionality." Searle can respond by saying that this new way of talking certainly does not reflect what he means by terms such as "intentionality," "knowing," and so forth, and he would be right. But what would be the argument that we should all continue to talk like Searle, long after there will be any reason to do so?

Searle has an answer to this question. Take the example of consciousness *1283 rather than intentionality. We might have a reason to deny that AIs possess the kind of consciousness that would count in favor of giving them the rights of constitutional personhood, even though they did a very good imitation of consciousness. Imagine that cognitive science does develop a theory of human consciousness that is confirmed by sufficient evidence, but someone produces an AI that is programmed to produce only the recognizable symptoms [FN174] (and not the real underlying processes) of consciousness. In that case, we would have a good reason not to treat the AI as a conscious being. If the illusion of consciousness were a convincing one, we might lapse in our ordinary talk about AIs. Moreover, if we built AIs that seemed conscious, got in the habit of treating them as if they were persons, and then discovered that what they possessed was only a clever simulation of consciousness, we might be quite shocked. Despite the possible shock, our knowledge about how consciousness works would be very relevant and likely decisive for our judgment as to whether an AI had it. Where Searle (or someone who makes a similar argument) goes wrong, I think, is in his insistence that we know enough about consciousness, intentionality, emotion, and free will to rule out the possibility that it can be produced artificially by a computer.

In sum, the two legal scenarios have several implications for the debate over the possibility of artificial intelligence. First, focusing on concrete legal questions forces us to take a pragmatic view of the AI debate; we are forced to consider what hangs on its outcome. Second, the trustee scenario suggests that AIs will need to become very competent indeed before we are tempted to treat them as possessing human-quality intelligence suitable for use as a means to human ends. Third, questions about true intentionality or real consciousness are not relevant to the inquiry in the trustee scenario. Fourth, the constitutional personhood scenario suggests that these questions about mental states will indeed be relevant if we ask whether AIs ought to be treated as ends in themselves. Fifth, the answer to the personhood question is likely to be found two places--in our experience with AI and in our best theories about the underlying mechanisms of the human mind.

Today, we lack both experience with really capable AIs and well- confirmed theories of how the human mind works. Given these gaps, Turing's suggestion that we put aside the question whether AIs can think was a good one. Perhaps we have not put it far enough aside.

*1284 VI. PERSONHOOD RECONSIDERED

Finally, I would like to raise some questions about the implications of the AI debate for controversial questions in legal, moral, and political theory. Cognitive science may provide us with a better understanding of our concept of a person. Some of the most intractable questions in jurisprudence, as in ethics and politics, have concerned the borderlines of status--what is a person and why we do give human persons such strong legal protection? [FN175] Should animals have stronger legal rights? How should we treat criminal defendants with multiple personalities? [FN176] What should be the legal status of a fetus? Should trees have standing? Many of these questions remain unsettled. Disagreement about their proper answers has persisted and even intensified.

It seems that developments in cognitive science might eventually be brought to bear on some of these questions. For example, it could be argued that personhood is identical with humanity--that possession of the genetic material of homo sapiens is a necessary and sufficient condition for personhood. But cognitive science may give us a very different picture of personhood--a picture that casts doubt on the equivalence between humans and persons. At the other end of the spectrum, AI research might give us insight into the claim that groups have rights that are not reducible to those of individuals.

Thinking about the question whether AIs should ever be made legal persons does shed some light on the difficult questions the law faces about the status of personhood. It is not that we have discovered a theory of personhood that resolves hard questions about the borderlines of status. Rather, thinking about personhood for AIs forces us to acknowledge that we currently lack the resources to develop a fully satisfactory theory of legal or moral personhood. [FN177] There are reasons for our uneasiness about the hard cases at the borderline of status, and the thought *1285 experiment in which we have engaged can help us to get a firmer grasp on these reasons.

The first reason for our uneasiness concerns the relationship between our concept of personhood and our concept of humanity. All of the persons we have methave been humans, and the overwhelming majority have been normal humans who give clear behavioral evidence of being conscious, having emotions, understanding meanings, and so forth. Given this coincidence (in the narrow sense), it is not surprising that our concept of person is fuzzy at the edges. For most practical purposes, this fuzziness does not get in our way. We treat humans as persons, and we need not worry about why we do so.

There are, however, occasions on which this strategy fails. Two of the most prominent cases occur at the beginning and the end of human life. Abortion and the cessation of life-sustaining treatment for humans in permanent vegetative states both raise questions about the status of personhood that cannot be answered by a simple comparison with a normal human adult. A third case is that of those higher mammals that seem most likely to have a mental life that is similar to that of humans.

In these cases, we can see the second reason for the persistence of uneasiness about the borderline of personhood. With respect to fetuses, humans in vegetative states, and higher mammals, we lack the sort of evidence we would need to establish a clear-cut case of personhood. Fetuses and humans in permanent vegetative states do not behave as normal *1286 human adults do, but they are humans. [FN178] Similarly, we have not been able to communicate with higher mammals in a way that yields clear behavioral evidence of a mental life of human quality, and higher mammals like whales are clearly not humans. In none of these cases is the behavioral evidence sufficient to establish that persons are (or are not) present.

There is a third reason for our persistent doubts about the borderline of personhood. Cognitive science, so far, has not yielded well-confirmed theories of the brain processes that underlie mental states like consciousness, emotion, and so forth. Absent well-confirmed theories of underlying processes, we cannot make confident judgments that the elements of personhood are lacking in particular cases.

Our thought experiment does suggest what sort of evidence might be decisive. If AIs behaved the right way and if cognitive science confirmed that the underlying processes producing these behaviors were relatively similar to the processes of the human mind, we would have very good reason to treat AIs as persons. Moreover, in a future in which we interact with such AIs or with intelligent beings from other planets, we might be forced to refine our concept of person.

The question then becomes what do we do about the hard cases that arise today? Thoughts about the shape of an answer can begin with the nature of justification and argumentation, both moral and legal. Our unreflective intuitions and well-considered moral and legal judgments are rooted in particular cases. These paradigm cases are the stuff of ordinary practical discourse. We make analogies to the familiar cases. We try to bring order to our particular judgments by advancing more general theories. We seek reflective equilibrium between our considered judgments and general theories. Ordinary practical discourse is shallow in the sense that it can be (and usually is) limited to arguments rooted in our common sense and ordinary experience.

What do we do when we must decide a case that goes beyond these shallow waters--the tranquil seas where theories are connected to the ocean floor by familiar examples and strong intuitions? In deep and uncharted waters, we are tempted to navigate by grand theories, grounded on intuitions we pump from the wildest cases we can imagine. This sort of speculation is well and good, if we recognize it for what it is--imaginative theorizing. When it comes to real judges making decisions in real legal cases, we hope for adjudicators that shun deep waters and recoil *1287 from grand theory. When it comes to our own moral lives, we try our best to stay in shallow waters. [FN179]

The thought experiments in this Essay have taken us beyond the shallow waters of our intuitions and considered judgments. One way of expressing the result of our journey is this: Our theories of personhood cannot provide an a priori chart for the deep waters at the borderlines of status. An answer to the question whether artificial intelligences should be granted some form of legal personhood cannot be given until our form of life gives the question urgency. But when our daily encounters with artificial intelligence do raise the question of personhood, they may change our perspective about how the question is to be answered.

And so it must be with the hard questions we face today. Debates about the borderlines of status--about abortion, about the termination of medical treatment, and about rights for animals--will not be resolved by deep theories or the intuitions generated by wildly imaginative hypotheticals. Of course, many of us do believe in deep theories; we subscribe to a variety of comprehensive philosophical or religious doctrines. But in a modern, pluralist society, the disagreement about ultimate questions is profound and persistent. Resolution of hard cases in the political and judicial spheres requires the use of public reason. We have no realistic alternative but to seek principled compromise based on our shared heritage of toleration and respect. If there is no common ground on which to build a theory of personhood that resolves a hard case, then judges must fall back on the principle of respect for the rights of those who mutually recognize one another as fellow citizens.

[FNa1]. Professor of Law and William M. Rains Fellow, Loyola Law School, Loyola Marymount University. B.A. 1981, University of California at Los Angeles; J.D. 1984, Harvard Law School. I owe thanks to Scott Altman, Ken Anderson, Don Brosnan, Don Crenshaw, Zlatan Damnjanovic, Michael Fitts, Kent Greenawalt, Sharon Lloyd, Shelley Marks, David Millon, Elyn Saks, and Paul Weithman for comments made on earlier versions of this essay. My colleagues Dave Leonard, Sam Pillsbury, Dave Tunick, and Peter Tiersma have been generous in sharing criticisms and suggestions. Bill Mulherin of the William M. Rains Law Library and Jai Gohel of the Loyola Law School Class of 1992 provided valuable research assistance. Finally, I am grateful to the editors of this review for their many helpful suggestions.

[FN1]. For an introduction to cognitive science and the philosophy of mind, see OWEN J. FLANAGAN, JR., THE SCIENCE OF THE MIND 1-22 (2d ed. 1991). For the purposes of this essay, I will not address the question as to which computer architectures could produce artificial intelligence. For example I will not discuss the question whether parallel, as opposed to serial, processing would be required. Similarly, I will not discuss the merits of connectionist as opposed to traditional approaches to AI. For a comparison of parallel distributed processing with serial processing, see id. at 224-41. These issues are moot in one sense. A digital computer can, in principle, implement any connectionist or parallel approach. On the other hand, there could be one very important practical difference: the parallel architecture could turn out to be much faster.

[FN2]. There is a debate within the artificial intelligence community as to the goal of AI research. The possibilities range from simply making machines smarter to investigating the nature of human intelligence or, more broadly, the nature of all intelligence. See Bob Ryan, AI's Identity Crisis, BYTE, Jan. 1991, at 239, 239-40. Owen Flanagan distinguishes four programs of AI research. Nonpsychological AI research involves building and programming computers to accomplish tasks that would require intelligence if undertaken by humans. Weak psychological AI views computer models as a tool for investigating human intelligence. Strong psychological AI assumes that human minds really are computers and therefore in principle can be duplicated by AI research. Suprapsychological AI investigates the nature of all intelligence and hence is not limited to investigating the human mind. See FLANAGAN, supra note 1, at 241-42. This Essay discusses the philosophical foundations for Flanagan's third and fourth categories of AI research.

[FN3]. See generally Catharine W. Hantzis, Legal Innovation Within the Wider Intellectual Tradition: The Pragmatism of Oliver Wendell Holmes, Jr., 82 NW. U. L. REV. 541, 561-75, 595-99 (1988) (discussing Holme's jurisprudential focus on concrete issues rather than generalities); Steven D. Smith, The Pursuit of Pragmatism, 100 YALE L.J. 409, 409-12 (1990) (discussing the renewed popularity of legal pragmatism); Symposium: The Renaissance of Pragmatism in American Legal Thought, 63 S. CAL. L. REV. 1569 (1990) (collecting articles espousing diverse views of legal pragmatism). The law is a "pragmatic" context in the sense that legal decisions are made for practical purposes with consequences in mind and in the sense that foundationalist philosophical theories do not play a role in legal reasoning. This assertion does not depend on the further claim that legal actors have adopted American pragmatism as part of their world view.

[FN4]. In addition, the Essay advances a more modest hypothesis. Examining the debate over the possibility of AI through legal examples illuminates the consequences of the arguments made in the debate, and this pragmatic assessment has a bearing on the arguments, even if it is not decisive. This Essay surely does not suffice to confirm the ambitious hypothesis in the text. I hope that it has established the more modest claim presented in this footnote. The proof, of course, is in the pudding.

[FN5]. The point is related to that made by Sir John Fortescue about the common law of England. There is a presumption in favor of its wisdom, because it has been tested by long experience. See SIR JOHN FORTESCUE, DE LAUDIBUS LEGUM ANGLIE 39-41 (S.B. Chrimes ed., Wm. W. Gaunt & Sons 1986) (1537); see also J.G.A. POCOCK, THE MACHIAVELLIAN MOMENT: FLORENTINE POLITICAL THOUGHT AND THE ATLANTIC REPUBLICAN TRADITION 9-18 (1975) (offering a critique of Fortescue's laudatory expositions of English law and arguing that because English law is based on accumulated experiences--elevated to the level of custom--its very existence presumes its validity, thus preempting rational scrutiny of English law's assumption that it is well suited to the needs of the English).

[FN6]. Here I adopt the view of Ronald Dworkin. See RONALD DWORKIN, LAW'S EMPIRE 91-96, 147-50, 276-400 (1986); RONALD DWORKIN, A MATTER OF PRINCIPLE 3- 42 (1985); RONALD DWORKIN, TAKING RIGHTS SERIOUSLY 1-80 (1977).

[FN7]. Cf. SUSAN L. HURLEY, NATURAL REASONS 15-18, 225-53 (1987) (advocating coherence approach to reasoning in general and legal reasoning in particular); S.L. Hurley, Coherence, Hypothetical Cases, and Precedent, 10 OXFORD J. LEGAL STUD. 221, 222-32 (1990) (same, with emphasis on legal reasoning).

[FN8]. See JOHN RAWLS, A THEORY OF JUSTICE 48-51 (1971).

[FN9]. I say "may provide" advisedly. We must be on guard against an easy or unthinking move from a legal conclusion to a moral one. In many circumstances, there are good reasons for answering a moral question differently from a legal one. Most obviously, the costs of legal enforcement of a norm are quite different than the costs of social enforcement of a moral norm. Moreover, inmany cases, the law will simply be morally wrong. The fact that a legal rule has survived a very long time does tell us that it has not led to the collapse of the society that enforces it, but it does not tell us directly whether that society would be better off without it. The legal case may bear on the moral one, but not being irrelevant is far from being decisive. I owe thanks to Elyn Saks for prompting me to qualify my argument in this regard.

[FN10]. See Christopher Stone, Should Trees Have Standing?--Toward Legal Rights for Natural Objects, 45 S. CAL. L. REV. 450, 453-57 (1972) [hereinafter Stone, Should Trees Have Standing?]. So far as I know, Stone became the first legal thinker to raise the questions asked by this Essay in a footnote to his famous 1972 essay. See id. at 456 n.26 (raising the question as to whether analysis applicable to natural objects such as trees would be appropriate to "computers"). Stone returned to this issue in 1987. See CHRISTOPHER STONE, EARTH AND OTHER ETHICS 12, 28-30, 65-67 (1987) [hereinafter STONE, EARTH AND OTHER ETHICS] (asking whether a "robot" should have standing and discussing criminal liability of AIs).

[FN11]. The phrase is Sir Edward Coke's. See Prohibitions Del Roy, 12 Coke Rep. 63, 65, 77 Eng. Rep. 1342, 1343 (1608).

[FN12]. THOMAS HOBBES, ELEMENTS OF PHILOSOPHY (1655), reprinted in 1 THE ENGLISH WORKS OF THOMAS HOBBES 1, 3 (William Molesworth ed., London, J. Bohn 1839); see also THOMAS HOBBES, LEVIATHAN (1670), reprinted in 3 THE ENGLISH WORKS OF THOMAS HOBBES 1, supra, at 29-32 [hereinafter LEVIATHAN] (equating reason with computation or "reckoning of the consequences"). Hobbes uses "ratiocination" to mean reasoning.

[FN13]. RENE DESCARTES, DISCOURSE ON THE METHOD OF RIGHTLY CONDUCTING ONE'S REASON AND SEEKING TRUTH IN THE SCIENCES (1637), reprinted in THE ESSENTIAL DESCARTES 138 (Margaret D. Wilson ed., 1969). This passage was likely inspired by Descartes' experience with the French Royal Gardens, which included a miniature society inhabited by hydraulically animated robots. As visitors walked along garden paths they set the robots bodies in motion. The robots actually played musical instruments and spoke. See FLANAGAN, supra note 1, at 1-2.

[FN14]. See Feng-hsiung Hsu et al., A Grandmaster Chess Machine, SCI. AM., Oct. 1990, at 44, 44.

[FN15]. See, e.g., ANN VON DER LIETH GARDNER, AN ARTIFICIAL INTELLIGENCE APPROACH TO LEGAL REASONING 1-24 (1987); RICHARD E. SUSSKIND, EXPERT SYSTEMS IN LAW: A JURISPRUDENTIAL INQUIRY (1987); ALAN TYREE, EXPERT SYSTEMS IN LAW 7-11 (1989); L. Thorne McCarty, Artificial Intelligence and Law: How to Get There From Here, 3 RATIO JURIS 189, 189-200 (1990); Edwina L. Rissland, Artificial Intelligence and Law: Stepping Stones to a Model of Legal Reasoning, 99 YALE L.J. 1957, 1961-64 (1990).

[FN16]. Overviews of the debate over the possibility of AI are found in JAMES H. FETZER, ARTIFICIAL INTELLIGENCE: ITS SCOPE AND LIMITS 3-27, 298-303 (1990); JOHN HAUGELAND, ARTIFICIAL INTELLIGENCE: THE VERY IDEA 2-12 (1985); RAYMOND KURZWEIL, THE AGE OF INTELLIGENT MACHINES 36-40 (1990); and in the essays collected in THE PHILOSOPHY OF ARTIFICIAL INTELLIGENCE (Margaret A. Boden ed., 1990) and in THE ARTIFICIAL INTELLIGENCE DEBATE: FALSE STARTS, REAL FOUNDATIONS (Stephen Graubard ed., 1988). For a strong statement of the view that AI is impossible, see HUBERT DREYFUS, WHAT COMPUTERS CAN'T DO: THE LIMITS OF ARTIFICIAL INTELLIGENCE 285-305 (rev. ed. 1979).

[FN17]. See Alan M. Turing, Computing Machinery and Intelligence, 59 MIND 433 (1950), reprinted in THE PHILOSOPHY OF ARTIFICIAL INTELLIGENCE, supra note 16, at 40 (subsequent citations to pagination in anthology). For a recent discussion and defense of the Turing Test, see Daniel C. Dennett, Can Machines Think?, in KURZWEIL, supra note 16, at 48. For a recent critique, see Donald Davidson, Turing's Test, in MODELLING THE MIND 1 (K.A. Mohyeldin Said et al. eds., 1990). For a report on a recent competition testing present-day computers and programs in the Turing format, see Carl Zimmer, Flake of Silicon, DISCOVER, Mar. 1992, at 36, 36-38.

[FN18]. See Turing, supra note 17, at 41-48.

[FN19]. See JOHN R. SEARLE, MINDS, BRAINS AND SCIENCE 28-41 (1984); John R. Searle, Author's Response, 3 BEHAVIORAL & BRAIN SCI. 450 (1980); John R. Searle, Is the Brain a Digital Computer?, 64 PROC. & ADDRESSES AM. PHIL. ASS'N, Nov. 1990, at 21, 21; John R. Searle, Minds, Brains & Programs, 3 BEHAVIORAL & BRAIN SCI. 417 (1980), reprinted in THE PHILOSOPHY OF ARTIFICIAL INTELLIGENCE, supra note 16, at 67 [hereinafter Searle, Minds, Brains & Programs; subsequent citations to pagination in anthology]; John R. Searle, "The Emperor's New Mind": An Exchange, N.Y. REV. BOOKS, June 14, 1990, at 58 (letter to the editor with response from John Maynard Smith).

[FN20]. Searle, Minds, Brains & Programs, supra note 19, at 70.

[FN21]. Id.

[FN22]. Id. at 83-84.

[FN23]. Id. at 70-71.

[FN24]. See, e.g., Robert P. Abelson, Searle's Argument Is Just a Set of Chinese Symbols, 3 BEHAVIORAL & BRAIN SCI. 424 (1980); Ned Block, What Intuitions About Homunculi Don't Show, 3 BEHAVIORAL & BRAIN SCI. 425 (1980); Bruce Bridgeman, Brains + Programs = Minds, 3 BEHAVIORAL & BRAIN SCI. 427 (1980); Arthur C. Danto, The Use and Mention of Terms and the Simulation of Linguistic Understanding, 3 BEHAVIORAL & BRAIN SCI. 428 (1980); Daniel Dennett, The Milk of Human Intentionality, 3 BEHAVIORAL & BRAIN SCI. 428 (1980); John C. Eccles, A Dualist-Interactionist Perspective, 3 BEHAVIORAL & BRAIN SCI. 430 (1980); J.A. Fodor, Searle on What Only Brains Can Do, 3 BEHAVIORAL & BRAIN SCI. 431 (1980); John Haugeland, Programs, Causal Powers, and Intentionality, 3 BEHAVIORAL & BRAIN SCI. 432 (1980); Douglas R. Hofstadter, Reductionism and Religion, 3 BEHAVIORAL & BRAIN SCI. 433 (1980); B. Libet, Mental Phenomena and Behavior, 3 BEHAVIORAL & BRAIN SCI. 434 (1980); William G. Lycan, The Functionalist Reply (Ohio State), 3 BEHAVIORAL & BRAIN SCI. 435 (1980); John C. Marshall, Artificial Intelligence--The Real Thing?, 3 BEHAVIORAL & BRAIN SCI. 435 (1980); Grover Maxwell, Intentionality: Hardware, Not Software, 3 BEHAVIORAL & BRAIN SCI. 437 (1980); John McCarthy, Beliefs, Machines, and Theories, 3 BEHAVIORAL & BRAIN SCI. 435 (1980); E.W. Menzel, Jr., Is the Pen Mightier than the Computer?, 3 BEHAVIORAL & BRAIN SCI. 438 (1980); Marvin Minsky, Decentralized Minds, 3 BEHAVIORAL & BRAIN SCI. 439 (1980); Thomas Natsoulas, The Primary Source of Intentionality, 3 BEHAVIORAL & BRAIN SCI. 440 (1980); Roland Puccetti, The Chess Room: Further Demythologizing of Strong AI, 3 BEHAVIORAL & BRAIN SCI. 441 (1980); Zenon W. Pylyshyn, The "Causal Power" of Machines, 3 BEHAVIORAL & BRAIN SCI. 442 (1980); Howard Rachlin, The Behavioralist Reply (Stony Brook), 3 BEHAVIORAL & BRAIN SCI. 444 (1980); Martin Ringle, Mysticism as a Philosophy of Artificial Intelligence, 3 BEHAVIORAL & BRAIN SCI. 444 (1980); Richard Rorty, Searle and the Special Powers of the Brain, 3 BEHAVIORAL & BRAIN SCI. 445 (1980); Roger C. Shank, Understanding Searle, 3 BEHAVIORAL & BRAIN SCI. 446 (1980); Aaron Sloman & Monica Croucher, How to Turn an Information Processor Into an Understander, 3 BEHAVIORAL & BRAIN SCI. 447 (1980); William E. Smythe, Simulation Games, 3 BEHAVIORAL & BRAIN SCI. 448 (1980); Donald O. Walter, The Thermostat and the Philosophy Professor, 3 BEHAVIORAL & BRAIN SCI. 449 (1980); Robert Wilensky, Computers, Cognition and Philosophy, 3 BEHAVIORAL & BRAIN SCI. 449 (1980).

[FN25]. ROBERT A. HEINLEIN, THE MOON IS A HARSH MISTRESS 13-14 (1966). Copyright 1966 by Robert A. Heinlein. Reprinted by permission of the Berkley Publishing Group.

[FN26]. See JOHN CHIPMAN GRAY, THE NATURE AND SOURCES OF THE LAW (Roland Gray ed., MacMillan 1921) (1909); see also Stephen C. Hicks, On the Citizen and the Legal Person: Toward the Common Ground of Jurisprudence, Social Theory, and Comparative Law as the Premise of a Future Community, and the Role of the Self Therein, 59 U. CIN. L. REV. 789, 808-21 (1991) (discussing the construct of the legal person in the context of social theory); Richard Tur, The 'Person' in Law, in PERSONS AND PERSONALITY: A CONTEMPORARY INQUIRY 116, 116-27 (Arthur Peacocke & Grant Gillett eds., 1987) (providing a concise summary of the concept of the person within several areas of the law).

[FN27]. GRAY, supra note 26, at 27.

[FN28]. This statement is not quite correct. As Christopher Stone points out, X may be given the legal status of personhood in order to confer rights on Y. Thus, giving a fetus the status of personhood might confer the right to sue in tort for injury to it on its parents. See STONE, EARTH AND OTHER ETHICS, supra note 10, at 43.

[FN29]. GRAY, supra note 26, at 46.

[FN30]. See id. at 48-49; Stone, Should Trees Have Standing?, supra note 10, at 5.

[FN31]. See STONE, EARTH AND OTHER ETHICS, supra note 10, at 22 (citing Mullick v. Mullick, 52 I.A. 245, 256-61 (P.C. 1925)).

[FN32]. See David Millon, Theories of the Corporation, 1990 DUKE L.J. 201, 206 (discussing historical development of the theory of corporations).

[FN33]. GRAY, supra note 26, at 48-49, 53. The corresponding question about the point of making an inanimate thing the subject of a legal right is easier to answer. Giving temples or trees rights that can be enforced by guardians or private attorneys general has an obvious objective--to protect the tree or temple. See generally, Stone, Should Trees Have Standing?, supra note 10 (discussing implications of legal rights for the environment). Of course, the same sort of argument can be made for making inanimate objects the subjects of legal duties. The tree can be made liable for damage done by a falling branch to induce a natural person to take preventative action--calling the tree trimmers.

[FN34]. GRAY, supra note 26, at 50-51.

[FN35]. Id. at 52.

[FN36]. It is important to remember that the question whether something should be given legal personhood is distinct from the question whether it has moral rights. (I use the term "moral right" to refer to moral claim rights, that is, to moral claims that individuals have on one another, with "moral" used in contrast with "legal.") Thus, the fact that corporations are legal persons with constitutional rights--such as the rights to freedom of speech, due process, and equal protection of the laws--does not entail the conclusion that corporations have equivalent moral rights. Vice versa, the possession of moral rights does not lead automatically to the conclusion that there should be corresponding legal rights. See STONE, EARTH AND OTHER ETHICS, supra note 10, at 43, 73. This point is a narrow one. The factors that bear on the decision to grant legal rights may bear on the question whether corresponding moral rights exist, but the relationship between the two sorts of rights is not one of entailment in either direction.

[FN37]. For those unfamiliar with the common-law term "trust," it is defined as "a fiduciary relationship with respect to property, subjecting the person by whom the title to property is held to equitable duties to deal with the property for the benefit of another person, which arises as a result of a manifestation of an intention to create it." RESTATEMENT (SECOND) OF TRUSTS § 2 (1959). The trustee is the legal person who administers the trust--invests trust assets, and so forth. The beneficiary is the person for whom the trust is maintained, for example, the person who receives income from the trust. The settlor is the person who establishes the trust. The terms of a trust are the directives to the trustee in the document or instrument creating the trust.

[FN38]. See Christina Toh-Pantin, Wall Street Sees Tide Turing on Program Trading, Reuters Financial Report, Oct. 27, 1989, available in LEXIS, Nexis Library, FINRPT File; Anise C. Wallace, 5 Wall St. Firms Move to Restrict Program Trades, N.Y. TIMES, May 11, 1988, at A1.

[FN39]. At this stage, the question might be raised whether the trustee would violate the duty not to delegate the administration of the trust by failing to exercise independent judgment. See RESTATEMENT (SECOND) OF TRUSTS § 171. The answer is probably no, for two reasons. First, this duty not to delegate can be overridden by the terms of the trust. See Henshie v. McPherson & Citizens State Bank, 177 Kan. 458, 478, 280 P.2d 937, 952 (1955) (holding that settlor can waive the duty not to delegate by including such a waiver in the terms of the trust instrument); RESTATEMENT (SECOND) OF TRUSTS § 171 cmt. j. Second, in this scenario the trustee is not delegating the administration to another person. Rather, the trustee is using the program as an instrument; the law might consider the program to be part of the terms of the trust.

The development of the legal standard for delegation of trust duties is suggestive. The traditional view was based upon how the courts classified the delegated powers. If they are merely "ministerial" the court may allow such a delegation. See Morville v. Fowle, 144 Mass. 109, 113, 10 N.E. 766, 769 (1887). More recently, courts have decided the issue based upon whether the delegation is a matter of usual business practice. See Walters-Southland Inst. v. Walker, 222 Ark. 857, 861, 263 S.W.2d 83, 84-85 (1954). Thus, if the use of AIs to perform the functions of trustees became more common, the courts would become more accepting, reasoning that such use had become usual business practice.

[FN40]. The fact that an AI is owned should not, by itself, preclude it from serving as a trustee. Corporations are owned by stockholders, but they are legally entitled to serve as trustees. See infra text accompanying note 43. The analogy between an AI and its owner and a corporation and its stockholder can be extended. For example, the role of an AI as a trustee would, like a corporation, be constrained by the scope of powers given to the corporation by the "owner." In the case of a corporation, the owners are the stockholders; in the case of an AI, the owner would be the creator, the creator's employer, or the purchaser. In a corporation, as long as the stockholders approve of the corporation's activities as trustee, the corporation is acting properly within the scopeof its power. See Hossack v. Ottawa Dev. Ass'n, 244 Ill. 274, 295, 91 N.E. 439, 447 (1910). In a similar manner, as long as AI the acts within the scope contemplated by its owner, it too could be acting within the scope of its trusteeship power.

[FN41]. See RESTATEMENT (SECOND) OF TRUSTS § 89. The Restatement specifically provides that married women, see id. § 90, infants, see id. § 91, insane persons, see id. § 92, aliens, see id. § 93, and nonresidents, see id. § 94, may serve as trustees. But see Clary v. Spain, 119 Va. 58, 61-62, 89 S.E. 130, 131 (1916) (removing infant as trustee).

[FN42]. See RESTATEMENT (SECOND) OF TRUSTS § 95 (specifying the United States or a state can be trustee).

[FN43]. See id. § 96. The Restatement also has provisions dealing with unincorporated associations, see id. § 97, and partnerships, see id. § 98.

[FN44]. Of course, just as a corporation has stockholders and directors, an AI could have owners and programmers. Perhaps the difference between the case of an AI and a corporation, with respect to the role for humans, is not as significant as it might at first appear.

[FN45]. For example, at this stage in my argument, I do not want to consider the question whether an AI would be competent to administer a charitable trust, the terms of which required the trustee to make aesthetic judgments about the worthiness of competing applicants for grants to produce operas or ballets. Interestingly, however, the law itself will rarely second guess such complex judgments. The courts generally will not interfere with the selection of the beneficiary made by the trustee as long as the general description left by the settlor gives the court enough guidance to determine if the trustee's administration was proper. See GEORGE T. BOGERT, TRUSTS § 55, at 210 (6th ed. 1987).

[FN46]. Assume further that if fulfillment of the terms is impossible, the trust instrument provides for the termination and distribution of assets according to explicit instructions.

[FN47]. As to publicly traded securities, this assumption may not require a very "smart" expert system. If the market truly takes a random walk, then any reasonably diversified portfolio of publicly traded securities is as good as any other.

[FN48]. This task, of course, is not a simple one. A trustee may receive clear and unambiguous notice of the death of a beneficiary, but this need not be the case. The AI might need to engage a private detective if benefit checks were returned unopened or were not cashed for a substantial period of time.

[FN49]. This objection was called to my attention by Catharine Wells and Zlatan Damnjanovic.

[FN50]. See RESTATEMENT (SECOND) OF TRUSTS § 174 (1959).

[FN51]. See id. § § 201, 205. Failure to meet the standard of care and skill may result in a finding of negligence and assessment of damages against the trustee, a reduction in the trustee's compensation, or removal of the trustee from office. See, e.g., Riegler v. Riegler, 262 Ark. 70, 77, 553 S.W.2d 37, 40-41 (1977); Neely v. People's Bank, 133 S.C. 43, 47, 130 S.E. 550, 551 (1925).

[FN52]. See RESTATEMENT (SECOND) OF TRUSTS § 205.

[FN53]. Of course if the AI were infallible then we might suppose this issue to be moot. But this assumption is unrealistic. For example, the program might have a bug that caused the program to make a bad investment or to waste the trusts assets by churning--i.e., by buying and selling repeatedly in a short period of time--thus incurring large broker's fees. We surely cannot rule out the possibility of such bugs in advance. Even lengthy experience without the appearance of such bugs does not make them impossible.

[FN54]. See L. Nancy Birnbaum, Strict Products Liability and Computer Software, 8 COMPUTER/L.J. 135, 143-55 (1988); Michael C. Gemignani, Product Liability and Software, 8 RUTGERS COMPUTER & TECH. L.J. 173, 189-99 (1981); Lawrence B. Levy & Suzanne Y. Bell, Software Product Liability: Understanding and Minimizing the Risks, 5 HIGH TECH L.J. 1, 8-15 (1990).

[FN55]. If we take the common-law approach, potential for criminal liability would not be a prerequisite for service as a trustee. The traditional common- law view was that a trustee could not be held liable because larceny required an initial trespass and trover. Because the trustee has legal title, there is no trespass, and therefore no larceny. See People v. Shears, 158 A.D. 577, 580, 143 N.Y.S. 861, 863, aff'd, 209 N.Y. 610, 103 N.E. 1129 (1913). Modern statutes, however, do hold the trustee criminally responsible. See, e.g., Cal. Penal Code § 506 (West 1988).

[FN56]. See generally C.L. TEN, CRIME, GUILT, AND PUNISHMENT 7-85 (1987) (discussing, evaluating, and comparing various theories of punishment).

[FN57]. I should note a possible exception that has been ruled out by the description of the first scenario. I have assumed that the expert trust administration system is programmed to achieve the purposes of the trust. It would be possible, however, to program an expert system to steal or commit some other crime. Moreover, a sufficiently complex and intelligent AI might commit a crime on its own initiative. For example, our trustee program might discover that it can garner information from other AIs that possess inside information and run afoul of the federal securities laws. Cf. HANS MORAVEC, MIND CHILDREN: THE FUTURE OF ROBOT AND HUMAN INTELLIGENCE 49 (1988) (describing intelligent robot that commits burglary to gain access to power supply in home of neighbor of robot's owner).

[FN58]. For exploration of nonutilitarian punishment theory, see Samuel H. Pillsbury, Emotional Justice: Moralizing the Passions of Criminal Punishment, 74 CORNELL L. REV. 655, 658-74, 685-98 (1989); Samuel Pillsbury, Evil and the Law of Murder, 24 U.C. DAVIS L. REV. 437, 440-47 (1990).

[FN59]. The classic statements of retributive or desert-based theories of punishment are those by Kant and Hegel. See IMMANUEL KANT, THE METAPHYSICS OF MORALS 140-145 (Mary Gregor ed., Cambridge University Press 1991) (1797) (also available in an earlier translation of a portion of the original work, IMMANUEL KANT, THE METAPHYSICAL ELEMENTS OF JUSTICE 99-107 (John Ladd trans., 1965) (1797)); GEORG HEGEL, ELEMENTS OF THE PHILOSOPHY OF RIGHT 127-31 (Allen W. Wood ed. & H.B. Nisbet trans., Cambridge University Press 1991) (1821) (also available in the earlier translation, GEORG HEGEL, PHILOSOPHY OF RIGHT 68-74 (T.M. Knox trans., 1952) (1821)). For a recent statement of retributive theory, see JEFFRIE G. MURPHY, RETRIBUTION, JUSTICE, AND THERAPY: ESSAYS IN THE PHILOSOPHY OF LAW 77-127 (1979).

[FN60]. This point about desert suggests a related objection. The concept of moral duty arises in a particular human context. One picture of moral duties is that they exist where there is a temptation to be overcome. For example, we might think that there is a duty not to steal the property of another because there are temptations to do so. It might be argued that an AI could not be the subject of this sort of duty because it lacks the necessary moral psychology. In particular, an expert trust administration system could not be tempted and therefore could not have a duty to overcome temptation. The point of this objection is not that there is some practical problem with making artificial intelligences trustees, but is instead that we ought not speak about them as having duties if we want that concept to retain its ordinary moral meaning. The law speaks of trustees as having legal duties, and with natural persons these legal duties respond to the same feature of human moral psychology, i.e., temptation, as do moral duties. Applying the concept of legal duty to AIs would thus drive a wedge between the concepts of legal and moral duty. Of course, we can choose to do this, but should we? Would it be better to create a new legal category for expert systems that have human-like competencies but lack some features of human moral psychology?

I do not want to suggest that I am committed to the picture of duty as correlative to temptation that is hypothesized in this footnote. For example, if moral duties are correlative to temptation, then God could not be the subject of moral duties, a conclusion many theists would reject. Nonetheless, the questions raised seem important and unanswered. This issue was brought to my attention by Sharon Lloyd. The doubt about the picture of duty as correlative to temptation was raised by Paul Weithman.

[FN61]. There are two sorts of educative theories. The first sort maintains the purpose of punishment is the education of the individual who is punished. See Herbert Morris, A Paternalistic Theory of Punishment, in PATERNALISM 139, 140-44 (Rolf Sartorious ed., 1983). The second sort of educative theory maintains that punishment educates those who witness or learn of the punishment of others. See EMILE DURKHEIM, THE DIVISION OF LABOR IN SOCIETY 86 (George Simpson trans., 1933); Charles Nesson, The Evidence or the Event? On Judicial Proof and the Acceptability of Verdicts, 98 HARV. L. REV. 1357, 1359-60 (1985); Lawrence B. Solum & Stephen Marzen, Truth and Uncertainty: Legal Control of the Destruction of Evidence, 36 EMORY L.J. 1085, 1167 (1987).

[FN62]. This point was called to my attention by Jeff Sherman. There is a problem with the statement of the objection: it assumes that an AI would consist only of a system of rules. But this need not be the case. Neural net technology, for example, does not operate this way. See generally FLANAGAN, supra note 1, at 224-41 (discussing parallel distributed processing, including neural nets).

[FN63]. See RESTATEMENT (SECOND) OF TRUSTS § 167(1) (1959). In addition, the law imposes a duty on the trustee to recognize any change in circumstances that would require some action by the trustee when that change is reasonably discoverable. See id. § 167(3).

[FN64]. Id. § 167 illus. 1.

[FN65]. Another example of change in circumstances is posed by the case in which a court permitted the sale of a school receiving money from a trust because the surrounding neighborhood became too dangerous to provide safety for the schoolchildren for whom the trust was created. See Anderson v. Ryland, 232 Ark. 335, 346, 336 S.W.2d 52, 58-59 (1960). We might imagine that an AI faced with declining enrollment would simply continue to serve fewer and fewer children--perhaps with a feeling of satisfaction at the increase in per pupil expenditures.

[FN66]. See Daniel C. Dennett, Cognitive Wheels: The Frame Problem of AI, in THE PHILOSOPHY OF ARTIFICIAL INTELLIGENCE, supra note 16, at 147, 148-50; see also FLANAGAN, supra note 1, at 250-52 (discussing problem of giving computers common sense and proposing that one cannot program a computer with a set of rules from which it can draw inferences; rather, subtle features from particular situations would stimulate neural network that would respond with common sense appropriate for situation).

[FN67]. See RESTATEMENT (SECOND) OF TRUSTS § § 183, 232 (1959).

[FN68]. In the situation presented, the courts usually let such a decision stand as long as it was made in good faith. See Dumaine v. Dumaine, 301 Mass. 214, 222, 16 N.E.2d 625, 629 (1938); In re Frances M. Johnson Trust, 211 Neb. 750, 755, 320 N.W.2d 466, 469 (1982).

[FN69]. I owe this example to Michael Fitts, who has pressed it quite forcefully.

[FN70]. Imagine that the AI accesses the Martindale-Hubbell Law Directory on line, and that it has a law firm selection formula based on area of specialization, lawyer experience and qualifications, and so forth. The legal capacity of the AI to enter into an agreement with an attorney depends on whether the legal system will treat an AI as a legal person.

[FN71]. Perhaps for this reason, the duty of care the law imposes on a human trustee in such situations is limited. A human trustee has a duty to obtain the advice of an expert such as an attorney and will be protected from personal liability if she takes reasonable care in the selection of the advisor. See In re Davis, 183 Mass. 499, 501, 67 N.E. 604, 605 (1903).

[FN72]. The most famous example, of course, is the fictional case of Jarndyce v. Jarndyce. See CHARLES DICKENS, BLEAK HOUSE (1853).

[FN73]. This would make the rights of the beneficiaries under the trust legally unenforceable. The option of termination is not wholly out of line with existing practice, however. For example, the courts will terminate a trust when the settlor's purpose has been frustrated. See Hughes v. Neely, 332 S.W.2d 1, 8 (Mo. 1960).

[FN74]. Cases exist in which the settlor has appointed an adviser to the trustee, the consent of whom the trustee must obtain before making certain types of decisions. See Gathright's Trustee v. Gaut, 276 Ky. 562, 564-65, 124 S.W.2d 782, 783-84 (1939). Hypothetically, the AI would serve as the actual legal trustee, with the non-AI entity playing the role of adviser whose consent must be given in certain crisis situations. Or the terms of the trust might give the adviser power to authorize or review the AI's discretionary decisions and to reverse or change them. Advisory trustees would have "strictly limited capacities and duties, that is, an assistant to the trustee limited in his capacity by the terms of the trust, having no right or authority further than the capacity of advising as provided in the instrument." Id. at 565, 124 S.W/2d at 784.

[FN75]. If an AI could not respond to questions posing hypothetical legal decisions, such questions could be used to distinguish the AI from the human in the Turing Test.

[FN76]. The evidence for this proposition is negative. I have found no authority that indicates that a category of limited-purpose trustee currently exists. Of course, some people are not really competent to serve as the trustee for complex trusts even though they may be competent to serve for simpler trusts. Despite the real variations in the ability level of natural persons, the law seems to be that all natural persons can serve as trustees for all sorts of trusts. Cf. RESTATEMENT (SECOND) OF TRUSTS § 89 (1959) (stating unqualified capacity of natural persons to serve as trustees).

[FN77]. This is not to say that an AI would need to be competent to serve as a trustee for every conceivable sort of trust in order to recognized as a legal person. Legally, any natural person has the capacity to serve as a trustee for any trust. But many humans would be unable to do a very good job of carrying out a trust that required complex judgment or specialized competencies.

[FN78]. The full Turing Test would require human-like competence in response to questions on any topic, but a trustee does not need such omnicompetence. An AI could be a competent trustee, but be unable intelligently to discuss either baseball or cake-baking. See infra text accompanying notes 171-73 (discussing the Turing Test and the possibility that it is biased).

[FN79]. For example, in a case in which a trustee is selected for the limited purpose of taking title to an author's literary property, that trustee must still be able to make the required discretionary decisions in the management of copyrights, royalties, and the like. See In re Estate of Hellman, 134 Misc. 2d 525, 528-30, 511 N.Y.S.2d 485, 486-88 (N.Y. County Sur. Ct. 1987). Thus, merely limiting the scope of the trustee's power does not guarantee a solution to the problem of capacity to make discretionary decisions.

[FN80]. In comments on an earlier version of this Essay, Michael Fitts made this point.

[FN81]. Felix S. Cohen, Transcendental Nonsense and the Functional Approach, 35 COLUM. L. REV. 809, 809 (1935).

[FN82]. For this point, I am indebted to David Millon.

[FN83]. ARTHUR C. CLARKE, 2001: A SPACE ODYSSEY 155-56 (1968). Copyright by the author. Reprinted by permission of the author and the author's agents, Scott Meredith Literary Agency, Inc., 845 Third Avenue, New York, New York 10022.

[FN84]. How far in the future? We do not know, and I certainly do not know enough to make an educated guess. Raymond Kurzweil estimates that an AI will pass the Turing Test between 2020 and 2070. See KURZWEIL, supra note 16, at 483. Hans Moravec, Director of the Mobile Robot Laboratory of Carnegie Mellon University, predicts that "robots with human intelligence will be common within fifty years." MORAVEC, supra note 57, at 6.

[FN85]. EUGENE CHARNIAK & DREW MCDERMOTT, INTRODUCTION TO ARTIFICIAL INTELLIGENCE 7 (1985).

[FN86]. See JOHN L. POLLOCK, HOW TO BUILD A PERSON: A PROLEGOMENON 1-12 (1989).

[FN87]. See Bob Ryan, Dynabook Revisited with Alan Kay, BYTE, Feb. 1991, at 203, 203-06. The concept of "agents" plays a different role in Marvin Minsky's theory of intelligence. See MARVIN MINSKY, THE SOCIETY OF MIND 17-23 (1985).

[FN88]. Of course, the question arises whether the AI will hold the copyright in the romance novels that it writes. The National Commission on New Technological Uses of Copyrighted Works has taken the position that the author of a computer-generated work is the human user of the computer program. See NATIONAL COMM'N ON NEW TECHNOLOGICAL USES OF COPYRIGHTED WORKS, FINAL REPORT 112 (CCH) (1978). That conclusion has been challenged. See Pamela Samuelson, Allocating Ownership Rights in Computer-Generated Works, 47 U. PITT. L. REV. 1185, 1200-04 (1986); Timothy L. Butler, Note, Can a Computer Be an Author? Copyright Aspects of Artificial Intelligence, 4 HASTINGS COMM. & ENT. L.J. 707, 734-47 (1982).

[FN89]. See generally KENT GREENAWALT, SPEECH, CRIME, AND THE USES OF LANGUAGE 9-39 (1989) (discussing rationales for freedom of speech); Lawrence B. Solum, Freedom of Communicative Action: A Theory of the First Amendment Freedom of Speech, 83 NW. U. L. REV. 54, 68-86 (1989) (same). Analogously, utilitarian justifications might be developed for other rights of constitutional personhood that could be applied to AIs. See generally Kent Greenawalt, Utilitarian Justifications for Observance of Legal Rights, in ETHICS, ECONOMICS, AND THE LAW: NOMOS XXIV 139 (J. Roland Pennock & John W. Chapman eds., 1982) (discussing the relationship of morality and legal rights); Douglas Laycock, The Ultimate Unity of Rights and Utilities, 64 TEX. L. REV. 407, 413 (1985) (discussing the need to incorporate into an analysis of individual rights the utility of actions that threaten those rights). Of course, it may turn out that the utilitarian justifications of some rights are dependent on the utility to the right holder. In that case, we would be required to answer the question whether AIs can possess utilities.

[FN90]. I owe this example to Kent Greenawalt. Utilitarian arguments can be made that could justify the extension of just about any right to AIs on the ground that humans would benefit. For example, if AIs were more productive when unowned, then a utilitarian case could be made for extending the Thirteenth Amendment to AIs.

[FN91]. It should be noted, however, that free speech rights for AIs could be justified by deontological arguments, without making assumptions about the moral status of AIs themselves. For example, it might be argued that freedom of speech for AIs promotes the autonomy of human listeners. Cf. Thomas Scanlon, A Theory of Freedom of Expression, 1 PHIL. & PUB. AFF. 204, 215-20 (1972), reprinted in THE PHILOSOPHY OF LAW 153 (Ronald M. Dworkin ed., 1977) (exploring listener autonomy justification for freedom of speech); Solum, supra note 89, at 77-79 (same).

[FN92]. The concept of personhood has proven elusive. For illustrative attempts to gain purchase on it, see Tur, supra note 26, at 121-29 (exploring legal personhood), and Marcel Mauss, A Category of the Human Mind: The Notion of Person; the Notion of Self, in THE CATEGORY OF THE PERSON 1 (Michael Carrithers et al. eds. & W.D. Halls trans., 1985) (discussing the idea of a person by examining how various societies define the concept).

[FN93]. U.S. CONST. amend. XIV, § 1.

[FN94]. See id.; Madden v. Kentucky, 309 U.S. 83, 90 (1940); Colgate v. Harvey, 296 U.S. 404, 428-29 (1935), overruled on other grounds by Madden, 309 U.S. at 93.

[FN95]. See U.S. CONST. amend. XIV, § 1; Santa Clara County v. Southern Pac. R.R., 118 U.S. 394, 396 (1886). It might be argued that AIs should not be considered bearers of constitutional rights, because the framers of the Fourteenth Amendment did not have a specific intention to include them. Of course, the framers probably lacked any intentions at all with respect to artificial intelligences. Given the general principles they espoused, the question whether their intentions support giving AIs constitutional rights will turn initially on what general principles lie behind the framers' idea of personhood and then on more particular questions about consciousness, interests, and other qualities addressed below. See generally Michael Perry, The Legitimacy of Particular Conceptions of Constitutional Interpretation, 77 VA. L. REV. 669, 674-94 (1991) (arguing for a conception of originalism based on general principles); Lawrence B. Solum, Originalism as Transformative Politics, 63 TUL. L. REV. 1599, 1612-16 (1989) (same).

[FN96]. See, e.g., The Pipe Line Cases, 234 U.S. 548, 562-63 (1914) (White, C.J., concurring); Cotting v. Kansas City Stock Yards Co., 183 U.S. 79, 86 (1901).

[FN97]. See First Nat'l Bank v. Bellotti, 435 U.S. 765, 784-85 (1978).

[FN98]. This point is controversial. Compare Roger Scruton, Corporate Persons I, in 63 SUPPLEMENTARY VOLUME: PROCEEDINGS OF THE ARISTOTELIAN SOCIETY 239 (1989) (arguing that corporate persons have moral responsibilities that cannot be reduced to those of constituent natural individuals) with John Finnis, Corporate Persons II, in 63 SUPPLEMENTARY VOLUME: PROCEEDINGS OF THE ARISTOTELIAN SOCIETY 267 (arguing against this thesis).

[FN99]. See DAVID WIGGINS, SAMENESS AND SUBSTANCE 148-89 (1980); Christopher Gill, Introduction to THE PERSON AND THE HUMAN MIND 1, 2-12 (Christopher Gill ed., 1990); Adam Morton, Why There is No Concept of a Person, in THE PERSON AND THE HUMAN MIND, supra, at 39, 39-59; Amelie O. Rorty, Persons and Personae, in THE PERSON AND THE HUMAN MIND, supra, at 21, 27-33; Peter Smith, Human Persons, in THE PERSON AND THE HUMAN MIND, supra, at 61; David Wiggins, The Person as Object of Science, as Subject of Experience, and as Locus of value, in PERSONS AND PERSONALITY: A CONTEMPORARY INQUIRY, supra note 26, at 56, 69- 72.

[FN100]. See Harry G. Frankfurt, Freedom of the Will and the Concept of a Person, 68 J. PHIL. 6 (1971), reprinted in HARRY G. FRANKFURT, THE IMPORTANCE OF WHAT WE CARE ABOUT: PHILOSOPHICAL ESSAYS 11 (1988). The discussion of Kant's definition of personhood that appears below may be helpful. See infra note 137 and accompanying text.

[FN101]. E.O. Wilson espoused the view that promoting the human gene pool is a fundamental moral principle. See EDWARD O. WILSON, SOCIOBIOLOGY: THE NEW SYNTHESIS 120 (1975); see also FLANAGAN, supra note 1, at 265-305 (discussing Wilson's position); MORAVEC, supra note 57, at 2 (predicting the possibility of intelligent machines that could reproduce themselves and beat human DNA in evolutionary race).

Another variant of the anthropocentric argument could be made in religious form: humans are persons because God created humans in God's own image. This argument could not prevail in our pluralist society for reasons explored below. See infra text accompanying notes 145-47.

[FN102]. My position in this regard is similar to Kant's. Kant believed that humans would have moral duties to nonhuman persons. See infra note 137.

[FN103]. See TERMINATOR 2: JUDGMENT DAY (TriStar Entertainment 1991).

[FN104]. See MORAVEC, supra note 57, at 100.

[FN105]. This point might not be accepted by some animal rights activists with respect to higher mammalian life forms such as whales. See Anthony D'Amato & Sudhir K. Chopra, Whales: Their Emerging Right to Life, 85 AM. J. INT'L L. 21, 27 (1991). Of course, two other qualifications should be noted. First, we frequently encounter nonhuman candidates for personhood in fiction. Second, many people hold religious beliefs that there are nonhuman intelligences, and some people believe that they have personally encountered such intelligences.

[FN106]. See FLANAGAN, supra note 1, at 254. John Haugeland calls this the "hollow shell strategy." John Haugeland, Semantic Engines: An Introduction to Mind Design, in MIND DESIGN 1, 32 (John Haugeland ed., 1981).

[FN107]. See Turing, supra note 17, at 49-50. Although Turing dismissed the objection, he gave an answer in theological terms. Id. at 50. If God is omnipotent, she can give an AI a soul. Id.

[FN108]. A full explanation of the justification for the requirement of public reason is beyond the scope of this Essay, but the following two arguments are the most essential. First, modern society is characterized by the fact of pluralism: differences over comprehensive religious and philosophical conceptions of the good will persist without intolerable use of coercive force. See John Rawls, The Idea of an Overlapping Consensus, 7 OXFORD J. LEGAL STUD. 1, 4 (1987). Second, given the fact of pluralism, respect for citizens as free and equal members of society requires that the state give reasons for its conduct that all can accept as reasonable, given the plurality of fundamental beliefs. See Lawrence B. Solum, Pluralism and Modernity, 66 CHI.-KENT L. REV. 93, 99 (1991). But see Steven A. Gardbaum, Why the Liberal State Can Promote Moral Ideals After All, 104 HARV. L. REV. 1350, 1364-69 (1991) (criticizing the coercion argument); Michael J. Perry, Toward an Ecumenical Politics, 20 CAP. U. L. REV. 1, 17-18 (1991) (critiquing both the stability argument and the respect-for-persons argument in the context of religious reasons).

[FN109]. See Lawrence B. Solum, Faith and Justice, 39 DEPAUL L. REV. 1083, 1105-06 (1990); John Rawls, The Idea of Free Public Reason, Address at the inaugural Abraham Melden Lectures, Department of Philosophy, University of California at Irvine (Feb. 27 & Mar. 1, 1990). Kent Greenawalt has made a plausible case against this version of the requirement of public reasons. See KENT GREENAWALT, RELIGIOUS CONVICTIONS AND POLITICAL CHOICE 56-76 (1988) (arguing that nonpublic reasons may be employed when questions about status cannot be resolved by public reasons). Greenawalt might argue that the question as to whether AIs should be given constitutional rights is underdetermined by public reason. Therefore, nonpublic reasons, including religious reasons, legitimately can be brought to bear on the question.

[FN110]. 410 U.S. 113 (1973).

[FN111]. The implausibility of dualism is almost a dogma of philosophy of mind in the analytic tradition. See DANIEL C. DENNETT, CONSCIOUSNESS EXPLAINED 33-39 (1991); FLANAGAN, supra note 1, at 57-59, 216-24. But see W.D. HART, THE ENGINES OF THE SOUL 1-8 (1988) (defending a dualist position in philosophy of mind).

[FN112]. See Turing, supra note 17, at 52-53. The philosophical literature on consciousness is substantial. See, e.g., DENNETT, supra note 111; RAY JACKENDOFF, CONSCIOUSNESS AND THE COMPUTATIONAL MIND 275-327 (1987) (concluding, inter alia, that a computer would not be conscious in the sense humans are, regardless of the extent of its conceptual complexity); WILLIAM G. LYCAN, CONSCIOUSNESS 1-8 (1987) (discussing theories of consciousness, including dualism, behaviorism, functionalism, and the identity theory); COLIN MCGINN, THE PROBLEM OF CONSCIOUSNESS: ESSAYS TOWARDS A RESOLUTION 202-13 (1991) (posing question whether a machine could be conscious, and arguing that by duplicating the human brain--which has an unknown something which confers consciousnenss--one ought to be able to create an entity capable of experiencing the world around it).

[FN113]. See DANIEL C. DENNETT, BRAINSTORMS 149-50 (1978) (explaining why cognitive psychologists have given the subject relatively minimal attention).

[FN114]. Ludwig Wittgenstein makes a similar point in his discussion of pain. See LUDWIG WITTGENSTEIN, PHILOSOPHICAL INVESTIGATIONS ¶ ¶ 281-84, at 97e- 98e (G.E.M. Anscombe trans., 3d ed. 1958).

[FN115]. In one sense, the question is an easy one. If biological science progresses to the point one can build a human being from scratch, so to speak, then it is quite likely that an artifact could be conscious. See MCGINN, supra note 112, at 203-04.

[FN116]. This premise is subject to a qualification that has already been noted. Rights may be granted to X (which may be a person, with or without ends of her own, or even a thing) in order to protect the ends of Y. See supra note 28. Thus, one may give utilitarian or deontological justifications for granting rights to AIs that do not assume that AIs have their own ends. See supra notes 89-91 and accompanying text.

[FN117]. I am assuming here, contrary to Aristotle, that biological systems such as trees do not have ends, goals, or aims in the same sense that humans do. When we say that the oak is the telos of the acorn, we are using telos (end, goal, or aim) in a different sense than when we say that the physician's aim is to restore health. See ARISTOTLE, ON THE SOUL, reprinted in 1 THE COMPLETE WORKS OF ARISTOTLE, 661, at 415b1-20, 432b21 (Jonathan Barnes ed., Princeton University Press 1984); ARISTORLE, GENERATION OF ANIMALS, reprinted in 1 THE COMPLETE WORKS OF ARISTOTLE, supra, at 1203-04, at 778a16-b19.

[FN118]. See PHILLIP N. JOHNSON-LAIRD, MENTAL MODELS 448-77 (1983). But see MCGINN, supra note 112, at 209-13.

[FN119]. Evolution can produce biological mechanisms capable of performing similar functions, even though different underlying mechanisms are used. This is called "convergent evolution." See MORAVEC, supra note 57, at 39. For example, the octopus has developed a nervous system that evolved independently of the vertebrate version possessed by humans. See id. at 42. I suspect that some readers with strong intuitions that true machine intelligence is impossible may not possess these intuitions with respect to invertebrate intelligence. But if invertebrate animals might become intelligent without brains that share an evolutionary heritage with human brains, why not machines?

[FN120]. See DANIEL C. DENNETT, THE INTENTIONAL STANCE 327-28 (1987). Although transistors may be faster than neurons, the massively parallel structure of the brain or the possibility that processing may be accomplished within neurons may make the brain capable of performing a vastly greater number of operations per second than any transistor based system. Id. We simply do not know yet. Hans Moravec estimates that it will take roughly ten trillion calculations per second to match the calculations performed by the whole human brain. See MORAVEC, supra note 57, at 59. As of 1988, this was about one thousand times faster than the fastest supercomputers. Id. at 59-60. If current growth rates in processing speed and cost are extrapolated into the future, this amount of raw processing power would become economical for routine use in about the year 2030. See id. at 64, 68.

[FN121]. Of course, there would be a preliminary question as to how this would take place. It would be argued by the defendant that the AI is an exhibit and not a witness. I want to put this problem aside.

[FN122]. Cf. DENNETT, supra note 111, at 33-39 (discussing the distinction between mind and brain).

[FN123]. Cf. WITTGENSTEIN, supra note 114, ¶ 281, at 97e ("[O]nly of a living human being and what resembles (behaves like) a living human being can one say: it has sensations; it sees; is blind; bears; is deaf; is conscious or unconscious.").

[FN124]. This objection is different from the "consciousness" objection, assuming that intentionality is not essential to consciousness. Thus, we can imagine something that has "raw feelings," such as pains and pleasures, but lacks propositional attitudes and other intentional states.

[FN125]. Searle provides the following definition: "Intentionality is by definition that feature of certain mental states by which they are directed at or about objects and states of affairs in the world. Thus, beliefs, desires, and intentions are intentional states; undirected forms of anxiety and depression are not." Searle, Minds, Brains & Programs, supra note 19, at 72 n.3. On the difference between the ordinary concept of intentionality and the technical philosophical concept, see DENNETT, supra note 120, at 271.

[FN126]. Searle's definition of intentionality as that "feature of certain mental states by which they are directed at or about objects and states of affairs in the world," may not seem to be directed at meaning or understanding. See Searle, Minds, Brains & Programs, supra note 19, at 72 n.3. Searle is assuming a theory of meaning that connects the meaning of a statement to its reference, that is, to what it is about in the world. Given a referential theory of meaning, the connection between intentionality and meaningfulness is conceptual.

[FN127]. See M'Naghton's Case, 8 Eng. Rep. 718, 719 (1843).

[FN128]. Perhaps we do this only because infants and the insane are human. Our principle may be that all humans and only humans should have the rights of constitutional personhood. If so, then the example does not really bear on the intentionality objection, which would no longer carry any force of its own against constitutional personhood for AIs.

[FN129]. See ROBERT NOZICK, PHILOSOPHICAL EXPLANATIONS 574-75 (1981).

[FN130]. Would it make sense to say that an AI might struggle to live a meaningful "life" (as opposed to a meaningful "existence")? The problem, of course, is that our concept of life seems tied to particular biological forms.

[FN131]. See DENNETT, supra note 120, at 13-35.

[FN132]. Cf. id. at 29 & passim (discussing beliefs of thermostats).

[FN133]. Daniel Dennett calls such hypotheticals "intuition pumps." DANIEL C. DENNETT, ELBOW ROOM: THE VARIETIES OF FREE WILL WORTH WANTING 12 (1984). "Such thought experiments (unlike Galileo's or Einstein's, for instance) are not supposed to clothe strict arguments that prove conclusions from premises. Rather, their point is to entrain a family of imaginative reflections in the reader that ultimately yields not a formal conclusion but a dictate of 'intuition.' " Id.

[FN134]. This condition requires qualification. We treat our cats like intentional systems, but we do not think they have the rights of constitutional personhood. In order to rebut the presumption, AIs would have to exhibit intentional behaviors implying a level of intelligence that we associate with humans. Of course, the key would be use of language. If cats could talk, and if they demanded constitutional rights, they might get them.

[FN135]. See FLANAGAN, supra note 1, at 252-54. This objection is related to the objection from lack of consciousness and the objection from lack of intentionality, but should be categorized separately. It is not clear whether emotions are intentional states. It seems plausible that emotions require consciousness, but it is not evident that consciousness requires emotions. Id.; cf. MCGINN, supra note 112, at 202 (noting the existence of unconscious beliefs and desires).

[FN136]. My Trekkie (or, more properly, Trekker) friends indicate that my analysis of Star Trek is overly simplistic. For example, Ken Anderson contends that Spock possesses repressed emotions and that McCoy believes that Spock's personhood (and not just his humanity) is dependent on his having an emotional life. My bottom line is that McCoy would be wrong if he made this latter judgment. A Spock without emotions would still deserve to be treated as a person.

[FN137]. See ROGER J. SULLIVAN, IMMANUEL KANT'S MORAL THEORY 68 (1989). Kant often refers to rational beings other than humans. See IMMANUEL KANT, GROUNDWORK OF THE METAPHYSICS OF MORALS 57 (H.J. Paton trans., Harper & Row 1964) (1797). Kant defines person as follows: "A person is the subject whose actions are susceptible to imputation. Accordingly, moral personality is nothing but the freedom of a rational being under moral laws (whereas psychological personality is merely the capacity to be conscious of the identity of one's self in the various conditions of one's existence.)" KANT, supra note 59, at 24; see LESLIE A. MULHOLLAND, KANT'S SYSTEM OF RIGHTS 168 (1990).

[FN138]. See Nancy Sherman, The Place of the Emotions in Kantian Morality, in IDENTITY, CHARACTER, AND MORALITY 149, 154-62 (Owen Flanagan & Amelie O. Rorty eds., 1990).

[FN139]. This statement can be challenged. For example, Colin McGinn argues that certain behaviors are linked to our very concept of emotion:

Think here of facial expressions: these are so integral to our notion of an emotion that we just do not know what to make of the suggestion that an IBM 100 might be angry or depressed or undergoing an adolescent crisis. The problem is not that the IBM is inanimate, not made of flesh and blood; the problem is that it is not embodied in such a way that it can express itself (and merely putting it inside a lifelike body will not provide for the right sort of expressive link up).

MCGINN, supra note 112, at 207. But of course if our AI did have the right sort of behaviors linked up to the right sort of internal processes, this objection would no longer hold. See id. Moreover, I am not quite sure that McGinn is right about facial expressions. Radio plays and books seem to be able to convey human emotions without visual representations of facial expressions, and the blind perceive emotions without the ability to see facial expressions (although touching faces might come into play in this case). The range of human emotions that can be conveyed through verbal means should not be underestimated.

[FN140]. See FLANAGAN, supra note 1, at 253.

[FN141]. See Aaron Sloman, Motives, Mechanisms, and Emotions, in THE PHILOSOPHY OF ARTIFICIAL INTELLIGENCE, supra note 16, at 231, 231-32.

[FN142]. See FLANAGAN, supra note 1, at 253.

[FN143]. Georges Rey, Functionalism and the Emotions, in EXPLAINING EMOTIONS 163, 192 (Amelie Oksenberg Rorty ed., 1980).

[FN144]. Of course, this assumes that we have gotten past the consciousness objections.

[FN145]. "Privileged first-person access" is another way of saying that you cannot get inside your neighbor's head to find out what she is really feeling.

[FN146]. See JOHN FINNIS, NATURAL LAW AND NATURAL RIGHTS 85-90 (1980).

[FN147]. See Solum, supra note 109, at 1087-89. John Rawls has explored this state of affairs, which he calls "the fact of pluralism." Rawls, supra note 108, at 4.

[FN148]. See FLANAGAN, supra note 1, at 255; see also Arie A. Covrigaru & Robert K. Lindsay, Deterministic Autonomous Systems, AI MAG., Fall 1991, at 110, 111-13 (arguing that "an entity is autonomous if it is perceived to have goals, including certain kinds of goals, and is able to select among a variety of goals that it is attempting to achieve").

[FN149]. There is a large body of philosophical literature on the concept of autonomy. See GERALD DWORKIN, THE THEORY AND PRACTICE OF AUTONOMY 3-62 (1988); Frankfurt, supra note 100. Of course, the classic discussion is Kant's. See KANT, supra note 59 at 98-100; SULLIVAN, supra note 137, at 46-47.

[FN150]. See Robert N. Shapiro, Of Robots, Persons, and the Protection of Religious Beliefs, 56 S. CAL. L. REV. 1277, 1286-90 (1983).

[FN151]. See DENNETT, supra note 133, at 20-21.

[FN152]. A more developed conception of autonomy for AIs can be found in Covrigaru & Lindsay, supra note 148, at 112-17. They summarize the criteriaas follows:

A goal directed system will be perceived to be autonomous to the degree that (1) it selects tasks (top level goals) it is to address at any given time; (2) it exists over a period of time that is long relative to the time required to achieve a goal; (3) it is robust, being able to remain viable in a varying environment; (4) some of its goals are homeostatic; (5) there are always goals that are active (instantiated but not achieved); (6) it interacts with its environment in an information-processing mode; (7) it its a variety of complex responses, including fluid, adaptive movements; (8) its attention to stimuli is selective; (9) none of its functions, actions, or decisions need to be fully controlled by an external agent; and, (10) once the system starts functioning, it does not need any further programming.

Id. at 117. Some of the criteria offered by Covrigaru and Lindsay do not really seem to be criteria for autonomy. For example, some humans may lack fluid motion (criterion seven on the list) because of a physical condition, but we do not believe that this destroys their autonomy.

[FN153]. For example, we would not hold such persons criminally or civilly liable for those actions produced by the controller--unless perhaps they voluntarily submitted themselves to the implantation procedure and they either foresaw or should have foreseen that the consequence of such submission would be the action to which liability attaches.

[FN154]. Although, one day someone may get on a computer-simulated virtual surfboard and ride a virtual wave.

[FN155]. See William G. Lycan, Introduction to MIND AND COGNITION 3, 3-13 (William G. Lycan ed., 1990).

[FN156]. See Roger Penrose, Matter Over Mind, N.Y. REV. BOOKS, Feb. 1, 1990, at 3-4; Paul Weiss, On the Impossibility of Artificial Intelligence, 44 REV. METAPHYSICS 335, 340 (1990).

[FN157]. Put in possible-worlds talk, the argument establishes that there is a possible world in which an AI behaves as if it is conscious but is not really conscious. The argument does not establish that there is no possible world in which an AI is really conscious.

[FN158]. The phrase "natural slave" is borrowed from Aristotle, but my use of it is ironic, since AIs are artifacts and hence not natural in the same sense as the human beings enslaved in ancient Greece. See ARISTOTLE, POLITICS, reprinted in 2 THE COMPLETE WORKS OF ARISTOTLE, supra note 117, at 1986-87, at 1252a32 ("[T]hat which can foresee by the exercise of mind is by nature lord and master, and that which can by its body give effect to such foresight is a subject, and by nature a slave . . . ."). Curiously, Aristotle says that tools are "inanimate slaves." ARISTOTLE, EUDEMIAN ETHICS, reprinted in 2 THE COMPLETE WORKS OF ARISTOTLE, supra note 117, at 1968, at 1241b23. The phrase "inanimate slaves" would be more apt, of course, for an AI than for a hammer.

[FN159]. See JOHN LOCKE, TWO TREATISES OF GOVERNMENT § § 25-51, at 285-302 (Peter Laslett ed., 1988) (1690).

[FN160]. Id. § 27, at 287. But see 1 Corinthians 6:19-20 (St. Paul, stating "You are not your own property; you have been bought and paid for."); LEVIATHAN, supra note 12, at 110 (arguing that in the state of nature, "every man has a Right to every thing; even to one another's body"). See generally STEPHEN R. MUNZER, A THEORY OF PROPERTY 41-44 (1990) (discussing property rights of persons in their bodies).

[FN161]. LOCKE, supra note 159, § 27, at 287-88.

[FN162]. Id. § 27, at 288. This conclusion does not follow automatically, as Locke may have believed. "[W]hy isn't mixing what I own with what I do not own a way of losing what I own rather than a way of gaining what I don't?" ROBERT NOZICK, ANARCHY, STATE, AND UTOPIA 174-75 (1974); see also JEREMY WALDRON, THE RIGHT TO PRIVATE PROPERTY 184-91 (1988) (discussing the results of mixing one's labor). Stephen Munzer advances an argument that might substitute for this premise but is based on an appeal to desert rather than mixing. See MUNZER, supra note 160, at 254-91; see also Stephen Munzer, The Acquisition of Property Rights, 66 NOTRE DAME L. REV. 661, 674-86 (1991) (discussing interpretations of Locke's theory of property acquisition).

[FN163]. Of course, the actual situation might be very complicated. Real AIs may be the product of the labor of many, many persons--some or all of whom may have contracted away their property rights in the software of the AI in exchange for a salary. In addition, in order to operate, an AI requires hardware, which may be the property of others. Furthermore, later-generation AIs may be the product of the creative work of earlier-generation AIs. I will assume that these complications do not affect the outcome of the argument.

[FN164]. See ROBERT FILMER, Patriarcha, in PATRIARCHA AND OTHER WRITINGS 1, 6-7 (Johann P. Sommerville ed., 1991).

[FN165]. LOCKE, supra note 159, § 52, at 178.

[FN166]. Id. § 53, at 179.

[FN167]. Although this view was attributed by Locke to Filmer, it may not be Filmer's own.

[FN168]. Of course, some theists may believe that the personhood of humans comes from their soul, and that souls are made by God. But many theists do not accept the conclusion that it is this feature of human personality that defeats the Filmerian argument. One might take the position that even if souls were made by humans and not God, parents would not own their children.

[FN169]. In addition to the Lockean argument explored in the text, there is a utilitarian argument that could be advanced in favor of property status for AIs. The premise of the argument is that unless AIs are property, there will be no incentive to create them. AI research is expensive, and without incentives the market will not produce AIs. This case is unlike the case of natural persons, because humans are constructed so as to have strong natural incentives to reproduce. It should be noted, however, that in the case of slavery for natural persons, most of us do not accept that if slavery maximized the utility of slaves, then slavery would be morally correct. (Mad-dog utilitarians are an exception.) Of course, once AIs gained the ability to reproduce themselves the need for the incentive might disappear, and the utility to AIs of their own freedom might then outweigh any benefits of additional incentives for humans to produce AIs.

If the premise of the utilitarian argument is correct, it raises further questions. Suppose that the only way that AIs will be brought into being is if the legal system guarantees that they will be the property of their creator. Given that fact, what would be our obligations toward AIs? One might argue that we have an obligation to them not to bring them into the world as slaves.

[FN170]. WILLIAM GIBSON, NEUROMANCER 131-32 (1984). Copyright 1984 by William Gibson. Reprinted by permission of the Berkley Publishing Group.

[FN171]. Turing, supra note 17, at 42.

The game may perhaps be criticized on the ground that the odds are weighted too heavily against the machine . . . . This objection is a very strong one, but at least we can say that if, nevertheless, a machine can be constructed to play the imitation game satisfactorily, we need not be troubled by this objection.

Id.

[FN172]. Robert M. French, Subcognition and the Limits of the Turing Test, 99 MIND 53, 53-54 (1990).

[FN173]. See id. at 58.

[FN174]. By symptoms, I mean the surface behaviors--those that could be observed without examining the underlying mechanisms.

[FN175]. See GREENAWALT, supra note 109, at 120-43 (discussing the valuation of the life of a fetus).

[FN176]. See Elyn R. Saks, Multiple Personality Disorder and Criminal Responsibility, 25 U.C. DAVIS L. REV. 383 (1992).

[FN177]. In this sense, I do not think that considering the philosophical debate about the possibility of AI yields any clear answers for current debates about personhood. But see Steven Goldberg, The Changing Face of Death: Computers, Consciousness, and Nancy Cruzan, 43 STAN. L. REV. 659, 680 (1991). Goldberg has argued that artificial intelligence research may shape the outcome of at least one legal question about the borderline of status--the definition of death. He begins with the premise that humans have a strong preference for seeing the human species as unique. He then argues that if a social consensus were reached that computers are self-aware, we would then seek another characteristic of humans to distinguish ourselves as unique in the universe. He suggests that this characteristic may be the capacity for social interaction. Id. at 680. This leads Goldberg to the conclusion that self- aware computers would make it more likely that courts would adopt capacity for social interaction as a definition for death. Id. at 681-82.

Goldberg's essay is provocative, but his argument is tenuous. First, Goldberg states but does not argue for the assumption that "any concept of human death depends directly on those qualities thought to make humans unique." Id. at 663 (citing ROBERT M. VEATCH, DEATH, DYING AND THE BIOLOGICAL REVOLUTION: OUR LAST QUEST FOR RESPONSIBILITY 29-42 (1976)). Second, Goldberg asserts but does not provide evidence for the proposition that a psychological need for humans to see themselves as unique caused the shift from heart-function to brain- function definitions of death. Id. at 660-70. Third, Goldberg does not consider other possible chains of causation. Consider two other possibilities. One possibility is that there may have been practical concerns for the cost of sustaining "life" without possibility of recovery. A second possibility is that consciousness may be a condition of personhood, as personhood is understood by the best available moral theory. The development of life-sustaining technology that permits the maintenance of heart function without consciousness for extended periods may have forced the issue of how to define death, which would have only been theoretical before the development of the new technology. Fourth, AIs that are capable of producing consciousness also may be capable of social interaction. Therefore, the same developments in artificial intelligence research that would prompt a move away from a consciousness-based definition of death would also prompt a move away from a social-interaction definition. Fifth, Goldberg's argument seems to imply that the move to a social-interaction definition would be based on a conceptual mistake or some form of wishful thinking. But if Goldberg can see this, why will courts be unable to do so?

[FN178]. Fetuses might have some behaviors that are associated with feeling, but they clearly do not engage in behavior that establishes the concurrent presence of consciousness, intentionality, emotion, and free will.

[FN179]. Compare John Rawls, The Independence of Moral Theory, 48 PROC. & ADDRESSES AM. PHIL. ASS'N 5 (1975) (arguing that moral theory should be independent of metaphysics) with Robert Stern, The Relationship Between Moral Theory and Metaphysics, PROC. ARISTOTLIAN SOC'Y 143 (1992) (arguing that moral theory is dependent upon metaphysics).

END OF DOCUMENT