Letter From Moravec to Penrose

Date: 9 Feb 90 02::02:22 GMT
From: Hans.Moravec@rover.ri.cmu.edu
Subject: Dear Roger,
Newsgroups: sci.nanotech

This is an open letter, distribute at will.
Comments are solicited. Thanks. -- Hans Moravec

To: Professor Roger Penrose, Department of Mathematics, Oxford, England

Dear Professor Penrose,

Thank you for sharing your thoughts on thinking machinery in your new
book The Emperor's New Mind, and in the February 1 New York Review of
Books essay on my book Mind Children.  I've been a fan of your
mathematical inventions since my high school days in the 1960s, and was
intrigued to hear that you had written an aggressively titled book about
my favorite subject.  I enjoyed every part of that book -- the
computability chapters were an excellent review, the phase space view of
entropy was enlightening, the Hilbert space discussion spurred me on to
another increment in my incredibly protracted amateur working through of
Dirac, and I'm sure we both learned form the chapter on brain anatomy. 
You won't be surprised to learn, however, that I found your overall
argument wildly wrong headed!

If your book was written to counter a browbeating you felt from
proponents of hard AI, mine was inspired by the browbeaten timidity I
found in the majority of my colleagues in that community.  As the words
"frightening" and "nightmare" in your review suggest, intelligent
machines are an emotion-stirring prospect, and it is hard to remain
unbrowbeaten in the face of frequent hostility.  But why hostility? Our
emotions were forged over eons of evolution, and are triggered by
situations, like threats to life or territory, that resemble those that
influenced our ancestors' reproductive success.  Since there were no
intelligent machines in our past, they must resemble something else to
incite such a panic -- perhaps another tribe down the stream poaching in
our territory, or a stronger, smarter rival for our social position, or
a predator that will carry away our offspring in the night.  But is it
reasonable to allow our actions and opportunities to be limited by
spurious resemblances and unexamined fears? Here's how I look at the
question.  We are in the process of creating a new kind of life.  Though
utterly novel, this new life form resembles us more than it resembles
anything else in the world.  To earn their keep in society, robots are
being taught our skills.  In the future, as they work among us on an
increasingly equal footing, they will acquire our values and goals as
well -- robot software that causes antisocial behavior, for instance,
would soon cease being manufactured.  How should we feel about beings
that we bring into the world, that are similar to ourselves, that we
teach our way of life, that will probably inherit the world when we are
gone? I consider them our children.  As such they are not fundamentally
threatening, though they will require careful upbringing to instill in
them a good character.  Of course, in time, they will outgrow us, create
their own goals, make their own mistakes, and go their own way, with us
perhaps a fond memory.  But that is the way of children.  In America, at
least, we consider it desirable for offspring to live up to their
maximum potential and to exceed their parents. 

You fault my book for failing to present alternatives to the "hard AI"
position.  It is my honest opinion that there are no convincing
scientific alternatives.  There are religious alternatives, based on
subjective premises about a special relation of man to the universe, and
there are flawed secular rationalizations of anthropocentrism.  The two
alternatives you offer, namely John Searle's philosophical argument and
your own physical speculation, are of the latter kind.  Searle's
position is that a system that, however accurately, simulates the
processes in a human brain, whether with marks on paper of signals in a
computer, is a "mere imitation" of thought, not thought itself. 
Pejorative labels may be an important tool for philosophy professors,
but they don't create reality.  I imagine a future debate in which
Professor Searle, staunch to the end, succumbs to the "mere imitation"
of thought and emotion.  Your own position is that some physical
principle in human brains produces "non- computable" results, and that
somehow this leads to consciousness.  Well, I agree, but the same
principle works equally well for robots, and it's not nearly as
mysterious as you suggest. 

Alan Turing's computability arguments, now more than fifty years old,
were a perfect fit to David Hilbert's criteria for the mechanization of
deductive mathematics, but they don't define the capabilities of a robot
or a human.  They assume a closed process working from a fixed, finite,
amount of initial information.  Each step of a Turing machine
computation can at best preserve this information, and may destroy a bit
of it, allowing the computation to eventually "run down," like a closed
physical system whose entropy increases.  The simple expedient of
opening the computation to external information voids this suffocating
premise, and with it the uncomputability theorems.  For instance,
Turing, proved the uncomputability of most numbers, since there are only
countably many machine programs, and uncountably many real numbers for
them to generate.  But it is trivial to produce "uncomputable" numbers
with a Turing machine, if the machine is augmented with a true
randomizing device.  Whenever another digit of the number is needed, the
randomizer is consulted, and the result written on the appropriate
square of the tape.  The emerging number is drawn uniformly from a real
interval, and thus (with probability 1) is an "uncomputable" number. 
The randomizing device allows the machine to make an unlimited number of
unpredetermined choices, and is an unbounded information source.  In a
Newtonian universe, where every particle has an infinitely precise
position and momentum, fresh digits could be extracted from finer and
finer discriminations of the initial conditions by the amplifying
effects of chaos, as in a ping pong ball lottery machine.  A quantum
mechanical randomizer might operate by repeatedly confining a particle
to a tiny space, so fixing its position and undefining its momentum,
then releasing it and registering whether it travels left or right. 
Just where the information flows from in this case is one of the
mysteries of quantum mechanics. 

The above constitutes a basic existence proof for "uncomputable" results
in real machines.  A more interesting example is the augmentation of a
"Hilbert" machine that systematically generates inferences from an
initial set of axioms.  As your book recounts, a deterministic device of
this kind will never arrive at some true consequences of the axioms. 
But suppose the machine, using a randomizer, from time to time concocts
an entirely new statement, and adds it to the list of inferences.  If
the new "axiom" (or hypothesis) is inconsistent with the original set,
then sooner or later the machine will generate an inference of "FALSE"
from it.  If that happens the machine backtracks and deletes the
inconsistent hypothesis and all of its inferences, then invents a new
hypothesis in its place.  Eventually some of the surviving hypotheses
will be unprovable theorems of the original axiom system, and the
overall system will be an idiosyncratic, "creative" extension of the
original one.  Consistency is never assured, since a contradiction could
turn up at any time, but the older hypotheses are less and less likely
to be rescinded.  Mathematics made of humans has the same property. 
Even when an axiomatic system is proved consistent, the augmented system
in which the proof takes place could itself be inconsistent,
invalidating the proof!

When humans (and future robots) do mathematics they are less likely to
draw inspiration from rolls of dice than by observing the world around
them.  The real world too is a source of fresh information, but
pre-filtered by physics and evolution, saving us some work.  When our
senses detect a regularity (let's say, spherical soap bubbles) we can
form a hypothesis (e.g., that spheres enclose volume with the least
area) likely to be consistent with hypotheses we already hold, since
they too were abstracted from the real world, and the real world is
probably consistent.  This brings me to your belief in a Platonic
mathematical reality, which I also think you make unnecessarily
mysterious.  The study of formal systems shows there is nothing
fundamentally unique about the particular axioms and rules of inference
we use in our thinking.  Other systems of strings and rewriting rules
look just as interesting on paper.  They may not correspond to any
familiar kind of language or thought, but it is easy to construct
machines (and presumably animals) to act on their strange dictates.  In
the course of evolution (which, significantly, is driven by random
mutations) minds with unusual axioms or inference structures must have
arisen from time to time.  But they did poorly in the contest for
survival and left no descendants.  In this way we were shaped by an
evolutionary game of twenty questions -- the intuitions we harbor are
those that work in this place.  The Platonic reality you sense is the
groundrules of the physical universe in which you evolved -- not just
its physics and geometry but its logic.  If there are other universes
with different rules, other Roger Penroses may be sensing quite
different Platonic realities. 

And now to that other piece of mysticism, human consciousness.  Three
centuries ago Rene Descartes was a radical.  Having observed the likes
of clockwork ducks and the imaging properties of bovine eyes, he
rejected the vitalism of his day and suggested that the body was just a
complex machine.  But lacking a mechanical model for thought, he
exorcised the spirit of life only as far as a Platonic realm of mind
somewhere beyond the pineal gland -- a half-measure that gave us
centuries of fruitless haggling on the "mind-body" problem.  Today we do
have mechanical models for thought, but the Cartesian tradition still
lends respectability to a fantastic alternative that comforts
anthropocentrists, but explains nothing.  Your own proposal merely
substitutes "mysterious unexplained physics" for spirit.  The center of
Descartes' ethereal domain was consciousness, the awareness of thought
-- "I think therefore I am."

You say you have no definition for consciousness, but think you know it
when you see it, and you think you see it in your housepets.  So, a dog
looks into your eyes with its big brown ones, tilts its head, lifts an
ear and whines softly, and you feel that there is someone there.  I
suppose, from your published views, that those same actions from a
future robot would meet with a less charitable interpretation.  But
suppose the robot also addressed you in a pained voice, saying "Please,
Roger, it bothers me that you don't think of me as a real person.  What
can I do to convince you? I am aware of you, and I am aware of myself. 
And I tell you, your rejection is almost unbearable." This performance
is not a recording, nor is it due to mysterious physics.  It is a
consequence of a particular organization of the robot's controlling
computers and software.  The great bulk of the robot's mentality is
straightforward and "unconscious." There are processes that reduce
sensor data to abstract descriptions for problem solving modules, and
other processes that translate the recommendations of the problem
solvers into robot actions.  But sitting on top of, and sometimes
interfering with, all this activity is a relatively small reflective
process that receives a digest of sensor data organized as a
continuously updated map, or cartoon- like image, of the robot's
surroundings.  The map includes a representation of the robot itself,
with a summary of the robot's internal state, including reports of
activity and success or trouble, and even a simplified representation of
the reflective process.  The process maintains a recent history of this
map, like frames of a movie film, and a problem solver programmed to
monitor activity in it.  One of the reflective process' most important
functions is to protect against endless repetitions.  The unconscious
process for unscrewing a jar lid, for instance, will rotate a lid until
it comes free.  But if the screw thread is damaged, the attempt could go
on indefinitely.  The reflective process monitors recent activity for
such dangerous deadlocks and interrupts them.  As a special case of
this, it detects protracted inaction.  After a period of quiescence the
process begins to examine its map and internal state, particularly the
trouble reports, and invokes problem solvers to suggest actions that
might improve the situation. 

The Penrose house robot has a module that observes and reasons about the
mental state of its master (advertising slogan: "Our Robots Care!"). 
For reasons best known to its manufacturer, this particular model
registers trouble whenever the psychology module infers that the master
does not believe the robot is conscious.  One slow day the reflective
process stirs, and notes a major trouble report of this kind.  It runs
the human interaction problem solver to find an ameliorating strategy. 
This produces a plan to initiate a pleading conversation with Roger,
with nonverbal cues.  So the robot trundles up, stares with its big
brown eyes, cocks its head, and begins to speak.  To protect its
reputation, the manufacturer has arranged it so the robot cannot
knowingly tell a lie.  Every statement destined for the speech generator
is first interpreted and tested by the reflective module.  If the robot
wishes to say "The window is open," the reflective process checks its
map to see if the window is indeed labeled "open." If the information is
missing, the process invokes a problem solver, which may produce a
sensor strategy that will appropriately update the map.  Only if the
statement is so verified does the reflective process allow it to be
spoken.  Otherwise the generating module is itself flagged as
troublesome, in a complication that doesn't concern this argument.  The
solver has generated "Please, Roger, it bothers me that you don't think
of me as a real person." The reflective process parses this, and notes,
in the map's schematic model of the robot's internals, that the trouble
report from the psychology module was generated because of the master's
(inferred) disbelief.  So the statement is true, and thus spoken.  "What
can I do to convince you?"-like invoking problem solvers, asking
questions sometimes produces solutions, so no lie here.  "I am aware of
you, and I am aware of myself" -- the reflective process refers to its
map, and indeed finds a representation of Roger there, and of the robot
itself, derived from sensor data, so this statement is true.  "And I
tell you, your rejection is almost unbearable" -- trouble reports carry
intensity numbers, and because of the manufacturer's peculiar
priorities, the "unconscious robot" condition generates ever bigger
intensities.  Trouble of too high an intensity triggers a safety circuit
that shuts down the robot.  The reflective process tests the trouble
against the safety limit, and indeed finds that it is close, so this
statement also is true.  [In case you feel this scenario is far fetched,
I am enclosing a recent paper by Steven Vere and Timothy Bickmore of the
Lockheed AI center in Palo Alto that describes a working program with
its basic elements.  They avoid the difficult parts of the robot by
working in a simulated world, but their program has a reflective module,
and acts and speaks with consciousness of its actions.]

Human (and even canine) consciousness undeniably has subtleties not
found in the above story.  So will future robots.  But some animals
(including most of our ancestors) get by with less.  A famous example is
the Sphex wasp, which paralyzes caterpillars and deposits them in an
underground hatching burrow.  Normally she digs a burrow, seals the
entrance, and leaves to find a caterpillar.  Returning, she drops the
victim, reopens the entrance, then turns to drag in the prey.  But if an
experimenter interrupts by moving the caterpillar a short distance away
while the wasp is busy at the opening, she repeats the motions of
opening the (already open) burrow, after shifting the prey back.  If the
experimenter again intervenes, she repeats again, and again and again,
until either the wasp or the experimenter drops from exhaustion. 
Apparently Sphex has no reflective module to detect the cycle.  It's not
a problem in her simple, stereotyped life, malicious experimenters being
rare.  But in more complex niches, opportunities for potentially fatal
loops must be more frequent and unpredictable.  The evolution of
consciousness may have started with a "watchdog" circuit guarding
against this hazard. 

I like thinking about the universe's exotic possibilities, for instance
about computers that use quantum superposition to do parallel
computations.  But even with the additional element of time travel (!),
I've never encountered a scheme that gives more than an exponential
speedup, which would have tremendous practical consequences, but little
effect on computability theorems.  Or perhaps the universe is like the
random axiomatic system extender described above.  When a measurement is
made and a wave function collapses, an alternative has been chosen. 
Perhaps this constitutes an axiomatic extension of the universe --
today's rules were made by past measurements, while today's
measurements, consistent with the old rules, add to them, producing a
richer set for the future. 

But robot construction does not demand deep thought about such
interesting questions, because the requisite answers already exist in
us.  Rather than being something entirely new, intelligent robots will
be ourselves in new clothing.  It took a billion years to invent the
concept of a body, of seeing, moving and thinking.  Perhaps fundamentals
like space and time took even longer to form.  But while it may be hard
to construct the arrow of perceived time from first principles, it is
easy to build a thermostat that responds to past temperatures, and
affects those of the future.  Somehow, without great thought on our
part, the secret of time is passed on to the device.  Robots began to
see, move and think almost from the moment of their creation.  They
inherited that from us. 

In the nineteenth century the most powerful arithmetic engines were in
the brains of human calculating prodigies, typically able to multiply
two 10 digit numbers in under a minute.  Calculating machinery surpassed
them by 1930.  Chess is a richer arena, involving patterns and strategy
more in tune with our animal skills.  In 1970 the best chess computer
played at an amateur level, corresponding to a US chess federation
rating of about 1500.  By 1980 there was a machine playing at a 1900
rating, Expert level.  In 1985, a machine (HiTech) at my own university
had achieved a Master level of 2300.  Last year a different machine from
here (Deep Thought) achieved Grandmaster status with a rating of 2500. 
In past each doubling of chess computer speed raised the quality of its
play by about 100 rating points.  The Deep Thought team has been adopted
by IBM and is constructing a machine on the same principles, but 1000
times as fast.  Though Kasparov doubted it on the occasion of defeating
Deep Thought in two games last year, his days of absolute superiority
are numbered.  I estimated in my book that the most developed parts of
human mentality -- perception, motor control and the common sense
reasoning processes -- will be matched by machines in no less than 40
years.  But many of the skills employed by mathematics professors are
more like chess than like common sense.  Already I find half of my
mathematics not in my head but in the steadily improving Macsyma and
Mathematica symbolic mathematics programs that I've used almost daily
for 15 years.  Sophomoric arguments about the indefinite superiority of
man over machine are unlikely to change this trend. 

Well, thank you for a stimulating book.  As I said in the introduction,
I enjoyed every part of it, and its totality compelled me to put into
these words ideas that might otherwise have been lost. 

                            Very Best Wishes,
                            Hans Moravec
                            Robotics Institute,
                            Carnegie Mellon University,
                            Pittsburgh, PA  15213 USA
                            Arpanet: hpm@rover.ri.cmu.edu
                            Fax: (412) 682-1793
                            Telephone: (412) 268-3829