Topics in Machine Learning: Robot Learning
Web Projects by:

Brad Rosenberg
bradr at cs dot brandeis dot edu
11/12/2002

The following work is a companion piece to Karl Sims' paper, "Evolving 3D Morphology and Behavior by Competition".

Evolving Competitive Virtual Creatures

With the intent to explore interesting spaces, Karl Sims developed a system in which he evolved competiting virtual creatures. In the past, the focus of evolutionary tactics were used in hopes of simple optimization. Sims' interest lied more in developing a diverse population that exhibited "interesting" behavior.

The Competition
The actual competition consisted of two creatures, facing off in a physically simulated three-dimensional environment. The creatures were placed facing each other, equidistant from the center goal object, a cube. The object of the competition is for a creature to make its way closest to the cube and control as much of the cube as possible. This was done by at start time, activating the nervous sytem of the creature until a set time which determined the match had ended. The following equation was used to determine the fitness of a creature:

Fitness1 = 1.0 + ((d2 - d1) / (d1 + d2))
Fitness2 = 1.0 + ((d1 - d2) / (d1 + d2))
where di denotes creature i's distance from the cube.

Pictures of a competition starting [2]:

Determining Competition Patterns
Sims hypothesized upon how different competition patterns would affect the evolution of creatures. In light of the fact that evolution of these creatures was performed on a Connection Machine (CM-5), a massive parallel processing supercomputer, processing time was still major contributing factor to how to setup each creature to compete. Sims describes four single-species patterns: all vs all, random pairing, tournament, and all vs best. All vs all takes every individual and plays them again all others in the population. However, this requires a large number of total competitions. Random pairing takes each individual and pairs them with a single opponent. While this pattern requires a small number of competitions, fitness is quite subjective since it is based on a single opponent. Sims describes this situation as, "... fitness can be more dependent on the luck of receiving a poor opponent than on an individuals actual ability." A tournament pattern allows each winning individual to compete against other winning individuals. Sims final suggestion, which he establishes as the more interesting pattern in his particular experiment, is for each individual to compete against the previously most fit individual in a "king of the hill" competition.

Creature Morphology
Sims' creatures were created using directed-graphs for a variable-length genotype. The creature develops by starting at the root node of the graph and following connections from this node to other nodes using the information contained therein. This allowed for cyclic and recursive parts. Sims goes on to describe how the information within the node allowed for various types of joints and parameters, along with child parts and their relative properties to their parents.

Creature Behavior
Not only was the physical structure of the creature evolved, but the underlying behavioral mechanism (the nervous system) was also evolved in conjunction. Sims brains of the creatures used input sensor values and produced output effector values. In between these inputs and outputs, neurons existed that could modify and provide calculations before being sent to various effectors. Sims used three sensors: joint-angle sensors which provided the degree of freedom of each joint, contact sensors that fired when a portion of the body of the creature was in contact with anything else, and photosensors which allowed the creature to "see" the cube and his opponent. Attached to these senors were neurons, which allowed computation to be performed on incoming senors. Finally, output was placed to effectors, which simply exerted a joint force.

Creature Evolution and Results
Once a random population was generated, creatures are paired off, competitions are held, and the fitness of each creature is evaluated. From here on out, parents are selected based on fitness, children are bred through crossover, asexual reproduction, and mutation based on their parents genotypes. These possible children are then tested for survivability and survivors are kept for the following population.

Sims noticed in his results that different runs showed different evolutionary progress. Sometimes it took many generations for a creature to reach the cube at all, other times a successful strategy was discovered quickly. Aside from rate of evolutionary progress, interesting behavior developed amongst the creatures. Strategies for reaching the cube consisted of extending arms, falling on top of the cube, even crawling and rolling developed. When both competition creatures had evolved to reach the cube, strategies and counter-strategies developed as in swatting the cube away and chasing after it, smothering the cube to block access, and even offensive moves against opponents before snatching the cube. Below are some snapshots of competitions in play. The first involves an arm-like creature wrapping itself around the cube when facing a crab-like creature. The second shows a creature with arms trying to bat the cube to one arm.

Video Interview with Karl Sims - (-1:30min - Creatures from the Cube game)

Sims - The Next Step
After this particular project, Sims went on to expand upon this work a develop creatures that evolved locomotion in terms of walking, swimming, jumping and following. His paper, "Evolving Virtual Creatures" [3] describes the methods in which he went about this experiment. A 3-minute movie showing creatures from these experiments can be seen here [4]. Beyond this, Sims has moved in a direction more towards interactive art and is currently leading GenArts Inc.

Robot Learning

Robot Learning, like AI, is a vast topic with no set definition. However, the field of robotics is a popular one, with work being done in both the private and public sector. Robots are used to perform such operations that are either tedious and repetitive for humans, too dangerous for humans, or that are beyond the capability of humans. By sensing their environment and affecting it, robots make direct changes to the real world. By making these machines intelligent and able to learn, these machines can increase their usefulness in reality.

Rodney Brooks' group of individuals working to enhance the abilities of their robots. Featured here is the famous Cog, a humanoid robot which tries to mimic its visitors faces, with two eyes, two hands, and dozens of limbs and joints [video]. One of the major goals of this project is the collaborations of dozens of sensors and effectors. One of the other most popular robots at MIT's AI Lab is Kizmet, a disembodied head who's goal is to socialize with its visitors. Kizmet receives information through visual and auditory sensors, and responds through gaze, facial expression, posture, and vocal expressions. [video]. The newest robot to the AI Lab camp is Coco, a mobile robot.

One of the more fascinating groups working on robot learning. Research in this group focuses on developing robots that can adapt to changes in the environment and within their own capabilities. One experiment performs obstacle avoidance and exhibits reactive behavior using evolutionary algorithms [video]. Shown here, this robot makes his way through the obstacles visiting a series of yellow spots painted on the floor. In another experiment, a "shepard" robot is evolved to herd a random/obstacleavoiding "sheep" into a pasture. This, too, was involved using evolutionary algorithms [video]. Research is also being performed in continous and embedded learning. In one experiment, a robot is asked to navigate through a doorway with changing internal conditions. In a beginning situation, all sensors were enabled. In sequential setups, its sonar sensor is disabled or debilitated, and through virtual simulations, the robot learns how to navigate without it [video].

Other Interesting Links
The Multi-Robot Lab at Carnegie Mellon University
The Mobile Robotics Research Group at University of Edinburgh
The University of Southern California Robotics Research Lab

References

1) "Evolving 3D Morphology and Behavior by Competition" [paper]

2) http://www.isd.atr.co.jp/~ray/pubs/fatm/node10.html - Color snapshots of Sims' evolved creatures.

3) "Evolving Virtual Creatures" [paper]

4) Evolving Virtual Creatures - The Movie (http://wwwradig.in.tum.de/personen/mauss/creatures/)

5) The Humanoid Robotics Group at MIT

6) Adaptive Systems Group at the Navy Center for Applied Research in Artificial Intelligence