MMVR 2002 Presentation
Using Virtual Agents, Avatars and Worlds
for Treatment and Healing
(click here for slides)

Galen Brandt

I’m here to talk about the work of Dr. Dorothy Strickland, who could not be here today, in using virtual reality to treat autism, and about how virtual agents, avatars and worlds may be useful in the treatment of patients and in the healing process. 

In 1994, Dr. Strickland, a computer scientist, engineer and visual artist, was searching for a patient population that could take advantage of the relatively simple, cartoonish graphics generated by early VR systems.  Her research led her to Shannon, a seven-year-old girl with autism.  Disconnected from her surroundings and easily upset by change, Shannon could not cross a street by herself, or recognize and use a fork.

Working with a team from the University of North Carolina and NC State, Strickland designed a purposely simple, unchanging, infinitely repeatable virtual street scene, and immersed Shannon in it via headset.  After just a few five-minute sessions, Shannon had tracked a moving virtual car, walked down the sidewalk, and stopped at the stop sign to let the “bu ca” pass. 

Success – but would it generalize to the real world?  In a follow-up study, Strickland trained Shannon and several other children with autism to recognize and use everyday objects in a virtual kitchen – a spatula, a cup, a fork.  After several sessions, the children were able to use the same objects in the real world, often for the first time. 

So we see that virtual environments can be of real use in building spatial navigation, object recognition and learning skills in those with special needs. 

But how to bring these environments and associated treatments to children with special needs all over the world?  Strickland created and launched an NIH-funded website – dotolearn.com. – dedicated, in her words, “particularly to those individuals who may simply see the world in ways we do not yet understand.”  The site provides free online information and solutions to parents, teachers and health professionals, and learning tools for those with a range of special learning needs.

In particular, the site features the first virtual reality child learning and safety games ever delivered via the web. These games combine navigation in virtual worlds, virtual object recognition and manipulation, training by animated agents, and in some cases, video clips to aid in generalizing the learning to the real world.  All have been designed by a team of experts in special needs, spectrum disorders and special learning skills, and tested via controlled studies in the public schools.

The fire safety game is Strickland’s response to a FIMA directive which advises parents to give their children a “meeting place” to go to in case of a house fire where they can safely wait for help.  The children are guided in their play by an animated agent, a friendly virtual dog named “Buddy,” who patiently, repeatedly leads and encourages the children, without the strains, fears and misunderstanding that can accompany human interaction for children with autism.  Let’s watch.

These anecdotal results and Strickland’s controlled studies show that special needs children do learn navigation and safety skills in a virtual world guided by animated agent teachers, and the learning appears to generalize to the real world.

Strickland will soon launch a street crossing and safety game on the site. In this game, children, led by the animated agent of their choice – a boy, girl, rabbit, or Buddy the dog – learn to explore their backyard safely via a treasure hunt, cross their driveway, walk through a neighborhood and down a busy city street.  The game uses the same principles as the fire safety game:  there are multiple levels and menu options of increasing complexity, the children can practice first with instruction by the agent and then by themselves, and they are rewarded through points, scoring and verbal encouragement for safe exploration of virtual worlds.  The games use the WildTangent platform, and can be ordered on the site for $14.95. 

Also on the site are a series of games which teach language development, vocabulary and skills building. They include customizable menus in which animated agents help the children to practice simple, key actions in the virtual scene such as put the cup on the table and put the book in your backpack, and video clips of real children performing these actions to aid in generalization of learning to the real world.  These games are available by subscription as a one-time download to be played off-line.

Strickland has just received a social skills grant, and plans to develop online shared virtual spaces populated by a range of animated agents and supplemented by video clips to help children practice appropriate behaviors in social settings:  for example, being seated and ordering a meal in a restaurant. 

Also on Strickland’s site is Ken Perlin’s face applet, which lets you build facial expressions so you can learn to link these expressions with their associated emotions – eyebrows up and lips curved up equals happy, and so on.  Strickland encourages feedback from site users, and here is a letter she received recently from the mother of a boy with autism:   

Dear DoToLearn: 

I want to thank you so much for your "Facial Expressions" program. About six months ago, I introduced it to my seven year old son who has Aspergers/HFA and he loved it.  He had no trouble figuring out how to manipulate the face (which, by the way, we think looks eerily like Ann Curry from NBC News!) both with selected emotions and individual facial features.  My son has a great deal of difficulty "reading" people's nonverbal expressions, and this game seemed a perfect tool for him.  He played with the game here and there for about a month and I wasn't sure if he was getting anything out of it.  Then one day he came in and told me about something unexpected that happened at school.  I asked if his teacher was surprised.  He pondered a moment, then said, "Why yes!  She was very surprised!  Her eyebrows went up and her mouth made into an oh!"  CLICK!  That was the first time he related a facial expression with an motion!  I can't tell you how thrilled I was! He spent a lot of time with the game over the next few weeks, and now he only plays with it occasionally.  I think that's because he no longer needs it!  He can figure out for himself what people are saying with their faces. Once again, thank you for this vitally useful tool.  It truly works wonders! 

Sincerely,
C. A.,
Michigan
So we see that a human face, represented as a relatively static, but manipulatible, emotive agent, can dramatically
increase emotional literacy in those with special needs.  

Now let’s combine spatial navigation in virtual worlds, plus the emotive power of the human face in building emotional literacy, and place these in the dynamic, social setting of the avatar virtual worlds platform called DigitalSpace Traveler.  This is a 3D, immersive “talking heads” platform, populated by highly emotive, realtime avatars.  Phonemic extraction is used to drive the avatars, so that as you talk into your computer microphone, your avatar lip syncs accordingly, and people at the other end inworld hear your real voice through their computer speakers.  

The goal is to model a powerful, new form of social, dynamic virtual communication and community, based on the most basic, naturalistic, humanistic, real world metaphors:  live, face-to-face verbal communication in a 3D, realtime graphic setting – in particular:     

·        The animated face, which blinks, breathes, makes simple expressions like a “real person”

·        Live voice, our most comfortable, natural form of expression, which gives us cues to our emotional state

·        Distanced, attenuated, stereo voice for the “cocktail party” effect

·        Immersive, 1st and 3rd person POV and simple 3D navigation for personal and group dynamics such as staging and maneuvering

These real world anchors and metaphors emulate natural social processes, and are designed to help “bind the pair” of the real person and her virtual avatar, enhance sense of presence, and facilitate individual and group communication and community. What results is a powerful mechanism for the organization of social relations:  avatars evolve natural pathways through spaces, gather in natural conversational and social groups, orient towards the speaker, make eye contact, hang out, pause, glance, dance, maintain personal distance, avoid physical discomfort – everything we do with each other in the real world with relative ease…unless we are social phobics or agoraphobics, who can find such tasks excruciating or impossible.  That’s why Dr. Brenda Weiderhold and her associates are placing social phobics in DigitalSpace Traveler, so they can rehearse and play out aspects of social contact within its safe and distanced, but realtime and compelling virtual worlds…and they are finding that this virtual socializing does in fact activate the fear structure sufficiently to permit desensitization and treatment.  

Let me close with two stories not of clinical, but of personal self-healing and collective healing.  Imagine living in a neighborhood with a closeknit community, where the homes, streets, parks and village squares are narrative expressions of your feelings and those of your fellow citizens, of past community events, and of homages to friends. You spend your time in these narrative settings, living out a particular and very real part of your identity that can regenerate back into the brick and mortar of the community.  Prple Tears, a longtime citizen of Traveler, and her community, are living out this virtual narrative of people and spaces everyday in Traveler.  Tears creates virtual spaces as testaments to their community life – here is a getwell card sent by the community to their friend CyberStar, and here is their inworld funeral for CyberStar, when she eventually dies of cancer. Together, Tears and her community explore and extend their “real” needs and hopes and dreams via this narrative, expression-based system where the story and the real have become one.  

AlphaBit is another longtime citizen of 3D avatar virtual worlds, but she makes her virtual home in a platform called ActiveWorlds.  Here is what she has to say about personal and community healing in cyberspace.

My husband Alpha and I dated for 8 months in Activeworlds, then got together IRL. After approximately a year, we married IRL and Virtual. Two months later, Alpha Died of Cancer. Activeworlds Citizens built a prayer meeting site, where a lot of folks met the morning of his death in a last attempt to heal him thru prayer. Odd as it seems, Alpha passed away within minutes of the silent prayer moment.  Later a memorial site was built for folks to say their final goodbyes. At the time, I was in Ohio, far from my family. However, I had my virtual family near me at all times in Activeworlds. I could not have gotten through the pain of loss without them.
The tears flowed as the databits streamed to encourage me to go forward for Alpha's sake.  Therapists will tell you that it is a good thing to talk to people about your loss. Being able to talk...and talk...and talk, helped me a lot.

I am constantly seeing folks who through Activeworlds are able to deal with their losses, amongst friends and loved ones, no matter what the miles between them. On occasion, I return to Alpha's memorial site and just stop and reminisce. Keeps me going:) 

Love,

AlphaBit

Thank you. 

(click here for slides)

Return to Virtual Galen Home

(c) Copyright Galen Brandt, 2002, All Rights Reserved