By: Barbara Robertson

When we think about the first films to convince directors that visual effects created with computer graphics could open their imaginations, two films immediately come to mind: James Cameron’s The Abyss, in which a transparent CG character communicated with an actor, and Cameron’s Terminator 2, which starred a digital, liquid terminator and is lauded as the first movie to show the power of a digital pipeline. Both films won visual effects Oscars, as did Cameron’s Alien before, and Titanic after. Titanic, released in 1997, still holds the record for the largest box-office revenue: $1.8 billion. It was the last feature film Cameron had made. Until now.

A new facial motion-capture system devised by Weta Digital captured actor Zoe Saldana’s
facial expressions and mouth movements to help animators give Neytiri, a CG character, an
emotional performance. The long-awaited and highly anticipated Avatar, written, directed, and produced by Cameron and released by Twentieth Century Fox, pushes digital filmmaking into new worlds. It will immerse audiences in an alien environment, one created entirely with computer graphics and projected, in theaters so equipped, in stereo 3D. Cameron used a Pace Fusion 3D camera to film the live-action segments, but they comprise a small percentage of the film. Weta Digital created the alien planet Pandora and the CG characters and creatures that inhabit it, animating the characters using data from actors’ performances on motion-capture sets. Will it have the same impact on visual effects as did Cameron’s earlier films?

“It certainly changed the way we do things,” says Joe Letteri, senior visual effects supervisor at Weta Digital. “We had to go through a complete re-tooling and re-architecting.” Now a partner at Weta, Letteri has won visual effects Oscars for two episodes of The Lord of the Rings and for King Kong, along with an Oscar nomination for his work on I, Robot while at the New Zealand studio.

In particular, Letteri notes, the studio revamped systems for real-time facial motion capture and muscles, created methods for growing a rain forest in which most of the movie takes place, implemented new lighting techniques, built a compositing pipeline to handle stereo 3D, and more. “We could not allow ourselves to cheat anything,” he says. “Everything had to be done correctly; there was no place to hide.”

Weta used an absorption-based subsurface scattering routine to give the blue-skinned avatars and Na’vi a fleshy, believable look.

In the film, Jake Sully (actor Sam Worthington), a paraplegic war veteran, is given the opportunity to inhabit the athletic body of an avatar. He opts in. His avatar is an alien, a Na’vi, a race of humanoids that populate the planet Pandora. He, like all Na’vi, is blue. A 10-foot-tall biped with a stretched, cat-like body. Almond-shaped eyes. Tail. Pointed ears. Through his avatar, Jake immigrates to Pandora, a lush planet filled with waterfalls, jungles, and six-legged creatures, some of which fly. There he meets the beautiful Neytiri (actor Zoe Saldana) and assimilates into the Na’vian culture.

Everything on Pandora—every plant, creature, and character—is digital, created by artists using computer graphics tools and moved by animators working with keyframe and motion-capture data.

“The planet was really inspired by Jim’s [Cameron] underwater dives,” Letteri says. “There’s bioluminescence. The creatures have blue skin, and the animals have vivid patterns. We all know the rules: Big animals don’t have vivid colors. But, they do underwater, and Jim said they can exist on this planet. So we brought that color palette to the surface and made it believable. However, the big thing was that Jim wanted to do facial motion capture.”

Performing Characters

For Gollum in The Lord of the Rings, Weta had captured Andy Serkis’s body, not his face. For King Kong, they glued markers on Serkis’s face and captured him in a high-resolution volume, and then retargeted the motion data to Kong’s CG face. “Jim didn’t want to go that route,” Letteri says. “He was more interested in a video head rig.”

To make a head-mounted system that would encumber the actors as little as possible, Weta decided to create software that could track facial movements using one camera. Then they took it a step further by re-projecting the motion onto a 3D model in real time.

“We knew Jim would have real-time motion capture on the stage for the characters, and would be recording the faces,” Letteri says. “We thought, wouldn’t it be cool if we could do real-time faces? We knew he was coming in six weeks, so we did some all-nighters and got a system working.” When Cameron arrived, he could see actors on stage wearing a head rig that was driving the facial expressions for a CG character in real time.

Stephen Rosenbaum—who had been on the crew at Industrial Light & Magic for The Abyss as a CG artist, was a CG animator on Terminator 2, and who had won a visual effects Oscar for Forest Gump—was the liaison between Cameron and his Lightstorm group in Los Angeles and Weta in New Zealand. He helped integrate Weta’s creatures, avatar puppets, and facial-capture system into previs and the real-time motion systems developed by Lightstorm and Giant Studios. Rosenbaum was one of six visual effects supervisors at Weta who worked with Letteri on the film. The other five were Dan Lemmon, Eric Saindon, Wayne Stables, Chris White, and Guy Williams.

“Lightstorm created environments at a previs level,” Rosenbaum explains. “We created the creatures and character puppets at Weta that they used within the environments. Giant used our puppets during motion capture. And, when they had scenes where actors needed to interact with creatures, we also provided pre-animated characters so they could see the action during motion capture.”

Giant and Lightstorm performed the real-time motion capture that allowed Cameron to see the CG version of the film at a game-quality level as the actors performed in a motion-capture volume approximately 40 feet wide by 70 feet long. Giant set up the volume using close to 120 industrial cameras from Basler Vision, and handled the re-targeting, in real time, of motion from actors onto the rendered, 10-foot-tall aliens. Lightstorm’s virtual cinematography system, developed by Glen Derry, blended the characters into the virtual set using Autodesk’s MotionBuilder for real-time rendering.

“We could tie into the body capture and add our facial capture simultaneously,” Rosenbaum says. “So [Cameron] could see the body performance and the facial gestures happen [on the CG characters] with the dialog, which was a nizce feature.”

The real-time facial performances weren’t always practical—video projection onto the characters’ faces was sufficient for all but the most subtle scenes. However, Letteri believes it’s game changing.

Weta modeled all the plants in the rain forest on Pandora, seen here virtually, using a rule-based growth system. Some plants have as many as one million polygons.

“It’s one of those things,” Letteri says. “You can see a motion-capture demo, and it’s kind of interesting. But, on set, seeing actors and CG characters performing at the same time, well, that’s really cool. It doesn’t even demo well in a video. When you’re there, it’s a whole different feeling. You have to see it in person.”

Rosenbaum estimates that more than 80 percent of the film is virtual. “We’re delivering about 110 minutes of full CG,” he points out. “I would guess that another 20 minutes have a combination of CG and live action. And, there are some other VFX facilities helping out. We sent some flying creatures, Na’vi, environments, and vehicles to ILM, Framestore, and a few other vendors, as well. But, the bulk of the CG work is being done at Weta.” The list of other vendors that worked on previs and postvis for the film includes BUF, Halon, Hybride, Hydraulx, Lola, Pixel Liberation Front, Stan Winston Studio (now Legacy Effects), and The Third Floor.

Capturing Faces

Each actor captured on set wore a helmet with a lipstick camera attached to a boom arm, and green makeup dots on his or her face. The crew positioned the camera between the actor’s nose and upper lip to capture the mouth movement and to see the eyes. To paint the dots, the makeup artists used a vacuform mask cut with small holes designed for each actor. “We’d put the mask on the face, draw a pen mark for the dots, pull it away, and paint on the green dots,” Rosenbaum says. “The actors loved it. It took only five or 10 minutes and they were back on stage.”

To plot the dot pattern, the facial motion-capture crew had first taken video of the actors doing a FACS session—creating particular expressions, mouthing phonemes, doing prescribed facial gestures—and, if they had dialog, saying their lines. The FACS analysis helped the crew identify major muscle groups for each face so they could position the dots, sometimes as many as 70, most effectively.

For the eyes, Weta developed software to track the pupils. “We had an LED array around the camera so we could illuminate the face and see the pupil clearly,” Rosenbaum says. “And if we couldn’t get good data, we’d track the pupils from the video. Traditional facial capture has always been a problem, but I think our eye movement is fantastic. It sells the characters.”

The eye movement was particularly important because although the avatars have eyebrows, the Na’vi didn’t, so their eyes needed to express much of their emotion. Yet, the iris in the Na’vi eyes was so big, the white of their eyes showed only when they were shocked.

“We ended up adding a stripe pattern to suggest eyebrows,” says Andy Jones, animation director. “We studied Zoe’s [Saldana] expression, and found it was really tricky to get the same feeling on her CG character without eyebrows. To prove it to [Cameron], I roto’d Zoe’s eyebrows out of her face, and he realized what we were up against. That’s when we textured in a pattern to get the feeling of eyebrows back in there.”

The motion captured from the actors on stage drove a facial system developed by Jeff Unay on their corresponding CG characters. To help with the lip sync, character designers had created the lips on the Na’vi to match those of the actors performing them. “We kept the characteristics of the actors and reshaped them into alien characters,” Letteri says. “That gave us a good basis.”

“Solving” software applied the data to Weta’s facial system, and a facial-solving team adjusted the result. The motion data worked best for lip sync and mouth movement; animators spent more time tweaking brow and eye animation. “When the overall expression straight out of the facial solve was not what it should have been, the team would push the data around to get the right poses and extremes, yet still keep the live feeling of the data,” Jones says. “As the team adjusted poses with sliders—they called it ‘tuning’ because they tuned the solve on various frames—the solving software learned which poses to use.”

Unay based the underlying system on blendshapes. “We started with a dynamic muscle rig for the faces, but although it was good at preserving volume, it was coming up short in terms of level of detail,” Jones says. “[Cameron] was very specific. If he saw tension in Zoe’s mouth, he wanted exactly that [in Neytiri]. We had to art-direct and sculpt her face.”

So, Unay modeled blendshapes to mimic a volume-based system using FACS, which describes the muscle groups that control parts of the face. Thousands of shapes. The resulting rig for Neytiri, for example, has 1500 blendshapes. “The animators use sliders that control only about 50 shapes at a time,” Jones says. “The system switches to banks of shapes depending on which muscle sliders they move. It all happens under the hood without the animators knowing. The combinations of shapes look amazing; the skin looks like it’s pressing and pulling.”

As the animators worked in Autodesk’s Maya, they could bring up, on their screens, reference video shot in HD from multiple angles. “We could see the skin and get the timing from the helmet camera, but it distorted the face too much to see the overall mood,” Jones says. “We needed cameras farther away.”

Animators at Weta persuaded director James Cameron to add a stripe pattern to suggest eyebrows on the Na’vi’s faces to help give the computer-generated characters the same emotional feeling as the actors performing them.

Animating Performances

Animators also keyframed Na’vi ears and tails. “We’d whip their tails around if they were upset, and use them as a counterbalance when they ran,” says Jones. “They were like another appendage. We also found the ears really useful for adding emotion to the character.” The ears tell when a Na’vi is angry or shocked, just as they do for cats and dogs.

For the Na’vi bodies, the motion capture worked extremely well. “Giant’s body capture was fantastic,” Jones says. “We still had to animate their hands and fingers, but the offsets and targeting and retargeting was well done. They kept the weight. And, the data was clean.”

The characters’ design might have helped with the retargeting. Rather than completely altering the human proportions, the designers created the Na’vi with similar proportions to humans, but with slim hips, narrow shoulders, and long necks. “It made the retargeting process easier,” Jones says.
Oddly, although animators often use motion-captured data to add the tiny movements to help bring alive a character that is standing still, Weta’s animators found themselves adding jitter to the mocapped data in some cases.

“When someone was yelling or screaming, the high-frequency jitters were often filtered out,” Jones explains. “The system couldn’t distinguish between muscle shake and noise precision. So we would animate it back in, and all of a sudden it felt like the characters were screaming, not just opening their mouths. We had the body muscle rig, but when a bicep fires, there needs to be a jitter. When [Cameron] saw us doing that, he really loved it.”

The muscle rig is new, developed at Weta specifically for this film. “It’s a dynamic system that simulates muscles properly,” Saindon says. “It calculates the fat layers and colliding volumes much more accurately than in the past.”

Prior to this, after animation, the character TDs needed to fine-tune the look of the character and fix problems—intersections, muscles that didn’t look right, and so forth—by sculpting the character on a shot-by-shot basis. With the new system, that was rarely necessary.

“We’d get something much more accurate and realistic straight out of the box,” Saindon says. “We had to do little in the way of going back and fixing things.”

Creatures

In addition to the characters, Weta animators performed approximately 10 creatures, a hellfire wasp, and thousands of insects. “Every single frame has something alive in it, whether it’s a moving plant or bugs,” Williams says.

Of the creatures, four fly and most have six legs. “Our first approach was typically to hide the middle legs, animate the animals as quadrupeds, and then bring the middle legs back in,” Jones says. The animators might animate a horse-like creature by having the leg movement cascade, or change the gait by changing the offset. A cat-like creature might arch its back, lift its front legs, and use them as arms and hands.

Jake learns to ride a creature that looks like a flying horse, and for those shots, the crew used a gimbaled motion-control rig. “The good thing about motion capture was that it gave us the posing [Cameron] liked for the character on top of the creature, where the character should be looking, and the riding style,” Jones explains. “But it was obvious that his legs weren’t reacting to his chest popping up and down, so we couldn’t use the motion capture completely.”

Am I Blue?

Facial capture was perhaps the biggest challenge. The second biggest challenge for the technical team was keeping the aliens from looking like someone had poured blue paint on them. “It was a tricky problem,” Letteri says. “They needed to have warmth under their skin, so we had to find the right shades of blue and blood color that would look good in firelight, blazing sun, overcast skies, and rain. Blue skin quickly wants to look like plastic.”

For skin texture reference, the crew did photo shoots under controlled lights of young people with the most perfect skin they could find. “We discovered that even someone with nearly flawless skin still has lots of imperfections in displacement and color. They have nodules, bumps, pink around their eyes, and blotchy layers,” Williams says. Painters added these imperfections to the texture maps and created a pore structure for the aliens that looked realistic. All this helped make their skin come alive.

As for the color, even though the aliens had blue skin, the crew put red blood in their veins, and did so without turning their skin purple. “Before, we had more of an analytical approximation for subsurface scattering,” Williams says. “We went to an absorption-based subsurface scattering routine. The system we use now does proper frequency-based scattering.”

Because they used the actual wavelength for red transmission through the nose, ears, and pores of the skin, the red blood didn’t cause the blue skin to turn purple. They also added a little red to the skin tone. Then, they applied some of the same techniques and shaders written in Pixar’s RenderMan to the plants.

Deep in the Jungle

“We cross-pollinated the efforts,” Williams says. “The plant shader now uses the skin shader.” The plants, however, aren’t blue, even though they started that way. Blue light from a blue sky bouncing off blue plants onto blue-skinned characters created uninteresting images.

“We needed to have other colors hitting the characters’ skin to give them the kind of complexity that helps make them look real,” Williams says.

At night, as the characters walk through the jungle, the plants glimmer with bioluminescence. The CG artists used subsurface scattering to cause thick plants to glow like a wax candle. “Some plants just have a glowing moss over them,” Saindon says. “It depended on the plant and how [Cameron] felt it should look.”

To create the rain forest, the Weta artists started with FBX files from Lightstorm that they imported into Maya scenes. “We had simple representations for where the trees and plants were,” Saindon says. “Jim moved and placed things where he wanted for camera angles. So, we did a one-to-one match at first to get a layout that he specifically liked.”

Because the plants needed to be dynamic, all of them are models created using a rule-based growth system. Although they average 100,000 polygons, some have as many as one million polygons.
“The plant-growing tools were almost like a modeling tool,” Williams says. “Once we grew a plant, we could instantly create variants by changing the seed value for the random functions.” The variants might change the number of branches and sub-branches, the height, the silhouette, the age, or other parameters.

The crew planted the jungle using painting techniques to place trees, shrubs, and grass. “It’s similar to [Maya’s] Paint Effects, but we aren’t creating geometry,” Saindon says. “The system is taking pre-existing geometry and placing actual full-res models at correct angles on the ground.”

They also used Massive’s software to grow forests. When artists planted seeds on a terrain, Massive would simulate a forest growing and competing for light and space. Bigger trees grew quickly, smaller plants died, and shade-loving ferns grew around the base of the large trees.

Jake Sully (actor Sam Worthington) prepares to inhabit the avatar body resembling a Na’vi, seen forming in the tank behind. The color palette for the film reflects James Cameron’s fascination with the underwater world.

“We’d create large areas, and then on a shot-by-shot basis, would sculpt scenes to play well for the camera and the depth of the scene,” Williams explains. “All of our show is done inside Maya, and everything in the jungle is 3D, so when you move the camera around in Maya, you get a real 3D sense.”

To light and render the massive jungle, Weta implemented two techniques: stochastic pruning and spherical harmonics. The stochastic pruning threw away unnecessary geometry on the fly as a plant moved away from camera. “It might take a fern with a million polygons and push it back to a few pixels when it’s in the distance,” Saindon says.

Spherical harmonics, a technique used for real-time rendering in video games, made it possible to light the rain forest. “Basically, we store coefficients for angles,” Saindon says. “We calculate the harmonics for each individual plant, all the lighting angles, and store that on the geometry. That allows us to drop simple lights into the scene and still get proper occlusion from each plant. The plant does its own self-occlusion using its own harmonics, seeing what should be occluding what, and stores the information. That means we can light an entire jungle with one light. We could get complex lighting with a very simple setup. We couldn’t have done the movie without it.”

Even so, the data processing requirements for the show were enormous. In addition to the characters, Weta created volumetric explosions, fireballs, 3D water simulations, and other effects. “Joe [Letteri] set down the hard line,” Williams says. “He told us not to plan on cheating anything.” At one point during postproduction, the studio was generating 110gb of data an hour.

“Jim Cameron’s expectations are extremely high, and he demands a lot,” Rosenbaum says. “The scope of CG movies is getting so large and the time constraints too tight, that people tend to compromise, but Jim doesn’t compromise. He insists on a high standard. When I worked on The Abyss, it took us six months to create 90 seconds with the pseudopod. We went into it with the same question we had on this film: How the hell will we do this? And we had the same mind-set: We’ll put our heads together and figure it out. He’s always one to push a VFX company. And he certainly did it on this one.”

http://www.cgw.com/Publications/CGW/2009/Volume-32-Issue-12-Dec-2009-/CG-In-Another-World.aspx

In this article, I’m getting right at the meat of what technologies are, getting us closer to that goal, and what this really means for the future of not only motion capture filmmaking, which Steve Perlman of Mova (www.mova.com), refers to as volumetric cinematography, but also the impact it will have on live-action films and mixed media films, like James Cameron’s upcoming Avatar. Mocap is finally coming of age, but the future will be even more exciting.

In the simplest terms, the future of mocap will be a seamlessly integrated motion capture experience where the limitations imposed by current technologies are overcome and the mocap process blends into the background. This will allow us to capture full body and facial motion data, as well as capture the movement and texture of real clothing, skin, props, and environments, while permitting realtime compositing or superimposing with live-action elements.

The idea is to create a mocap environment with all of the advantages of traditional filmmaking — with few of the drawbacks and incredible flexibility in post — not available in traditional cinematography. This is an ambitious though daunting order, but it may not be far off.

NEW WAY OF VIEWING MOCAP

Traditionally, mocap has been viewed as the capture of body and facial motion, translated into animation that drives the motion of 3D characters. While this is still a very big part of mocap, in reality, it goes far beyond character animation. Motion capture often starts with capturing body and facial data, but in many cases, virtual cameras also simultaneously use motion capture to create a perspective and live preview of the actions being performed by the actors. In post production, additional camera motion tracking may be used to set up camera placement and movements. Since each element can be manipulated independently, replacing dialogue, or doing reshoots may only require mocap for one character since the existing mocap for all other characters likely does not need to be changed.

MIXED MEDIA MOCAP

While directors like Robert Zemeckis have embraced mocap for every aspect of their filmmaking environment, others, like James Cameron have used it in conjunction with traditional live-action environments and actors. This creates a whole new set of challenges, but also opens up a lot of new opportunities for interactively combining live-action characters with mocap characters. Glenn Derry, virtual production supervisor on James Cameron’s Avatar (December 2009) explains that although the movie is being filmed with the stereoscopic Pace Cameron Fusion camera system, and works with live-action characters, 75 percent of the movie is virtual. In order to achieve their filming style, they have had to develop a number of tools that allow them to mix live-action actors with mocap virtual characters, as well as combine traditional sets with virtual environments. For example, if they were filming a scene in which an avatar is interacting with an actor, they might be working on location, but using a motion capture set-up that combines an active optical marker tracking system, with inertial sensors, and camera tracking.

Cameron called on Atlanta- and LA-based Giant Studios (www.giantstudios.com)  for its proprietary, realtime mocap technology. The Avatar stage is in Playa Vista, CA.
Derry describes their shooting workflow as very director-centric, meaning they use a digital camera with a virtual eyepiece, which shows a live view of the combined superimposition of the live-action elements along with the virtual characters, which are being rendered and superimposed in realtime, as well as composited greenscreen elements, whose movements are matched to the camera movements based on realtime camera tracking. In other words, the director sees all of the elements of a shot combined in realtime, for a live preview. In this way, the director chooses his shots during filmmaking, which has the advantage of the actor acting for the camera. This can be helpful since an actor adjusts the size of his movements based on the framing of the camera. It is also necessary to use this approach when combining live-action characters since the view of them is fixed by the position of the camera while filming the live-action sequences.

In many ways, they are already achieving many of the goals of future mocap in that they have broken free from filming in a mocap volume. They have done this by combining multiple technologies, and have also succeeded in realtime preview of the superimposed images, making many aspects of the motion capture, just part of the background process.
what’s new?

There are several new technologies that promise greater accuracy and resolution. Others offer better realtime preview and data management, while others offer the flexibility to do them anywhere. Some even approach the capture from a completely different perspective, and promise to capture the motion and texture of cloth and skin, while having scalable resolution options.

HMC

When I visited ImageMovers Digital (www.imagemoversdigital.com) and the set of Robert Zemeckis’s A Christmas Carol this past April, I got to see all of their new toys, including their new facial motion capture system, the HMC or Helmet Mounted Camera set-up, which they developed with Vicon. The HMC is a system with four small cameras attached to small booms, placed low on the face, just below the view of normal visual perception. These are mounted to a lightweight skullcap helmet. This captures multiple perspective video of black dots marked on the actors’ faces, which allows for individual facial motion tracking data to drive the facial motion of an individual character. This can significantly speed up the post process by requiring fewer cameras to capture body motion and keeps the facial motion data for each character separate. This allows for much more reasonable amounts of data to track, solve and retarget. This also helps with realtime preview since selectively, only the body motion has to be solved. It also is useful for early versions of scene renders since facial animation can be postponed until camera positioning and movement have been determined.

The HMC is not dependent on a capture volume, since the cameras go with the actor  and don’t require any sort of fixed stage. This could ultimately be used to drive the facial movement of a character interacting on a normal set or location, assuming facial replacement is the only requirement.

The HMC isn’t the only kid on the block, though. Glenn Derry has been using a single camera facial capture rig they call Headrigs, in Avatar, which predates the HMC. “It also employs a skullcap helmet, but uses a single camera mounted from the side like a headset microphone,” he says.

While lacking the multipositional perspective for tracking, as well as the resolution of a four-camera system over a one camera system, it makes up for it with realtime facial motion solving and playback, which Derry explains is what is most important for their application, and provides them with the results they need in their streamlined, realtime workflow.

RESTROSPECTIVE VERTEX TRACKING

Mova’s Steve Perlman has created a unique approach, using some new and existing technologies, which has some real potential to do things that haven’t really been done before… at least not like this. The Contour Reality Capture system takes a bit of a step backwards, in that, for now, anyway, it places the actor in a very limited environment, similar to the earliest days of facial mocap where the actor had a very limited range of movements. From there, however, it takes a couple of big steps forward, and assuming they can increase the volume to a reasonable size, it could have a big impact on the mocap industry.

The system uses some tried and true concepts, but uses them in a unique way. Several high resolution black and white cameras set in sync with UV lights flickering 120 times a second to capture the phosphorescent glow of UV makeup applied to skin, and UV dye applied to clothing. These multiple perspective cameras will act as the motion capture sources, but the points that are being tracked haven’t yet been determined.

Using a technique referred to as Retrospective Vertex Tracking, Mova’s software uses pattern tracking of the random pattern made by the sponge application of the phosphorescent make-up on the skin. Since the points that are tracked aren’t determined until post, different resolution solves can be made based on the need for more or less resolution, and greater resolution can be applied on areas of the face that need more tracked points. Because of the way it chooses tracked points, it has potential of having much higher resolution solves that traditional tracking marker systems.

At the same time, a UV image is being captured for motion capture. Several regular color cameras are set directly out of sync with the UV lights, but in sync with 5600K fluorescent bulbs. These effectively capture the texture map images which are projected back onto the 3D mesh, basically translating the video into a 3D video of the person’s face, adding  some of the subtle details like minor wrinkles and semi transparent skin.
These are things that don’t register in mocap, but can be seen on video. The result is a 3D mocap, which looks very real. This also works with fabrics and uses no visible markers.

THE FUTURE OF MOCAP VOLUMES

While the ability to capture motion data outside of a mocap volume is critical for some applications, the mocap volumes still have big advantages in terms of accuracy and the ability to view multiple subjects from any angle. This is what still makes them the tool of choice for non-mixed media apps. With this in mind, those who use passive-optical-based mocap volumes are constantly improving and upping the ante with their stages. An example of this is Vicon’s House of Moves (www.moves.com), which has just completed construction on their new mocap volume.

Their new stage can be configured in 30-x-50-feet for full body capture only, or 30-x-30-feet for full body, plus facial and finger capture. This stage differs from their old stage in that it is all white, with 270 near infrared cameras, which is more comfortable to work in since it is easier on the eyes. More importantly, however, was the design of the volume as a traditional soundstage in that it is sound proofed and designed to capture production grade audio, eliminating the need for most ADR.

The cameras are configured in two separate systems so that finger and facial capture is separate from the body motion capture. This allows Autodesk Motion Builder to be used to drive the animation of a virtual character in realtime. They have streamlined their workflow to timecode sync video, animation and audio, and have integrated custom virtual camera rigs with realtime preview to allow the director a tactile, interactive perspective. Hand-held virtual cameras add to the functionality. In addition, Vicon has integrated greenscreens in their volumes for intermixing live-action characters with motion capture characters.

Imagemovers Digital’s mocap stage, which was built for A Christmas Carol, also uses near infrared lighting, but had a separate truss system built for wirework so the mocap cameras wouldn’t be effected by truss movement. Instead of upping the camera count, they relied on 100 cameras, using the HMCs for facial motion capture. Lightstorm Entertainment in conjunction with Giant Studios also relies heavily on mocap volumes, using a 70-x-36-foot volume for performance capture for Avatar, and use a single camera facial capture system they call Headrigs.

JUST THE CAMERA PLEASE

One of the areas of motion capture, which is seeing a rapid increase in demand, has nothing to do with character animation and everything to do with camera placement and movement. This is camera tracking, which has actually been around for a while now, originally used for greenscreen apps so virtual cameras would match the live-action camera movements for compositing live-action characters into 3D environments. Now, however, some of the same systems, like Intersense’s IS900 studio camera tracking system, are being used by directors like Zemeckis to set up camera angles and movements in post.

When it comes time in the post process to bring in the camera, Zemeckis breaks out his IS900, which he has hooked up to three machines. Then, one at a time, he uses the virtual camera to frame his shots and create camera movement. He moves from one computer to another so he can optimize his time, allowing each 3D artist to tweak the movements while he sets up the next shot. In this way, he knows that the shot he is working with is a perfect take, so he can focus all of his attention on getting the right angle, framing and movement. It is a critical part of his post workflow.

Traditional passive optical mocap stages all use specialized virtual camera rigs that act as a physical representation of a virtual camera with some camera and navigation controls. These feed a viewfinder playing back the virtual camera’s view. Some directors prefer to set up their shots as they record the motion capture, whereas others prefer to focus on the capture and deal with the camera set-up details in post. Each has its advantages, and represents different directing and workflow styles and preferences.

INERTIAL TRACKING

Inertial tracking is another mocap technique that is seeing greater implementation. Inertial trackers generally work by having accelerometers and gyroscopes in them that detect motion in any direction along any axis. These provide all sorts of useful information. These sensors can be used independently, with inertial trackers placed at joints and other logical locations on a motion capture suit, like those used by Moven. Or they can provide supplemental information when used with optical systems to provide directional information to help with solving tracking points when they have been blocked from view. Intersense uses a combination of acoustic tracking and inertial trackers in their IS900 camera tracking system.

The advantage inertial sensors have over most other motion capture systems is that they can provide 3D movement data for each inertial sensor without having to be observed by outside sources, like cameras in optical systems or acoustic sensors in acoustic systems. They can be used under clothing, and in most shooting environments. The limitation is: they work well for body capture, but have no solution for facial capture and aren’t practical for finger capture. Used in conjunction with other technologies, however, they have a lot of potential for use in mocap production.

NEXT-GEN WORKFLOW

In many ways, the techniques used by James Cameron on Avatar, now being employed by Weta Digital for Steven Spielberg and Peter Jackson’s latest mocap film, Tintin, is a model of what can be expected to come. On Avatar, the approach is to shoot everything in the same way as you would a normal movie. What they have done, however, is mix in the mocap technologies and feed the characters and environments created in 3D, which are driven by the camera tracker and performance mocap, back into the viewfinder in a live environment in realtime.

The director knows what he is shooting and how it will all be composed when completed. In fact, as they are filming the movie, Cameron is cutting the film, so what is sent off to Weta has already been cut to the frame. This director-centric approach gives the director a very tangible view of what he is capturing. What is lost is the freedom that you have in a completely mocap production. In essence, each of these tools had been designed to best suit both the production style of the directors and, more importantly, work within the environment, restrictions, and requirements demanded.

BIGGER, BETTER, FASTER

As it stands, in the world of mocap there is always a tug of war between increased accuracy and realtime performance. The greater the resolution and number of cameras, the more points can be captured with greater the accuracy of the points and the reduced incidents of occlusion. Unfortunately, this comes at the cost of realtime preview. The flexibility of shooting anywhere, especially outside of a soundstage or volume restricts the normal flexibility of 3D motion capture in that it often solves only the mocap from certain perspectives, which is fine when you are mixing with a traditional camera since its perspective is determined at the time of recording.

As processing gets faster and the tools, which combine multiple technologies, are developed, we can expect higher resolution while still providing realtime playback. There will also be a larger number of tools, and a larger pool of talent. The systems will be more flexible, quicker to set up, provide more accurate data, and provide more integrated workflows.

WHAT IT ALL MEANS

As technical as motion capture is, ultimately you will hear the same thing from virtually all camps, “It’s all about story.” The purpose of these technologies is to allow us to create characters and effects that we could not do before, and do it more realistically and efficiently than ever before. They allow us to create and interact with virtual worlds and blend the lines between reality and fantasy. The tools are becoming more flexible, more powerful and are changing the way a lot of films are being made.

Heath Firestone is a producer and director for Firestone Studios LLC (www.firestonestudios.com), which specializes in mixed media 3D compositing and camera tracking.

http://www.postmagazine.com/ME2/dirmod.asp?sid=&nm=&type=Publishing&mod=Publications::Article&mid=&tier=4&id=EBB3ED7515214734AA0BAA618B57011E