Because of the huge scale of The Avengers, different visual effects houses produced various characters and FX shots for the film. Industrial Light & Magic scored the Hulk and was also asked to update Iron Man’s suit.

In Wired’s video interview, ILM special effects supervisor Jeff White reveals how actor Mark Ruffalo becomes the best cinematic incarnation of the Hulk yet (and also tells how the angry green giant helped wreak havoc on New York City — with only four days of principal photography).

The Avengers, directed by Joss Whedon and rated PG-13, is now playing in theaters worldwide.

Steven Spielberg’s collaboration with Peter Jackson on The Adventures of Tintin was the first time in his career he had worked that closely with another filmmaker, he told The Hollywood Reporter’s executive editor, features Stephen Galloway during an exclusive interview with the two men in Paris.

Spielberg — who directed the picture, which Jackson produced — said he was struck by the humor Jackson brought to the project, based on the intrepid young journalist who features in a series of comic books known around the world. The character was brought to life through motion capture created by Jackson’s Weta Digital in New Zealand.

At the same time, Jackson said he was amazed by his friend’s continuing passion for film. “Steven just walks onto the set every day like it’s the first time,” he said.

In the interview, which can be read in full in the upcoming issue of The Hollywood Reporter, Jackson said that, as a young director, “I started to read about Steven doing the Tintin film and I was waiting and waiting to see it. Then eventually, six or seven years ago, Steven called me.” He admitted he was “a little nervous.”

While Jackson was an expert on motion capture thanks to Lord of the Rings, Spielberg said he learned about it from his friend James Cameron, who invited him to the Marina Del Rey studio or “volume” while Avatar was being made.

“I watched for a while and Jim let me play with — you can’t even call it a camera: it looks like a game controller, with a little television screen and an X and Y control to move the camera around,” he said. “When you walk, the camera dollies; when you go forward, the camera moves in. I was able to play around. Then Peter and Weta devised an entirely new system that was the most remarkable I had ever seen.”

While Spielberg was filming in Los Angeles, Jackson would check in via teleconference at the crack of dawn in New Zealand. “Once, it was very funny, there was one moment where we were rehearsing the actors and Peter had just come down about 4 a.m. his time and I wanted to consult with him about a change in dialogue — and there was Peter on the monitor,” Spielberg laughed. “I said, ‘Peter?’ And Peter was sort of sitting there, but his eyes were closed. I said, ‘Peter! Wake up!’ And he didn’t move.”

The biggest problem the filmmakers confronted, he added, was the script: “We kept changing the script all through the shoot. We had an entire subplot we cut out. We shot it to thicken the plot, because the plots in all the Hergé books are very easy to understand and we tried to overly complicate them and realized that Hergé was right and we were wrong.”

References:http://www.hollywoodreporter.com/news/tintin-steven-spielberg-peter-jackson-249495

The Academy of Motion Picture Arts and Sciences, which gives out the Oscars, is generally loathe to make changes to their award categories. But they’ve shown more willingness in recent years, adding a category for Animated Feature Film, and expanding the roster of Best Picture nominees from five to 10 (and then changing it again).

Given their new-found willingness to adapt with the times, I would petition the Academy that it’s time to add another category to keep up with the way technology is changing movie-making: digital acting.

Maybe “digital” isn’t the right term; perhaps “virtual” acting is better. But this would cover any performance in which the performer’s own body is not viewed on the screen. Instead, they’re represented by some animation, puppet or other special effect.

This would cover motion capture performances like Andy Serkis’ stupendous turn as Caesar the chimpanzee in “Rise of the Planet of the Apes.” Motion capture is where the actor physically performs all or most of the action for a part wearing a special suit. The performance is filmed, and then digital animators “draw” over the actor’s body to change him or her into someone (or something) that looks completely different.

But the category would also cover traditional voice acting, too — a mode of thespian performance that has been not so much derided as ignored.

Voice acting, when it’s done right, does more than simply lend a character a voice. It fills them with a heart and soul that little blips of pixels or splashes of paint could never hope to do. I think of Ellen DeGeneres’ role as Dory the dim-witted fish in “Finding Nemo.” She gave that character such a resonant emotional center, Dory seemed to jump right off the screen.

Or think of Frank Oz as Yoda in the “Star Wars” films. Although the puppetry of “The Empire Strikes Back” and “The Return of the Jedi” seems crude by today’s CGI standards, Oz managed to give the little green guy the commanding presence of a great wizard/scholar.

Serkis has become practically a one-man band at giving his characters presence through motion capture, granting Caesar a cunning intelligence and empathetic heart. He pioneered this type of acting with Gollum in the “Lord of the Rings” trilogy, creating a character who was both utterly malicious and yet pitiable.

In fact, the only major objection I can think of to establish an Oscar for digital acting is that Serkis would probably win it most every year!
Reference
http://www.helloindianapolis.com/commons/pages/articles/opinion/academy_needs_an_oscar_for_digital_acting/237518/

“We’re shooting two films back-to-back, so I’m writing two scripts, not one, which will complete a free-film story arc – not really a trilogy, but just an overall character arc. “We’re doing a lot of preliminary work right now on new software and new animation techniques and so on. We’re creating a new facility in Manhattan Beach so everybody that’s not already dead is coming back.”

And in time when so many visual effects artists are unemployed, Cameron’s projects, due for release in December of 2014 and December 2015, are a welcome change.
“It’s absolutely going to generate jobs. There is a plethora of visual effects artists and a lot of them are out of work right now, so it’s a really great place to get a lot of talented people,” Peter Gend, visual effects instructor at the Art Institute of California in Los Angeles said. “What goes into his motion capture process is really different than what anyone else does.”
Jon Landau, Cameron’s co-producer of the “Avatar” films at Lightstorm Entertainment, is anticipating that more than 700 jobs will be created for the two productions, which are slated to be bigger and better than ever.
“Another thing the Manhattan Beach studio affords us is the ability to expand. Should we need to pick up and do a live-action shoot, should we need more space, they have it,” he said.

And with the first “Avatar” earning the title of the highest-grossing film worldwide, Cameron is indeed feeling the pressure.
“There’s always an expectation. I had to deal with that after ‘The Terminator’ back in 1984. All of a sudden I had a big hit movie and it was ‘what are you doing next?’ But my job is take the audience on a journey and entertain them,” he said. “The second I am sitting down writing, I just go to Pandora. I don’t think about that stuff, about standing on a red carpet. It has its own life, really. The characters have their own lives.”
And on the subject of “The Terminator,” Cameron is sticking by his star, Arnold Schwarzenegger, despite his recent personal problems.
“He and I have spoken a lot,” the director said. “You have to stand by your friends. He and I have talked and I said, ‘look man, I stand by you.’ He doesn’t need my advice. He knows how to manage his image and to say what he needs to say, and he’s going to say it on his own terms when he is ready. It is that simple.”
Cameron’s 1997 blockbuster “Titanic” is being re-released in 3D, so it seemed only natural he weigh in on the news, especially in light of his well-documented hate for 2D films that have been converted to 3D in post-production. “It is actually a little frustrating because it would have been so easy to shoot ‘Titanic’ in 3D, if we’d had cameras back then and if there had been theaters,” he told us. “It’s actually more work (to convert to 3D in post-production) and I don’t really enjoy the process, but I enjoy the result.”
“We have spent several years and millions of dollars trying to create a time machine so that I could go back and shoot it in 3D and it didn’t work out,” he joked. “So we just have to convert it.”

Reference
http://www.foxnews.com/entertainment/2011/06/21/james-cameron-to-generate-hundreds-jobs-by-filming-avatar-sequels-domestically/#ixzz1QCHlWY7V

Production has commenced in Wellington, New Zealand, on “The Hobbit,” filmmaker Peter Jackson’s two film adaptation of J.R.R. Tolkien’s widely read masterpiece.

“The Hobbit” is set in Middle-earth 60 years before Tolkien’s “The Lord of the Rings,” which Jackson and his filmmaking team brought to the big screen in the blockbuster trilogy that culminated with the Oscar-winning “The Lord of the Rings: The Return of the King.”

The two films, with screenplays by Fran Walsh, Philippa Boyens, Guillermo del Toro and Peter Jackson, will be shot consecutively in digital 3D using the latest camera and stereo technology. Filming will take place at Stone Street Studios, Wellington, and on location around New Zealand.

“The Hobbit” follows the journey of title character Bilbo Baggins, who is swept into an epic quest to reclaim the lost Dwarf Kingdom of Erebor, which was long ago conquered by the dragon Smaug. Approached out of the blue by the wizard Gandalf the Grey, Bilbo finds himself joining a company of thirteen dwarves led by the legendary warrior, Thorin Oakensheild. Their journey will take them into the Wild; through treacherous lands swarming with Goblins and Orcs, deadly Wargs and Giant Spiders, Shapeshifters and Sorcerers.

Although their goal lies to the East and the wastelands of the Lonely Mountain first they must escape the goblin tunnels, where Bilbo meets the creature that will change his life forever… Gollum.

Here, alone with Gollum, on the shores of an underground lake, the unassuming Bilbo Baggins not only discovers depths of guile and courage that surprise even him, he also gains possession of Gollum’s “precious” ring that holds unexpected and useful qualities… A simple, gold ring that is tied to the fate of all Middle-earth in ways Bilbo cannot begin to know.

Martin Freeman takes the title role as Bilbo Baggins and Ian McKellen returns in the role of Gandalf the Grey. The Dwarves are played by Richard Armitage (Thorin Oakenshield), Ken Stott (Balin), Graham McTavish (Dwalin), William Kircher (Bifur) James Nesbitt (Bofur), Stephen Hunter (Bombur), Rob Kazinsky (Fili), Aidan Turner (Kili), Peter Hambleton (Gloin), John Callen (Oin), Jed Brophy (Nori), Mark Hadlow (Dori) and Adam Brown (Ori). Reprising their roles from “The Lord of the Rings” trilogy are Cate Blanchett as Galadriel, Andy Serkis as Gollum and Elijah Wood as Frodo. Jeffrey Thomas and Mike Mizrahi also join the cast as Dwarf Kings Thror and Thrain, respectively. Further casting announcements are expected.

“The Hobbit” is produced by Peter Jackson and Fran Walsh, alongside Carolynne Cunningham. Executive producers are Ken Kamins and Zane Weiner, with Philippa Boyens as co-producer. The Oscar-winning, critically acclaimed “The Lord of the Rings” trilogy, also from the production team of Jackson and Walsh, grossed nearly $3 billion worldwide at the box office. In 2003, “The Return of the King” swept the Academy Awards, winning all of the 11 categories in which it was nominated, including Best Picture – the first ever Best Picture win for a fantasy film. The trilogy’s production was also unprecedented at the time.

Among the creative behind-the-scenes team returning to Jackson’s crew are director of photography Andrew Lesnie, production designer Dan Hennah, conceptual designers Alan Lee and John Howe, composer Howard Shore and make-up and hair designer Peter King. Costumes are designed by Ann Maskrey and Richard Taylor.

Taylor is also overseeing the design and production of weaponry, armour and prosthetics which are once again being made by the award winning Weta Workshop. Weta Digital take on the visual effects for both films, led by the film’s visual effects supervisor, Joe Letteri. Post production will take place at Park Road Post Production in Wellington.

“The Hobbit” films are co-produced by New Line Cinema and MGM, with New Line managing production. Warner Bros Pictures is handling worldwide theatrical distribution, with select international territories as well as all international television licensing being handled by MGM. The two films are planned for release in late 2012 and 2013, respectively.

Ian McKellen Talks ‘The Hobbit’ In 3D

Gandalf himself, Ian McKellen, gave an update from near the Wellington set of ‘The Hobbit’ on his official site:

I’ve seen Bilbo — in three dimensions.

I was visiting old friends in the Stone Street offices and heard Martin Freeman was just round the corner by the permanent greenscreen, done up as Bilbo, testing his costume in front of the 3D cameras. Indeed, there he was in the open air, mostly oblivious to the camera, though turning this way and that as required. Martin improvised a hobbity gait, padding back and forth, testing his big hairy Hobbit feet, pointy ears and little tum.

Beneath the shade of a tent, in a sun hat, Andrew Lesnie was remotely controlling the two lenses within the mighty camera which digitally records in 3D. His screen showed the familiar 2D image but next to it, above the director’s chair, was a large colour screen in full magical three dimensions, much as it will appear in the cinema — courtesy of the spy-glasses that transform the blurred outlines onscreen to the high definition exactitude of the 3D effect.

Three Bilbos simultaneously, two performances on screen and the actor beyond: which was the real one? Martin Freeman was transmuting into a character whose reality will soon be as authentic as his own.

Update: DS got to chat with James Nesbitt on being cast as dwarf Bofur in ‘The Hobbit:’

You’re over in New Zealand filming The Hobbit at the moment – how’s that going?

“Good. We haven’t actually started filming yet. We got here quite early – we were always going to spend a number of weeks mainly kind of getting fit because it’s a huge project and will involve a lot of stamina and stuff. And also getting used to the different types of work we’ll be doing.”

How’s Peter Jackson doing after his surgery?

“Peter got a bit ill but he’s well and on the mend now so we start in a few weeks. It’s been a fantastic experience. Obviously it’s very sad here at the moment with the earthquake in Christchurch. It’s been very keenly felt all over the islands. But Kiwis are a tight-knit group and everyone’s offering a lot of support. But yes, it’s terribly beautiful. It’s a wonderful opportunity. It’s great for me just to be involved in something this epic and to have the opportunity to work with all these great people. It really is something I’m very grateful for.

You said before you were having dwarf training – what’s that been like?

“Yeah, it’s good! I mean, we’re all playing dwarves – we’re playing Tolkien dwarves, Bilbo Baggins goes on this journey with 13 dwarves, so it’s great. We’re just really doing different things.”

What have they been making you do?

“We’re doing lots of horse riding and stunts and stuff. We’re basically playing! It’s a very good way to earn a living, it’s great. They’re working us hard I have to say! But they’re very down to earth. It has such a feeling of being a little company, it just happens to be a little company of thousands of people! But everyone is enjoying it. Every day’s a little treat.”

Who is the best and worst at the stunts, horseriding and so on? <

“Richard Armitage is very good at the old horse riding because of course he did it in Robin Hood, so he’s very good at that. Of course he’s playing our leader, so it’s right that he’s good at that. Aidan Turner and Rob Kazinsky, they’re fit young men. But we’re all shapes and all sizes and we all have our own skills.”

Were you a fantasy film fan before you joined The Hobbit?

“No I wasn’t really, to tell you the truth. But obviously coming out to do this now I’ve immersed myself in it a bit. It’s funny – as a company, we watched the three films together, so we were in the cinema all together. It’s a very exciting prospect to be watching those and thinking, ‘Oh my God, I’m going to be in something like that’. It is a chance to be a kid again.”

The Hobbit movies are due for release in December 2012 and December 2013

Source: http://www.comingsoon.net/news/movienews.php?id=75416, http://www.flicksnews.net/2011/03/ian-mckellen-talks-hobbit-in-3d.html

By Richard Verrier, Los Angeles Times When Gore Verbinski was directing his upcoming movie, “Rango,” a spaghetti western-like tale set in a desert town overrun by bandits, he did what he typically does: have his principal actors, led by Johnny Depp and fellow cast members that include Harry Dean Stanton, Abigail Breslin and Ray Winstone, act out key scenes.

The actors wore western costumes — Depp sported a giant cowboy hat and bandana and Winstone packed a sidearm.

They had the usual array of props, including whiskey glasses and sawhorses, on a stage at Universal that also featured

a saloon with a 40-foot-long wooden bar and the requisite swinging doors and even a chuckwagon.

This wasn’t a run-through for another one of Verbinski’s big-budget live- action movies. It was all done as part of a 20-day shoot

to capture the voice tracks for his first animated film, “Rango,” about a chameleon played by Depp with an identity crisis.

In animated movies, actors usually voice the lines of their characters in a recording booth. But Verbinski figured he’d draw out more lively dialogue

if the actors physically performed their scenes onstage — just like on a live action set. “It was just like rehearsing a high school play,” said

Verbinski, best known for directing the first three “Pirates of the Caribbean” movies. “Why give up on what we do in live action?”

With the extensive use of computer-generated animation, or CG, in movies such as the “Pirates” franchise, “Avatar” and “Alice in Wonderland,”

the lines are blurring between live-action and animated pictures in a way that Walt Disney himself could have scarcely imagined. That has created opportunities for directors, cinematographers and even production designers to transfer their skills from one medium to another.

“As live-action filmmaking, in terms of its process and tools, comes closer and closer to the way we’ve always made our animated movies, the crossover has been made much easier for filmmakers,” said Bill Damaschke, co-president of production for Glendale-based DreamWorks Animation. “It’s probably exploded over the last two or three years.”


In a sign of that crossover, DreamWorks Animation recently partnered with Guillermo del Toro, director of such dark fantasy films as “Pan’s Labyrinth” and such supernatural action movies as “Hellboy.”  Del Toro spends at least two days each week at DreamWorks, where he is writing and directing his first animated feature, “Trollhunters,” a story about kids experiencing growing pains in a magical world.

“It’s almost an irresistible medium to play in,” said Del Toro. “I’m a filmmaker who is interested in truth and not reality, and I think there is great emotional truth and power to be found in animation.”

Steven Spielberg and Peter Jackson, two of Hollywood’s biggest names, are making two films based on the popular graphic novel series “Tintin” that combine 3-D performance-capture technology and computer animation. Spielberg is directing the first, “The Adventures of Tintin: The Secret of the Unicorn,” due in late 2011; Jackson will direct the second (a third film is also a possibility).

Today, animation is a significant profit source for Hollywood’s studios and, not surprisingly, attracting the interest of filmmakers.  In 2010 alone, four of the top 10 movies at the box office were animated films, including “Toy Story 3″ and “ ShrekForever After.” This year, Hollywood will release 15 animated films — up from 12 in 2010 and close to the record number of 17 reached in 2002, according to Hollywood.com.

“For a long time, people in live action viewed animation as a sleepy backwater that really wasn’t considered mainstream filmmaking,” said Steve Hulett, a former Disney animator and business representative for the Animation Guild.  “Today, the attitude is much different. I think people have a lot more respect and a little bit of awe.”

By hiring filmmakers who have worked in live action, animation studios hope to bring more realism to their movies. “There’s a whole wealth of experience of telling stories in live action that is now being applied to animation, from the movement of cameras, to how shots are framed and the mode of lighting,” said Roger Deakins, a cinematographer who worked as a visual consultant on the DreamWorks movie “How to Train Your Dragon.”

Industrial Light & Magic, the visual-effects house owned by filmmaker George Lucas, turned to Verbinski — with whom the studio had worked on the first three “Pirates” movies — to create a distinctive “photographic look” for “Rango,” which Paramount Pictures will release in March. Verbinski was familiar with the process of storyboarding and working with computer animators — about 60% of the third “Pirates” film used computer animation —  but other aspects were foreign.

In making digital animated movies, directors don’t set foot on a studio set and cinematographers don’t frame shots from behind the camera on a dolly. Instead, the picture is made entirely on a computer, where the filmmakers use software programs and technology that simulate the functions of a set or a camera.

Directors help shape the initial “story reel” (the rough drawings that lay out the story), guide actors during voice recordings and work closely with animation supervisors and technicians as they create digital characters and scenes one frame at time. A finished animated film can total 130,000 frames and takes two years or more to make.

At DreamWorks, Del Toro worked as a consultant on its recent computer-animated film  “Megamind” and the second installment of “Kung Fu Panda” and has been tapped to be executive producer on two other upcoming films, “The Guardians” and the Shrek spinoff “Puss in Boots.”

“Guillermo brings a unique vision and unmatched level of inventiveness to his storytelling,” DreamWorks’ Damaschke said. He citied Del Toro’s advice in creating a visually more dramatic opening sequence in “Megamind,” with the inept villain in perilous freefall, and tightening the pace of the film, which was cut by several minutes.

It’s not only live action directors who are venturing into animation.

On “Rango,” for example, Verbinski was joined by visual effects supervisor John Knoll and production designer Mark McCreery, both of whom worked with him on the “Pirates” movies. McCreery, who created the Davy Jones character in “Pirates,” crafted similarly lifelike creatures in “Rango,” including a turkey named Gory that “looks so real you feel like you could reach out and touch it,” said Knoll.

The team spent hours watching spaghetti westerns such as “The Good, the Bad, and the Ugly” to absorb director Sergio Leone‘s style, including how he filmed campfire and desert scenes. Verbinski revisited the town in Mexico, Real de Catorce, where he filmed “The Mexican” for ideas on how the fictional town of Dirt should look.

Knoll and his team then created a three-dimensional computer model of Dirt  and used a motion-capture stage at ILM that was equipped with a monitor called a virtual camera that allowed Verbinski to view the town from different angles and then frame the best shots and angles to guide the animators. “We were using a lot of the same visual shorthand that we developed during the ‘Pirates’ pictures,” Knoll said.



Even though animated movies don’t use physical cameras on the set, cinematographers are still needed to decide how shots should be framed with a “virtual camera” inside the digital world as well as how best to light them.

For example, “How to Train Your Dragon” producer Bonnie Arnold tapped Deakins, known for his use of atmospheric lighting in such films as “No Country for Old Men,” to create naturalistic lighting for a variety of scenes in the DreamWorks film — from a moonlight flying sequence to a tender scene between a father and son in a workshop illuminated only by candlelight.

“We created something that mimics candlelight and wraps around the characters faces’ so you can see their expressions,” Deakins said. “You want the audience to feel like these characters are in the real world.”

References:

http://www.motioncapturesociety.com/blog/press/395-the-lines-are-blurring

http://www.rangotrailer.net/

TRON: Legacy, a high-tech adventure set in a digital world that is unlike anything ever captured on the big screen. Directed by Joseph Kosinski, “TRON: Legacy” stars Jeff Bridges, Garrett Hedlund, Olivia Wilde, Bruce Boxleitner, James Frain, Beau Garrett and Michael Sheen and is produced by Sean Bailey, Jeffrey Silver and Steven Lisberger, with Donald Kushner serving as executive producer, and Justin Springer and Steve Gaub co-producing. The “TRON: Legacy” screenplay was written by Edward Kitsis and Adam Horowitz; story by Edward Kitsis & Adam Horowitz and Brian Klugman & Lee Sternthal; based on characters created by Steven Lisberger and Bonnie MacBird.

Joseph Kosinski is very clear in defining his approach to the making of “TRON: Legacy”: “My goal was to really make it feel real. I wanted it to feel like we took motion picture cameras into the world of TRON and shot it. So I wanted to build as many sets as possible. I wanted the materials to be real materials: glass, concrete, steel, so it had this kind of visceral quality to it.”


To achieve the exciting, iconic look for “TRON: Legacy,” Kosinski gathered around him artists from diverse disciplines. “We pulled people from the world of architecture, from automotive design, people who have never worked in movies before. We flew people in from all over the world,” says Kosinski.

Kosinski and his team knew they would be pushing the boundaries of what current effects technology can achieve to make “Legacy” in the spirit of “TRON.” The result is a complicated blend of techniques, from blue screen to 3D cameras, that Kosinski and his team have melded together for the film. Kosinski explains, “I’m going more on instinct rather than experience, but a lot of the technology we’re using is stuff I’ve used bits and pieces of in commercials. However, this is the first time we’re using it simultaneously at this scale.”

In addition to the technological complexities of “TRON: Legacy,” it is also produced in 3D. As Bailey comments, “3D is definitely a challenge technically; the cameras are bigger and heavier and there are a lot of extra variables that you have to take into account, so it definitely slows the process down. But I think it’s a great reason to go to the movies because it’s an experience that you just can’t recreate on an iPhone or your laptop or at home.”

Kosinski continues, “It is a lot more work to shoot in 3D; the VFX are being finished in 3D, which is also a challenge. Having to create separate imagery for both eyes makes it that much more work.”For shooting “TRON: Legacy” in 3D, the filmmakers employed the newest generation of camera, built specifically for them, and used a 3D technique that is a combination of technologies—completely digital motion-capture of a character and the live-action camera system.

The filmmakers and designers of “TRON: Legacy” let their creativity soar to develop an exciting aesthetic for the film that would immerse audiences in a stunning visual landscape never before seen—or imagined.With director Joseph Kosinski at the helm to steer the film’s look and Darren Gilford tapped as production designer, it was clear to both of them that keeping the first film’s spirit alive was key. “The first film established a look that was so iconic,” Gilford explains, “and a lot of that was because of the limitations of the computer, what they really could do back in the ’80s. It was very geometric, very simplistic. With the computer technology we have now, it’s limitless what we can do. But we made a conscious decision that we would not go totally organic. We’d soften shapes and forms where we could, but we would definitely try and maintain those basic ‘TRON’ geometric shapes.”

To accomplish this, heavyweight talents were required, including concept artist David Levy. It was his job to convert Kosinski’s ideas to drawings and designs and establish the new film as its own world. “Joe’s vision evolved the visuals from the first film. He wanted the Grid to feel exactly like our reality, but with a twist,” Levy says.


Kosinski’s aim was to blend the real and the unreal without anyone noticing. “I don’t want the audience to know where the line rests, so sometimes I’m going to shoot everything completely practical, and then sometimes, it will be one practical set piece surrounded by blue screen. And if we do it right, it should be unnoticeable; it should be seamless,” says Kosinski.

In this respect, “TRON: Legacy” strays far away from the original. “The marriage of photorealistic computer-generated images and actual practical sets really gives you a sense of the world that you’re in,” says Jeff Bridges. “In the original ‘TRON’ we didn’t have that because it was basically black duvetyn with white adhesive tape marking things; we never got the feeling of where we actually were. There’s nothing like walking onto the set for the first time and seeing it all dressed.”

Twenty to 25 designers in various art departments churned out concepts and from those Kosinski and his team created the sets—from real-world locations, mixtures of real architecture with blue screen, to fully digital sets. Gilford estimates that there are between 60 and 70 unique settings in the film, split between 15 impressive fully-constructed sets and varying levels of computer-created landscapes.

Finally, since “TRON: Legacy” will be released in 3D, filmmakers were confronted with a unilateral challenge, one which would influence every decision made on the visual aspects of the film. Production designer Gilford says, “There are certain aspects that we had to design around and certain rules we had to obey. For example, when moving the 3D camera rig, one camera could reveal a light source a split second before the other. It can be a nightmare.”

Much care and foresight was also taken in the production design to incorporate iconic images from the mythology of “TRON.” For example, the art department incorporated many of the original film’s images and props into Flynn’s secret lab beneath the arcade. Those with a sharp eye will recognize the Master Control Program desk caddy from the original film, the tabletop computer interface and a condensed version of the Shiva laser, which takes Sam into the Grid. Others will make out a map of the Grid embedded in the code of the background image and Sam’s drawings from childhood on the wall.

While creating the look wasn’t always easy, Gilford admits that it proved to be immensely gratifying: “For a designer, this was a dream, and for my team as well. I really felt that we were able to assemble one of the most unbelievable art departments for ‘TRON: Legacy.’ It was incredibly challenging—but we had a blast doing it.”

The most important unifying element in the Grid is light. “In our film light links everything together. There are ribbons of light that form beneath the street then crawl up sidewalks and buildings, continuing for miles up into the city,” comments production designer Gilford. “Streetlights arise out of and wrap over the street to give the illusion they’re cradling the street.”

At the End of Line Club at the top of the Grid’s mile-high skyscraper, light is embedded in almost every surface: ribbons of light wrap around the floor and ceiling and around the booths. Even the drinks are illuminated. And the club’s roof and walls are glass, offering a view of the city lights and the beacon of the distant portal.But the element of light is perhaps best identified in the lit suits, which were a challenge to construct. Lead concept artist Neville Page and director Kosinski believed the suits they conceived could be made and shot “practical,” that is, without the use of CGI—so the designers went to work to make them a reality.

In the end the lighted suits were created by using electroluminescent lamps made from a flexible polymer film. Most of the form-fitting suits were made out of foam latex, but the Sirens’ suits were made by spraying balloon rubber over spandex, giving an incredible, super-sleek shape. The actors wearing those suits had to be severely compressed within the suits to compensate for the bulk of the electronics.

“In addition to the main cast costumes,” adds Christine Bieselin Clark, who worked with the film’s costume designer Michael Wilkinson, “we also built all of our background costumes. Once you go into the Grid, everybody has some element of light. We ended up making over 140 foam suits, which there is no precedent for”.The body-molded suits with their distinct lighting patterns are influencing clothes and shoe designers, with “TRON: Legacy” fashion elements showing up on runways and in fashion magazines. The distinctive hairstyles, such as upswept hair, and the bold eye-makeup treatments are setting new style trends around the world.

Lightcycles are an important and vital part of the TRON mythology. One of the designers who worked on the sleek, reconfigured Lightcycles in “TRON: Legacy” is Daniel Simon, a former car designer for Bugatti, who used, as a basis, the original sketches by Syd Mead, the designer of the Lightcycles for “TRON.”

Simon explains the challenges: “A Lightcycle forms a visual unit with its rider. His helmet and body become part of the bike design and stance—but you still need to give him freedom to move. That’s not in your catalogue; you have to start from scratch.” Moreover, adds Simon, “The Lightcycles are created out of a baton, so I had to design the entire inside of the bike, every screw and gear, so Digital Domain could transform it in animation. That was interesting, developing the look of how a vehicle might grow.”

Other vehicles in “TRON: Legacy” include the Light Runner, on the Grid a powerful racing car but in the Outlands a tough off-roader; the Recognizer, a huge, U-shaped vehicle that roams the streets looking for wayward programs; Solar Sailers that are flying cargo ships; and Clu’s Rectifier battleship, which is three times larger than any aircraft carrier in the real world, holding Clu’s entire army.

And while one would expect these amazing vehicles to be entirely computer-generated creations, many were also practically built for certain scenes, in keeping with Kosinski’s desire to constantly blur the line between CGI and reality. The filmmakers contracted a company called Wild Factory, who builds prototypes for Volkswagen, to take on the task of bringing some of the vehicles to reality.

On the back of every illuminated suit is a light disc and as in “TRON,” the discs play very significantly in “TRON: Legacy.” The light disc represents the power source, essence and memory bank of every program.Used as a weapon, it will return to its user like a boomerang. The light discs that were created for the film consist of 134 LED lights, are radio-controlled and attached to the light suits with a magnet. They also house the batteries and electronics that power the light suits.

For Grid-dwellers the baton is an important piece of equipment that can be used as a weapon or to store huge amounts of data and utility programs, and by keying in codes, it can create a Lightcycle or other vehicle around the user. As director Kosinski explains,“The baton is the Swiss army knife of this world. It can form any number of vehicles and it can also form different weapons depending upon your specialty.”

“TRON: Legacy” is a showcase for today’s technology and features some firsts in cinematic history: It is the first 3D movie to integrate a fully digital head and body to create the younger version of Jeff Bridges’ character; the first to make extensive use of self-illuminated costumes; the first to create molded costumes using digital sculpture exclusively, creating molds directly from computer files using CNC (Computer Numerical Cutting) technology; and the first 3D movie shot with 35mm lenses and full-35mm chip cameras.

“TRON: Legacy” took the technology known as facial capture to an extraordinary new level. Using a 3D scan of Jeff Bridges, a mold of his face was built and from that a mask was made with 52 holes in it, acting as a template for the facial marker dots tracked by four lipstick cameras attached to a carbon-fiber custom helmet. Meanwhile, a three-dimensional digital version of Bridges was created by Digital Domain using dozens of photographs of Bridges in his early 30s, its movements correlated with the 52 facial markers on the performance mask.It was the filmmakers’ biggest technical hurdle. As director Kosinski says, “I don’t think there is anything more difficult than creating a digital human that’s going to be in the same scene with other real human beings. And to top that off, it’s a digital human that people know…and we must capture all the charisma and personality of Jeff Bridges.”

When playing Clu, Bridges had the 52 markers drawn on his face and wore the Helmet Mounted Camera (HMC); his facial movements fed into the computer and were used to control the expressions and movements of the digital head. Thus, the digital performance of a younger Bridges was controlled by the real Bridges’ performance, as if the younger Bridges were actually on screen. The information sent to the computer made it possible to instruct the digital head to speak and emote in the exact same way Jeff Bridges would on set.

“Clu had to look, feel, breathe and act exactly like the young Jeff,” comments Academy Award® winner Eric Barba, the film’s visual effects supervisor. “Jeff gave us some really great performances to do that with, but it had to be a believable, realistic human—and in this case a perfect early-1980s Jeff Bridges. We took our E-motion Capture technology and pushed it far beyond anything we’ve done. It raised the bar higher than we’ve seen before.”

“TRON: Legacy” is the first film to use the Helmet Mounted Camera in live action, allowing the actor to interact with others in the scene. The technique, as producer Sean Bailey points out, “enabled us to come up with scenes that weren’t possible. And we had a different challenge than ‘Benjamin Button’: what Brad Pitt looks like at eighty years old is speculative, but most people know what Jeff Bridges looked like when he was in ‘Against All Odds,’ so we had to match that. It wasn’t just technology for technology’s sake; it enabled us to write in a whole new way.”

As technology strives to create ever more realistic immersive experiences, the question arises as to how far the integration of humans and computers can really go. Does the premise of “TRON: Legacy” bear any relation to reality? The filmmakers wanted the movie to be grounded in a sense of reality and have a sense of scientific truth. They felt if the audiences feel there is some underlying scientific premise that has been broken, then the story won’t feel real.So through producer Jeffrey Silver the filmmakers reached out to the National Academy of Sciences’ Science and Entertainment Exchange to advise them, asking questions like: Could you inject a digital version of a human being into a computer? And, could a digital personality be reconstituted into human form? They brought scientists in for a roundtable discussion just to talk about some of the fundamental concepts of “TRON: Legacy.”


The answers surprised them. It seems that if one had enough computing power and employed the principles of quantum physics in a theoretical process known as quantum teleportation, then it could happen. “We were delighted; it set off our imaginations. Science fiction is not supposed to be reality; it’s an extrapolation of what is possible, intended to ignite the imagination,” says Silver.

The 3D experience of “TRON: Legacy” will bring viewers into the Grid, and along for the ride, more than any other film in cinematic history. Using the next generation of 3D technology developed after “Avatar,” “TRON: Legacy” will allow the audience to experience the digital grid and be part of the action in the highly stylized landscape.

The “TRON: Legacy” audiences will discover an exciting, evolved grid that pulsates with the latest technology, stunning visual effects and leading-edge design. With an insightful father-son story that’s grounded in cultural reality, a strong, relatable female lead and a unique, one-of-a kind style, “TRON: Legacy” blends what’s real with super-imaginative, eye-popping visuals and 3D action.

Source: http://www.artooz.com/featured-posts/the-making-of-tron-legacy-2010-movie/

Tintin to get Weta treatment

Steven Spielberg has wrapped up shooting the first Tintin movie in Hollywood and will now pass it to Peter Jackson to complete the visual effects.

Spielberg last week completed 32 days of shooting the actors, including Daniel Craig as pirate Red Rackham, using special performance capture technology, Variety reported.

The Adventures of Tintin: Secret of Unicorn will now continue at Weta Digital in Wellington under the eye of Jackson, the film’s producer.

Spielberg and Jackson have been tight-lipped on exactly how performance capture technology a more advanced version of how Andy Serkis was used as the model for the computer-generated Gollum in The Lord of the Rings is being used for the first of three Tintin movies. But it is understood it will be used to make the actors look similar to the characters depicted in the series by Herge.

Requests to visit the Los Angeles set of the film featuring the plucky Belgian reporter have been repeatedly turned down. “You have to see it to understand [the technology]. It really can’t be described,” said Spielberg spokesman Marvin Levy.

Kathleen Kennedy, who is also producing the series, said it was hard to describe exactly the world Spielberg and Jackson were creating in the film. “It’s extremely difficult to explain to someone unless they are standing next to me and usually then their reaction is, ‘Oh my God’.”

Jackson is likely to direct the second Tintin movie.

The special-effects house behind Avatar reveals a bit of its magic

Joe Letteri has served as visual effects supervisor at both Industrial Light & Magic and Weta Digital. In doing so, he’s had a hand in creating the most innovative and creative visual effects in film history. His career spans from The Abyss to the Oscar-nominatedAvatar. Screen Junkies caught up with him at the VES Awards to discuss Steven Spielberg’s first entry in the Tintin trilogy he’s tackling with Peter Jackson.


First up, he discusses what we can expect Tintin to look like:

“We’re experimenting with a number of different looks. When you do 3D you have a range of options to go with, slightly sort of cartoony. You go more Pixar style where there’s realism but still exaggeration. The problem with going completely photoreal with human characters is you want to honor the comicness of it. So we’re still feeling our way around it.”

MORE FROM JOE AFTER THE JUMP.

He then went on to describe introducing Spielberg to the Avatar technology used on the film:

“Tintin was a similar process because Peter [Jackson] sort of suggested to Steven the difficulty of doing a film that’s based on a comic character, trying to do live-action is always difficult to cast and to get the feel of the characters properly. So he suggested the idea of trying to do it as essentially a performance capture film. So Jim [Cameron] showed Steven and Peter what we were doing with ‘Avatar’ and they shot a couple days on the stage and loved the process. We shot ‘Tintin’ the same way. People are definitely picking up on the process. Jim has shown it to a lot of people. A lot of people came through the ‘Avatar’ set. He was pretty open and happy to demonstrate the whole technique. Because he’s designed it like a live-action set, it’s actually easier to pick up than most people think. From the outside it might look daunting but once you get in there on that stage and start working with it, it’s very much like shooting a live action. That was the whole point. You have all this technology in a way to remove the technology.”

He makes it sound so easy. Something must be holding up the works though since we shouldn’t expect to see The Adventures of Tintin: The Secret of the Unicorn in theaters before late 2011. Spielberg, Jackson, and Cameron probably spend most of their time gathering for regular let’s-rub-our-beards-together sessions. What, you think this kind of technology comes out of thin air?

Eighteen months ago, Guillermo del Toro had a 10-year-plan. His life was mapped out, and it had nothing to do with JRR Tolkien’s lovingly rendered cartography of Middle-earth.

“I was calmly laying out the next decade of my life when The Hobbit appeared,” he laughs. “I was preparing all these things and all of a sudden The Hobbit shows up and takes over my life.”

Make no mistake: The Hobbit is his precious. Del Toro knows more than anyone that this diptych could – should – define his career.

And so the director has been busy building a world that not only honours JRR Tolkien’s book and Peter Jackson’s Lord Of The Rings trilogy but will emerge assuredly, triumphantly, his own.

Our very own cuddly character, Jamie Graham, snuggled up to Del Toro at his Wellington base of operations, and talked exclusively about the biggest films of next decade.

The Hobbit has taken much longer to design than your other movies…

It took almost a year. Which for me is very, very long because normally I take about a third of that time to design movies like Hellboy. And if you actually take into account we have three or four times the number of artists… [chuckles].

We produced hundreds, literally hundreds, of drawings; dozens and dozens of maquettes; dozens of material tests. It’s epic. And we are still going to be designing into production…

How did it work with the writing of the script? Presumably you’ve had as much input as Peter Jackson, Fran Walsh and Philippa Boyens?

Many, many months ago we sat down to discuss the structure with 3in by 5in cards and we laid out the two movies.

We were meeting on a daily basis at 9am and we would go at it for hours, into the afternoon. Then in the afternoon I would go to check on design.

Then at one point we split into two teams: I did one pass at things and they did a pass at things; it’s pretty much the way I’m used to co-writing.

But I must say what was great and what made a big difference was the amount of great ideas that I felt were generated in a day – it was staggering.

We could have written three or four versions of The Hobbit [laughs].


You mentioned the structure. Will the book make up the first movie, with the second movie plucked from the appendices and maybe even your imagination? Or will parts of the book be saved for the second movie?

We are respecting the structure established by Professor Tolkien because the order of the adventures in The Hobbit is well known to generations and generations of kids. You don’t want to be moving stuff like that.

But we will be integrating Gandalf’s comings and goings because he does disappear in the book quite often.

So, as opposed to the book, we see where he goes and what happens to him

You and Peter are both visionary filmmakers who will fight for those visions. What happens when you clash?

So far we haven’t come to such a crossroads. We argue and we win at different stages. But I think Peter has been, so far, the perfect producer.

Two filmmakers have produced me in my life, both named Peter. One was Pedro Almodóvar and one is Peter Jackson.

Both times my experience has been that they are perfect producers because they understand the producer is not a producer/director.

A producer is a producer. If there’s an emergency, if everything goes wrong, then the producer can – and should – have a strong opinion.

But while everything is going well, on time, on budget and is creatively solid, there’s no need for that.

Presumably working with Peter is not that much different to working with Mike Mignola on the Hellboy movies?

You nailed it. I’d say Mike is as opinionated as if he was another director because essentially he directs on the page. And Mignola, like Pedro and Peter, knows the process – they all know that at some point you’re going to be alone with the beast [laughs].

You’re going to be the guy and you can only trust your own instincts.

You’re not going to be making a phone call from a remote location to ask a question; you’re going to have to make a decision yourself.

So how arduous has it been commuting between LA and Wellington? You’re now in New Zealand full-time, yes?

Yes. I go to LA very seldom now. It is, however, an incredibly easy commute for me. I’m used to it. I’m used to London-LA and in the same way I’m used to Wellington-LA. I blob out on the plane [laughs] and I have 13 hours all to myself, so it’s a privilege.

I write, or prepare emails, or read, so it’s a really great working day.

And the great advantage between LA and Wellington is that you are essentially in your time zone. You lose a day but you go to sleep in your night in LA and you wake up the next morning in Wellington.

Do you find time to sneak in the odd movie on the plane?

I do! But I try to watch television mostly because it doesn’t need you to have a big screen!

You love creating your creatures and obviously The Hobbit offers some great opportunities. There’s the dragon Smaug, the spiders of Mirkwood, the Wargs, Beorn the bear-man…

The way I phrased it to Weta, I said we would keep the DNA in the same gene pool as the Rings trilogy, but that we would generate a different type of character. For example, in the trilogy most of the creatures are brutish or inarticulate.

In The Hobbit, the creatures speak: Smaug has beautiful lines of dialogue; the Great Goblin has beautiful lines of dialogue; many creatures do. So we had to design them with a different approach because you are not just designing things that are scary.

I also wanted some of the monsters in The Hobbit to be majestic.

I wanted the Wargs to have a certain beauty so that you don’t have a massively clear definition: what is beautiful is good and what is ugly is not. Some of the monsters are absolutely gorgeous.

Smaug won’t be like the dragons in Reign Of Fire, say. Was it a big challenge to communicate his character?

I think one of the designs I’m the proudest of is Smaug. Obviously he took the longest.

It’s actually still active: we’re finishing his colour palette and a little bit of the texture. But the bulk of the design took about a year, solid. It’s because of the unique features of the dragon.

Early in production I came up with a very strong idea that would separate Smaug from every other dragon ever made. The problem was implementing that idea. But I think we’ve nailed it.

What was the idea?

I cannot tell you what it was because it would be a massive spoiler! But I’m 100 per cent happy with Smaug. If there is such as thing as 110 per cent, then I’m there!

What about the spiders? How faithful are they to Shelob from Return Of The King?

Well, they are the progeny of Shelob, but Shelob was quite a promiscuous girl [laughs]. She mated with many partners. And insects and spiders are incredibly adaptable creatures. There will be spiders… [Laughs]

That sounds like a Paul Thomas Anderson s

equel: There Will Be Spiders! But they are visually quite striking and in a different way to Shelob.

I wish I could tell you more but I would be spoiling it again. They are very different. They are more creatures of the shadow, more creatures of the deep forest. They are not earth nesting. They are nesting in the canopies so physically they have adapted to that environment.

Will the sequences involving Smaug and the spiders be genuinely scary?

I think so. I hope so. At least that’s the way we’re approaching it. Every good children’s movie, be it early Miyazaki or Disney, always has a thrilling scene or two. When I read The Hobbit as a kid… Well, you have the moments like when Beorn has the heads of goblins on spikes outside his house [laughs].

Tolkien made no bones about that. There is no way to have a dragon attack a town that’s not scary. It’s the same for the spiders: there is no way of making giant spiders cocooning people so it would be gentle!

Have you been studying real spiders? There are some big ones in New Zealand!

We have been. We have a couple of the guys in the design team who are obsessed with spiders.

They actually do their own little documentaries and features and they go out and capture spiders and they shoot their mouthparts and this and that with macro-lenses.

The main problem with the spider designs is how do you translate the weight into a design so nimble or so long-legged, because a spider has long legs. With Shelob, she was quite low to the ground so she moved like a tank. Our spiders have to feel massive but be very nimble.

Are you OK with the real spiders?

No. I adore insects, and I’m completely fascinated by spiders… But I am completely and absolutely horrified by them, too [laughs].

It’s something that Peter and I share!

How about the scale of The Hobbit? You’ve done big action sequences in Mimic, the Hellboy movies and Blade II, but you’ve never tackled anything like the climactic Battle of Five Armies…

No – and I think that I’m really quite eager to go and do that. But at the same time there were so many battles in the trilogy. So one of the first things is how do we make the battles or the action in The Hobbit feel different from that?

Because it was fresh when the trilogy came out, to see those enormous valleys or fortresses being invaded by warriors.

But then after the trilogy you had Troy, Narnia, everything. It has become quite common seeing two massive CG armies attacking each other.

So we came up with a good solution, I think. It will make the battles stand out.


Is it going to be more intimate?

I wish I could spoil it! All I can say is that we have an incredibly good team of people who know we are not making another Rings. We are not trying to make a quadrilogy, or a pentilogy. We’re tying to make two films that flow with those but that stand on their own completely.

We want to avoid stuff that is not part of the DNA, that is not part of the lexicon, but we also don’t want people to feel “We’ve seen this”.

Except where that familiarity is comforting, like Hobbiton or Rivendell – then you want to feel like you’re coming back home to a movie that you love and cherish.

Will you be using the same palette as the trilogy, dark and fertile?

I think The Hobbit is a bit more colourful. And a bit more operatic. And whimsical. One of the things the book marks very strongly is the seasons, so we’re using that as the basis of our thought.

Presumably it will also be a bit more magical? Have a stronger fairytale vibe?

It is in many ways just what you enjoy in the book. You enjoy an almost chamber piece, like when the stone trolls talk about cooking the dwarves.

It’s such a small piece but at the same time it’s magical and it’s almost a comedy, that you have these enormous creatures talking about cooking these dwarves!

It wouldn’t be a Guillermo del Toro movie unless it possessed a poetic quality, surely?

There is a lot of magic in the film. Peter has the eye of a strong historian, in the sense that the trilogy is incredibly accurate to a world that was created. He’s like an archaeologist who’s digging

something that existed. I think that The Hobbit has a little bit more poetic licence.

It has… How can I say it? It has a little bit more flamboyance.

The Hobbit begins shooting in late spring 2010 and will open in 2011.

http://www.totalfilm.com/features/guillermo-del-toro-on-making-the-hobbit

by: Dominic von Riedemann

After years of uncertainty, AMPAS has decided that motion capture films are ineligible for the Best Animated Feature Film Award.

The Academy of Motion Picture Arts and Sciences laid a 4-year old controversy to rest when they announced that motion-capture films are no longer considered eligible for the Best Animated Feature Film category.

“I just adored it. It made me more like a painter than ever before. I got a chance to do so many jobs that I don’t often do as a director. You get to paint with this device that puts you into a virtual world, and allows you to make your shots and block all the actors with a small hand-held device only three times as large as an XBOX game controller. When Captain Haddock runs across the volume [the name for the motion capture stage], the cameras capture all the information of his physical and emotional moves. So as Andy Serkis runs across the stage, there’s Captain Haddock on the monitor, in full anime, running along the streets of Belgium. Not only are the actors represented in real time, they enter into a three-dimensional world.”

With all the buzz around the skyrocketing popularity of 3-D after the record breaking Avatar made it big, there should be a much bigger reception for the far more impressive and prominent work in performance capture and the animation that accompanies it. There’s a big problem when a spectacular performance from Zoe Saldana doesn’t get one bit of serious awards consideration despite plenty of campaigning. LA Times’ Hero Complex has some great bits from Steven Spielberg speaking with great wonder and passion about his work onThe Adventures of Tintin, which was shot in 3-D with performance capture just like Avatar.

Word from Spielberg comes from a follow-up to a big front page story in the LA Times about the angst of Hollywood actors as more filmmakers embrace performance capture (it’s a hell of a read). Spielberg clears up right from the get-go that the choice to shoot in 3-D and use performance capture isn’t just a gimmick or part of a growing trend. “It was based on my respect for the art of Hergé and wanting to get as close to that art as I could.” Hergé, of course, is the man responsible for creating the comic series, which follows a young intrepid Belgian reporter and his canine sidekick Snowy, mostly taking place in pre-World War II Europe.

Spielberg says the performance capture technique is what helps make the Tintin world more accessible:

“Hergé wrote about fictional people in a real world, not in a fantasy universe,” Spielberg said. “It was the real universe he was working with, and he used National Geographic to research his adventure stories. It just seemed that live action would be too stylized for an audience to relate to. You’d have to have costumes that are a little outrageous when you see actors wearing them. The costumes seem to fit better when the medium chosen is a digital one.”

Interesting, because you would almost think that creating a stylized world based on Hergé’s art through motion capture and animation might be a little harder to accept because of its artistic style, but I get what he’s saying anyway. It’s the same reason something likeScooby-Doo, Rocky and Bullwinkle, and Alvin and the Chipmunks look so damn silly on screen. So rather than dressing actors Jamie Bell (Tintin), Andy Serkis (Captain Haddock), Daniel Craig (Red Rackham), Simon Pegg and Nick Frost (Thomson and Thompson) in silly outfits in a realistic world, you get a completely custom crafted universe where everything feels right.

Not only does it feel right, but it feels genuine as, much like Cameron’s success in Avatar’s presentation of performance capture, the head-rigging captures every bit of an actor’s performance, especially on the face, which avoids the glass-eyed, moving doll look that Robert Zemeckis can’t seem to avoid. For Spielberg, it was pure magic seeing the actor’s performance come alive, not simply watching them with green screen and equipment, but on the digital presentation in the animated world (created by co-producer Peter Jackson’s Weta Workshop) which appears on monitors as filming takes place. Spielberg praised the experience:

“I just adored it. It made me more like a painter than ever before. I got a chance to do so many jobs that I don’t often do as a director. You get to paint with this device that puts you into a virtual world, and allows you to make your shots and block all the actors with a small hand-held device only three times as large as an XBOX game controller. When Captain Haddock runs across the volume [the name for the motion capture stage], the cameras capture all the information of his physical and emotional moves. So as Andy Serkis runs across the stage, there’s Captain Haddock on the monitor, in full anime, running along the streets of Belgium. Not only are the actors represented in real time, they enter into a three-dimensional world.”

But the most encouraging and important thing that Spielberg says about motion capture is what everyone, especially acting awards shows, need to understand. No matter how different someone like Jamie Bell looks on-screen with Tintin’s likeness, “it will be Jamie Bell’s complete physical and emotional performance. If Tintin makes you feel something, it’s Jamie Bell’s soul you’re sensing.” I couldn’t have said it better myself. But we still have to wait two more years, until December 23rd, 2011, which is when The Adventures of Tintinfinally hits theaters in the US.

http://latimesblogs.latimes.com/herocomplex/2010/02/steven-spielberg-on-tintin-technology-it-made-me-more-like-a-painter-than-ever-before-.html

James Cameron and Peter Jackson are the kings of the CGI world. Cameron, of course, directed Titanic, the highest-grossing movie of all time—which he says he’d make with no ship if he were filming today. Jackson was the guy behind bringing Middle-earth to the big screen in the Lord of the Rings trilogy. Now they are back with Avatar and The Lovely Bones, two of the most-hyped films of the holiday season. Newsweek asked them about their new films and how technology is changing Hollywood. An excerpt of the transcript is printed below:

Cameron: So how’s the road trip been on The Lovely Bones?

Jackson: It’s all right. Not too bad. Having a harder job getting over the jet lag than I normally do, but never mind. Getting older, I guess. I’m in … Berlin.

Cameron: Ha, ha! You had to think about it for a minute!

Jackson: I did! I’m flying to Paris as soon as this phone call is over. So we’re talking about technology and movies?

Cameron: People often ask us about the future of filmmaking because we’ve both been innovators in the last few years, creating cutting-edge stuff that gets widely or narrowly adopted. I think the simple answer is that filmmaking is not going to ever fundamentally change. It’s about storytelling. It’s about humans playing humans. It’s about close-ups of actors. It’s about those actors somehow saying the words and playing the moment in a way that gets in contact with the audience’s hearts. I don’t think that changes. I don’t think that’s changed in the last century.

Jackson: There’s no doubt that the industry is in a weird position. It’s not just Hollywood—it’s international. The loss of the independent distribution companies and the finance companies, and the lack of ability to get medium-budget films these days. The studios have found comfort in these enormous movies. The big-budget blockbuster is becoming one of the most dependable forms of filmmaking. It was only three or four years ago when there was a significant risk with that kind of film. Now, especially last summer, we saw blockbuster after blockbuster be released, and they all had significant budgets and they’re all doing fine. It almost doesn’t matter if the film is a good film or a bad film, they’re all doing OK. They’ve lost the ability to have that happen with a low-budget movie and with midrange-budget movies.

Cameron: But they’ve also lost the courage to make, frankly, a movie like Avatar, which is a blockbuster-scaled movie not based on prior arc. All the blockbusters of the last four years, like Transformers, Harry Potter, Spider-Man—they’re all films based on other films or part of a franchise. The idea of making a film of that scale that’s a unique piece has been lost. In the meantime, we have all these increases in technology. And there’s no clear way to pay for these blockbuster movies in the old traditional way. It’s not clear that the technology will come down in price in the near future.

Jackson: People are holding on to the idea of lowering the price. The vast majority of the CGI budget is labor. Unless everything goes to China or Eastern Europe in the sweatshops, that sort of approach, labor is never going to go down. It’s only going to go up.

Cameron: Because computers don’t create beautiful images. People do. Down at your place in Wellington [New Zealand], we had 800 people working on Avatar for the past six months.

Jackson: The ones that are conscious anyway.

Cameron: I’m sure there was a big night at the Wellington pubs a few days ago when they turned over their last shot.

Jackson: I think there were a few pillows and sleeping bags under desks. A lot of media attention is switching to technology in the wrong way. They’re saying the industry is in trouble; will 3-D save it? That really doesn’t have anything to do with it. The industry is in trouble, but it has nothing to do with technology, nor is technology going to necessarily be the savior.

Cameron: No, it can’t: 3-D may help define the idea of the big show at the cinema, the cinematic experience, but I think the heart of the cinematic experience is the group experience. It’s the psychology of sitting in a dark room with a bunch of people and reacting to something, and feeling like your reaction is the same as the rest of the group, a way of proof-checking your emotions are normal.

Jackson: Or not.

Cameron: If you’re the one guy laughing out of 400 people, you’re obviously out of step. I don’t think that’s going to change. People have been downloading films, watching films on laptops, watching films on iPods for quite some time now. Ticket sales are not dropping at the same rate that those other methods of media are rising. I came to filmmaking in the early ’80s, and it was a time of deep economic recession. It was a time when VHS home video was taking money from the theaters. The film industry was depressed. That’s what I knew—a state of upheaval and change. It all sorted itself out. These things always sort themselves out. The fundamental question is: is cinema staying or is it going away? I think it shows no signs of going away. I feel quite confident you and I are going to make the kinds of films we love 10 and 20 years from now.

Jackson: I do too. In addition to the theatrical experience, we will be seeing a lot of other forms of distribution and delivery, which is going to be interesting. We have things like Xbox Live with all the subscribers. It’s not going to be too much longer before Xbox Live produces programming. There are so many opportunities there. Everybody is playing a defensive game. Nobody is going on the attack and being brave and courageous, apart from you.

Cameron: They always say pioneers are those guys flying on planes with arrows in their backs. 3-D will find its place. It’s like color. Color didn’t affect the career of a single actor. And then people will find out about the intimacy to 3-D that can add to a dramatic film that’s not even on the radar of the Hollywood studios right now.

Jackson: I find personally that within 10 minutes I forget that it’s in 3-D, in a good way. The only thing about 3-D is the dullness of the image. But that’s a relatively simple technical hurdle to overcome. It’s just brightening the image.

Cameron: It’s already been overcome. The new technology has already solved the light-level problem. We think it looks fantastic.

Jackson: How far away are we from taking glasses out of the 3-D equation?

Cameron: I’ve seen displays at a laptop size and a relatively modest plasma size that work quite well. You have to situate your head to the sweet spot so you don’t get the double image. But people are always turning the laptop display to get the best image. I can imagine three or four years from now an iPhone that’s 3-D-enabled that doesn’t require glasses that you can watch a movie on. Certainly laptops will be here before that. I think the ones that succeed in the marketplace are the ones that initially make their sets, their displays, to be able to use the glasses. If you’re going to do a Super Bowl party you’re going to have a bowl of glasses on your coffee table, and they’re going to be the disposable kind. And then eventually I think the glasses have to go away for home use. I think that will happen within five years.

Jackson: I’m seeing there’s a lot of misunderstanding about motion capture at the moment.

Cameron: The irony is that one of the first examples of motion capture that worked so beautifully is Gollum in the second and third of your films. Suddenly this new idea had burst on the scene, that a quasi-human creature could be created with such heart and soul.

Jackson: With Gollum and Kong, the key thing that we did was the eyes. I think Gollum and Kong represented the best eyes that I’ve seen in a CGI film.

Cameron: The experience of creating a soulful performance is through the eyes: knowing how to rig eyes, how to light for eyes, get the reflections and refractions in the eyes. Of course, we had big-eyed characters, which we did on purpose. We couldn’t accomplish the character we’re doing in Avatar through any kind of makeup means. That’s been explored for 30 years of Star Trek and Star Wars. But I think the thing I hope that the media can convey to audiences is that this is an actor-driven process. Nayteri, in my film, for example—she is what Zoe [Saldana] created 100 percent. Initially I thought we want to keep the technique under wraps. We don’t want to pull the curtain aside and show people how we’ve done this; we just want to show you my magic. But I’ve recently changed my tune. I want people to see a side-by-side image of Nayteri in a scene and Zoe doing the scene, so they understand that it’s a physical and facial performance. Zoe took months of training at archery and martial arts, so she could move a certain way and have a certain grace. It’s something she created that just translated to her character. This is a highly actor-driven process.

Jackson: Actors will never be replaced. The thought that somehow a computer version of a character is going to be something people prefer to look at is a ludicrous idea. It’s just paranoia. What is great, when you would have used prosthetic makeup, you have motion capture to do a more emotive version. That’s great for nonhuman characters, but in terms of creating nonhuman beings—why on earth would anyone want to do that? It’s so expensive. It’s 20 times more than an actor’s going to cost.
Cameron: The other thing that people aren’t talking about, you can take an actor of a given age, and you can transform their age. Additive makeup can age somebody, but it’s hard to make someone younger. Let’s say you have a novelistic storyline where you cast an actor in their 40s, but the first time you see them they’re 15 years old and the last time you see them they’re 80. This is the Benjamin Button idea. Clint Eastwood could do another Dirty Harry movie and look the way he looked in the ’70s. He would still be making all the performance choices. It would be his voice. We’d just make him 30 years younger. If I did Titanic today, I’d do it very differently. There wouldn’t be a 750-foot-long set. There would be small set pieces integrated into a large CGI set. I wouldn’t have to wait seven days to get the perfect sunset for the kiss scene. We’d shoot it in front of a green screen, and we’d choose our sunset.

Jackson: There are all great tools that people haven’t quite gotten their heads around yet. But one of the things that has happened [is that] people focus on technology. Probably the film industry has been guilty; there’s more attention spent on the technical aspects than the story. That’s led to a self-fulfilling prophecy. People regard CGI as a gimmick, they almost blame CGI for a bad story or a bad script. They talk about CGI as if it’s responsible for a drop in standards. We’ve gotten to a point now where there isn’t nothing else we haven’t seen. We’ve seen dinosaurs, we’ve seen aliens; with Avatar we’ve seen realistic creatures. I think we’re going to enter a phase where there’s less interest in the CGI and there’s a demand for story again. I think we’ve dropped the ball a little bit on stories for the sake of the amazing toys that we’ve played with.

Cameron: I think you’re right. What’s interesting in the marketing evolution of Avatar is that we put out a teaser trailer that was all about the imagery, and people were less than satisfied, because they weren’t learning enough about the story. We put out a story trailer that set the stage and told you what the main character was, and all of a sudden people were wildly excited about the movie. There’s the proof within the marketing evolution of a single film.

Director James Cameron had many reasons to be happy the morning that this year’s Oscar nominations were announced: His blockbuster movie “Avatar” tied for the most with nine, including best picture and best director. But he was dismayed that his cast, including stars Zoe Saldana, Sam Worthington and Sigourney Weaver, was shut out.

In fact, unlike the great majority of best picture nominees, the “Avatar” actors have not nabbed one major critic’s award, or guild prize. The snubs reflect the apparent ambivalence of the film community — especially actors — to “Avatar” and its revolutionary use of “performance capture,” a new technology that combines human actors with computer-generated animation to create the blue, 10-foot-tall creatures who are the heart of the movie.

To the uninitiated, it raises basic questions: Is this acting, or is it animation? And, does this suggest that actors could become obsolete? It’s an issue that provokes a strong response from Hollywood figures, from best actor nominees Jeff Bridges and Jeremy Renner to directors Cameron and Steven Spielberg.

“I’m sure they could do it now if they wanted. Actors will kind of be a thing of the past,” Bridges told Tribune Newspapers the day nominations were announced. “We’ll be turned into combinations. A director will be able to say, ‘I want 60 percent Clooney; give me 10 percent Bridges; and throw some Charles Bronson in there.’ They’ll come up with a new guy who will look like nobody who has ever lived and that person or thing will be huge,” he said.


Renner, nominated for ” The Hurt Locker,” put it this way: “Some movies are actors’ kind of movies and some movies are more directors’ movies. ‘Avatar’ is a spectacle. It’s a beautiful experience, but it’s not really an actors’ kind of movie. It doesn’t really allow for an actor to truly tell a story. The director’s telling the story in that one.”

Perhaps mindful that actors make up the largest Oscar voting bloc, Cameron fiercely promotes the contributions of his cast to the success of “Avatar.” He and other advocates of performance capture (known as “motion capture” in its previous, less sophisticated incarnation), including Spielberg, say not enough actors have experienced the process to appreciate it.

“There’s a learning curve for the acting community, and they’re not up to speed yet,” Cameron said. “We didn’t get out and proselytize with the Screen Actors Guild as we probably should have to raise awareness. Not only should they not be afraid of it, they should be excited about it. There is a new set of possibilities, after a century of doing movie acting in the same way.”

Cameron describes it as “an actor-driven process.”

“I’m not interested in being an animator. … That’s what Pixar does. What I do is talk to actors. ‘Here’s a scene. Let’s see what you can come up with,’ and when I walk away at the end of the day, it’s done in my mind. In the actor’s mind, it’s done. There may be a whole team of animators to make sure what we’ve done is preserved, but that’s their problem. Their job is to use the actor’s performance as an absolute template without variance for what comes out the other end.”

“I like to think of it as digital makeup, not augmented animation,” said Spielberg, who is using Cameron’s “Avatar” technology in his new movie, “The Adventures of Tintin: The Secret of the Unicorn.” “It’s basically the actual performance of the actual actor, and what you’re simply experiencing is makeup.”

In the case of “Avatar,” Spielberg said, “the digital makeup is so thin you actually see everything that Zoe (Saldana) is doing. Every nuance of that performance comes through digitally.”

Spielberg and Cameron say that making a movie in performance capture is, for the actors, very similar to performing a play.

“Motion capture brings the director back to a kind of intimacy that actors and directors only know when they’re working in live theater,” Spielberg said.

Recording takes place on a spare motion-capture stage called the volume. Actors wear skin-tight bodysuits with reflective markers; every movement is tracked by an array of more than 100 fixed cameras. A specialized head-rig camera records the actor’s face and eyes.

“The virtual camera is always active,” explained “Avatar” producer Jon Landau. Gone is the need for camera and lighting setups, makeup retouches and costume fittings. Scenes do not need to be shot repeatedly from different camera angles. Instead, the camera data are fed into a computer that creates a 3-D replica of the actor’s every movement, and the director can just add his camera moves — from any perspective — digitally.

“There’s a purity to it. You can’t rely on anything else but your own skill as an actor; (it) enables the actor to shoot the scene in one take without worrying where the camera is,” said Andy Serkis, a veteran British stage actor who pioneered motion-capture acting as Gollum in Peter Jackson’s “Lord of the Rings” trilogy. Serkis also took the title role in Jackson’s remake of “King Kong” and is performing in Spielberg’s “Tintin.”

“If you don’t have the performance, the rest is dressing,” Serkis said. “You can’t enhance a bad performance with animation. You can’t dial it up, lift the lip or the eyebrow. It has to be right at the core moment. It’s the same as conventional shooting.” For actors to not recognize “performance capture as acting is bad and disrespectful. It’s also Luddite.”

In the case of “Avatar,” some complain that Cameron’s characters are too one-dimensional to merit their actors a nomination, but others believe that “Avatar” star Saldana, whose every minute on screen is in performance capture, was robbed of recognition.

“Zoe played Neytiri with such strength, grace and force. If the audience realized just how much, they would have appreciated the performance more,” said “Avatar” co-star Sigourney Weaver. “The technology is so innovative, and it will just continue to get more innovative. We might as well recognize (the contributions of actors) now.”

From a director’s standpoint, recording in performance capture is unusually free and fast. On a typical day of a live-action production, a director might complete a dozen or so scenes in which the lights, cameras, scenery and actors are repositioned. Spielberg said that on “Tintin,” he completed 75 setups a day on the motion-capture stage and finished principal photography in 30 days. That’s less than half the time it would have taken to shoot a live-action version of the film.


“It allows the director and cast to focus on the performance,” Spielberg said. “The director sits right on the floor (with the actors). Because he’s not wearing a motion-capture suit, he appears invisible.”

“One hundred percent of my focus is on the actors,” Cameron said. “I’m not thinking about the lighting, the dolly, or waiting around … to light the shot.”

Though performance-capture veterans speak enthusiastically about the technique, questions remain. Many wonder whether Saldana will get the kind of career boost usually associated with co-starring in a box-office bonanza. The Screen Actors Guild recently appointed a committee to look into what SAG President Ken Howard described as “pay and recognition” issues associated with performance capture in movies and video games. In fact, studios haven’t formally recognized SAG’s jurisdiction over the work, leaving it up to each employer to decide whether the performers receive standard union benefits such as minimum pay or meal breaks.

Moreover, the actors are not the only ones unsure about their primacy in the process. There’s also a branch of animators who don’t want their contributions overlooked. Cameron points out that it took a team of 20 or more animators at the Weta Workshop in New Zealand nine months to fully animate each “Avatar” character.

“The academy has to come to terms with where (performance capture) goes,” said director Henry Selick, whose ” Coraline” is nominated for best animated film. “Is it animation? Is it a new category? I’m like the academy. I don’t know where it fits. I will tell you this: Animators have to work very, very hard with the motion-capture data. After the performance is captured, it’s not just plugged into the computer which spits out big blue people. It’s a hybrid.”

Tribune Newspapers writers Richard Verrier, Amy Kaufman and Yvonne Villarreal contributed to this report.

http://www.chicagotribune.com/entertainment/sc-ent-0224-avatar-actors-20100224,0,1272757.story

CG In Another World

By: Barbara Robertson

When we think about the first films to convince directors that visual effects created with computer graphics could open their imaginations, two films immediately come to mind: James Cameron’s The Abyss, in which a transparent CG character communicated with an actor, and Cameron’s Terminator 2, which starred a digital, liquid terminator and is lauded as the first movie to show the power of a digital pipeline. Both films won visual effects Oscars, as did Cameron’s Alien before, and Titanic after. Titanic, released in 1997, still holds the record for the largest box-office revenue: $1.8 billion. It was the last feature film Cameron had made. Until now.

A new facial motion-capture system devised by Weta Digital captured actor Zoe Saldana’s
facial expressions and mouth movements to help animators give Neytiri, a CG character, an
emotional performance. The long-awaited and highly anticipated Avatar, written, directed, and produced by Cameron and released by Twentieth Century Fox, pushes digital filmmaking into new worlds. It will immerse audiences in an alien environment, one created entirely with computer graphics and projected, in theaters so equipped, in stereo 3D. Cameron used a Pace Fusion 3D camera to film the live-action segments, but they comprise a small percentage of the film. Weta Digital created the alien planet Pandora and the CG characters and creatures that inhabit it, animating the characters using data from actors’ performances on motion-capture sets. Will it have the same impact on visual effects as did Cameron’s earlier films?

“It certainly changed the way we do things,” says Joe Letteri, senior visual effects supervisor at Weta Digital. “We had to go through a complete re-tooling and re-architecting.” Now a partner at Weta, Letteri has won visual effects Oscars for two episodes of The Lord of the Rings and for King Kong, along with an Oscar nomination for his work on I, Robot while at the New Zealand studio.

In particular, Letteri notes, the studio revamped systems for real-time facial motion capture and muscles, created methods for growing a rain forest in which most of the movie takes place, implemented new lighting techniques, built a compositing pipeline to handle stereo 3D, and more. “We could not allow ourselves to cheat anything,” he says. “Everything had to be done correctly; there was no place to hide.”

Weta used an absorption-based subsurface scattering routine to give the blue-skinned avatars and Na’vi a fleshy, believable look.

In the film, Jake Sully (actor Sam Worthington), a paraplegic war veteran, is given the opportunity to inhabit the athletic body of an avatar. He opts in. His avatar is an alien, a Na’vi, a race of humanoids that populate the planet Pandora. He, like all Na’vi, is blue. A 10-foot-tall biped with a stretched, cat-like body. Almond-shaped eyes. Tail. Pointed ears. Through his avatar, Jake immigrates to Pandora, a lush planet filled with waterfalls, jungles, and six-legged creatures, some of which fly. There he meets the beautiful Neytiri (actor Zoe Saldana) and assimilates into the Na’vian culture.

Everything on Pandora—every plant, creature, and character—is digital, created by artists using computer graphics tools and moved by animators working with keyframe and motion-capture data.

“The planet was really inspired by Jim’s [Cameron] underwater dives,” Letteri says. “There’s bioluminescence. The creatures have blue skin, and the animals have vivid patterns. We all know the rules: Big animals don’t have vivid colors. But, they do underwater, and Jim said they can exist on this planet. So we brought that color palette to the surface and made it believable. However, the big thing was that Jim wanted to do facial motion capture.”

Performing Characters

For Gollum in The Lord of the Rings, Weta had captured Andy Serkis’s body, not his face. For King Kong, they glued markers on Serkis’s face and captured him in a high-resolution volume, and then retargeted the motion data to Kong’s CG face. “Jim didn’t want to go that route,” Letteri says. “He was more interested in a video head rig.”

To make a head-mounted system that would encumber the actors as little as possible, Weta decided to create software that could track facial movements using one camera. Then they took it a step further by re-projecting the motion onto a 3D model in real time.

“We knew Jim would have real-time motion capture on the stage for the characters, and would be recording the faces,” Letteri says. “We thought, wouldn’t it be cool if we could do real-time faces? We knew he was coming in six weeks, so we did some all-nighters and got a system working.” When Cameron arrived, he could see actors on stage wearing a head rig that was driving the facial expressions for a CG character in real time.

Stephen Rosenbaum—who had been on the crew at Industrial Light & Magic for The Abyss as a CG artist, was a CG animator on Terminator 2, and who had won a visual effects Oscar for Forest Gump—was the liaison between Cameron and his Lightstorm group in Los Angeles and Weta in New Zealand. He helped integrate Weta’s creatures, avatar puppets, and facial-capture system into previs and the real-time motion systems developed by Lightstorm and Giant Studios. Rosenbaum was one of six visual effects supervisors at Weta who worked with Letteri on the film. The other five were Dan Lemmon, Eric Saindon, Wayne Stables, Chris White, and Guy Williams.

“Lightstorm created environments at a previs level,” Rosenbaum explains. “We created the creatures and character puppets at Weta that they used within the environments. Giant used our puppets during motion capture. And, when they had scenes where actors needed to interact with creatures, we also provided pre-animated characters so they could see the action during motion capture.”

Giant and Lightstorm performed the real-time motion capture that allowed Cameron to see the CG version of the film at a game-quality level as the actors performed in a motion-capture volume approximately 40 feet wide by 70 feet long. Giant set up the volume using close to 120 industrial cameras from Basler Vision, and handled the re-targeting, in real time, of motion from actors onto the rendered, 10-foot-tall aliens. Lightstorm’s virtual cinematography system, developed by Glen Derry, blended the characters into the virtual set using Autodesk’s MotionBuilder for real-time rendering.

“We could tie into the body capture and add our facial capture simultaneously,” Rosenbaum says. “So [Cameron] could see the body performance and the facial gestures happen [on the CG characters] with the dialog, which was a nizce feature.”

The real-time facial performances weren’t always practical—video projection onto the characters’ faces was sufficient for all but the most subtle scenes. However, Letteri believes it’s game changing.

Weta modeled all the plants in the rain forest on Pandora, seen here virtually, using a rule-based growth system. Some plants have as many as one million polygons.

“It’s one of those things,” Letteri says. “You can see a motion-capture demo, and it’s kind of interesting. But, on set, seeing actors and CG characters performing at the same time, well, that’s really cool. It doesn’t even demo well in a video. When you’re there, it’s a whole different feeling. You have to see it in person.”

Rosenbaum estimates that more than 80 percent of the film is virtual. “We’re delivering about 110 minutes of full CG,” he points out. “I would guess that another 20 minutes have a combination of CG and live action. And, there are some other VFX facilities helping out. We sent some flying creatures, Na’vi, environments, and vehicles to ILM, Framestore, and a few other vendors, as well. But, the bulk of the CG work is being done at Weta.” The list of other vendors that worked on previs and postvis for the film includes BUF, Halon, Hybride, Hydraulx, Lola, Pixel Liberation Front, Stan Winston Studio (now Legacy Effects), and The Third Floor.

Capturing Faces

Each actor captured on set wore a helmet with a lipstick camera attached to a boom arm, and green makeup dots on his or her face. The crew positioned the camera between the actor’s nose and upper lip to capture the mouth movement and to see the eyes. To paint the dots, the makeup artists used a vacuform mask cut with small holes designed for each actor. “We’d put the mask on the face, draw a pen mark for the dots, pull it away, and paint on the green dots,” Rosenbaum says. “The actors loved it. It took only five or 10 minutes and they were back on stage.”

To plot the dot pattern, the facial motion-capture crew had first taken video of the actors doing a FACS session—creating particular expressions, mouthing phonemes, doing prescribed facial gestures—and, if they had dialog, saying their lines. The FACS analysis helped the crew identify major muscle groups for each face so they could position the dots, sometimes as many as 70, most effectively.

For the eyes, Weta developed software to track the pupils. “We had an LED array around the camera so we could illuminate the face and see the pupil clearly,” Rosenbaum says. “And if we couldn’t get good data, we’d track the pupils from the video. Traditional facial capture has always been a problem, but I think our eye movement is fantastic. It sells the characters.”

The eye movement was particularly important because although the avatars have eyebrows, the Na’vi didn’t, so their eyes needed to express much of their emotion. Yet, the iris in the Na’vi eyes was so big, the white of their eyes showed only when they were shocked.

“We ended up adding a stripe pattern to suggest eyebrows,” says Andy Jones, animation director. “We studied Zoe’s [Saldana] expression, and found it was really tricky to get the same feeling on her CG character without eyebrows. To prove it to [Cameron], I roto’d Zoe’s eyebrows out of her face, and he realized what we were up against. That’s when we textured in a pattern to get the feeling of eyebrows back in there.”

The motion captured from the actors on stage drove a facial system developed by Jeff Unay on their corresponding CG characters. To help with the lip sync, character designers had created the lips on the Na’vi to match those of the actors performing them. “We kept the characteristics of the actors and reshaped them into alien characters,” Letteri says. “That gave us a good basis.”

“Solving” software applied the data to Weta’s facial system, and a facial-solving team adjusted the result. The motion data worked best for lip sync and mouth movement; animators spent more time tweaking brow and eye animation. “When the overall expression straight out of the facial solve was not what it should have been, the team would push the data around to get the right poses and extremes, yet still keep the live feeling of the data,” Jones says. “As the team adjusted poses with sliders—they called it ‘tuning’ because they tuned the solve on various frames—the solving software learned which poses to use.”

Unay based the underlying system on blendshapes. “We started with a dynamic muscle rig for the faces, but although it was good at preserving volume, it was coming up short in terms of level of detail,” Jones says. “[Cameron] was very specific. If he saw tension in Zoe’s mouth, he wanted exactly that [in Neytiri]. We had to art-direct and sculpt her face.”

So, Unay modeled blendshapes to mimic a volume-based system using FACS, which describes the muscle groups that control parts of the face. Thousands of shapes. The resulting rig for Neytiri, for example, has 1500 blendshapes. “The animators use sliders that control only about 50 shapes at a time,” Jones says. “The system switches to banks of shapes depending on which muscle sliders they move. It all happens under the hood without the animators knowing. The combinations of shapes look amazing; the skin looks like it’s pressing and pulling.”

As the animators worked in Autodesk’s Maya, they could bring up, on their screens, reference video shot in HD from multiple angles. “We could see the skin and get the timing from the helmet camera, but it distorted the face too much to see the overall mood,” Jones says. “We needed cameras farther away.”

Animators at Weta persuaded director James Cameron to add a stripe pattern to suggest eyebrows on the Na’vi’s faces to help give the computer-generated characters the same emotional feeling as the actors performing them.

Animating Performances

Animators also keyframed Na’vi ears and tails. “We’d whip their tails around if they were upset, and use them as a counterbalance when they ran,” says Jones. “They were like another appendage. We also found the ears really useful for adding emotion to the character.” The ears tell when a Na’vi is angry or shocked, just as they do for cats and dogs.

For the Na’vi bodies, the motion capture worked extremely well. “Giant’s body capture was fantastic,” Jones says. “We still had to animate their hands and fingers, but the offsets and targeting and retargeting was well done. They kept the weight. And, the data was clean.”

The characters’ design might have helped with the retargeting. Rather than completely altering the human proportions, the designers created the Na’vi with similar proportions to humans, but with slim hips, narrow shoulders, and long necks. “It made the retargeting process easier,” Jones says.
Oddly, although animators often use motion-captured data to add the tiny movements to help bring alive a character that is standing still, Weta’s animators found themselves adding jitter to the mocapped data in some cases.

“When someone was yelling or screaming, the high-frequency jitters were often filtered out,” Jones explains. “The system couldn’t distinguish between muscle shake and noise precision. So we would animate it back in, and all of a sudden it felt like the characters were screaming, not just opening their mouths. We had the body muscle rig, but when a bicep fires, there needs to be a jitter. When [Cameron] saw us doing that, he really loved it.”

The muscle rig is new, developed at Weta specifically for this film. “It’s a dynamic system that simulates muscles properly,” Saindon says. “It calculates the fat layers and colliding volumes much more accurately than in the past.”

Prior to this, after animation, the character TDs needed to fine-tune the look of the character and fix problems—intersections, muscles that didn’t look right, and so forth—by sculpting the character on a shot-by-shot basis. With the new system, that was rarely necessary.

“We’d get something much more accurate and realistic straight out of the box,” Saindon says. “We had to do little in the way of going back and fixing things.”

Creatures

In addition to the characters, Weta animators performed approximately 10 creatures, a hellfire wasp, and thousands of insects. “Every single frame has something alive in it, whether it’s a moving plant or bugs,” Williams says.

Of the creatures, four fly and most have six legs. “Our first approach was typically to hide the middle legs, animate the animals as quadrupeds, and then bring the middle legs back in,” Jones says. The animators might animate a horse-like creature by having the leg movement cascade, or change the gait by changing the offset. A cat-like creature might arch its back, lift its front legs, and use them as arms and hands.

Jake learns to ride a creature that looks like a flying horse, and for those shots, the crew used a gimbaled motion-control rig. “The good thing about motion capture was that it gave us the posing [Cameron] liked for the character on top of the creature, where the character should be looking, and the riding style,” Jones explains. “But it was obvious that his legs weren’t reacting to his chest popping up and down, so we couldn’t use the motion capture completely.”

Am I Blue?

Facial capture was perhaps the biggest challenge. The second biggest challenge for the technical team was keeping the aliens from looking like someone had poured blue paint on them. “It was a tricky problem,” Letteri says. “They needed to have warmth under their skin, so we had to find the right shades of blue and blood color that would look good in firelight, blazing sun, overcast skies, and rain. Blue skin quickly wants to look like plastic.”

For skin texture reference, the crew did photo shoots under controlled lights of young people with the most perfect skin they could find. “We discovered that even someone with nearly flawless skin still has lots of imperfections in displacement and color. They have nodules, bumps, pink around their eyes, and blotchy layers,” Williams says. Painters added these imperfections to the texture maps and created a pore structure for the aliens that looked realistic. All this helped make their skin come alive.

As for the color, even though the aliens had blue skin, the crew put red blood in their veins, and did so without turning their skin purple. “Before, we had more of an analytical approximation for subsurface scattering,” Williams says. “We went to an absorption-based subsurface scattering routine. The system we use now does proper frequency-based scattering.”

Because they used the actual wavelength for red transmission through the nose, ears, and pores of the skin, the red blood didn’t cause the blue skin to turn purple. They also added a little red to the skin tone. Then, they applied some of the same techniques and shaders written in Pixar’s RenderMan to the plants.

Deep in the Jungle

“We cross-pollinated the efforts,” Williams says. “The plant shader now uses the skin shader.” The plants, however, aren’t blue, even though they started that way. Blue light from a blue sky bouncing off blue plants onto blue-skinned characters created uninteresting images.

“We needed to have other colors hitting the characters’ skin to give them the kind of complexity that helps make them look real,” Williams says.

At night, as the characters walk through the jungle, the plants glimmer with bioluminescence. The CG artists used subsurface scattering to cause thick plants to glow like a wax candle. “Some plants just have a glowing moss over them,” Saindon says. “It depended on the plant and how [Cameron] felt it should look.”

To create the rain forest, the Weta artists started with FBX files from Lightstorm that they imported into Maya scenes. “We had simple representations for where the trees and plants were,” Saindon says. “Jim moved and placed things where he wanted for camera angles. So, we did a one-to-one match at first to get a layout that he specifically liked.”

Because the plants needed to be dynamic, all of them are models created using a rule-based growth system. Although they average 100,000 polygons, some have as many as one million polygons.
“The plant-growing tools were almost like a modeling tool,” Williams says. “Once we grew a plant, we could instantly create variants by changing the seed value for the random functions.” The variants might change the number of branches and sub-branches, the height, the silhouette, the age, or other parameters.

The crew planted the jungle using painting techniques to place trees, shrubs, and grass. “It’s similar to [Maya’s] Paint Effects, but we aren’t creating geometry,” Saindon says. “The system is taking pre-existing geometry and placing actual full-res models at correct angles on the ground.”

They also used Massive’s software to grow forests. When artists planted seeds on a terrain, Massive would simulate a forest growing and competing for light and space. Bigger trees grew quickly, smaller plants died, and shade-loving ferns grew around the base of the large trees.

Jake Sully (actor Sam Worthington) prepares to inhabit the avatar body resembling a Na’vi, seen forming in the tank behind. The color palette for the film reflects James Cameron’s fascination with the underwater world.

“We’d create large areas, and then on a shot-by-shot basis, would sculpt scenes to play well for the camera and the depth of the scene,” Williams explains. “All of our show is done inside Maya, and everything in the jungle is 3D, so when you move the camera around in Maya, you get a real 3D sense.”

To light and render the massive jungle, Weta implemented two techniques: stochastic pruning and spherical harmonics. The stochastic pruning threw away unnecessary geometry on the fly as a plant moved away from camera. “It might take a fern with a million polygons and push it back to a few pixels when it’s in the distance,” Saindon says.


Spherical harmonics, a technique used for real-time rendering in video games, made it possible to light the rain forest. “Basically, we store coefficients for angles,” Saindon says. “We calculate the harmonics for each individual plant, all the lighting angles, and store that on the geometry. That allows us to drop simple lights into the scene and still get proper occlusion from each plant. The plant does its own self-occlusion using its own harmonics, seeing what should be occluding what, and stores the information. That means we can light an entire jungle with one light. We could get complex lighting with a very simple setup. We couldn’t have done the movie without it.”

Even so, the data processing requirements for the show were enormous. In addition to the characters, Weta created volumetric explosions, fireballs, 3D water simulations, and other effects. “Joe [Letteri] set down the hard line,” Williams says. “He told us not to plan on cheating anything.” At one point during postproduction, the studio was generating 110gb of data an hour.

“Jim Cameron’s expectations are extremely high, and he demands a lot,” Rosenbaum says. “The scope of CG movies is getting so large and the time constraints too tight, that people tend to compromise, but Jim doesn’t compromise. He insists on a high standard. When I worked on The Abyss, it took us six months to create 90 seconds with the pseudopod. We went into it with the same question we had on this film: How the hell will we do this? And we had the same mind-set: We’ll put our heads together and figure it out. He’s always one to push a VFX company. And he certainly did it on this one.”

http://www.cgw.com/Publications/CGW/2009/Volume-32-Issue-12-Dec-2009-/CG-In-Another-World.aspx

Avatar – on the Cutting Edge

The director of Terminator and Titanic explains how movies will be transformed by motion-tracking and 3D technology

Three-time Academy Award-winning director James Cameron is a pioneer in the field of motion capture. In the mid-’90s he used the nascent technology to create the massive crowd scenes and stunts in his blockbuster Titanic. These days he’s still at the cutting edge of the technology, but he prefers to call motion capture “performance capture” because, as he points out, “actors don’t do motion, they do emotion.”

Cameron is in the midst of his latest film project, Avatar, which is his most technologically innovative film to date. The futuristic movie about an ex-Marine will be released in 2009 simultaneously with a massive, multiplayer, video game based on the film.

Business Week couldn’t catch up to Cameron for a sit-down interview, since he’s busy creating Avatar, but reporter Aili Mc Connon was able to engage the director, via e-mail, in a discussion of how motion-capture technology has spurred innovation in cinema and made filmmaking more cost-effective. The following are excerpts from their virtual conversation:

What has motion capture meant to the film industry and to your work?

Performance capture (Perfcap) in recent years has enabled such stunning [computer generated] characters as Gollum (in Lord of the Rings parts 2 and 3), “King Kong,” and Davy Jones (in Pirates of the Caribbean) to be brought to life. The technology is critical to the realization of my dream project, Avatar.

In fact, Avatar wasn’t possible when it was first written 11 years ago, and only through pushing the technology to new levels over the past year and a half have we reached the point where the film is finally possible to make.

What innovations have you developed for Avatar?

We have greatly enhanced the size of the performance-capture stage, which we call The Volume, to six times the size previously used. And we have incorporated a real-time virtual camera, which allows me to direct [computer-generated] scenes as I would live-action scenes. I can see my actors performing as their characters, in real-time, and I can move my camera to adjust to their performances.

In addition, we have pioneered facial performance capture, in conjunction with our visual effects partner, Weta Digital. This technique eliminates hours in the makeup chair, and various other discomforts, for the actors. Previously, actors needed to have hundreds of tiny spherical markers glued to their faces, and they couldn’t touch their own faces throughout the shooting day as a result. With the new system, a lightweight head-rig can be donned minutes before shooting.

We have had great success, and other filmmakers such as Steven Spielberg and Peter Jackson have worked on our virtual stage doing tests for their upcoming films, and given high praise to the system.

Does the rig cover the whole head, including your face? Does it capture fine facial movements?

The rig is a small skull cap, made from a cast of the actor’s head, so that it fits comfortably while being tight enough to avoid shifting. It acts as a base for a strut which resembles a concert microphone (visualize Madonna in concert), except instead of a mike in front of the face, it has a tiny camera. The key to it is the software, which interprets the movement of the actor’s face, pupils, and eyelid responses as the image flows in from the video feed of the head-rig camera.

In what directions do you see the technology going in the short term?

Improvements to the software and higher computation speeds and storage densities will enable us to have more realistic environments and more refined facial emotions and hand movements. Hand movement, for example, is still at a crude state.

On Avatar, we’re working on-stage at a reality level equal to an ’80s video game. At the end of the day, after a year and a half of post production, the images seen by audiences will be 100% photo-real, i.e. indistinguishable from photography. But for our day-to-day shooting, the image can be improved a lot.

Another area which needs improvement is the lighting. We need to improve its ability to handle cinematic lighting, the casting of shadows and so on. All of this can be improved as Moore’s Law raises the speed of processing and as upgrades to the software become available.

In addition, we’re developing ways for [computer-generated] characters to interact with actors who are being photographed on real, live-action set. We will have real-time stereo (three-dimensional stereoscopic, or 3D) composites of characters, which will be viewed by me in the eyepiece of the camera while I’m shooting a live-action scene. This will be revolutionary. We’re not quite there yet, but we hope to have that by August, in time for our live-action shoot in Wellington, New Zealand.

Long term, what do you expect?

I expect that more filmmakers will embrace the technique and apply it to different types of scenarios. For the creation of fantasy and science-fiction characters, Perfcap will largely replace makeup and prosthetics.

Actors need not feel threatened by this change in technology. It doesn’t replace acting, in fact it’s designed to empower the acting and directing process, as opposed to the traditional [computer-generated] animation process, which uses only the actor’s voice, and in which a committee of animators perform the character, operate the camera, and do the lighting.

I believe it will make fantasy filmmaking much more user-friendly for filmmakers, actors, and studios, and ultimately bring down costs. It’s just now possible to create photo-real human [computer-generated] characters, but it isn’t cost effective.

Many other fields, from medicine to automotive design, now use similar motion-capture systems (though on a smaller scale). Do you ever run across or dream up non-entertainment applications yourself?

I’m bore-sighted on the cinematic process. While one can generally imagine all the industrial and science applications, I’m not interested in developing them. However I can visualize a number of uses for the technique in advanced forms of entertainment, at theme parks and so on.


What role will 3D play in the future of film?

Here’s what can happen, although it’s too early to say if it will: 3D can become ubiquitous as digital cinema replaces film. As digital cinema rolls out, stereo follows—and in some cases leads the charge, as we have seen recently with the digital 3D releases of Chicken Little and Monster House forcing the installation of hundreds of new digital projectors.

There will eventually be major titles available from all studios at some screens in almost all multiplex cinemas worldwide. I would say the ho

rizon for this is five years. 3D can become a fully accepted way in which audiences view movies. It will become another consumer choice, like premium or regular gas. The premium experience of 3D will be the preferred viewing experience for action, animated, fantasy, and science-fiction films.

3D’s broad acceptance at theaters will generate enough content that consumer-electronics manufacturers will make home players and monitors available. The technology exists now, but is not readily available as off-the-shelf products. 3D display will become a must for video and computer games.

In 20 years, stereo media may become the preferred method for displaying all information, including news and other broadcast media. The density of information one can place on a small screen becomes much higher if it’s stacked in three dimensions.

Is there something beyond 3D in film? Could we ever see in cinema the same kind of physical participation we’re starting to see in video-game consoles like Nintendo’s

Wii?

Imagine a movie in which the viewer is swept along by a narrative, following the action from place to place, but without the intervention of a camera. You can choose which character to watch in a scene, as if you’re an invisible witness standing there while a real event plays out. This is still years away, at a level of realism people would consider cinematic, but certainly not decades away.

I can imagine the dense fantasy worlds I like to create for movies having an equal or greater life in a world of interactive play, authored by others, in a partnership. Of course, add massive multiplayer capability to this, and people will never leave their homes.

Bill Desowitz speaks with Oscar-winning visual effects supervisor Joe Letteri about staying on the cutting edge of digital acting and 3D environments at Weta Digital.

After winning an Oscar for vfx on King Kong, Joe Letteri has remained at Weta Digital to supervise work on The Water Horse (Sony/Revolution, Dec. 7, 2007), including the CG sea creature, and Avatar, James Cameron’s long-awaited, first feature since Titanic. Letteri also discusses early work on the CG Silver Surfer from Fantastic Four: Rise of the Silver Surfer (Fox, June 16, 2007). Under the vfx supervision of Kevin Rafferty, Weta has reportedly enhanced its CG animation process that employs performance capture techniques to add further dimensionality to the liquid-metal hero performed by Doug Jones.

Bill Desowitz: Please fill us in on the state of the industry with regard to digital actors and 3D environments.

Joe Letteri: I think as far as digital actors go, in terms of characters rather than digital doubles, we’ve come pretty far. Especially with motion capture and facial motion capture, you can really work with an actor to develop the character. It’s an extension of what we did with Andy Serkis with Gollum and Kong. There is still a huge call for an animation team to work the actors and help develop the digital characters the rest of the way, because, typically, there are characters that are non-human and invariably there are things that a human actor cannot do. It’s too dangerous or physically not possible because of the configuration of the character. So you need this integration with the actors and the animators to pull everything together. I don’t see that going away anytime soon. I’m not sure you want it to go away because it’s really a great combination to have.

BD: In terms of techniques, how do you see it evolving between performance capture and keyframe?

JL: What we’ve always done here is to try to make the path as two-way as possible so we can start off with performance capture but then layer on the same set of tools that the animators are used to working with. So it’s always the call of the animators to make requested changes. Sometimes it sails straight through based on what the motion editors are doing. Other times, if you get a request to change the performance, you just have to decide at some point the data is too heavy and it’s going to be easier to reanimate it. The performance is a guide. But again, by having the data there, you’ve got a good starting point for the character and what you need to do next.

BD: What do you think of the various techniques that are available, including what Sony is doing with Imagemotion, what ILM did with Imocap on Dead Man’s Chest, Face Robot from Softimage and the new Contour from Mova that was introduced at SIGGRAPH?

JL: I think those are all great ways to go. They all bring a little bit of something to the toolkit that you have and each has its own strengths and weaknesses, so you can tune which technique you use to the situation. In the course of a large feature, you’re probably going to use several of those techniques. It’s great to see all of these things being developed in different ways to look at an important part of the problem for a particular task and try and solve that.

BD: I had a chance to ask Andy Serkis about performance capture recently and he thinks there will be a time in the future when a director can look through a viewfinder of a handheld camera and see in realtime physical and facial capture. Is this something you’re keen on?

JL: Yes, I think that would be a great thing to have to be able to work with the actor to get as much of his performance as possible. For example, on Kong and even with Gollum, we had a lot of the body stuff working in realtime. But looking at the facial, there was a problem of translating and learning the character, particularly with Kong. Yes, I would like to see that happen. One thing that it means, though, is that all that character development has to be done upfront if you’re going to do it all on stage with a director. We spent weeks and weeks with Andy on the motion capture stage for Kong and we got to digest all of that and turn that into his character. So it means shifting the way you’re doing things. It’s another one of those paradigms where what we used to call post-production pushes more and more into preproduction.

BD: How close are we to conquering “The Uncanny Valley?”

JL: You mean the real/not real, the human/not human? I think it depends on the application. Obviously we have seen digital doubles done really well using a lot of image-based techniques, for example, where you couldn’t tell the difference between the original performance of the actor and the digital performance. So from that point of view, it’s almost already been hit. But to take a human performance and have it actually be a human character? I don’t know. I’ve never been in that situation before. We usually just hire a real actor. We’re always looking at creating the other characters.

BD: And that includes Avatar? (The sci-fi film is about Jake, a paraplegic war veteran who is brought to another planet inhabited by a humanoid race at war with humans.)

JL: Well, we’re early days, so there could certainly be that coming up. To me it’s not the problem that’s most interesting [about the project]. So you’re only going to use that in those situations where it’s too dangerous for an actor. So if it does come up, I’m sure we’ll dig into it and tackle it. But the idea is to get a full-on performance that you can carry a movie with.

BD: What can you tell us about Avatar at this point?

JL: I think it’s going to be a combination of a lot of the things that we’ve been doing that you and I have been discussing. There’s performance capture techniques out there that really allow us to work in realtime with actors that allow us to develop the characters. And then just allowing the time to work with it after the fact, because the characters — no matter what you do on a performance capture stage — really come alive when you get them into the scenes and get them lit and rendered and see what they really look like and how they behave. And that always influences your perception of them. You start making adjustments to bits of the performance that you might need to do just to bring them alive. In a CG performance sometimes you need to add a little life, so you might add a little movement to the eyes. It’s stuff you start to define once you’re in there working with it that alters it in a subtle way. The final result is what you’re going to see on screen.

BD: What kind of toolset fine-tuning are you planning?

JL: We’ll be building on what we have here because we have a pretty robust toolset for character animation. Again, we’ll look at that a lot more once we see what the performances are and see what we need to do to bring it all alive.

BD: And does the added immersion with 3-D excite you?

JL: Yeah, the 3-D stuff looks really cool. We’ve been doing some tests for ourselves just internally. When you do everything in a 3-D world digitally, then you can play with different things and figure out what works, and start to answer the questions about how to watch a 3-D movie without getting a headache. We’re learning about how to do it properly and it’s been really good to chew through those problems.

BD: Moving on to the 3D environments, what are you looking to improve?

JL: Probably in general to use the same technique we used to create New York City in Kong [Maya-based software from Chris White dubbed “CityBot” rebuilt the city, floor-by-floor, section-by-section, block-by-block, adding intricate and period-accurate detail to the low-res dataset.] And extend it to build any type of environment, not just cityscapes. As you get more and more down the line, different locations that are called for are hard to reproduce. And it’s not just fantasy. It’s getting more and more difficult to do big scenes like New York in a real location. The amount of cleanup and replacement and set extension that you have to do dovetails into, “Gee, we just replaced that digitally — we could’ve done the whole thing that way.” So we’re looking at that for various types of environments to give us the freedom to answer that question.

BD: With Avatar, there was work done by Rob Legato in creating a virtual production studio in L.A. How does that fit in with your current plans?

JL: That’s still the basis of everything he’s doing and that dovetails nicely with the system that we’ve developed down here. We wound up using a lot of the same technologies and things. There’s not that much of a difference. Probably the real difference is one of approach. Rob has created an immersive system, which is necessary for Avatar. Whereas our system was designed mostly to work with things that had been shot on set and to add motion capture into those. We’re finding it very compatible, which allows us to keep a consistent workflow back and forth. We both use the Giant system for motion capture. The rest involves developing an infrastructure around that to support the film.
BD: What can you tell us about The Water Horse and the mysterious Scottish sea creature?

JL: We’re right in the middle of that… going through our first pass on the animation blocking and getting everything up to speed. Where it plays into what we’ve been discussing is in the area of character design and character performance. The main character is really fun to work with and it’s been great to continue what we did [with Kong].

BD: And how is Fantastic Four 2 going?

JL: We’re still early days on that but the obvious question is you want something with a cool silver look to it. We’ve been coming up with some things on that. So that’s more like taking what they’re doing onset and use that to drive the Silver Surfer.

Bill Desowitz is editor of VFXWorld.

In this initial feature for the launch of SIGGRAPH Quarterly’s online magazine, Sony Pictures Imageworks’ Rob Engle and Rob Bredow discuss the subject of stereoscopic 3D film production and presentation, and offer their ideas as to where this increasingly important technology may be heading in the future.

Article author: Eden Ashley Umble All images courtesy of Sony Pictures Imagesworks unless otherwise stated
Combined left & right eye final shot – IMAX
Left eye camera render	Combined left & right eye camera render	Right eye camera render

LCD shutter glasses (active)

Films using a variety of 3D technologies – from 3D animation to stereoscopic Real-D and IMAX 3D presentation – have grown in popularity with both audiences and filmmakers in recent years. For audiences, the 3D experience can provide thrills that are visceral as well as visual, while filmmakers are using this technology to tell their stories in a way that is more immediate, more detailed, more real than ever before, allowing them to push the boundaries of filmmaking to the limits of their imaginations. In this initial feature for the launch of SIGGRAPH Quarterly online magazine, Sony Pictures Imageworks’ Rob Engle and Rob Bredow discuss the subject of stereoscopic 3D film production and presentation, and offer their ideas as to where this increasingly important technology may be heading in the future.

Red & cyan filtered glasses

Crew: Sony Pictures Imageworks’ IMAX 3D artists and staff
Engle was the Digital Effects Supervisor on the groundbreaking adaptation of Robert Zemeckis’ theatrical 2D film “The Polar Express” for IMAX 3D, which was the first feature-length all-CG project to be created in stereoscopic 3D (and at 96 minutes, the longest IMAX 3D film ever made). A separate team of 60 artists and support staff worked for 6 months to create the IMAX 3D version, all while the 2D film was still being finished. The IMAX 3D version of “The Polar Express” opened on November 10, 2005, the same day as the theatrical feature, winning critical acclaim and setting box office records for an IMAX attraction, grossing $35 million domestically on just 60 IMAX screens. Bredow was a Digital Effects Supervisor on the theatrical release of the film and was instrumental in the early phases of testing the viability of the IMAX 3D project.
Rob Engle	Engle is presently supervising the IMAX 3D version of “Monster House”, which will be released using the Real-D stereoscopic projection system. The second film to employ the Imagemotion(TM) performance capture innovation developed by Sony Pictures Imageworks, “Monster House”, directed by Gil Kenan, will be released July 21, 2006. Bredow has been promoted to Visual Effects Supervisor on the CG animated feature “Surf’s Up”, the second film being produced by Sony Pictures Animation. “Surf’s Up” is directed by Ash Brannon (“Toy Story 2″) and Chris Buck (“Tarzan”) and will be released in June 2007.	Rob Bredow
The demand for stereoscopic 3D films is growing. What 3D projects are currently in production at Sony Pictures Imageworks? Rob Engle: We have “Monster House” in REAL-D [set for release July 21st, 2006] and “Open Season”, set for release Sept. 29, 2006] as an IMAX 3D film. Rob Bredow: Basically, I don’t think there’s a project in house where 3D isn’t discussed. There are thoughts about just about every one of our shows because of the renewed interest in 3D across the board. I think this is certainly in part because of the success of the IMAX 3D “The Polar Express” . It’s becoming something that’s on everybody’s mind for every one of our projects, particularly our CG features; it looks like every one of our CG features could possibly have a 3D version. I think it’s interesting to look at the different marketing angles that have yet to be explored, in terms of having a 3D version of your movie in combination with your 2D version, and how you release those. Day and date is how we’ve been delivering so far and that’s cool because you get great word of mouth on the 3D version since it’s a phenomenal experience, and then the nice thing is, if the IMAX theater is sold out, they can still go see it in the 2D theater. In the case of “The Polar Express”, by all accounts the 3D version added significantly to revenues. How are Stereoscopic 3D films perceived by our eyes? Engle: There are really 2 phenomena that the eyes use to perceive depth. One is where they focus, meaning specifically at what depth our individual eyes are adjusted to see, just like a camera lens focuses. The other aspect is called convergence, which is basically the phenomena where your eyes cross or uncross in order to bring two similar features in an image together. You are constantly adjusting your convergence and your focus to tell you the relative depth of objects. That’s how we see things in the real world. In stereoscopic films, where they’re projected flat on a screen, your brain is being asked to separate those two phenomena. It’s being asked to focus on a fixed point, which is usually 20 to 30 feet away, and then converge independently.
Imageworks’ crew
How does stereoscopic film projection differ from conventional film presentation? Engle: Normally, we would render a single camera point of view and project that and that’s how you see a CG film in a theater. To create a stereoscopic movie we render two viewpoints and project both of those viewpoints simultaneously. What’s important is that the projection system is capable of delivering an independent image to the left eye from the right eye. There are a wide variety of technologies out there for achieving this, and almost all of them use glasses in one form or another. In active systems the viewer wears a set of glasses that have electronically triggered filters over each eye. The filters switch between opaque and transparent in synch with a projector that is alternating between the left eye and the right eye. In passive systems the viewer wears glasses with fixed filters, which have some unique property per-eye that selects the image to pass. For example, with the anaglyph system a red filter allows only one color light to one eye while a cyan filter allows the rest of the light to the other eye. The more sophisticated IMAX and Real-D systems use color-neutral polarizing filters to select the left and right eye images. There are also some systems out there that are called autostereoscopic displays where you don’t have to wear the glasses at all. Do stereoscopic 3D projection systems use one or two projectors? Engle: It depends on the system you’re talking about, but it can be either way. For an active system (shutter glasses) there is almost always only one projector which is alternating between the left and right eye images. The Real-D system uses a single projector but places an active filter, which can switch between two different types of polarization, in front of the projection lens. This allows the viewer to wear much simpler and more lightweight passive glasses. The IMAX system uses a two projector system where each lens has a fixed polarizing filter over it. They integrate the two film transports and lenses with one lamp house which makes it look like one GIANT projector. Older 35mm projection single-projector systems used a split lens which would polarize the top half of the frame one way and the bottom half another and then superimpose the two halves on the screen. The primary issues that help decide which system to use are synchronization of the left and right eyes, the amount of light that needs to reach the screen, and cost. Synchronization has become less of an issue now that we have digital cinema but was a real problem for two-projector 35mm systems. The reason light output is an issue is that the extra filters added to the system can cause a significant reduction in light level. A two projector system will cost more but produce more light. One advantage of the Real-D system is that it allows theater owners to upgrade for stereo presentation without buying another projector.

How do the experiences of viewing IMAX and Real-D differ? Engle: I think the biggest difference is that when you’re in an IMAX theater, you’re usually immersed in the screen without even looking at any content. Once you sit down, it takes a good turn of the head to look from one end of the screen to the other. The result of that is that you generally feel like you’re in the image on an IMAX screen, more so than you are in a conventional theatre screen. The Real-D system currently is targeted at multiplex type theaters with 40 to 50 feet wide screens where you can see the edges. What that means in terms of the experience for the audience and for the way in which you create the content is that an IMAX theater can be much more immersive, and of course, that’s why they call it “the IMAX 3D experience”. Contrast that with a multiplex theater, where it’s literally as if you were looking through a window and experiencing a deep world. We’re capable of pushing things out of the screen but that effect really depends on how things are composed. Fundamentally, IMAX will feel like you’re more in the world, and multiplex Real-D will feel like you’re watching the world. Both IMAX and Real-D offer compelling 3D experiences for their audiences. As co-creators, Imageworks is always trying to find the best way to match the director’s vision to the best use of stereoscopic presentation. Sometimes that will mean IMAX and sometimes it will mean Real-D.

Cross section view of a typical IMAX theater

Representation of overall size of IMAX screen compared to the viewers’ field of vision

How was the IMAX 3D conversion process accomplished on “The Polar Express“? Engle: For the 2D theatrical release of the movie, [the main Imageworks filmmaking team] would produce the content in the first place, and they did that though a process which was basically a combination of Imagemotion, performance capture and hand animation to produce the final shot. The 3D team would pick it up from there. We would start by looking at their final animation files and we would produce a stereo camera that matched their camera as much as possible. We were trying to preserve their movie without changing the composition of their shots if we didn’t have to. We wanted to be as faithful to the original as possible. We were unsure if people could sit through an hour and a half of stereoscopic material without getting headaches, so we were very careful about trying to produce something that would be comfortable. Once we approved the camera, we had a team of people whose job was basically to resurrect how the original shot was produced, and reproduce it, but for two eyes. And it wasn’t simply a matter of taking the original movie and using that say, as the left eye, because one of the other things that was very important to give a better 3D experience was doing things like dialing back the use of depth of field in shots. Bredow: On “The Polar Express”, which was the first all-CG feature created at Imageworks, we had our hands full figuring out how to do the 2D version of the movie. When the 3D version came along, initially it was extremely daunting because we were just up to our eyeballs in work. And one of the things we discovered was, in fact, we could do it, based on setting up two teams, and relying on some of the core pipeline technology that Imageworks has built over a long period of time: the way the data for a shot gets recorded, so another team could come along and re-create the shots. It was technically possible. That was one kind of exciting thing we learned. At the same time, we walked away learning that if you know in advance you’re going to do a stereo version of your film, you can set it up a whole lot better in advance. That’s one of the things, going forward, we’re looking at all of our movies, saying, ‘Are we set up to do this in 3D in an efficient way, if the client requests going day and date with both versions?’ What sort of challenges does stereoscopic 3D presentation pose to faithfully re-creating a filmmaker’s vision? Engle: Most filmmakers will use depth of field to try to direct the viewer to look at a specific object, but if the viewer wants to have a true 3D experience, then depth of field is actually your enemy in that respect. On [the IMAX 3D version of] “The Polar Express”, we would go in and dial back those kinds of effects, we would adjust things like transparency that can be confusing when looking at a stereo image, we would adjust things here and there, but basically produce something that was faithful to the original and was just stereo. [One particular asset of] a Zemeckis movie, is on average, he likes to use very long shots. The average length of a shot on “The Polar Express” was something like 7 seconds, which in this kind of MTV, commercial world is almost unheard of. Usually shots are on the order of 3 seconds, and it turns out that that a longer shot really works to your advantage in a 3D film, because it allows people’s eyes to grow accustomed to whatever’s in the shot before you yank it out from underneath them and switch to a different shot. When Imageworks adapts a film for stereoscopic presentation, we very carefully work so that each shot (and thus, the movie as a whole) is the best 3D it can be. We cut very few corners… In most cases we re-render every element to ensure that the shot has the most detail possible. The result is a stereoscopic experience that is very rich and (hopefully) gives the audience what they came to see… A motion picture experience unlike anything they have seen before.

“Polar Express“ – Sony Pictures Imageworks (combo image)

Can you talk about working with a filmmaker as creative as Robert Zemeckis? Bredow: It’s interesting to get to work with somebody who’s obviously experimental and innovative in filmmaking. Basically, when you get to work with someone like Robert Zemeckis, his focus is to be able to tell stories with whatever means are available to him. When he sees an actor like Tom Hanks, and wants him to play a six year-old kid, you start to realize the extremes that he’s interested in going, to be able to tell his stories, which makes it a lot of fun. That starts with things like acting and characters and who’s playing his main characters, and goes all the way to technical innovations in terms of how to make the audience experience his movie firsthand in 3D. That was my experience on “The Polar Express”. Are there any striking differences between the pipelines created at Imageworks to process IMAX and Real-D? Engle: The primary difference between the Real-D and IMAX pipelines are in the ways the cameras are created. With a Real-D presentation you need to be more aware of how the edge of the screen can interfere with the 3D effect. You need to adjust the overall depth of the scene into the screen plane. Can you talk about the look and production of “Monster House“ in Real-D?

“Monster House“ from Columbia Pictures will be presented in Real-D

Engle: I think “Monster House” is a unique creature in respect to its look. The look of the film is very much like a Claymation miniature, and there’s a lot of use of global illumination, bounce lighting and very intricate shadow detail that wasn’t used as much on “The Polar Express”. It’s a very different look. In the Real-D world, in a multiplex, it’s more of a window environment, and we’re trying to direct that much more carefully on “Monster House” than we did on “Polar”. We’e using a different renderer and a different lighting package on this show, so behind the scenes there’s a lot going on to make sure we can do this show, but the basic concepts are the same as “The Polar Express”. Should all films be in stereoscopic 3D, or do some films possess characteristics that specifically call for a 3D viewing experience? Engle: I think that CG features have a special quality which lend themselves to stereoscopic presentation I look forward to seeing a few live action blockbusters in stereo but do we really want to see every film in 3D? Maybe I’m too old, but I saw a clip of a classic movie musical not too long ago which had been converted to 3D and I thought to myself, yeah that’s neat, but there’s something to be said for leaving those historical gems alone. Maybe it comes back to the whole question of colorizing a film, for example, do you do it or not? If you did a version of “Casablanca” that was in color and 3D, it would just be a different movie; I mean, why bother? I’m generally against it. I would much rather leave it alone and let people enjoy it for what it was. I suppose there’s the other argument that says if you make it 3D or in color, then it reaches a new audience that wasn’t there before, but I like to think that people are cleverer than that. Bredow: I think “The Polar Express” was an example of a movie that was particularly well suited for 3D. It was “stereo friendly” for a lot of reasons. Robert Zemeckis loves to move the camera, he loves to use really wide camera angles, long shots, and all those things are great for a 3D movie. I think that’s one of the reasons it was such a good showcase piece for this kind of film. Not all films are going to be as well suited as “The Polar Express”. Interestingly, I think CG features generally have a better shot than your average live action movie, just because of the way that they tend to be cut; they tend to not move at the same sort of pace of some of the more fast-cutting live action movies. On [the IMAX 3D version of] “The Polar Express”, what was so successful for me personally when I went to the movie theater and watched it, was that it was so immersive. By the time the movie had come out I had seen the 2D version a lot of times, but seeing the 3D version was honestly like seeing another movie. There’s something so immersive about that, especially when you’ve got the opportunity to do stereo and a huge screen. When you can fill the audience’s peripheral vision, it really does do something different in terms of putting them inside the movie, which is fun. How do you see 3D technology being applied to everyday communication and other consumer applications in the future? Engle: I certainly think the technology is moving forward to the point where we will have autostereoscopic displays in the consumer’s hands. Right now, they’re very expensive, but it will happen. The most interesting question in my mind is whether or not we can make use of the third dimension to make computer user interfaces more accessible. Imagine if the desktop on your computer was actually dimensional. Would it be much more cluttered, or would it be better organized? I don’t know, but I certainly think if people haven’t experimented with it, they should be. As far as how it’s going to change communication, I wouldn’t be surprised if someday maybe everybody will have a stereo camera phone and you can have a stereo telephone conferencing. I don’t know, but I certainly think that 3D will get better and cheaper, and as a result, it’ll be everywhere. Bredow: I think what everybody thinks of when they think of 3D, of course, is the glasses and the hassle that they can be. Anytime you have to put glasses on to carry on a conference call or something like that, I think that that’s something that people are not going to choose to do very often. There were a lot of different things in SIGGRAPH this year, various monitors that did some sort of 3D experience, like a single monitor without glasses, and actually all of them had the disadvantage of they weren’t very sharp or very detailed, but it was interesting to see various prototypes. Most of them were showing short little test animations, or things that had been acquired in 3D, or still images. When you can perfect the idea of not having to put on glasses, I think you’re going to see more broad application. In terms of integrating it into other areas of people’s lives, it’s a good question. And the simple matter is, it costs at least twice as much as your standard projection setup, just because you’re going to have two projectors instead of one, in most setups. Can you imagine, for example, people buying Real D glasses in the same way they purchase their reading glasses now?

Engle: It’s funny that you mention that, because I’m always joking with our digital production manager that I want to get polarized contact lenses, so I don’t have to take my glasses off. On the future of 3D as a growing visual effects process, can you see Real-D technology being used in a classroom application soon? Engle: Absolutely. I think not necessarily specifically Real-D, as much as the more it becomes commonplace, I can certainly see it being used in the classroom. What I find interesting is the use of the technology to produce compelling content, for example, imagine National Geographic specials shot in stereo where you feel like you’re actually in the lion’s den, as opposed to just seeing it. I personally think that would be amazing. Of course, if you’ve seen any 3D IMAX films, you know that you feel like you’re there.

Bredow: It’s not impossible, but just generating the content for stereo rather than 2D is easily twice the work so when you’re talking about a Power Point presentation or something like that, there has to be a really specific reason that you’d need to present it in stereo just to get your point across. It may not be worth the time. Just having, for instance, fonts floating over a background probably wouldn’t be worth the effort, whereas if you’re trying to describe something that’s inherently 3D, then that could have some payoff, for sure. Real-D specifically is a very high-end theater based system. The projector’s the size of a normal film projector and the cost of course is up there too. But the basic concepts behind the technology which are similar between many of the different 3D options, with polarized lenses, 2 projectors with different sort of polarization and a special screen — there’s nothing keeping anybody from setting that up with a few thousand dollars’ worth of hardware in a classroom setting and very cheap polarized glasses. It’s technically feasible. How practical is it in terms of generating the content, that’s probably the biggest question. Do you think stereoscopic 3D technology will soon be a viable addition to current curriculum for digital art and animation students? Bredow:With the trend of a lot more movies going into 3D, there are good opportunities from the education side. It would be great to have more education about the way stereo works, the way our eyes perceive 3D, and the way you can trick the eye with various techniques, [such as] whether you aim their cameras toward a focal point, how you handle depth of field, etc. There’s lots of different ways of thinking about these kinds of things, and there’s some good research out there, but I don’t think there are lots of people covering that currently in schools because the recent popularity of the medium is a relatively new thing. 3D in general is an interesting topic for people to explore. Another
interesting opportunity from the educational perspective, and for people just getting into the industry, is that often the 2D movie is done by a team of people and then the 3D team comes along and picks up the assets and makes the stereo version. The first time that happened to us, it was simply by necessity of the schedule. We had our hands full making the 2D version of “The Polar Express” and late in the schedule, they decided to make the 3D version, so we started a whole other team to do it. It turned out that that was actually a pretty efficient way to work. One of the good opportunities for people from an educational standpoint is there’s a higher number of positions that require less experience across the entire board, who can work on some of these shows. [A stereoscopic 3D film] can be one of the first shows that people might get to work on who don’t have the traditional 5 years’ experience in feature film. So, one nice thing about these shows is they do create some entry level positions. Engle: Absolutely. In general, with the introduction of any new technology there is the need to train people on the best way to use it. Creating films for stereoscopic presentation is not a new field but, with the wider availability of 3D venues, there will be a stronger demand for good 3D content.
	About the author: Eden Ashley Umble is a writer based in Northern California, where she lives with her husband and two kids. She worked in film production and publicity for fifteen years on films such as “Edward Scissorhands”, “The Long Kiss Goodnight” and “Fat Man and Little Boy.”

http://www.massivesoftware.com

Andy Serkis is the reigning master of performance for motion-capture — the recording of an actor’s every move and facial nuance for use by animators to enliven CG characters. In his acclaimed star turns as the ring-addicted Gollum in the Lord of the Rings trilogy and the noble mega-ape in King Kong, 43-year-old Serkis invested his digital roles with the power of old-school stagecraft at its best. The London-based actor has also recently ported his skills to the gaming world, appearing in the new PlayStation 3 title Heavenly Sword, which he co-produced. Now that even Angelina Jolie is getting in on the sensors-and-greenscreen action — for Robert Zemeckis’ upcoming take on Beowulf — Wired spoke to Serkis about digital acting, the future of mo-cap, and why Gollum didn’t score an Oscar.

Wired: What tips would you offer an actor doing motion-capture for the first time?

Serkis: For digital roles, the actor is manipulating their character like a puppet. It’s really useful to have time on a monitor to work with the CG model — to play around with your puppet before the actual shoot. It’s like having a third eye on yourself. Actors have to learn to demand that time.

Wired: Did your role in Heavenly Sword expand over time, as in The Lord of the Rings?

Serkis: I got more and more involved in the character development and the writing. When I took the actors down to New Zealand to rehearse, we sat in a circle and performed the whole game, from beginning to end, as a play for each other. By treating it as theater, we could see how all the characters were inter related, figure out where scenes weren’t working, and feel the whole arc.

Wired: Were you already an avid gamer when you took the gig?

Serkis: No, but Heavenly Sword got me much more into it. I’m not bothered by hack-and-slash games, but what I really enjoy is being taken on a journey to other realities. I have a strong desire to create games from Shakespeare — play as Romeo, play as Juliet. Macbeth is an amazing story. Maybe I should be keeping these ideas to myself (laughing). One thing that’s going to change in the next few years is that scripts for games are going to come more from the dramatic arena. They’ll be more like film scripts. You can’t just come up with an idea for a game and stick the drama on top. It all has to be one driving thrust.

Wired: While you were growing up, you spent a lot of time in the Middle East.

Serkis: My father’s Iraqi — he’s a doctor, retired now. My mum moved me and my older sisters to London when I was a year old, but my father still had a practice in Iraq. I stayed in Baghdad every summer until I was 14. My dad’s sister is still there, but many of my relatives have managed to get out. People forget that there are still people there who are not radicalized in any particular direction, trying to live normal lives in a very difficult situation.

Wired: What experiences in your early acting career prepared you to do motion-capture for Gollum and Kong?

Serkis: My first job when I got my equity card was acting in 14 plays back-to-back. Playing that many roles, you look for ways of differentiating the characters physically, which goes hand in hand with understanding them psychologically. In 1992, I played a homeless kid called Dogboy in a play at the Royal Court Theatre called Hush. When his dog is killed, he allows the creature’s spirit to possess him, and he breaks into this middle-class household to avenge his spirit. I was naked for the entire performance. There was a lot of Dogboy in Gollum.

Wired: Were you surprised at how much input you ended up having on The Lord of the Rings?

Serkis: It was very much an organic process. I got a call from my agent who said, “They’re looking for someone to do a voice for a completely digital character. It’s going to be three weeks’ work.” But then I met Peter Jackson, and he said, “No, we’re looking for someone to be Gollum on set, because we want real chemistry with the other actors.” I learned that the only way that I could generate Gollum’s voice was by fully inhabiting the character.

My first day, I was climbing down the side of a 6,000-foot volcano in a Lycra suit, and the crew was like, “We thought Gollum was going to be animated. Who the hell is this guy who looks like he just walked out of a fetish shop?” That was terrifying. But as everything came together — the motion-capture, rotoscoping, animation, voice and breath work — the process became very exciting. Nothing like it had ever really been done before.

Wired: Have years of cyber-acting changed your approach to stage acting?

Serkis: It’s made me more still. My natural bent is to have an overabundance of energy, and motion-capture essentializes your every breath, your every move. Seeing yourself through that mask, you realize how far you can pull back and make the performance even more powerful.

Wired: What’s on your wish list as a digital actor?

Serkis: The environment you’re working in for performance-capture is very clinical. There’s no stimulation from sets or costumes; you’re working in a black box with lots of lights around you. I want to be able to shoot a scene in costume instead of a Lycra suit. We need motion-capture studios that let directors use lighting, back projection and other forms of stimulation to help the actors feel immersed in the world of the film.

Wired: What projects are you working on now?

Serkis: I’m in the early stages of a film called Freezing Time about Eadweard Muybridge, the Victorian photographer who was really the forefather of cinema. Digital animators still treat his images like the Bible. He was a very obsessed man. He tried to have a relationship with his wife, but it wasn’t fully consummated, so she ended up having an affair with this dashing guy called Harry Larkins. Muybridge shot him dead in a fit of jealousy but was acquitted because the murder was considered a crime of passion.

I’m also working on a movie called Inkheart with Brendan Fraser and Helen Mirren based on a book by Cornelia Funke, who is like the German J. K. Rowling. It’s about an antiquarian bookbinder who has the ability to “read” characters out of books. I play a very dark character called Capricorn who is accidentally read out of a book and doesn’t want to go back in. Then the bookbinder’s wife falls in. That’s coming out next year.

Wired: You got dissed by the Academy because Gollum was considered a collaboration with the animators at Weta Digital. Will a CG character ever win an Oscar?

Serkis: For The Elephant Man, a whole team of prosthetics artists worked on John Hurt’s character to help him create that performance. Whether or not the Academy can learn to see ones and naughts as a digital form of prosthetics — that is the question.

http://www.wired.com/entertainment/hollywood/magazine/15-10/pl_serkis

At TEDIndia, Pranav Mistry demos several tools that help the physical world interact with the world of data — including a deep look at his SixthSense device and a new, paradigm-shifting paper “laptop.” In an onstage Q&A, Mistry says he’ll open-source the software behind SixthSense, to open its possibilities to all.

http://www.ted.com/talks/pranav_mistry_the_thrilling_potential_of_sixthsense_technology.html

http://www.ted.com/talks/ed_ulbrich_shows_how_benjamin_button_got_his_face.html

The digital acting in King Kong was a huge leap forward, because it was the first such performance that really brought emotional weight. © 2005 Universal Studios.

Several years ago, I got to spend some quality time alone with Ray Harryhausen. It was only about half an hour, but I count it as a career highlight. Although I don’t recall his exact words, Harryhausen told me that he always had a deep commitment to the animated performance, believing that a great one was as engaging and emotionally telling as life itself… or at least a great human performance. The thing that convinced me was his work on Mighty Joe Young. I know that he’s right – it’s not just movement; it’s performance.

Following in Harryhausen’s footsteps is Dr. Mark Sagar, who was responsible for much of the wonderful facial animation in Peter Jackson’s King Kong. Sagar too has a commitment to performance. I’ve been fairly outspoken about Kong’s MoCap performance being just awful. He looks like a 4,000-pound gorilla bouncing around Times Square like a 190-pound man. It just doesn’t sell. But the face was a different matter. I said to myself, “Kong’s face doesn’t look MoCapped.” MoCap in general still has a “look” that I’d rather not see. Kong’s face — and all the other truly great facial performances so far — arise mostly from the brilliance of the individual master character animator(s). Each subtle movement of the face imbued with the loving touch of a dedicated and talented artist. I understand that as much as 75% of Kong’s face was keyframed, and I think it made the movie. But keyframing is an expensive, time consuming luxury and it’s all changing.

Perhaps the finest example ever of this hybrid approach with MoCap tracking being used with keyframe animation is found in Pirates of the Caribbean: Dead Man’s Chest. I believe ILM has crossed the valley into the land of believable virtual human performance, but more about this later on.

With the unprecedented popularity of animated performances these days, we need a way increase production, believability and magic. When it takes the delicate hand of a great character animator – and they are few and far between – you’re not going to see useful increases in output. It certainly would not be possible to keep up with even current demand and tight schedules. So we end up with what happens all too often these days, inconsistent animation of the same character. Shrek, as successful as it was, went from great (for example, the eye animation on Princess Fiona) to mediocre at best, with inconsistencies within characters. Shrek is not alone: the Star Wars franchise has had its uncanny digital actors as well. Not to pick on them, but nearly all virtual performances in features have been inconsistent because of the number of people working at different levels on them. But they were entertaining and they set the ball in motion. We now need better virtual performances. Imagine going back to the old Felix the Cat animations: you might still find them entertaining, but the animation would likely be annoying…it is to me.

Taking a large portion of the responsibility for life-like performance out of the hands of lower level animators is changing the business big time. In a way, it’s sad. Many animators are developing a whole new set of skills. It actually takes a different, more technical type of person to be a character animator these days. Some of the great artistry is being lost, and I, for one, think that artistry can never be fully replaced by technology. I call these newbies tech-animators. Most of the one’s I’ve talked with recently truly have hand animation talent, but they’ve been trained to use technology to power through a ton of shots quickly. I believe in the near future, this new breed of tech-animator will evolve into a population with little or no knowledge of traditional animation, digital or otherwise. They will know the technology and how to use it to create life-like performances. Thus we are bound see a decline in animated character performances a la Chuck Jones et al.

With the demand putting pressure on producers, it’s necessary that we move on. I’m thinking there will be a place for master character animators for a long time in tweaking Hero characters to give them that special animated personality look, that’s bigger and more engaging than any recorded performance. But I’ve already seen a small army of tech-animators on the rise.

Getting back to technology, though, there are three basic approaches that I’m looking at, and all involve MoCap. First, the long standing standard MoCap approach and its newest refinements that is making capture more accurate. The second uses fewer capture markers to give the general performance, with the detailed nuances created by underlying virtual face structure. The third is the use of virtual sensory perception and Artificial Intelligence (AI) to automatically create behavior streams on-the-fly using MoCap libraries and behavior blending.

Where Kong wowed us with one powerful performance, Narnia was able to bring to life a whole world of expressive characters. © Disney Enterprises Inc. and Walden Media Llc. All rights reserved.

A Time Of Transition
Sony Pictures Imageworks is one of the most respected vfx/animation, and I think they have a great attitude about virtual performance. They’ve been responsible for some of the wonderful virtual performance work done in movies such as Narnia and Monster House and many more. Imageworks seems to believe in both the power of MoCap and magic of talented people. It’s one place where you’ll find a lot of talented animators who are working hand-in-hand with new technology. According to Debbie Denise, evp of production infrastructure/exec producer at Sony Pictures Imageworks, the studio’s basic philosophy toward animation and character performance strongly impacts their R&D and staffing efforts. It may sound cheesy, but it’s that philosophy that makes all the difference in what you see on screen.

“Imageworks’ approach and philosophy is that through motion capture and animation tools, we try to preserve the essence of what actors and directors bring to the performance of a character. This applies to body and/or facial capture. To this end, we are always trying to improve our ability to get that performance on screen as faithfully as possible, taking into account the design and feel of the animated character.”

I asked if she thought all the equipment and greenscreen claptrap hampered performance at all: “We feel that it’s critical to allow the performer and the director to be unimpeded by the technology we deploy to capture the data. That’s why our R&D group is working hard to develop new and better ways to capture the data with less invasive technology and in creative environments.” Naturally, I wasn’t able to get many details of the actual R&D efforts or how next year’s Beowulf improves the process. However, this is what she offered: “As you know, there are several ways to capture a performance. Generally we go about getting the character’s skin to animate by tracking drive points on the mesh skin surface directly with the markers (on the actors) and another approach is to develop software that analyzes the movement of the markers to trigger muscles or shapes to deform the surface.

“Our current solution uses both of these techniques. We work directly with the data in an extremely efficient way, and animators can alter it to reflect any changes in dialogue, sight lines or exaggerated or toned down movement. And we can choose the degree of application of each method in all shots. This seems to us to be the best of both worlds.

“We are also working on methodologies that would give us ways to capture the facial performance with as much data as we currently capture, but with few or no markers on the performer… in all types of environments or stages… Not too much to ask!”

I knew that Imageworks has long taken character animation seriously on many levels. I asked Denise to comment in general on the artistry vs. technology of their philosophy: “No matter how great the technology is, the animators are still the ones who turn the data into magic. To that end, we build “animator friendly” tools that allow the animators to enhance the data intuitively, rather than through overly complex user-interfaces.” She stressed that for Sony, it’s as much about art as it is about technology.

The Magic of MoCap
Until recently, the MoCap approach has required actors to don shiny little balls or other markers that could be tracked by multiples special video cameras, within a confined space. In some cases, it still does, but as demands for accuracy and comfort are heard, some new trends are emerging. Recent photometric approaches involve tracking facial characteristics in new ways. This sometimes involves the actor having their face done up in strange makeup in lieu of markers. These new approaches claim higher data density, accuracy and comfort for the performance actors.

With MoCap, as you probably know, human actors perform the character’s lines and business while it’s captured in video and translated to a flow of mathematical motion data. The data is filtered and interpreted by 3D animation software on a point-by-point basis using standard plug-ins or proprietary pipeline elements. Essentially the cleaned motion data is used to manipulate the 3D character mesh. As the actor’s face or body moves, so goes the 3D mesh doppelganger. In the best productions, master character animators tweak the performance before final render and compositing so that it has character that can’t yet be captured. The results can be spectacular.

The biggest problem with this approach has been capturing the subtleties of the face, especially the eyes. It’s often impractical to get a master face animator to do the magic necessary for a believable performance. Polar Express is an example of how disturbing inappropriate eye movement can be. I’m happy to say that ILM did a spectacular job of eye capture in Dead Man’s Chest.

One of the most effective refinements of the standard MoCap approach comes in the form of “Stretchmark,” a software system being developed by Pendulum. It’s designed to use high-definition MoCap data in an innovative way to produce highly realistic character performances.

Artist/animator/engineer Robert Taylor is a co-owner of Pendulum, an animation studio down in San Diego. Taylor recognizes the need for more and better character animation in ever-shorter time frames. “We don’t use a model of underlying muscle systems because I’ve seen that tried and so far I haven’t been so impressed. We take a more global approach using the captured data stream to control a custom set of blend-shapes. We did a lot of trial and error work to come up with about 45 base blend-shapes for the human face mesh. This base group can be blended to reproduce a huge variety of emotional expression.

Studio Pendulum’s Mark Anthony character displays the results of the Stretchmark system. © Studio Pendulum.

“We like a lot of data; for example, Mark Anthony (an impressive demonstration of concept) had 90 data points on the face. The work was done at House of Moves using the latest Vicon system. First, we apply a data fit to the mesh using our Stretchmark software and then run the Cap data to see how it goes. It’s never ideal, so we then we apply what we call a corrective ID. The base blend-shapes cover most of the performance, but we will see breaks in the performance because the base blend-shapes couldn’t cover a particular expression. Our sculptor then creates any needed custom blend-shapes to cover those. It’s a quick process taking only about a half-day or so. With the new shapes the performance will be smooth and complete. We have a lot of tweaking tools that we can use to then customize the animation. We can also control the weighting or influence each point has on the mesh. We can add multipliers to get cartoonish behavior, or we can even modify the final performance using puppeteering tools or hand animation to tweak the performance. We set it up that way because we’re animators, and we love to tweak the final performance.”

Some New Approaches
MoCap is the basis for all the tech-animation approaches. One interesting and effective new approach is Face Robot, which is available from Softimage. It’s a software approach that sits on top of Softimage XSI, bringing you that remarkable set of tools, making it a full-face animation system. You can do everything from sculpting to final render with this setup. “Face Robot is a high-end stand alone product that incorporates Softimage within it,” explains special projects manager Michael Isner. “Our solver has two inputs: the MoCap, and our soft tissue model. This is the first face performance software that contributes to the performance. Our soft tissue model is tweaked in cooperation with the director and art director’s input, to reflect what they want from the face movements. It’s an artistic process. The artist works with what I describe as ‘soft IK in a jelly fish.’ We use a Wizard to create the core model of the face and how that underlying jellyfish will behave. Then we attach the MoCap data to specific points. The beauty of it is that we use a small number of markers because all the in-between portions of the face move in a life-like way on their own. They’re not locked into a hard set of rules nor are they dependent on static morph targets. The facial expressions are more dynamic, more life-like than you get by any other means.”

In a sense, the artist creates a face personality that will create a look and performance that is original, though based upon a face actor’s performance. I’ve seen some early attempts at using underlying muscle structure etc. to help create emotional expressions before, but none of them have been very impressive. It’s the uncanny look of slightly inappropriate movement that kills it. Face Robot doesn’t have it down perfectly yet, but it is certainly a unique approach that is capable of giving fine performances and saving a lot of money in the long run.

Virtual Acting: The Innovations Are Real

Image Metrics’ most high profile work can be seen in Warner Bros.’ Polar Express.

Twice Negative
One of the most technologically cutting edge vfx and animation houses in the U.K. is Double Negative. I spoke with Paul Franklin about the DoubleNeg character animation pipeline. “Over the past couple of years Double Negative has been developing a new facial motion capture system. We’ve been collaborating with another U.K.-based company called Image Metrics where they have devised a unique method of analysing video footage of human faces in motion. This analytical process produces very detailed data, and the team at Double Negative has worked out how to take this very dense, abstract data stream and plug it directly into our proprietary character animation pipeline.”

I had heard that their proprietary system uses sort of a biological substructure of virtual muscle and bone that is controlled by the dense Image Metrics data stream. “Yes, the key to this system is an understanding of how the underlying muscle groups in the human face combine to produce recognisable expressions and emotion; by sculpting the shapes resulting from the contraction and relaxation of the muscles in unison a powerful character rig can be built with a minimum of animation controls. The same philosophy informs the approach to analysing data generated by the video capture so there is a bridge between the two ends of the process.

I’d also heard that the Image Metrics approach to MoCap, called “CyberFace,” doesn’t require all those nasty face markers. Way back some time ago, Famous Faces had a system that would track facial characteristics without markers, but they never really went forward with it. I thought it a great idea at the time. Image Metrics has definitely taken this approach to a high level. “Yes, perhaps the most striking aspect of the whole methodology is that the capture sessions are marker free — the actor only wears light makeup so as to emphasise various key facial features, Franklin adds. “This provides a major advantage over other techniques where the large numbers of markers placed on the actor’s face can often prove to be an unwelcome barrier between the director and the performance. We have also worked out how to run the video capture simultaneously with standard full-body optical marker based capture so we can record the entire performance at the same time.

“Despite the process being marker-free it produces a very detailed recording of the subtleties of a performance. This is down to the fact that rather than sampling discreet points on the human face and then interpolating the missing data; our technique records continuous moving shapes taken from the eyes, mouth, cheeks etc… The analytical process then relates this detailed, yet localized, data to a comprehensive database of human expression, generating the animating muscle combinations that went into making the recorded shapes. This animation data then goes directly onto the same character controls used by our animators when they are keyframing a performance from scratch. The data can be left in its raw state for an unedited version of the performance or it can be worked on using a suite of in-house tools that allow our animators to use as much or as little of the captured performance to build the final character.”

One of the main complaints with MoCap is all the rigmarole that goes with it. I’ve heard actors and dancers complain about the costumes and markers, crew can get frustrated setting up multiple capture cameras and in general MoCap is a pain way below the neck. I asked Franklin if any effort or thought had been put into making the system easier on the director, crew and actors: “One of the most exciting aspects of this new approach to motion capture is that it eliminates a lot of that complex preparation generally associated with the whole field of performance capture. It also removes many restrictions placed upon performance — previously actors might have to keep their heads completely still or be confined to a very limited area on the stage. At its most simple this process can work with a single video camera with basic lighting. Adding more cameras and higher resolution enables more coverage and looser framing which in turn allows performances to flow naturally — another barrier removed between the director and the drama.”

The ability to capture a face with one camera sounds remarkable. I assume the software provides the 3D movement. In that case since you don’t have 3D position tracking. Of course, in most cases they use multiple high- resolution cameras. Franklin says they’re currently using this technology on several high profile movies and he believes the results will be spectacular.

As Animal Logic did on Happy Feet, many vfx companies are going to in-house solutions for motion capture needs. © 2006 Warner Bros. Ent. Inc. All rights reserved.

Animal Logic
I have a lot more research to do on Animal Logic, but I just had to mention them here. They were greatly responsible for the spectacular virtual performances in Happy Feet. Audience appeal of this movie has been spectacular in no small part because of the amazing dance sequences, fluffy stars and cute personalities. I understand that the crowd scenes were primarily animated using new proprietary crowd animation software known in-house as Horde, which takes multiple performances with variation and then randomizes them further through time and space warping. Very naturalistic crowd scenes were made possible through this tool. More procedural and cycle-based crowd work was handled by Massive, which also must be mentioned. More on Animal Logic down the road.

Massive
I’ve written about Massive, the intelligent crowd simulation system, on several occasions. It’s so good it floors me. Briefly, it uses artificial intelligence and virtual perception to select appropriate MoCap sequences, blends them and attaches them to characters in a highly realistic way. The result is that sequences animate themselves. In speaking with the Massive team at Rhythm & Hues, I discovered that there is always a ton of excitement when they gather to review the final render. They have no idea what the characters will do, but are always amazed at how flawless it usually looks. Massive is available for licensing if you need smart, believable crowds.

However, in chatting with Stephen Regelous, founder and product manager at Massive, he suggests that the application of AI need not be limited to crowd behavior. After all, each individual crowd character has to perform in believable ways. Blend-shapes could be scripted within Massive Prime, to emulate believable face expressions and emotions. Last year he implied that he had an interest in pulling high quality intelligent Hero performances out of Massive. Imagine using artificial intelligence animation software with Hero characters and getting believable performances. Having seen what Massive can do, I believe it’s not far off. In fact, I spoke with R&H Massive supervisor Dan Smiczek, who says: “It’s all there, built right into Massive Prime. It can handle very highly detailed face models using blend-shapes. You can get extremely fine animation details like eye blinks and impressive emotional expression. The really neat thing is that the blend-shapes are controlled directly by the AI. So you can have like one character yell at an other character and the one yelled at will ‘hear’ that and react appropriately, say with jerk and a nasty face.” I asked Smiczek if they’d been using Massive in this way and he adds: “Not yet. R&H has its own outstanding face animation software that we’ve been using, but it can be done.”

I know Regelous has had his sights set on intelligent automatic Hero animation for some time. He built an unexpected amount of face animation capability into Massive and many users don’t have the slightest idea of how powerful it is. You can develop an extensive library of FaceCap data sets and then script the Hero character, using the built in visual and auditory perception, to react. Remember you can also hand animate on top of this, if you like. I suspect this will shortly become an area of heavy use as studios learn that Heroes too can be intelligently autonomous.

ILM’s work done for Davy Jones’ eyes fooled even veterans in the vfx world. All images © 2006 Disney Enterprises Inc and Jerry Bruckheimer, Inc. Photo credit: ILM.

Uncanny Eyes
I think creepy eye movement has bothered me more than anything else in the virtual acting world. However, I think the guys at ILM have finally captured the holy grail under the brilliant leadership of supervisor John Knoll, R&D director Steve Sullivan and director Gore Verbinski. Davy Jones is a kind of hybrid character created by English actor Bill Nighy and a vast team of amazingly talented tech-animators and a director with the eye. In a sense, Davy Jones is probably the most advanced case of digital makeup ever conceived. With a ton of innovative approaches that I’m still exploring, Verbinski & ILM managed to extract and compile perhaps the most perfect virtual performance in history to date.

Jones is the character with the beard made of octopus-like tentacles. What you see is virtually all digital: the entire performance. It’s lively, exciting and real…even magical, but it’s all virtual. Or is it? I honestly don’t know how to classify it. Jones is a MoCap/keyframe hybrid because even though he is virtual, Nighy is fully represented in the performance of that digital makeup, which is tracked to the actor throughout the action. Can you say that virtual makeup gives a performance? I think we have to in this case. And the eyes — even in the close up I’m convinced the eyes are really Nighy’s, they’re all digital…amazing.

Meanwhile, the tentacles were animated using an articulated ridged body solver and flesh simulation developed by Ron Fedkiw’s team at Stanford. Nevertheless, I’m told much of the performance was keyframed and tracked flawlessly to the actor. But what techniques they used to get it this perfect is beyond me.

ILM had to develop an innovative way to track the virtual makeup onto Nighy as well. It was done, not in the typical greenscreen environment, but live on set and on location during a regular shoot, surrounded by actors in costume. Nighy had to wear something like a black-and-white checkered tracksuit with a skull cap and headband for tracking. His face had tracking dots and black rings around his eyes, so he looked a bit peculiar. He was out there in the water and on the beach, acting with the other players, who were resplendent in their wonderful costumes.

Hank is part of the Artificial Actors project at the Filmakademie Baden-Württemberg. © Filmakademie Baden-Württemberg.

It was necessary to track the virtual makeup to Nighy with great precision. To do this, Knoll, Sullivan and the R&D team developed what they call the Imocap process. Each performer is put in a suit with special tracking marks on it. Two high-resolution “witness” cameras are positioned on either side of the film camera. Parallax data is then used to triangulate positions of the special markers. Knoll credits the R&D team for coming up with remarkably clever software capable of reconstructing skeletal motion from the relatively straightforward triangulation data. What makes it even more amazing is that it was all done on location without the normal sources of power, often in knee-deep water. The entire system is portable, rugged and clearly robust.

I’d like to add a few words about Fedkiw, the Stanford professor who advises ILM on deep technology issues, often providing tools used for some of their amazing vfx and animation work. Fedkiw and his graduate students are using some of this virtual human simulation in very interesting ways. For example, one of his students has been working with surgeons from Iraq in designing facial reconstructions. They often work with the medical school at Stanford. Interestingly, it was this work that lead to the development of some remarkable technology for virtual acting. It’s based on biometrics. Fedkiw says: “We actually built a model of the human head using MIR images that give us a look at the head’s internal muscle structure. They come out a little bit warped and we had to correct for that. We also acquired the Visible Human data set to help refine it. We modeled the inside of the human head with all the muscles and bones etc.” Working with Motion Analysis, they developed some trial motion data streams. Using Motion Analysis’ acquired SIMM biomechanical animation software, they linked up the MoCap data to their virtual head. Motion Analysis used about 200 markers to develop some hi-res MoCap data. The data controls how much and in what way the virtual muscles of the head respond, yielding facial expressions that track the face actor with remarkable accuracy. This biologically accurate functional 3D model of the head shows great promise both in medicine and in entertainment. It is the most technically and medically robust physical head simulation I’ve seen, and perhaps that is why it works so well, when others have failed. Those of you, who thought Fedkiw and his team only did fluid dynamics, think again.

Something for Everyone
There is actually a sophisticated free facial animation system available for download. It’s being used professionally by production companies and it’s worth taking a look at. Volker Helzle and I have had coffee and chats on several occasions over the past few years as I followed the development of a very interesting facial animation system using an underlying blend-shape library with some 65 control sliders capable of creating virtually any emotional expression you can imagine. This one reminds me a little bit of the Pendulum system, and yet it’s very much its own system. This is part of the Artificial Actors project of the highly respected Institute of Animation, Visual Effects and Postproduction at the Filmakademie Baden-Württemberg in Germany. The documentation is available in English. It comes as a remarkable tool set that you can download here: “http://aistud.filmakademie.de/actor/88.0.html” at no charge. Don’t think that because it’s free, it’s not a very valuable tool. It’s been developed at enormous expense by top engineer artists, and I think you’ll be impressed. Helzle tells me he would very much like to have you join in their development effort by downloading and using the tools and reporting back with suggestions and complaints. I can’t tell you about the latest developments because they haven’t been announced, but this is cutting edge stuff.

When undertaking back-to-back sequels to Disney’s surprise blockbuster, Pirates of the Caribbean: The Curse of the Black Pearl, Industrial Light & Magic quickly realized that it neWith the help of the Imocap system, Bill Nighy’s creepy Davy Jones is the next great CG performance after Gollum and King Kong. All images © 2006 Disney Enterprises Inc and Jerry Bruckheimer, Inc. Photo credit: ILM.

eded to significantly raise the bar. Not only did the shot count triple from 324 to 979 on this summer’s record-breaking Dead Man’s Chest, but also the CG creatures were more complex and closer to the action. This required several R&D wrinkles and getting the creature pipeline up to speed on the new Zeno platform in San Francisco.As most of you have seen by now, the results of the character animation are very impressive. They’ve already begun talking about the creepy Davy Jones as the next great CG peformance beyond Weta’s Gollum and King Kong. Lord of the Deep and commander of the mysterious Flying Dutchman ghost ship, Jones is a delicious mutation: part human and part squid, with a beard full of wiggly tentacles, and crab-like claws.

Unable to rely on traditional MoCap or hand animation, ILM created an innovative new system called Imocap that allowed onset and on location motion capture to elicit the most believable look and performance possible out of actor Bill Nighy.

“The characters required a lot of careful examination of human performances and then trying to combine that with the animation,” explains animation supervisor Hal Hickel. “We knew that there were going to be actors cast to play Davy Jones and his crew, and that those actors would be on set in the plates that we were going to be put those CG characters into and that somehow we had to extract the motion of the performances without having to reshoot later. We didn’t want to bring the mocap stage onto the set. So the R&D and MoCap groups came up with a solution: special [sensor-studded] suits that would be worn by Bill Nighy and other actors playing his crew. We would take reference cameras onto the sets and untethered cameras out on location with lightweight tripods and position them at angles off of what the main taking camera was seeing. This allowed us to track the movements and provided great data from the hero plates with the actors in them, casting their real shadows and making good eye contact with the live actors, and then we were able to extract their motion and apply it to our CG characters and put those characters right on top of the actors. There’s still a lot of animation artistry in there because there’s a lot of interpretation. This is just about getting the skeletal motion of the character; we still did all of the facial animation by hand [in Zeno].”

A second R&D project at ILM involved creating Davy Jones’ tentacle beard itself. Many tests were done to get the behaviors right.

But more about Imocap in part two. Suffice it to say, Davy Jones is the most complex and human looking CG character in Dead Man’s Chest — and he’s all-CG. The animators incorporated as much of Nighy’s face as possible. However, the eyes proved to be an interesting test case. “As a backup, [director] Gore [Verbinski] asked us to put some makeup in a T-zone around his eyes and mouth, in case he wanted to do a blend for an extreme close-up,” Hickel continues. “But we never used it. We knew it would be difficult, but we figured we could get there pretty quickly. What was just as difficult was the whole spark of life. There’s always that last percent of realism that’s hardest to capture. The closer you get to the goal of it being real enough that people will stop worrying about it and thinking about it, the more glaring the omissions are. On top of which, there’s the gray area of his performance. The thing about Bill was he wasn’t a stone-faced villain. It was a very mercurial performance — he was constantly changing his expression and delivery. Nobody expected it. Every scene we’d stare at it and study it. I know there are animators that are leery of any technique that takes away some of their authorship. I totally understand that. Pure animation is wonderful. But I also think the collaboration between an animator and a live actor is an exciting thing too. I imagine it’s what makeup artists feel.”Zeno added an additional challenge. ILM came into Dead Man’s Chest with only a small portion of its creature pipeline function intact. War of the Worlds and The Island, the two previous projects done with Zeno, were primarily hard surface works. The creature work on those didn’t need cloth, sim, flesh or hair. A large part of the effort was re-enabling the pipeline, particularly the facial animation.

Meanwhile, the second R&D project involved the tentacle beard itself. “Our R&D folks worked with James Tooley, our sim guru [creature development supervisor], and Karin Derlich [creature technical director], who came up with behaviors,” Hickel adds. “And we’d do tests and I’d say, ‘This one is too tentacly and this one feels too much like an elephant’s trunk and this one feels too much like a snake.’ We would look at tons of octopus references. After we got it, then those behaviors were added to the solver through what we call ‘Joint Motors,’ so all the tentacles were divided into little joint segments and each segment was essentially a little motor that was directed to move this way or that way. So those joint motor impulses were sent out at the same time the tentacles were receiving force information: I need to swing this way, I need to swing that way… and so it would all happen together.

The crew of The Flying Dutchmen features a cast of characters with visual references to the ocean and its creatures, including coral, sea sponges, barnacles, mussels, hammerheads and puffer fish.

“Once we added those behaviors to our sim engine, the last thing we needed was something called ‘Sticktion,’ which is a combination of friction and stickiness. The problem was that without Sticktion, the tentacles would just slide onto each other. We really wanted them to be this heap of viscous tubes that would stick to each other and stick to his chest. And the ones at the bottom of the stack would stick there in a big matte. The biggest ones out in front that hang from his chin and moustache-like tentacles could really swing around.”

“Imagine a piece of spaghetti sticking to a leather jacket,” suggests visual effects supervisor John Knoll. “That was the effect I wanted to get. R&D added this subtle stickiness to the engine.”

“The great thing,” Hickel continues, “is that as complicated as it was, once Karin came up with basic settings for all of the controls, the sim artists got up and running very quickly. I’m pretty amazed by that, actually, because this was very stressful for me. Back in December, when we really didn’t have this working yet, there was no plan B. We couldn’t animate it by hand and we looked at other sim possibilities, but they didn’t achieve what we had in mind. There are more than 200 shots and 15 minutes of screen time of Davy and we had only one artist who knows how to do this.”

With as many as 50 animators working together on a total of 18 CG characters, there were plenty of technical and artistic challenges. “What makes these characters so complicated is that they are encrusted with sea life and we had to figure out ways to cover them with barnacles and such,” Hickel observes. “We wrote tools that the modelers used where they had a sea life picker, where they could pick a mussel or a barnacle. As our model supervisor, Jeff Campbell, said, it was a little like flower arranging. And they also used ZBrush for displacement textures for the sea life and for the characters themselves and our usual suite of modeling and paint tools.”

The crew of The Flying Dutchmen include Ogilvey, who has a sea sponge head; Palafico, whose head is a red fan coral and very translucent; Koleniko, in which one side of his face is a puffer fish and can puff up with spines; and Knoll’s favorite: a crab-like creature whose head rotates in and out of the shell.

The Kraken’s tentacles modeled in Maya. The creature was keyframe animated with some flesh sim enhancements in Zeno, courtesy of the new creature pipeline.

Then there’s the Kraken, the mythological squid monster that most are familiar with from 20,000 Leagues Under the Sea, which plays a prominent role in Dead Man’s Chest as the instrument of Jones’ destruction. Tentacles were crucial here as well. Modeled in Maya, the Kraken was keyframe animated with some flesh sim enhancements in Zeno, courtesy of the new creature pipeline. They even had to procedurally tweak the suckers on each row of tentacles because they were too clean looking, so they randomly replaced suckers that were more rough and worn looking. Utilizing ILM’s new fluid dynamics engine, developed in cooperation with the Stanford University research program, Dead Man’s Chest, like Poseidon, contains improved CG water, in which nifty algorithms are put through multiple processors. And thanks to Zeno, which has been described as “Maya on steroids,” you can introduce particle controls, Soft Body, Rigid Body controls and other techniques.

“We started out in parallel with Poseidon, but they got into a bit of a crisis and we loaned them my entire water crew,” Knoll admits. “They wrapped in April and I got them back to finish my shots. They really pushed the envelope. The development they did at the end of Poseidon really paid off here. We did a lot of difficult water shots right up to the last day. The crew really knew what nobs to turn to get it to look good. We used CG water around the bases of the tentacles when they’re sloshing back and forth underwater. The Flying Dutchman travels underwater and reaches the surface like a submarine, so those shots were done with CG water as well, and the Dutchman is 380 feet long. We got realistic droplet size and realistic dispersion of particles.”


The scenes on Cannibal Island, where Captain Jack Sparrow (Johnny Depp) narrowly escapes, contain a large number of shots where you see different variations of the same view under different lighting conditions, so Knoll and matte supervisor Susumu Yukuhiro needed to think about a 3D solution. “We saw ads for a product called Vue. It’s designed for organic landscapes and getting realistic renders. We started playing around with it and it became our primary tool for big, exotic landscapes.”

Overall, Knoll believes Dead Man’s Chest takes character animation another step forward at ILM, especially considering Nighy’s performance. “There are not as many shots numerically as on Sith, but it’s [a greater accomplishment] in terms of the amount of shots in the time that we had. Sith had 2,400 shots in about two years and this had 1,000 shots in about five months, but the average shot complexity was higher than on Star Wars.”

Looting CG Treasure From Dead Man’s Chest — Part 2
Bill Desowitz concludes our two-part coverage of Pirates of the Caribbean: Dead Man’s Chest with a report on ILM’s innovative Imocap system.
By Bill Desowitz
[ Posted on July 17, 2006 ]

Printable Version

ILM’s new Imocap was used to create the CG characters of Davy Jones and his crew in Dead Man’s Chest. All images © 2006 Disney Enterprises Inc and Jerry Bruckheimer Inc. Photo credit: ILM.

Tasked by director Gore Verbinski to come up with more complex and authentic-looking CG characters in Dead Man’s Chest, since Davy Jones and the crew of The Flying Dutchman would be interacting closely with the live actors, Industrial Light & Magic put its R&D team to work on a new incarnation of its proprietary motion capture system, dubbed Imocap. The results of Jones are so impressive, in fact, that people have already begun talking about the sea-encrusted villain with his creepy tentacle beard as the next great CG performance breakthrough.

	Early concept art for Davy Jones.

“We’ve done a lot of computer vision work here in R&D for the last several years and we were hoping to apply that to motion capture work outside of the MoCap studio some day,” remarks Steve Sullivan, director of R&D at ILM. “Dead Man’s Chest provided an opportunity for [remote MoCap] and a clear case of [requiring] that same quality on set where we needed those actors together in a scene for those hero performances. So we worked with the production team to nail down constraints of what we could get away with and what’s off-limits.” Imocap became a new protocol for measuring the actors and obtaining data during the actual shoot for the creation of skeletal motion in the computer. The software contained added functionality and new ways of tracking data. Special sensor-studded suits for the actors playing CG characters were created, which were more comfortable than typical MoCap outfits, as the actors were required to wear them in a variety of simple and treacherous conditions. “…On set, I wore a gray suit, which had reference points comprised of white bubbles and strips of black-and-white material, so that when they come to interpret your physical performance, they’re better placed to do so,” adds Bill Nighy, who plays Davy Jones.

According to Sullivan, “the suits needed to be ‘dignified.’ They had to be comfortable and not look ‘stupid.’ There were a few iterations of the material itself, which started out as a cotton blend but ended up being a stretchy, semi formfitting material. And we arrived at a neutral gray to help with our lighting calculations… and we used some markers and bands to help with the capture process itself. Those needed to be comfortable as well. Cameras were based on location and shooting conditions.”

“For shots where we used reference cameras, Kevin Wooley, our Imocap lead, housed some cameras in watertight enclosures and wired them to a computer for storing the images,” explains animation supervisor Hal Hickel. “This was great for the onset stuff. For beaches and jungles, we used untethered cameras with lightweight tripods. They were a little more trouble on the backend because they weren’t synchronized to each other, but both solutions worked well, and will continue to be used on the third Pirates movie [At World’s End].”

Davy Jones and the crew of The Flying Dutchman interact closely with live actors throughout the film.

Thus, by integrating the MoCap process with the actual shoot — providing the animators with hero plates with the actors in them, casting their real shadows and making good eye contact with the live actors — they were able to create, for instance, a more expressive, nuanced performance out of the maniacal Davy Jones, with the help, of course, of Nighy.

“We had new ways for the computer to analyze the images,” Sullivan continues. “The software piggy backed on MARS, the matchmoving [and tracking] solver. It understood what the actors could and couldn’t do. Our process is more holistic than traditional MoCap. We try to capture the whole body at once from different kinds of information, and that allows the flexibility to use many kinds of cameras and to work with partial information sometimes.

“The product of Imocap comes out as an animated skeleton, just like regular MoCap, and the animators do with that whatever they want, with artists in the middle running the post process. Sometimes they’ll need to cheat the body to get a better composition of the image. But the advantage is that the animators are overriding things and animating for performance reasons rather than just getting the basic physics and timing down. That all comes from the actor.”


Although ILM is currently developing its own facial performance capture system, Hickel determined this wasn’t the time to introduce yet another R&D component. “We have a lot of confidence in our facial animation, so we decided to do it by hand. The creature pipeline was being moved over to Zeno and most of the faces were different enough from the actors anyway.”

During the actual shoot on beaches and jungles, ILM used untethered cameras with lightweight tripods to measure the actors and obtain data that was then used to create the characters’ skeletal motions in the computer.

Hickel adds that there’s still a lot of animation artistry at work. “The CG characters weren’t 1:1 proportional copies of the actors, so there’s a lot of reinterpreting their motion and figuring out how to get a good performance out of a guy who’s head is made of coral. We had a little more freedom with some of the background actors because their faces are so different, such as Ogilvy, whose head is basically a giant sea sponge and he has one eye in some weird orifice. Davy Jones is the most complex and human-looking CG character. He’s 100% CG — even his eyes. We knew it would be difficult, but we figured we could get there pretty quickly. What was just as difficult was the whole spark of life. The thing about Bill was he wasn’t a stone-faced villain. It was a very mercurial performance — he was constantly changing his expression and delivery. Nobody expected it. Every scene we’d stare at it and study it.” Concludes visual effects supervisor John Knoll: “For us, it’s taken character animation another step forward with Davy Jones and how nice Bill Nighy’s performance comes through.”