My (exceedingly minor) contribution to Avatar’s 3D effects

Tuesday, 26 January 2010 05:54 by The Lunatic

 

I met with James Cameron a few times when I worked in the Windows Media Division at Microsoft.  Mr. Cameron had just finished the 3D IMAX documentary “Ghosts of the Abyss” and was looking at different technologies to use in an upcoming “big budget 3D production”. This was back in 2003 or so.

In the first meeting, we just provided a broad overview of the technologies that Microsoft was working on, and had a general discussion on what he was looking for.  I must say, I was exceedingly impressed with Jim’s technical knowledge. Most “Writer/Director” types I’ve interacted with are great on the creative side but fairly light on the technical side. I pointed out that the digital camera that they used for the “Ghosts of the Abyss” had a subsampled horizontal resolution, and he jumped right in and told me how they solved that problem ... by tapping into the actual image sensor itself, bypassing the rest of the storage capabilities of the camera, they were able to capture the higher resolution image which the sensor was capable of, and store it directly to hard drives. I knew about this solution, but he had all the exact specs right off the top of his head and was able to describe, in perfect detail, how it worked. I scored some points with him as well, and he was in complete agreement with my opinion that full chroma resolution, known in the industry as 4:4:4, is absolutely critical for matting, chroma keying, and compositing special effects during production. But no one will ever see a difference between 4:4:4 and 4:2:2 (where the chroma information is subsampled by half) in a final distribution copy – even if it’s projected on a big screen in a movie theater.

At the time, we (Microsoft) had just finished developing the Windows Media 9 Series Professional Video Codec (known as WMV9, which was later standardized as SMPTE specification VC-1). I was demonstrating the differences between WMV9 and the only other high performance codec that could handle full HD video, which is called h.264 (also known as AVC). One of the biggest advantages of WMV9/VC-1 is that it can handle very fine detail – it is much better than h.264 in this regard. We had a “split screen” clip of an old Disney movie that was shot on 16mm film, which was quite grainy.  The demonstration clearly showed that the h.264 codec (encoded with the exact same bitrate, buffer parameters, etc) could not handle the randomness of the film grain, and most of the fine detail was lost – while the WMV9 codec did a far better job with this kind of a stress test.

Jim was getting visibly agitated during the demo, and I wasn’t sure what was going on. He was fidgety, turning red, and finally he blew up at me. He got right up in my face, about six inches away, and said “I’m looking right at you and your face is perfectly clear. I don’t see any film grain. I don’t want to see any in my movies either!!!!!”

It was classic James Cameron – you always hear these stories about his outbursts, but I really love the fact that he so exceedingly passionate about making the absolute best, technically perfect, image quality that he can. He is a true perfectionist. I pointed out that the demonstration was just to show how much better our codec was at handling very fine, random detail. I wasn’t “in favor” of film grain in the slightest, but if your codec can’t reproduce this, it can’t reproduce other fine image details either. He calmly conceded that point, and we continued with the meeting as if nothing happened.

(Side note: Both h.264 (aka AVC) and VC-1 were accepted as encoding standards for HD-DVD, and eventually Blu-Ray (although Toshiba unceremoniously dropped HD-DVD after Sony paid Warner Bros $500 million, and another $100 million to Fox, to exclusively support Blu-Ray). But every Blu-Ray player made today has the VC-1 decoder (technology which I worked on) and every HD movie released on Blu-Ray is encoded with one of these two codecs (although some use MPEG2 – the old codec used on regular DVD’s). See http://www.blu-raystats.com/Stats/TechStats.php for a list of what movie is encoded with what codec.)

But back to the story ... Jim made it very clear that he never wanted to shoot on film again, everything was going to be digital, and this next project was going to be in 3D. I didn’t realize the production was going to take six more years!

At our next meeting, we had prepared for a more thorough discussion of 3D and how we might be able to help him. Although a 3D effect requires separate images for the right and left eyes, the vast majority of the picture information is duplicated between those two images. Video codecs are exceedingly good at analyzing video and finding redundant information which can be compressed to save bandwidth, and a 3D picture could easily be encoded without having to distribute two complete, separate, full resolution video streams. With a little bit of engineering work, we were able to show a pretty good “proof of concept” for this.

We had an extremely interesting discussion. To create the final 3D effect on playback, you need to be able to “direct” the correct image to each eye. That’s all well and good, but twice as many images means twice as many video frames, which has a cost associated with it. And the technology to direct the image to each eye will almost always be the limiting factor. There are various ways to do this: red/blue or green/magenta chroma separation (poor quality image, but cheap and easy), polarization (ok for movie theaters, but problematic in the home as you need two synchronized projectors), and alternating frames with a synchronized LCD shutter (which is the technology used for 3D at home).  However, if you use the alternating frames without increasing the frame rate, you reduce the temporal bandwidth – basically, your brain only sees it as half the framerate, and the flicker will be very obvious. So, the total framerate needs to go up.

At the time, circa 2003, it was actually quite difficult to double the framerate. Digital Cinema Production was relatively new, and the image sensors in those early generations of high resolution digital cameras couldn’t offload the data fast enough for each picture, before it needed to snap the next frame. (this is not a problem now, of course, even some high end “consumer” HD cameras today have high speed modes which outperform the early models that Cameron was using, which cost over $120,000 each at the time). 

So we were brainstorming some ideas on a whiteboard, and I said, “How about if you scan the image vertically, instead of horizontally, and direct alternating vertical scan lines to each eye.” Everyone stopped talking and pondered the idea. I had hit upon something interesting. Basically, this would have the effect of reducing the horizontal resolution of each image, instead of the temporal resolution (which reduces framerate), but the eye is much less sensitive to horizontal subsampling than it is to temporal subsampling.  Indeed, at our first meeting I had pointed out that the camera they had used for “Ghosts of the Abyss” does just that – when you record onto the internal HDCam tape, the image is compressed in the vertical domain using non-square pixels. (and our own tests at Microsoft had previously confirmed that, in general, this is ok – if you don’t have enough bandwidth to encode/record a full 1080P image, you are better off downsampling in just the horizontal domain rather than making the entire frame smaller and going down to a 720P image).

The problem with this idea, unfortunately, is that directing the proper information to each eye becomes far more difficult. Instead of flashing the shutter on the synchronized 3D glasses each frame (on the order of 20 to 40 milliseconds) you’d need to do it for each scan line (about a thousand times faster). So while my idea could help reduce the storage requirements, it created a different problem which would be even more complex – but it was a great brainstorming session and it was an interesting concept to discuss even though it wasn’t really feasible. (and as everyone knows, storage space has continued to get cheaper and cheaper and cheaper. In 2002, when we were producing Muffin Man, I spent over $10,000 for a ten drive SCSI RAID5 disc array, which had one terabyte of storage. Last month I bought one terabyte hard drive for eighty four bucks!)

Mr. Cameron and his production crew continued to work with Microsoft Studios over the next year on a few things, mostly exploration work, and I had limited involvement with the project after that. I did get a call from one of the engineers one time, saying that Jim asked about me, which was nice.

A few years later, I finally put two and two together when reports about Avatar started leaking into the public awareness, and I realized this was the “big project” that he was working on. So I was totally thrilled to finally see it in the theaters last month. And I was also thrilled, near the very end of the credits, to see a “Special Thanks to Microsoft”. I’m not sure if I really contributed anything personally that actually made any impact on the production, but I think I earned at least a tiny part of that “Special Thanks”.

And the movie itself?  Well, Avatar seems to be one of the most reviewed films of all time, so I don’t have too much to add.  For the most part, overall reviews have been very positive, and while most reviewers have some issues with various aspects of the story line – they all pretty much say “it’s a beautifully produced movie and WOW, what incredible 3D effects!!!”

So here’s my take. I really loved it overall, and I too had some issues with certain aspects of the story line.  Specifically for me, what is unobtainium and why is it so important? Is it something that will “save” humanity (i.e., it has medicinal use) or is it just a valuable item, like diamonds or jewels that can just be traded for a profit? Is it REALLY worth killing for?  To me, this was a great distraction, and I had a very hard time believing that more of the military crew didn’t have moral and ethical problems with the fact that they were rampantly killing intelligent beings, destroying their world, just for this “unobtainium”.

(I’ll bet that this had been explained, and that some scenes got cut from the final edit to make the movie shorter.  My guess is that you’ll probably see some deleted scenes on the “special features” of the Blu-Ray disc when it comes out.)

But WOW, it really is a beautifully produced movie - and what incredible 3D effects!!!

And I am happy to report that there is not one single spot of “film grain” to be seen anywhere in the movie!

 

Categories:   Miscellaneous | Science
Actions:   E-mail | Permalink | Comments (14) | Comment RSSRSS comment feed

Comments (14) -

January 29. 2010 01:34

What a great post! I agree that there was limited background info on the unobtainium but it didn't really bother me that much. Drives me crazy with all these reviews that say "It's basically the same story as Pocahontas/Dances with Wolves/David and Goliath/Alien/whatever. Yes, there are elements of the battle story that are borrowed elements, but the overall "world" that he created is quite unique. I'm not sure I would liked getting yelled at by James Cameron. I usually try not to upset anyone with a net worth more than 100 times my own Smile

Bill Fields

January 30. 2010 17:46

Ha! When I was Microsoft, many people considered it a "Badge of honor" to get yelled at by Bill Gates. I only had a few personal interactions with him and they were all very pleasant. I had to solicit a special guest to come in and yell at me!

The Lunatic

January 30. 2010 20:19

I can yell at you if you want.  Smile

Geoff

February 20. 2010 18:58

Update on 3D TV – I’ve seen a number of articles about “new technologies” that will bring the 3D experience into the home.  This is actually fairly easy, no new technologies need to be invented – but the industry does need to standardize a few things to eliminate incompatibilities between different manufacturers.

All that is needed for 3D TV is to double the frame rate on both the Blu-Ray player (unfortunately, this is not an option on regular DVD players) and the TV display, and have an infra-red LED that blinks on and off to synchronize the LCD shutter glasses.  This can be the exact same type of LED that’s used in every remote control, it’s just coming OUT of the system and aimed towards the viewers. The LED will turn “on” for the frame that shows in one eye and “off” for the frame that shows in the other eye.

If it’s done right, the manufacturers will standardize on a scheme where the synchronization LED is in the TV and not in the disc player – this way, you can use the same configuration for broadcast and satellite programming as well.  My fear is that some of the hardware manufacturers will try to “jump start” the process with their own proprietary system to gain a competitive advantage, which could result in incompatible (and most likely more expensive) glasses, and possibly even programming that only works if you have a player and TV from the same manufacturer.

With an appropriate industry standard, it would be easy to give the user the choice of displaying content in either 3D or 2D (by just displaying every other frame, you will get a “regular” image), inexpensive glasses that you can buy anywhere, and programming that can be from an over the air broadcast source, satellite, or played back from disc.

The Lunatic

April 3. 2010 17:54

I saw a documentary on Jim Cameron and learned that he has a degree in Physics so no big surprise that he is such a techy. Our boss from GVG days yelled at me too, maybe it's a physics thing?

Avatar was the first 3D movie I watched without eye strain so whatever was done, it was good.

I'll thank you, David.
Mike

Mike

April 9. 2010 16:00

David,

There was an episode of 3rd Rock from the Sun many years ago that had several sequences in 3D - it was broadcast over regular TV.  I even bought a VHS tape of it, along with the requisite glasses.  I have no idea how they did it, but it was very fun and quite effective.  

Geoff

April 9. 2010 18:55

I hadn't heard about that, but the only way I know of to do it is with chroma separation - which does work, but the director is limited in the colors that can be used. I can see why 3rd Rock from the Sun would be a good show for this - the characters mainly wore brown shirts and the sets were mostly "neutral" colors!

The Lunatic

April 9. 2010 21:35

I just looked it up.  Not chroma at all, it was something they call the Pulfrich effect, which is a medical condition that many people have, but can be enhanced with classes, one dark lens and one clear lens.  Unfortunately it only works with objects moving and only from side to side, not up and down.  I don't understand it very well, but the Wiki article is here: http://en.wikipedia.org/wiki/Pulfrich_effect - there are also some YouTube videos that demonstrate it.  Just look through one lens of a pair of sunglasses, with no lens on the other eye.

I'm going to have to go back and watch that 3rd rock again, I think.

Geoff

April 10. 2010 02:57

Hmm, very interesting.  I am familiar with the effect, it's a popular demonstration at science museums (but I never knew the name for it). They usually have a ball hung from a string, which is swinging back and forth. You put on the glasses and look at the shadow of the swinging ball, and the shadow appears to be swinging towards you in a circular motion. I had never thought of using this for 3D video - but it is still quite limited by the fact that it only works on images with certain lateral motion. As soon as the motion stops, the effect disappears!

The Lunatic

April 10. 2010 06:46

Yeah.  I remember some of that show looking wonderful, but some of it being odd.  Now I know why.  As I say, I'm going to have to dig the tape out and watch it again.  It's in the garage, somewhere!

Geoff

May 6. 2010 07:29

This has has to be one of the greatest posts I've read. I love Cameron's movies, but he seems to jump the gun when it comes to dealing with people.  Avatar, great effects, ok storyline thats' been done a thousand times.  It also was cliched, but sometimes that works in movies.  I am fascinated with anything 3d though, can't wait to see the new technology involving it.  A 60 frames per second 3d movie would rock everyone's socks off.

Best 3d Animator

June 2. 2010 02:38

Wow, great story.  I heard that Cameron  is even assisting with the oil well in the gulf these days.

Erik

June 4. 2010 06:17

Second update on 3D TV – In the six months since I originally wrote this post, 3D TV's and Blu-Ray disk players have taken the market by storm. The first two manufacturers out of the gate with complete offerings are Samsung and Panasonic.  I finally got a demo of a big-screen Samsung system this weekend, and it is very impressive - as close to a 3D "theater" experience as you can imagine.

I did not get a chance to try it myself, but I am told that the 3D enabled Blu-Ray players will work with any 3D enabled TV's from other manufacturers - which is great news! However, the manufacturers have implimented proprietary signaling systems for the LCD shutter glasses, which means that you need to buy extra (or replacement) glasses from the manufacturer of your TV (which are bound to be more expensive).  To me, this is very anti-consumer. I have a feeling they will get it sorted out and will eventually standardize - but I'm not sure how long this will take.  In the meantime, a company called Xpand Cinema (http://www.xpandcinema.com) sells glasses that they claim will work with any TV set.

The Lunatic

October 18. 2010 08:43

I'm looking for a life-size cardboard cutout of a Na'vi. That way, for halloween, I can hold it in front of me and go as an Avatar character as seen in a 2D movie theater!

The Lunatic

Add comment