I met with James Cameron a few times when I worked in the Windows Media Division at Microsoft. Mr. Cameron had just finished the 3D IMAX documentary “Ghosts of the Abyss” and was looking at different technologies to use in an upcoming “big budget 3D production”. This was back in 2003 or so.
In the first meeting, we just provided a broad overview of the technologies that Microsoft was working on, and had a general discussion on what he was looking for. I must say, I was exceedingly impressed with Jim’s technical knowledge. Most “Writer/Director” types I’ve interacted with are great on the creative side but fairly light on the technical side. I pointed out that the digital camera that they used for the “Ghosts of the Abyss” had a subsampled horizontal resolution, and he jumped right in and told me how they solved that problem ... by tapping into the actual image sensor itself, bypassing the rest of the storage capabilities of the camera, they were able to capture the higher resolution image which the sensor was capable of, and store it directly to hard drives. I knew about this solution, but he had all the exact specs right off the top of his head and was able to describe, in perfect detail, how it worked. I scored some points with him as well, and he was in complete agreement with my opinion that full chroma resolution, known in the industry as 4:4:4, is absolutely critical for matting, chroma keying, and compositing special effects during production. But no one will ever see a difference between 4:4:4 and 4:2:2 (where the chroma information is subsampled by half) in a final distribution copy – even if it’s projected on a big screen in a movie theater.
At the time, we (Microsoft) had just finished developing the Windows Media 9 Series Professional Video Codec (known as WMV9, which was later standardized as SMPTE specification VC-1). I was demonstrating the differences between WMV9 and the only other high performance codec that could handle full HD video, which is called h.264 (also known as AVC). One of the biggest advantages of WMV9/VC-1 is that it can handle very fine detail – it is much better than h.264 in this regard. We had a “split screen” clip of an old Disney movie that was shot on 16mm film, which was quite grainy. The demonstration clearly showed that the h.264 codec (encoded with the exact same bitrate, buffer parameters, etc) could not handle the randomness of the film grain, and most of the fine detail was lost – while the WMV9 codec did a far better job with this kind of a stress test.
Jim was getting visibly agitated during the demo, and I wasn’t sure what was going on. He was fidgety, turning red, and finally he blew up at me. He got right up in my face, about six inches away, and said “I’m looking right at you and your face is perfectly clear. I don’t see any film grain. I don’t want to see any in my movies either!!!!!”
It was classic James Cameron – you always hear these stories about his outbursts, but I really love the fact that he so exceedingly passionate about making the absolute best, technically perfect, image quality that he can. He is a true perfectionist. I pointed out that the demonstration was just to show how much better our codec was at handling very fine, random detail. I wasn’t “in favor” of film grain in the slightest, but if your codec can’t reproduce this, it can’t reproduce other fine image details either. He calmly conceded that point, and we continued with the meeting as if nothing happened.
(Side note: Both h.264 (aka AVC) and VC-1 were accepted as encoding standards for HD-DVD, and eventually Blu-Ray (although Toshiba unceremoniously dropped HD-DVD after Sony paid Warner Bros $500 million, and another $100 million to Fox, to exclusively support Blu-Ray). But every Blu-Ray player made today has the VC-1 decoder (technology which I worked on) and every HD movie released on Blu-Ray is encoded with one of these two codecs (although some use MPEG2 – the old codec used on regular DVD’s). See http://www.blu-raystats.com/Stats/TechStats.php for a list of what movie is encoded with what codec.)
But back to the story ... Jim made it very clear that he never wanted to shoot on film again, everything was going to be digital, and this next project was going to be in 3D. I didn’t realize the production was going to take six more years!
At our next meeting, we had prepared for a more thorough discussion of 3D and how we might be able to help him. Although a 3D effect requires separate images for the right and left eyes, the vast majority of the picture information is duplicated between those two images. Video codecs are exceedingly good at analyzing video and finding redundant information which can be compressed to save bandwidth, and a 3D picture could easily be encoded without having to distribute two complete, separate, full resolution video streams. With a little bit of engineering work, we were able to show a pretty good “proof of concept” for this.
We had an extremely interesting discussion. To create the final 3D effect on playback, you need to be able to “direct” the correct image to each eye. That’s all well and good, but twice as many images means twice as many video frames, which has a cost associated with it. And the technology to direct the image to each eye will almost always be the limiting factor. There are various ways to do this: red/blue or green/magenta chroma separation (poor quality image, but cheap and easy), polarization (ok for movie theaters, but problematic in the home as you need two synchronized projectors), and alternating frames with a synchronized LCD shutter (which is the technology used for 3D at home). However, if you use the alternating frames without increasing the frame rate, you reduce the temporal bandwidth – basically, your brain only sees it as half the framerate, and the flicker will be very obvious. So, the total framerate needs to go up.
At the time, circa 2003, it was actually quite difficult to double the framerate. Digital Cinema Production was relatively new, and the image sensors in those early generations of high resolution digital cameras couldn’t offload the data fast enough for each picture, before it needed to snap the next frame. (this is not a problem now, of course, even some high end “consumer” HD cameras today have high speed modes which outperform the early models that Cameron was using, which cost over $120,000 each at the time).
So we were brainstorming some ideas on a whiteboard, and I said, “How about if you scan the image vertically, instead of horizontally, and direct alternating vertical scan lines to each eye.” Everyone stopped talking and pondered the idea. I had hit upon something interesting. Basically, this would have the effect of reducing the horizontal resolution of each image, instead of the temporal resolution (which reduces framerate), but the eye is much less sensitive to horizontal subsampling than it is to temporal subsampling. Indeed, at our first meeting I had pointed out that the camera they had used for “Ghosts of the Abyss” does just that – when you record onto the internal HDCam tape, the image is compressed in the vertical domain using non-square pixels. (and our own tests at Microsoft had previously confirmed that, in general, this is ok – if you don’t have enough bandwidth to encode/record a full 1080P image, you are better off downsampling in just the horizontal domain rather than making the entire frame smaller and going down to a 720P image).
The problem with this idea, unfortunately, is that directing the proper information to each eye becomes far more difficult. Instead of flashing the shutter on the synchronized 3D glasses each frame (on the order of 20 to 40 milliseconds) you’d need to do it for each scan line (about a thousand times faster). So while my idea could help reduce the storage requirements, it created a different problem which would be even more complex – but it was a great brainstorming session and it was an interesting concept to discuss even though it wasn’t really feasible. (and as everyone knows, storage space has continued to get cheaper and cheaper and cheaper. In 2002, when we were producing Muffin Man, I spent over $10,000 for a ten drive SCSI RAID5 disc array, which had one terabyte of storage. Last month I bought one terabyte hard drive for eighty four bucks!)
Mr. Cameron and his production crew continued to work with Microsoft Studios over the next year on a few things, mostly exploration work, and I had limited involvement with the project after that. I did get a call from one of the engineers one time, saying that Jim asked about me, which was nice.
A few years later, I finally put two and two together when reports about Avatar started leaking into the public awareness, and I realized this was the “big project” that he was working on. So I was totally thrilled to finally see it in the theaters last month. And I was also thrilled, near the very end of the credits, to see a “Special Thanks to Microsoft”. I’m not sure if I really contributed anything personally that actually made any impact on the production, but I think I earned at least a tiny part of that “Special Thanks”.
And the movie itself? Well, Avatar seems to be one of the most reviewed films of all time, so I don’t have too much to add. For the most part, overall reviews have been very positive, and while most reviewers have some issues with various aspects of the story line – they all pretty much say “it’s a beautifully produced movie and WOW, what incredible 3D effects!!!”
So here’s my take. I really loved it overall, and I too had some issues with certain aspects of the story line. Specifically for me, what is unobtainium and why is it so important? Is it something that will “save” humanity (i.e., it has medicinal use) or is it just a valuable item, like diamonds or jewels that can just be traded for a profit? Is it REALLY worth killing for? To me, this was a great distraction, and I had a very hard time believing that more of the military crew didn’t have moral and ethical problems with the fact that they were rampantly killing intelligent beings, destroying their world, just for this “unobtainium”.
(I’ll bet that this had been explained, and that some scenes got cut from the final edit to make the movie shorter. My guess is that you’ll probably see some deleted scenes on the “special features” of the Blu-Ray disc when it comes out.)
But WOW, it really is a beautifully produced movie - and what incredible 3D effects!!!
And I am happy to report that there is not one single spot of “film grain” to be seen anywhere in the movie!