Like most modern films, Avatar was shot digitally in 16:9 with safety protection in most shots for several different aspect ratios. It was released in either 1.85:1 or 1.77:1 in 3D, apparently depending on the theater.

However, Cameron has stated in at least one interview that he prefers the 2.39 aspect ratio for 2D projection:

For Avatar we’re shooting in a 16:9 ratio, we’re extracting a cinemascope ratio from that for 2D theatrical exhibition, and for 3D theatrical exhibition we will do, in the theaters that can, we’ll be in the 16:9 format and the theaters that can’t we’ll be in the scope format. Because I actually think that the extra screen height really works well in 3D. It really pulls you through the screen. So I’m actually going back on years of kind of eschewing the kind of 1.85 format, now saying 1.85 – or actually, it’s 1.78:1 – actually works really well in 3D. But only in 3D. I still like the scope ratio compositionally for flat projection.

And I have to agree that, if nothing else, the Blu-ray version looks off. There’s too much space on the top and bottom of the picture in general, and it looks more like an open matte version than something that the cinematographer would intentionally shoot for such an AR.