HDR Stage

I am proud to announce I have completed my Master's thesis!

I have made many changes based on feedback that I have received, in addition to applying the correct layout to the thesis. (The thesis now weighs in at 69 pages!)

The thesis is currently under consideration (weighing in at 54 pages).

Thesis draft 1, weighing in at 52 pages, is nearly ready for review.

Work continues on writing the thesis. I'm up to about 38 pages of writing and pictures, 48 including the title page, etc.

Relighting college 8 with light maps, as previously described. The model now sits on an aerial image of College 8.

Although the visual light map (left column) is shown to the user, the actual lighting is done using the diffuse light map (right column). Shown are the forest light probe (top row) and the Grace Cathedral light probe (bottom row).

When light hits a highly polished surface, it all bounces off in the same direction because any point on the surface has the same orientation. However, a rough surface will cause light to bounce off in many directions because each point on the surface may be oriented in a different direction.

Surfaces that are perfectly smooth tend to be highly reflective (like a mirror, for example). The reflection of a scene in a mirror is sharp and clear. If the mirror was put in the forest environment (top image), the mirror would perfectly reflect the scene.

We assume that buildings are mostly rough and therefore the light that hits the building scatters in many directions. This means that using the forest environment would make the building look shiny, like a mirror, and that is not desired. Instead, a precomputed diffuse light map (bottom image) is used to simulate this property.

Here we show the lighting maps used for relighting models. The environment light map (the photographic image) forms the background around the model. The diffuse light map is formed from the environment light map. The diffuse light map is what is actually used for lighting the model.

The actual use of these two light maps is fairly abstract but it will become more clear after screen shots of a model lighted with these images are posted. The light maps are presented in latitude-longitude form (like a map of the world), but internally they are used in cube map form.

Environment light map

Diffuse light map

The equation relating one image to its two component intrinsic images (lighting L and reflectance R for all pixels x and y) is:

	=		x
Image (Frame 10)		Lighting (Frame 10)		Reflectance

What a difference a data set makes! Show below is a comparison between reflectance images calculated using a subset of the wall images and the full set of images. Clearly an image set with a wider variety of lighting conditions show a dramatic increase in quality.

All 26 photographs of a wall at College 8 taken throughout the day.

Reflectance from 11 images (left) and 26 images (right)

The color reflectance image calculated using a subset of 11 of 26 images (left),
and the reflectance image calculated using the full set of 26 images (right).

As described in the previous post, the reflectance of an image is calculated for each color channel, namely red, green, and blue. The following shows the reflectance for each color channel and the combined RGB reflectance image: Darker, blacker colors indicate low reflectivity while brighter, lighter colors indicate high reflectivity. Observe that wood siding tends to be darker while glass windows tend to be lighter.

Update (May 29): I've recalculated the reflectance images using the full set of 26 images instead of the subset of five images as used in the previous post.

	+		+
Red channel		Green channel		Blue channel

=
	Full color (Red + Green + Blue)

Reflectance images for red (top left), green (top middle) and blue (top right) channels,
which when combined, form a single color reflectance image (bottom).

I spent an entire day, from before sunrise to mid-afternoon, taking photos of a wall at College 8 at UCSC. I took a photo about every ten or twenty minutes or whenever the lighting changed significantly, and stopped when the wall was in complete shadow for the rest of the day.

The photographs of the wall will eventually be used to texture the 3D model of College 8. The problem is that the photographs have the light and shadows "baked-in" at the time the photographs were taken, so lighting the 3D model with an arbitrary light source will look wrong (for examples, the shadows might be in the wrong direction). We need a way to light the 3D model (and therefore textures) for any light source at any location.

Our goal is to break a photograph into two components: the light image and the reflectance image. We can then take the reflectance image to let us apply our own custom lighting to the photograph. Yair Weiss provides a very simple but very powerful algorithm to break down a series of images into a single reflectance and multiple lighting images, which is described in his paper, Deriving intrinsic images from image sequences (1.9Mb pdf). Conceptually, we take the derivatives of the images in both the x and y direction, then take the median values of the derivative images, recombine, and apply an inverse convolution to get the desired images.

The following is a sample result using just five images (out of a sequence of thirty images captured over a nine-hour timespan). Note that Weiss' algorithm generates five illumination images but only one reflectance image. Only the results from the red color channel are shown, but the green and blue color channels look very similar.

Photographs of a wall at College 8 taken thoughout the day. All of these images are used to calculate the reflectance and lighting images, shown below. Frame 1 is the first image on the left.

The red channel illumation of the wall in Frame 1 (left) and red channel reflectance of the wall (right). Only one reflectance image is generated from the five photograph, while there is an illumination image created for each photograph used as input. Note that the green and blue channels are similar but are omitted here to reduce visual clutter.

HDR light probes are 360 degree high dynamic range images. Light probes can be constructed from two 180 degree images using Photoshop or similar image editing program. The key operation is rectangular to polar (and polar to rectangular) conversion, along with image resizing and merging.

Light probes (360 degree images) captured around Santa Cruz. The scenes are (in left-to-right, top-to-bottom order):
College 8, Lower East Field, Crown-Merrill fire road 01,
Crown-Merrill fire road 02, West Cliff, and Seabright Beach.

Here's a mockup image of my new College 8 model (courtesy Andy Ames et al) lighted using the Grace Cathedral light probe:

A model of College 8 lighted (left) using the Grace Cathedral HDR light probe (right).

I constructed a HDR image of the wall material (painted wood) at College 8, then calculated the diffuse component using HDRShop.

The next step is to use the diffuse image as a reflectivity mask when rendering the lighting and textures. However, better diffuse images will need to be created to match the visible texture before they can be applied.

An HDR image of a College 8 wall (left) and the diffuse component (right).

I constructed light probes (360 degree photos) from two hemispherical images (using a camera with a fisheye lens) using Photoshop. The source images are preprocessed (semiautomatically) in Photoshop before HDRShop creates the final HDR light probe.

The two source omnicam images taken in opposing directions.
The Boardwalk is on the left while Seabright Beach is on the right.

Resulting light probes of the two views combined. While both images contain the same 360 degree image, different focal emphasis can be put on the center of the image. The Boardwalk is the focus in the left image, while Seabright Beach is the focus on the right image.

Take a look at the overview of HDR Stage. This is the high-level look at what is involved in creating HDR images, geometry, and relighting scenes.

View the HDR Stage overview here

Diffuse HDR lighting is key to relighting scenes. I extracted the diffuse lighting component from the HDR environment cubemap using HDRShop, which is an extremely processor-intensive operation. A 256x256 image took approximately 41 minutes to process on a computer with an Athlon 1.13Ghz processor.

The results are subtle (and look similar to just the diffuse base texture on the model) but in motion it is much more convincing. Better HDR environments and more complex models will help show the power of diffuse lighting.

The next steps are to display the HDR environment cube map while lighting with the diffuse component, in addition to a reworking of the internal model-shader architecture.

College 8 lighted by a diffuse HDR image (based on the Eucalyptus grove)

HDR cube map (left) and HDR diffuse lighting cube map (right)

I never actually attended College 8 so I can't actually reflect on it. Fortunately HDR Stage now pseudo-lights the model of College 8 using reflection mapping. This rendering of College 8 would look great if it were made completely from glass. However, it is not, mostly consisting of wood siding and windows. Clearly this rendering is not acceptable, so diffuse lighting is the next step.

College 8 reflecting HDR lighting environment

Models can now be drawn with multiple textures! Supports loading and manipulating multiple models (although that hasn't been exposed in the UI yet). Standard OpenGL lighting is currently being used (and none of that fancy HDR lighting yet).

College 8 in an HDR environment

A long and productive session:

• Better OBJ loading and drawing architecture.
• Exploring browsers in FLTK in preparation for loading multiple models into a scene.
• Merged OBJ loading code into HDRStage.
• Added a scene and texture manager to HDRStage.

Added OBJ models to HDR Stage

I designed a pretty good modular framework with (mostly) well-written and well-documented code, solid components, an error logger, and a GUI (thanks to FLTK). This will let me easily test, modify, and extend the current code base. Getting to this point has taken a lot of work but it'll definitely be worth it.

The main visual change is the result of multitexturing. I don't have any 3d model loading code included yet, so I'm just using a procedural sphere as a placeholder. The model is textured with an image of College 8, while reflecting the HDR environment around it.

Relighting a College 8 sphere