PROJECTS:
Light capture : Image-based re-lighting of multi-camera captured action

In our previous work we developed a studio system that allowed us to capture action in our experimental studio from 12 cameras synchronously in a chroma-keying environment. A 3D model of the actor is generated using a visual hull computation. The aim was to integrate the captured action into a different scene, e.g. for special effects. A limiting factor was that the original images were taken under the studio illumination and could not be re-lit to match a different lighting situation.

As an extension of this work we are proposing to capture a high dynamic range illumination map of the studio that allows to compute the diffuse surface reflection parameters of the foreground action. Once we have the surface properties we can integrate the studio scene into a different scene illumination captured again with a HDR illumination map. Both illumination maps are generated from a series of images of the studio and the target scene using a camera equipped with a spherical (fish-eye) lens.

	Fig. 1: a) input image, b) wireframe rendered image of the 3D model, c) irradiance image and d) the estimated diffuse reflectance.

Fig. 1 shows results of a test production. A scene with a boy was captured using 12 cameras simultaneously (just one camera image shown). Using a visual hull computation this gives a 3D model of the scene as depicted in image b) of Fig. 1. The model is made of 3000 triangles and shows some typical artefacts due to principle limitations of the visual hull reconstruction.

The picture c) of Fig. 1 shows the irradiance image rendered from the 3D scene model and an illumination map of the studio with a global illumination approach. The irradiance image is then used to compute the diffuse reflectance component from the input image as shown in picture e). Most of the shading effects have been compensated. The remaining problems (like the area on the boy's chest) are mainly due to errors in the visual hull.

	Fig. 2: An example of usage of the reflectance model using the original image (left) and our reflectance model (right).

Fig. 2 gives an example of the usage of the computed reflectance model in a different illumination environment. The boy is sitting on a 'space scooter' and is moving forward from the inside of a room (top image) into bright sunlight. The pictures are rendered with a commercial animation and rendering package. The bottom left of Fig. 2 shows the use of the original camera image in this situation and the right image shows the use of the reflectance map computed with our method. It can be seen that the use of our reflectance model is producing more realistic results under this changed lighting conditions.

The results show that the proposed approach is increasing the range in which the illumination can be changed from the original studio lighting.
The additional operational overhead for achieving that is relatively low since only the illumination map has to be captured in addition to the set-up of the multi-camera system.

A limiting factor of the method is the quality of the 3D models. In particular the surface normals that can be derived from the visual hull computation are not very precise. The diffuse component of the reflection can be still computed quite robustly since it is integrating over the hemisphere of each object surface point.

However more work will be carried out in the future to increase quality of the surface reflectance parameters. This will mainly target the accuracy of the 3D reconstruction that would allow better estimation of the surface normals and finally the consideration of the specular components.