Iso Photography
Iso Photography
Embedded Files
The iso photography module is designed to calibrate and compare images in order to validate the iso photography process.
The window contains 2 widgets :
the settings widget on the left contains two tabs, one for the calibration of the images, and the other for the validation and comparison between photos and simulations, see details bellow,
the preview widget contains has no use when the calibration tab is selected, and displays a preview of the selected images when the validation tab is selected. The footer of the preview widget contains the display settings to choose and define the comparison mode.
NB : the documentation for this widget is included in the "Analyse" section at the end of this page, you can skip the "Preparation" section at the begining if you already have all the required pictures and simulations.
The preview widget in the iso photography module works exactly as the one in the image comparison module. See the Image Comparison module documentation for more details on the preview widget footer.
Prepare the required files for Iso-photography
Prepare the required files for Iso-photography
An iso-photograph ("iso" formerly "equal to") is a computer simulated image that is virtually equal to a photograph. A photograph and iso-photograph, viewed side by side, cannot be differentiated by an human eye.
An iso-photograph is an objective prediction of the appearance of objects and scenes as they would be captured by a digital camera.
Beyond the magic, taking iso-photographs allows you to make the right decision about the choice of materials for your prototypes. Iso-photographic simulation is also a key step in the validation process of the accuracy of your rendering software. Last but not least, video compositing with iso-photograph renderings is more natural and does not require any adjustment effort.
The iso-photography process relies on several steps :
1) Taking reference pictures to determine the intrinsics parameters of the camera and actually find these settings,
2) Taking a picture that will be the reference image to compare with the simulation. This picture is also used to determine the camera position,
3) Measure all the materials that are present in the scene, as well as the light spectral behaviour,
4) Configure the 3D scene that will be used for the simulated image : import and place all the geometries in the scene, import the camera intrinsic settings and place it in the scene, assign the materials and light emissions that were measured,
5) Generate the simulated image and export the raw output from Predict Engine,
6) From the raw outputs from the camera and the simulation, apply the same calibration to both files in order to get RGB images from the RAW bayerised inputs,
7) Apply the required post process to each image and compare them visually and using the DeltaE metric.
Figure 1 : example of an iso-photograph
Figure 2 : photograph the Figure 1 is simulating
1) Camera intrinsics settings
1) Camera intrinsics settings
In order to define the camera intrinsics settings (focal length, field of view and distorsion), we need reference pictures :
It is recommended to take between 3 to 20 pictures of a flat chessboard where it appears in whole with as many different sizes, positions and orientations as possible,
The exposure of the chessboard should be correctly set and it is better to avoid any lighting variation such as the projection of shadows by other elements across its surface in order to facilitate its detection,
These reference pictures do not need to be taken with the same lighting condition as the scene,
These reference pictures must be taken with the same camera and lens settings (focal length, zoom, focus distance, aperture) that will be used for the entire iso-photography process.
Figure 3 : example of intrinsic calibration reference pictures
Once the reference pictures are ready, the intrinsics settings of the camera can be determined using Predict Cam (standalone) :
Run ∽camera.exe
Choose Track Object -> Chessboard.
Set the settings of the chessboard (width, height, edge length, margin, depth).
Choose Resize -> No in order to use the full image resolution.
Open a set of images. A number of images between 3 to 12 is needed to get a fine calibration: The more the better. The pictures should be focused on the chessboard with no motion blur so that the edges appear sharp and the borders are fully visible. The viewpoints of the chessboard should cover the whole field of view with multiple inclinations and rotations.
Press Camera View in order to see the superposition of the virtual chessboard over the picture.
Before calibration, the alignment is not right.
Go through each image using the Next Frame button (or the right arrow) and press Push to Buffer. If the chessboard is correctly detected, the Buffer Size is incremented and the alignment between the virtual chessboard and the picture is updated.
Save the calibration data to an .intrinsic file using Save Intrinsic in the File section.
2) Reference picture and camera extrinsic settings
2) Reference picture and camera extrinsic settings
The simulation generated with Predict Engine will be compared to a reference picture. Usually, we use this same picture to determine the camera position : this only works if the picture contains the same chessboard used for the intrinsic calibration of the camera.
If it is not the case, an additional picture must be taken with the camera properly positionned for the iso-photographic process and the chessboard added in its field of view.
The position of the chessboard should be known as much as possible as it will help positioning the 3D geometries in the scene later on.
The reference picture for the extrinsic calibration (camera position) should follow these rules :
The same chessboard and camera (with its lens) used for the intrinsic calibration must be used and the camera position must be exactly the same as in the final image for which one wants to perform the iso-photography,
The chessboard must appear free of occlusions by any other element in the scene,
If multiple camera positions are needed, this reference picture of the chessboard must be taken each time the camera is moved,
The exposure of the chessboard should be correctly set.
Figure 4 : example of a reference picture for the extrinsics camera calibration
The chessboard is better detected facing the camera and may probably not be detected at a shallow angle.
Once the reference picture is ready, the position of the camera can be determined using Predict Cam (standalone) :
Run ∽camera.exe
Load an intrinsic calibration file using Load Intrinsic (see the previous section on how to compute the intrinsic parameters if needed).
Choose Track Object -> Camera so that the position of the chessboard is fixed and the camera is moving.
Set the following settings of the chessboard: width, height, edge length, margin, depth.
Open the image Camera -> Stream -> Single Image. Once loaded, the virtual chessboard should coincide with the real chessboard. You may have to make the plane of the image closer or further by changing the value of Image Plane Distance. If the chessboard is not detected, try changing the resolution of the input image in Tracking-> Resize or toggle the Tracking-> Fast checkbox.
Load a mesh representing the scene using File -> Import Geometry and place the virtual chessboard at the correct position in the virtual scene. Note that only OBJ format is supported. From the camera viewpoint, the geometry of the scene should coincide with the picture.
Save the extrinsic parameters using Save Extrinsic as well as the settings of the chessboard using Save Chessboard.
Export the camera to the UVR format using Export Camera.
3) Materials and light measurements
3) Materials and light measurements
All materials and lights in the scene should be measured in order to ensure an output iso-photographic image of high quality and accuracy:
Glass and metal-like materials are usually measured using an ellipsometer which outputs spectral complex index of refraction.
Measures of optical indexes for standard glass and metallic materials like BK7 glass can be obtained from databases but may not correspond to actual materials which can sometimes include additional coatings (for protection for instance).For opaque specular and diffuse materials, it is the reflective properties of the material (the BRDF) that should be measured. This property is usually measured with a specific device like a goniometer.
The measurement of light properties encompasses their spectral and angular distributions. It is of extreme importance that it is done accurately since it has a large impact on the colors and brightness of the rendered scene.
It is recommended to use measurements that are as close as possible to the true materials used in the scene.
It is strongly recommended to use spectral measurement instead of RGB to gain in accuracy.
See the Physics section for more details on spectral rendering.
4) 3D scene configuration
4) 3D scene configuration
The 3D scene for the simulation must contain all the geometries properly positionned and using the right material, the lights using the right emission, and the camera with the right intrinsic settings and position.
Start by placing all the required geometries in the scene, you can start by placing them roughly.
You must include in the scene the chessboard that was used for the camera extrinsic calibration. The chessboard geometry can come from an existing geometry file (.fbx, .obj,... in this case make sure the origin of the GameObject is placed on a white corner of the chessboard), or you can create it directly inside Unity, especially if you have access to the .chessboard file from Predict Cam Standalone :
Instantiate the chessboard GameObject by right clicking anywhere in your Unity scene and selecting the menu "PredictSuite/Camera/Instantiate Chessboard",
The opened Chessboard Editor enables you to create a geometry with a custom chessboard. The chessboard should match the one you used to take the reference pictures with your camera. Import the chessboard settings using the button on the top right corner, or fill in the required fields :
the Square Count field is the number of squares (width*height) in the chessboard,
the Square Dimension field is the dimension of one square (width*height) in Unity units,
the Margin Dimension field is the dimension of the white margin around the chessboard (X is left, Y is right, Z is bottom, W is top) in Unity units,
the Origin Color field is the color of the square in the bottom left corner,
the Origin outside the margin field defines whether the origin (red dot on the preview) is at the very outside of the mesh, or if it is at the corner of the first square,
the Depth field is the actual depth of the geometry that will be created, in Unity units.
Click the "Create Chessboard" button to create and instantiate the geometry in your scene.
Place the chessboard where you want it to be in the 3D scene : if you have the .chessboard file for your current setup, you can use the same method as to place the camera in the scene (right click on the GameObject, menu PredictSuite/Transform/Import Extrinsic Matrix, give a reference to the .chessboard file)
Then place and configure the camera in the scene :
Import the intrinsic camera calibration : add the UVR Projective Camera component to the camera, and give it a reference to the .intrinsic file that was generated by Predict Cam Standalone,
Import the extrinsic camera calibration : right click on the camera GameObject, menu PredictSuite/Transform/Import Extrinsic Matrix, give a reference to the .extrinsic file and import the Transform. NB : you must select the option "Convert to Y up".
To be able to place the geometries precisely in the scene, import the reference picture and place it as a transparent overlay in front of your camera :
Import the reference picture in your Unity project.
Right-click on your camera in the "Hierarchy" tab, menu "PredictSuite/Camera/Add reference picture plane".
In the selection window, select your reference picture (the reference picture must be in the Assets folder).
Once the overlay is ready, move your geometries in the scene to fit precisely their position in the reference picture.
You can change the transparency of the plane by changing the "Color" field on the "Image" component.
When all the geometries are placed precisely, make sure to disable the Canvas GameObject. Otherwise it will be visible when you start the Predict Engine render.
All geometries in the scene must be assigned a measured material. In Predict Engine, a material defines the separation between two mediums on each side of the mesh the material is applied on.
When an object is made of multiple parts with different materials, its mesh must be separated into separate sub-meshes such that each sub-mesh is associated with a single material.
In the specific case of a Dielectric of Conductor partially contained within another Dielectric, the surfaces of the meshes should be cut along the change of mediums. For instance, a partially filled glass of water needs 3 materials and sub-meshes for each combination of mediums between air, water and glass: air-water, air-glass, water-glass.
It is the direction of the normal of the surface that indicates the external medium. See the detailed documentation for more details on normals.
Some material require properly defined texture coordinates on the mesh to function as expected.
Lights are usually built from AreaLights with a specific mesh as a support for the shape of the light source. They need to be positioned and oriented in the same way as their real counterpart and be associated with the correct material and emission. See the lights section for more details.
5) Simulation and export
5) Simulation and export
The last step before starting the rendering is to define the camera as a Photographic Camera or Bayer Camera and use the measured response curves of the camera sensor for the channel R, G and B. Once your scene is ready, you can start the application to get the Predict Engine render. Don't forget to set the correct resolution and aspect ratio in the Unity Game tab.
Analyse the iso-photography files
Analyse the iso-photography files
Once all of the pictures and simulations are ready, you should have a set of RAW files from the camera and EXR files from the Predict Simulations.
7) RAW files calibration
7) RAW files calibration
RAW pictures and simulations must first be calibrated :
De-bayerisation : For each pixel of a camera, the sensor is actually made of 4 different photosensors arranged in a grid, each photosensor is specialized in detecting either red, green or blue wavelengths. The raw output of the camera is actually a grayscale image that contains these 4 photosensors output for each required pixel. In order to get an RGB image as we are used to, the 4 subpixels must be recombined to get an RGB color. During the calibration process, the raw image is de-bayerised in order to fallback to an RGB image.
Color management : once debayerised the colors are expressed in the raw colorspace of the camera so a conversion to the SRGBD65 colorspace is applied as it is the most commonly used.
Exposure management : when comparing a photo with a simulation, it is important that both image are rendered with the same exposure. If needed, during the calibration process, the exposure of the simulation is corrected in order to match the reference picture exposure.
Figure 5 : example of a bayer photosensor grid
https://en.wikipedia.org/wiki/Bayer_filter
The calibration tab enables you to :
List all the pairs of photos and simulations you want to process : the files must be given by pair as they will be treated simultaneously. You can reorder the list manually, or match the files by their names using the reordering button at the top. You can import the files one by one or select an entire folder at once.
Give a reference to the folder that contains the calibration files of the camera : the folder should at least contain a subfolder named "XYZ" that contains the "color_matrix.json" file. The json file must contain the camera matrix in the following example format : {"matrix":[[1.5,0.3,0.06],[0.6,0.9,-0.2],[0.07,-0.1,1.4]]}.
Choose which files you want to export,
Choose the folder in which calibrated files will be exported : a subfolder named "01_calibrated" will be created inside the given folder, containing one folder for the photos and one for the simulations.
Export the calibrated images.
7) Post process and validation
7) Post process and validation
Once raw images were pre-processed they can be properly compared.
The validation tab enables you to :
List all the pairs of pre-processed photos and simulations you want to process : the files must be given by pair as they will be treated simultaneously. You can reorder the list manually, or match the files by their names using the reordering button at the top. You can import the files one by one or select an entire folder at once. You can select a pair using the little eye on the right to preview it in the preview widget. The preview works as the one in the Image Comparison widget : you can choose to preview files with an horizontal slider, in a parrallel view, or by displaying the DeltaE between the two images.
Define the tone mapping process that will be applied to each image :
White balance : you can define the color that will be used as the reference white in the balance for each file separately.
When using the picking button, the value will be picked from the corresponding image. The first click will select a central pixel and the second click will define a zone around this pixel. The color used as white reference will be the average of every pixel in this zone.
If the white reference as been set, you can enable the "Preserve exposure" option: in this case the image is divided by the white reference, then multiplied by the average of the white reference RGB channels, in order to preserve the global exposure.
Exposure : you can apply a linear exposure factor to the images, the same factor will be applied to the both of them.
Delta E : The DeltaE is a metric that defines the distance between two colors. It is an analysis metric used to define whether or not two colors can be differentiated : typically it is accepted that when the DeltaE is inferior to 2.0 the colors cannot be differentiated by the human eye. We are currently using the CIEDE2000 formula published in 2001. The DeltaE display settings can be defined in the settings widget on the left and/or in the preview widget footer :
the color map used to visualize the DeltaE values,
the minimum and maximum values displayed by this color map,
the threshold value (displayed with a white line on the colormap preview),
a "View threshold" option to highlight points that are above/bellow the given threshold.
Choose which files you want to export,
Choose the folder in which calibrated files will be exported :
a subfolder named "02_propcessed" will be created inside the given folder, containing one folder for the photos and one for the simulations,
a subfolder named "03_delta_e" will be created inside the given folder, containing all the deltaE outputs
Export the post processed images.
Page updated
Report abuse