Build the Ultimate Special Effects Light Tool with RICOH THETA and OpenCV

RICOH THETA is an extremely popular camera that captures 360° images, video, and even live streams. Combined with the OpenCV library for computer vision, this lightweight, low-cost and highly portable device can become the ultimate tool for visual special effects (VFX).

RICOH THETA runs the Android OS on an eight-core Arm Cortex-A53 platform. This is more than enough juice to perform advanced graphics processing within the camera, including creating HDR files, tracking objects in real-time, or identifying and classifying objects with deep learning.

In addition to OpenCV, the camera can also supports the Qualcomm FastCV library internally to take advantage of the Qualcomm Adreno 506 GPU inside.

With so much processing power and advanced image sensors with fisheye lenses, the camera can be modified for use in a dazzling array of applications that deal with images, light, or video.

One powerful application of the 360° camera is creating HDR files. The light information from the HDR file is then utilized to generate reflection on 3D objects. In other words, VFX professionals are not using the camera to take pictures. Instead, they are employing the camera to grab lighting data from a specific time and location that they can use when build CG objects. This makes the CG objects look extremely realistic.

You can get a better idea of how to use an image as a light source in the following example with a 2D (non-360°) image.

In this example, a 2D image is used to project light onto a glass surface.

The light source is reflected on the glass.

If the image is in 360 degrees, then the entire scene will be illuminated. In the scene below, there is a test mirrored ball.

The HDR file for the light is produced by taking 11 shots at different exposures.

The images are combined together into a single HDR file inside the camera. The image retains all light information from all the exposures. The image is used as the skybox of the scene.

You can adjust the light direction.

Kasper Oerlemans created the open source plug-in HDR2EXR that runs inside the RICOH THETA. It is focused at VFX professionals who need to jump on-set, grab highly accurate lighting data without getting in the way of the production flow. The highly accurate lighting information will then be applied to CG characters later.

After automatically combining 11 images, the HDR file will be 160MB in size and contain 32-bit light information.

You can tinker with the open source code in Android Studio.

You can easily edit these constants at the top of the code in MainActivity.java:

• numberOfPictures — number of pictures in the HDR file. Set to 11 pictures by default

• shutterSpeedSpacing — adjust shutter speed between each of the images

The plug-in relies on the Android NDK toolset to run OpenCV natively inside of the camera.

To receive a guide with over 25 source code examples — including HDR2EXR, FastCV example, OpenCV example, and TensorFlow demos — sign up for the RICOH THETA Dream and Build Contest. Aside from the code examples, you’ll receive an asset pack of 360 video and images for you to experiment on. If you don’t have RICOH THETA, you can still test out the concepts in an Android Virtual Device (AVD).

The RICOH THETA Dream and Build Contest is currently live on Hackster. 10 participants who apply before April 16th will receive a RICOH THETA to develop with. Then you’ll have time and resources to build a customized plug-in to make the camera do what you want.

The friendly and active theta360.guide developer community is available to get assistance from other developers just like you.