Intel RealSense 2.0 Camera Setup

Introduction #

Just like the Kinect, the Intel RealSense can be used to with Notch in a number of ways, such as generating meshes, emitting particles, generating fields, and using the depth and colour images in video processing chains.

The suitability of RealSense for production environments is entirely at your own risk. Internal and external user testing has highlighted significant issues in aligning operating system, driver and firmware versions. Driver management is extremely difficult with the device due to two paths of driver delivery (packaged in Windows 10 OS updates and separate driver downloads). We advise significant caution when considering this device for any production environment.

Notch has been updated to support the RealSense SDK 2.31.0 - using RealSense cameras with newer or older driver SKDs may have unpredictable results.

Setup #

Firstly, you will need an Intel RealSense depth camera. Notch Uses the official SDKs to integrate with the RealSense camera. Please note that you should download the SDKs first before downloading any other softwares or using the camera. The Intel RealSense SDK’s support cameras SR300 and D400 Series. (“SDK V2.0”:https://realsense.intel.com/sdk-2/#install)

Connect your RealSense camera to your PC and open Notch Builder. Go to menu Devices > VideoIn/Camera/Kinect Settings to find the Video In settings. In the Subheading of Intel RealSense tick “RealSense Enabled” and press OK.

To see if the camera is working you will need to connect the Kinect Mesh node to the root, then link the Kinect Source node to the Kinect mesh Colour Image Input as shown in the example below. Press play and the camera should be working.

Adding RealSense sources to your scene #

Add a Video Kinect Source Node to the Nodegraph (Video Processing > Input Output). This is the source node for any RealSense related outputs. This node provides both the RGB and depth image from the RealSense camera, and by default, it generates an alpha channel which is clipped using the depth image and the near and far planes set on the node. You may also output depth from this node as luminance in the RGB channel. Connecting it to a Kinect Mesh node will allow you to visualize what the RealSense camera can see in 3D space. You need one Video Kinect Source for each physical RealSense camera in use.

Nodes that use Intel RealSense source #

Video nodes can use Kinect RGB or Depth images as source as inputs. However, the following nodes are specifically made for use with the RealSense camera:

• Depth Camera / Kinect Colour Key
• Depth Camera / Kinect Mesh, Generates a mesh from a RealSense source.
• Depth Camera / Kinect Source, source node for any effect using Kinect.

Using multiple RealSense cameras with Notch #

While Notch supports multiple RealSense cameras (up to 4), the PC hardware / USB buses have limitations on the number of RealSense cameras, It depends on what machine you are working with. This has not been tested however, a user has managed to get up to 4 working.

Use of a RealSense Camera is mutually exclusive to any other depth camera (e.g. Kinect) - it is not possible to mix the RealSense cameras, Kinect1 and Kinect2 in one project.

Skeletal Data #

There is no Skeletal data with the RealSense camera.

Recording RealSense Data #

Notch does not currently support recording RealSense data.

Considerations for the use of the RealSense camera in performance environments #

Working with a RealSense Camera can take some persistence. The cameras were not designed specifically for performance environments and the driver/firmware provided by the vendor is sometimes in flux. If you do not have any experience with the use of these cameras we highly recommend engaging with an experience interaction/integration specialist. Specific considerations:

• IR Interference: Incandescent light sources (including the sun, and some models of strobe) significantly degrade the real sense depth image, making it unusable.
• Signal Extension - as the camera is fairly new we do not know the achievability of the camera’s extension.
• Field of View : Both versions of the realsense cameras have very particular field of views and both versions depth and RGB fov’s differ.

Troubleshooting #

My RealSense camera does not show up in Notch. What should I do?

• Does the camera work outside of Notch? There are two tool softwares that come with the SDK such as Intel Depth Quality Tool and Intel RealSense Viewer.
• Close all other software that may be accessing the camera (including Intel’s own tools).
• Replugging the RealSense camera while using Notch will not work. Please restart Notch.
• Check you are using a USB 3.0 port and cable.
• Please turn off all anit-virus software. We have seen some issues with popular anit-virus softwares which can interfear with the camera.

The frame rate of the RGB camera appears poor, even though Notch’s frame rate is fine..

• Please be aware that the Intel RealSense Cameras frame rate is limited speed by your USB Bus. Having multiple cameras on one PC will also affect this, try disconnecting a few cameras.
• If your scene only requires the depth camera try disabling the colour channel in the Video In/ Kinect settings dialog. This will reduce the amount of bandwidth used.
• There is a known issue with the RGB camera not working smoothly on the Disguise GX range.

If you are still struggling to use your RealSense camera with Notch please get in contact with us at support@notch.one.