This node searches the supplied image or video for faces. if a face is found, a mesh is generated then tracked along with the image.
Face tracking using this method is based on machine learning. It uses a preprocessed database generated from hundreds of different images of faces to recognise new faces in the images given to it. This database is required by the Face Tracker node; it is loaded as a “Machine Learning Database” resource and is set via the “Training Database” parameter.
Face tracking with this method is very is CPU-intensive; face recognition takes 1-2 frames on CPU. In order to keep rendering performant, this is performed in parallel to the rest of the graphics processing in the frame. As such, there is an inherent additional latency in the output of the node – typically two frames, but this may be reduced to one frame using the “Low Latency” option at the cost of reduced frame rate. To simplify use, the image output from the Face Tracking node is delayed so it is in sync with the geometry generated by the node: if you are using the face geometry from the detected face with a Face Tracking Geometry node, you can use the image stored on the Face Tracker node to texture it and they will be in sync.
NVIDIA Face Tracking (RTX Only)
This method uses the NVIDIA RTX AI based face tracking solution to generate a face mesh, and track the points between frames. This features requires an NVIDIA RTX enabled graphics card, as well as the NVIDIA AR drivers (available here). Running on the GPU, this method is far faster than the CPU based alternative and can be used with a 3D Face mesh instead of a 2D mesh.
This node uses the NVIDIA Broadcast Engine AR SDK as well as a dataset that needs to be manually downloaded, compiled, and copied into the NVIDIA AR SDK folder (due to licensing). This node will not work unless you follow these steps to set up your system prior to launching Notch Builder. These steps are taken directly from the official NVIDIA documentation and are repeated here for convenience.
- Make sure Notch Builder is not running
- Download and install the NVIDIA Broadcast Engine AR SDK
- Generate a 3D Morphable Face Model (3DMM)
- Create a temporary folder and download all the following files:
- Download the ConvertSurreyFaceModel.exe utility from the BROADCAST-AR-SDK project page on Github. Click “ConvertSurreyFacemodel.exe, then click the “Download” button.* Important: do not right-click and choose “Save as..”* as this will not work and you will get an error when you try to run the executable later.
- Download the following Surrey Face Model files from the eos project page on Github — OR: download the entire repo by clicking here. You’ll get a few more files than you need, but you’ll avoid download errors. The files you want are in the archive, in the “share” folder. If you want to download just the files you need, these are the ones:
- sfm_3448_edge_topology.json (beware that some browsers corrupt pure text files like JSON on download!)
- sfm_model_contours.json (beware that some browsers corrupt pure text files like JSON on download!)
- Open a Command Prompt (Start menu, CMD) and navigate to the folder where the files downloaded in step 1 and 2 can be found.
- Copy and paste this strong into your Command Prompt and hit enter to run it: ConvertSurreyFaceModel.exe --shape=sfm_shape_3448.bin --blend_shape=expression_blendshapes_3448.bin --topology=sfm_3448_edge_topology.json --contours=sfm_model_contours.json --ibug=ibug_to_sfm.txt --out=face_model0.nvf
- Copy the newly generated face_model0.nvf to the “models” folder inside your NVIDIA AR SDK models folder (C:\Program Files\NVIDIA Corporation\NVIDIA AR SDK\models).
Error: “No shape file was specified”
Make sure you’re copy-pasting the entire command from step 3d above exactly as written. There should be no extra spaces or other characters added. When run correctly, there should be no feedback in the Command Prompt after you’ve run the command, you should just have a new file in the folder.
Error: ERR#8 or “Syntax Error at DOCTYPE”
Your download of one or more of the files from Github has failed. You can download the entire directory by clicking here. You’ll get a few more files than you need, but you’ll avoid download errors.
Why does this happen? Well, when you are downloading the files from Github as described in step 3b above, make sure you do not have any content blockers or extra security settings in your browser, as they tend to block or sanitize JS-related files like JSON under certain conditions. They need to be downloaded as their RAW source format to work. One way to do this especially for the .json files is to click the “RAW” option on Github for those files, and then go to “Save page as..” in your browser.
Error: The “Face Tracking Geometry” node is marked as red in the node graph
You’re missing the NVIDIA Broadcast Engine AR SDK. Please download and install it according to the setup guide above.
Error: “Unsupported 16-bit Application” when running ConvertSurreyFaceModel.exe
Your download from Github has failed, most likely because it wasn’t downloaded using the “Download”-button, and instead using right-click “Save as..” on the wrong object. Re-download the file from step 3b in the list above, and it’ll work.
When the Face Tracking node finds a face it generates a mesh which may be used as geometry by the Face Tracking Geometry node, or used to deform a matching 3D object’s vertices using the Face Tracking Deformer node. The image output by the Face Tracking node can either pass through the original input; or output a mask from the detected face geometry in either the RGB or Alpha channels. Only one face can be detected at a time, so multi-face tracking is currently not supported.
The Face Tracker also analyses the detected face pose to determine whether the eyes and mouth are open, the positions of several key points such as the eyes, and the overall approximate rotation and scale of the face. These values are available as envelopes that can be extracted from the node via an Extractor modifier and can, for example, be linked to Image2D nodes in order to attach 2D images to points on the face. The Face Found envelope is set to 0 if no face is found, or 1 if there is.
To improve performance further, there is the option to downsample the input image before face detection – as this typically does not have to be processed at full resolution to find faces. Downsample Steps defines the number of downsample iterations: 1 means half width, half height and so on.
It must be noted that face tracking, while reasonably solid and robust, is not perfect. There is a degree of tolerance but care must be taken with the input feed to ensure that the algorithm is able to perform to its best. In particular:
- Face detection works best in good, balanced, even lighting, with a subject facing straight on to the camera and good image quality. The technique is not intended to work with moving cameras but is tolerant to some degree.
- Excessive facial hair, hats, novelty glasses, low frame rates and fast motion may impact the result.
- The face must be at least 300 pixels high in the source image. Smaller faces will most likely not be detected.
- The face detection algorithm can detect multiple faces in a single image, but at present only one face is tracked. To keep consistency the largest face in the image is chosen each frame.
|Preview In Viewport||Preview the effect blended with alpha in the viewport.|
|Preview RGB In Viewport||Preview the RGB values in the viewport.|
|Preview Alpha In Viewport||Preview the alpha values in the viewport.|
|Active||Turn the effect on or off. Values < 0.1 will turn the effect off, and deferred rendering must be on.|
|Downsample Steps||The number of times the input image is halved in size before face detection is executed – to increase performance.|
|Frame Skips||The number of input frames that are skipped. A 60hz feed may be processed at 30hz to increase overall system performance.|
|Low Latency||Reduce the number of frames between processing and getting the result, at the cost of reduced overall performance.|
|Output Image Mode|| Defines the image generated by the node.
|Show Bounding Boxes||Render bounding boxes of detected faces on screen.|
|Reset Face When Lost||Reset the face data when a face cannot be found.|
|Solver|| Which solver to use for Face tracking.
|Training Database||The Machine Learning Database resource – required for CPU face tracking to work.|
|Name||Description||Typical Node Input|
|Effect Mask||Mask out areas that Post-FX applied to this node won’t be applied.||Video Loader|
|Alpha Image||Add an image to control the alpha values of the image.||Video Loader|
Outputs an image or video that can be connected to any relevant video input, most commonly an Image 2D node.