Method

This node searches an image for faces. When a face is found a pre-defined arrangement of vertices, forming a simple 2D mesh, are searched for and tracked across multiple frames.

Example Image
Example Nodegraph

Face tracking is based on machine learning. It uses a preprocessed database generated from hundreds of different images of faces to recognise new faces in the images given to it. This database is required by the Face Tracker node; it is loaded as a “Machine Learning Database” resource and is set via the “Training Database” parameter.

When the Face Tracking node finds a face it generates a mesh which may be used as geometry by the Face Tracking Geometry node, or used to deform a matching 3D object’s vertices using the Face Tracking Deformer node. The image ouput by the Face Tracking node can either pass through the original input; or output a mask from the detected face geometry in either the RGB or Alpha channels. The Face Tracker also analyses the detected face pose to determine whether the eyes and mouth are open, the positions of several key points such as the eyes, and the overall approximate rotation and scale of the face. These values are available as envelopes that can be extracted from the node via an Extractor modifier and can for example be linked to Image2D nodes in order to attach 2D images to points on the face. The Face Found envelope is set to 0 if no face is found, or 1 if there is.

Face tracking is CPU-intensive; face recognition takes 1-2 frames on CPU. In order to keep rendering performant this is performed in parallel to the rest of the graphics processing in the frame. As such there is an inherent additional latency in the output of the node – typically two frames, but this may be reduced to one frame using the “Low Latency” option at the cost of reduced frame rate. To simplify use, the image output from the Face Tracking node is delayed so it is in sync with the geometry generated by the node: if you are using the face geometry from the detected face with a Face Tracking Geometry node, you can use the image stored on the Face Tracker node to texture it and they will be in sync.

In order to improve performance further, there is the option to downsample the input image before face detection – as this typically does not have to be processed at full resolution to find faces. Downsample Steps defines the number of downsample iterations: 1 means half width, half height and so on.

It must be noted that face tracking, while reasonably solid and robust, is not perfect. There is a degree of tolerance but care must be taken with the input feed to ensure that the algorithm is able to perform to its best. In particular:
- Face detection works best in good, balanced, even lighting, with a subject facing straight on to the camera and good image quality. The technique is not intended to work with moving cameras but is tolerant to some degree.
- Excessive facial hair, hats, novelty glasses, low frame rates and fast motion may impact the result.
- The face must be at least 300 pixels high in the source image. Smaller faces will most likely not be detected.
- The face detection algorithm can detect multiple faces in a single image, but at present only one face is tracked. To keep consistency the largest face in the image is chosen each frame.

Attributes

Name Description
Preview In Viewport Preview the effect blended with alpha in the viewport.
Preview Alpha In Viewport Preview the alpha values in the viewport.
Downsample Steps The number of times the input image is halved in size before face detection is executed – to increase performance.
Frame Skips The number of input frames that are skipped. A 60hz feed may be processed at 30hz to increase overall system performance.
Low Latency Reduce the number of frames between processing and getting the result, at the cost of reduced overall performance.
Output Image Mode Defines the image generated by the node.
  • Original Image, The original image is output with no modifications.
  • Face Mask, A white mask is generated against a black background.
  • Face Mask + RGB, A mask is generated and multiplied against the original image, against a black background.
  • Original Image with Alpha Mask, the original image is output, with the faecemask applied to the alpha.
Show Bounding Boxes Render bounding boxes of detected faces on screen.
Training Database The Machine Learning Database resource – required for face tracking to work.

Inputs

Name Description Typical Node Input
Effect Mask Mask out areas the effect wont affect the image. Image Plane
Alpha Image Add an image to control the alpha values of the image. Video Loader

Outputs

There are no outputs from this node.