Multi Machine Considerations and Determinism

Updated: 28 Nov 2025

When making real-time content for large canvasses with large resolutions, it is pretty common to need to render your content from multiple machines. By using different machines for different sections of the canvas, you can make sure that you still get good real-time performance across a large resolution, as each machine is only responsible for rendering a part of the total resolution.

In practice, this necessitates some considerations and technical requirements in order to deliver seamless content across the entire canvas. In any section that has a seam between where two separate outputs meet, you need to ensure that all content is perfectly synchronised and identical in order to avoid any visible seams. To do this, you need to be aware of both hardware requirements and design choices within Notch. These requirements relate to time, resolution and pixels, and input data.

Determinism #

If a system is fully deterministic, that means that given a common (identical) set of input variables, the resulting output will be identical every time. This is a really important consideration when working with multi-machine setups, as we need all of our content to be completely identical across all machines. Notch has been designed to ensure all systems are as deterministic as possible.

In many cases, performance is always fully deterministic, without any need for further considerations. However, there are still cases where certain requirements need to be met in order to assure that looks are deterministic across multiple machines. These requirements relate to resolution, and both time and input synchronisation.

Considerations Relating to Time and Time Deltas #

Its important to make sure that systems are able to provide the conditions needed for looks to be split across multiple machines.

Consider a simulation, such as a smoke simulation, or a particle system running on two or more independent servers. These systems are going to start at a given time, and then develop, frame-by-frame. In order to ensure that these systems are identical across all of these machines, we need to be sure that:

• The simulations start at the same time.
• The simulations progress at the same rate.
• The two machines are rendering frames at the same time.

In order to achieve this on a hardware level, we need to use Genlock and Frame lock to make sure that the clocks and outputs of all the servers are both synchronised. Genlock keeps all servers’ clocks aligned; Frame lock ensures GPUs output the same frame at the same moment. The setup of these will differ depending on the media server, but essentially we need to ensure that every machine is receiving a timing signal (usually from a designated piece of hardware). This timing signal is then used by every machine to dictate when it will render frames, ensuring that the output of all servers remains synchronised.

With the hardware in place, the next considerations are around the performance of the content in Notch. To put it simply, we need to make sure that none of the servers drops a frame. If one machine drops a frame, and the others don’t, there is a risk that that server’s output will now be out of synch by one frame. In a particle system for example, that could mean that the movement of all the particles is delayed by one frame on that server compared to all of the others. This would become visually obvious on the canvas where the output of that server meets another’s. The particles might jump positions as they move from one screen to another, resulting in a visual seam.

For that reason it’s essential to make sure that any look used across multiple machines are optimised and able to run with enough headroom that we can ensure that no server will drop a frame. There is detailed information on assessing and optimising performance and render time in Managing Performance in Media Severs.

Considerations Relating to Pixels, Resolutions and Screen Boundaries #

Other areas of graphics require considerations around the size and resolutions of our canvas, especially in areas where two separate outputs meet. This usually becomes a consideration where a process’ output value for any given value is determined by the input values of the pixels that surround it.

A Post-FX node’s output works by taking the values of all of the pixels for a region, and performing an operation, that then results in the output of the Post-FX. In a few cases, such as the Glow Post-FX, the output value of each pixel is determined by the pixels around it as well. In a single server setup, this is absolutely fine. But in a multi-server setup, this creates an additional factor to consider. If we think of the two pixels either side of a boundary, where the output of two separate servers meets: The pixel on on side of the boundary is not friendly with their neighbours - they know nothing about their neighbouring pixels on the other side of the boundary. Because of that, they will have a different output than they would if there was no border there. The result of this is a visible seam, at the boundary where the outputs meet.

Luckily, there is a way to resolve this, which is introducing over-scan. What that means is, we get each server to render more content than is strictly needed for it’s section of the canvas. Then, in the media server, we crop out the extra content, so that the outputs meet at the same boundary again. In doing this, we’ve allowed those pixels on either side of the border to meet their neighbours, get friendly, and work together to create a consistent result. This then removes any visible seam. The downside of this is that we’ve had to render more pixels on each server than was strictly necessary, which will have an impact on performance. The amount of over-scan that we need (the amount of extra content we need to start rendering on each machine), will be determined by the effect size. If we want our effect to work across larger areas, we will need to introduce more over-scan.

Considerations Relating to Input Data and Timing #

From our definition of determinism: Given a common (identical) set of input variables, the resulting output will be identical every time. It is fairly obvious that in order to ensure it, we need to make sure that all of the input variables to all of our machines are identical in both values and timing.

This means that anywhere Notch content’s output is determined by incoming data, that incoming data needs to be arriving at exactly the same time, with exactly the same values, to every server. This applies to video feeds, tracking data, audio data, and anywhere else that an input is being used to make content.

For video inputs, a synchronised video system is needed that ensures any external feeds are arriving at the exact same time.

For any hardware, tracking systems or network messaging, a network must be capable of reliably getting all of the necessary packets of data to the right place at the right time.

Similarly, any audio system that is driving Notch content must ensure that the latency is consistent, and that audio is arriving at all machines at the exact same time.