Rob Chandler, Full-Service Virtual Production Founder – Morden Wolf & Starting Pixel, February 12, 2024
“Virtual production (VP) blends traditional filmmaking with cutting-edge technology such as LED walls, motion capture, augmented reality (AR), and advanced computer graphics. Even perceived ‘old’ tech such as green screens have developed, allowing for stunning live-to-tape filmmaking. Whether filming a movie, music video, advert, corporate explainer, or social media reel, VP allows directors to push creative boundaries beyond budget when combined with locations and sets, or as an entire virtual production.
However, this is a double-edged sword. The greater creative freedom given by VP also carries greater technical knowledge and risks. This can be a challenge for directors, who are responsible for the overall success of any type of production.
Although, this shouldn’t deter directors from dipping their toes into VP for all or part of a production (or taking the whole plunge). With careful preparation and proper management, directors can harness the full potential of virtual production while mitigating the risks associated with unfamiliar technology.
The Director’s Role in Virtual Production
In any type of traditional production (movie, music video, advert, corporate explainer, or social media reel etc), the director is the creative leader who shapes the narrative, guides talent, makes artistic decisions, and oversees the film’s overall visual style to bring the script to life.
In virtual production, the director must do all of that – and more. They must not only understand the narrative and performance aspects but also have a solid grasp of ‘the art of the possible’ when it comes to VP.
This new addition to their craft includes but is not limited to understanding how camera movements interact with CGI elements in real-time, impact of physical and virtual lighting, when and how to use 2D vs 2.5D vs 3D backplates, the numerous ways to capture ‘in camera’ or take it to post, the effective communicating with the technical team to deliver the vision and being familiar with the capabilities, limitations and when to use the VP tools and workflows to hand. (The list goes on, and there’s far too much to put here).
Once a director has a solid grasp of the tech possibilities (and challenges), their ability to fulfil their traditional role is greatly enhanced. For example, through the use of Unreal Engine and VR headsets, directors can explore the virtual environment development as they are built, long before the shoot. Also, if so inclined, armed with some basic know-how, a director can contribute to or own the physicality of a virtual environment themselves. Specifically for indie filmmakers, this is an incredibly democratising use of the technology.
Furthermore, unlike a traditional film set, virtual environments can be adjusted on set, lighting, movement of windows, deleting a tree etc, allowing space for instant creative decisions. However, this approach has to be taken with great caution, and requires experience and clear communications with the on-set VP tech team. Moving a virtual prop a just a few metres seems only to take seconds, but the untested knock-on effect may cause lighting/ colour abortions or hardware performance issues that will inevitably cause delays on set.
This is why collaboration is so much more important in a VP shoot. Directors must work more closely with visual effects teams, virtual art departments (VAD), technical directors and virtual production supervisors throughout the production, especially in the pre-production phase. This means balancing their needs, tapping into expertise and communicating clearly and effectively in a language that can be understood on both sides. For example, actors and expense have a very different meaning in the world of game development.
Just like traditional filming, the type, complexity and style of a production dictates how hands-on a director can or should be. Larger productions have more roles and teams. The same goes for VP, allowing for roles and departments such as VAD (virtual art department), VP supervisors, Head of Virtual Production, VP Producer, Virtual Cinematographer and even niche roles such as Virtual Lighting Gaffer. Small productions will inevitably combine roles, the VP supervisor may also be the UE tech artist, virtual gaffer and key-puller. Just like a DP will cam op and gaffer. Likewise the director maybe the scriptwriter, UE VFX artist and colourist/editor.
One key stumbling block to consider is who is responsible for what and when. Currently, even within the key VFX companies, there is an inconsistency to VP related job titles,roles and responsibilities. The good news is that ‘Virtual production supervisor’ is seen as the key title on a production, however, some see this as an on-set hands-on role, others (and especially those in the VP world) see the role as being integral part of a production from the very start and wholly responsible for the virtual production aspect. Quite often, they straddle the two, especially with smaller productions.
As a result, directors can become paralysed by analysis, and rightly so, unclear who will be doing what and when, with a huge reliance on trust.
One key role the director needs to understand the role of VP is the producer. A simple misunderstanding can lead to the VP supervisor being engaged too late to advise on technology choices, creative options, and methodologies which may have saved vast amounts of time and money to achieve the director’s vision. This is typically seen in the commercial productions, as producers are typically engaged after strategy and creative decisions have been made, when they are tasked with the delivery. To get the most out of the magic of virtual production, briefing media strategists and creative teams on ‘the art of the possible’ will not only save time and money, but broaden the creative sandpit, increasing variations and allowing for innovation and exceeding client expectations.
Pre-Production Challenges and Solutions
In traditional pre-production planning, directors focus on script breakdown, storyboarding, scheduling, casting, location scouting, and collaborating with the heads of departments to develop the visual style and feel of the film.
Much of this is the same for VP. For any VP parts of the shoot, there will be less physical location scouting due to the ability to use virtual environments. These environments can be 3D scans (the use of gaussian splatting has greatly reduced the entry point cost), early drafts of fictional realistic locations, abstract sets, digital twins of a real location )for call-backs or unrealistic to film in (eg Time Sq at any time). So phases like storyboarding, the use of virtual location scouting (via a VR headset) and pre-viz within Unreal Engine will positively impact time, budget and CO2 impact.
The biggest difference, however, is that in VP, pre-production planning takes on even greater importance due to the VP workflow which moves from a traditional “fix in post” methodology to a “fix in pre” one. Taking this approach dramatically increases productivity, saving budget and time.
However, this means directors must have a clear vision much earlier in the process. They must work closely with VP Supervisor or VP HOD from the start to understand the capabilities and limitations of the VP tools and shots on a VP stage at their disposal.
This shift brings about new challenges, too. Directors must ensure that the technology serves the story rather than dictates it, manage the often higher costs associated with tech, and plan with detail to avoid costly changes later.
Directors must be involved in decisions about the scale, limitations and technical specifications of LED volumes, why and when green screen virtual production is the right choice (no matter how many groans there are), the camera setup, and the overall visual aesthetic of the virtual environment. This process also includes a highly valuable pre-visualization (previs) phase to shoot scenes, shot block and test camera movements, resulting in a pre-viz roughcut that significantly reduces the need for on-set adjustments.
Pre-light, tech, shot and virtual scout days are equally, if not more, important, as they provide an opportunity to test the combination of physical and virtual lighting, camera movements, camera settings, set layout, and shot angles, ensuring that the filming days aren’t delayed by foreseeable and assumptive challenges.
Likewise, understanding colour management in virtual production is crucial. The approach needs to be fully considered before the technology and type of virtual production are chosen. Another important reason to get a VP supervisor involved at an early stage.
Size, set-up, and format of an LED volume will impact the type of in-camera photography, while green screen VP is more flexible in post while affording on-set PreVis and instantly available slap comps for rough cuts. Awareness of these options is key. While making decisions too late will inevitably impact time and budget resources, but also a missed opportunity to really lean in to the possibilities that combining live action with immersive CGI can offer.
Remember that these decisions can be unique to the scene or even shot. Like a decision on location vs set build, virtual production gives a director options, and it doesn’t need to be one choice for the entire production. This is being helped by the growth of ‘smart stages’ where manageable LED volumes are located in the same space as a green or blue screen, allowing for incredible flexibility and agility.” Continue reading more
Rob Chandler’s Linkedin page: https://www.linkedin.com/pulse/directing-virtual-productions-whats-difference-rob-chandler-pxvce/