1st Edition

The VES Handbook of Virtual Production

Edited By Susan Zwerman, Jeffrey A. Okun Copyright 2024
    454 Pages 172 Color Illustrations
    by Focal Press

    454 Pages 172 Color Illustrations
    by Focal Press

    454 Pages 172 Color Illustrations
    by Focal Press

    The VES Handbook of Virtual Production is a comprehensive guide to everything about virtual production available today – from pre-production to digital character creation, building a stage, choosing LED panels, setting up Volume Control, in-camera compositing of live action and CG elements, Virtual Art Departments, Virtual Previs and scouting, best practices and much more.

    Current and forward-looking, this book covers everything one may need to know to execute a successful virtual production project – including when it is best to use virtual production and when it is not. More than 80 industry leaders in all fields of virtual production share their knowledge, experiences, techniques, and best practices. The text also features charts, technical drawings, color images, and an extensive glossary of virtual production terms.

    The VES Handbook of Virtual Production is a vital resource for anyone wishing to gain essential knowledge in all aspects of virtual production. This is a must-have book for both aspiring and veteran professionals. It has been carefully compiled by the editors of The VES Handbook of Visual Effects.

     Chapter 1. What is Virtual Production

    Definition of Virtual Production / Types of Virtual Production

    What Types of Scenes Work Well for Virtual Production

    Greenscreen for Virtual Production

    Chapter 2. Advantages and Disadvantages of Shooting in Virtual Production

    Advantages and Disadvantage of Using LED Volumes for Production

    Chapter 3. How the Virtual Production Supervisor and Producer Relate to Other Departments

    Virtual Production Supervisor Relationships

    How the VFX Team on a Virtual Production Show Interfaces with Other Departments

    Production Design for Virtual Production - What Has Changed, What Has Not?

    Chapter 4. How To Shoot and Edit Animation Using Live-Action Virtual Production

    Introduction: The Creative Story Process

    Performance and Motion Capture for Animation

    Virtual Camera and Director’s Layout

    Creative Editorial

    Chapter 5. Visualization

    What is Storyvis

    Previs/Mocap Pipeline and Flow chart

    VR Scouting

    VCAM - Virtual Camera System Used During Prep

    Techvis and Motion Control Techvis


    VP Tools for On-Set Visualization



    Chapter 6. What Roles Are Needed for Virtual Production?

    Overview of Virtual Production Staffing and Organization

    Producing Virtual Production

    Chapter 7. VAD (Virtual Art Department)

    Definition of VAD / Workflow Integration

    Artists for Asset Creation/Sets

    Building Assets in Game Engines/Unreal

    Managing Asset Quality and Performance Needs for Virtual Production

    Virtual Scouting with Production Design - VR System

    Virtual Pre-lighting with the DP

    Multi-user Virtual Location Scouting – Pre-production

    What Are the Differences Between a Multi-User Virtual Scout and Other Scouting Processes?

    Techvis for Art Departments

    Optimization and Delivery

    Digital Content Creation Software Used in VAD

    Content Library - Digital Backlot of Assets

    Version Control Software

    Chapter 8. Digital Asset Creation

    Building 3D Assets in a VFX Facility

    Capturing Surfaces with Photogrammetry to Create Assets

    Procedural Digital Asset Creation

    Move AI - Markerless Motion Capture

    Using Miniatures for Virtual Production

    Preparing 2D Plates for Virtual Production

    Chapter 9. Volumetric Capture for Humans

    Overview of Volumetric Capture

    Volumetric Capture / Considerations for Production - Wardrobe, Hair, Makeup, Blocking

    Volumetric Capture with Actors for Virtual Production

    Chapter 10. How to Make Characters for Use in Virtual Production

    Smart Simulated Characters (Digital Extras)

    MetaHumans: Creating Digital Humans for Production

    Animating MetaHumans with Motion Capture

    Chapter 11. How to Capture Environments for LED Walls

    Photogrammetry and LiDAR Sets for VP

    Photogrammetry Optimization

    Plate Photography for Playback on LED Walls

    Chapter 12. Process Work on LED Stage

    General Understanding of Process Work

    Break it Down Into Techniques

    Chapter 13. Projects Shot on an LED Stage

    LED Wall ICVFX

    LED Volume - 3D Tracking Full Volume versus 2D Playback on Set

    Chapter 14. Challenges and Limitations of Shooting in a Volume

    Limitations To Be Aware of When Shooting In a Volume

    Moiré, Viewing Angle, Banding, Latency, Artifacts, and Frame Sync

    Color Gamut and Color Shift

    Sound Dampening

    Post Cleanup of the LED Wall

    Chapter 15. Setting Up an LED Volume Stage - Permanent versus Pop Up

    Setting up a Full Volume for LED Walls

    Stage Designs

    Mobile LED Stages and Walls

    LED Display Wall - LED Modules, Processors, and Rigging

    Display Considerations

    Chapter 16. LED Display Technology and Hardware

    Characteristics of LED Displays

    Color Consistency and Predictability

    LED Display Validation Protocols

    What Are the Aspects of Choosing a Panel

    Types of Tools That Connect Cameras, Monitors and LED Displays to Computers

    Key Characteristics of LED Display Processors

    LED, OLED, and Playback Servers

    Chapter 17. Volume Control - Brain Bar Labor

    Description of Labor types for the Brain Bar

    Rendering and Control Computers

    Chapter 18. Software for Virtual Production

    Virtual Production Software

    Types of Content for On-Set Virtual Production

    On-Set Applications

    Media Servers for Filmmaking

    Chapter 19. Cameras for Virtual Production

    Characteristics of Cameras for Virtual Production

    LED Processing for Camera -  Frame Remapping for ICVFX

    LED Processing for Camera - GhostFrame

    Calibration and Lens Metadata

    Chapter 20. Camera Tracking for Virtual Production

    Overview of Inside-Out and Outside-In Tracking Systems

    Outside-In Tracking Systems

    Physical Tracker - Inertial / Optical / Hybrid / Encoder Based Solutions

    Chapter 21. Introduction to Color Management for LED Walls

    The Definition for Matching

    Rendering a Virtual Scene to Produce Linear Light

    Creating a Signal from Linear Light

    LED Processor Conversions

    Photography of LED Panels

    Preview of Camera Look in Real-Time Engine

    Limitations of LED Wall Panels

    Chapter 22. Art Dept Integration in the Digital and Physical World

    Getting the Environment Ready for the Cinematographer and Lighting in the Volume

    Blending the Physical and Digital World

    Workflow of Shooting on LED Wall

    Real-Time Compositing Between Physical and Digital World

    Chapter 23. Lighting Types for Virtual Production (LED Volume)

    DPs on Lighting for a LED Volume

    Lighting for the Outer Frustum

    Using the LED Wall as a Direct Reflection Lighting Tool

    External Lighting on Stage versus LED Wall Lighting

    On-Set Image Based Lighting (OS IBL) / Lights with Special Integration for Virtual Production

    Lighting Control Systems, Pixel Mapping and Protocols with Virtual Production

    Chapter 24. Epic Training (Unreal Engine)

    Training In-Camera Visual Effects (ICVFX) for Virtual Production

    Chapter 25. Unity Software - Overview/Aspects of Virtual Production


    World Building in Unity

    Camera Tracking in Unity

    Simulcam with GS Shooting - Live Set Composting

    Ziva Dynamics- Photographic (Realistic) People, Animals, and Creatures

    Chapter 26. The Future of Virtual Production

    Overview of the Future

    Beyond Flat Screens for On-Set Virtual Production Stages

    Alternative Display Technology: Projection Based Volume Using White Cyc

    High Resolution Depth (HD3D)

    MicroLED - High-End Display and Resolution

    Full Gamut Color- RGBW

    NVIDIA Omniverse and Artificial Intelligence

    What is Next? - Beyond the Volume


    Appendix A: Charts and Technical Diagrams

    Appendix B: : The Virtual Production Glossary


    Susan Zwerman, VES, is an experienced Visual Effects Producer with a passion for cutting-edge film production. She is highly respected for her expertise in visual effects and virtual production budgeting and scheduling. As chair of the DGA UPM/AD VFX Digital Technology Committee, Susan organizes virtual production seminars to introduce members to this exciting and evolving new technology. Susan received the Frank Capra Achievement Award in recognition of career achievement and service to the industry and the Directors Guild of America in 2013. She is a member of the Academy of Motion Picture Arts and Sciences, the Producers Guild of America, the Directors Guild of America, and a member and Fellow of the VES.

    Jeffrey A. Okun, VES, is an award-winning Visual Effects Supervisor who is more than conversant with virtual production. He is a member and Fellow of the VES and a member of The Academy of Motion Pictures, Arts and Sciences, the American Society of Cinematographers (ASC), the Television Academy, and the Editor’s Guild. Okun created visual effects tracking and bidding software in 1992 that is still in wide use within the industry today, as well as the revolutionary visual effects techniques dubbed the "PeriWinkle Effect" and the "Pencil Effect" – a predictive budgeting tool. He is also a noted 60s, 70s, and 80s rock and roll photographer.