More Thoughts on VR

I attended a one day conference on VR & Film, sponsored by SIRT and ETV Film Inc in mid-June.  There were hardware and software demonstrations, but I was particularly interested in the talks, where people involved in creating virtual reality explained the storytelling issues they encountered.

Having worked in cgi in the early years, I'm confident that the technical problems of VR will be solved over time.  My personal interest runs more towards how VR is going to communicate with audiences as a narrative medium.

Pretty much everyone agrees that VR has a resemblance to theatre in that the audience is free to look where they wish.  The stage has various techniques for directing the audience's attention, lighting being a major one.  If only one part of the stage is lit, the audience will naturally look there.  Jeff Preyra of 360 Storylabs pointed out that with a 360 degree camera, it was impossible to place lights, as they would always be visible.  For this reason, he felt that the future of VR storytelling would be motion capture avatars in cgi environments.  The virtual lights in cgi are invisible to the camera, so in a cgi environment, directors could still control lighting.

Preya felt that establishing shots would have to be longer as the audience would naturally want to look around an environment and take stock of who is present before watching whatever dramatic action is going to unfold.  This makes sense, though when returning to established locations it shouldn't be necessary and if you want to surprise the characters and the audience, you could start the action of a scene immediately to prevent the audience from knowing everything that was present.

Preyra also felt that musical scoring didn't work in VR.  As the viewer was in the scene, any music needed to have a visible source.  I'm not sure about this.  In the early years of talkies, there was music under the opening and closing titles, but none during the film unless there was a onscreen source such as a radio, phonograph or visible musicians.  However, by 1933, just a few years after talkies became the dominant form of movies, King Kong had dramatic scoring by Max Steiner throughout the film.  By the late 1930s, composers like Steiner, Newman, Korngold, Waxman, Hageman and Tiomkin were hard at work scoring films throughout their run times.

Ian Tuason of CFC Media Lab said that cameras could only move in straight lines, as any change in the camera's direction might clash with head movements of someone wearing a VR headset.  This makes sense on the face of it, but again a look at film history leads me to believe that it can be done.  Films in the 1930s and '40s routinely shot in ways where the camera's position operated separately from the camera's view.  In other words, the camera's location would physically move while the camera itself would change what it was pointing at.  What was standard, however, was someone moving on screen that gave the audience a focus.  So if a camera was tracking through a restaurant before stopping at a table where the main action was to take place, the camera would follow a waiter while it was moving.  In a VR situation, if there is an obvious center of interest on screen, like a character, the camera could move, changing it's spatial and angular relationship so long as the audience has a reason to stay focused on that character.

I feel the same way about cutting to a closer view.  If the audience is looking at a speaking character, cutting to a closer shot should not disorient the audience any more than it does in a conventional film.


The next five to ten years are going to be very interesting with regard to VR.  No doubt gaming will be a leader, as it's a natural for putting a player inside the game.  But just as games have evolved cut scenes to provide the player with narrative information, VR is going to evolve storytelling grammar in order to do the same.  Once that grammar exists, we'll find out if VR is going to be successful with audiences as a storytelling medium or if it's just a fad like stereoscopic 3D.

For my earlier thoughts on VR, go here.

Some Thoughts on VR Storytelling

I am not an expert in virtual reality and frankly haven't been following it that closely.  However, it's beginning to pop up more and more, so I've started to think about how it might affect animated storytelling.

You've probably seen the video of Glen Keane drawing in three dimensions.  I've just seen a video from Occulus Medium about sculpting in a virtual reality.  I don't doubt that video games will take advantage of VR before any other form of animated storytelling, but how will stories that are not interactive accommodate themselves to VR?

The simplest approach will probably be to let the viewer wander through a scene, looking wherever he or she wants to at any given moment.  This is the closest to what now occurs on the stage.  When watching a play, you are free to look where you wish.  There are many directorial tricks to influence where the audience is looking, but there's nothing to stop an audience member from staring at a performer's shoe for the entire play.

(Will the hardware will track head or eye positions?  We move our eyes much faster than we turn our heads.  Besides speeding up the audience experience, tracking eyes will prevent the feeling of vertigo when the entire world spins while you turn your head.  Will the viewer glide through the scene like a film camera or will the viewer have to walk or run to keep up with the action?  Both have emotional validity, but will the action scale itself to the viewer's ability to keep up with it?)

There are two issues with giving the viewer the freedom to wander through a scene that I see.  The job of director exists specifically to shape a story visually and direct the audience to what is important at any given moment.  If you give the audience the freedom to look where they please without strategies to influence where they look, you could end up with stories that are less dramatic or comic than intended, simply because the audience is not catching the important action.

The other issue, and one that will be common to all forms of VR, has to do with increased production demands.  Right now, if you cut from character A to character B, character A does not have to be animated so long as he is off-screen.  With VR giving the viewer the ability to look where he pleases, character A must always be alive in case the viewer is looking in his direction.  This will require more animation for each character in a scene and may also require more background animation (wind, water, etc.)  If the characters will truly be present in the story, and not just on some kind of breathing cycle or other nondescript movement, this will require more work and larger budgets.  It will also force animators to learn how to have characters listen, something live actors learn early on, but that animated films typically avoid except for reaction shots.

Another alternative to this would be something similar to directing live action TV dramas from the 1950s.  Imagine that every character has a virtual camera focused on her as well as a camera that covers the master shot.  The viewer could switch between cameras, choosing who to look at.  The danger here, as above, is that on a first viewing the viewer doesn't know where the story is headed. She might miss important details or emotional moments.  Will a story have to be viewed multiple times in order to fully understand its dynamics?  And once understood, will the viewer have evolved a preferred way of experiencing a particular story that will be different from other viewers' approach to the same material?  Imagine a shared VR experience where you are looking through the eyes of the person next to you, experiencing how they experience a story.  Then you switch places.

Another possibility is the subjective camera where the viewer is in the story.  This edges more into an interactive approach, where characters would address the viewer, even if the viewer's responses were prerecorded audio.  In this way, it would be possible to be in the film as whatever character you desired and see the story through their eyes.  On additional viewings, you could take the part of another character.  Would Star Wars be more or less interesting if you could take the part of any of the characters and see the film through that character's eyes?

While videogames have been around for decades, there is still a separation between games and narrative media.  With VR, will they bleed into each other more?  Besides playing a character, will the viewer have hands to interact with the environment?  And if so, will the viewer be able to influence the story based on actions taken?  A hit or a miss with a weapon could have a major impact for how a story unfolds.  With each action taken, a parallel story possibility exists, perhaps making this approach far too expensive to produce.  If it turns out "these ARE the droids we're looking for," or the trash compactor kills one or more characters or the death star doesn't blow up, you have three other movies.  Multiply that by every fork in the story and how long would it take to produce the film?

Eventually, conventions will develop.  There's no reason that red means stop and green means go except that we all agree on it.  There's no reason that a right mouse click does certain things compared to a left mouse click except that we've been trained across various software packages to accept this.  Just as there are storytelling conventions in film, such as the cut and the dissolve, VR will inevitably develop its own conventions so that audiences feel confident as to the best way to experience a story.  What those conventions will be and how long they will take to develop is unknown at this point, but that's the next frontier.  People look back in wonder at the development of animation in the 1930s and the development of cgi in the '80s and '90s.  The next explosion of creativity in the animation field is going to be the development of VR as a storytelling medium.