The Gap is Gone: Sound in the New VR Frontier
Why VR?
What’s the fascination with Virtual Reality? Until you’ve tried it for yourself it’s hard to appreciate why strapping a phone to your face has anything to offer beyond gimmickry. Yet it’s what happens to both body and mind when you’re ‘inside’ the VR world that defies expectations. We encounter all the physiological and emotional reactions that we do in the real world. Perched on the edge of a virtual cliff, you’re seized by fear no matter how much you tell yourself “this is not real!”. Take this recent example, where viewers find themselves having to rescue a helpless kitten high on a ledge.
Simply put, VR is the first wholly new form of media to emerge since 1910. Where film was about manipulating time, VR is about manipulating space and our place in it. Before it we’d seen only enhancements to existing forms of moving image and sound. That elusive nirvana of every creative conversation — how to create ‘immersive’ experience — is suddenly obsolete. Because in VR you are immersed, like it or not. The gap is gone.
VR is also a completely new medium for telling stories, for giving experiences, and for social interaction. It is different from film because all of a sudden we are in the movie (we have presence), and we are free to look and move around inside it the way we choose (we have agency). A whole new language needs to written for the creators of VR and for its users, and a key part of this vocabulary is sound.
The Role of Sound in VR
Many of the new VR production companies emerging are offshoots of visual effects outfits, for whom expanding imagery from a forward placed screen to a 360º field is a relatively accessible process. Because their skillsets are primarily visual, sound was getting left behind. But ask any VR developer, from Jaunt to Industrial Light and Magic, and they will tell you: sound is far more important in VR than it ever was in film or tv. In the vast majority of VR experiences, we are using only two of our senses: sight and hearing. So sound is 50% of the experience to be had, yet it conveys almost all the emotion. In the VR experience just as in life, it can convey feelings or drive a story without even needing to look at it, because sound is by nature immersive, and our ears never blink. In this way, sound has found an important role in the VR world: to lead the user’s attention towards something you want them to see that may be out of view (the human visual system sees only a 180º field of view, and just 114º of that in 3D.)
Made Music Studio in VR
As long-time evangelists for the power of sound, Made Music Studio embraced the sonic possibilities of the VR space early on, building partnerships with developers of new audio tools and creating unique workflows to enhance the sound & music component of VR experiences. Great advances in the simulated positioning of sound (spatialization) mean that we can place sounds anywhere in space, even when listening on regular headphones. In the VR experience this spatialized soundscape can now track head movement correctly so the sound remains in place, just as in real life. We can go further, creating an ‘inner experience’ for the user, an internal dialogue or song, by making it sound distinct from the sounds that are ‘out there’.
Our Approach
Our long experience in creating meaningful & memorable music and sonic identities for entertainment, general market brands and physical spaces from stores to stadiums, finds a perfect home in the VR world. We can bring these diverse elements together, creating experiences that use space in entertaining and engaging ways. As with all our work at Made Music Studio, strategy is our foundation, and this is ever more important in VR, where the canvas is seemingly limitless. Our development process begins with understanding the larger concept before moving into any creative development. What emotion do we want to evoke? What story are we telling?
If the VR project requires live location recording, we use ambisonic microphones to capture a ‘soundfield’, which together with detailed post-produced soundscapes can then be refined and manipulated to accurately represent a true-to-life, responsive sonic experience. On the music side, we are able to go beyond the conventions of stereo and surround, and create real musical interactions with the user. When it comes to playback, we then work with our clients to determine the ideal delivery platforms, whether GearVR, Oculus Rift, HTC Vive, Google Cardboard and beyond, and facilitate integration of the necessary codecs.
Featured Work
Some of our recent work which we are particularly proud of includes Studio Transcendent’s VR film A Brief History of Flight. This unique piece was nominated for the 2015 Proto Award for Best Educational VR Experience at the most recent VRLA Conference. Viewers are brought on a cinematic journey through the history of aviation, from witnessing the Wright Flyer’s first leap, to being strafed by WWI fighters, buzzed by the leading-edge F22 Raptor, and nearly engulfed by the world’s largest plane. Every sound of every aircraft is historically accurate, carefully spatialized to give the user a real-life experience of this amazing airshow. The experience is underscored by a soaring orchestral soundtrack composed by Made Music Studio’s team.
In our work on Applied VR’s Pain RelieVR app — a post-surgery VR experience currently being tested in hospitals in the US — we drew from research in music therapy to create a patterned musical soundscape that helps distract patients without overwhelming them. Placed in the VR setting with a simple, visual task-oriented game, patients experience as much as a 40% reduction in pain. This project, developed in collaboration with Cedars-Sinai and Children’s Hospital Los Angeles is now entering its next phase based on patient feedback, and we have begun work on a second VR patient experiences aimed at anxiety management.
Our live-action pieces with SilVR Thread challenge users to speed down a ski terrain park, go zip-lining or sailing. Throughout these journeys, the music and sound amplifies the thrill and the realism. So much so, that sometimes this happens!
We’re deliberately pushing the envelope to usher in a new kind of experience — Sonic VR: 360º experiences of space that are driven by sound and music, with all the meaning and emotionality they bring. The possibilities are endless and exciting. So go ahead & strap that phone to your face, but don’t forget the headphones!
Joel Douek is Creative Director, West Coast and Chief Scientist at Made Music Studio. Follow him on Twitter to learn more about VR: @joeldouek.