We’ve been working through ideas for documentary narratives for 360 video using CG environments with the goal of creating immersive 360 Video experiences for education. In these early tests I’ve been using Skybox Studio v2 among other tools to animate static POV environments. These basic test sets are leading to some very promising ideas. The challenge is to create a workflow and UX that works broadly, from headsets to holding one’s phone up.
On August 3rd, I spoke at TEDx Melbourne about how digital narrative, a small corner of the storytelling universe is changing the fundamental format of Story.
Held in the main plenary of the MCEC. Here’s a transcript.
I’m going to talk today about how a small corner of the storytelling universe is changing and how this change is truly great for education.
I’m a writer and visual artist, I’m not specifically an educator, but I have been involved for years in different types of digital narrative:
Writing and designing story worlds, showing other storytellers and program makers how their work can be enhanced, expanded and improved upon in the digital space, and also facilitating access to other people’s stories.
The one thread through all of them has been how over the years, the digital revolution has been supplying storytellers with an increasingly amazing toolset.
Stories are essentially a form of memory. They have this unspoken, fundamental importance in our lives because as social creatures, sharing memories in entertaining and instructive ways fulfil fundamental drives in us.
Actually, here we are at TED, a hugely social facility for ideas and storytelling, and which relates very specifically to the idea I want to share.
You see, whilst I was trying to make this talk (I want to say good but let’s go with concise, I went and watched one of TED’s most seminal talks, which is ten years old now, given by Sir Ken Robinson. Only has 40 million plus views, so no big deal.
It’s a great talk, but the thing about it was there were three hooks in it that really tied things up for me. And I thought, yes, my talk is about answering the call of another TED Talk’s’ challenges. Challenges that after 10 years are still very much out there.
There are three pretty provocative observations made by Sir Ken’s:
Creativity is as important as literacy in education;
We tend to get get educated out of creativity;
Ideas of value, more often than not come about through an interaction between different disciplinary ways of seeing.
The new craft of digital narrative can respond to these challenges brilliantly, because it provides a state for creative learning that is adaptable, fluid and dynamic and social. Like us.
I want to touch on two types of this storytelling today:
Before we get to them, I’ve got a metaphor in two parts to share and hopefully it helps in explaining how a digital approach really brings something new to the fundamental form of a story.
It’s got a bit of philosophy of mind and it’s got a bit of Oscar Wilde.
And it starts with a question: why is narrative, in its manifold forms, so foundational to the human experience anyway?
Well, perhaps because our ability for self-reflection combines with the biological drive to be social, leading us to reimagine memories and experiences in entertaining and instructive ways – in other words, generate and consume stories together and seemingly as soon as we can.
Let me just read to you what two eminent academics, Roger Schank, Robert Abelson have said about this: “The reason that humans constantly relate stories to each other is that stories is all they have to relate. Put another way, when it comes to interaction in language, all of our knowledge is contained in stories and the mechanisms to construct them and retrieve them.”
Stories are forms of memories which we share with each other and also format into media: books, cinema, television, you name it.
By now we can safely say narrative is a fundamental aspect to the experience of life and how we learn about it. And it’s developed out of our earliest forms of communication.
So let’s retreat to the bright bright corner of the storytelling universe that I’m talking about, and ask what if narrative was formatted to be more like us? Not static, but adaptive, dynamic and social?
You could say this is already happening, with information, in the way we consume it daily – it’s dynamic and social.
But what about at the deeper level of storytelling, and where this time you can have an influence on the nature of the story in a way that encourages creativity and interaction between different disciplinary ways of seeing. This would be amazing as a learning environment.
Now I want to add in Oscar Wilde’s The Portrait of Dorian Gray.
Oscar Wilde’s famous story is about a fortunate young man influenced to pursue a narcissistic life of hedonism and aesthetics, just as an extraordinary portrait of him is being completed. When it’s finished, Dorian keeps the painting of himself out of sight, in the attic and lives large.
Over time, he notices that the portrait changes to reflect the morally corrupt actions he commits in real life. And all the while he continues to resemble his unscathed self in the portrait when it was painted. So spoiler alert – it ends badly!
But, it’s the use of the ‘magical portrait’ that we’re interested in here. The digital narrative toolset enables a story device, in this case a portrait that can magically reflect Dorian’s moral self, in completely new ways.
So, a minute amount of audience participation – which is a bit rich coming from someone who dreads audience participation! But it’s pretty low-key.
Bring to mind a picture you’re really familiar with, maybe one that’s been in your life for some time.
Maybe you’ve looked at that picture and caught yourself thinking – something’s different this time.
For sure, the picture is exactly the same, because it’s you who’s changed since you last saw it. Your experiences that make that picture seem different.
Bring to mind a book or a film in the same way. When you read or watch it again years later it might feel different because your memories and your experiences in the world have made you different.
Digital storytelling allows the picture or story itself to change, not just your perception of it.
A digital toolset makes possible a story that is adaptive not static. Potentially much more like memory and more like us.
Which is a really interesting proposition for education, learning environments and shared creativity.
Digital has touched many of the existing forms of story in amazing ways. Books are not just in print anymore, radio and television have decoupled from appointment listening and viewing and cinema has evolved as a medium to have a longer life span across many different screens.
The first of the new story types I’m talking about is still emerging.
About five years ago, I worked on Bluebird AR. The AR stood for Alternate Reality.
It was a story in which the audience were a crucial part of the outcome – the story was what came out of the collision between storytellers and audience.
Bluebird is an example of what happens when the photograph or narrative, as in the metaphor, is changed when it’s looked at.
Bluebird was played out online across websites, social media and a central social hub in which the audience worked together to unravel plot, control the pacing and to some extent the direction of the story.
And it responded to Sir Ken’s observations perfectly.
It put maximum emphasis on creative problem solving and interaction between different disciplinary ways of seeing, and rewarded it with authorship in progressing the story.
One day whilst Bluebird was in full swing, we received an email from a leading climate scientist at Carnegie Institute for Science, who had been one of our advisors during pre-production.
He was forwarding an email he’d received from a Year 11 student of Melbourne High School.
As this student researched and built theories in the world of Bluebird, he was asking a real-world leading expert in geoengineering if he knew what the images were of and whether they had the potential to be viable geoengineering solutions.
Sully, as the student was known in the game world, was one of Bluebird’s star players.
I think the lessons of Bluebird are more relevant than ever, not for entertainment but for education.
I think they articulate a highly creative approach to learning, especially complex ideas and the interaction between different disciplinary ways of seeing and in a learning environment that is dynamic and social and, perhaps most importantly digitally native.
So the second type of digital narrative I want to introduce today is the important flip side to the Bluebird model, in which the stories of the past can be revisited and expressed in this new language.
We’ve called this factual multimedia for some time, meaning that you can bring previously separate, often ephemeral items of media together to tell a more complete story of actual events.
I’ve been incredibly fortunate to revisit the story of the Sydney Opera House on two occasions in recent years.
The thing about Sydney Opera House is that the story of the building and how it came about articulates why it is such an incredible place – your appreciation of the monumental effort that went into its creation massively enhances your understanding of the place, but also much broader ideas and lessons in humanity, creativity, Australian social and political history, architecture, modernism, for example.
You begin to understand that its significance as a performance venue and world heritage listed masterpiece, is informed by an incredible backstory. You’re informed view is now contextualised, more complete, more 360 in perspective.
What the craft of digital narrative is doing here is taking the multiple threads of storytelling and binding them into a meta-story device that gives you the audience a deeper, contextual understanding.
So here’s an example. Here, we’re taking a look at one aspect of the story, the important discovery of the theory for the roof of Sydney Opera House, the Spherical Solution.
This is of course a fundamental element of the building – why is Sydney Opera House so recognisable? Because of the profile of the shells.
In this one chapter of the story, resides the context for understanding the principal, its back story, and its enduring significance. And this content exists in a variety of forms from archival photographs and film to newly created animations of the principle, to writing and interviews. All brought together.
A few of weeks ago I finished working on an app that revisits the little known battles of Fromelles and Pozières in this same meta-story context.
The events of the battle are recorded at the time and place where they occurred, whilst a chaptered story leads you through the central narrative. The depth of detail drawn from the experience is completely up to you, from watching the story of the battle, through to understanding the machinery of war, the diaries and profiles of soldiers, and modern day analysis. All in one app on one device.
Why I think these new models fit so well with education is that the audience behaviour is active rather than passive. It involves reaching into the experience.
You have, to varying degrees, to work with the narrative in order to be part of the story. You learn through the experience of participating. Just like life.
This year is marked by another jump for digital narrative, into virtuality.
It’s early days for Virtual Reality. Even though the ideas and versions of the tech have been with us for decades.
I worked on this magazine cover 22 years ago when photoshop was 4 years old.
But its emergence this year as a consumer device is a hugely significant moment in this digital revolution. And why, although I’ve been talking about stories, I chose the title of this talk to reflect this new wave.
We may see it as a novelty now, but the principle and potential that VR points to is likely to become a more significant a shift in perception than was seeing the earth from space. It will push human experiences forward in incredible ways.
VR promises to be a great evolution for some of the ideas I’ve been talking about for the past few minutes.
I reckon this is the most exciting time to be interested in telling stories.
Earlier in the year I was commissioned to create a “moving canvas” for a permanent installation at Sydney Opera House. “The Lounge, enabled by Samsung” is part of the Renewal Program, that will see a significant evolution of some of the internal spaces of Sydney Opera House.
This was an amazing brief on all levels: an invitation to artfully interpret a subject I know pretty deeply, and have the work on permanent display at SOH for thousands of people to see every day. It was a rare opportunity to work at the aesthetic end of the communication spectrum.
Samsung supplied a high density LED screen (comprised of a multiple of smaller units) that allows for a diode per pixel in the display of a 1920×1080 image. Yes, that’s 2,073,600 diodes/pixels! When the films are projected as 30Mbps H264 videos, they do so with remarkable quality – colour depth, steady frame rate and fine line detail.
In collaboration with SOH staff, we identified six themes that would ground the abstract visuals, and which aligned with the philosophy behind the broader Renewal Program. These included the Creative Spirit that the place embodies and hosts; a Nexus, in terms of intersection between so many cultural elements; Design Heritage, Meeting Place, Renewal and Performance. The slideshow below shows an image each from the broader films that exemplify these themes:
A Giant Sphere
I had for some time imagined a set of images and sequences that related to Sydney Opera House that I hadn’t been able to put into a specific project, comprised of things Reuben and I had seen when studying and working with the forms in the architecture, design and engineering of the building. They were exciting aesthetically, but not really appropriate or were too abstract for what we were working on at the time. This was the perfect project to explore them.
The main image was of a giant sphere, its internal and external forms reflecting the interior and exterior of the shells. The sphere in its entirety is of course the ultimate expression of the geometry of the building, and covering its external surface in the chevron pattern of the tile lids, illustrates the perfect geometric solution that was found for articulating the spherical geometry of the final roof form.
Similarly, the interior of the sphere is ribbed with the same form as the finished building, culminating at a hemisphere (the ridges of the shells) and in a union of pedestals, where each of the shells gather at their base. Removing the final wedge of each of the pedestals gave us two points of entry/exit between external and internal spaces, at the poles of this giant sphere.
Other key imagery celebrated the aesthetic of the Yellow Book, produced to illustrate the final roof geometry.
I also explored intersection between Utzon’s designs and Peter Hall’s by filming them in 4K and mirroring them to celebrate the union and the repetition of the form, which is an aesthetic in its own right.
The recreation of an ancient Bennelong Point also enabled us to tell the visual story of the location as a meeting place stretching back thousands of years. Alongside paintings by Conrad Martens and Joseph Lycett, the CG sequence imagines the forested promontory and the tidal island, heaped with middens that would eventually become the location of Sydney Opera House.
For the sound design, we wanted something that could drift through the space, that would reprise the various visual beats. We worked with Andrew Stevenson and Hylton Mowbray of We Love Jam to produce a composition, blending diegetic and non-diegetic elements. The idea of the massive volume to the sphere was a central theme of the sound design, conveying wonder, space, and the universality of the geometry of the building. Below you can listen to an excerpt:
The project was created using Adobe products, including After Effects, Photoshop, Illustrator and Cinema 4D. We shot 4K video using a Red camera setup.
The complex causes of World War One are often swept aside to expose the fate of Archduke Franz Ferdinand as the personification of the outbreak of the Great War. Behind this textbook depiction is the story of a nonconformist’s tragic death alongside his life’s love, Sophie, both caught up in events that on reflection seem destined to have occurred.
This dark providence was only strengthened as it passed from one conspirator to another, through a hapless chain of transactions culminating on a bright summer’s day in Sarajevo, 100 years ago. Bosnia was primed for violent expressions of nationalism, having first passed under the crushing rule of the Ottomans and now that of the Austro-Hapsburg Empire. The Black Hand, part terrorist cell, part branch of Serbia’s military and government, had targeted the Archduke, nephew to the emperor and heir to the empire, for assassination, and then vetoed the idea, but too late to dissuade the Young Bosnia members who would go through with it. (more…)
2016 has turned immediately into the year of living virtually.
In 2015 there was the sense that it was definitely going to be that way, both for proper VR and its cousin 360° video, but it’s still surprising how fast it’s happened: headlong without the necessity for language or codes. Naturally it’s a time of experimentation.
Describing the spectrum of experience: VR – Virtual Experience – 360° video
Clear parlance and definitions are really important. A pureVR experience is a quite separate thing to a 360° video, although the former can incorporate the latter.
Witness ADR1FT for instance, which although it can be played in 2D, shines through an HMD (head mounted display) – a really seminal experience of VR’s new wave.
Then there is a form of media that intersects both, which here is referred to as a virtual experience and commonly comprises a dimensional space of simple geometry, one that shares polygons and video elements, and which The Guardian’s intense 6×9: Solitary Confinement is a great example (also featuring excellent binaural sound design).
Basically, the geometry defines a space around you that enhances the sense of perspective and therefore presence, but is in immersive terms a step below purer forms of VR – trading immediacy for qualities associated with higher production values.
Simple geometry is commonly used to enhance the space and therefore presence of a 360° experience. Car ads abound, and it’s an emerging solution for fast turnaround communication media, such as advertorial and real estate.
However, as below, it’s also an excellent opportunity for more sophisticated storytelling, and a brilliant approach to experimenting within the evolving language of the medium.
Finally, at the other end of this spectrum form pure VR is pure 360° video.
As an aside in this post, the output equipment across an array of cameras and lens types, is a wholly exciting subject in its own right (behold Lytro’s Immerge).
The distinctions along this spectrum of experience are fluid, but it’s really important, when describing the output or end product to distinguish between them: from pure VR to the more basic 360° video, and the intersecting media between. Otherwise the quality of presence, immersion and subtleties of these high fidelity formats become arbitrarily mixed up with the lo-fi.
And to show how tricky this can be, here’s my favourite virtual experience to date – much more than the 360° video it pegs itself as. Watching this experience on Oculus DK2 is completely different to watching on Google Cardboard.
Dead slow cross-dissolves work
Just as I write, the tools for editing and post production are beginning to emerge. When I was sitting down to work out logistically what could be achieved with the small time and smell-of-an-oily-rag-budget we had to produce the work (the enthusiasm and effort by great and talented people around you are as always the trick), a big consideration was flow.
It turns out that the humble cross-dissolve is ideal for establishing a transition in a 360° environment which put a general audience at ease. Dead slow pace also opens up interesting aesthetics in the transition moments – not only graphic forms in motion, but it affects the sense of travel, presence and temporality in about as positive and soothing a way as possible. Dissembling the world by cutting away parts of its structure to reveal the scene the audience would be seeing next – a sort of breaking down of the walls of one scene to reveal another – just felt artificial and flat, next to a dead slow dissolve, which retains the sense of volume between locations nicely.
If not binaural, pan instead
Although we recorded binaural sound across the scenes, we weren’t able to implement it in the final film. Andrew Stevenson, sound designer for the project remixed elements of the sound to emulate the experience of spatial sound instead. It may be a poorer cousin but it’s huge entirely worth it. Even for the below-conscious level acknowledgement that the drift of Nicole Car’s voice emerges from the location of the Joan Sutherland Theatre (to the user’s left), it’s a nice detail.
I also used sound as a pre-cursor and intermediary to visual transition, basically a thread to carry the user through the experience. Throughout the video, we either hear the next scene before seeing it or are led through it wit the existing soundscape.
Check your heights
Maintaining, or being very conscious of the reason you’re changing the consistency of the height of the camera (and therefore the end experience) is vital, and a basic checklist item when moving between setups. If you want to maintain consistency for the user across the experience, this is a crucial point and especially for fixed camera work.
It also gives you a powerful position to break from. I had initially wanted to come into the Joan Sutherland Theatre via the stage left well – coming up onto stage, with Nicole singing to the camera. When that was not possible, we inverted the camera rig, strapped it to a mid-stage fly line and sent it up over Nicole’s head instead. Especially if you’re viewing on a good headset, this experience of gentle lift works nicely as a disruptive thrill to the rest of the locked-off shots.
Spatial elements, grade
We paired right back on ambitions to define geometry and move the 360° video production more toward a virtual experience, but took the opportunities where they were available. I used perspective typography to contextualise the performances, and the grade study for the scenes gave them a slightly enhanced depth and saturation, to offset the lack of defined space – the ‘flatness’ of the final video surface. This works particularly well in the organ space of the Concert Hall, and also in the Utzon Room, with the intense golden light of early morning that pours into the room at that hour.
Title cards and credits are also a good place to break out into a simple sense of 3D. At the end of the film, the credits are set at the four cardinal points.
This was a production that was very much characterised by what we could do with the means to hand. Sure, one of the most incredible buildings in the world is a good start! and being surrounded by enthusiastic and talented individuals is the key. But what I mean is the intention of understatement and incidental observation really made the limitations strengths rather than the weaknesses they always promise to be.
It was a such a privilege to be given the opportunity to make what’s essentially a small arty film about Sydney Opera House. Huge thanks to crew, artists and friends who poured their efforts into the production. Please check out them out in the credits above.
Finally, here’s how I describe the finished video:
The Sydney Opera House 360° video intends to lead its viewer through the Sydney Opera House from daybreak to evening, immersing them in a handful of incidental experiences along the way. Deliberately low-key and vérité in its approach, the idea of observing the ‘nature’ of the house is explored – an experience that is not commonly available to visitors, but is the magic of the place that people who are around it all the time can easily recognise. Making this experience universally available is the core value proposition.
The performances arose in the vein of this approach: Nicole Car’s enthusiasm to ad lib in-between performances; Sydney Symphony Orchestra’s generous permission to let us record them in rehearsal; Benjamin Schwartz giving up his time to noodle in the Utzon Room early one morning. The spirit of generosity and creativity that is poured into the Sydney Opera House by all who work and perform there is what makes it a special place and infuses the visitor/audience experience. The 360° experience reflects on these qualities, the sense that behind the performance are the talents, enthusiasm and generosity of people who are inspired by the extraordinary place they work and perform.
Musically, the pieces all resonate with this intention. Rimsky-Korsakov’s orchestral work, Scheherazade, speaks to the grand romance of the building itself, a place and experience that is by its nature bigger than any of us, classical yet contemporary, an immersive feeling that sweeps us off our feet! Dvořák’s Song to the Moon is an operatic love letter to the incredible legacy of intimate opera in the Joan Sutherland Theatre. Benjamin Schwartz noodles around in the ‘key of Bach’, evoking one of Utzon’s inspirations for his mural, which Ben faces.