Wednesday, May 15, 2013

How Movies Move: The Perception of Motion

On a recent trip to Princeton University, I had the opportunity to visit the Thomas Edison National Historic Park, in West Orange, New Jersey (about an hour drive from the campus). At this large laboratory complex, Edison invented the Kinetoscope, one of the first commercially successful motion picture viewers (the one in the photo is from the George Eastman House in Rochester, NY). Edison's interest in moving pictures was sparked in 1888 by a visit from Eadweard Muybridge who had already built a motion picture projector but one that could only present a dozen or so images in succession (e.g., a galloping horse). Edison's device was not a movie projector but instead a large one-person viewing console in which about 40 ft of 35 mm film strip passed by a peephole. The film was illuminated by stroboscopic flashes that were produced by a spinning opaque disk with an open slit that was placed between the film strip and a lamp. In this way, frames were flashed as instantaneous still images in rapid succession.  

At West Orange, Edison and William Dickson (the true film expert at Edison's lab) constructed the "Black Maria," a small studio where short films were shot for the Kinetoscope. Many of Edison's silent "flicks" were shot there, including the first U.S. copyrighted film, a 5-sec clip of an Edison employee, Fred Ott, sneezing. In the spring of 1894 in New York City, the first movie parlor opened, and people paid a nickel to peer into a Kinetoscope and watch a short clip (~20 sec) of a vaudeville act, such as a dancer or contortionist. The movie craze thus began, and Kinetoscope parlors cropped up in cities around the country and across the Atlantic. Yet within a couple of years the popularity of these peep shows faded as movie projectors were invented which could present the 35 mm films to large audiences.

It was of course the remarkable depiction of naturalistic movement that enthralled these early moviegoers as it does so today. The “illusion” of seeing movement from the rapid succession of still images, or what psychologists call apparent motion, is based on a not completely understood set of processes studied early on by Max Wertheimer, the German Gestalt psychologist. In 1912, Wertheimer published experiments in which a vertical line is followed a horizontal line at various lag times between presentations. If the lag time (or inter-stimulus interval) was very fast (less than 3/100 sec) then the viewer perceived both lines  simultaneously, which appeared to form a right angle. In this case, visual persistence of the first line overlapped with the presentation of the second line so that the two were perceived as being presented simultaneously. If the lag between stimuli was long (greater than 1/5 sec) then the viewer simply saw two separate lines presented sequentially. However, between these two lag times, particularly around 1/20 sec (or 50 msec), the vertical line looked as if it moved and swung down to a horizontal position.

The perceptual mechanisms underlying apparent motion are multifaceted and still studied by psychologists and neuroscientist. Some brain regions are particularly sensitive to movement, both real and apparent. In fact, at the initial entry point where visual input enters the cortex—an area identified as V1—some neurons are activated when objects are moving at a specific direction. From V1, two main paths process visual information. The dorsal path courses up to the parietal cortex and is involved in spatial processing ("where" things are). The ventral path courses down into the temporal cortex and is involved in object processing ("what" things are). For example, regions in the inferotemporal (IT) cortex  become active when we recognize objects, such as faces, cars, and furniture. The ventral and dorsal paths work together to integrate visual information allowing us to recognize objects placed in a spatial environment.

In the cortex, two regions are particularly involved in motion perception. The first, MT (middle temporal region, also called MT+ or V5) is sensitive to directional movement of a pattern or object, such as viewing a car moving across an intersection. In an fMRI study, individuals watched two squares blinking on and off with the lag time between presentations varying between 50 and 62 ms. Recall that in the Wertheimer study such time lags are often perceived as a single object moving—in this case a square appearing to move from left to right. With longer lags the apparent motion disappears, and the two squares simply appear to blink on and off in sequence. When individuals perceived a square to move from one position to the other, area MT was particularly active. The STS (superior temporal sulcus) is another motion-related region. It becomes active when individuals perceive bodily movements, including moving torsos, heads, and even eyes.

 It is still amazing to sit in a darkened theater and experience an action-packed Hollywood blockbuster on the big screen. As we watch such movies, multiple brain regions allow us to interpret movement from these flashing still images and "see" objects flying around or "feel" as if we are moving in the scene. With the advent of digital animation and other forms of computer-generated imagery, filmmakers have developed graphic techniques that make the completely artificial appear as natural as if we were viewing a scene outside our house. The analysis of various methods of digitally constructing realistic scenes, such as rendering shading and perspective and mimicking natural movements through subtle blurring of motion, might help scientists understand better how we perceive the real world. Natural viewing is somewhat akin to viewing a stop-action movie, as we are always fixating and then moving our eyes (about three times a second), and each time we do so, it is as if we capture a single-shot frame of the world. Thus, as much as science might help us understand filmmaking, filmmaking itself might help us understand how the brain works.

Wednesday, May 1, 2013

Directing Eye Gaze: The Filmmaker's Sleight of Minds

Walter Murch, the preeminent editor of such movies as Apocalypse Now (1979), The Unbearable Lightness of Being (1988), The English Patient (1996), and Cold Mountain (2003), wrote an incisive book on film editing called In the Blink of an Eye. In it, he said that editors must be preoccupied with "misdirection" and keep in mind the following questions: "What is the audience going to be thinking at any particular moment? Where are they going to be looking? What do you want them to think about? And, of course, what do you want them to feel?" (Murch, 2001, page 21). In the fast-paced rhythm of current Hollywood blockbusters where cuts are interspersed every 3-5 seconds, film editors must ask themselves these questions about 1200 times per movie.

The essential skill of a film editor is to create shot transitions that are so smooth that we are blind to them. In this way, outstanding editors keep us from being aware of their own craft. We are familiar with great actors, directors, and even producers, but how many award-winning editors do you know? (At least you now know one—Walter Murch.) In my mind, film editing defines in large part what makes movies distinct from other art forms. Murch's description of the editor's role is much like that of a psychologist who must be attuned to the thoughts and feelings of others. His notion of "misdirection" resembles the ploy of magicians who also must be aware of the viewer's attention and thoughts.

The psychologist Tim Smith has conducted some of the best research in psychocinematics. By studying eye movement behavior, he has shown that movies are extraordinarily potent stimuli in driving and directing our attention. As we view the world (including movies), our eyes fixate on a spot for about a third of a second and then jump to another spot, fixate, then jump again, and so on. These jumps are called saccadic eye movements, which are rapid, taking only 3/1000 of second to complete. Occasionally our eyes will track a moving object, but about 95% of the time they are motionless and fixated at a point in space. During any fixation, you can only clearly see a relatively small focused area, which amounts to a span of three or four words as you read this text. Thus, the position and sequence of eye movements offer a record of what your attention is focused on at any given moment.

Smith and colleagues recorded eye fixations of groups of subjects while they watched clips of feature films, such Blade Runner and There Will Be Blood. When they analyzed the data what became evident was that the subjects all tended to fixate on the same spot and move in synch from moment to moment. The figure below shows a frame from the movie There Will Be Blood with the circles representing where subjects were fixated at the moment of viewing that frame. The cluster of circles shows that there was a strong overlap or coherence of eye fixations among subjects as they watched the movie. Smith refers to this gaze attraction as attentional synchrony. In fact, as the subjects watched a film these clusters seemed to move together as if all were being guided by the moving images (for a video demo see Tim Smith's blog). 

As with Uri Hasson's fMRI analysis of coherent brain activity while watching movies (see March 15 blog), Smith has shown that filmmakers have found ways to control our attentional focus as we follow the plot. Thus, as the Murch quote above suggests, a good filmmaker understands the cognitive demands of viewing movies and knows how to guide the viewer so as to drive the storytelling in a forceful manner both perceptually and emotionally. In some respect we allow the filmmaker to guide and move us, as if we are on a fantastic vehicle that transports us through the movie. When the plot is riveting, we all seem to move together and enjoy the same magical ride.