How Movies Move: The Perception of Motion

On a recent trip to Princeton University, I had the opportunity to visit the Thomas Edison National Historic Park, in West Orange, New Jersey (about an hour drive from the campus). At this large laboratory complex, Edison invented the Kinetoscope, one of the first commercially successful motion picture viewers (the one in the photo is from the George Eastman House in Rochester, NY). Edison's interest in moving pictures was sparked in 1888 by a visit from Eadweard Muybridge who had already built a motion picture projector but one that could only present a dozen or so images in succession (e.g., a galloping horse). Edison's device was not a movie projector but instead a large one-person viewing console in which about 40 ft of 35 mm film strip passed by a peephole. The film was illuminated by stroboscopic flashes that were produced by a spinning opaque disk with an open slit that was placed between the film strip and a lamp. In this way, frames were flashed as instantaneous still images in rapid succession.  

At West Orange, Edison and William Dickson (the true film expert at Edison's lab) constructed the "Black Maria," a small studio where short films were shot for the Kinetoscope. Many of Edison's silent "flicks" were shot there, including the first U.S. copyrighted film, a 5-sec clip of an Edison employee, Fred Ott, sneezing. In the spring of 1894 in New York City, the first movie parlor opened, and people paid a nickel to peer into a Kinetoscope and watch a short clip (~20 sec) of a vaudeville act, such as a dancer or contortionist. The movie craze thus began, and Kinetoscope parlors cropped up in cities around the country and across the Atlantic. Yet within a couple of years the popularity of these peep shows faded as movie projectors were invented which could present the 35 mm films to large audiences.

It was of course the remarkable depiction of naturalistic movement that enthralled these early moviegoers as it does so today. The “illusion” of seeing movement from the rapid succession of still images, or what psychologists call apparent motion, is based on a not completely understood set of processes studied early on by Max Wertheimer, the German Gestalt psychologist. In 1912, Wertheimer published experiments in which a vertical line is followed a horizontal line at various lag times between presentations. If the lag time (or inter-stimulus interval) was very fast (less than 3/100 sec) then the viewer perceived both lines  simultaneously, which appeared to form a right angle. In this case, visual persistence of the first line overlapped with the presentation of the second line so that the two were perceived as being presented simultaneously. If the lag between stimuli was long (greater than 1/5 sec) then the viewer simply saw two separate lines presented sequentially. However, between these two lag times, particularly around 1/20 sec (or 50 msec), the vertical line looked as if it moved and swung down to a horizontal position.

The perceptual mechanisms underlying apparent motion are multifaceted and still studied by psychologists and neuroscientist. Some brain regions are particularly sensitive to movement, both real and apparent. In fact, at the initial entry point where visual input enters the cortex—an area identified as V1—some neurons are activated when objects are moving at a specific direction. From V1, two main paths process visual information. The dorsal path courses up to the parietal cortex and is involved in spatial processing ("where" things are). The ventral path courses down into the temporal cortex and is involved in object processing ("what" things are). For example, regions in the inferotemporal (IT) cortex  become active when we recognize objects, such as faces, cars, and furniture. The ventral and dorsal paths work together to integrate visual information allowing us to recognize objects placed in a spatial environment.

In the cortex, two regions are particularly involved in motion perception. The first, MT (middle temporal region, also called MT+ or V5) is sensitive to directional movement of a pattern or object, such as viewing a car moving across an intersection. In an fMRI study, individuals watched two squares blinking on and off with the lag time between presentations varying between 50 and 62 ms. Recall that in the Wertheimer study such time lags are often perceived as a single object moving—in this case a square appearing to move from left to right. With longer lags the apparent motion disappears, and the two squares simply appear to blink on and off in sequence. When individuals perceived a square to move from one position to the other, area MT was particularly active. The STS (superior temporal sulcus) is another motion-related region. It becomes active when individuals perceive bodily movements, including moving torsos, heads, and even eyes.

 It is still amazing to sit in a darkened theater and experience an action-packed Hollywood blockbuster on the big screen. As we watch such movies, multiple brain regions allow us to interpret movement from these flashing still images and "see" objects flying around or "feel" as if we are moving in the scene. With the advent of digital animation and other forms of computer-generated imagery, filmmakers have developed graphic techniques that make the completely artificial appear as natural as if we were viewing a scene outside our house. The analysis of various methods of digitally constructing realistic scenes, such as rendering shading and perspective and mimicking natural movements through subtle blurring of motion, might help scientists understand better how we perceive the real world. Natural viewing is somewhat akin to viewing a stop-action movie, as we are always fixating and then moving our eyes (about three times a second), and each time we do so, it is as if we capture a single-shot frame of the world. Thus, as much as science might help us understand filmmaking, filmmaking itself might help us understand how the brain works.

Directing Eye Gaze: The Filmmaker's Sleight of Minds

Walter Murch, the preeminent editor of such movies as Apocalypse Now (1979), The Unbearable Lightness of Being (1988), The English Patient (1996), and Cold Mountain (2003), wrote an incisive book on film editing called In the Blink of an Eye. In it, he said that editors must be preoccupied with "misdirection" and keep in mind the following questions: "What is the audience going to be thinking at any particular moment? Where are they going to be looking? What do you want them to think about? And, of course, what do you want them to feel?" (Murch, 2001, page 21). In the fast-paced rhythm of current Hollywood blockbusters where cuts are interspersed every 3-5 seconds, film editors must ask themselves these questions about 1200 times per movie.

The essential skill of a film editor is to create shot transitions that are so smooth that we are blind to them. In this way, outstanding editors keep us from being aware of their own craft. We are familiar with great actors, directors, and even producers, but how many award-winning editors do you know? (At least you now know one—Walter Murch.) In my mind, film editing defines in large part what makes movies distinct from other art forms. Murch's description of the editor's role is much like that of a psychologist who must be attuned to the thoughts and feelings of others. His notion of "misdirection" resembles the ploy of magicians who also must be aware of the viewer's attention and thoughts.

The psychologist Tim Smith has conducted some of the best research in psychocinematics. By studying eye movement behavior, he has shown that movies are extraordinarily potent stimuli in driving and directing our attention. As we view the world (including movies), our eyes fixate on a spot for about a third of a second and then jump to another spot, fixate, then jump again, and so on. These jumps are called saccadic eye movements, which are rapid, taking only 3/1000 of second to complete. Occasionally our eyes will track a moving object, but about 95% of the time they are motionless and fixated at a point in space. During any fixation, you can only clearly see a relatively small focused area, which amounts to a span of three or four words as you read this text. Thus, the position and sequence of eye movements offer a record of what your attention is focused on at any given moment.

Smith and colleagues recorded eye fixations of groups of subjects while they watched clips of feature films, such Blade Runner and There Will Be Blood. When they analyzed the data what became evident was that the subjects all tended to fixate on the same spot and move in synch from moment to moment. The figure below shows a frame from the movie There Will Be Blood with the circles representing where subjects were fixated at the moment of viewing that frame. The cluster of circles shows that there was a strong overlap or coherence of eye fixations among subjects as they watched the movie. Smith refers to this gaze attraction as attentional synchrony. In fact, as the subjects watched a film these clusters seemed to move together as if all were being guided by the moving images (for a video demo see Tim Smith's blog). 

As with Uri Hasson's fMRI analysis of coherent brain activity while watching movies (see March 15 blog), Smith has shown that filmmakers have found ways to control our attentional focus as we follow the plot. Thus, as the Murch quote above suggests, a good filmmaker understands the cognitive demands of viewing movies and knows how to guide the viewer so as to drive the storytelling in a forceful manner both perceptually and emotionally. In some respect we allow the filmmaker to guide and move us, as if we are on a fantastic vehicle that transports us through the movie. When the plot is riveting, we all seem to move together and enjoy the same magical ride.

Kuleshov Revisted: Emotional Reactions Across Time

Among aficionados of early silent films, Lev Kuleshov, the Soviet film director, is known for his use of editing to evoke different emotions. He took shots of an actor posing with a neutral expression and placed them after various scenes, such as one of a bowl of soup, a child's coffin, or a seductively dressed woman. Depending on the preceding shot, the actor's "neutral" expression seemed to change—from one of hunger to sadness and then to lust (see YouTube video).

Facial expressions offer potent displays of emotions, though the "Kuleshov effect" shows that expressions can be ambiguous and thus depend upon the context or situation. Following a close-up shot of a person crying a zoom out could show a funeral scene, though it could just as likely reveal a wedding procession. Even in the same setting, identical expressions may be perceived differently. Check out the clever drawing by psychologist Roger Shepard which shows a perceptual illusion--the two monsters are physically the same size though in the context of the tunnel, linear perspective makes our brains interpret the background monster to appear much larger than the one in the foreground. There is also an emotional illusion as the facial expressions are identical, yet the monster in the foreground appears scared, whereas the one in the background appears angry. 

In a recent set of psychological experiments, Diane Marian and I played on a kind of Kuleshov effect by presenting short film clips of facial expressions that changed over a few seconds. Let's say you are at a gathering and you look up from a conversation and see an unfamiliar face turn and smile at you. Your emotions would be buoyed. Now consider that expression in reverse—you look up and see someone smiling at you, but then the person turns and the expression fades to a blank stare. In our experiment, we showed a variety of short clips in which expressions moved from neutral to happy, happy to neutral, neutral to angry, and angry to neutral. A neutral face that had just been smiling at you actually looks rather grumpy (see video).

           

Conversely, when an angry face moves to a neutral expression, the person at the end appears to have a rather pleasant disposition.

These effects have the appearance of a perceptual illusion as the very same "blank" expression actually looks different (grumpy or pleasant) depending on whether it started as a happy or angry expression. In fact, we never interpret facial expressions as static, momentary images. In everyday experiences, facial expressions move within a rich contextual environment as they track and signal social interchanges. Of course, movies play on a character's facial expressions as the plot thickens. Jack Nicholson's evolution into madness in The Shining comes immediately to mind. These influences reinforce the idea that much of our emotional involvement at the movies is an unfolding or dynamic experience that occurs as much in the brain as on the screen.

Miyazaki Magic: From Up on Poppy Hill (Kokuriko-zaka Kara)

Those familiar with Hayao Miyazaki's animated films, such as Princess Mononoke, Spirited Away, Ponyo, and Kiki's Delivery Service, will be prepared for the U.S. release of From Up on Poppy Hill (Kokuriko-zaka Kara). The movie was directed by his son, Gorō Miyazaki, and co-scripted by Hayao Miyazaki and Keiko Niwa. It is set in Yokohama 1963 and tells the story of the burgeoning relationship between Umi and Shun as they try to prevent the demolition of a dilapidated clubhouse (called the "Latin Quarter") next door to Umi's high school. The story is wonderfully engaging and the visuals stunning. The characters are depicted with typical "anime" features (big eyes, flat shading), though the background scenery is vivid with extraordinary detail or at times with a painterly, watercolor style.

For those not familiar with Hayao Miyazaki, he is often compared to Walt Disney, as his full-length animations have been cherished by generations of children in Asia. A relative from Japan once told me that her kids watched their VHS recording of My Neighbor Totoro so often that the tape wore out from use. Indeed, even before last month's U.S. release of From Up on Poppy Hill, the movie had already grossed $60 million. What is special about movies by Miyazaki is that one can find playful stories aimed primarily toward young children, such as Totoro and Kiki, as well as darker tales, such as Princess Mononoke and Nausicaä of the Valley of the Wind, which are directed more toward older children (of course all of these are suitable for adults!). If one looks beyond to other animations from Studio Ghibli--the production company co-founded by Miyazaki and Isao Takahata--one can find Grave of the Fireflies by Takahata, a tale of two children trying to survive in Japan during World War II. This adult-oriented story is so heart-wrenching that I would have difficulty watching it again. The broad use of animation to describe a manifold of thoughts and emotions likely comes from the cultural foundation of manga, the comic books of Japan, which span the market from children's stories to adult pornography.

Although comparisons between Miyazaki and Disney are valid, what is apparent while watching a Miyazaki movie is how much our sense of storytelling is culturally based. For one, a Miyazaki movie typically involves a young girl as the main character who must confront a coming-of-age situation, which is true for Umi in Poppy Hill. For Umi, however, the relationship with Shun becomes quite complicated to the point where in the middle of the story Shun comments, "It's like a cheap melodrama"! The heroines in these movies are refreshing, especially compared to the "princesses" in Disney movies; yet, of course Miyazaki's characters are themselves grounded in their own cultural myths of gender and youth.

More interesting to me, from a cultural standpoint, is the lack of a true villain in a Miyazaki movie. In Western philosophy, we are ingrained with a sense of polarity—there's good and evil, right and wrong. We expect to have a defined bad guy that the hero must overcome and defeat. Eastern philosophies tend to acknowledge that goodness or badness often depends upon the context or situation. Psychologist Kaiping Peng has considered how such cultural influences play our thoughts and feelings. Early on, I had problems with Miyazaki movies as there wasn't a defined "bad guy." In fact, sometimes a character that started out seeming to be the evil archenemy suddenly works to assist the hero. In these movies, such as in Poppy Hill, there are of course questions and confrontations that must be addressed and resolved, yet the success of these stories is not the overthrow or defeat of some evil villain. In these movies we don't even need a final chase scene. My point: for a charming and engaging movie that is universally appealing, go see From Up on Poppy Hill.