IRA FLATOW, HOST:
This is SCIENCE FRIDAY, I'm Ira Flatow. I'm going to go out on a limb here and guess that because you are a public radio listener and listen to a lot of music on public radio, that you have great musical taste. And you'd be able to pick out the better musician just by listening to that person. Well, I'm going to test you out right now. Get your ears ready. These are two clips from the 7th International Franz Liszt Piano competition and I want you to tell me which pianist do you think won.
I'm going to play you two clips. I'm going to hit Player 1 now.
(SOUNDBITE OF MUSIC)
FLATOW: Short clip. Okay. This was Player 1. Let's listen to Player 2.
(SOUNDBITE OF MUSIC)
FLATOW: Did you have a favorite clip? Do you know who won? Well, if you picked the last clip, you are correct. That was from the winning performance of Yingdi Sun and you've judged one of them better, right, just by listening to it. But if you were to judge them in concert, actually watch them being played, you might choose otherwise, because new research, which studied piano playing found that watching, watching how the music was performed was more important than picking a winner than actually listening to the music.
Yeah, a paper was published in the proceedings of the National Academy of Sciences. Chia-Jung Tsay is an assistant professor in management science innovation at University College in London and she's author of the paper. She joins us from BBC studios in London. Welcome to SCIENCE FRIDAY.
CHIA-JUNG TSAY: Hi, thanks for having me.
FLATOW: So tell us what - what inspired this study?
TSAY: So it was actually my own music experience. For the last two decades or so I've been performing and participating in various competitions, and it was through this experience that I observed, depending on the type of evaluation process involved, whether the competitions required audio recordings or video recordings, there could be very different outcomes.
And so this led me to the empirical investigation of how much visual information really matters.
FLATOW: Now, Dr. Tsay, tell us exactly what you found, because it was very surprising.
TSAY: Yes, yes. So I was surprised both as a musician and an academic. So basically I gathered a number of excerpts from these major international competitions and I made three versions of these excerpts, so silent videos without any sound, audio recordings, and videos with sound. And in the initial tests I had novice participants and I randomly assigned them to one of these three versions of the excerpts.
And I was surprised to find that it was only through the silent videos that the novices were able to identify the actual winners, even though I think there's a wide consensus that sound is so central to the judgment of music performance.
FLATOW: So just to repeat that, the novices who never heard the performances, they just saw them...
TSAY: That's right.
FLATOW: ...could judge which ones the judges pick?
TSAY: Exactly, exactly. And for this first set of experiments, it might be just that these novices simply didn't have the training or experience to use sound and to evaluate sound appropriately. So in the next set of experiments, I had professional musicians who also were randomly assigned to one of these three versions of the excerpts, and it turns out that it was very similar and even with these experts it was only through the silent videos that they were able to identify the actual winners.
And there weren't significant differences in how the experts versus the novices did.
FLATOW: Wow. So all the hand movements, gesticulation, the drama, counted more than the music?
TSAY: From the results it appears that there's various visual markers that allowed for these inferences of how good people were, the quality of performance.
FLATOW: Any particular ones, visual markers, that stood out?
TSAY: So especially passion, so passion that seemed visible allowed people to make inferences about whether someone was better or worse, and other traits such as involvement or creativity or uniqueness. There are just certain traits or characteristics that seem to be more visually perceptible than they were through sound.
And people tended to associate these traits with better or worse quality.
FLATOW: Do you play yourself in front of audiences?
TSAY: Oh, absolutely. I love performance.
FLATOW: So have you learned - are you going to incorporate any of these findings into your own performance?
TSAY: Oh, so I think it was really my own experience that informed my thoughts about this research, and so for me being on stage and being able to respond in real time to audiences, it very much resonated with me in a way that recording in a sound studio didn't really do for me. So I've been long attuned to how much visual information can be powerful, both for the performer as well as for the audience.
But I think the degree to which visual information matters so much even for these professional musicians, that remains surprising to me.
FLATOW: Yeah, because you would think, you know, professional musicians, I know what's going on here. I'm a professional, I'm performing, they're watching me and I'm interested in the sound. Maybe we can compensate for that and close our eyes and listen to the music. But no.
TSAY: Right, right. And we're very much trained to value sound and to create better sound and we have auditions and ear training classes, so everything about this industry, this domain, says that yes, it is sound that matters the most. And in fact, the professional musicians who were randomly assigned to receive the silent videos, actually they were quite upset with me, so they were very frustrated and expressed this high lack of confidence in their judgment when they didn't realize they would actually do much better with the silent videos in identifying the actual winners.
FLATOW: I'm wondering if this kind of, you know, visual information over auditory might play into other kinds of science and art things, you know? I'm trying to think of what other kinds of things that might be helpful.
TSAY: Right. I think the domain of music, it's such a unique opportunity because it's the one domain in which everyone would probably agree that sound is most critical to this domain. And so because of that, given that there were such robust findings in the domain of music, I would imagine that there would be wide implications for a number of domains and decisions, just any type of professional judgment in which there is access to visual information but perhaps other types of information would be much more relevant or predictive of actual levels of performance.
FLATOW: Well, Dr. Tsay, thank you very much for taking time to talk with us. A fascinating study. I'm sure you're getting a lot...
TSAY: Thank you, Ira.
FLATOW: ...feedback from that.
TSAY: Thank you, Ira.
FLATOW: Chia-Jung Tsay is an assistant professor in management science and innovation at University College in London. Transcript provided by NPR, Copyright NPR.