Robert Harley of The Absolute Sound on Blind Listening Tests

Filed under: Home Audio


- Image: HifiBooks.com

I've long held the belief that the double-blind "ABX" tests often used to try and determine if there are differences between audio formats, amplifiers, etc. are fundamentally flawed. The roughly 50/50 results that always result supposedly show no audible differences between CD and high-res audio, or digital and analog recordings or whatever the test subjects are only show that the tests themselves don't work. Robert Harley of The Absolute Sound has posted an interesting editorial both online and in the upcoming issue #183 of the magazine that also takes this view. He mentions first the most recent example of High Resolution Audio which we covered back in April.

The following is an excerpt from Robert Harley's post on AVGuide.com.
"I contend that such tests are an indictment of blind listening tests in general because of the patently absurd conclusions to which they lead. A notable example is the blind listening test conducted by Stereo Review that concluded that a pair of Mark Levinson monoblocks, an output-transformerless tubed amplifier, and a $220 Pioneer receiver were all sonically identical. ("Do All Amplifiers Sound the Same?" published in the January, 1987 issue.)

A good example is the listening test conducted by Swedish Radio (analogous to the BBC) to decide whether one of the low-bit-rate codecs under consideration by the European Broadcast Union was good enough to replace FM broadcasting in Europe. The test involved 60 'expert' listeners spanning 20,000 evaluations over a period of two years. Swedish Radio announced in 1991 that it had narrowed the field to two codecs, and that "both codecs have now reached a level of performance where they fulfill the EBU requirements for a distribution codec." In other words, Swedish Radio said the codec was good enough to replace analog FM broadcasts in Europe. This decision was based on data gathered during the 20,000 'double-blind, triple-stimulus, hidden-reference' listening trials.

After announcing its decision, Swedish Radio sent a tape of music processed by the selected codec to the late Bart Locanthi, an acknowledged expert in digital audio and chairman of an ad hoc committee formed to independently evaluate low-bit rate codecs. Using the same non-blind observational-listening techniques that audiophiles routinely use to evaluate sound quality, Locanthi instantly identified an artifact of the codec. After Locanthi informed Swedish Radio of the artifact (an idle tone at 1.5kHz), listeners at Swedish Radio also instantly heard the distortion.

How is it possible that a single listener, using non-blind observational listening techniques, was able to discover -in less than ten minutes- a distortion that escaped the scrutiny of 60 expert listeners, 20,000 trials conducted over a two-year period, and elaborate "double-blind, triple-stimulus, hidden-reference" methodology, and sophisticated statistical analysis? The answer is that blind listening tests fundamentally distort the listening process and are worthless in determining the audibility of a certain phenomenon." - Robert Harley, The Absolute Sound


If you're interesting in reading more (a lot more) about this subject, Harley has also posted an exhaustive paper which he submitted to the Audio Engineering Society called The Role of Critical Listening in Evaluating Audio Equipment Quality.