Looking into listening: Using eye-tracking to establish the cognitive validity of the Aptis Listening Test

Holzknecht, F.; Eberharter, K.; Kremmel, B.; Zehentner, M.; McCray, G.; Konrad, E.; Spöttl, C.

Abstract

This study investigated the cognitive processing of 30 test-takers while completing the Aptis Listening Test. The research studied test-takers’ processes according to ones targeted at the different item
levels in the Aptis Test.

Specifically, it examined whether test-takers’ cognitive processes and types of information used corresponded to the ones targeted at the different CEFR levels. To this end, a detailed analysis of test-takers’ verbal recalls was conducted, which were stimulated by a replay of their eye-traces while they had been solving the items. The study also explored the usefulness of quantitative analyses of eye-tracking metrics captured during listening tests.

The stimulated recall findings indicate that the Aptis Listening Test successfully taps into the range of cognitive processes and types of information intended by the test developers. The data also shows, however, that the differences between the CEFR levels in relation to the intended cognitive processes could be more pronounced, and that the process of “discourse construction” could be more evident for B2 items. It is, therefore, suggested that a different item type could help elicit this type of higher-order processing. In terms of types of information used by candidates, a clear difference and progression between the CEFR levels to answer items correctly was observed.

The quantitative analysis of the eye-tracking metrics revealed interesting results. A linear mixed effects model analysis, with visit duration on response options as the dependant variable, showed that testtakers looked at the response options of higher-level items significantly longer than at the response options of lower-level items. The results also showed that response options higher up on the screen were looked at significantly longer than response options lower down, regardless of item level. In addition, it was found that better readers focused on the response options significantly longer than poorer readers.

• Report Type: Research Report
• Publication Date: 2017
• Deposit Date: Mar 1, 2024
• Publisher URL: https://www.britishcouncil.org/exam/aptis/research/publications/arags/looking-listening-using-eye-tracking