Full bibliography

Visual information on a talker's face can influence what a listener hears. Commonly used approaches to study this include mismatched audiovisual stimuli (e.g., McGurk type stimuli) or visual speech in auditory noise. In this paper we discuss potential limitations of these approaches and introduce a novel visual phonemic restoration method. This method always presents the same visual stimulus (e.g., /ba/) dubbed with a matched auditory stimulus (/ba/) or one that has weakened consonantal information and sounds more /a/-like). When this reduced auditory stimulus (or /a/) is dubbed with the visual /ba/, a visual influence will result in effectively 'restoring' the weakened auditory cues so that the stimulus is perceived as a /ba/. An oddball design in which participants are asked to detect the /a/ among a stream of more frequently occurring /ba/s while either a speaking face or face with no visual speech was used. In addition, the same paradigm was presented for a second contrast in which participants detected /pa/ among /ba/s, a contrast which should be unaltered by the presence of visual speech. Behavioral and some ERP findings reflect the expected phonemic restoration for the /ba/ vs. /a/ contrast; specifically, we observed reduced accuracy and P300 response in the presence of visual speech. Further, we report an unexpected finding of reduced accuracy and P300 response for both speech contrasts in the presence of visual speech, suggesting overall modulation of the auditory signal in the presence of visual speech. Consistent with this, we observed a mismatch negativity (MMN) effect for the /ba/ vs. /pa/ contrast only that was larger in absence of visual speech. We discuss the potential utility for this paradigm for listeners who cannot respond actively, such as infants and individuals with developmental disabilities.

8 citations (Crossref) [2023-10-31] Place: Netherlands Irwin, Julia. Haskins Laboratories, New Haven, CT, USA. Irwin, Julia. Southern Connecticut State University, New Haven, CT, USA. Avery, Trey. Haskins Laboratories, New Haven, CT, USA. Brancazio, Lawrence. Haskins Laboratories, New Haven, CT, USA. Brancazio, Lawrence. Southern Connecticut State University, New Haven, CT, USA. Turcios, Jacqueline. Haskins Laboratories, New Haven, CT, USA. Turcios, Jacqueline. Southern Connecticut State University, New Haven, CT, USA. Ryherd, Kayleigh. Haskins Laboratories, New Haven, CT, USA. Ryherd, Kayleigh. University of Connecticut, Storrs, CT, USA. Landi, Nicole. Haskins Laboratories, New Haven, CT, USA. Landi, Nicole. University of Connecticut, Storrs, CT, USA.