Face-to-face communication involves auditory signals, such as speech, and visual signals, such as visual speech and hand gestures. Especially these visual signals can enhance speech comprehension in adverse listening conditions, such as in noise. However, despite the abundance of visual expressions in language, most models and theories on the neurobiology of language are based on characteristics of (clear) speech and text, and they rarely consider multimodal signals. In this talk, I will argue that we need a multimodal view on the neurobiology of language, and that these visual signals are often taken on board immediately by listeners in creating and shaping an interpretation of the linguistic input. This talk will center around the question of how we, as language users, integrate auditory and visual signals into a coherent message, how this is orchestrated in the brain, and how we do this in both clear and adverse listening conditions. I will focus on how this is achieved both within and between conversational partners, and will discuss what is so ‘special’ about face-to-face communication.