
Contents
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
45.1 Introduction 45.1 Introduction
-
45.2 Multimodal/Multimedia Input Interpretation 45.2 Multimodal/Multimedia Input Interpretation
-
45.2.1 Mechanisms for Integrating Modalities 45.2.1 Mechanisms for Integrating Modalities
-
45.2.2 Criteria for Modality Integration 45.2.2 Criteria for Modality Integration
-
45.2.3 Natural-Language Technology as a Basis for Multimodal Analysis 45.2.3 Natural-Language Technology as a Basis for Multimodal Analysis
-
45.2.4 Reconsidering Phenomena of Natural-Language Dialogue in a Multimodal Context 45.2.4 Reconsidering Phenomena of Natural-Language Dialogue in a Multimodal Context
-
45.2.5 Analysis of Emotional Signals in Natural-Language Dialogue 45.2.5 Analysis of Emotional Signals in Natural-Language Dialogue
-
-
45.3 Generation of Multimedia Output Including Natural Language 45.3 Generation of Multimedia Output Including Natural Language
-
45.3.1 Natural-Language Technology as a Basis for Multimedia Generation 45.3.1 Natural-Language Technology as a Basis for Multimedia Generation
-
45.3.2 Multimodal/Multimedia Coordination 45.3.2 Multimodal/Multimedia Coordination
-
-
45.4 Language Processing for Accessing Multimedia Data 45.4 Language Processing for Accessing Multimedia Data
-
45.4.1 NL-Based Video/Image Analysis 45.4.1 NL-Based Video/Image Analysis
-
45.4.2 Natural-Language Access to Multimodal Information 45.4.2 Natural-Language Access to Multimodal Information
-
45.4.3 NL Summaries of Multimedia Information 45.4.3 NL Summaries of Multimedia Information
-
-
45.5 Conclusions 45.5 Conclusions
-
Further Reading and Relevant Resources Further Reading and Relevant Resources
-
References References
-
-
-
-
-
-
-
-
-
-
45 Multimodal Systems
Get accessElisabeth André is a Professor of Computer Science and Founding Chair of Human-Centered Multimedia at Augsburg University, Germany. She has a long track record in multimodal human–machine interaction, embodied conversational agents, social robotics, and social signal processing. In 2010, she was elected a member of the prestigious Academy of Europe, and the German Academy of Sciences Leopoldina. In 2013, she was awarded a EurAI fellowship. In 2017 she was elected to the CHI Academy, an honorary group of HCI leaders. Since 2019, she has served as the Editor-in-Chief of IEEE Transactions on Affective Computing.
Jean-Claude Martin is Professor of Human–Computer Interaction, and Head of the Pluridisciplinary Research Group on ‘Cognition Perception Use’ at LIMSI-CNRS (Paris, France). He conducts research on the sensory-motor bases of social cognition in humans and in multimodal interfaces. He studies social skills training; autism; job and medical interviews; virtual coaches; sports, and e-health. He is Editor-in-Chief of the Springer Journal on Multimodal User Interfaces. He has been involved in several projects about blends of expressions of emotions in several modalities. He has supervised and co-supervised 15 defended PhD theses.
-
Published:05 December 2014
Cite
Abstract
Recent years have witnessed a rapid growth in the development of multimodal systems. Improving technology and tools enable the development of more intuitive styles of interaction and convenient ways of accessing large data archives. Starting from the observation that natural language plays an integral role in many multimodal systems, this chapter focuses on the use of natural language in combination with other modalities, such as body gestures or gaze. It addresses the following three issues: (1) how to integrate multimodal input including spoken or typed language in a synergistic manner; (2) how to combine natural language with other modalities in order to generate more effective output; and (3) how to make use of natural language technology in combination with other modalities in order to enable better access to information.
Sign in
Personal account
- Sign in with email/username & password
- Get email alerts
- Save searches
- Purchase content
- Activate your purchase/trial code
- Add your ORCID iD
Purchase
Our books are available by subscription or purchase to libraries and institutions.
Purchasing informationMonth: | Total Views: |
---|---|
October 2022 | 10 |
November 2022 | 8 |
December 2022 | 7 |
January 2023 | 10 |
February 2023 | 4 |
March 2023 | 5 |
May 2023 | 7 |
June 2023 | 7 |
July 2023 | 4 |
August 2023 | 6 |
September 2023 | 6 |
October 2023 | 10 |
November 2023 | 4 |
December 2023 | 2 |
February 2024 | 1 |
March 2024 | 14 |
April 2024 | 2 |
May 2024 | 7 |
June 2024 | 1 |
July 2024 | 6 |
August 2024 | 13 |
September 2024 | 10 |
October 2024 | 7 |
December 2024 | 4 |
January 2025 | 6 |
March 2025 | 4 |
April 2025 | 8 |
Get help with access
Institutional access
Access to content on Oxford Academic is often provided through institutional subscriptions and purchases. If you are a member of an institution with an active account, you may be able to access content in one of the following ways:
IP based access
Typically, access is provided across an institutional network to a range of IP addresses. This authentication occurs automatically, and it is not possible to sign out of an IP authenticated account.
Sign in through your institution
Choose this option to get remote access when outside your institution. Shibboleth/Open Athens technology is used to provide single sign-on between your institution’s website and Oxford Academic.
If your institution is not listed or you cannot sign in to your institution’s website, please contact your librarian or administrator.
Sign in with a library card
Enter your library card number to sign in. If you cannot sign in, please contact your librarian.
Society Members
Society member access to a journal is achieved in one of the following ways:
Sign in through society site
Many societies offer single sign-on between the society website and Oxford Academic. If you see ‘Sign in through society site’ in the sign in pane within a journal:
If you do not have a society account or have forgotten your username or password, please contact your society.
Sign in using a personal account
Some societies use Oxford Academic personal accounts to provide access to their members. See below.
Personal account
A personal account can be used to get email alerts, save searches, purchase content, and activate subscriptions.
Some societies use Oxford Academic personal accounts to provide access to their members.
Viewing your signed in accounts
Click the account icon in the top right to:
Signed in but can't access content
Oxford Academic is home to a wide variety of products. The institutional subscription may not cover the content that you are trying to access. If you believe you should have access to that content, please contact your librarian.
Institutional account management
For librarians and administrators, your personal account also provides access to institutional account management. Here you will find options to view and activate subscriptions, manage institutional settings and access options, access usage statistics, and more.