Abstract
The pulvinar is a heterogeneous thalamic nucleus, which is well developed in primates. One of its subdivisions, the medial pulvinar, is connected to many cortical areas, including the visual, auditory, and somatosensory cortices, as well as with multisensory areas and premotor areas. However, except for the visual modality, little is known about its sensory functions. A hypothesis is that, as a region of convergence of information from different sensory modalities, the medial pulvinar plays a role in multisensory integration. To test this hypothesis, 2 macaque monkeys were trained to a fixation task and the responses of single-units to visual, auditory, and auditory–visual stimuli were examined. Analysis revealed auditory, visual, and multisensory neurons in the medial pulvinar. It also revealed multisensory integration in this structure, mainly suppressive (the audiovisual response is less than the strongest unisensory response) and subadditive (the audiovisual response is less than the sum of the auditory and the visual responses). These findings suggest that the medial pulvinar is involved in multisensory integration.
Introduction
Our perception of the environment is multisensory. Horn and Hill (1966) were among the firsts to report on the existence of multisensory neurons while recording in the superior colliculus. Single-units recordings in this subcortical structure later enabled Stein and Meredith to define the rules of multisensory integration (Meredith and Stein 1983, 1986). Associative cortical areas of the temporal, frontal, and parietal lobes were later shown to have multisensory properties (Stein and Stanford 2008; Meredith et al. 2020). For example, the superior temporal sulcus, the orbitofrontal cortex, and area 7 of the primate receive inputs from many unisensory cortical areas and individual neurons in these regions can respond to stimuli from several sensory modalities (Benevento et al. 1977; Leinonen et al. 1979; Bruce et al. 1981; Barraclough et al. 2005; Schlack et al. 2005; Sugihara et al. 2006; Avillac et al. 2007; Falchier et al. 2011 for a review). More recently, evidence that even unisensory cortical areas have multisensory properties has been reported (Cappe, Rouiller, et al. 2009 for a review): anatomical connections between unisensory areas of different modalities have been observed (Falchier et al., 2002; Cappe and Barone 2005; Falchier et al. 2011). Moreover, studies showed that the responses of neurons in the auditory (Lakatos et al. 2007; Kayser et al. 2008), visual (Wang et al. 2008), and somatosensory (Zhou and Fuster 2004) cortices can be modulated by the addition of another modality. All these results led Ghazanfar and Schroeder to the provocative proposition that the whole cortex might be multisensory (Ghazanfar and Schroeder 2006). Besides cortex, subcortical structures may also be involved in multisensory integration. In addition to the superior colliculus, the inferior colliculus also shows evidence for multisensory interaction (Gruters and Groh 2012 for a review). Audiotactile interaction has been observed in the cochlear nucleus (Heeringa et al. 2018; Ansorge et al. 2021). Multisensory neurons have been found in several nuclei of the thalamus. For example, in the ventral posterior medial region, somato-visual neurons were recorded in the mouse (Allen et al. 2017) and visuo-tactile integration was observed in rats (Bieler et al. 2018). Unisensory auditory, visual, and tactile neurons, along with bimodal and trimodal neurons, were found in the centre median and parafascicular nucleus of the macaque (Matsumoto et al. 2001).
Among thalamic nuclei, the medial pulvinar appears as an ideal candidate for multimodal integration. Located posteriorly in the thalamus, the medial pulvinar presents reciprocal connections with unisensory cortical areas (auditory, visual, and somatosensory cortices) (Cappe et al. 2007; Cappe, Rouiller, et al. 2009; Homman-Ludiye et al. 2020), polysensory areas (superior temporal cortex, prefrontal cortex, posterior cingulate cortex, insular cortex) (Romanski et al. 1997; Gutierez et al. 2000; Homman-Ludiye et al. 2020), and motor areas (dorsal premotor cortex) (Morel et al. 2005; Homman-Ludiye et al. 2020). Furthermore, among the thalamic nuclei receiving inputs from several modalities, the medial pulvinar is the one with the densest connectivity and with the greatest overlap (Cappe, Morel, et al. 2009). It also receives afferents from subcortical structures such as the superior colliculus (Trojanowski and Jacobson 1975; Benevento and Standage 1983) and is interconnected with the amygdala (Jones and Burton 1976). These numerous connections suggest that it is a region of convergence of information from different sensory modalities.
Relatively little is known about the functions of the pulvinar (Froesel et al. 2021 for a review). First, lesion studies in humans (Ward and Arend 2007; Snow et al. 2009; Wilke et al. 2018) and inactivation studies in monkeys (Wilke et al. 2010) showed deficits in visual attention and distractor filtering as well as deficits in localizing visual targets and in the control of visually guided eye and limb movements. However, these behavioral studies did not examine sensory modalities other than visual. Second, the pulvinar is involved in the communication between cortical areas (Saalmann and Kastner 2011; Benarroch 2015). Third, electrophysiological studies showed that neurons in the pulvinar respond to auditory (Gattass et al. 1978; Yirmiya and Hocherman 1987), somatosensory (Mathers and Rapisardi 1973; Gattass et al. 1978; Avanzini et al. 1980), and visual stimuli (Mathers and Rapisardi 1973; Gattass et al. 1978; Benevento and Miller 1981; Wilke et al. 2009; Maior et al. 2010; Komura et al. 2013; Nguyen et al. 2013; Van Le et al. 2013, 2014, 2016). However, only three studies tested stimuli of different modalities on the same neurons. The first one (Mathers and Rapisardi 1973) reported responses to visual and somatosensory stimuli in the pulvinar of the squirrel monkey but did not report on multimodal interaction in this structure. The second one (Gattass et al. 1978) found convergence of auditory, visual, somatic, and olfactory inputs on neurons of various regions of the pulvinar but did not examine multisensory interaction. The third one (Avanzini et al. 1980) reported visual, acoustic, and somesthesic responses in neurons of the pulvinar-lateralis posterior complex of the cat’s thalamus. This study reported an inhibitory interaction between visual and somesthesic stimuli in the pulvinar nucleus, but interactions between auditory and visual stimuli in this structure were not explored. Thus, to the best of our knowledge, the features of audiovisual interactions in the primate pulvinar are currently unknown. In the present study, we recorded single-units in the medial pulvinar of 2 awake macaques performing a fixation task. Single-unit data confirmed the presence of auditory, visual, and audiovisual neurons in the medial pulvinar. Furthermore, it revealed the existence of a new type of multisensory neurons, which we named “complex multisensory”. These neurons responded to more than one modality, but the modality to which they responded was different depending on the stimulus category. We also provide evidence of multisensory integration in this structure, which was mostly suppressive and subadditive.
Materials and methods
Animals
Two adult male rhesus monkeys (Macaca mulatta), weighting 7 and 5 kg and 5-years-old, participated in this experiment. They were naive to all procedures. They were housed together at night and separated during the day in cages for food and water delivery. Water was available ad libitum as soon as they were in cage, whereas access to food (fruits, vegetables, and cereals) was controlled and given after each training or recording session. A weight loss criterion of 10% was decided for stopping the experiment, but this never happened. All procedures were approved by the National Committee for Ethical Reflection on Animal Testing in compliance with the guidelines of the European Community on Animal Care (authorization number: 01000.02).
Experimental procedures
The monkeys were trained in a fixation task. They were sitting in a primate chair (Crist instruments), with the head fixed, in a darkened, sound-attenuated box, in front of a 31-cm distant screen (BenQ, 60-cm diagonal, 1920 × 1080 pixels, 120 Hz). Two loudspeakers (Creative Gigaworks t20 serie II) were positioned one on each side of the screen. The training took place 5 days a week, on the morning. As the monkeys were food deprived, the training was their first access to food. Each correct trial was rewarded with diluted compote (0.05 mL/trial).
The task was as follows. To initiate a trial, the monkey had to fixate a 0.5° point into a square fixation window of 2 × 2° for a random time between 500 and 1200 ms. Then, a stimulus was presented for 250 ms (auditory, visual, or audiovisual stimulus). The monkey had to maintain fixation during stimulus presentation, while the fixation point was extinguished. If fixation was not broken, the monkey was rewarded after a random delay (300–700 ms). Otherwise, the trial was aborted. The intertrial interval was 1000 ms. The task was controlled with EventIDE software (Okazolab Ltd), and eye tracking was performed with an eye-tracker (ISCAN ETL 200, Woburn, MA, USA).
A total of 9 stimuli was used: 3 auditory, 3 visual, and 3 audiovisual stimuli. The auditory stimuli consisted in white noise, in a macaque “coo” call, and in a rattlesnake rattle. The auditory stimuli were stereo 44–48 kHz waves normalized to 60 dB, with a 3-ms fading-in and fading-out. The visual stimuli were a picture of random dots, a rhesus macaque head photograph, and a rattlesnake picture. All images were normalized in RGB colors with a color depth of 24, sized 453 × 453 pixels with a 72 × 72 dpi resolution (final size 19 × 19°), with a mean luminance of 120 cd/m2, and presented in the center of the screen. There was only one exemplar of each stimulus (macaque, snake, noise). The audiovisual stimuli were the synchronous presentation of an auditory and a visual stimulus, always semantically congruent. As pulvinar neurons can respond to both low level features (Mathers and Rapisardi 1973; Gattass et al. 1978; Benevento and Miller 1981) and high level stimuli (Maior et al. 2010; Nguyen et al. 2013; Van Le et al. 2013), it cannot be determined whether individual neurons responded to the components of the stimuli or to the stimuli as a whole. The stimuli were randomly presented, about 20 times each. A 4 × 4° white square appeared in the upper right corner of the screen (hidden from the monkey’s view by black tape) at the same time as the stimulus presentation, to precisely indicate onset and offset of the stimulus.
Surgical procedures
A first surgery was performed to implant an MRI-compatible headpost (Crist Instrument) to the skull. After 1 month of recovery, the monkeys were trained in the fixation task. When they reached 90–95% correct trials, a second surgery was performed to stereotaxically implant a footed stainless-steel recording chamber (Crist Instrument) above the right somatosensory cortex (S2) (centered on average AP = 6.5 and ML = 0.75). The skull within the chamber was removed to allow direct access to the brain. The dura mater was left intact and was protected by a sterile silicone patch.
All surgeries were performed under anesthesia, induced with a mixture of tiletamine/zolazepam (Zoletil 50®, 5 mg/kg) and glycopyrrolate bromide (Robinul®, 0.01 mg/kg), and maintained with isoflurane (1.5%) after intubation. Analgesics were administered (Tolfedine 4 mg/kg and buprenorphine chlorhydrate [Vetergesic®] 0.01 mg/kg) during surgery and the following days. An antibiotic treatment (Amoxicillin [Clamoxyl® LA] 15 mg/kg) was administered during the first week.
The location of the recording chamber was determined before surgery by comparing stereotaxic 3 T anatomical MRI scans to sections of stereotaxic atlases of the brain of M. mulatta (Paxinos et al. 2000; Saleem and Logothetis 2007). After implantation, the locations of the recording sites were confirmed by comparing the atlas sections to stereotaxic anatomical MRI scans of the monkeys’ head with a tungsten microelectrode inserted at target coordinates. The locations of the recording sites are relative to the zero coordinates defined in the stereotaxic atlases.
Recordings
Neuronal activity was recorded with tungsten microelectrodes (5–7 MΩ at 1 kHz, Frederick Haer Company, Bowdoinham, ME). The electrodes were inserted daily into the medial pulvinar (ML between 4.9 and 9.8 mm, AP between 5 and 9.8 mm, depth between 17.9 and 23.8 mm below the cortical surface). The electrode insertion was performed with an oil hydraulic micromanipulator (Narishige MO-972) attached to the recording chamber.
The recorded signal was sampled at a frequency of 40 kHz with a 1401 power acquisition interface (CED, Cambridge, UK), amplified with a gain of 1000 (NL104) and band-pass filtered between 200 Hz and 8 kHz (NL125) by a Neurolog system (Digitimer, Hertfordshire, UK). The 50-Hz noise was eliminated with a Humbug device (Digitimer). The signal was displayed and recorded with Spike2 software (CED). Data were analyzed with custom scripts in the Spike2 software environment.
Single-unit isolation
The collected data usually contained spikes issued from several neurons. For single-unit isolation, spike sorting was performed with the PCA-based cluster analysis of Spike2 software. First, 1- to 1.5-ms-wide portion of the voltage trace was extracted online around putative spikes whenever the peak amplitude exceeded a threshold set at 3 times the noise level. The trace chunks were then analyzed offline by PCA. A 3D rendering of the PCA results allowed visualizing one or several clusters, where each cluster groups spike shapes that are similar. Spikes grouped in a given cluster were ascribed to single-unit activity if the corresponding interspike interval distribution showed a clear refractory period, i.e. no interval < 1.2 ms. In contrast, multiunits show no refractory period since they group several independent neurons. Based on our statistical analysis (see below), we found that in all but 3 cases, single-units simultaneously recorded at a given site exhibited distinct response patterns with respect to stimuli categories and modalities.
Classification of neurons
Only neurons with sufficiently high activity (more than 800 spikes on the whole recording period) were analyzed. To determine if there was a response to the stimuli, we first calculated, for each stimulus presentation, the mean firing rate per trial during stimulus presentation (number of spikes occurring during the 0.25-s-long stimulus presentation divided by 0.25 s). We similarly calculated the mean spontaneous activity per trial during the period preceding each stimulus presentation (−500–0 ms). The distributions of the ≈20 values of spontaneous and evoked firing rates per trial obtained for each type of stimulus were then compared with the Wilcoxon signed-rank test. A P-value of 0.05 was used as a threshold to indicate the presence of significant excitatory or inhibitory responses. Responses to different stimuli were then compared using the Kruskal–Wallis test and the Mann–Whitney test applied to the distributions of mean firing rates calculated for each stimulus presentation. Again, the threshold P-value was 0.05, and a Bonferroni correction was applied for multiple comparisons.
This allowed classifying neurons in 4 classes:
The first class corresponds to unisensory visual neurons. These neurons showed significant responses to both visual and audiovisual stimuli, but the response to visual and audiovisual stimuli did not differ significantly.
The second class corresponds to unisensory auditory neurons, characterized by a significant response to both auditory and audiovisual stimuli, but without significant differences between these 2 responses.
The third class corresponds to multisensory neurons that we named “classic” because they have already been encountered and described in previous multisensory studies (e.g. Horn and Hill 1966; Meredith and Stein 1983; Allman et al. 2009; Kayser et al. 2011; Olcese et al. 2013; Meredith et al. 2020). This class includes several subtypes of bimodal neurons:
(i) Bimodal neurons that responded significantly only to the congruent presentation of auditory and visual response and that did not respond when either auditory or visual stimuli were presented alone (AV cells).
(ii) Bimodal neurons that responded to both auditory and visual stimuli presented alone, but not to the combination of the two modalities (A, V cells).
(iii) Bimodal neurons that responded to both unisensory stimuli and to their combinations (A, V, AV cells).
The fourth class, which has not been described previously, corresponds to “complex” multisensory neurons. We named these cells “complex” because of the profile of their responses. Depending on the stimulus category, these neurons displayed either unisensory or multisensory responses. Complex multisensory neurons were further subdivided according to the modalities to which they responded.
A first group of complex neurons behaved as unimodal auditory neuron with 1 or 2 stimulus categories and as unimodal visual neurons with the remaining categories (A & V cells). For example, a cell that responded significantly to the sound of the rattlesnake and to the image of the monkey would belong to this subtype. The other 3 groups of complex neurons presented multimodal responses to 1 or 2 stimulus categories and unimodal responses to the remaining stimulus categories.
Thus, neurons could be unimodal visual for 1 or 2 stimulus categories and bimodal for the remaining ones (V & AV cells). For example, a cell could respond as a unimodal visual cell for the noise stimuli and as a bimodal cell for the snake and/or rhesus stimuli.
Likewise, neurons could be unimodal auditory for 1 or 2 stimulus categories and bimodal for the remaining ones (A & AV neurons).
The fourth type of complex audiovisual neurons corresponds to auditory–visual-audiovisual neurons (A & V & AV); these cells were auditory for one stimulus category, visual for another category, and audiovisual for the last category. This subclassification is independent of the number of stimulus categories used, as it is based only on the modalities that actually activated the cells.
The statistical tests to classify the neurons were performed on the individual stimuli (i.e. visual macaque) and with all the stimuli of a given modality pooled together (i.e. visual). Thus, a neuron could be classified as visual because it responded significantly to one visual stimulus or because it responded when all responses to visual stimuli of all stimuli were grouped together.
We also computed peri-stimulus time histograms (PSTHs) to illustrate the time course of the responses. These PSTHs were computed following conventional methods, by summing the number of spikes occurring within 10-ms-wide bin across stimulus repeats, then normalizing this sum by the bin width to obtain a firing rate in spikes/s. The PSTHs were then smoothed with a 45-ms moving average.
Neurons’ responses characterization
When neurons were multisensory for more than one stimulus category, the corresponding indexes were averaged. With the formula presented above, ADI and AMI are restricted to the range −100% to +100%.
Latencies
Latencies were calculated using the half-rise latency method (Levick 1973; Nowak et al. 2010). To improve the temporal resolution while preserving a good signal-to-noise ratio, 10 PSTHs with a bin width of 10 ms were computed with a 1-ms offset increment for each calculation. The 10 PSTHs were then averaged with a 1-ms resolution. The peak amplitude and the half-peak amplitude were determined. Only peaks with a half-peak amplitude higher than the mean + 3 SD in PSTH prestimulus baseline (−500–0 ms) were considered. The time after stimulus onset at which the half-peak amplitude is first reached corresponds to the half-rise latency. A further constraint was that the amplitude remained ≥ half-peak height for at least 7 consecutive ms. Latencies have been obtained for 69 neurons, yielding a total of 160 latencies as a given neuron could respond to several stimulus categories and modalities. Note that this method, which requires assessment of significance over the time course of the response, is more constrained than that used for determining presence of significant response over the whole stimulus presentation (“classification of neurons” above). This explains why the number of neurons with significant latencies is less than the number of neurons with significant responses. To examine the influence of stimulus modality, we reduced the dataset by taking, for each neuron, the shortest latency within each modality.
Data availability
The datasets supporting the current study have not been deposited in a public repository because they are still used for the analysis of local field potentials but are available from the corresponding author on request.
Results
Medial pulvinar contains auditory, visual, and audiovisual neurons
We recorded spiking activity in response to visual, auditory, and audiovisual stimuli in the medial pulvinar of 2 macaques engaged in a fixation task (Fig. 1). 374 single-units were recorded in 164 sites and 328 analyzed for responsiveness to the stimuli (46 neurons were excluded because of too low activity or fluctuating baseline). Most (213/328, 65%) medial pulvinar neurons responded to auditory, visual, and/or audiovisual stimuli. These single-units were all spontaneously active, with a median spontaneous firing rate of 3.9 spikes/s. During stimulus presentation, the neurons either showed no change in firing rate (no response), an increase of firing rate (mean + 4.2 spikes/s relative to spontaneous activity level, global population response Fig. 2A), or a decrease in activity (mean −1.2 spikes/s, global population response Fig. 2B).
Experimental design. A) The monkey sat in a primate chair in front of a computer screen. After fixation of the central point (for a random duration comprised between 500 and 1200 ms), an auditory, visual, or audiovisual stimulus was presented for 250 ms. The monkey had to maintain fixation during stimulus presentation to obtain a reward. B) Three static visual stimuli were presented (always the same picture for each): A picture of random dots, a macaque face and a rattlesnake. C) T3 MRI slices of the brain of monkey C in a frontal section (top, left) and in a sagittal section (top, right) showing electrode position. Drawings below MRI pictures show the cerebral regions delimited by the boxes in the MRI slices, labeled according to macaque brain atlases (Paxinos et al. 2000; Saleem and Logothetis 2007). PuM: Medial pulvinar, PuL: Lateral pulvinar, PuI: Inferior pulvinar, PuA: Anterior pulvinar, BrSC: Superior colliculus brachium, LV: Lateral ventricle, cc: Corpus callosum.
Global responses and individual neuron examples. a and b) Global PSTHs representing the mean firing rate (sp/s) of neuronal populations according to the response types: increase (A) or decrease (B) in firing rate. Significant responses were averaged over all stimuli and all modalities. Since a given neuron could yield between 1 and 15 significant responses (9 individual stimuli, 3 modalities with all stimuli pooled, 3 categories with all modalities pooled), the total number of responses (159 + 170) is larger than the number of neurons with significant responses (213). Black line corresponds to the mean and gray shading to ±95% CI. c–e) Examples of auditory neuron (C), visual neuron (D), and classic bimodal audiovisual neuron (E). Top: PSTHs were calculated across all trials (all stimulus categories pooled) for a given modality (auditory in light gray, visual in gray, and audiovisual in black), with a bin width of 10 ms and smoothed with a 45-ms moving average. Bottom: Boxplots summarizing distributions of mean firing rates per trial for each stimulus presentation in response to auditory (A), visual (V), and audiovisual (AV) stimuli. Paired spontaneous activity measured during 500 ms before stimulus onset has been subtracted from each response. Responses were considered as significant when the distribution of firing rates during stimulus presentation was different from the distribution of preceding spontaneous firing rates, as tested with a Wilcoxon signed rank test. Sharps below boxplots indicate significance level: #: P < 0.05, ##: P < 0.01, and ###: P < 0.001. Modulation of response amplitude by the modalities was evaluated by the Kruskal–Wallis test, followed by Mann–Whitney tests between modality (P-values adjusted by Bonferroni correction). Asterisks above boxplots indicate significance level: *: P < 0.05, **: P < 0.01, and ***: P < 0.001.
The 213 responsive neurons were individually found to be auditory, visual, audiovisual, or nonspecific to one modality (19 neurons for which the response to at least one stimulus category was significant only when all modalities were pooled together—these neurons were not included in the analysis of multimodal interactions). Unisensory cells responded to only one sensory modality and responded identically to audiovisual stimuli. The 2 neurons presented in Fig. 2C, D are unisensory cells (2.C auditory, 2.D visual). The neuron presented in Fig. 2C was classified as a unisensory auditory neuron: the auditory stimuli induced an increase in activity that was statistically significant when compared with baseline activity. Similarly, there was a statistically significant increase in activity in response to the bimodal audiovisual stimuli, but the activity level was not statistically different from the one obtained in response to the auditory stimuli alone. Presentation of the visual stimuli had no significant effect on activity. The neuron presented in Fig. 2D was a unisensory visual neuron. The activity increased by the same amount in response to visual and bimodal audiovisual stimuli, but the activity did not change in the presence of the auditory stimuli presented alone. At the population level, the medial pulvinar appears to have slightly more auditory than visual unisensory neurons (χ2 = 6,94, df = 1, P = 0.008): among the 213 responsive neurons, 66 (31%) were classified as auditory, and 39 (18%) as visual (Fig. 3A).
Proportions of single-units classified by sensory modalities and multimodal integration types. A) Pie chart representing the proportions of single-units classified as unimodal visual, unimodal auditory, classic audiovisual, complex audiovisual, or nonspecific to a modality. B) Pie chart representing the proportions of complex audiovisual neurons, classified according to the modalities to which they responded.
We found 53 audiovisual neurons (25% of the responsive neurons, Fig. 3A) whose response features corresponded to those described in previous studies (e.g. Stein and Meredith 1993; Allman et al. 2009; Kayser et al. 2011; Olcese et al. 2013; Meredith et al. 2020). In order to account for this anteriority, we named these neurons “classic” audiovisual neurons. These neurons could be further subclassified in 3 classes that correspond to the different types of bimodal neurons that have been previously described. The first class includes neurons that did not respond significantly to either auditory or visual stimuli when presented alone but that responded to the combined auditory–visual stimuli (Benevento et al. 1977; Meredith and Stein 1986; Olcese et al. 2013). It represents 81% of the classic audiovisual neurons in our sample. The second class, which represents 11% of the classic audiovisual neurons, groups cells which responded in all sensory conditions; an example is presented in Fig. 2E. For this neuron, the visual, auditory, and audiovisual stimuli induced an increase in activity that was statistically significant in all cases when compared with baseline activity. Such neurons have been described in many studies, for example in the superior colliculus (e.g. Horn and Hill 1966; Meredith and Stein 1983, 1986; Bell et al. 2001; Populin and Yin 2002). The third class corresponds to neurons with multisensory suppression (e.g. Meredith and Stein 1983, 1986; Meredith et al. 1987): they responded to both unimodal auditory and visual stimuli, but their response to the bimodal stimuli was not significant. They represent 8% of the classic audiovisual neurons.
Medial pulvinar contains complex audiovisual neurons
Thirty-six neurons were found to possess “complex” audiovisual responses (17% of the responsive neurons, Fig. 3A). These are cells for which the sensory modality that evoked a response depended on the stimulus category. For example, the neuron in Fig. 4 presented a statistically significant increase in activity in response to the noise stimulus in the visual, auditory, and audiovisual modalities. This neuron would therefore be classified as bimodal when the stimulus was the noise stimulus in either modality. Likewise, the neuron displayed a bimodal response when the snake stimulus was used, as the activity increased in both visual, auditory, and audiovisual modalities. However, when the macaque stimulus was used, a significant increase in activity was observed only in the visual and audiovisual modalities; this cell did not respond to the auditory macaque stimulus. Furthermore, it responded equally well to the visual and audiovisual macaque stimuli. Therefore, when macaque images or sounds were used as stimuli, this neuron showed features of a unisensory visual neuron. Given this dual response pattern, it was classified as visual-audiovisual complex multisensory neuron (V & AV), like 28% of the complex audiovisual neurons. We found 3 other types of complex audiovisual neurons (Fig. 3B): auditory-audiovisual (A & AV, 56% of the neurons), which responded as unisensory auditory neurons for 1 or 2 stimulus categories, and as bimodal neurons with the remaining categories (A & AV neurons). We also encountered auditory–visual neurons (A & V, 8%), which behaved as unisensory auditory neuron with 1 or 2 stimulus categories and as unisensory visual neurons with the remaining categories. The last type of complex audiovisual neurons corresponds to auditory–visual-audiovisual neurons (A & V & AV, 8%); these cells were auditory for one stimulus category, visual for another category, and audiovisual for the last category. This subclassification is independent of the number of stimulus categories used, as it is based only on the modalities that actually activated the cells.
Example of a complex audiovisual neuron. First column: Responses to noise stimuli; second column: Responses to macaque stimuli; third column: Responses to snake stimuli. Boxplots summarizing distributions of mean firing rate per trial (spontaneous activity subtracted) for each stimulus are shown below each corresponding PSTHs (auditory in light gray, visual in gray and audiovisual in black). This complex neuron was assigned to the visual modality for macaque stimulation and to the multisensory modality for noise and snake stimuli. Significance of responses was tested using Wilcoxon test (#: P < 0.05, ##: P < 0.01, and ###: P < 0.001). There was no significant modulation (enhancement or suppression) with the bimodal stimulation (Mann–Whitney test, adjusted P-values > 0.05).
Medial pulvinar neurons are poorly selective
The sparseness and the SEIs indicate that medial pulvinar neurons responded to many stimuli and tended to be poorly selective. The SPI quantifies the variability of the responses to all 9 stimuli used. It is indicative of a sparse representation when the index approaches 0 and of a distributed representation when the index reaches values close to 1. In our sample, it ranges from 0.42 to 1, with a median at 0.93 (Fig. 5). The response features of unisensory neurons may bias the SPI toward high values. Indeed, unisensory neurons responded similarly to the unisensory and the corresponding audiovisual stimuli. However, the distribution of the SPI remained similar when computed only on multisensory neurons (minimum: 0.42, median: 0.93, and maximum: 0.99; not illustrated).
Neuronal selectivity and sparseness. The SPIs of the 213 responding cells are plotted as a function of the SEIs (bottom left). Indexes distributions are shown on the top (SEI) and on the right (SPI). A SPI close to 1 is indicative of a neuron responding similarly to all stimuli. The SEI indicates whether the neuron responded to only one stimulus (index close to 1) or whether it responded equally well to the preferred and least preferred stimulus (index close to 0).
The SEI compares the response of the neuron to the best and the worst stimulus. An index close to 1 is indicative of a very selective neuron, which responded to only one stimulus, whereas an index close to 0 is indicative of a poorly selective neuron, which responded almost as well to the worst as to the best stimulus. The SEI ranges from 0.10 to 1, with a median at 0.43 (Fig. 5).
Figure 6 illustrate averaged PSTHs for each stimulus and modality. Although some neurons could respond selectively to one stimulus category, there was no overall preference for one or another category at the population level. In addition, at the population level response, amplitudes were not different between snake, macaque, and noise stimuli, whatever the modality (Kruskal–Wallis test, all P-values > 0.05).
Global responses of neuronal populations according to the response types. Global PSTHs representing the average firing rate (sp/s) of neuronal populations according to the responses types, for each stimulus and modality: Excitatory (A) or inhibitory (B) responses. Each row of PSTHs (top to bottom) corresponds to auditory, visual, audiovisual, and pooled modality conditions. Each column of PSTHs (left to right) corresponds to noise, macaque, snake, and pooled stimuli conditions. For each PSTH, we only used the neurons showing a significant response to the corresponding stimulus (or pooled stimuli). Responses were averaged over all neurons showing a statistically significant excitatory (A) or inhibitory (B) response for each stimulus, without (white background) or after (gray background) pooling across stimuli or modalities. The mean spontaneous activity has been subtracted before averaging. The PSTH with the dark gray background represents population average for all enhanced or all decreased responses (all the responses used for the PSTHs with white backgrounds) (same as Fig. 2A, B). Black line corresponds to the mean and gray shading to ±95% CI.
Multisensory integration in the medial pulvinar is subadditive and suppressive
Additivity and AMIs, calculated for each multisensory cell, revealed subadditive and suppressive multisensory integration at the population level in the medial pulvinar. The AMI allows determining if the multisensory response is enhanced (index > 0) or suppressed (index < 0) compared with the best unisensory response. The ADI informs about the linearity of interaction and can reveal subadditive (index < 0), additive (index = 0), or supra-additive (index > 0) interactions between unisensory and multisensory responses.
AMI ranges between −61% and 34%. At the population level, the AMI revealed a dominance of suppressive interaction (median = −3%; sign test, P = 0.004).
ADI ranges between −74% and 13%. Unsurprisingly, at the population level, the ADI revealed an essentially subadditive audiovisual interaction (median = −34%; sign test, P = 10−23). There is a continuum from subadditive and depressed to supra-additive and enhanced responses (Fig. 7) and the majority of the cells are subadditive depressed cells.
Multisensory integration: additivity and AMIs. The AMIs of 96 multisensory cells are plotted as a function of ADIs (bottom left). Indexes distributions are shown on the top (ADI) and on the right (AMI). The brackets are merely indicative of the meaning of the index values and are not intended to indicate statistical significance.
Faster responses for auditory and audiovisual than for visual stimuli
Stimulus modality strongly affected response latencies (Kruskal–Wallis test, P < 0.0001). The medians of the half-rise latencies for auditory and audiovisual responses were 89 ms (n = 35) and 75 ms (n = 46), respectively. The distributions for auditory and audiovisual latencies largely overlapped and did not differ significantly (Posthoc Dunn’s test, P > 0.99). Visual latencies, with a median equal to 141 ms (n = 30), were longer than both auditory (P = 0.003) and audiovisual latencies (P < 0.0001). The cumulative distribution shows a systematic lag for visual responses by about 50 ms (Fig. 8).
Latencies of medial pulvinar neurons. A) Cumulative distribution of half-rise latencies for each modality: Visual (V, 30 latencies) in gray, auditory (A, 35 latencies) in light gray, and audiovisual (AV, 46 latencies) in black. In order to have a single value per neuron and modality, only the shortest latency within each modality was taken into account. B) Distribution of half-rise latencies for visual responses produced by the presentation of noise (n = 13), macaque (n = 18), and snake (n = 15) pictures. Dots represent individual data, horizontal central bars correspond to the medians, and error bars indicate the 95% confidence interval of the medians. Latencies were significantly longer for responses to macaque stimuli. Asterisks indicate significance level obtained with Posthoc Tukey’s test: *: P < 0.05; **: P < 0.01.
Latencies were also compared between stimuli in each modality. These comparisons did not reveal differences for the auditory and audiovisual modalities (mixed effect model, P = 0.13 and 0.09, respectively). However, as summarized in Fig. 8B, differences between stimuli were observed in the visual modality (P = 0.01) and revealed longer latencies for macaque stimulus (median 183 ms, n = 18) compared with noise (median 137 ms, n = 13) and snake (median 150 ms, n = 15) (Posthoc Tukey’s test: P = 0.002 for comparison between macaque and noise, P = 0.02 between snake and macaque, P = 0.06 between noise and snake).
Sensory modalities are intermingled in the medial pulvinar
A 3D map was constructed to report the location of the recording sites in the explored region (2D representation in Fig. 9). No topographic organization is visible for the sample of recorded neurons: neurons of all types were present in variable proportions at each recording site. This was confirmed by a PCA in 3D space by using the three dimensions of recording site coordinates (AP, ML, and depth).
Sensory modalities are intermingled in the medial pulvinar. Topographic representation of neuronal responses to each modality. Representation of response modalities of the 213 cells with significant responses as a function of the anteroposterior (y axis, from 2 to 6 mm) and mediolateral (x axis, from 5 to 9 mm) coordinates of electrode penetration sites. Response modality is gray-scale-coded: Light gray corresponds to auditory neurons, dark gray to visual neurons and black to classic and complex audiovisual neurons pooled together. The radius of the circles is proportional to the number of neurons that responded to each modality.
Discussion
Unisensory and multisensory neurons in the medial pulvinar
In the present study, we investigated neuronal responses to auditory, visual, and audiovisual stimuli in the medial pulvinar of awake monkeys. Numerous studies have demonstrated neuronal responses to visual stimuli in the pulvinar (Mathers and Rapisardi 1973; Gattass et al. 1978; Avanzini et al. 1980; Benevento and Miller 1981; Wilke et al. 2009; Maior et al. 2010; Komura et al. 2013; Nguyen et al. 2013; Van Le et al. 2013, 2014, 2016), although few studies have specifically examined the medial part of the pulvinar (Maior et al. 2010; Van Le et al. 2013). A few studies have also shown the presence of responses to auditory (Gattass et al. 1978; Yirmiya and Hocherman 1987), somatosensory (Mathers and Rapisardi 1973; Gattass et al. 1978; Avanzini et al. 1980), and olfactory stimuli (Gattass et al. 1978) in the pulvinar. Only one study has documented multisensory interactions in the cat pulvinar (Avanzini et al. 1980); suppressive interactions were reported for somesthetic and visual stimuli. We confirmed that pulvinar neurons are responsive to visual stimulations and auditory stimulations. We also reported multisensory audiovisual neurons and we described and quantified multisensory integration in the medial pulvinar. Most of the multisensory neurons that we encountered correspond to previously described bimodal cells (e.g. Horn and Hill 1966; Benevento et al. 1977; Stein and Meredith 1993; Olcese et al. 2013; Meredith et al. 2020). There exist another type of multimodal cells, the modulatory (or subthreshold) neurons: neurons responding to a single modality when tested with unimodal stimuli (similarly to unisensory cells), but for which the unisensory response would have been different from that obtained with the audiovisual stimuli. Although often observed in cortex (Allman and Meredith 2007; Avillac et al. 2007; Meredith et al. 2020), we did not find such cell in our sample. However, some of our bimodal neurons were modulated and presented responses statistically different between the unisensory and the audiovisual stimuli.
Another novelty in our results lies in the identification of complex audiovisual neurons. The modality to which these neurons responded depended on the stimulus category. This type of neurons has not been described previously. Indeed, identifying these neurons requires different types of multisensory stimuli. Instead, previous studies most often relied on the use of a single multisensory stimulus.
Multisensory integration in the medial pulvinar
The presence of numerous audiovisual neurons prompted us to examine their multisensory integration properties. Multisensory integration was globally suppressive (AMI < 0) and subadditive (ADI < 0). Studies on multisensory integration performed in the superior colliculus of anesthetized and awake cats and monkeys (Meredith and Stein 1983, 1986; Meredith et al. 1987; Wallace et al. 1996, 1998; Bell et al. 2001; Perrault Jr et al. 2003) reported either response suppression or response enhancement, depending on stimulus features. Enhanced responses were most often observed when stimuli were coincident in space and time and when stimulus intensities were in the low to medium range. Studies examining multisensory integration in cortex reported varying degrees of bimodal interactions. Some studies reported a dominance of suppressive effect (Benevento et al. 1977; Sugihara et al. 2006; Avillac et al. 2007; Kayser et al. 2008), other studies a dominance of enhanced responses (Wallace et al. 1992; Allman and Meredith 2007; Olcese et al. 2013) and still other studies found roughly equivalent numbers of suppressed and enhanced cells (Barraclough et al. 2005; Meredith et al. 2020). As in the superior colliculus, manipulation of stimulus congruency and intensities can alter and possibly reverse the sign of multimodal interactions (Avillac et al. 2007; Kayser et al. 2008; Olcese et al. 2013).
In addition to the recording sites and stimulus features, the task can also influence multisensory interactions: Bell et al. (2003) showed that the mere presentation of the fixation point could decrease multisensory integration in the superior colliculus. However, in our study, the fixation point was extinguished during the presentation of the visual and auditory stimuli. Another explanation for the dominance of suppressive multisensory integration is the strength of the stimuli: our stimuli were well above perception threshold. According to the law of inverse effectiveness (Meredith and Stein 1983, 1986), weak stimuli benefit more from multisensory integration and favor the occurrence of enhanced responses (Meredith and Stein 1986; Perrault Jr et al. 2003; Kayser et al. 2008). Finally, in our behavioral task, the audiovisual stimuli had no more value than the unisensory stimuli. Multisensory integration was not mandatory for the task. This could also have contributed to the occurrence of subadditive and suppressive audiovisual interactions.
Latencies
Our study disclosed median latencies of 89 ms for auditory stimuli, 75 ms for audiovisual stimuli, and 141 ms for visual stimuli. One study reported a range of visual latencies comparable to ours (Maior et al. 2010), while shorter latencies have been reported in the pulvinar in other studies (Petersen et al. 1985; Nguyen et al. 2013; Van Le et al. 2013). Differences between studies could explain this discrepancy: first, our method for latency measurement is different from that used in these studies; second, our recordings were restricted to the medial pulvinar.
We found that visual latencies were much longer, by about 50 ms, than auditory latencies. Longer visual latencies have been observed in the superior colliculus in awake (Wallace et al. 1998; Populin and Yin 2002) and anesthetized (Meredith et al. 1987; Wallace et al. 1996) animals, with comparable or even larger differences (50–80 ms). Studies exploring multimodal cortical areas have also shown longer latencies for visual responses compared with auditory responses (Bruce et al. 1981; Schlack et al. 2005).
It is known that the strength of cross-modal interactions depends on the relative timing of each stimulus modality (Meredith et al. 1987). However, we did not adjust the stimulus onsets to account for the difference in latency between modalities. One reason for not doing this is that we had no a priori knowledge of what the latency difference would be in the medial pulvinar. Another reason is that, in natural settings, objects may manifest as simultaneous auditory and visual cues, although our daily lives furnishes many instances of visual cues preceding auditory cues, and vice versa. It is to be noticed that interactions are quite close to optimal with a temporal disparity of ±100 ms (Allman and Meredith 2007). Moreover, the temporal overlap of the responses to auditory and visual stimuli (Fig. 6) suggests that multisensory interactions would have ample room to develop despite differences in onset latency.
Functions of the medial pulvinar
The functional role of the pulvinar remains largely unknown, yet some hypotheses have been put forward (Froesel et al. 2021).
Anatomical studies have shown many sensory connections with the pulvinar (Cappe et al. 2007; Cappe, Rouiller, et al. 2009), overlapping territories of projections from different sensory cortices to the medial pulvinar (Cappe, Rouiller, et al. 2009) and projections from the medial pulvinar to the premotor cortex (Cappe, Morel, et al. 2009). This suggested that the medial pulvinar is a place of sensorimotor convergence. Our study indeed demonstrates that the medial pulvinar contains multisensory audiovisual neurons with various multisensory integration characteristics.
It has been hypothesized that the pulvinar plays a major role as mediator or modulator of signal transfer between corticalareas (Saalmann and Kastner 2011; Benarroch 2015). Indeed, the replication principle postulates that all directly linked cortical areas would also be indirectly connected via the pulvinar (Sherman 2016, 2017). These cortico-pulvino-cortical pathways would improve the signal-to-noise ratio of information transmitted by modulating the synchrony between these areas (Fries 2015). These cortico-thalamo-cortical pathways would be faster than cortico-cortical projections. The multimodal convergence and interactions we report at the single-unit level suggest that the replication principle could also apply to multisensory–motor interactions between distant cortical regions (Sherman and Guillery 2002; Cappe, Morel, et al. 2009).
Considering abovementioned 3 points, the pulvinar would be involved in sensory distractor filtering (Fischer and Whitney 2012) and in visual selection and attention (Saalmann and Kastner 2009), allowing the detection of important stimuli (behaviorally relevant or requiring fast motor responses for example). In addition, the pulvinar may be an important modulator of behavioral flexibility, contributing to the selection of sensory inputs (Froesel et al. 2021).
Current hypotheses state that the pulvinar has evolved by selection pressure to allow fast and accurate detection of predators such as snakes or emotional faces (Isbell 2006; Maior et al. 2010; Van Le et al., 2013, 2014, 2016; Arend et al. 2015). Moreover, threat recognition is altered in patients with lesions restricted to the medial part of the pulvinar (Ward et al. 2007). Unlike in the studies by Van Le et al. (2013, 2014), we found that the strength of the response to snake and macaque stimuli were similar. This discrepancy may be explained by the fact that our monkeys were keeping fixation while passively viewing or listening to the stimuli. We also repeatedly used a single potentially threatening stimulus to which the monkey had probably quickly become accustomed. On the other hand, and in agreement with these previous studies (Van Le et al. 2013, 2014), we found that visual latencies were faster for snake compared with macaque faces. Our multisensory indexes revealed mostly suppressive and subadditive interactions. Although beyond the scope of our data, we hypothesize that relevant and unexpected multimodal stimuli would lead to response enhancement and possibly supra-additive integration in order to trigger faster adaptive responses. One perspective to explore this function would be to record neuronal activity in response to stimuli of different saliency and of varying relevance for the monkey while monitoring its behavioral response.
Acknowledgments
We would like to thank Amirouche Sadoun and the technical team for animal care.
Funding
This work was supported by the French National Research Agency (ANR) (ANR-12PDOC-0008 and ANR-16-CE37–0009-01).
Conflict of interest statement: None declared.
