Interactive multimodal integral field spectroscopy

ABSTRACT

Using sonification on scientific data analysis provides additional dimensions to visualization, potentially increasing researchers’ analytical capabilities and fostering inclusion and accessibility. This research explores the potential of multimodal integral field spectroscopy applied to galaxy analysis through the development and evaluation of a tool that complements the visualization of data cubes with sound. The proposed application, ViewCube, provides interactive visualizations and sonifications of spectral information across a 2D field-of-view, and its architecture is designed to incorporate future sonification approaches. The first sonification implementation described in this article uses a deep learning module to generate binaural unsupervised auditory representations. The work includes a qualitative and quantitative user study based on an online questionnaire, aimed at both specialized and non-specialized participants, focusing on the case study of data cubes of galaxies from the Calar Alto Integral Field Spectroscopy Area survey. Out of 67 participants who completed the questionnaire, 42 had the opportunity to test the application in person prior to filling out the online survey. 81 per cent of these 42 participants expressed the good interactive response of the tool, 79.1 per cent of the complete sample found the application ‘Useful’, and 58.2 per cent rated its aesthetics as ‘Good’. The quantitative results suggest that all participants were able to retrieve information from the sonifications, pointing to previous experience in the analysis of sound events as more helpful than previous knowledge of the data for the proposed tasks, and highlighting the importance of training and attention to detail for the understanding of complex auditory information.

1 INTRODUCTION

The combination of visual and auditory displays can offer a better understanding of a phenomenon (Enge et al. 2024), making the use of sound for the representation of physical quantities an established area of research (Dubus & Bresin 2013). It can allow holistic interpretations of the data facilitating the discovery of previously unseen relationships (Cooke et al. 2017), as well as single datum analytic tasks, both involving point estimation and comparison, trend identification, and data structure analysis (Walker & Nees 2011).

Sonification has proven to be a very useful tool to assist in the analysis of hyperspectral data, generating sonic time-series related to the spatial and spectral content near user-selected mouse positions (Bernhardt, Cowell & Oxford 2007). It can also improve the perception of density in visualization of complex data in parallel coordinates and scatter plots (Rönnberg & Jimmy 2016), and ensure simple access to information for blind and non-blind people, enhancing the accessibility of astronomical data as well as the work of the scientist accepting complementary exploration methodologies (Casado, Diaz-Merced & García 2024).

Furthermore, the enhancement of visual information with sonification allows sighted and blind or low vision (BLV) communities to have astronomy experiences at similar levels (Arcand et al. 2024). Including blind users from the beginning of the design process, Casado & García (2024) proposed the case study of the galaxies from the Sloan Digital Sky Survey (SDSS) to show the possibilities of sonoUno. This multimodal application for displaying sound and images from any data set allowed for the discovery of the variable star UCAC4 459–09273 by blind students using sonification. As a blind researcher, Foran, Cooke & Hannam (2022) reported the use of StarSound on 1D high-redshift galaxy work for the verification and initial analysis of the rest-frame ultraviolet (UV) spectra of distant galaxies, and developed the touch-based sonification tool VoxMagellan to analyse 2D images and multidimensional data sets.

In the photometric and spectroscopic analysis field, Trayford et al. (2023) proposed the audification of spectral data cubes (direct conversion of data into audible frequencies), to demonstrate that physical information can be extracted directly from sound with STRASUSS (Trayford & Harrison 2023). Using Star Sounder, Huppenkothen et al. (2023) provided an interactive sonification of the Hertzsprung–Russell diagram based on the cross-match between the Kepler Stellar Table and Gaia Data Release (DR2). The introduction of a sonic perspective has also been valued on spectroscopic analysis of Quasars by Hansen, Burchett & Forbes (2020), finding that sonification can enable more rapid discovery and identification of intergalactic/circumgalactic medium (IGM/CGM) system candidates than visually scanning through spectra.

Including multimodal interactivity, Starks (2018) explored the data cubes of the Antennae Galaxies radio-image from the Atacama Large Milimeter/submilimeter Array (ALMA), using Galaxy player within the Soniverse project. Additionally, spatialized sonifications have been used on immersive representations of Antarctic astronomy data (West et al. 2018), and highlighted by Quinton, McGregor & Benyon (2020, 2021) as an effective parameter mapping strategy with the potential to detect sudden changes between multiple sources within the field of exosolar planetary search.

Framed in this context of inclusive and immersive scientific representations, this article presents ViewCube, a multimodal interactive binaural tool for the analysis of data cubes with headphones. The application includes an unsupervised sonification approach, based on autoencoders. Furthermore, the work explores the potential of multimodal integral field spectroscopy (IFS) using the case study of the Calar Alto Legacy Integral Field Spectroscopy Area (CALIFA) survey (Sánchez et al. 2012a; Sánchez et al. 2016).

Providing quantitative and qualitative feedback from specialized and non-specialized users in Astronomy and Music, this paper aims at demonstrating the usefulness of sound in multimodal displays on IFS analysis to: (a) estimate the position of the user-selected spaxel within the galaxy, identifying whether it is located to the left/right or front/rear in the virtual soundscape; (b) estimate the spaxel’s relative distance from the centre of the galaxy, indicating whether it is near or far from this reference point; and (c) identify the type/age of a spectrum, determining whether the spaxel corresponds to a star-forming region (spectrum with multiple and relatively strong emission lines), to an intermediate age galaxy, or to a retired galaxy. The work is expected to make IFS more accessible for BLV researchers while also enhancing the overall capabilities for datacube analysis.

2 MULTIMODAL IFS

This section describes the design strategy and implementation of ViewCube. The application combines graphical and auditory displays for the interactive multimodal analysis of data cubes. Using the case study of the CALIFA survey data cubes, the work aims at enhancing 3D spectroscopy representation with immersive sonification.

2.1 Case study

The CALIFA (Sánchez et al. 2012b, 2016; Walcher et al. 2014) is a public legacy survey of over 600 galaxies with an r-band isophotal major axis between 45 and 79.2 arcsec and a redshift 0.005 |$\lt $| z |$\lt $| 0.03, selected from the Sloan Digital Sky Survey (SDSS) DR7 photometric catalogue. Aimed at helping in the study of galaxy evolution in the Local Universe through cosmic time, it uses integral field spectroscopy (IFS; Allington-Smith 2006) to provide a wide-field IFU survey of galaxies that includes all morphological types, covering masses between 10^8.5 and 10^11.5 M_|$\odot$| (Sánchez et al. 2016).

The observations were obtained with the integral-field spectrograph PMAS/PPak mounted on the 3.5 m telescope at the Calar Alto observatory. The wavelength range between 3700 and 7500 Å is sampled using two different spectral setups, a low resolution V500 mode (3745–7500 Å) with a spectral resolution of 6.0 Å (full width at half maximum, FWHM), and a medium-resolution V1200 mode (3650–4840 Å) with a spectral resolution of 2.3 Å (FWHM; García-Benito et al. 2015). CALIFA’s third Data Release (DR3; Sánchez et al. 2016) provides to the public 646 objects in the V500 setup, 484 in the V1200, and the combination of the cubes from both setups (COMBO).

The morphology references provided for each galaxy used in this work are extracted from Walcher et al. (2014).

2.2 Standalone application

ViewCube  ¹ is a lightweight, standalone application written entirely in Python, designed for the efficient browsing of data cubes. Originally developed for the quick assessment of the quality and physical characteristics of data cubes from the CALIFA survey, and for the rapid exploration of high-level data products generated by the PyCASSO pipeline (de Amorim et al. 2017), ViewCube’s functionality has since expanded. The application now supports data cubes from any provenance, thanks to its general and flexible FITS reader. This reader is agnostic to the source of the data cubes, enabling it to handle a wide variety of data cubes from different instruments (e.g. MUSE) and surveys (e.g. MaNGA, CALIFA) across various wavelength ranges (optical, radio).

Despite its name, ViewCube is also capable of rendering Raw Stacked Spectra (RSS) formats, provided that a file mapping the positions of the fibres is available. Visualizations within ViewCube are rendered using the Matplotlib module (Hunter 2007). The primary objective of ViewCube is to facilitate a fast and effective inspection of data cubes, either for a quick quality assessment or for a focused examination of their characteristics.

As shown in Fig. 1, the user interface features two main windows: an image window, which presents a 2D map of the datacube convolved through a chosen passband, and a spectral window, which displays the spectrum corresponding to the location of the mouse pointer. Users can select different spaxels or fibres for comparison, generate an integrated spectrum, and save both individual and integrated spectra. Additionally, users can modify the filter used to convolve the data cube to produce the image in the image window, and adjust the central wavelength of the filter by dragging and dropping the filter passband shown in the spectral window.

In keeping with its initial exploratory purpose for quality assessment, ViewCube allows for the interactive comparison of spectra from two data cubes at the same spaxel, provided the data cubes share the same dimensions. To enhance the exploration of individual spectra and perform more advanced operations, ViewCube offers the possibility of integrating with other packages such as PySpeckit (Ginsburg et al. 2022) and PyRAF (Science Software Branch at STScI 2012). Future versions of ViewCube are planned to include a faster rendering visualization engine, as well as additional menu options for improved functionality.

2.3 Sonification module

This section describes the sound module implemented within ViewCube to allow the sonification of the spectra associated with each spatial element of a datacube. The module, named SoniCube, provides an open, comprehensive, and general-purpose multimodal tool for IFS analysis, supporting the development of future sonification techniques focused on specific potential features within data cubes. The aim of the SoniCube interface is to offer a diverse ‘palette of sonifications’, akin to the range of colour palettes available for visual 2D maps in ViewCube. This variety of sonification options allows users to extract or enhance different data characteristics by selecting specific sonification methods, much like how a colour palette reveals visual details. In this paper, we present the first sonification method implemented within SoniCube’s palette, providing an autonomous fast representation of the spectra. This first sonification ‘palette’ implementation is designed to replicate the purpose of the visual ViewCube counterpart, providing a quick qualitative overview of the data cube.

Also implemented in Python, SoniCube controls a sound synthesizer developed in CSound via Open Sound Control (OSC) (Wright 2002), using the python-osc native module and the ctcsound interface (Ctcsound 2022). The module provides a real-time interactive sonification of the spectrum associated to each user-selected spaxel. In this first sonification implementation, each spectrum is converted into sound using the deep learning approach described in Section 2.4. This process provides an unsupervised unique auditory footprint conveying the information of each spectrum in one single sound event. Each sound event is generated with an additive synthesizer using six independent oscillators, fed by a 6D latent vector which is generated by an autoencoder (Baldi 2012).

The module generates ‘on the fly’ a 6D latent vector from each user-selected spectrum. The components of this vector are interpreted as fundamental frequencies for the six oscillators that synthesize the sound. The six components are multiplied by a factor of 10 000 for scaling the latent values to audible frequencies, generating comprehensible accurate sonifications. For a formalized description of the synthesizer see Appendix  A.

Additionally, the module calculates the azimuth and radial distance from user-selected spaxels to the reference spaxel, which corresponds to the centre of the galaxy on each data cube. The azimuth is used to locate the auditory footprint of the spectrum within the binaural soundscape (Møller 1992) generated for each data cube, providing an immersive representation of its spectra with the listener located at the centre of the galaxy. For more information about binaural encoding see Appendix  B.

The distance is used to feed the direct-to-reverberant energy ratio of a reverberation emulator (Gardner 1998), which provides the cognitive sensation associated with the sound field that can be found in large indoor environments. This effect is used to generate a virtual auditory cue for distance perception based on direct-to-reverberant energy ratio (Lu & Cooke 2010), related to the proximity from the user-selected spaxel to the reference spaxel in the centre of the galaxy.

The amplitude of each sonification is calculated from the absolute fluxes of the represented spectrum. To solve the wide and variable dynamic range of flux density commonly found in real sky observations, SoniCube includes two operating modes for representing relative sound amplitudes. In the flux sensitive mode (default), the amplitude of each sonification is calculated from the logarithmic median of the absolute flux, normalized using feature scaling. This mode preserves the apparent relation of fluxes within the data cube. On the other hand, if the sensitive mode is deactivated, all spectra in the data cube are represented with the same amplitude, allowing the appreciation of regions with relatively low absolute fluxes. An additional two-stage broad-band dynamic range limiter/compressor (Kates 2005) is implemented in both modes to keep extremely salient values controlled, preventing from hearing damage when analysing unexplored data cubes.

To compensate for the non-linear response of human hearing in frequency and amplitude domains, an A-weighted curve is applied to the array of amplitudes using the librosa module (McFee et al. 2015). This function calculates the normalized amplification factors that provide an equal loudness response on each footprint, presenting good results for the implementation of this first sonification module evaluated with the CALIFA survey. Nevertheless, alternative loudness contours are being explored for critical future implementations (Charbonneau et al. 2012).

Finally, using the Open Sound Control (OSC) protocol, the module sends to CSound all the variables needed to synthesize the auditory footprint in a binaural soundscape. The block diagram of Fig. 2 summarizes the sound generation process. Notice the use of azimuth and distance of the selected spaxel in the binaural and reverberation blocks, as well as the normalized median flux, and the frequencies with A-weighted factors of the corresponding spectrum in the additive synthesizer.

Addressing some additional aesthetic aspects of the sonification, background echo, and flanger effects were added to the workflow, facilitating the binaural localization of the footprint and smoothing fast transitions between spectra.²

2.4 Autoencoding CALIFA

As mentioned in the previous section, this first sonification implementation of SoniCube uses an autoencoder architecture to provide accurate sonifications of the spectral information of a data cube. Based on the gradient descent algorithm, these networks allow the reduction of data dimensionality better than other approaches such as principal components analysis (PCA; Hinton & Salakhutdinov 2006). Autoencoders are neural network models with the potential to learn an approximation to the identity function, providing an output that is similar to their input (Ng et al. 2011). By reducing the number of hidden units of the intermediate layer of the network, a model can learn relevant structures of the data, which can also be reconstructed from this intermediate lower dimensional representation, named latent space (Goodfellow 2016).

The dimension of the latent space depends on the data set and the architecture used. Aimed at obtaining stellar parameters using convolutional neural networks, Mas-Buitrago et al. (2024) proposed a 32D latent space autoencoder, displayed in a 8×4 matrix, for the reduction of CARMENES spectra, and ACES synthetic spectra modelled with Mas-Buitrago, González-Marcos, Solano, Passegger, Cortés-Contreras, Ordieres-Meré, Bello-García, Caballero, Schweitzer, Tabernero et al. (1990). On the other hand, the reconstruction from 4D latent vectors was enough for Xiang, Gu & Cao (2022) to analyse the stellar magnetic activity using variational autoencoders on the Large Sky Multi-Object Fiber Spectroscopic Telescope (LAMOST) K2 spectra. Demonstrating the potential of variational autoencoders, Portillo et al. (2020) summarized galaxy spectral information with only six latent variables on the Sloan Digital Sky Survey (SDSS). This dimension also worked effectively for the Calcium II Triplet library (CaT) reduced with sparse autoencoders (García Riber & Serradilla 2024), which also agrees with our preliminary tests on the CALIFA survey galaxies.

Intensive testing was done with different configurations of both architectures using the COMBO (V500+V1200) data cubes of the DR3 CALIFA survey. Fig. 3 provides a comparative example for the data cube of the spiral (Scd) galaxy NGC 5406 with a six-layer 6D autoencoder, and a four-layer 6D VAE, both implemented using TensorFlow (Abadi et al. 2016).

On each encoded data cube, we calculated the coefficient of determination (R²) between the original and the reconstructed sets of spectra, providing a measure of the accuracy of the reduction. As for the duration of their training processes, the VAE required 5.5 times more computation time per epoch than the sparse autoencoder to provide lower results. Respectively, R² = 0.96 (VAE) versusR² = 0.98 (sparse) for the complete data cube, with 4.92 per cent (VAE) versus 39.12 per cent (sparse) of the spectra with R²|$\gt $| 0.9, and R² = 0.96 versus R² = 0.99 for the represented spaxel (34,34).

Based on these tests, a six-layer 6D sparse autoencoder module was included in SoniCube to represent ‘in real time’ the spectral information of the data cubes with low-dimensional vectors. This architecture allowed the reduction of each input spectrum X_i|$\in$||$\mathbb {R}$|¹⁹⁰¹ (each data cube contains around 5540 spectra with 1901 flux values per spectrum), to a 6D representation Z_i|$\in$||$\mathbb {R}$|⁶, and the reconstruction of X_i from the latent vector Z_i, |$\hat{X}$|_i|$\in$||$\mathbb {R}$|¹⁹⁰¹.

The model was trained on each data cube independently with around 5540 spectra. Fig. 4 provides two examples of the original and reconstructed spectra from the data cubes of NGC 5784 (Sbc) and NGC 5682 (E4).³

Fig. 5 shows three examples of the learning curves obtained during the training process with the spectra from the data cubes of NGC7047 (Sab), UGC10331 (E1), and UGC03960 (E5). These galaxies illustrate the performance of the autoencoder, respectively corresponding to the best, the medium, and the worst encoding results. The coefficients obtained ranged from 0.998 to 0.882 along the complete data set, with 49 per cent of the data cubes presenting an R² higher than 0.96, and 4.78 per cent presenting an R² under 0.92.

3 EVALUATION

To evaluate the potential utility of the previously described approach for the auditory analysis of galaxy data cubes, specifically using CALIFA data cubes as representative data, we conducted an anonymous online survey provided here for reference.⁴ The questionnaire was complemented by training videos and, to a lesser extent, in-person interactive demonstrations, targeting both specialized and non-specialized participants. All participants received the same online form, where they indicated whether or not they had experienced the application in person.

3.1 Survey design

The online survey was administered to volunteer participants from 2024 April 15 to July 31. The questionnaire featured five training videos, which could be replayed as needed, including one providing a general overview of the application and one for each specific section. The survey comprised four sections with video-supported questions that analysed various aspects of the proposal. Additionally, 12 questions were included to gather demographic information, participants’ self-reported levels of expertise in Astronomy and Music, and three qualitative assessments concerning the application’s interactivity, usefulness, and the aesthetics of the sounds employed in the sonification.

All participants were advised to use headphones and to check their correct placement in left and right ears. There was no time limit to complete the survey. The following describes the sections and questions included in the survey. Each question included several sonifications with no graphics, generated from the spectra of the galaxies NGC 5784 (Sbc), NGC 5732 (Sbc), NGC 5682 (E4), NGC 6060 (S0a), NGC 7562 (Sbc), NGC 7671 (S0), NGC7800 (Ir), NGC 2638 (Sb), and UGC 00148 (Sb). The participants could compare the questions with the examples presented in the training videos as many times as needed.

Section 1. Sound Location. This first section consisted of four questions designed to analyse the possibilities of the application to estimate sound location. Within the virtual binaural soundscape provided, the listener is virtually placed in the centre of the galaxy, looking at the upper position. The training videos of this section provided examples around the spiral galaxy (Sa) NGC 7549.

Section 2. Distance to the centre of the galaxy. This section also included four questions aimed at studying the possibilities of the application to provide auditory information about the distance from the moving cursor to the centre of the galaxy within the virtual spectral soundscape. The training videos of this section provided examples from the spiral galaxy (Sbc) NGC 5732.

Section 3. Age/Galaxy type. This section explored how multimodal representation could aid in differentiating between various Age/Galaxy types. The training videos presented three examples, illustrated in Fig. 6. These examples include: the spectrum (37,34) from a star-forming region of the spiral galaxy (S0) NGC 3395; the spectrum (36,34) near the centre of the intermediate-age spiral galaxy (Sd) NGC 2347; and the spectrum (35,32) from a region near the centre of the retired spiral galaxy (Sb) NGC 6125.

Section 4. Combined questions. Finally, two multiple choice questions analysed the potential of the application to allow the identification of the position of the represented spectrum (left/right), its distance to the centre (close/far), and if it corresponded to a star-forming region or to a retired galaxy.

Qualitative question 1. If you have tried the application in person, please rate the multimodal experience. If not (you only saw the training videos of this questionnaire), please skip this question. Options: Very bad, Bad, Acceptable, Good, or Very good.

Qualitative question 2. Rate the potential usefulness of the multimodal display for the exploration of the CALIFA Survey. Options: Useless, Doubtfully useful, Useful, or Very useful.

Qualitative question 3. Rate the aesthetics of the sonifications. Options: Intolerable, Bad, Acceptable, Good, or Nice Sounding.

3.2 General results

The survey was completed by 67 participants,⁵ including 31 professional astronomers, two of them identified as blind or low vision (BLV), and 36 non-astronomers. Their ages ranged from less than 21 (1) to more than 60 yr old (10), with most of the participants ranging between 21 and 30 (18), and between 41 and 50 (21). They were mainly from Spain (50 participants) but also from Mexico, USA, Japan, Germany, China, Malta, Australia, and UK. Participants were asked about their music preferences and native language to explore whether language influences the ability to recognize sound features. Although the study included speakers of eleven different languages, the sample size was too small and diverse to draw any definitive conclusions. The same limitation applied to the analysis of music preferences.

As shown in the first graph of Fig. 7 the mean global success rate obtained by 67 participants was 0.516, with a Jeffreys confidence interval of (0.498, 0.534), standard deviation 0.169, and 68.3 per cent of uncertainty. The subgroup formed by professional astronomers (31 participants) obtained a mean success rate of 0.554 with a Jeffreys confidence interval of (0.528, 0.579), standard deviation of 0.188, and the subgroup of non-astronomers (randomly downsampled from 36 to 31 participants to allow direct comparison) obtained a mean success rate of 0.481 with a Jeffreys confidence interval of (0.455, 0.507), standard deviation 0.171, and 68.3 per cent of uncertainty.

Although the sample was too small to establish statistical significance, these indicative results appear to confirm that all participants were able to understand the information from the sonifications thanks to the training videos, even without having previous experience in Astronomy. Agreeing with the results obtained using auditory graphs by Smith & Walker (2005), the training and context provided in the survey enhanced the performance of the participants. This is particularly notable in the analysis of the combined questions, in which non-astronomers performed 1.33 times better than professional astronomers, as can be noticed in the left-down graph of Fig. 7. This result could be related to the decision of allowing participants to repeat the training videos as many times as they wanted, that could benefit low-intermediate prior knowledge participants (van Riesen et al. 2022).

As for the performance of BLV professional astronomers (only two participants), their results on simple questions were similar, although slightly higher (8 per cent), than those obtained by an equivalent random sampled group of non-BLV professional astronomers. This suggests that the application could help in bringing IFS analysis closer to BLV astronomers. In the combined questions, none of the two BLV astronomers succeeded in the Type/Age section. In the following, their results are included in the group of professional astronomers.

3.3 Subgroup quantitative analysis

To provide further analysis of the recorded feedback, the participants were divided into four subgroups according to their self-declared level of expertise in Astronomy and Music. The Astronomers musicians subgroup included professional astronomers also identified as professional or amateur musicians. A second group of astronomers no musicians was used to analyse the influence of sound analysis experience on potential experts in the proposed analysis tasks. The Musicians no astronomers group was formed by participants identified as professional and amateur musicians. Finally, the non-experienced group included the rest of the participants declaring no experience in any of the fields. Table 1 summarizes the results obtained on simple and combined question sections.

As shown in the up-right graph of Fig. 7, Astronomers no musicians were the best performers in the simple question sections, obtaining an average success rate of 0.577, with a Jeffreys confidence interval of (0.573, 0.616), standard deviation 0.187, and 68.3 per cent of uncertainty. It is worth mentioning that the non-experienced group performed as well as the Astronomer musicians, achieving an average success rate of approximately 0.55 with a standard deviation of 0.186, under the same conditions of uncertainty. This performance was 1.14 times better than that of the Musicians, which may suggest that the additional focus required to learn about unfamiliar fields helped the non-experienced participants with the proposed tasks.

The number of correct responses by group of expertise is provided in Fig. 8. As for the combined questions, the success rates obtained were notably lower than those obtained in simple questions for all groups, with the exception of Musicians no astronomers. This suggests that the experience in the analysis of sound events was more helpful than previous knowledge of the data for the proposed task.

Further analysis of the combined responses revealed how participants failed to answer all three aspects of the combined questions simultaneously, but performed well per section. From the complete sample, 74.63 per cent of the participants located the sonification correctly, 46.27 per cent marked the correct distance to the centre of the galaxy, and 36.57 per cent successfully interpreted the age of the galaxy. The respective averaged success rates from the simple questions per section were 53.2 per cent for the location questions, 63.30 per cent for distance analysis questions, and 40.7 per cent for the type of galaxy questions.

As illustrated by Fig. 9, the Type/Age questions had the lowest relative success rates across all groups, possibly due to the level of abstraction involved in these tasks. This result suggests that interpreting a galaxy’s type or age through sound in this specific sonification implementation requires more training than the other tasks, which were more intuitive and aligned with the participants’ prior experience. It is worth mentioning the exception of the astronomers no musicians, who obtained the worst results in the interpretation of the distance to the centre of the galaxy. This fact has a positive correlation with the lack of accuracy in distance perception, when compared to horizontal localization, as discussed by Middlebrooks (2015).

Analysing the results of the combined questions by age (down-right graph of Fig. 7), the group ranging from 51 to 60 yr old (6 participants) performed 1.56 times better than the 41–50 subgroup (21 participants), and 1.73 times better than the 31–40 subgroup (9 participants), suggesting that experience and attention to detail played an important role in the understanding of complex auditory information. Notice that the size of the samples do not provide statistical significance.

Fig. 10 provides an additional comparison between the results of the 42 participants that could try the application in person and the 25 participants that only used the training videos. Live testers presented a global success rate of 0.51 versus the 0.43 of only video trained participants. Nevertheless, the best results were obtained by astronomers non-musicians subgroup (4 participants) using only the training videos with no repetition restrictions (0.65), followed by non-experienced subgroup (13 participants) testing the application live (0.60). These results suggest that both training methods were useful for the proposed tasks.

3.4 Qualitative feedback

To evaluate the interactivity, usefulness, and aesthetics of the proposal, the three qualitative questions described in Section 3.1 were included in the survey. Of the 67 participants that completed the survey, 42 tested the application in person. As shown in Fig. 11, 81 per cent of these participants expressed the good interactive response of the application, 79.1 per cent of the complete sample of participants (67) found the application ‘Useful’ or ‘Very useful’, 19.4 per cent ‘Doubtfully useful’, and one participant (1.49 per cent) considered it ‘Usefulness’. Regarding the aesthetics, 58.2 per cent rated it as ‘Good’ or ‘Nice sounding’, 34.33 per cent as ‘Acceptable’, and 5.97 per cent as ‘Bad sounding’. The subgroup results are also available in Table 2. Additionally, seven participants explicitly expressed that they found it difficult to differentiate the sounds, and eight participants explicitly expressed their enthusiasm about the project.

4 CONCLUSIONS

The design and evaluation of fast and efficient multimodal interactive tools for the exploration of IFS data cubes can help in the analysis of current massive spectroscopic surveys, complementing the possibilities of visual representations, and fostering inclusion and accessibility.

This article provides a summary of the motivations and design strategies used in the development of the interactive multimodal binaural application ViewCube, and a user study with specialized and non-specialized participants analysing selected galaxies from the CALIFA survey. The complete work was aimed at exploring the potential of multimodal IFS for the analysis of data cubes, with the motivation of making them more accessible for blind and low vision (BLV) researchers, and more immersive for complementary exploration through sound.

The tool allows the exploration of a wide variety of data cubes from different instruments and surveys across various wavelength ranges and includes a deep learning sonification approach to provide accurate comprehensible sonifications of the spectral information of the data cubes. The approach serves as an initial, general-purpose sonification tool designed to provide an overview of spectral properties, particularly in terms of stellar age and emission lines. In this context, the autoencoder effectively captures the general characteristics of both the strong emission lines (if present) and the continuum.

Regarding the qualitative feedback obtained from the 42 participants (including 21 professional astronomers) who evaluated the application in person, it can be concluded that the interactivity of the application is good or very good (80.95 per cent), with 79.1 per cent of the complete sample of participants (including 31 professional astronomers) finding it ‘Useful’ or ‘Very useful’, and 58.21 per cent declaring it as ‘Good’ or ‘Nice sounding’.

The quantitative evaluation of the application was conducted through an online survey featuring five training videos. The questionnaire was structured in four blocks aimed at analysing the potential of the application for the estimation by sound of: (1) the position of a user-selected spaxel in the virtual soundscape generated by a data cube (left, right, front, or rear); (2) the distance of the user-selected spaxel to the centre of the represented galaxy (close to the centre, intermediate distance, or far from the centre); (3) the type/age of the spectrum of the user-selected spaxel (star-forming region, intermediate age, or retired galaxy); and (4) all three characteristics combined.

Although the sample was too small to provide statistical significance, the results suggest that all participants (experienced and non-experienced) were able to retrieve information from the sonifications, presenting an average success rate of 0.516, with professional astronomers performing 1.15 times better than non-astronomers.

The sample included two professional astronomers self-declared BLV. This group presented an average success rate of 0.541, 8 per cent higher than a random sampled subgroup formed by two professional astronomers, and 5 per cent higher than the mean of the complete sample, suggesting that the application can improve the access of BLV astronomers to IFS analysis. In the following analysis, BLV participants were included in professional astronomer subgroups.

The subgroup analysis revealed how the astronomers no musicians obtained the higher success rates, followed by astronomers musicians and non-experienced participants, who performed 1.14 times better than musicians. These results suggest that the additional attention required for learning aspects of unfamiliar fields could have helped the non-experienced participants with the proposed tasks.

Concerning the combined questions, the non-experienced group performed even slightly better than the astronomers (1.33 times). In these questions, musicians were the top performers, suggesting that, for the proposed task, experience in analysing sound events was more beneficial than prior knowledge of the data. Analysing the results by age, participants between 51 and 60 yr old were the top performers, suggesting that experience and attention to detail played a significant role in understanding complex auditory information.

In conclusion, although further research with additional sonification approaches and alternative spectroscopic surveys is planned, the promising trends outlined in this article suggest that the use of multimodal IFS displays can enhance the datacube analysis process and make 3D spectroscopy more accessible to BLV researchers. ViewCube has demonstrated its ability to convey information about CALIFA’s galaxies, which was understood by both experienced and non-experienced users.

ACKNOWLEDGEMENTS

We extend our gratitude to the anonymous referee for their valuable and insightful feedback, which has contributed to enhancing the quality and clarity of our manuscript.

We want to thank all the anonymous volunteers who made this research possible by testing the application and completing the survey.

This study uses data provided by the Calar Alto Legacy Integral Field Area (CALIFA) survey (https://califa.caha.es/). Based on observations collected at the Centro Astronómico Hispano Alemán (CAHA) at Calar Alto, operated jointly by the Max-Planck-Institut für Astronomie and the Instituto de Astrofísica de Andalucía (CSIC).

RGB acknowledges financial support from the Severo Ochoa grant CEX2021-001131-S funded by MCIN/AEI/10.13039/501100011033 and PID2022-141755NB-I00.

DATA AVAILABILITY

The data cubes from the CALIFA Survey are available at https://califa.caha.es/FTP-PUB/reduced/COMB/reduced_v2.2/.

The encoded cubes generated for the sonification can be found at https://zenodo.org/records/10570065.

The feedback recorded from the survey and its analysis notebooks are available at https://github.com/rgbIAA/ViewCube-Evaluation/tree/main/2024.

Footnotes

REFERENCES

APPENDIX A: SYNTHESIZER DESCRIPTION

The equation of the six-oscillator additive synthesizer implemented in SoniCube can be expressed for each sonification as:

where A_i is the A-weighting coefficient for each frequency, F is the median of absolute flux of the represented galaxy spectrum,r is the ratio or slope of the dynamic range limiter/compressor used to control salient flux values, f_i are the fundamental frequencies obtained through the autoencoder dimension reduction process (six dimensions), and |$\phi$|_i, the relative phases of the oscillators (in our case, all set to zero).

Attending the loudness of each sonification, A-weighting coefficients are calculated for each one of the six frequencies or formants as (IECI 2013):

with R_A(f_i) calculated from the expression:

Fig. A1 shows the transfer level curve of the two-stage dynamic range limiter/compressor included in SoniCube for preventing ear damage. The slope of each stage is represented by r in equation (A1).

A stereo reverberation processor based on eight delay lines (Kahrs & Brandenburg 1998) is added to the signal flow to provide information about the distance from the selected spaxel to the centre of the galaxy. The final signal sent to the binaural encoder can be expressed as:

where S(t) is the output of the additive synthesizer, d is the distance of the user-selected spaxel to the reference point (centre of the galaxy), g is the gain of the feedback loop (in our case set to 0.9 to provide a long reverberation effect), and |$\tau$| is the fixed delay time applied to each line.

APPENDIX B: BINAURAL ENCODING

Human hearing can locate sound sources in a 3D space through the analysis of interaural level differences (ILD), and interaural time delays (ITD). The alterations produced in a sound when travelling from the source to the listener can be characterized using head related transfer functions (HRTF) (Brown & Duda 1997). Binaural systems are based on the convolution of the HRTF of both ears with the sound sources, allowing the spatialization of static and dynamic locations (Lazzarini & Carty 2008).

The relationship between the sound source and the signal reaching the listener ears can be expressed in terms of the azimuth (AZ) and elevation (EL) angles, the distance to the source (d), and the angular frequency (w), as shown in the following equation:

where Y_L,R are the audio spectra of acoustic signals at listener’s ears, H_L,R is the HRTF and X_L,R are the audio spectra of the sound source.

Sonicube uses the binaural hrtfmove Csound opcode (Carty 2008) to represent the spectral information of the data cubes in an immersive 2D soundscape correlated to ViewCube’s UI. The azimuth is calculated from the (x,y) coordinates and the elevation is set to zero to discard the third dimension. Distance is not used to keep the levels of the sonification flux-dependent, reducing the general expression to: