Decoding roughness perception in distributed haptic devices

Abstract

The ability to render realistic texture perception using haptic devices has been consistently challenging. A key component of texture perception is roughness. When we touch surfaces, mechanoreceptors present under the skin are activated and the information is processed by the nervous system, enabling perception of roughness/smoothness. Several distributed haptic devices capable of producing localized skin stretch have been developed with the aim of rendering realistic roughness perception; however, current state-of-the-art devices rely on device fabrication and psychophysical experimentation to determine whether a device configuration will perform as desired. Predictive models can elucidate physical mechanisms, providing insight and a more effective design iteration process. Since existing models (1, 2) are derived from responses to normal stimuli only, they cannot predict the performance of laterally actuated devices which rely on frictional shear forces to produce localized skin stretch. They are also unable to predict the augmentation of roughness perception when the actuators are spatially dispersed across the contact patch or actuated with a relative phase difference (3). In this study, we have developed a model that can predict the perceived roughness for arbitrary external stimuli and validated it against psychophysical experimental results from different haptic devices reported in the literature. The model elucidates two key mechanisms: (i) the variation in the change of strain across the contact patch can predict roughness perception with strong correlation and (ii) the inclusion of lateral shear forces is essential to correctly predict roughness perception. Using the model can accelerate device optimization by obviating the reliance on trial-and-error approaches.

Introduction

Tactile textural information is multidimensional in nature (4–6), and one of the key perceptual dimensions of texture is roughness. During the tactile exploration of surfaces, the SA-1 (Merkel cells), RA-1 (Meissner's corpuscles), SA-2 (Ruffini endings), and RA-2 [Pacinian corpuscles (PC)] mechanoreceptors present under the glabrous skin of our fingers are activated. The physiological properties of each mechanoreceptor allow them to provide information about various features of the tactile world (7, 8), and they each produce unique spike patterns in response to stimuli (Figure 1A). The information from these mechanoreceptors is then summarized and processed by the sensory cortex in the brain to give us a measure of how rough or smooth a surface feels (Figure 1B). The combined effect of the spatial and temporal variations across SA-1, RA-1, and RA-2 mechanoreceptors has been shown to correlate well to the roughness perception produced by a wide variety of textures (9, 10).

Although surface haptic devices have been successful in rendering tactile cues for actions such as turning a knob or moving a slider, rendering realistic textural information using tactile cues has been more challenging (11, 12). Even when the bulk vibrations and lateral forces experienced during the tactile exploration of textured surfaces were recorded and replayed on the whole finger with a very precise matching, the original feeling of roughness could not be reproduced (12). This led to the hypothesis that localized variations in skin strain within the contact patch might be critical in conveying textural information.

In this study, we investigate three state-of-the-art distributed surface haptic devices capable of producing localized skin stretch. Device 1 (3) is a distributed pin array driven by voice coil actuators and capable of moving each pin by a specified distance in the normal direction. Device 2 (13) is a magnetically actuated distributed pin array which can move each pin by a specified distance in the lateral direction. Device 3 (14) is a wearable device consisting of an array of electroadhesive pucks. When one puck is engaged, the nonengaged pucks and bulk finger surrounding it move, causing relative motion. The relative motion between the actuators in each of these devices is transferred to the skin, which ends up producing localized skin strains across the contact patch. Schematics of the three devices are shown in Figure 2.

Psychophysical experiments showed that when the adjacent pins in Device 1 were normally actuated with a relative phase difference between them (i.e. when one pin moves up, the adjacent pin moves down), the perceived roughness increased with an increase in the phase difference. Increasing the number of instances of localized skin stretch by dispersing the pin actuations spatially across the contact patch also resulted in an increase in roughness perception. A similar observation was reported for Device 2—an augmentation in the perceived intensity with the increase in the phase decorrelation between neighboring pins. When the pin movements are decorrelated (i.e. when one pin moves left, the adjacent pin moves right), a localized skin stretch is produced, which, in turn, increases the perceived intensity. However, the same effect of phase delay between neighboring pucks on perceived roughness was not observed in Device 3. A 0° phase difference between the voltages applied to neighboring pucks produced the highest roughness perception, which peaked at 90° but dropped sharply to a minimum at 180°. A schematic explaining the device configuration at different phase differences between neighboring actuators is shown in Fig. S6. If localized skin stretch is indeed a critical contributor to roughness perception and the amount of skin stretch produced increases monotonically with the phase difference between adjacent actuators, we would expect the perceived roughness to show the same monotonic increase. However, this was not the case. A proper understanding of how the skin mechanics trigger the mechanoreceptors that contribute to roughness perception can help us design haptic devices that produce predictable and consistent roughness perception.

To this end, we developed a mechanics model that can predict the perceived roughness for arbitrary spatiotemporal stimuli in both normal and tangential directions. We validate these predictions against psychophysical experimental results from the literature for different haptic devices under a wide range of experimental conditions. The model also sheds light on the importance of considering shear forces in predicting the roughness perception in laterally actuated tactile devices.

Results

First, we used a multilayered tissue mechanics model to estimate the extent of mechanoreceptor activation. The slowly adapting type 1 (SA-1) Merkel cells produce a sustained response during all phases of mechanical stimulus (onset, hold, and offset), while the rapidly adapting type 1 (RA-1) Meissner corpuscles respond only during changes in mechanical stimulus (onset and offset) (Figure 1A). Different measures such as stresses (15), strains (2, 15), and strain energy densities (16, 17) have all been shown to correlate well to neural firing rates. Since strain provides a direct measure of the amount of stretch produced in the skin, we used the strain as a proxy for SA-1 activation, although the other measures would yield identical predictions because of the linear material properties assumed in this model. The change of strain was used as a proxy for RA-1 activation based on its responses to changes in indentation. These values were calculated at the interface between the epidermis and the dermis, where the mechanoreceptors are assumed to be located. Since roughness perception is determined by the spatial variation in SA-1 and RA-1 responses (10), we next computed the spatial variation in the strain and the change of strain by convolving it with a 1D Gabor filter, similar to the approaches used in (9, 10, 18, 19). Finally, we used the area under the curves as a measure of the average mechanoreceptor activation translating to roughness perception.

We tested how the model estimates performed against psychophysical experimental results for different device configurations (normal and lateral actuations) and experimental conditions (spatial dispersion and phase delay). Figure 3 shows an exemplary modeling workflow of how roughness perception is predicted for external stimuli applied to Device 2 with 0°, 90°, and 180° phase difference. Figure 4 shows the strains and the change of strains at the mechanoreceptor locations computed by the mechanics model, for the three devices that were investigated in this study. Figure 5 shows the correlation between the predictions of the model developed in this study and the experimental results reported in the literature.

Psychophysical experiments have shown that incorporating a phase delay between neighboring pin displacements or spatially dispersing the actuation can cause up to a 2-fold increase in roughness perception (3). It was hypothesized that this was due to the increased localized skin stretch produced by incorporating a phase delay, and the increased instances of skin stretch produced by incorporating a spatial dispersion.

Studies have shown that SA-1 responses can account for the roughness perception produced by dot patterns (2), and the variation in SA-1 activation is predictive of the roughness perception of coarse textures (9, 10). However, the variation in strain was unable to account for the roughness perception produced by stimuli involving spatial dispersion or phase differences between the actuators (Figure 5A). While the perceived roughness and variation in strain are well correlated for Device 2 (R² = 0.989), the correlation is very poor for Device 1 (R² = 0.0004) and a negative correlation is predicted for Device 3. Thus, the variation in strain is not a good candidate to predict roughness for distributed tactile stimuli.

It has been observed that all three afferent classes contribute to roughness perception (9, 10) and that for most natural textures, the variation in RA-1 activation is a more accurate predictor of roughness than SA-1 activation (10). Our model predictions show (Figure 5B) that the perceived roughness and variation in change of strain are very well correlated for all the three devices and experimental conditions reported in the literature (R² = 0.876, 0.987, and 0.959 for Devices 1, 2, and 3, respectively).

While a monotonic increase in roughness was observed with an increase in phase difference for both Device 1 and Device 2, it was not observed for Device 3. Device 3 was force-controlled, wherein the modulation of friction forces using electroadhesion was used to produce relative motion between the pucks. The displacements corresponding to each phase difference were experimentally characterized by imaging the finger–device interface, as given in Table 1. The phase difference between the voltages applied to the pucks does not translate linearly to the absolute and relative displacements produced. Since the absolute and relative displacements both contribute to the strains (and, in turn, the change of strains), the cumulative sum of the absolute and relative displacements should correlate to the roughness perception if it is indeed governed by the variation in change of strain. Computing the cumulative displacements from Table 1, we notice that this is indeed consistent with the psychophysical experimental results—180° phase difference produces the least roughness perception, followed by 0°, and then by 90° producing the highest roughness perception.

Thus, the inconsistency in Device 3 (nonmonotonic dependence of roughness perception on phase difference) was because the phase difference between the applied voltages did not translate linearly to the displacements produced in the finger. We hypothesize that this could have been due to several reasons: (i) the stiffness of the material (latex) in between the pucks might have mechanically coupled them, affecting their relative motion; (ii) the deformation produced by the low shear modulus of the material (Ecoflex 00-10) between the top of the pucks and the bottom of the finger might not have transferred the forces from the device to the finger; and (iii) certain vibration modes produced by the device dynamics led to nonlinearities in the system response.

Role of shear forces in roughness perception

While it is possible to predict the roughness perception produced by static dot patterns and other similar textures without including the shear forces (1, 2), including the frictional interactions and the concomitant shear forces is essential to correctly predict the roughness perception for laterally actuated distributed haptic devices. When two adjacent actuators move out-of-phase, they stretch the skin in between and produce a localized strain. Without the frictional shear forces, the actuators would just slide on the finger surface without producing any skin stretch (Figure 6A and B). To show the importance of shear forces in predicting roughness perception, we applied the same set of stimuli on Device 2 (adjacent pins actuated with 0°, 90°, and 180° phase delays) with and without frictional forces at the finger–device interface. We can see that without shear forces (Figure 6A), the variation in strain shows a negative correlation and the variation in change of strain contribution shows a weak correlation with perceived roughness. In contrast, the effect of phase difference on roughness perception is accurately captured when the shear forces are included (Figure 6B). In Device 1, the pins move only normally, and hence, additional movements in the normal direction dominate over the effect of shear forces. Hence, the effect of omitting shear forces on Device 1 is not pronounced. Since Device 3 is force-controlled and its working principle hinges on the transfer of shear forces across the finger–device interface to produce a skin stretch, a frictionless interface in this case cannot produce any skin motions and hence cannot make any predictions about roughness perception.

Discussion

We developed a model that can predict the roughness perception produced under a wide range of stimuli, and it is able to predict important phenomena such as the enhancement of roughness perception by applying spatial dispersion and phase difference between actuators, as shown in prior experimental work (3, 13, 14). We showed that the variation in the change of strain can predict the perceived roughness with strong correlation, which we believe is indicative of the importance of the role of the RA-1 mechanoreceptor activation in recreating roughness perception using tactile stimuli relying on localized skin stretch. For laterally actuated devices, it is also essential to include the shear forces due to interfacial friction to make correct predictions about roughness perception. We hope that the model can be used by haptic device designers to predict the efficacy of devices in rendering roughness perception and fast-track the iterative design process using predictive modeling rather than relying on trial-and-error for optimization.

Although the model can successfully predict the roughness perceived for a wide range of conditions, it has a few limitations. First, the finite element method (FEM) model assumes linear elastic material properties, which performs well for the experimental conditions that the model was validated against in this study. However, to accurately capture dynamic effects such as the role of actuation frequency of roughness perception, it might be important to include nonlinear material properties such as hyperelasticity and viscoelasticity in the model. Second, the model is based on SA-1 and RA-1 activation and performs well for the devices investigated in this study. However, it is well known that PC activation also plays a role in roughness perception, and roughness percepts can be best explained using a cumulative contribution of all three afferent types (1, 9, 10). Therefore, the model can be extended to include the contribution of PC afferents, especially for devices where vibration is an important contributing factor.

Materials and methods

Finger mechanics model

A 2D FEM model was developed in Abaqus to simulate the contact mechanics at the finger–device interface. The finger was modeled as a multilayered structure consisting of the bone, nail, nail bed, subcutaneous tissue, viable epidermis, dermis, and stratum corneum. The bone was assumed to be rigid, and the thicknesses and mechanical properties of the other layers were taken from the literature (20, 21). The dimensions and material properties of the different skin layers are given in Table 2. The pins in Device 1 and Device 2 were made of carbon fiber and aluminum, respectively, which are much stiffer than the skin tissue, and thus assumed to be rigid. Device 3 was made of silicone rubber which had an elastic modulus (∼1 MPa) comparable to the epidermis and affects how the shear forces are transferred to the finger; hence, it was modeled as deformable. Linear elastic material properties were assumed, and the domain was discretized using 2D plane strain elements.

Stimulus

All three devices were preindented until all four actuators were completely in contact with the finger surface. After that, the pins/pucks were moved by small amounts in either the normal or lateral directions, depending on the device configuration. For Device 1, the pins were moved 20 μm in the vertical direction toward or away from the skin, depending on the phase difference being simulated. Since actuation displacements for Device 2 were not reported in the paper, we used the same value that was reported for Device 1 and moved the pins 20 μm in the horizontal direction, either left or right. Device 3 is a force-controlled device, and hence, the absolute and relative displacements were phase-dependent, unlike Device 1 and Device 2 which are displacement-controlled and phase-independent. The displacements corresponding to each phase difference were experimentally characterized by imaging the finger–interface (14) (Table 1) and were used as the boundary conditions in the model.

Predicting roughness perception from the mechanical response of the finger

SA-1 mechanoreceptors respond to indentation, and RA-1 mechanoreceptors respond to changes in indentation (22). Therefore, we first computed the maximum principal strains and the change of these strains at the mechanoreceptor locations (the interface between the epidermis and the dermis) and used them as proxies for SA-1 and RA-1 activation, respectively. The strain plots shown in Figs. S1, S2, andS3 correspond to low, medium, and high spatial dispersion for Device 1, Fig. S4 corresponds to Device 2, and Fig. S5 corresponds to Device 3. Roughness perception has been shown to correlate well to the spatial variation in afferent spiking responses (10). A Gabor filter provides a measure of localized, periodic variation and has been widely used in the literature to explain the spatial and temporal codes of roughness perception (9, 10, 18, 19). Therefore, we next convolved the strains and change of strains with a 1D Gabor filter given by:

Here, λ is the spatial period and determines the peak sensitivity of the filter and σ determines the attenuation distance of the filter. We used $λ=2.8mm$ for the strain and $λ=1.15mm$ for the change of strain, based on the innervation densities of SA-1 and RA-1 mechanoreceptors, and a value of $σ=1.12mm$ was used. These values were taken from the literature, and no further optimization was performed. Finally, we computed the area under the curve after the convolution operation to compute the average roughness perception.

Supplementary Material

Supplementary material is available at PNAS Nexus online.

Funding

The authors acknowledge the Texas A&M University and Texas A&M Engineering Experiment Station startup funds, the Governor's University Research Initiative Grant No. GURI 2018-1, the Chancellor's University Research Initiative, the National Science Foundation under Grant Nos. 2106866 and 2106191, and the J. Mike Walker ‘66 Department of Mechanical Engineering Graduate Student Summer Research Grant Program.

Author Contributions

S.C., S.T., M.A.P., J.E.C., and M.C.H. did the conceptualization; S.C. and S.T. performed the formal analysis and investigation; S.C., C.C., A.K., and G.W. did the validation; J.E.C. and M.C.H. provided the supervision; and S.C. and M.C.H. did the writing (original draft, and review and editing, respectively).

Data Availability

Raw simulation and analysis files for the study are available in the Texas Data Repository. DOI/accession number(s): https://doi.org/10.18738/T8/WJLCC0.

References

Author notes