Sensory feedback, which can be presented in different modalities - single and combined, aids task performance in human-robot interaction (HRI). However, combining feedback modalities does not always lead to optimal performance. Indeed, it is not known how feedback modalities affect operator performance under stress. Furthermore, there is limited information on how feedback affects neural processes differently for males and females and under stress. This is a critical gap in the literature, particularly in the domain of surgical robotics, where surgeons are (...) under challenging socio-technical environments that burden them physiologically. In the present study, we posited operator performance as the summation of task performance and neurophysiological cost of maintaining that performance. In a within-subject design, we used functional near-infrared spectroscopy (fNIRS) to assess cerebral activations of 12 participants who underwent a 3D manipulation task within a virtual environment with concurrent feedback (visual and visual + haptic) in the presence and absence of a cognitive stressor. Cognitive stress was induced with the serial-7 subtraction test. We found that while task performance was higher with visual than visual + haptic feedback, it degraded under stress. The two feedback modalities were found to be associated with varying neural activities and neural efficiencies, and these were stress- and gender-dependent. Our findings engender further investigation into effectiveness of feedback modalities on males and females under stressful conditions in HRI. (shrink)