An investigation of visualization in enhancing empathic communication through intelligence augmentation in the context of narrative medicine

Narrative Medicine (NM) cares about patients’ experiences and stories in the medical context. In NM, physicians “reach” and “join” their patients in their illnesses and empathize with their situations by reading and receiving patients’ stories. However, challenges, such as physicians’ limited empathic capability, lack of initiative, and time constraints, impede the practice in NM. 

Data-driven medical visualization has the potential to support healthcare in scientific analysis and diagnosis with the assimilation of statistical evidence data. While in NM, visualization with anecdotal evidence, such as narrative elements, relating the experiences and emotions is required. The research reviewed visualization studies for NM, defining four visualization research domains and five leading visualization solutions for NM and exploring future opportunities since there is no survey article on this field. A future work framework of visualization for NM is concluded. As per the framework, the thesis defines the research context as synchronous (real-time) face-to-face empathic medical communication, one of the trends of interest in visualization for NM. Few previous visualization works have focused on it. Intelligence Augmentation (IA) represents a burgeoning field of AI that seeks to empower humans by fostering human and AI collaboration instead of displacing human labor. Therefore, this thesis is devoted to investigating if and how IA-based interactive visualization can enhance empathic communication in real-time face-to-face medical conversation.

The research first investigates the requirements through a scenario-based interview and a field-based study based on the EMVIS I (EMotional VISualization) visualization. The results indicate information requirements: patients’ emotional transition and background conditions. Four user-supportive factors highlight the design targets of visualization, and specific function requirements are discussed. 

Then, it offers a novel IA framework, EOPD (Empathic Opportunity Perception and Distinction), to support physician-AI collaboration. The multi-modal corpus is constructed as the first corpus in this area, and the two ML models have been built and trained to target specific empathic tasks, which are also the first baseline for further model improvement in this context. Next, EMVIS II, based on EOPD, is designed with “shared” visualization, involving an interaction framework and three main visualization components. Interaction channels for co-recognition and co-collection, as well as the mixed visual forms and scenario-based visualization structure, are highlighted for design considerations.

Finally, the study delves into the impact of EMVIS II through a field-based empirical study. The results uncover that IA-based visualization can facilitate and enhance real-time face-to-face empathic medical dialogues. By using the visualization, physicians show improvements in their efforts of empathy and perceived empathy performance. Its impacts include augmenting complex affective empathy, facilitating and complementing empathy concern and providing tailored support to junior and senior physicians.