People with disabilities or special needs can benefit from AI-based conversational agents, which are used in competence training and well-being management. Assessment of the quality of interactions with these chatbots is key to being able to reduce dissatisfaction with them and to understand their potential long-term benefits. This will in turn help to increase adher-ence to their use, thereby improving the quality of life of the large popula-tion of end-users that they are able to serve. We systematically reviewed the literature on methods of assessing the perceived quality of interactions with chatbots, and identified only 15 of 192 papers on this topic that included people with disabilities or special needs in their assessments. The results also highlighted the lack of a shared theoretical framework for assessing the per-ceived quality of interactions with chatbots. Systematic procedures based on reliable and valid methodologies continue to be needed in this field. The cur-rent lack of reliable tools and systematic methods for assessing chatbots for people with disabilities and special needs is concerning, and may lead to un-reliable systems entering the market with disruptive consequences for users. Three major conclusions can be drawn from this systematic analysis: (i) re-searchers should adopt consolidated and comparable methodologies to rule out risks in use; (ii) the constructs of satisfaction and acceptability are differ-ent, and should be measured separately; (iii) dedicated tools and methods for assessing the quality of interaction with chatbots should be developed and used to enable the generation of comparable evidence.
Preliminary Results of a Systematic Review: Quality Assessment of Conversational Agents (Chatbots) for People with Disabilities or Special Needs
de Filippis, Maria Laura;Federici, Stefano
;Mele, Maria Laura;Borsci, Simone;Bracalenti, Marco;
2020
Abstract
People with disabilities or special needs can benefit from AI-based conversational agents, which are used in competence training and well-being management. Assessment of the quality of interactions with these chatbots is key to being able to reduce dissatisfaction with them and to understand their potential long-term benefits. This will in turn help to increase adher-ence to their use, thereby improving the quality of life of the large popula-tion of end-users that they are able to serve. We systematically reviewed the literature on methods of assessing the perceived quality of interactions with chatbots, and identified only 15 of 192 papers on this topic that included people with disabilities or special needs in their assessments. The results also highlighted the lack of a shared theoretical framework for assessing the per-ceived quality of interactions with chatbots. Systematic procedures based on reliable and valid methodologies continue to be needed in this field. The cur-rent lack of reliable tools and systematic methods for assessing chatbots for people with disabilities and special needs is concerning, and may lead to un-reliable systems entering the market with disruptive consequences for users. Three major conclusions can be drawn from this systematic analysis: (i) re-searchers should adopt consolidated and comparable methodologies to rule out risks in use; (ii) the constructs of satisfaction and acceptability are differ-ent, and should be measured separately; (iii) dedicated tools and methods for assessing the quality of interaction with chatbots should be developed and used to enable the generation of comparable evidence.I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.
