Dialogue Act Classification via Transfer Learning for Automated Labeling of Interviewee Responses in Virtual Reality Job Interview Training Platforms for Autistic Individuals
Authors
MDPI Signals Special Issue: Deep Learning and Transfer Learning . 2023 5 19; 4(2).
Abstract
Computer-based job interview training, including virtual reality (VR) simulations, have
gained popularity in recent years to support and aid autistic individuals, who face significant
challenges and barriers in finding and maintaining employment. Although popular, these training
systems often fail to resemble the complexity and dynamism of the employment interview, as the
dialogue management for the virtual conversation agent either relies on choosing from a menu of
prespecified answers, or dialogue processing is based on keyword extraction from the transcribed
speech of the interviewee, which depends on the interview script. We address this limitation through
automated dialogue act classification via transfer learning. This allows for recognizing intent from
user speech, independent of the domain of the interview. We also redress the lack of training data
for a domain general job interview dialogue act classifier by providing an original dataset with
responses to interview questions within a virtual job interview platform from 22 autistic participants.
Participants’ responses to a customized interview script were transcribed to text and annotated
according to a custom 13-class dialogue act scheme. The best classifier was a fine-tuned bidirectional
encoder representations from transformers (BERT) model, with an f1-score of 87%.