Spring 2023
Duration:
One Semester
Lab:
HCI/IC/EPFL
Goals:
Curate a chitchat oriented dialogue dataset containing aspect-level emotion annotations and develop a classifier capable of performing aspect level emotion analysis on chit-chat oriented dialogue data using large language models and prompting.
Assistant:
Anuradha Welivita (kalpani DOT welivita AT epfl DOT ch)
Student Name:
Taken
Abstract:
Conventional sentiment analysis mainly predicts the sentiment at the sentence or document level. In the conventional setting, it is assumed that a single sentiment is conveyed towards a single topic in the given text, which may not be the case in practice. For example, consider the following sentence:
“This actor is the only failure in an otherwise brilliant cast.”
In this single sentence, two opposite sentiments are attached to two different entities as follows:
- Entity: actor; Opinion: failure; Sentiment: Negative
- Entity: case; Opinion: brilliant; Sentiment: Positive
Therefore, the need for recognizing more fine-grained aspect-level opinions and sentiments dubbed as Aspect Based Sentiment Analysis (ABSA), has received increasing attention in the recent past (Zhang et al., 2022, Poria et al., 2020). ABSA thus builds a comprehensive opinion summary at the aspect level, which provides useful fine-grained sentiment information for downstream applications.
Researchers have experimented with numerous rule-based and neural network based approaches for ABSA. The advent of pre-trained language models (PLMs) such as BERT (Devlin et al., 2018) and RoBERTa (Liu et al., 2019) has brought substantial improvements on a wide range of ABSA tasks in recent years. Specially with the introduction of the PLM GPT3 (beta.openai.com/docs/models/gpt-3), most research is being conducted in using prompting for ABSA (Li et al., 2021).
Even though ABSA has gained popularity, there is comparatively less attention paid towards Aspect Based Emotion Analysis (ABEA) (De Geyndt et al., 2022). This is particularly due to lack of datasets containing aspect-level emotion annotations. In particular, there is quite a few research conducted on ABEA in dialogues (Song et al., 2022). ABEA on dialogue data can be very useful in understanding dialogues and generating emotionally targetted dialogue responses. An example is as follows:
Speaker: I passed the interview and got the job!
Listener: Wow that’s wonderful. How do you feel?
Speaker: I’m happy I got the job. But I’m anxious how the new boss is going to be.
(entity: job; emotion: happy) (entitiy: new boss; emotion: anxious)
Listener: Don’t worry, the boss will be fine. Congratulations!
(Targetted response addressing each emotion expressed in the previous turn)
The student is expected to:
- Curate a chitchat-oriented dialogue dataset containing aspect-level emotion annotations
- Develop a classifier that is able to perform aspect-based emotion analysis on chitchat-oriented dialogue data. The idea is to use a small-scale dataset developed in the previous step and use this dataset to fine-tune a large pre-trained language model such as GPT3 using prompting (See few-shot learning (Brown et al., 2020) and GPT3-based prompting: video link)
Keywords:
Dataset curation; aspect-based emotion understanding; prompting; large language models; empathetic chatbots; natural language processing
Suitable for:
Master student. Interested students should contact Anuradha Welivita (kalpani DOT welivita AT epfl DOT ch) along with a copy of your CV.
Duration:
One Semester
Lab:
HCI/IC/EPFL
Goals:
Curate a dataset of auto-generated dialogs between large language models and empathetic chatbots using prompting and benchmark it with existing automatic evaluation metrics
Assistant:
Ekaterina Svikhnushina (ekaterina DOT svikhnushina AT epfl DOT ch)
Student Name:
Taken
Abstract:
Building an empathetic chatbot is an important objective in dialog generation research. However, progress in this area is impeded by the lack of reliable automatic evaluation metrics allowing for the cost- and time-efficient prototyping.
Recently, a number of automatic metrics for open-domain dialog evaluation have been proposed (Y-T. Yeh et al., 2021). The majority of them rely on pre-generated human-chatbot dialogs. Therefore, they still suffer from the need to recruit human workers to generate dialogs for evaluation.
Some works investigated the use of self-played dialogs of a chatbot with itself for this purpose (A. Ghandeharioun et al., 2019). This is a promising approach, but so far it has only been tested on generic open-ended conversations, which are not necessarily emotionally colored. Moreover, the roles of a speaker and a listener in these dialogs were played by the same dialog model, which may not be representative of the corresponding interlocutors’ discourse in real life.
Building on top of the existing research and in order to address the specified gap, this project suggests creating and dataset of auto-generated dialogs, which are grounded in emotional prompts with a clear distinction of an emotional speaker and an empathetic listener roles. This dataset should be further benchmarked using the existing automatic evaluation metrics to understand their generalization ability to emotionally colored contexts.
The student is expected to:
- Curate auto-generated dialogs between a large language model (LLM) and several empathetic chatbots. This requires the application of promoting (P. Liu et al., 2021) to make the LLM play the role of an emotional speaker. Emotional prompts are expected to be based on the grounding scenarios from (E. Svikhnushina et al., 2022). LLM could be accessed through API, such as the one available from OpenAI.
- Review the literature on existing automatic metrics for open-domain dialog evaluation (Y-T. Yeh et al., 2021, A. Ghandeharioun et al., 2019) and benchmark the publicly available ones on the collected dataset.
- If time allows, run exploratory data analysis to identify the distinctive features between good and bad dialogs.
Keywords:
Dataset curation; prompting; large language models; empathetic chatbots; natural language processing
Suitable for:
Master student. Interested student should contact Ekaterina Svikhnushina (ekaterina DOT svikhnushina AT epfl DOT ch) and Pearl Pu (pearl DOT pu AT epfl DOT ch) along with a copy of your CV.