Learning to Identify Emotions in Text

ABSTRACT
This paper describes experiments concerned with the automatic
analysis of emotions in text. We describe the construction
of a large data set annotated for six basic emotions:
anger, disgust, fear, joy, sadness and surprise, and we
propose and evaluate several knowledge-based and corpusbased
methods for the automatic identification of these emotions
in text.
Categories and Subject Descriptors
I.2.7 [Natural Language Processing]: Text analysis
General Terms
Algorithms,Experimentation
Keywords
emotion annotation, emotion analysis, sentiment analysis
1. INTRODUCTION
Emotions have been widely studied in psychology and behavior
sciences, as they are an important element of human
nature. They have also attracted the attention of researchers
in computer science, especially in the field of human computer
interaction, where studies have been carried out on
facial expressions (e.g., [3]) or on the recognition of emotions
through a variety of sensors (e.g., [13]).
In computational linguistics, the automatic detection of
emotions in texts is becoming increasingly important from
an applicative point of view. Consider for example the tasks
of opinion mining and market analysis, affective computing,
or natural language interfaces such as e-learning environments
or educational/edutainment games.
For instance, the following represent examples of applicative
scenarios in which affective analysis could make valuable
and interesting contributions:
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. To copy otherwise, to
republish, to post on servers or to redistribute to lists, requires prior specific
permission and/or a fee.
SAC’08 March 1620,
2008, Fortaleza, Cear´a, Brazil
Copyright 2008 ACM 9781595937537/
08/0003 ...$5.00.
• Sentiment Analysis. Text categorization according to
affective relevance, opinion exploration for market analysis,
etc., are examples of applications of these techniques.
While positive/negative valence annotation is
an active area in sentiment analysis, we believe that
a fine-grained emotion annotation could increase the
effectiveness of these applications.
• Computer Assisted Creativity. The automated generation
of evaluative expressions with a bias on certain
polarity orientation is a key component in automatic
personalized advertisement and persuasive communication.
• Verbal Expressivity in Human Computer Interaction.
Future human-computer interaction is expected to emphasize
naturalness and effectiveness, and hence the
integration of models of possibly many human cognitive
capabilities, including affective analysis and generation.
For example, the expression of emotions by synthetic
characters (e.g., embodied conversational agents)
is now considered a key element for their believability.
Affective words selection and understanding is crucial
for realizing appropriate and expressive conversations.
This paper describes experiments concerned with the emotion
analysis of news headlines. In Section 2, we describe
the construction of a data set of news titles annotated for
emotions, and we propose a methodology for fine-grained
and coarse-grained evaluations. In Section 3, we introduce
several algorithms for the automatic classification of news
headlines according to a given emotion. In particular we
present several algorithms, ranging from simple heuristics
(e.g., directly checking specific affective lexicons) to more
refined algorithms (e.g., checking similarity in a latent semantic
space in which explicit representations of emotions
are built, and exploiting Na¨ıve Bayes classifiers trained on
mood-labeled blogposts). Section 4 presents the evaluation
of the algorithms and a comparison with the systems that
participated in the SemEval 2007 task on “Affective Text.”
It is worth noting that the proposed methodologies are
either completely unsupervised or, when supervision is used,
the training data can be easily collected from online moodannotated
materials.
2. BUILDINGA DATA SET FOREMOTION ANALYSIS
For the experiments reported in this paper we use the data
set we developed for the SemEval 2007 task on “Affective
Text” [14].
The task was focused on the emotion classification of news
headlines extracted from news web sites. Headlines typically
consist of a few words and are often written by creative
people with the intention to “provoke” emotions, and
consequently to attract the readers’ attention. These characteristics
make this type of text particularly suitable for
use in an automatic emotion recognition setting, as the affective/
emotional features (if present) are guaranteed to appear
in these short sentences.
The structure of the task was as follows:
Corpus: News titles, extracted from news web sites (such
as Google news, CNN) and/or newspapers. In the case
of web sites, we can easily collect a few thousand titles
in a short amount of time.
Objective: Provided a predefined set of emotions (anger,
disgust, fear, joy, sadness, surprise), classify the
titles with the appropriate emotion label.1
The task was carried out in an unsupervised setting, and
consequently no training was provided. The reason behind
this decision is that we wanted to emphasize the study of
emotion lexical semantics, and avoid biasing the participants
toward simple “text categorization” approaches. Nonetheless
supervised systems were not precluded from participation,
and in such cases the teams were allowed to create their
own supervised training sets.
Participants were free to use any resources they wanted.
We provided a set of words extracted from WordNet Affect
[15], relevant to the six emotions of interest. However,
the use of this list was entirely optional.
2.1 Data Set
The data set consisted of news headlines drawn from major
newspapers such as New York Times, CNN, and BBC
News, as well as from the Google News search engine. We
decided to focus our attention on headlines for two main reasons.
First, news have typically a high load of emotional content,
as they describe major national or worldwide events,
and are written in a style meant to attract the attention of
the readers. Second, the structure of headlines was appropriate
for our goal of conducting sentence-level annotations
of emotions.
Two data sets were made available: a development data
set consisting of 250 annotated headlines, and a test data
set consisting of 1,000 annotated headlines.2
2.2 Data Annotation
To perform the annotations, we developed a Web-based
annotation interface that displayed one headline at a time,
together with six slide bars for emotions and one slide bar
for valence. The interval for the emotion annotations was
set to [0, 100], where 0 means the emotion is missing from
1The task also included a valence classification track.
2The data set and more information about the
task can be found at the SemEval 2007 web site
http://nlp.cs.swarthmore.edu/semeval.
the given headline, and 100 represents maximum emotional
load.
Unlike previous annotations of sentiment or subjectivity
[18, 12], which typically rely on binary 0/1 annotations, we
decided to use a finer-grained scale, hence allowing the annotators
to select different degrees of emotional load.
The test data set was independently labeled by six annotators.
The annotators were instructed to select the appropriate
emotions for each headline based on the presence of
words or phrases with emotional content, as well as the overall
feeling invoked by the headline. Annotation examples
were also provided, including examples of headlines bearing
two or more emotions to illustrate the case where several
emotions were jointly applicable. Finally, the annotators
were encouraged to follow their “first intuition,” and to use
the full-range of the annotation scale bars.
2.3 InterAnnotator Agreement
We conducted inter-tagger agreement studies for each of
the six emotions. The agreement evaluations were carried
out using the Pearson correlation measure, and are shown
in Table 1. To measure the agreement among the six annotators,
we first measured the agreement between each annotator
and the average of the remaining five annotators,
followed by an average over the six resulting agreement figures.
Emotions
anger 49.55
disgust 44.51
fear 63.81
joy 59.91
sadness 68.19
surprise 36.07
Table 1: Pearson correlation for inter-annotator
agreement
2.4 Finegrained and Coarsegrained Evaluations
Provided a gold-standard data set with emotion annotations,
we used both fine-grained and coarse-grained evaluation
metrics for the evaluation of systems for automatic
emotion annotation.
Fine-grained evaluations were conducted using the Pearson
measure of correlation between the system scores and
the gold standard scores, averaged over all the headlines in
the data set.
We also ran coarse-grained evaluations, where each emotion
was mapped to a 0/1 classification (0 = [0,50), 1 =
[50,100]). For the coarse-grained evaluations, we calculated
precision, recall, and F-measure.
3. AUTOMATIC EMOTION ANALYSIS
3.1 Knowledgebased
Emotion Annotation
We approach the task of emotion recognition by exploiting
the use of words in a text, and in particular their cooccurrence
with words that have explicit affective meaning.
As suggested by Ortony et al. [11], we have to distinguish
between words directly referring to emotional states (e.g.,
“fear”, “cheerful”) and those having only an indirect reference
that depends on the context (e.g., words that indicate
possible emotional causes such as “killer” or emotional responses
such as “cry”). We call the former direct affective
words and the latter indirect affective words [16].
As far as direct affective words are concerned, we follow
the classification found in WordNet Affect.3. This is an
extension of the WordNet database [5], including a subset of
synsets suitable to represent affective concepts. In particular,
one or more affective labels (a-labels) are assigned to a
number of WordNet synsets. There are also other a-labels
for those concepts representing moods, situations eliciting
emotions, or emotional responses. Starting with WordNet
Affect, we collected six lists of affective words by using the
synsets labeled with the six emotions considered in our data
set. Thus, as a baseline, we implemented a simple algorithm
that checks the presence of this direct affective words in the
headlines, and computes a score that reflects the frequency
of the words in this affective lexicon in the text.
Sentiment analysis and the recognition of the semantic
orientation of texts is an active research area in the field of
natural language processing (e.g., [17, 12, 18, 9]). A crucial
aspect is the availability of a mechanism for evaluating
the semantic similarity among “generic” terms and affective
lexical concepts. To this end we implemented a semantic
similarity mechanism automatically acquired in an unsupervised
way from a large corpus of texts (e.g., British National
Corpus4). In particular we implemented a variation of Latent
Semantic Analysis (LSA). LSA yields a vector space
model that allows for a homogeneous representation (and
hence comparison) of words, word sets, sentences and texts.
For representing word sets and texts by means of an LSA
vector, we used a variation of the pseudo-document methodology
described in [1]. This variation takes into account also
a tf-idf weighting schema (see [6] for more details). In practice,
each document can be represented in the LSA space
by summing up the normalized LSA vectors of all the terms
contained in it. Thus a synset in WordNet (and even all
the words labeled with a particular emotion) can be represented
in the LSA space, performing the pseudo-document
technique on all the words contained in the synset. In the
LSA space, an emotion can be represented at least in three
ways: (i) the vector of the specific word denoting the emotion
(e.g. “anger), (ii) the vector representing the synset of
the emotion (e.g. {anger, choler, ire}), and (iii) the vector
of all the words in the synsets labeled with the emotion.
In this paper we performed experiments with all these three
representations.
Regardless of how an emotion is represented in the LSA
space, we can compute a similarity measure among (generic)
terms in an input text and affective categories. For example
in a LSA space built form the BNC, the noun “gift” is highly
related to the emotional categories joy and surprise. In
summary, the vectorial representation in the LSA allows us
to represent, in a uniform way, emotional categories, generic
3WordNet Affect is freely available for research purpose
at http://wndomains.itc.it See [15] for a complete description
of the resource.
4BNC is a very large (over 100 million words) corpus
of modern English, both spoken and written (see
http://hcu.ox.ac.uk/bnc/). Other more specific corpora
could also be considered, to obtain a more domain
oriented similarity.
terms and concepts (synsets), and eventually full sentences.
See [16] for more details.