In [ ]:
! pip install datasets
In [3]:
import datasets
from datasets import load_dataset

Convieniently, the airline sentiment dataset from Kaggle is available on Hugging Face. It consits of over 14,000 tweets from airline customers collected in Feb 2015.

In [ ]:
ds = load_dataset("osanseviero/twitter-airline-sentiment")
In [5]:
ds = ds['train']

The dataset includes several columns. We only need the sentiment labels and the actual tweets (text).

In [6]:
ds
Out[6]:
Dataset({
    features: ['tweet_id', 'airline_sentiment', 'airline_sentiment_confidence', 'negativereason', 'negativereason_confidence', 'airline', 'airline_sentiment_gold', 'name', 'negativereason_gold', 'retweet_count', 'text', 'tweet_coord', 'tweet_created', 'tweet_location', 'user_timezone'],
    num_rows: 14640
})
In [7]:
ds= ds.select_columns(['text', 'airline_sentiment'])
In [8]:
ds = ds.rename_column('airline_sentiment', 'label')
In [9]:
ds
Out[9]:
Dataset({
    features: ['text', 'label'],
    num_rows: 14640
})

We'll use a roBERTa model from Hugging Face to do sentiment analysis.

In [ ]:
from transformers import pipeline

pipe = pipeline("sentiment-analysis", model="cardiffnlp/twitter-roberta-base-sentiment-latest")

Feed the tweets to the model to get predictions

In [12]:
outputs = pipe(ds['text'])
In [14]:
len(outputs), type(outputs)
Out[14]:
(14640, list)

Pull the labels from the tweet dataset

In [15]:
actual = ds['label']
In [16]:
len(actual), type(actual)
Out[16]:
(14640, list)

The labels are 'positive', 'negative' or 'neutral'

In [17]:
actual[0]
Out[17]:
'neutral'
In [18]:
outputs[0]['label']
Out[18]:
'neutral'

Compare the predictions to the actual labels and calculate the accuracy

In [19]:
correct = 0
total = 0
for i, label in enumerate(outputs):
    if label['label'] == actual[i]:
        correct += 1
    total += 1
In [20]:
print(f"Accuracy: {correct/total*100}")

Accuracy: 81.00409836065575