There is an age-old debate in the US about how the press and other media outlets may be politically biased. This has been particularly salient during the recent Clinton-Trump presidential campaign and since the election took place at the end of 2016. I have personally followed the whole campaign quite closely and have tried to regularly watch several TV news channels in order to absorb different views and perspectives. I've also developed my own subjective sense of what side of the political spectrum each channel tends to lean toward. Yet, most channels claim to be independent and unbiased (e.g. Fox News' slogan is "Fair & Balanced"). So I thought it'd be interesting to try and find objective ways to measure how biased media outlets actually are.
In this article, I describe a case study that I've recently worked on. First, I explain the study's methodology, which is based on the sentiment analysis of videos published on Youtube by a number of prominent American TV channels. I then present some details about the acquired dataset beforing laying out the study's results. Finally I discuss some limitations about the study and possible areas for improvement.
All the code that I've written to support this study has been published on Github (https://github.com/jphalip/media-bias). Feel free to download that code if you're interested in running similar studies yourself.
Methodology¶
For this study I've decided to analyze videos posted on Youtube by some of the most prominent American TV news channels, including the so-called "Big Three" (Fox News, CNN & MSNBC) and CBS News. One may argue that not all content aired on a given TV channel necessarily ends up being published on that channel's Youtube account, and therefore that this may skew the results. My counter-argument is that, by curating videos on their Youtube account for online consumption, a TV channel likely exposes even greater bias, therefore making the Youtube data even more pertinent for the purpose of this study.
It was fairly easy to download the video metadata for all channels (titles, descriptions, publication dates, etc.) using the Youtube API: https://github.com/jphalip/media-bias/blob/master/code/youtube_api.py
Once all the video metadata acquired, I selected a range of political topics ("Obama", "Clinton", "Trump", "Democrats", "Republicans", "Conservatives", "Liberals") and extracted the relevant videos that mentioned those topics. Titles that contained different variants of a same topic's word were flagged for that topic ("Mr. Obama Goes To Washington" and "The Obamas Vacation in N.C." were flagged for the "Obama" topic). This approach also means that, for example, videos about Melania Trump were flagged for the "Trump" topic and videos about Bill Clinton were flagged for the "Clinton" topic. Also, people's first names were not used for filtering and flagging videos as they are too generic and could have yielded false positives (e.g. "Donald" could have yielded videos about Donald Rumsfeld or Donald Sterling).
Then, I've analyzed the sentiment of all relevant video titles. The Oxford Dictionaries defines "sentiment analysis" as follows:
The process of computationally identifying and categorizing opinions expressed in a piece of text, especially in order to determine whether the writer's attitude towards a particular topic, product, etc. is positive, negative, or neutral. (Source: Oxford Dictionaries)
The goal is to find out whether sentiment analysis can help identify differences in how channels treat contentious political topics. This isn't the first time that sentiment analysis is used for that purpose (see for example this study showcased on the Washington Post). However, I think that my study is a bit unique in the way that it breaks down data into multiple topics.
When looking through the data I noticed that the video descriptions often contained some generic or irrelevant text (e.g. "Follow msnbc on Tubmlr" or "Check out FOX 411 for more entertainment news and gossip"). I wanted to prevent that irrelevant text from influencing the sentiment analysis results. Cleaning up that data would have been extremely time-consuming and tedious. So I ignored all the video descriptions and only analysed the sentiment of video titles. I would posit that Youtube users generally pay much more attention to the video titles than to their descriptions anyways.
The sentiment analysis was performed using the Google Natural Language API. It's pretty simple and straighforward: you send a piece of text to the API server, which then analyzes that text and returns a sentiment score between -1.0 (negative sentiment) and 1.0 (positive sentiment) as determined by Google's machine learning algorithms. The complete code for downloading sentiment scores for all videos and for saving the data to disk can be found here: https://github.com/jphalip/media-bias/blob/master/code/language_api.py
Note that the Google Natural Language API is not free. The function above was called for each of the ~30K relevant videos that I had collected, which in the end cost me around $30 US. If you're interested in running similar experiments, make sure to first refer to the API's pricing page for adequate budgeting.
Alright, now we're ready to do some exploration. Let's jump right in!
Data exploration¶
First, let's import all the code libraries that we need in order to perform our work:
from __future__ import division
from IPython.display import display
from datetime import datetime
import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
from code.utils import show_videos
from code.plotting import plot_channel_stats, plot_compressed_channel_stats, plot_sentiment_series
%matplotlib inline
Here are the channels considered for this study:
channels = pd.read_csv('channels.csv')
channels[['title', 'url', 'color']]
(Note: The color attributes are just arbitrary colors that will be used to visually differentiate the channels in all the graphs featured later in this article.)
Here are the topics that I've chosen to focus on:
topics = pd.read_csv('topics.csv')
topics[['title', 'slug']]
(Note: The slug attributes are the names used for the columns in the video dataset that flag videos relating to specific topics — See the video dataset in question right below.)
Now let's take a look at the video data. Here's a small sample below:
all_videos = pd.read_csv('videos.csv', parse_dates=['published_at'])
all_videos.head(3)
The youtube_id is the unique ID assigned by Youtube to each video (Tip: you may watch any video by visiting the URL https://www.youtube.com/watch?v=[insert the video youtube_id here]). Note that there is a separate column for each topic that is named after the topic's slug (e.g. obama, democrats, conservatives) and that flags videos as being relevant to the corresponding topic (i.e. the value is True if the topic is mentioned in the video title, or False otherwise). The relevant column is a flag that is True only if at least one of the topics is mentioned in the video titles — That column allows us to easily extract videos that are directly relevant to this study. You can see how all those columns were calculated and pre-processed by referring to the create_topic_columns() function published here: https://github.com/jphalip/media-bias/blob/master/code/utils.py
Now let's look at some general statistics about our dataset:
num_relevant = all_videos.relevant.sum()
num_total = all_videos.shape[0]
print 'Number of relevant videos: %s' % num_relevant
print 'Total number of videos: %s' % num_total
print 'Percentage of relevant videos: %0.2f%%' % (100*num_relevant/num_total)
So the chosen topics are covered in about 18% of all videos ever published by the selected channels, which I'd argue is sufficiently significant for the purposes of this study. Let's now drill down a bit further and see to what extent those topics are covered overall by each channel:
channel_stats = pd.DataFrame({
'relevant': all_videos.groupby('channel').relevant.sum().astype(int),
'total': all_videos.groupby('channel').size()
})
channel_stats['percentage_relevant'] = (100*channel_stats.relevant/channel_stats.total).round(2)
channel_stats.sort_values('percentage_relevant', ascending=False)
Fox News and MSNBC both cover those topics quite extensively (in about 40% of all their published videos), whereas CBS News and CNN both seem to cover all sorts of other topics (probably sport, science, entertainment, etc). This indicates that Fox News and MSNC are both quite focused on politics.
We can now see quantitatively how much (in absolute numbers) each individual topic is covered by those channels:
absolutes = all_videos.groupby('channel')[topics.slug].sum().astype(int)
display(absolutes)
Some initial observations can be made from the above table:
- CNN talks a lot about Obama, a lot more so than other channels.
- The term "Liberals" in video titles seems to be mostly used by Fox News. MSNBC almost never mentions it.
- Trump has been covered about twice as much as Clinton.
One way to evaluate how important each topic is to each channel, is to calculate the percentage of their videos covering that topic relatively to the entire number of videos that the channel has published:
totals = all_videos.groupby('channel').size()
relatives = 100 * absolutes.divide(totals, axis=0)
display(relatives)
Those percentages can illustrated as follows:
plot_channel_stats(relatives, topics, channels, title='Relative topic coverage\n(% of total # of each channel\'s videos)')
Based on those graphs we can draw a couple more observations:
- Obama is mentioned in 3-8% of videos from all channels. That is not too surprising given that he's been president for 8 years.
- Fox News and MSNBC have mentioned Trump in a quarter of their videos; they've mentioned Clinton twice as less.
(Note: If you're interested in checking out the code written to generate those graphs, please refer to: https://github.com/jphalip/media-bias/blob/master/code/plotting.py)
Sentiment analysis¶
Overall sentiments¶
Alright, this is where things get much more interesting. As mentioned earlier, the sentiment of all relevant videos was calculated using the Google Natural Language API. The sentiment scores were stored in a separate CSV file. Here is a small sample:
sentiments = pd.read_csv('sentiments.csv')
sentiments[['youtube_id', 'sentiment_score']].head()
The youtube_id column contains the unique video IDs. The sentiment_score column contains the scores (between -1 and 1) for all relevant videos. A score of 0 would correspond to neutral sentiment, a score of -1 to extremely negative sentiment, and a score of 1 to extremely positive sentiment.
Let's merge the sentiment scores into the main video dataset and then look at a small sample of videos with positive and negative sentiments:
videos = all_videos[all_videos.relevant].merge(sentiments, on='youtube_id')
# Some videos with negative sentiment:
videos.sort_values('sentiment_score')[['channel', 'title', 'sentiment_score', 'youtube_id']].head(4)
# Some videos with positive sentiment:
videos.sort_values('sentiment_score', ascending=False)[['channel', 'title', 'sentiment_score', 'youtube_id']].head(4)
Here is how the sentiment scores are distributed across the dataset:
sns.distplot(videos.sentiment_score, axlabel=False, ax=plt.gca())
plt.title('Sentiment scores distribution')
plt.gca().get_yaxis().set_visible(False)
plt.xlim(-1,1)
plt.show()
Note that a majority of video titles tend to have a positive sentiment in the 0-0.5 range.
We're now in a position to visualize the average sentiment of each channel toward each topic:
scores = pd.DataFrame(index=channels.sort_values('title').title, columns=topics.slug, )
for channel, group in videos.groupby('channel'):
for topic in topics.slug:
scores.loc[channel, topic] = group[group[topic]].sentiment_score.mean()
scores = scores.rename_axis('Topic', axis=1)
scores = scores.rename_axis('Channel', axis=0)
display(scores)
plot_channel_stats(scores, topics, channels, fig_height=10, y_center=True, title='Average sentiment by topic')
For a different perspective, the same data can be compressed into the following graph:
plot_compressed_channel_stats(scores, y_center=True, title='Average sentiment by topic')
The graphs above allow us to make a few interesting observations:
- MSNBC seems to have a positive tone overall.
- Obama has been generally spoken of in positive terms everywhere except on Fox News.
- Clinton has been spoken of in negative terms everywhere except on MSNBC.
- Trump has overall clearly been spoken of in positive terms across the board.
- Conservatives have generally been spoken of in positive terms.
- Each channel seems to cover Democrats with the roughly same tone as they do Republicans.
- Liberals have been covered in negative terms (except MSNC, which as mentioned before doesn't really use that word at all).
Evolution through time¶
The statistics that I've presented so far were all averages. While those give an idea of the overall sentiment on each topic, it would be quite interesting to also see how the sentiment has evolved through time, in particular over the past two years during the presidential campaign. Let's take a look at this evolution with the timeseries graphs below:
plot_sentiment_series(videos, topics, channels, start_date=datetime(2015, 1, 1), title='Sentiment evolution during the presidential campaign')
A few observations can be made from the above graphs:
- MSNBC appears to consistently cover all topics in a fairly positive tone.
- The sentiment about Republicans and Democrats regularly oscillates between positive and negative for all channels (except MSNBC).
- Obama is consistenly spoken of negatively by Fox News, positively by MSNBC, and mixedly by CBS News and CNN.
- Sentiment about Trump is starkly positive throughout.
- Clinton was mostly spoken of in negative terms during the campaign. Even the sentiment on MSBNC, which is generally mostly positive, was just above neutral on Clinton.
Left-wing vs Right-wing¶
For better or worse, it is quite common in the US to consider the political spectrum as bi-modal: left-wing and right-wing. In other words: "Liberals vs Conservatives", "Democrats vs Republicans" or "Obama & Clinton vs Trump". To see how sentiments from our dataset are distributed across this bi-modal spectrum, we can separate left-oriented topics from right-oriented topics and then calculate the averages for each channel:
# Separate left-oriented topics from right-oriented topics
left_topics = ['obama', 'clinton', 'democrats', 'liberals']
right_topics = ['trump', 'republicans', 'conservatives']
# Create two new flag columns, one for each mode
videos['left'] = np.any(videos[left_topics], axis=1)
videos['right'] = np.any(videos[right_topics], axis=1)
# Calculate average sentiments for each channel
modes = ['left', 'right']
scores = pd.DataFrame(index=channels.sort_values('title').title, columns=modes)
for channel, group in videos.groupby('channel'):
for mode in modes:
scores.loc[channel, mode] = group[group[mode]].sentiment_score.mean()
scores = scores.rename_axis('Topic', axis=1)
scores = scores.rename_axis('Channel', axis=0)
display(scores)
The same results can be represented graphically as follows:
plot_compressed_channel_stats(scores, color=['#50AFE8', '#E61B23'], y_center=True, title='Average sentiment: Left-wing vs Right-wing')
If sentiment analysis is to be trusted, then those channels all appear to be fairly conservative. Also, the overall tone is generally positive except for one notable exception, as Fox News tends to cover left-oriented topics in negative terms.
Limitations and future improvements¶
I will be the first to admit that sentiment analysis, as conducted in this study, isn't perfect and does have some limitations.
On subjectivity¶
Sentiment analysis is sometimes criticized for its subjectivity as two persons may disagree on whether a given sentence is negative or positive.
See for example the following videos that were estimated by the Google API to have a positive sentiment score:
show_videos(videos, ['NDq3Ojmk0mI', 'nLU12dCJpZ8'])
... and those videos with negative sentiment scores:
show_videos(videos, ['qwws4b22NIk', 'INNBVixrAgc'])
You may have your own opinion on whether the videos above were accurately rated. Some level of faith must be placed in the used algorithm (in this case Google's Natural Language API) to get more objective results overall if given sufficiently large data points. As data scientist Matthew Russell says:
It’s critical to mine a large — and relevant — sample of data when attempting to measure sentiment. No particular data point is necessarily relevant. It’s the aggregate that matters. (Source)
Here the hope is that the 30K videos that were analyzed constitute a large-enough dataset. But results could certainly be improved by integrating more videos from more channels to the research.
On context¶
The sentiment analysis was performed for each video title in a complete vacuum, without any context. Yet, context can sometimes be critical in accurately estimating sentiment, as entrepreneur AJ Bruno puts it:
Context also plays a big role in understanding a writer’s feelings on a subject. “That movie was bad!” is definitely negative sentiment from a 50 year old film critic, but it might be glowing praise from a 17 year old boy. (Source)
In this study, it's reasonable to assume that context is somewhat narrow anyways, as all data comes from news channels (i.e. not from comedy channels that may have used sarcasm, for example). However, the approach could perhaps be improved by taking notable political events into consideration in order to apply specific weights to certain sentiment scores.
On mixed topics¶
Sometimes the same video title may mention multiple opposite topics (e.g. Trump and Clinton, or Democrats and Republicans). When that occurs, the same sentiment score gets applied to each of the topics covered. Sometimes that is fine if the topics are treated the same way, for example in the following videos where both Trump and Clinton are criticized, or praised at the same time:
show_videos(videos, ['qmNDKUk-JuE', 'pL3HOH2YG9I'])
This is more problematic if one topic is treated more favorably than the other in the same sentence. For example in these videos:
show_videos(videos, ['Wjtm14fDAjQ', 'JmCmNHEGWWs'])
For the above videos, a positive score will be applied to both topics "Clinton" and "Trump". Similarly a negative score will be applied to both topics for the following videos:
show_videos(videos, ['f5NSJStPxEw', 'tRQm6tGyGbs'])
Ideally, in each of those problematic instances, different scores should be applied to each topic. To implement this we could perhaps try to automatically analyze the syntax of each video title and then try to determine how each topic is impacted by the overall sentiment of that title. The Google Natural Language API actually also provides a syntax analysis service that is worth looking into for that purpose.
Currently our dataset contains about 8% of video titles that cover more than one topic at the same time:
(np.sum(videos[topics.slug], axis=1) > 1).sum() * 100 / len(videos)
The dataset contains about 6% of video titles that cover both left-oriented and right-oriented topics at the same time:
np.sum(videos.left & videos.right) * 100 / len(videos)
If analyzing the syntax proves to be too difficult or tedious, then a simple solution would be to delete all problematic records in order to completely remove potential ambiguities.
Other possible improvements¶
Other improvements could be considered in future revisions of this study:
- In order to more accurately make comparisons between different channels, the data could be normalized by taking into account the overall sentiment of each channel, i.e. the average sentiment score for all videos ever published by each channel (even for videos that do not cover the studied topics).
- The algorithm I wrote to flag videos for each topic was fairly naive in that it only looked for exact matches. It's possible that some video titles contained typos and therefore were missed. The filtering algorithm could be improved by using fuzzy search instead.
- Last but not least, it'd be interesting to extend the study to other news channels, for example: Bloomberg, ABC News, NBC News, Fusion, Free Speech TV, or the Blaze.
Conclusion¶
This study provided an attempt at objectively evaluating bias in the media by analyzing the sentiment of video titles published a few prominent TV channels' Youtube accounts. The approach of using sentiment analysis is certainly not perfect. It is questionable whether positive or negative sentiment on a topic necessarily represents bias toward or against that topic. Similarly, neutral sentiment may not necessarily mean neutral political bias. My hope, however, is that this study offers some direction and inspiration for future research in this field.
Most of all, I hope you enjoy reading this article as much as I did working on it. Any feedback is welcome and appreciated!