Topic Modeling and Sentiment Analysis of 2020 News

Results

1. Topic modeling gave us two clear dimensions: politics and economics.

2. The most important topic was COVID-19 followed by an economy related topic. There is a significant overlap between these.

3. Other topics include: climate change and sustainable development, technological development, the European Union, international relations (with an emphasis on China and Iran), the Israeli-Palestinian conflict, North American politics, Russian and Eastern European politics, and a last topic that is composed of a series of open letters signed by academics, politicians and people related to international institutions. This last topic is linked to the effect that the pandemic had on the relationship between developed and developing countries.

4. During the sentiment analysis we detected two periods of the year 2020, in which negative feelings predominated. Between March and June due to the pandemic, and in October related to the North American presidential elections and the claims of electoral fraud.

5. Regarding emotions: fear and sadness went up in March, because of the arrival of COVID to North America. On the other hand, confidence and anticipation dominated in October due to the expectations that were generated by the electoral victory of Joe Biden. In the same period, anger, sadness and fear also increased due to the uncertainty that was triggered as a result of Donalds Trump’s refusal to accept the electoral results.

Intro

Project Syndicate is a website that publishes opinion columns with authors from all over the world. They define themselves as The World’s Opinion Page. On the about page they mention that:

“Project Syndicate produces and delivers original, high-quality commentaries to a global audience. Featuring exclusive contributions by prominent political leaders, policymakers, scholars, business leaders, and civic activists from around the world.”

For this reason, I considered it the perfect portal to analyze in order to find the important topics of 2020. With this objective, I did topic modeling using the LDA algorithm. In addition, I performed sentiment analysis (polarity and emotions) using the sentimentR package of R. The database was collected by doing web scrapping with theRselenium package.

Topic Modeling

In order to find the most important topics discussed in 2020, I used the Latent Dirichlet Allocation (LDA), which is a popular method for modeling topics. It is important to remember that it treats each article as a mixture of topics and each topic as a mixture of words. This allows documents to overlap in terms of content, rather than separating them into distinct groups.

The algorithm was run with k = 10, which means that it tried to identify 10 topics. There are probably more topics, to find an optimal number the model can be trained just how a machine learning model would be trained. Unfortunately, this is computationally expensive (takes a long time), so I decided to only search for 10 topics. Here are the results:

The intertopic distance map is a way of visualizing the topics on a 2-dimensional plane. The dimension of these topics, represented by the size of the circle, is proportional to the number of words that belong to each topic. The circles, which represent the topics, are plotted based on the words that they consist of . Therefore, when there are two topics close together, this means that they have many words in common. The horizontal dimension (PC2) separates ‘political’ topics (left) from ‘economics/COVID related’ topics (right). The vertical dimension (PC1) separates the documents that are opinion columns, below, from the “open letters”, above.

The bar graph on the right shows, by default, the 30 most important terms. The bars indicate the total frequency of the term across all articles. “Salient” is a specific measure that is defined below the graph, it can be thought of as a metric used to identify the most informative or useful words to identify the topics in a database. A higher “saliency” value indicates that the word is more useful for identifying a specific topic.

It is possible to adjust the words that appear in the bar graph, by moving the λ (lambda) slider. When the values are close to 0, the less frequent but more unique terms of the selected topic are highlighted. Lambda values closer to 1 highlight terms that are more frequent in the topic but may not be exclusive to it.

Topics

The first topic is clearly about the pandemic and it is the most important one (in terms of size) of the past year, which is not a surprise at all. The second topic is about economic issues and there is an overlap between these first two topics, which makes sense given the economic uncertainty that was unleashed in various parts of the world in the wake of the pandemic. Topic 3 is related to climate change and sustainable development, as suggested by the words: climate, global, development, food, africa, water or energy.

Topic 4 at first glance seems to be about economics just like topic 2. To compare these two topics,I used the per-topic-per-word probabilities, called β (“beta” ). To find the terms with the greatest difference in β between the topics, the rate of the logarithm of 2 was used: log2 (β2 / β1).

When comparing both topics, it seems that topic 4 (red) is about technology. One of the most important words in this topic is workers, which is explained by the number of articles written about the impact of technology on employment during the pandemic. It seems, on the other hand, that topic 2 (blue) has a greater focus on the international economy due to words like ECB (European Central Bank), G20 or Currency).

At first it seems that topic 5 is about China, but after a more detailed analysis I found out that Iran is also an important keyword of this topic. This may seem confusing at first but since other defining keywords of this topic (war, global, international, power, and security) are very common in literature on international relations, this topic is possibly about IR and or geopolitics.

Topic 6 is about American politics as the words Trump, Biden and Americans seem to suggest. Topic 7 is related to the European Union. It’s located in both horizontal dimensions, so it is possible to think that it is a topic that deals with both political and economic aspects. Number 8 deals with themes related to Russia and Eastern Europe. There is a certain proximity to topic 6, which may be due to the impeachment process to which Donald Trump was subjected. Topic nine appears to be about the Israeli-Palestinian conflict. Something interesting is that there are also many words related to gender issues. This may be a sign of poor semantic coherence, which, could be is a sign that the number of topics I have is likely to be less than optimal.

The most mysterious topic is number 9 (red). Looking at the most relevant terms of the topic it is difficult to get an idea about its content. Since some of the keywords in this topic (economics, global; world bank) are similar to those found in Topic 2 (blue) I decided to compare these two groups.

It is difficult to find any meaning when looking at the terms that differentiate the two topics. So I decided to explore the articles that have a high probability of belonging to topic 10.

Looking at the titles, I got the impression that the articles were about how the economic relationship between rich and developing countries was affected by the pandemic. I analyzed the articles in detail, I found that they were open letters. These were signed by many people. Next to the name of the signatories there was always a small mention of the position that they had, that is why the words, president, minister, professor, secretary or director were so frequent in this topic.

Topics Over Time

The graph above is a heatmap that shows the percentage of articles by topic during the months of the year. The first thing that stands out is that topics 1 and 2 have a peak between March and May, because that’s when COVID19 reached the United States. This caused concern about the effect that the pandemic could have on the economy, which is the reason why the percentage of topic 2 (economy) also increased.

On the other hand, topic 6, North American politics, has a predominance in the months of October, November and December due to the presidential elections and Donald Trump’s refusal to accept the results of these. The assassination of Qassem Suleimani (an Iranian military officer, who was a member of the Islamic Revolutionary Guard Corps) in January 2020 is what causes the topic of international relations to peak in that month. There is another small peak in July due to a series of events related to Chinese international politics.

First, Iran announced that it was negotiating a bilateral agreement with China that included areas such as: trade, energy, infrastructure, telecommunications and even military cooperation. Second, the UK decided to ban Huawei from its 5G network. Third, the decision to impose a national security law in Hong Kong. Fourth, the bilateral tensions that followed an armed confrontation between China and India in the Ladakh region, which resulted in the death of soldiers from both sides.

Topic 8, Russia and Eastern Europe, peaked in August due to the protests that occurred after the Belarusian president announced that he had won the presidencial election with 80% of the votes. The other peak occurred in February because at the beginning of that month the first impeachment of Donald Trump concluded. The impeachment was carried out in response to the Trump-Ukraine scandal, Trump, allegedly, pressured the Ukrainian government to investigate Hunter Biden, the son of then-presidential candidate Joe Biden.

Topic 3, sustainability, peaked in September due to an event called The Green Recovery, according to Project Syndicate this was a “virtual sustainability event” which was broadcasted live in September of 2020. Possibly there were a number of columns published that month that were linked to the event. The first day of the event can be seen here

Sentiment Analysis

A sentiment analysis was performed using the sentimentR package. First, a polarity analysis was carried out, negative feelings versus positive feelings. The sentimentR package calculates the feelings within each article analysing all the sentences from the text. The specific way in which a score is assigned to an article can be reviewed in this link. The important thing is to know that positive values correspond to positive feelings and negative values correspond to negative feelings. We calculated the average sentiment score of the articles by month, these are the results we found:

The first thing that stands out is the clear effect the pandemic had had on the feelings of Project Syndicate columnists. The score was negative from March to May. Another occasion when the score dropped was in October due to the presidential elections in the United States and specifically due to Donald Trump’s refusal to accept the results. The second thing that was analyzed were emotions, not just polarity, in order to see specifically what happened, here is a graph with the results.

With the arrival of COVID to the United States and Europe, emotions such as fear and sadness increased. It seems that sadness has a longer peak, remaining stable until May. While confidence, happiness, and anticipation increased- mainly due to the expectations generated by Joe Biden’s victory - Trump’s refusal to accept the results caused feelings of anger, sadness and fear to also increased.

Final thoughts

This exploratory analysis sheds some light on the news events of 2020. It is fascinating to see the effects of the most defining events of the year had: the Pandemic and the North American elections, both in topic modeling and sentiment analysis.

Looking at the most important events of the beginning of the year, I need to mention the assassination of Qassem Soleimani and the impeachment of Donald Trump. With the arrival of COVID to the United States and Europe, everything changed. The agenda was dominated by this topic and by the economic consequences that the pandemic was going to have. Towards the end of the year the US presidential elections and Donald Trump’s subsequent refusal to accept his defeat came to the fore.

Finally, in the sentiments analysis and in the topic modeling there is room for improvement. First of all, the dictionaries of sentiment and emotions can be better adapted to have more adequate results. Furthermore, it is possible to add more topics to better understand which were the topics that were present on public discussion in 2020.