607 Final Project-Analysis of Global Energy Usage
Chunhui Zhu & Nathan Cooper
December 2, 2017
Part II: The analysis of 1990-2014 import and export energy trade and energy consumption
Proposal
Global Energy efficiency usage already challenges our lives. As the human population and economy grows, so does our demand for energy. Some energy from fossil fuels (like Coal, Peat and Oil Shale) unfortunately are not replaceable energy, yet currently 80% global energy comes from burning fossil fuels. An estimation for fossil fuel reserves is only 50-120 years. In addition, burning fossil fuels product more carbon dioxide, which contributes to global warming. NASA Earth observatory predicted the global warming for next century is at least 20 times faster1.
Since then we found the impact of shortage energy for our lives, we already spent a decade to look for renewable energy and better solution for energy conservation.
Goals
We propose to analyze the changes in global energy usage (replacement and replacement energy), per capital use and energy trading flows to better understand the progress for energy efficiency usage. we will study what actions need to be taken to ensure that economic stability can be maintained in the future.
We will observe two largest population countries China and Indian energy consuming, since they are developing technology and economic in their highest speed. These two countries face more energy problems than other countries. In our research of data sets, we are going to discover what problem they are facing and what strategy they use.
To discover which countries are energy suppliers / consumer, what relationship they bound together.
Methods:
Based on Energy-Questionnaire-Guidelines, all category can be divided into five energy produts: Coal_Peat_Oil.Shale, Oil, Nautal_Gas, Electricity_heat, Biofuels_Waste. Every subcategory for each product of all natural source are listed in the guidelines. We used join function to find the component of products by category from data set “all_energy_statistics” which can match the subcategpry for every defined product from data set “fields”.
To get data set for import/export trade data for these five products, we use ‘grepl’ fuction to look for the words “import” and “export” in the column “commodity_transaction”. We also check if NA exists in after joined tables. Then we found “Electricity & heat” is not exist in import and export trade. Therefore the following, we only look for the other fours products transation.
We fine the total electricity consumption for top ten countries and the 10-year growth in 2004-2014.From comparing how much changes. We also find 2014 consumption by households per capital in these countries. We use the data of population from Wiki website. We calculate per capital consumption by using total_electricity (KW, million)/population of the country.
Data Wrangling
Data Resource: kaggle.com2; wikipedia.org3;
Objects: The top 10 world economies: USA,China,Japan,Germany,UK,India,France,Brazil,Italy,Canada.
suppressMessages(suppressWarnings(library('tidyverse')))
suppressMessages(suppressWarnings(library('data.table')))
suppressMessages(suppressWarnings(library('dplyr')))
suppressMessages(suppressWarnings(library('ggplot2')))
suppressMessages(suppressWarnings(library('reshape2')))
suppressMessages(suppressWarnings(library('scales')))
suppressMessages(suppressWarnings(library('rmarkdown')))
suppressMessages(suppressWarnings(library('rvest')))Data Overview: Raw “all_energy_statistics.csv” has 1048575 rows and 7 columns.
Step 1 : filted out top 10 countries from the large data set.
#use fread function to read in large table
raw<-fread('all_energy_statistics.csv')
head(raw)## country_or_area commodity_transaction year
## 1: Austria Additives and Oxygenates - Exports 1996
## 2: Austria Additives and Oxygenates - Exports 1995
## 3: Belgium Additives and Oxygenates - Exports 2014
## 4: Belgium Additives and Oxygenates - Exports 2013
## 5: Belgium Additives and Oxygenates - Exports 2012
## 6: Belgium Additives and Oxygenates - Exports 2011
## unit quantity quantity_footnotes
## 1: Metric tons, thousand 5 NA
## 2: Metric tons, thousand 17 NA
## 3: Metric tons, thousand 0 NA
## 4: Metric tons, thousand 0 NA
## 5: Metric tons, thousand 35 NA
## 6: Metric tons, thousand 25 NA
## category
## 1: additives_and_oxygenates
## 2: additives_and_oxygenates
## 3: additives_and_oxygenates
## 4: additives_and_oxygenates
## 5: additives_and_oxygenates
## 6: additives_and_oxygenates#eleminate low column 'footnote' for the survey
allEnergy<-raw[,-6]
#reduce the duplicated discription in 'commodity_transaction' column
allEnergy$commodity_transaction<-gsub(".*-", "", allEnergy$commodity_transaction)
#rename 'commodity_transaction' column as 'transaction'
names(allEnergy)[names(allEnergy) == 'commodity_transaction'] <- 'transaction'
#lowcase for the 'transaction' column
allEnergy$transaction <- tolower(allEnergy$transaction)
#convert dbl type to integer type for 'quantity'
allEnergy$quantity<-as.integer(allEnergy$quantity)## Warning: NAs introduced by coercion to integer range#select top 10 economic countries
subEnergy<- allEnergy[allEnergy$country_or_area %in% c("United States","China","Japan","Germany","United Kingdom","India","France","Brazil","Italy","Canada"), ]Step 2 : Coal_Peat_Oil.Shale, Oil, Nautal_Gas, Electricity_heat and Biofuels_Waste Charts -clisified energy products into five productes following Energy-Questionnaire-Guidelines.
fields<-fread('fields.csv')
head(fields)## Coal_Peat_Oil.Shale Oil
## 1: hard_coal conventional_crude_oil
## 2: anthracite natural_gas_liquids
## 3: coking_coal additives_and_oxygenates
## 4: other_bituminius_coal other_hydrocarbons
## 5: brown_coal aviation_gasoline
## 6: sub_bituminous_coal motor_gasoline
## Nautal Gas
## 1: natural_gas_including_lng
## 2: gasworks_gas
## 3: blast_furnace_gas
## 4: other_recovered_gases
## 5:
## 6:
## Electricity & heat
## 1: electricity_net_installed_capacity_of_electric_power_plants
## 2: heat
## 3: direct_use of_geothermal_heat
## 4: direct_use_of_solar_thermal_heat
## 5:
## 6:
## Biofuels & Waste
## 1: fuelwood
## 2: charcoal
## 3: bagasse
## 4: animal_ waste
## 5: other_vegetal_material_and_residues
## 6: municipal_wasteCoal_Peat_Oil.Shale<-fields[,1]
Oil<-fields[,2]
Nautal_Gas<-fields[,3]
Electricity_heat<-fields[,4]
Biofuels_Waste<-fields[,5]
CPO<-left_join(Coal_Peat_Oil.Shale,subEnergy, by = c("Coal_Peat_Oil.Shale"="category"))
OIL<-left_join(Oil,subEnergy,by = c("Oil"="category"))
NG<-left_join(Nautal_Gas, subEnergy,by = c("Nautal Gas"="category"))
EH<-left_join(Electricity_heat,subEnergy, by = c("Electricity & heat"="category"))
BW<-left_join(Biofuels_Waste,subEnergy, by = c("Biofuels & Waste"="category"))
impCPO<-CPO[grepl("imports",CPO[,3])==TRUE,]
impOIL<-OIL[grepl("imports",OIL[,3])==TRUE,]
impNG<-NG[grepl("imports",NG[,3])==TRUE,]
impBW<-BW[grepl("imports",BW[,3])==TRUE,]
expCPO<-CPO[grepl("exports",CPO[,3])==TRUE,]
expOIL<-OIL[grepl("exports",OIL[,3])==TRUE,]
expNG<-NG[grepl("exports",NG[,3])==TRUE,]
expBW<-BW[grepl("exports",BW[,3])==TRUE,]What energy products are!
Non-renewable resources
Renewable resources
Import & export Net Trade
Total Coal_Peat_Oil Net Trade (Export-Import):
Japan was the biggest buyer before 2011, with steady grew from 190,000 thousand metric tons in 1990, and had max import 250,000 in 2004, became constant after. China was exporting the product before 2008 and changed grew direction in a sudden. China became the largest buyer within four years, with max import above 300,000 2013. The following is India, third largest buyer showing strong grew in import after 2010. United States and Canada kept exporting all the time during 1990-2014. United States was the biggest seller in most of the time. Yet it reduced export amount from 1991, it was still the major supplier. After 2006, United States started to enlarge its export amount. Second suppler Canada kept 30,000-50,000 thousands metric tons per year, with steady export amount all the time. Germany, France, Italy and Brazil had constant small import each year.
impCPO.country<-impCPO %>% group_by(country_or_area,year) %>% summarise(totalImport=sum(quantity))
expCPO.country<-expCPO %>% group_by(country_or_area,year) %>% summarise(totalExport=sum(quantity))
netCPO<-merge(impCPO.country,expCPO.country, c("country_or_area","year"))
netCPO<-mutate(netCPO,netTransation=totalExport - totalImport)## Warning: package 'bindrcpp' was built under R version 3.3.3ggplot(data=netCPO, aes(x=year, y=netTransation, group = country_or_area, colour = country_or_area)) +
geom_line() +
geom_point( size=1, shape=10, fill="white")+
ggtitle("Total Coal_Peat_Oil.Shale net trade (Export-Import), Metric tons,thousand") +
scale_y_continuous(labels = comma)
Total Oil Net Trade (Export-Import):
United States is the largest buyer, kept grew amount from 360,000 thousand metric tons in 2009 and reached above 600,000 in 2005, then reduced the amount each year, yet it kept its first place until to 2014. Japan is the second largest buyer from 1990-2008, with steady import all the time, and slightly reduced the amount after 2010. China now becomes the biggest buyer in 2014, surpassed United States in 2014, and Japan in 2008. China had almost zero import before 1996, and had kept strong grew after. China imported 300,000 thousand metric tons oil in 2014. India also showed upward trend in importing more oil after 2005, from 200,000. Germany, Italy, France had constant import at around 100,000 per year, but started to reduce the amount after 2006. Canada is the only one country kept exporting oil at all the time, with small grew every year, and tended to increase the amount after 2010. United Kingdom kept selling oil before 2005, and became buyer after.
impOIL.country<-impOIL %>% group_by(country_or_area,year) %>% summarise(totalImport=sum(quantity))
expOIL.country<-expOIL %>% group_by(country_or_area,year) %>% summarise(totalExport=sum(quantity))
netOIL<-merge(impOIL.country,expOIL.country, c("country_or_area","year"))
netOIL<-mutate(netOIL,netTransation=totalExport - totalImport)
ggplot(data=netOIL, aes(x=year, y=netTransation, group = country_or_area, colour = country_or_area)) +
geom_line() +
geom_point( size=1, shape=10, fill="white")+
ggtitle("Total Oil net trade (Export-Import), Metric tons, thousand")+
scale_y_continuous(labels = comma)
Total Nautal Gas Net Trade (Export-Import):
United States is the biggest buyer from 1992-2013 with max import over 4,000,000 Terajoules in 2007 then reduced to 1,200,000 in 2014 ranking at fourth. Germany is the second largest buyer in most of time, yet its constantly import brought to the first place in 2014. Right after Germany, Italy had smooth grew before 2008 and surpassed united states in 2010. France had pretty steady grew and ranked at fourth before 2013. China showed very strong grew after 2008 which ranked at third in 2014. Similar to China, United Kingdom had enlarged their import from 2005. Japan did not have data in natural gas net trade. Canada is not only the biggest supplier, also the only country export Natural gas.
impNG.country<-impNG %>% group_by(country_or_area,year) %>% summarise(totalImport=sum(quantity))
expNG.country<-expNG %>% group_by(country_or_area,year) %>% summarise(totalExport=sum(quantity))
netNG<-merge(impNG.country,expNG.country, c("country_or_area","year"))
netNG<-mutate(netNG,netTransation=totalExport - totalImport)
ggplot(data=netNG, aes(x=year, y=netTransation, group = country_or_area, colour = country_or_area)) +
geom_line() +
geom_point( size=1, shape=10, fill="white")+
ggtitle("Total Natural Gas net trade (Export-Import), Terajoules")+
scale_y_continuous(labels = comma)
Total Biofuels & Waste Net Trade (Export-Import):
There are three forms import sources in this fields, United Kingdom took a lot import started from 2009 from 15,000 Terajoules. It grew with almost vertical speed, imported 70,000 Terajoules in 2014. Italy started to import biofuels and waste energy from 1998, and went up to 8,000 thousand cubic meters in 2014. Japan was out of this business. Others also had import or export, but the quantity of the trades are too small to be concerned.
impBW.country<-impBW %>% group_by(unit,country_or_area,year) %>% summarise(totalImport=sum(quantity))
expBW.country<-expBW %>% group_by(unit,country_or_area,year) %>% summarise(totalExport=sum(quantity))
netBW<-merge(impBW.country,expBW.country, c("country_or_area","year","unit"))
netBW<-mutate(netBW,netTransation=totalExport - totalImport)
ggplot(data=netBW, aes(x=year, y=netTransation, group = country_or_area, colour = country_or_area)) +
geom_line() +
geom_point( size=1, shape=10, fill="white")+
ggtitle("Total Biofuels & Waste net trade (Export-Import)")+
facet_grid(. ~ unit) +theme(axis.text.x = element_text(angle = 90, hjust = 1))+
scale_y_continuous(labels = comma)
Electricity Consumption (hourseholds)
2014 vs 2004 total electricity consumption for households
For 2014 the total electricity consumption for households’ data set, United States ranked at the first place with 1,416,977 million Kilowatt-hours and 10% grow in 10 years. United States almost double the total amount of the second country China, which had 717, 610 consumptions million Kilowatt-hours and two-time growth in past 10 years. The following is Japan which had 273,938 million Kilowatt-hours and constant consumption in 10 years. India ranked at fourth with 217,405 million Kilowatt-hours and 1.27 times growth in past 10 years. Canada is at fifth with 161,568 million Kilowatt-hours and 7% growth. France is at sixth with 149,426 million Kilowatt-hours and 4% growth. Brazil is at seventh with 132,302 million Kilowatt-hours and 68% the third fastest growth in past 10 years. The next three countries ranking in order are Germany, United Kingdom and Italy, with 129,600, 108,881 and 64,255 separately. These three countries reduced the total electricity consumption for households in past 10 years, with 8%, 12% and 4% dropped in usage.
Data Table [Note:Order by 2014_quantity ]
electricity2004<-subEnergy %>% filter(year=="2004") %>% filter(category=="total_electricity")
electricity2004<-electricity2004[grepl("households",electricity2004[,2])==TRUE,]
electricity2004<-electricity2004[order(electricity2004$quantity,decreasing = TRUE),]
electricity2004<-electricity2004[order(electricity2004$quantity,decreasing = TRUE),]
electricity2014<-subEnergy %>% filter(year=="2014") %>% filter(category=="total_electricity")
electricity2014<-electricity2014[grepl("households",electricity2014[,2])==TRUE,]
electricity2014<-electricity2014[order(electricity2014$quantity,decreasing = TRUE),]
electricity2014<-electricity2014[order(electricity2014$quantity,decreasing = TRUE),]
e<-merge(electricity2014,electricity2004, c("country_or_area"))
e<-data.frame(e$country_or_area,e$quantity.x,e$quantity.y,e$unit.x,e$transaction.x)
colnames(e)<- c("country_or_area","2014_quantity","2004_quantity","unit","transation")
#add new column "growth rate"
e["2004-2014growth"] <-round((e$`2014_quantity`-e$`2004_quantity`)/e$`2004_quantity`,2)
e<-e[order(e$`2014_quantity`,decreasing = TRUE),]
e## country_or_area 2014_quantity 2004_quantity unit
## 10 United States 1416977 1291984 Kilowatt-hours, million
## 3 China 717610 238449 Kilowatt-hours, million
## 8 Japan 273938 273923 Kilowatt-hours, million
## 6 India 217405 95660 Kilowatt-hours, million
## 2 Canada 161568 150978 Kilowatt-hours, million
## 4 France 149426 143380 Kilowatt-hours, million
## 1 Brazil 132302 78577 Kilowatt-hours, million
## 5 Germany 129600 140400 Kilowatt-hours, million
## 9 United Kingdom 108881 124200 Kilowatt-hours, million
## 7 Italy 64255 66592 Kilowatt-hours, million
## transation 2004-2014growth
## 10 consumption by households 0.10
## 3 consumption by households 2.01
## 8 consumption by households 0.00
## 6 consumption by households 1.27
## 2 consumption by households 0.07
## 4 consumption by households 0.04
## 1 consumption by households 0.68
## 5 consumption by households -0.08
## 9 consumption by households -0.12
## 7 consumption by households -0.04Electricity Consumption (hourseholds) 2004-2014 growth
ggplot(data=e, aes(x=reorder(country_or_area, -`2004-2014growth`), y=`2004-2014growth`,fill=country_or_area)) +
geom_bar(stat="identity")+
geom_text(aes(label=`2004-2014growth`), vjust=-1, color="black", size=3)+
ggtitle("Electricity Consumption by hourseholds 2004-2014 growth rate") +
xlab("country_or_area)") + ylab("2004-2014growth")
Per_Capital_Use(Kilowatt-hours)
For the graph of Per_Capital_Use(Kilowatt-hours), Canada and United States are at first and second places with very close amount of usage per capital. Their usage are close double amount of France and Japan usage per capital. China and India are at ninth and tenth, only 11.7% and 3.7% of the usage per capital in Canada.
url<-("https://en.wikipedia.org/wiki/List_of_countries_and_dependencies_by_population")
temp <- url %>%
html %>%
html_nodes("table")## Warning: 'html' is deprecated.
## Use 'read_html' instead.
## See help("Deprecated")population<-html_table(temp[1])
#formate 'population' table in r
population<-data.frame(population)
#rename 'Country (or dependent territory)' column as 'Country'
colnames(population)[2] <- "Country"
#remove [note] from country names in 'Country' column
population$Country <-gsub("\\[.*","",population$Country)
#remove "," from 'Population' column string, then covert char to interger data type
population$Population<-gsub("[^0-9]", "", population$Population)
population$Population<-as.integer(as.character(population$Population))
population10<- population[population$Country %in% c("United States","China","Japan","Germany","United Kingdom","India","France","Brazil","Italy","Canada"), ]
population10<-data.frame(population10$Country,population10$Population)
#rename all column names
colnames(population10) <- c("country_or_area","Population")
p<-merge(electricity2014,population10, c("country_or_area"))
#add new column "Per_Capital_Use"
p["Per_Capital_KW_hr"] <-round(p$quantity/p$Population*1000000,2)
p<-p[order(p$quantity,decreasing = TRUE),]
ggplot(data=p, aes(x=reorder(country_or_area, Per_Capital_KW_hr), y=Per_Capital_KW_hr,fill=country_or_area)) +
geom_bar(stat="identity")+
geom_text(aes(label=Per_Capital_KW_hr), vjust=-1, color="black", size=3)+
ggtitle("Electricity Consumption by hourseholds per capital") +
xlab("Per_Capital_Use(Kilowatt-hours)") + ylab("country")+
coord_flip()
Conclusion
Export & Import Trade Conclusion
In North America, US and Canada are the world largest energy consumers and suppliers in different products in most of the time of past 10 years. US and Canada both exported a lot of coal_peat_oil.shale. For oil and natural gas, Canada and United States were the best partner, which were the largest seller and buyer respectively in most time.
In Asian continents, China, Japan and India had dramatically enlarged the import of unreplacble energy source coal_peat_oil.shale and oil products in post 10 years, in which China and India largely contributed in the growth. Japan always kept the constant grew speed in energy trade. China also showed aggressively import more natural gas from 2008, yet Japan and India were not in natural gas trade. And Asian countries were not that much involved in trading renewable energy Biofuels and Waste.
UE had opposite reaction from Asia in energy trade. For unrenewable energy products, France, Germany, Italy and UK had kept the stable low quantity during the period. They started to import more renewable energy product biofuels and waste product in the past, thought the quantity was incomparable small to the unrenewable energy. United Kingdom showed very aggressive grew in renewable energy product starting from 2009. Italy also had slow grown as well.
Among top 10 countries, Brazil had very low active in energy trade. The quantities of coal_peat_oil.shale. , oil and biofuels and waste product had insignificant small.
Electricity Consumption by households Conclusion
From the grow rate in last decade, China and India were not surprise in ranking first and second growth in Electricity Consumption, and this reflected to their large import of coal, oil and gas. However, Brazil was surprise ranked at third without much trade in energy products. United States, Canada and France showed small grew in electricity Consumption. And Japan and Italy, and Germany seemed to control and reduced electricity consumption. From the households per capital consumption, Canada and United States were the highest amounts. Brazil, China and India were at the bottom with around 17% of Canada and United States per capital, because of the large population share in domestics.