This page demonstrates getting data from web using an Web API using an R program. New York Times provides web APIs that provide various content from nytimes.com. The Top Stories API is used in this program to fetch the top stories from NYT.
Three functions are developed to handle the Top Stories API.
Function: top_stories_uri
The function top_stories_uri constructs the URI for the top stories API given the section name and apikey
#Function to construct URI
top_stories_uri=function(section,apikey,response_format=".json"){
paste0("http://api.nytimes.com/svc/topstories/v1/",section,response_format,"?api-key=",apikey)
}Function:process_top_stories*
Function process_top_stories takes the json response from the top stories API and formats it into a dataframe
#Process json response
process_top_stories=function(response){
library(jsonlite)
#Convert json to nested dataframe
response_dfs=fromJSON(response)
#Get result count
count = response_dfs$num_results
#Initialize output data frame
topstories=data.frame(matrix(numeric(0), nrow = count))
results=response_dfs$results
topstories$Title=results$title
topstories$Abstract=results$abstract
topstories$url=results$url
topstories$By=results$byline
topstories$Tags=results$org_facet
#return output dataframe
topstories
}Function: get_nyt_top_stories
This is the main function that act as the interface for the NYT top stories API. This function invokes the API for the given topic and returns formatted dataframe. This makes use of the other two functions mentioned above.
get_nyt_top_stories=function(section,apikey){
library(RCurl)
raw_response=""
#Construct the URI for the request
uri=top_stories_uri(section,apikey)
raw_response=getURI(uri)
#The GET failed, could be bad URI or bad connection
if (raw_response==""){
print("Error:Get Failed")
return
}
#Able to get response, but response indicates it's error
if (substr(raw_response,3,7)=="error"){
print("Warning: Error response received")
return
}
#Process the json response
process_top_stories(raw_response)
}Let’s try to get the top stories
#Read api key
apikey=as.character(read.table("nytapikey.txt",header = F,stringsAsFactors = F))
#Get top stories
topstories=get_nyt_top_stories("home",apikey)
library(knitr)
#display top stories
kable(topstories,caption="NYT Top Stories")| Title | Abstract | url | By | Tags |
|---|---|---|---|---|
| Two Corruption Cases Will Put Culture of Albany on Trial | The trials of Assemblyman Sheldon Silver, the former New York State speaker, and Senator Dean G. Skelos, the former majority leader, will focus attention on the crisis of corruption in the state capital. | http://www.nytimes.com/2015/11/02/nyregion/in-two-corruption-cases-the-culture-of-albany-will-go-on-trial.html | By WILLIAM K. RASHBAUM and SUSANNE CRAIG | |
| In Arbitration, a ‘Privatization of the Justice System’ | Forced individual arbitration often bears little resemblance to court: The rules tend to favor businesses, and judges and juries are replaced by arbitrators whose rulings are difficult to appeal. | http://www.nytimes.com/2015/11/02/business/dealbook/in-arbitration-a-privatization-of-the-justice-system.html | By JESSICA SILVER-GREENBERG and MICHAEL CORKERY | American Arbitration Assn |
| Part I: Arbitration Everywhere | With a clause in complex contracts that few people read, corporations have insulated themselves from lawsuits and locked Americans into a system where arbitrators overwhelmingly favor business. | http://www.nytimes.com/2015/11/01/business/dealbook/arbitration-everywhere-stacking-the-deck-of-justice.html | By JESSICA SILVER-GREENBERG and ROBERT GEBELOFF | |
| Thousands Start Life Anew With Early Prison Release | Over the next few years, as a result of an across-the-board adjustment of federal drug penalties, tens of thousands more inmates may benefit from reductions in their terms. | http://www.nytimes.com/2015/11/02/us/with-early-release-thousands-of-inmates-are-adjusting-to-freedom.html | By ERIK ECKHOLM | United States Sentencing Commission |
| F.B.I. Program to Identify Extremists Faces Criticism | An interactive program, developed for teachers and students, is aimed at training them to prevent young people from becoming radicalized, however some religious leaders say it only focuses on Islamic extremism. | http://www.nytimes.com/2015/11/02/us/fbi-tool-to-identify-extremists-is-criticized.html | By LAURIE GOODSTEIN | Federal Bureau of Investigation, Muslim Public Affairs Council, American-Arab Anti-Discrimination Committee |
| Anger Over Tech Divide Infuses a Ballot | On Tuesday, San Francisco voters will weigh in on propositions linked to the technology industry and rising housing costs, including one to restrict Airbnb. | http://www.nytimes.com/2015/11/02/technology/san-francisco-ballots-turn-up-anger-over-the-technical-divide.html | By CONOR DOUGHERTY | Airbnb, San Francisco Board of Supervisors |
| Your Asia Briefing | We’re trying something new for our Asia readers. Email us at briefing@nytimes.com and tell us if you like it. | http://www.nytimes.com/2015/11/02/nytnow/your-asia-briefing.html | By VICTORIA SHANNON | |
| Fred Thompson, Former Senator and Actor, Dies at 73 | Mr. Thompson, a former Republican lawmaker from Tennessee who ran for president, had an unusual career that bridged the worlds of national politics and mass-market entertainment. | http://www.nytimes.com/2015/11/02/us/politics/fred-thompson-former-senator-actor-and-presidential-candidate-dies-at-73.html | By LIAM STACK | Senate, Republican Party |
| Thompson Took Own Path in the Senate (Sept. 30, 2007) | In his eight years in the Senate, Fred D. Thompson displayed little enthusiasm for the divisive battles that motivate religious Republican primary voters. | http://www.nytimes.com/2007/09/30/us/politics/30thompson.html | By JO BECKER | |
| If Cancer Becomes Biden’s Cause, Bold Doctor Is On Call | Dr. Patrick Soon-Shiong, who made a fortune developing a cancer drug, has developed a relationship with Vice President Joseph R. Biden Jr. as he weighs life out of office. | http://www.nytimes.com/2015/11/02/us/politics/if-cancer-becomes-bidens-cause-a-bold-but-polarizing-doctor-is-on-call.html | By PETER BAKER | |
| G.O.P. Campaigns Meet in Bid to Alter Debates | In the wake of complaints about the questions by CNBC’s moderators, the goal is to develop a strategy on how to exert more influence on the process. | http://www.nytimes.com/2015/11/02/us/politics/republican-campaigns-meet-in-an-effort-to-alter-debates.html | By ASHLEY PARKER | CNBC, Fox Business Network, Republican Party |
| Cruz and Rubio Grow Apart as Ambitions Expand | Ted Cruz and Marco Rubio brought similar backgrounds and Tea Party fervor to Washington, but any alliance between them frayed as they moved closer to seeking the White House. | http://www.nytimes.com/2015/11/02/us/politics/ted-cruz-and-marco-rubio-grow-apart-as-their-ambitions-expand.html | By JASON HOROWITZ | Republican Party |
| Ryan Says He Won’t Work With Obama on Immigration | In a series of interviews televised on Sunday, the new House speaker signaled an antagonism that is likely to appeal to the conservatives within the Republican Party. | http://www.nytimes.com/2015/11/02/us/politics/paul-ryan-says-he-wont-work-with-obama-on-immigration-reform.html | By NICHOLAS FANDOS | Republican Party, House of Representatives |
| Debris Suggests Russian Plane Disintegrated in Air | The potential explanations for the fatal Egyptian Sinai crash include several possibilities, from a sudden mechanical or structural failure to a terrorist attack. | http://www.nytimes.com/2015/11/02/world/europe/russian-airliner-evidently-disintegrated-while-aloft-officials-say.html | By NICOLA CLARK | Airbus Industrie, Kogalymavia |
| Erdogan’s Islamist Party in Turkey Regains Majority | The results of the parliamentary election represent a stunning electoral comeback for President Recep Tayyip Erdogan, and ensure another period of single party government in Turkey. | http://www.nytimes.com/2015/11/02/world/europe/turkey-elections-erdogan.html | By TIM ARANGO and CEYLAN YEGINSU | Islamic State in Iraq and Syria (ISIS), Justice and Development Party (Turkey) |
| Ohio Grapples With Specter of Marijuana Monopoly | Longtime marijuana advocates are conflicted over a legalization measure on the ballot that would grant exclusive rights to growing it commercially to a group of wealthy investors. | http://www.nytimes.com/2015/11/02/us/on-ballot-ohio-grapples-with-specter-of-marijuana-monopoly.html | By MITCH SMITH and SHERYL GAY STOLBERG | Drug Policy Alliance |
| Willis Carto, Far-Right Holocaust Denier, Dies at 89 | The Anti-Defamation League called Mr. Carto “one of the most influential American anti-Semitic propagandists” and “the mastermind of the hate network.” | http://www.nytimes.com/2015/11/02/us/willis-carto-far-right-figure-and-holocaust-denier-dies-at-89.html | By DOUGLAS MARTIN | |
| Militants Kill at Least 10 at Popular Hotel in Somalia | The Shabab, an Islamist group, claimed responsibility after fighters killed at least 14 people at the Sahafi Hotel in Mogadishu. | http://www.nytimes.com/2015/11/02/world/africa/-2015-11-02-world-africa-sahafi-shabab-militants-attack-hotel-somalia.html | By MOHAMMED IBRAHIM and JEFFREY GETTLEMAN | Shabab, African Union, Al Qaeda |
| Officers’ Boycott of Tarantino Films Grows | A boycott of “The Hateful Eight” and other works by Quentin Tarantino picked up strength with the endorsement by the National Association of Police Organizations. | http://www.nytimes.com/2015/11/02/business/media/officers-boycott-of-tarantino-films-grows.html | By MICHAEL CIEPLY | Weinstein Co, National Assn of Police Organizations |
| Review: In ‘King Charles III,’ Monarchy’s Future | Mike Bartlett’s prize-winning play comes to Broadway from London with its cast, Shakespearean stylings and free verse intact. | http://www.nytimes.com/2015/11/02/theater/review-in-king-charles-iii-glimpsing-the-near-future-of-monarchy.html | By BEN BRANTLEY |
Let’s check the sports stories..
sportstories=get_nyt_top_stories("sports",apikey)
kable(sportstories)| Title | Abstract | url | By | Tags |
|---|---|---|---|---|
| When Royals’ Wade Davis Comes In, the Game Is as Good as Over | Like the Giants’ Madison Bumgarner the year before, Wade Davis has been the immovable force of this postseason. | http://www.nytimes.com/2015/11/02/sports/baseball/when-royals-wade-davis-comes-in-the-game-is-as-good-as-over.html | By TYLER KEPNER | Kansas City Royals, New York Mets |
| No Stranger to Scrutiny, Terry Collins Keeps Perspective | The crushing loss in Game 4 did not appear to bother the Mets on Sunday, and Manager Terry Collins seemed unperturbed by criticism over his use of the bullpen. | http://www.nytimes.com/2015/11/02/sports/baseball/no-stranger-to-scrutiny-terry-collins-keeps-perspective.html | By TIM ROHAN | New York Mets, Kansas City Royals |
| Drew Brees Throws Seven Touchdown Passes as Saints Beat Giants in Final Seconds | In a high-octane game, Drew Brees tied the N.F.L. single-game record for touchdown passes, and he and Eli Manning combined for 13 touchdown passes, an N.F.L. single-game record. | http://www.nytimes.com/2015/11/02/sports/drew-brees-throws-seven-touchdown-passes-as-new-orleans-saints-beat-new-york-giants.html | By BILL PENNINGTON | New Orleans Saints, New York Giants |
| Ryan Fitzpatrick Is Injured and the Jets Are Routed by the Raiders | A promising season veers toward uncertainty after a second straight loss. | http://www.nytimes.com/2015/11/02/sports/football/oakland-raiders-beat-new-york-jets-ryan-fitzpatrick-injury.html | By BEN SHPIGEL | New York Jets, Oakland Raiders |
| Stanley Biwott of Kenya Wins New York City Marathon Men’s Race | With a burst of late speed, including running the 21st mile at a stunning 4:24, Biwott captured his first major marathon title. | http://www.nytimes.com/2015/11/02/sports/stanley-biwott-of-kenya-wins-new-york-city-marathon-mens-race.html | By JERÉ LONGMAN | |
| A Dazzling Encore for Mary Keitany, and a Promising Debut for Laura Thweatt | Mary Keitany of Kenya was the first woman to finish the New York City Marathon for the second straight year, while Laura Thweatt was the top American woman, finishing seventh in her marathon debut. | http://www.nytimes.com/2015/11/02/sports/mary-keitany-of-kenya-wins-another-new-york-city-marathon.html | By LINDSAY CROUSE | |
| These Competitors Measure Their Times in Years | Those who have run the New York City Marathon again and again and again are known as streakers, and there are some rivalries among them over their longevity. | http://www.nytimes.com/2015/11/02/sports/these-marathoners-cant-get-enough-of-new-york.html | By JULIET MACUR | New York Road Runners Club |
| Getting to the Start on Time: The Race Before the Race | For many marathoners, an early morning ride on the Staten Island Ferry is one of the miniraces that precede the 26.2-mile one. | http://www.nytimes.com/2015/11/02/sports/the-marathon-before-the-marathon-getting-to-the-start-on-time.html | By SETH BERKMAN | Staten Island Ferry |
| David Walters, 60, Runs to Beat Father Time, if Not His Own Time | Walters, who maintains a grueling training schedule despite knowing his best times are behind him, completed the New York City Marathon in 2:47:27. | http://www.nytimes.com/2015/11/02/sports/david-walters-60-runs-to-beat-father-time-if-not-his-own-time.html | By LINDSAY CROUSE | |
| Lightweight Blankets With a Big Footprint at the Marathon | The three-ounce metalized blankets for New York City Marathon finishers are designed to help ward off hypothermia, but are also marketing vehicles and keepsakes. | http://www.nytimes.com/2015/11/02/sports/lightweight-blankets-with-a-big-footprint-at-the-marathon.html | By SCOTT CACCIOLA | |
| Well-Known and Worn-Out Marathoners | The actor Ethan Hawke finished in 4 hours 25 minutes. | http://www.nytimes.com/2015/11/02/sports/well-known-and-worn-out-marathoners-ethan-hawke.html | ||
| For Royals’ Edinson Volquez, a Bittersweet Start in His Father’s Memory | Volquez prepared to take the mound for Game 5 of the World Series just days after his father’s death. | http://www.nytimes.com/2015/11/02/sports/baseball/for-royals-edinson-volquez-a-bittersweet-start-in-his-fathers-memory.html | By DAVID WALDSTEIN | Kansas City Royals |
| Daniel Murphy Hit Them High and Far, but Couldn’t Reach Down Low | For three weeks, Murphy, a baseball scrapper, soared through the playoffs. Then came the eighth inning of Game 4 of the World Series. | http://www.nytimes.com/2015/11/02/sports/baseball/daniel-murphy-hit-them-high-and-far-but-couldnt-reach-down-low.html | By MICHAEL POWELL | |
| Alex Rodriguez Is an October Surprise | The Yankees star continues his surprisingly strong season, only now as a World Series analyst. | http://www.nytimes.com/2015/11/02/sports/baseball/alex-rodriguez-is-an-october-surprise.html | By RICHARD SANDOMIR | Fox Sports |
| A Crushing Error, and Now a Daunting Hole | Daniel Murphy’s defensive miscue helped the Kansas City Royals take a 3-1 series lead against the Mets, but there were plenty of failures around him as well. | http://www.nytimes.com/2015/11/01/sports/baseball/after-daniel-murphys-error-mets-need-historic-comeback.html | By TYLER KEPNER | New York Mets, Kansas City Royals |
| Royals Aren’t Perfect, but Opportunistic Does Just Fine | Alex Rios made a mental blunder that helped the Mets pad their lead early in Game 4, but his teammates, as usual, waited for their chance and made it count. | http://www.nytimes.com/2015/11/01/sports/baseball/royals-arent-perfect-but-opportunistic-does-just.html | By BILLY WITZ | New York Mets, Kansas City Royals |
| Pay $740 for SRO at the World Series? ‘Ya Gotta Believe’ | Fans at Game 3 snapped up standing-room-only tickets that allowed them to soak up the atmosphere even if their views were sometimes obstructed and their legs grew tired. | http://www.nytimes.com/2015/11/01/sports/baseball/pay-740-for-sro-at-the-world-series-ya-gotta-believe.html | By BILLY WITZ | Citi Field (Queens, NY), Kansas City Royals, New York Mets |
| The Seahawks-Cowboys Matchup Should Have Been Great. It Wasn’t. | Both teams were N.F.C. favorites when the schedule was released, but distractions and injuries have made them mediocre. | http://www.nytimes.com/2015/11/02/sports/football/seattle-seahawks-beat-dallas-cowboys.html | By BENJAMIN HOFFMAN | Dallas Cowboys, Seattle Seahawks |
| Bengals Reach 7-0, Topping Ben Roethlisberger and Steelers | Cincinnati capitalized on two fourth-quarter interceptions by Roethlisberger in his return from a knee injury, rallying past Pittsburgh to secure its best start in franchise history. | http://www.nytimes.com/2015/11/02/sports/football/bengals-reach-7-0-topping-ben-roethlisberger-and-steelers.html | By THE ASSOCIATED PRESS | Cincinnati Bengals, Pittsburgh Steelers |
| Le’Veon Bell Carted Off After Injuring Knee in Steelers’ Loss | Bell, the Steelers’ leading rusher, was injured after catching a pass from quarterback Ben Roethlisberger, who had just returned from an injury. | http://www.nytimes.com/2015/11/02/sports/football/pittsburgh-steelers-leveon-bell-injured-cincinnati-bengals.html | By BENJAMIN HOFFMAN | Cincinnati Bengals, Pittsburgh Steelers |
| Monday’s Matchup: Colts (3-4) at Panthers (6-0) | This game should serve as a chance for the Panthers to show off just how improved their defense is this season. | http://www.nytimes.com/2015/11/02/sports/football/mondays-matchup-colts-3-4-at-panthers-6-0.html | By BENJAMIN HOFFMAN | Indianapolis Colts, Carolina Panthers |
| The Best of Week 8 in the N.F.L. | The top performers and key injuries in the league this week. | http://www.nytimes.com/2015/11/02/sports/football/the-best-of-week-8-in-the-nfl.html | Reported by THE NEW YORK TIMES | Houston Texans |
| Adulation Aplenty as American Pharoah Heads for the Stud Farm | Horse racing fans celebrated the Triple Crown winner on Sunday a day after he closed out his stellar career on the track. | http://www.nytimes.com/2015/11/02/sports/adulation-aplenty-as-american-pharoah-heads-for-the-stud-farm.html | By MELISSA HOPPERT | Ashford Stud |
| American Pharoah Ends Career Fittingly: Triumphant and Adored | It was the ninth victory in 11 career races for American Pharoah, the Triple Crown champion who will now retire. | http://www.nytimes.com/2015/11/01/sports/american-pharoah-ends-career-with-win-at-breeders-cup.html | By JOE DRAPE | |
| American Pharoah’s Final Triumph Wipes Away Effects of an Earlier Loss | American Pharoah’s victory in the Breeders’ Cup Classic at venerable Keeneland Racecourse capped his spectacular career and erased the memories of his defeat in the Travers Stakes in August. | http://www.nytimes.com/2015/11/01/sports/horse-racing/american-pharoahs-final-triumph-wipes-away-effects-of-a-startling-loss.html | By MELISSA HOPPERT | Keeneland Race Course |
| Fantastic Finishes, and a Domino Effect on the Rankings | Saturday’s results, although not directly involving most top teams, could still carry consequences for the College Football Playoff rankings, which come out Tuesday. | http://www.nytimes.com/2015/11/02/sports/-2015-11-02-sports-domino-effect-on-college-football-rankingshtml.html | By MARC TRACY | Clemson University, University of Notre Dame, University of Miami |
| A.C.C. Suspends Miami-Duke Officiating Crew for Allowing Final Play | After eight laterals, Miami’s Corn Elder rambled nearly the length of the field to beat Duke, but on Sunday the league said the game officials had made four errors. | http://www.nytimes.com/2015/11/02/sports/miami-duke-acc-suspends-officiating-crew.html | By VICTOR MATHER | Duke University, University of Miami |
| More Coaches Aren’t Lasting Out the Season | Seven midseason vacancies in the Football Bowl Subdivision is the same number as the previous three seasons combined. | http://www.nytimes.com/2015/11/01/sports/ncaafootball/more-coaches-arent-lasting-out-the-season.html | By MARC TRACY | |
| The Best Moments in College Football This Week | A look at some of the top story lines from the ninth week of the 2015 college football season. | http://www.nytimes.com/2015/11/01/sports/ncaafootball/the-best-moments-in-college-football-this-week.html | By THE NEW YORK TIMES | United States Navy |
| Patricia Canning Todd, Tennis Champion Who Refused to Play on Side Court, Is Dead at 93 | Todd, playing as a young mother, won four Grand Slam tennis tournaments in 1947 and 1948 but lost the chance for another in 1948 when she took a stand. | http://www.nytimes.com/2015/11/02/sports/tennis/patricia-canning-todd-tennis-champion-who-refused-to-play-on-side-court-is-dead-at-93.html | By DANIEL E. SLOTNIK |