Ultimate method of webscrapping is depicted here. We are going to sift thorugh the web scrapping process and extract the relevant data. Web scraping has a close relationship with data mining since it helps to extract various kinds of data for market research or any other reason where one needs a large quantity of data for further analysis. This topic is actually really fun! We’re often told how marketers are (or should be) creative, right?
Well, since you can do pretty much what you want with web scraping, you have carte blanche! Let’s say you have spotted one of your competitors on Instagram. And damn, they have a nice community of 15K+ followers . But you’re convinced your product is far better and that users could switch to you.
What do you do? You scrape! Find their Instagram page and start extracting every followers.With that list, you’ll be able to follow and DM them. It’s generally highly qualified since you know the profiles you collect are interested in what you do. As a matter of fact, you could do the same on Twitter or on any other social network.
The algorithm below is designed to scrap relevant information off the “Outland.no” website. The “Outland.no” is the norwegian-based retailer, selling books, toys and souvenirs to fans of science fiction and phantasy. The key libraries in the code are tidyverse, rvest and polite. The first 2 you should know well. If you want to scrape more than one page at a time — which you probably do — you should also install the polite library. This library will help you avoid overloading a site and potentially getting your web scraper blocked.
rm(list=ls())
library(rvest)
library(tidyverse)## ── Attaching packages ─────────────────────────────────────── tidyverse 1.3.2 ──
## ✔ ggplot2 3.4.0 ✔ purrr 0.3.5
## ✔ tibble 3.1.8 ✔ dplyr 1.0.10
## ✔ tidyr 1.2.1 ✔ stringr 1.4.1
## ✔ readr 2.1.3 ✔ forcats 0.5.2
## ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
## ✖ dplyr::filter() masks stats::filter()
## ✖ readr::guess_encoding() masks rvest::guess_encoding()
## ✖ dplyr::lag() masks stats::lag()library(polite)
titles <- c()
prices <- c()
sprachen <- c()
get_info <- function(page_number){
for (page_number in page_number){
#insert a website's url here
link <- paste0("https://www.outland.no/boker/fantasy?p=",page_number)
url <- bow(link, force=TRUE)
page <- scrape(url)
# collect the titles of the books
titles <-page%>%
html_nodes(".name .product-item-link")%>%
html_text(trim=TRUE)
# collect prices
prices <-page%>%
html_nodes(".special-price .price , .on-net-wrapper+ .weee .price")%>%
html_text(trim=TRUE)
# what language is the book
language <- page%>%
html_nodes(".language")%>%
html_text(trim=TRUE)
}
return(rbind(data.frame(titles,prices,language)))
}
# X is the number of pages to be scanned; apply the function we have created.
d <- lapply(X=1:5,FUN=get_info)
# compile the data here
data <- do.call(rbind.data.frame, d)
# clean the data: replace Språk with ""
data$language<- data$language%>%
gsub(pattern="Språk:", replacement = "")
library(gt)
# Final Output
data%>%
gt()%>%
gt:: tab_header(title="OUTLAND books")%>%
gt::cols_label(titles=md("**Titles**"),
prices=md("**Price**"),
language=md("**Language**")
)%>%
gt:: tab_caption(caption="Source:Outland.no")| OUTLAND books | ||
| Titles | Price | Language |
|---|---|---|
| The Sunbearer Trials | 135,15 kr | Engelsk |
| The Atlas Six | 135,15 kr | Engelsk |
| Fire & Blood (HBO Tie-in Edition): 300 Years Before A Game of Thrones | 101,15 kr | Engelsk |
| Ruination: A League of Legends Novel | 220,15 kr | Engelsk |
| The Atlas Paradox | 279,65 kr | Engelsk |
| The Atlas Paradox | 220,15 kr | Engelsk |
| Rule of Wolves (King of Scars Book 2) | 135,15 kr | Engelsk |
| Kingdom of the Feared | 220,15 kr | Engelsk |
| The Dragon's Promise: the Sunday Times bestselling magical sequel to Six Crimson Cranes | 254,15 kr | Engelsk |
| Six Crimson Cranes | 135,15 kr | Engelsk |
| Nona the Ninth | 254,15 kr | Engelsk |
| Babel: Or the Necessity of Violence: An Arcane History of the Oxford Translators' Revolution | 254,15 kr | Engelsk |
| Babel: Or the Necessity of Violence: An Arcane History of the Oxford Translators' Revolution | 220,15 kr | Engelsk |
| The Book of Accidents | 143,65 kr | Engelsk |
| She Who Became the Sun | 143,65 kr | Engelsk |
| The Atlas Six | 186,15 kr | Engelsk |
| Kingdom of the Wicked: TikTok made me buy it! The addictive and darkly romantic fantasy | 135,15 kr | Engelsk |
| The Shadow of the Gods | 143,65 kr | Engelsk |
| A Court of Thorns and Roses: The hottest fantasy sensation of 2022 | 135,15 kr | Engelsk |
| Fire and Blood: The inspiration for HBO's House of the Dragon (A Song of Ice and Fire) | 143,65 kr | Engelsk |
| The Priory of the Orange Tree: THE INTERNATIONAL SENSATION | 169,15 kr | Engelsk |
| The Poppy War (The Poppy War, Book 1) | 143,65 kr | Engelsk |
| The Silmarillion | 143,65 kr | Engelsk |
| The Fellowship of the Ring (The Lord of the Rings, Book 1) | 143,65 kr | Engelsk |
| Howl's Moving Castle | 126,65 kr | Engelsk |
| The Colour Of Magic: (Discworld Novel 1) | 109,65 kr | Engelsk |
| Throne of Glass | 135,15 kr | Engelsk |
| Coraline | 126,65 kr | Engelsk |
| Caraval: The mesmerising and magical Sunday Times bestseller | 135,15 kr | Engelsk |
| Critical Role: The Mighty Nein--The Nine Eyes of Lucien | 322,15 kr | Engelsk |
| The Whispering Dark | 254,15 kr | Engelsk |
| If You Could See the Sun | 220,15 kr | Engelsk |
| Can't Spell Treason Without Tea: A Cozy Fantasy Steeped with Love | 220,15 kr | Engelsk |
| Harry Potter og Føniksordenen - illustrert utgave | 499 kr | Norsk Bokmål |
| Slaughtered Gods | 143,65 kr | Engelsk |
| The Vermilion Emporium | 220,15 kr | Engelsk |
| Uncanny Times | 211,65 kr | Engelsk |
| The Voyage of the Forgotten | 279,65 kr | Engelsk |
| Big Bad: A Novel from the World of Buffy the Vampire Slayer | 305,15 kr | Engelsk |
| Blood of the Fold: Book Three of the Sword of Truth | 254,15 kr | Engelsk |
| Wizard's First Rule: Book One of the Sword of Truth | 245,65 kr | Engelsk |
| One Dark Window | 211,65 kr | Engelsk |
| A Forgery of Roses | 135,15 kr | Engelsk |
| The Dragon's Promise | 228,65 kr | Engelsk |
| Tread of Angels | 143,65 kr | Engelsk |
| Fjellkjempen Arkta | 249 kr | Norsk Bokmål |
| Midnattsbiblioteket | 399 kr | Norsk Bokmål |
| On Stranger Tides | 194,65 kr | Engelsk |
| Conan - Blood of the Serpent | 279,65 kr | Engelsk |
| Empire of the Vampire (Empire of the Vampire, Book 1) | 636,65 kr | Engelsk |
| Den uendelige historie | 349 kr | Norsk Bokmål |
| Season of Storms | 339,15 kr | Engelsk |
| The Lady of the Lake | 339,15 kr | Engelsk |
| Baptism of Fire | 339,15 kr | Engelsk |
| The Time of Contempt | 339,15 kr | Engelsk |
| City of Last Chances | 237,15 kr | Engelsk |
| What Hunts Inside the Shadows | 135,15 kr | Engelsk |
| The Citadel of Forgotten Myths | 254,15 kr | Engelsk |
| Legends & Lattes | 139 kr | Engelsk |
| Legends & Lattes | 254,15 kr | Engelsk |
| League of Dragons: Book Nine of Temeraire | 211,65 kr | Engelsk |
| Blood of Tyrants: Book Eight of Temeraire | 211,65 kr | Engelsk |
| Crucible of Gold: Book Seven of Temeraire | 211,65 kr | Engelsk |
| Celestial: An alternative history set at the height of the space race | 279,65 kr | Engelsk |
| Heart of the Sun Warrior (The Celestial Kingdom Duology, Book 2) | 211,65 kr | Engelsk |
| Heart of the Sun Warrior (The Celestial Kingdom Duology, Book 2) | 254,15 kr | Engelsk |
| A Game of Thrones: The Story Continues: The complete boxset of all 7 books (A Song of Ice and Fire) | 985,15 kr | Engelsk |
| The Stolen Heir: A Novel of Elfhame, from the author of The Folk of the Air series | 135,15 kr | Engelsk |
| The Silmarillion | 1486,65 kr | Engelsk |
| Jade Fire Gold | 135,15 kr | Engelsk |
| Girls of Fate and Fury | 135,15 kr | Engelsk |
| The Fall of Numenor: and Other Tales from the Second Age of Middle-earth | 1061,65 kr | Engelsk |
| The Fall of Numenor: and Other Tales from the Second Age of Middle-earth | 339,15 kr | Engelsk |
| Star Wars: Convergence | 288,15 kr | Engelsk |
| Loki: WICKED, VISCERAL, TRANSGRESSIVE: Norse gods as you've never seen them before | 220,15 kr | Engelsk |
| Loki: WICKED, VISCERAL, TRANSGRESSIVE: Norse gods as you've never seen them before | 254,15 kr | Engelsk |
| How To Succeed in Witchcraft | 220,15 kr | Engelsk |
| The Discord of Gods | 169,15 kr | Engelsk |
| The Stolen Heir: A Novel of Elfhame, from the author of The Folk of the Air series | 254,15 kr | Engelsk |
| Our Violent Ends: #1 New York Times Bestseller! | 135,15 kr | Engelsk |
| D&D Dungeon Academy vol. 2: Tourney of Terror | 109,65 kr | Engelsk |
| The Ivory Tomb: Rooks and Ruin, Book Three | 143,65 kr | Engelsk |
| The Gilded Cage: the thrilling, unputdownable sequel to The Prison Healer | 135,15 kr | Engelsk |
| The World We Make | 288,15 kr | Engelsk |
| The World We Make | 220,15 kr | Engelsk |
| The Depths | 220,15 kr | Engelsk |
| Soul of the Deep | 135,15 kr | Engelsk |
| Miss Peregrine's Museum of Wonders: An Indispensable Guide to the Dangers and Delights of the Peculiar World for the Instruction of New Arrivals | 220,15 kr | Engelsk |
| Illuminations: The Top 5 Sunday Times Bestseller | 288,15 kr | Engelsk |
| Illuminations: The Top 5 Sunday Times Bestseller | 220,15 kr | Engelsk |
| The Silmarillion | 636,65 kr | Engelsk |
| A Restless Truth | 220,15 kr | Engelsk |
| A Restless Truth | 279,65 kr | Engelsk |
| The Dark Talent: Alcatraz vs. the Evil Librarians | 109,65 kr | Engelsk |
| Wayward | 211,65 kr | Engelsk |
| The Rise of the Dragon: An Illustrated History of the Targaryen Dynasty, Volume One | 509,15 kr | Engelsk |
| The Rise of the Dragon: An Illustrated History of the Targaryen Dynasty | 509,15 kr | Engelsk |
| The Golden Enclaves: TikTok made me read it | 254,15 kr | Engelsk |
| Lucky Girl: How I Became A Horror Writer: A Krampus Story | 169,15 kr | Engelsk |
| Even Though I Knew the End | 169,15 kr | Engelsk |
| Djinn Falls in Love and Other Stories | 143,65 kr | Engelsk |
| A Dowry of Blood: THE GOTHIC SUNDAY TIMES BESTSELLER | 220,15 kr | Engelsk |
| Harry Potter and the Order of the Phoenix: Deluxe Illustrated Slipcase Edition | 2974,15 kr | Engelsk |
| Harry Potter and the Order of the Phoenix | 509,15 kr | Engelsk |
| Star Eater | 220,15 kr | Engelsk |
| Into the Riverlands | 169,15 kr | Engelsk |
| The Lost Metal: A Mistborn Novel | 313,65 kr | Engelsk |
| The Lost Metal: A Mistborn Novel | 254,15 kr | Engelsk |
| The Lost Metal: A Mistborn Novel | 339,15 kr | Engelsk |
| Master of Iron | 220,15 kr | Engelsk |
| Bloodmarked | 228,65 kr | Engelsk |
| Bloodmarked | 143,65 kr | Engelsk |
| A Venom Dark and Sweet | 220,15 kr | Engelsk |
| Charm | 228,65 kr | Engelsk |
| The White Wolf: The Elric Saga Part 3 | 373,15 kr | Engelsk |
| Tolkien Boxed Set | 959,65 kr | Engelsk |
| Master of One | 143,65 kr | Engelsk |
| Tress of the Emerald Sea | 254,15 kr | Engelsk |
| Tress of the Emerald Sea | 339,15 kr | Engelsk |
| The Rogue Crown (The Five Crowns of Okrith, Book 3) | 135,15 kr | Engelsk |
| The Witches' Blade (The Five Crowns of Okrith, Book 2) | 135,15 kr | Engelsk |
| The High Mountain Court (The Five Crowns of Okrith, Book 1) | 135,15 kr | Engelsk |
| The School for Good and Evil (The School for Good and Evil, Book 1) | 254,15 kr | Engelsk |
| Midnight in Everwood | 135,15 kr | Engelsk |
| Fireborn: Twelve and the Frozen Forest | 126,65 kr | Engelsk |
| Det andre oppdraget | 279 kr | Norsk Bokmål |
| The Lord of the Rings | 568,65 kr | Engelsk |
| Runestone Saga: Children of Ragnarok | 220,15 kr | Engelsk |
| Be the Serpent | 313,65 kr | Engelsk |
| The Fantasy Art of Frazetta: The Masterpieces 2023 16-month Wall Calendar | 249 kr | Engelsk |
| Black Panther: Panther's Rage | 271,15 kr | Engelsk |
| Ikke la de døde komme hit | 299 kr | Norsk Bokmål |
| A River Enchanted (Elements of Cadence, Book 1) | 135,15 kr | Engelsk |
| Eksperimentet | 299 kr | Norsk Bokmål |
| A Game of Retribution | 288,15 kr | Engelsk |
| Monsters Born and Made | 220,15 kr | Engelsk |
| Stone of Tears: Book Two of the Sword of Truth | 254,15 kr | Engelsk |
| On the Edge of the Dark Sea of Darkness | 169,15 kr | Engelsk |
| This Savage Song collectors hardback | 271,15 kr | Engelsk |
| The Secret Unicorn Club: Discover the Hidden Book within a Book! | 288,15 kr | Engelsk |
| Empire of Exiles | 211,65 kr | Engelsk |
| Bindle Punk Bruja: A Novel | 169,15 kr | Engelsk |
| The River of Silver: Tales from the Daevabad Trilogy (The Daevabad Trilogy, Book 4) | 211,65 kr | Engelsk |
| The River of Silver: Tales from the Daevabad Trilogy (The Daevabad Trilogy, Book 4) | 254,15 kr | Engelsk |
| Anno Dracula Signed 30th Anniversary Edition | 271,15 kr | Engelsk |
| Ashes of Man | 254,15 kr | Engelsk |
| Absynthe | 143,65 kr | Engelsk |
| Into the Forest: Tales of the Baba Yaga | 194,65 kr | Engelsk |
| Sorcerer's Edge: The Tethered Citadel Book 3 | 288,15 kr | Engelsk |
| Night of Demons and Saints | 143,65 kr | Engelsk |
| Morrighan: The Beginnings of the Remnant Universe; Illustrated and Expanded Edition | 254,15 kr | Engelsk |
| The Two Doctors Gorski | 203,15 kr | Engelsk |
| Will Do Magic for Small Change | 313,65 kr | Engelsk |
| Daughter of Darkness | 322,15 kr | Engelsk |
| Piranesi | 379 kr | Norsk Bokmål |
| Hevneren fra Helheim | 299 kr | Norsk Bokmål |
| Nothing But Blackened Teeth | 160,65 kr | Engelsk |
| The Dragon's Promise | 152,15 kr | Engelsk |
| The Witch and the Tsar | 305,15 kr | Engelsk |
| Signal To Noise | 143,65 kr | Engelsk |
| Malarkoi | 254,15 kr | Engelsk |
| Thistlefoot: A Novel | 305,15 kr | Engelsk |
| Isolation: The horror anthology | 135,15 kr | Engelsk |
| The Lost Girl King | 126,65 kr | Engelsk |
| Amari and the Great Game | 203,15 kr | Engelsk |
| The Complete Tolkien Companion | 339,15 kr | Engelsk |
| A Magic Steeped In Poison | 135,15 kr | Engelsk |
| No Gods, No Monsters | 143,65 kr | Engelsk |
| Forestfall | 135,15 kr | Engelsk |
| Ever Cursed | 126,65 kr | Engelsk |
| The War of Two Queens | 313,65 kr | Engelsk |
| The Tower of Swallows | 339,15 kr | Engelsk |
| Fire and Blood: The inspiration for HBO's House of the Dragon (A Song of Ice and Fire) | 109,65 kr | Engelsk |
| The Hollows | 143,65 kr | Engelsk |
| Zodiac Academy 5: Cursed Fates: Shadow Princess | 466,65 kr | Engelsk |
| The Children of Gods and Fighting Men | 220,15 kr | Engelsk |
| Tune in Tomorrow: The Curious, Calamitous, Cockamamie Story Of Starr Weatherby And The Greatest Mythic Reality Show Ever | 143,65 kr | Engelsk |
| Terciel & Elinor - The Old Kingdom 1: The newest adventure in the bestselling Old Kingdom series | 135,15 kr | Engelsk |
| Lost in the Never Woods | 135,15 kr | Engelsk |
| Cemetery Boys | 135,15 kr | Engelsk |
| Tongues of Serpents: Book Six of Temeraire | 211,65 kr | Engelsk |
| Victory of Eagles: Book Five of Temeraire | 211,65 kr | Engelsk |
| Empire of Ivory: Book Four of Temeraire | 211,65 kr | Engelsk |
| Magician's Gambit: Book Three Of The Belgariad | 143,65 kr | Engelsk |
| Tolkien Calendar 2023 | 199 kr | Engelsk |
| Empire of the Vampire (Empire of the Vampire, Book 1) | 143,65 kr | Engelsk |
| Leaf by Niggle | 101,15 kr | Engelsk |
| Mr Bliss | 203,15 kr | Engelsk |
| Tree and Leaf: Including MYTHOPOEIA | 135,15 kr | Engelsk |
| The Hood | 135,15 kr | Engelsk |
| Lord of Shadows: The stunning new edition of the international bestseller | 220,15 kr | Engelsk |
| Heat Wave | 220,15 kr | Engelsk |
| The Drowned Woods | 220,15 kr | Engelsk |
| The Monsters We Defy | 211,65 kr | Engelsk |
| Blood Like Fate | 228,65 kr | Engelsk |
| A Broken Blade | 211,65 kr | Engelsk |
| The Drowned Woods | 220,15 kr | Engelsk |
| Into the Broken Lands | 305,15 kr | Engelsk |
| Lapvona: A Novel | 296,65 kr | Engelsk |
| Longshadow | 135,15 kr | Engelsk |
As a result, one can see the output is the gt with all the relevant information compiled into a gt table. What you do with that information is up to you.