Web scraping

---

---

## Scraping the web

---

## Scraping the web: what? why?

- Increasing amount of data is available on the web
--

- These data are provided in an unstructured format: you can always copy&paste, but it's time-consuming and prone to errors

--
- Web scraping is the process of extracting this information automatically and transform it into a structured dataset

--
- Two different scenarios:
    - Screen scraping: extract data from source code of website, with html parser (easy) or regular expression matching (less easy).
    - Web APIs (application programming interface): website offers a set of structured http requests that return JSON or XML files.

---

## Web Scraping with rvest

---

## Hypertext Markup Language

- Most of the data on the web is still largely available as HTML 
- It is structured (hierarchical / tree based), but it''s often not available in a form useful for analysis (flat / tidy).

```html
<html>
  <head>
    <title>This is a title</title>
  </head>
  <body>
    <p align="center">Hello world!</p>
  </body>
</html>
```

---

## rvest

.pull-left[
- The **rvest** package makes basic processing and manipulation of HTML data straight forward
- It's designed to work with pipelines built with `%>%`
]
.pull-right[
<img src="img/rvest.png" width="230" style="display: block; margin: auto 0 auto auto;" />
]

---

## Core rvest functions

- `read_html`   - Read HTML data from a url or character string
- `html_element`  - Select a specified node from HTML document
- `html_elements`  - Select specified nodes from HTML document
- `html_table`  - Parse an HTML table into a data frame
- `html_text`   - Extract tag pairs' content
- `html_name`   - Extract tags' names
- `html_attrs`  - Extract all of each tag's attributes
- `html_attr`   - Extract tags' attribute value by name

---

## SelectorGadget

.pull-left-narrow[
- Open source tool that eases CSS selector generation and discovery
- Easiest to use with the [Chrome Extension](https://chrome.google.com/webstore/detail/selectorgadget/mhjhnkcfbdhnjickkkdbjoemdmbfginb) 
- Find out more on the [SelectorGadget vignette](https://cran.r-project.org/web/packages/rvest/vignettes/rvest.html)
]
.pull-right-wide[
<img src="img/selector-gadget/selector-gadget.png" width="75%" style="display: block; margin: auto;" />
]

---

## Using the SelectorGadget

---

---

---

---

---

---

## Using the SelectorGadget

Through this process of selection and rejection, SelectorGadget helps you come up with the appropriate CSS selector for your needs

## Scraping top 250 movies on IMDB

---

## Top 250 movies on IMDB

Take a look at the source code, look for the tag `table` tag:
<br>
http://www.imdb.com/chart/top

.pull-left[
<img src="img/imdb-top-250.png" width="100%" style="display: block; margin: auto;" />
]
.pull-right[
<img src="img/imdb-top-250-source.png" width="94%" style="display: block; margin: auto;" />
]

---

## First check if you're allowed!

```r
library(robotstxt)
paths_allowed("http://www.imdb.com")
```

```
## [1] TRUE
```

vs. e.g.

```r
paths_allowed("http://www.facebook.com")
```

```
## [1] FALSE
```

---

## Plan

---

## Plan

1. Read the whole page

2. Scrape movie titles and save as `titles`

3. Scrape years movies were made in and save as `years`

4. Scrape IMDB ratings and save as `ratings`

5. Create a data frame called `imdb_top_250` with variables `title`, `year`, and `rating`

---

## Step 1. Read the whole page

---

## Read the whole page

```r
page <- read_html("https://www.imdb.com/chart/top/")
page
```

```
## {html_document}
## <html lang="en-US" xmlns:og="http://opengraphprotocol.org/schema/" xmlns:fb="http://www.facebook.com/2008/fbml">
## [1] <head>\n<meta http-equiv="Content-Type" content="text/html ...
## [2] <body>\n<div>    <img height="1" width="1" style="display: ...
```

---

## A webpage in R

- Result is a list with 2 elements

```r
typeof(page)
```

```
## [1] "list"
```

- that we need to convert to something more familiar, like a data frame....

```r
class(page)
```

```
## [1] "xml_document" "xml_node"
```

---

## Step 2. Scrape movie titles and save as `titles`

---

## Scrape movie titles

---

## Scrape the nodes

```r
page %>%
  html_elements(".titleColumn a")
```
]
.pull-right[
<img src="img/titles.png" width="100%" style="display: block; margin: auto;" />
]

---

## Extract the text from the nodes

```r
page %>%
  html_elements(".titleColumn a") %>%
  html_text()
```
]
.pull-right[
<img src="img/titles.png" width="100%" style="display: block; margin: auto;" />
]

---

## Save as `titles`

```r
titles <- page %>%
  html_elements(".titleColumn a") %>%
  html_text()

titles
```
]
.pull-right[

```r
knitr::include_graphics("img/titles.png")
```
]

---

## Step 3. Scrape year movies were made and save as `years`

---

## Scrape years movies were made in

---

## Scrape the nodes

```r
page %>%
  html_elements(".secondaryInfo")
```
]
.pull-right[
<img src="img/years.png" width="100%" style="display: block; margin: auto;" />
]

---

## Extract the text from the nodes

```r
page %>%
  html_elements(".secondaryInfo") %>%
  html_text()
```
]
.pull-right[
<img src="img/years.png" width="100%" style="display: block; margin: auto;" />
]

---

## Clean up the text

We need to go from `"(1994)"` to `1994`:

- Remove `(` and `)`: string manipulation

- Convert to numeric: `as.numeric()`

---

## stringr

.pull-left-wide[
- **stringr** provides a cohesive set of functions designed to make working with strings as easy as possible
- Functions in stringr start with `str_*()`, e.g.
  - `str_remove()` to remove a pattern from a string
  
  ```r
   str_remove(string = "jello", pattern = "el")
  ```
  
  ```
  ## [1] "jlo"
  ```
  - `str_replace()` to replace a pattern with another
  
  ```r
  str_replace(string = "jello", pattern = "j", replacement = "h")
  ```
  
  ```
  ## [1] "hello"
  ```
]
.pull-right-narrow[
<img src="img/stringr.png" width="100%" style="display: block; margin: auto auto auto 0;" />
]

---

## Clean up the text

```r
page %>%
  html_elements(".secondaryInfo") %>%
  html_text() %>%
  str_remove("\\(") # remove (
```

---

## Clean up the text

```r
page %>%
  html_elements(".secondaryInfo") %>%
  html_text() %>%
  str_remove("\\(") %>% # remove (
  str_remove("\\)") # remove )
```

---

## Convert to numeric

```r
page %>%
  html_elements(".secondaryInfo") %>%
  html_text() %>%
  str_remove("\\(") %>% # remove (
  str_remove("\\)") %>% # remove )
  as.numeric()
```

---

## Save as `years`

```r
years <- page %>%
  html_elements(".secondaryInfo") %>%
  html_text() %>%
  str_remove("\\(") %>% # remove (
  str_remove("\\)") %>% # remove )
  as.numeric()

years
```
]
.pull-right[
<img src="img/years.png" width="100%" style="display: block; margin: auto;" />
]

---

## Step 4. Scrape IMDB ratings and save as `ratings`

---

## Scrape IMDB ratings

---

## Scrape the nodes

```r
page %>%
  html_elements("strong")
```
]
.pull-right[
<img src="img/ratings.png" width="100%" style="display: block; margin: auto;" />
]

---

## Extract the text from the nodes

```r
page %>%
  html_elements("strong") %>%
  html_text()
```
]
.pull-right[
<img src="img/ratings.png" width="100%" style="display: block; margin: auto;" />
]

---

## Convert to numeric

```r
page %>%
  html_elements("strong") %>%
  html_text() %>%
  as.numeric()
```
]
.pull-right[
<img src="img/ratings.png" width="100%" style="display: block; margin: auto;" />
]

---

## Save as `ratings`

```r
ratings <- page %>%
  html_elements("strong") %>%
  html_text() %>%
  as.numeric()

ratings
```
]
.pull-right[
<img src="img/ratings.png" width="100%" style="display: block; margin: auto;" />
]

---

## Step 5. Create a data frame called `imdb_top_250`

---

## Create a data frame: `imdb_top_250`

```r
imdb_top_250 <- tibble(
  title = titles, 
  year = years, 
  rating = ratings
  )

imdb_top_250
```

---

## Clean up / enhance

May or may not be a lot of work depending on how messy the data are

- Add a variable for rank

```r
imdb_top_250 <- imdb_top_250 %>%
  mutate(rank = 1:nrow(imdb_top_250)) %>%
  relocate(rank)
```

---

```r
imdb_top_250 %>%
  print(n = 20)
```

---

## What next?

---

```r
imdb_top_250 %>% 
  count(year, sort = TRUE)
```

---

```r
imdb_top_250 %>% 
  filter(year == 1995) %>%
  print(n = 8)
```

---

.question[
Visualize the average yearly rating for movies that made it on the top 250 list over time.
]

.panelset[
.panel[.panel-name[Plot]
<img src="img/imdbplot.png" width="65%" style="display: block; margin: auto;" />
]
.panel[.panel-name[Code]

```r
imdb_top_250 %>% 
  group_by(year) %>%
  summarise(avg_score = mean(rating)) %>%
  ggplot(aes(y = avg_score, x = year)) +
  geom_point() +
  geom_smooth(method = "lm", se = FALSE) +
  labs(x = "Year", y = "Average score")
```
]
]

---

## When you re-run all the previous 'scraping' codes on IMDb, it will not work. Why?!

---

<img src="img/imdb-top-250.png" width="100%" style="display: block; margin: auto;" />
]

<img src="img/imdb-new.png" width="100%" style="display: block; margin: auto;" />
]

---

<img src="img/imdb-top-250.png" width="100%" style="display: block; margin: auto;" />
]

<img src="img/imdb-new.png" width="100%" style="display: block; margin: auto;" />
]

- Solution 1: extract 25 movies at a time

- Solution 2: use a different source (but rating variable is then lost)
---

## Solution 1

```r
page <- "https://www.imdb.com/chart/top/"

doc <- read_html(page)
title <- doc %>%  
    html_elements("li h3.ipc-title__text") %>%  
    html_text()
year <- doc %>% 
    html_elements(".sc-b189961a-8.kLaxqf.cli-title-metadata-item") %>% 
    html_text()
year <- as.numeric(year[seq(1,length(year),3)])

ratings <- doc %>% 
    html_elements(".ipc-rating-star.ipc-rating-star--base.ipc-rating-star--imdb.ratingGroup--imdb-rating") %>% 
    html_text()

ratings <- as.numeric(str_extract(ratings,"[0-9]\\.[0-9]"))

imdb_top_25 <-tibble(
  title=title,
  year=year,
  rating=ratings)
```

---

<div id="htmlwidget-1b4ff99564eb6e8884a5" style="width:100%;height:400px;" class="datatables html-widget"></div>
<script type="application/json" data-for="htmlwidget-1b4ff99564eb6e8884a5">{"x":{"filter":"none","vertical":false,"data":[["1","2","3","4","5","6","7","8","9","10","11","12","13","14","15","16","17","18","19","20","21","22","23","24","25"],["1. Die Verurteilten","2. Der Pate","3. The Dark Knight","4. Der Pate 2","5. Die zwölf Geschworenen","6. Schindlers Liste","7. Der Herr der Ringe: Die Rückkehr des Königs","8. Pulp Fiction","9. Der Herr der Ringe: Die Gefährten","10. Zwei glorreiche Halunken","11. Forrest Gump","12. Der Herr der Ringe: Die zwei Türme","13. Fight Club","14. Inception","15. Star Wars: Episode V - Das Imperium schlägt zurück","16. Matrix","17. GoodFellas - Drei Jahrzehnte in der Mafia","18. Einer flog über das Kuckucksnest","19. Sieben","20. Interstellar","21. Ist das Leben nicht schön?","22. Die sieben Samurai","23. Das Schweigen der Lämmer","24. Der Soldat James Ryan","25. City of God"],[1994,1972,2008,1974,1957,1993,2003,1994,2001,1966,1994,2002,1999,2010,1980,1999,1990,1975,1995,2014,1946,1954,1991,1998,2002],[9.300000000000001,9.199999999999999,9,9,9,9,9,8.9,8.9,8.800000000000001,8.800000000000001,8.800000000000001,8.800000000000001,8.800000000000001,8.699999999999999,8.699999999999999,8.699999999999999,8.699999999999999,8.6,8.699999999999999,8.6,8.6,8.6,8.6,8.6]],"container":"<table class=\"display\">\n  <thead>\n    <tr>\n      <th> <\/th>\n      <th>title<\/th>\n      <th>year<\/th>\n      <th>rating<\/th>\n    <\/tr>\n  <\/thead>\n<\/table>","options":{"columnDefs":[{"className":"dt-right","targets":[2,3]},{"orderable":false,"targets":0},{"name":" ","targets":0},{"name":"title","targets":1},{"name":"year","targets":2},{"name":"rating","targets":3}],"order":[],"autoWidth":false,"orderClasses":false}},"evals":[],"jsHooks":[]}</script>

---
## Solution 2 
[https://icheckmovies.com/lists/imdbs+top+250/](https://www.icheckmovies.com/lists/imdbs+top+250/)

---
## Solution 2 
.middle[
<img src="img/sol2-2.png" width="90%" style="display: block; margin: auto;" />
]

---

## Solution 2

```r
page <- "https://www.icheckmovies.com/lists/imdbs+top+250/"
doc <- read_html(page)
title <- doc %>% 
    html_elements("h2 a") %>% 
    html_text()

title <- title[-1]

year <- doc %>% 
    html_elements(".info a:nth-child(1)") %>% 
    html_text()

imdb_top_250 <- tibble(title = title, year = year)
```
---

<div id="htmlwidget-8f836166d559454ecd73" style="width:100%;height:400px;" class="datatables html-widget"></div>
<script type="application/json" data-for="htmlwidget-8f836166d559454ecd73">{"x":{"filter":"none","vertical":false,"data":[["1","2","3","4","5","6","7","8","9","10","11","12","13","14","15","16","17","18","19","20","21","22","23","24","25","26","27","28","29","30","31","32","33","34","35","36","37","38","39","40","41","42","43","44","45","46","47","48","49","50","51","52","53","54","55","56","57","58","59","60","61","62","63","64","65","66","67","68","69","70","71","72","73","74","75","76","77","78","79","80","81","82","83","84","85","86","87","88","89","90","91","92","93","94","95","96","97","98","99","100","101","102","103","104","105","106","107","108","109","110","111","112","113","114","115","116","117","118","119","120","121","122","123","124","125","126","127","128","129","130","131","132","133","134","135","136","137","138","139","140","141","142","143","144","145","146","147","148","149","150","151","152","153","154","155","156","157","158","159","160","161","162","163","164","165","166","167","168","169","170","171","172","173","174","175","176","177","178","179","180","181","182","183","184","185","186","187","188","189","190","191","192","193","194","195","196","197","198","199","200","201","202","203","204","205","206","207","208","209","210","211","212","213","214","215","216","217","218","219","220","221","222","223","224","225","226","227","228","229","230","231","232","233","234","235","236","237","238","239","240","241","242","243","244","245","246","247","248","249","250"],["The Shawshank Redemption","The Godfather","The Dark Knight","The Godfather Part II","12 Angry Men","Schindler's List","The Lord of the Rings: The Return of the King","Pulp Fiction","The Lord of the Rings: The Fellowship of the Ring","Il buono, il brutto, il cattivo","Forrest Gump","The Lord of the Rings: The Two Towers","Fight Club","Inception","Star Wars: Episode V - The Empire Strikes Back","The Matrix","Goodfellas","One Flew Over the Cuckoo's Nest","Se7en","Interstellar","It's a Wonderful Life","Shichinin no samurai","The Silence of the Lambs","Saving Private Ryan","Cidade de Deus","La vita è bella","The Green Mile","Dune: Part Two","Terminator 2: Judgment Day","Star Wars","Back to the Future","Sen to Chihiro no kamikakushi","The Pianist","Gisaengchung","Psycho","Gladiator","Spider-Man: Across the Spider-Verse","The Lion King","The Departed","Léon","American History X","Whiplash","The Prestige","Hotaru no haka","Seppuku","The Usual Suspects","Casablanca","Intouchables","Nuovo Cinema Paradiso","Modern Times","Rear Window","C'era una volta il West","Alien","City Lights","Django Unchained","Apocalypse Now","Memento","12th Fail","WALL·E","Raiders of the Lost Ark","Das Leben der Anderen","Sunset Blvd.","Avengers: Infinity War","Paths of Glory","Spider-Man: Into the Spider-Verse","Witness for the Prosecution","The Shining","The Great Dictator","Aliens","Inglourious Basterds","The Dark Knight Rises","Dr. Strangelove or: How I Learned to Stop...","Coco","American Beauty","Oldeuboi","Amadeus","Toy Story","Avengers: Endgame","Das Boot","Braveheart","Good Will Hunting","Mononoke-hime","Joker","Kimi no na wa.","Tengoku to jigoku","Once Upon a Time in America","3 Idiots","Singin' in the Rain","Capharnaüm","Idi i smotri","Requiem for a Dream","Toy Story 3","Star Wars: Episode VI - Return of the Jedi","Eternal Sunshine of the Spotless Mind","Jagten","2001: A Space Odyssey","Oppenheimer","Reservoir Dogs","Ikiru","Lawrence of Arabia","The Apartment","Incendies","North by Northwest","Citizen Kane","M - Eine Stadt sucht einen Mörder","Scarface","Double Indemnity","Vertigo","Full Metal Jacket","Le fabuleux destin d'Amélie Poulain","Heat","A Clockwork Orange","Up","To Kill a Mockingbird","Jodaeiye Nader az Simin","The Sting","Indiana Jones and the Last Crusade","Die Hard","Taare Zameen Par","Metropolis","Snatch","Hamilton","L.A. Confidential","1917","Ladri di biciclette","Taxi Driver","Der Untergang","Dangal","Per qualche dollaro in più","Batman Begins","The Wolf of Wall Street","Some Like It Hot","Green Book","The Kid","The Father","Judgment at Nuremberg","The Truman Show","All About Eve","Top Gun: Maverick","Shutter Island","There Will Be Blood","Casino","Jurassic Park","Ran","The Sixth Sense","El laberinto del fauno","Unforgiven","No Country for Old Men","A Beautiful Mind","The Thing","Kill Bill: Vol. 1","The Treasure of the Sierra Madre","Yôjinbô","Monty Python and the Holy Grail","The Great Escape","Finding Nemo","Prisoners","Rashômon","Hauru no ugoku shiro","The Elephant Man","Chinatown","Dial M for Murder","Gone with the Wind","Lock, Stock and Two Smoking Barrels","V for Vendetta","El secreto de sus ojos","Inside Out","Raging Bull","Three Billboards Outside Ebbing, Missouri","Trainspotting","The Bridge on the River Kwai","Klaus","Fargo","Spider-Man: No Way Home","Catch Me If You Can","Warrior","Gran Torino","Tonari no Totoro","Million Dollar Baby","Harry Potter and the Deathly Hallows: Part 2","Bacheha-Ye aseman","12 Years a Slave","Blade Runner","Ben-Hur","Before Sunrise","The Grand Budapest Hotel","Barry Lyndon","Gone Girl","Hacksaw Ridge","Mad Max: Fury Road","The Gold Rush","Salinui chueok","Dead Poets Society","In the Name of the Father","Relatos salvajes","The Deer Hunter","The General","Monsters, Inc.","On the Waterfront","Sherlock Jr.","How to Train Your Dragon","Jaws","Le salaire de la peur","The Third Man","Mary and Max.","Smultronstället","Mr. Smith Goes to Washington","Ratatouille","Ford v Ferrari","Tôkyô monogatari","The Big Lebowski","Room","Det sjunde inseglet","Rocky","Logan","Spotlight","Hotel Rwanda","The Terminator","Platoon","La passion de Jeanne d'Arc","La haine","Before Sunset","The Best Years of Our Lives","Pirates of the Caribbean: The Curse of the...","The Exorcist","Jai Bhim","Rush","Network","The Wizard of Oz","Stand by Me","The Incredibles","Hachi: A Dog's Tale","Babam ve Oglum","Ah-ga-ssi","Into the Wild","The Sound of Music","To Be or Not to Be","La battaglia di Algeri","The Grapes of Wrath","Groundhog Day","Amores perros","Rebecca","The Iron Giant","Cool Hand Luke","The Help","It Happened One Night","Drishyam","Dances with Wolves","Gangs of Wasseypur","Aladdin"],["1994","1972","2008","1974","1957","1993","2003","1994","2001","1966","1994","2002","1999","2010","1980","1999","1990","1975","1995","2014","1946","1954","1991","1998","2002","1997","1999","2024","1991","1977","1985","2001","2002","2019","1960","2000","2023","1994","2006","1994","1998","2014","2006","1988","1962","1995","1942","2011","1988","1936","1954","1968","1979","1931","2012","1979","2000","2023","2008","1981","2006","1950","2018","1957","2018","1957","1980","1940","1986","2009","2012","1964","2017","1999","2003","1984","1995","2019","1981","1995","1997","1997","2019","2016","1963","1984","2009","1952","2018","1985","2000","2010","1983","2004","2012","1968","2023","1992","1952","1962","1960","2010","1959","1941","1931","1983","1944","1958","1987","2001","1995","1971","2009","1962","2011","1973","1989","1988","2007","1927","2000","2020","1997","2019","1948","1976","2004","2016","1965","2005","2013","1959","2018","1921","2020","1961","1998","1950","2022","2010","2007","1995","1993","1985","1999","2006","1992","2007","2001","1982","2003","1948","1961","1975","1963","2003","2013","1950","2004","1980","1974","1954","1939","1998","2005","2009","2015","1980","2017","1996","1957","2019","1996","2021","2002","2011","2008","1988","2004","2011","1997","2013","1982","1959","1995","2014","1975","2014","2016","2015","1925","2003","1989","1993","2014","1978","1926","2001","1954","1924","2010","1975","1953","1949","2009","1957","1939","2007","2019","1953","1998","2015","1957","1976","2017","2015","2004","1984","1986","1928","1995","2004","1946","2003","1973","2021","2013","1976","1939","1986","2004","2009","2005","2016","2007","1965","1942","1966","1940","1993","2000","1940","1999","1967","2011","1934","2015","1990","2012","1992"]],"container":"<table class=\"display\">\n  <thead>\n    <tr>\n      <th> <\/th>\n      <th>title<\/th>\n      <th>year<\/th>\n    <\/tr>\n  <\/thead>\n<\/table>","options":{"dom":"p","pageLength":8,"columnDefs":[{"orderable":false,"targets":0},{"name":" ","targets":0},{"name":"title","targets":1},{"name":"year","targets":2}],"order":[],"autoWidth":false,"orderClasses":false,"lengthMenu":[8,10,25,50,100]}},"evals":[],"jsHooks":[]}</script>
---

.midi[
.your-turn[
### .hand[Your turn!]
 - Use either of the alternative ways shown of scraping the top 250 movies and try (as best as you can) to answer the questions posed in this slide set!
 - Can you come up with a better solution?
]
]

---

## Ethics

---

## "Can you?" vs "Should you?"

.footnote[.small[
Source: Brian Resnick, [Researchers just released profile data on 70,000 OkCupid users without permission](https://www.vox.com/2016/5/12/11666116/70000-okcupid-users-data-release), Vox.
]]

---

## "Can you?" vs "Should you?"

---

## Challenges

---

## Unreliable formatting at the source

---

## Data broken into many pages

---

## Workflow

---

## Screen scraping vs. APIs

Two different scenarios for web scraping:

- Screen scraping: extract data from source code of website, with html parser (easy) or regular expression matching (less easy)

- Web APIs (application programming interface): website offers a set of structured http requests that return JSON or XML files

---

## A new R workflow

- When working in an R Markdown/Quarto document, your analysis is re-run each time you knit/render

- If web scraping in an R Markdown/Quarto document, you'd be re-scraping the data each time you knit, which is undesirable (and not *nice*)!

- An alternative workflow: 
  - Use an R script to save your code 
  - Saving interim data scraped using the code in the script as CSV or RDS files
  - Use the saved data in your analysis in your R Markdown document