Untitled

23.1 Introduction

• Overview of data rectangling:

  • Data rectangling involves converting hierarchical or tree-like data into rectangular data frames with rows and columns.

  • Hierarchical data is common in datasets sourced from the web.

  • Importance of understanding the structure of data to effectively manipulate and analyze it.

  • Introduction to key functions for rectangling:

  • tidyr::unnest_longer()

  • tidyr::unnest_wider()

  • Utilization of functions from the tidyr package, part of the tidyverse, along with datasets provided by repurrrsive for practical applications.

  • Incorporating jsonlite for reading JSON files into R lists.

23.2 Lists

• Definition and structure of lists:

  • Lists allow storage of heterogeneous types of objects within the same structure.

  • Creation of lists is done via list() function:

  • Example: x1 <- list(1:4, "a", TRUE)

  • Output:

    [[1]]
    [1] 1 2 3 4
  
    [[2]]
    [1] "a"
  
    [[3]]
    [1] TRUE
  

  • Lists can be named, similar to naming columns in a tibble.

  • Use of str() function for a compact display of the structure.

  • Hierarchical structure in lists:

  • Lists can contain other lists, allowing representation of tree-like structures.

  • Differentiation from c() which creates flat vectors.

  • Example of hierarchical list:

  • Code:

    x3 <- list(list(1, 2), list(3, 4))
    str(x3)
  

  • Output: A clear view of the list hierarchy.

23.2.1 Hierarchy

• Working with nested lists:

  • Example:
    x5 <- list(1, list(2, list(3, list(4, list(5))))

• Explanation on how View() function helps visualize complex lists in RStudio.

23.2.2 List-columns

• List-columns in tibbles:

  • Lists can be integrated into tibbles as list-columns, advantageous for storing non-compatible objects (e.g., model outputs).

  • Example of a simple list-column tibble:
    df <- tibble( x = 1:2, y = c("a", "b"), z = list(list(1, 2), list(3, 4, 5)) )

  • Commentary on the default print method for list-columns and how to view specific list-column details.

23.3 Unnesting

• Introduction to unnesting list-columns into rectangular formats.

• Two primary types of list-columns:

  • Named List-columns: Consistent naming across rows.

  • Unnamed List-columns: Length can vary across rows.

• Functions used for unnesting:

  • unnest_wider() for named lists.

  • unnest_longer() for unnamed lists.

  • Examples highlighting the usage of both functions:

  • Unnesting named columns: df1 <- tribble( ~x, ~y, 1, list(a = 11, b = 12), 2, list(a = 21, b = 22), 3, list(a = 31, b = 32) ) df1 |> unnest_wider(y)

    • Explanation of output and optional argument names_sep for disambiguation.

  • Unnesting unnamed lists:
    df2 <- tribble( ~x, ~y, 1, list(11, 12, 13), 2, list(21), 3, list(31, 32) ) df2 |> unnest_longer(y)

• Additional considerations when handling empty elements in lists and preserving data.

23.4 Case studies

• Application of unnesting techniques on real datasets:

  • The section showcases examples using datasets from the repurrrsive package, like gh_repos, which is a deeply nested list from the GitHub API.

• Exploring the nested structure using View()

• The conversion to tibble format using:

  repos <- tibble(json = gh_repos)

• Creating wider datasets with unnest_wider() and handling column information with names_sep.

23.4.1 Very wide data

• Importance of inspecting column names and handling duplicates during unnesting process.

23.4.2 Relational data

• Example involving the got_chars dataset from the Game of Thrones series.

  • Describing transformations required to connect various list-columns with the character dataset.

23.4.3 Deeply nested data

• Example using gmaps_cities:

  • Unnesting the geographical data retrieved from Google’s geocoding API.

  • Detailed itinerary of each unnest and rename step to organize data effectively for analysis.

23.5 JSON

• Discussion on JSON (JavaScript Object Notation) as the standard format for web API responses:

  • Basic structure of JSON, including key types like null, number, boolean, string, array, and object.

  • Differences between JSON and R’s data types.

  • Recommendations on using the jsonlite package and its functions:

  • read_json() for reading from disk.

  • parse_json() for handling JSON strings.

• Summary of transformations and syntax related to JSON parsing and converting into R data structures.

23.6 Summary

• Recap of the key concepts learned in the chapter:

  • Understanding and manipulating lists to create rectangular data frames.

  • Mastery of unnest_longer() and unnest_wider() functions for dealing with nested data structures.

  • Awareness of how JSON operates and how to handle it effectively in R.

• Transition to the next topic: web scraping and extracting data from HTML webpages.