Advanced R Lab: Processing Citi Bike API Data, JSON Structure, and Statistical Visualization

Understanding API Output:
- The initial downloaded data from an API is essentially a long text string.
- In the R environment, this output is often truncated, meaning it is shortened for display and does not show every piece of information at once.
- The structure of the raw API data shows that it is a collection of various columns, though not every column provided by the API is necessarily included in the final local dataset.
JSON Conversion:
- When the raw text string is converted to JSON format, the structure becomes more organized.
- The conversion allows users to see specific columns identified by the $ symbol.

Examining Objects with prettify:
- The jsonlite package contains a custom function called $prettify()$ to examine API output objects.
- Functionality: $prettify()$ makes raw JSON text significantly easier to read by adding line breaks and indentation.
- Output Volume: Running $prettify(API)$ can generate an enormous amount of output. For instance, when knitting a lab document to HTML or PDF, it can result in a document spanning nearly 700 pages if not handled correctly.
- Machine Performance Warning: Older machines or those with slow processing may struggle to process the large output of the $prettify()$ function.
- Best Practice: It is advised to use $prettify()$ for initial analysis purposes only. After viewing the structure, the line of code should be commented out (using #) before knitting the file to prevent the computer from attempting to render hundreds of pages of text.
Data Content in the Citi Bike Dataset:
- The output from $prettify()$ reveals individual station IDs for every entry in the Citi Bike system across New York City.
- Each entry contains unique identification and several datasets, including:
  - $num_bikes_available$
  - $num_bikes_disabled$
  - $num_docs_available$ (Available docking slots)

The str() Function:
- The $str()$ function (Structure) is used to determine the type of object and what information is available within it.
- Applying $str(station_status)$ reveals that the object is a data frame containing a specific number of observations and variables.
Data Frame Composition:
- Initially, the $station_status$ data frame may contain a subset of variables (e.g., 6 columns) filtered from the larger original API output.
Generating Histograms:
- Histograms are generated using the $hist()$ function.
- To specify a column within a data frame, use the dollar sign notation: $Dataset\$ColumnName$ (e.g., $station_status\$num_docs_available$ ).
- Live Data Variation: Because the Citi Bike API provides live data, visual outputs like histograms will vary depending on the exact time the data is pulled. For example, a histogram generated at one moment might look different from one generated five minutes later due to bike movements in the city.

Counting Specific Observations:
- To find how many stations meet a certain criteria, use the $sum()$ function combined with a logical operator.
- Example: To find stations with at least one e-bike available: sum(station_status\$num_ebikes_available > 0)
- Example Results from Transcript:
  - Instructor's result: 1,893 stations.
  - Sajid's result: 1,904 stations.
  - Another student's result: 1,890 stations.
- This variation confirms the dynamic nature of the dataset during a sunny day in New York City.
Creating Subsets (New Data Frames):
- To explore stations with at least one e-bike, a new data frame can be created using square bracket indexing.
- Example name: $ebikes_df$
- Syntax: `ebikes_df <- station_status[station_status\$num_ebikes_available > 0, ]
- The resulting data frame will have fewer observations than the original $station_status$ data frame because it only includes rows where the condition is true.

Calculating Means:
- The $mean()$ function can be applied to specific columns to compare data subsets.
- Comparing the mean of $num_docs_available$ for the filtered $ebikes_df$ versus the full $station_status$ helps identify if stations with e-bikes have higher or lower dock availability on average.
Creating the station_size Column:
- A new attribute called $station_size$ represents the total number of slots available for bikes (occupied or empty).
- This is calculated by summing several existing columns: $station_size = num_docs_available + num_bikes_available + num_ebikes_available + num_docks_disabled + num_bikes_disabled$
- Upon running this calculation, the environment will show the data frame variable count increasing (e.g., from 6 to 7 variables).
Calculating Occupied Docks:
- A new variable, $num_occupied_docs$ , is created to represent the number of docks currently in use.
- Instructional logic for this variable: $num_occupied_docs = station_size - num_docs_available - \text{disabled units}$

Using the plot Command:
- The $plot()$ function is used to produce an $x-y$ scatter plot.
- Mapping:
  - X-axis: $num_occupied_docs$ (Number of docks currently filled).
  - Y-axis: $num_bikes_available$ (Total number of bikes available for rent).
Visual Interpretation of the Citi Bike Scatter Plot:
- Clusters: Most stations tend to cluster between $10$ to $50$ occupied docks and $5$ to $40$ available bikes.
- Interpretation: This suggests many stations share a similar capacity range.
- Outliers: Points located farther to the right (higher X-axis values) or near the top (higher Y-axis values) represent "larger capacity stations."
- Specific Examples: Data points showing $80$ to $90$ occupied docks with nearly zero bikes available indicate stations that are effectively at capacity but currently empty of rentable bikes.

Knitting the Lab:
- Labs are typically knitted to HTML for submission.
- Final output can be opened in a browser and then printed or "Saved as PDF."
Troubleshooting Package Errors:
- Error Message: "could not find function 'fromJSON'"
- Cause: This usually occurs if the jsonlite package was not installed properly or the library was not loaded using $library(jsonlite)$ .
- Technical fix:
  1. Run $install.packages("jsonlite")$ .
  2. Run $library(jsonlite)$ .
  3. Crucial Step: Comment out the $install.packages()$ line before knitting to avoid errors during the rendering process.

Location Narratives:
- Hugo mentioned he is located in downtown New York City near Wall Street and Tribeca, noting it was a sunny day, which explains why the bike data was changing rapidly during the lab.
- A student in the chat mentioned it was raining in Georgia.
Troubleshooting with Ray:
- Ray encountered errors where R could not find the conversion functions.
- The instructor guided Ray to check the console for installation success. It was discovered that the jsonlite package had not finished installing.
- Ray was instructed to place the cursor at the end of the maintenance lines and use Control + Enter to force the execution of the install and library commands.
Filtering Issues with ebikes_df:
- A student encountered an error "ebikes_df not found" during step 10.
- The instructor clarified that the student must successfully run the code in step 9 (the filtering step) to create the object in the R environment before it can be used for calculations in step 10.