Strings / Regex / Dates (unfinished)

strings

• sequence of characters

• used to store/represent textual data

• string verbs function WITHIN dplyr verbs (like filter())

• Can wrap stringr functions around each other

  • Ex. mutate(state_key = str_squish(str_to_lower(state)))

str_detect()

• looks for something in a string

• returns TRUE/FALSE

  • looks thru what u tell it to, returns TRUE if it finds the value ur looking for and FALSE if not

    • ex. filter(str_detect(candidate, '"') looks for ““ in the candidate column

    • ex. filter(str_detect(candidate, '"') == FALSE) looks for rows where there is no ““ in candidate column

      • == TRUE is implied but FALSE is not, so need to add it

• NOT an absolutely equal to —> will return TRUE for ANY match that contains the desired value

  • Will read TRUE for “Kennedy” if the string has “JF Kennedy” “John F Kennedy” or “Kennedy”

• CASE MATTERS → it will only return true of the CASE is the same (all caps, no caps, etc)

• Find all the different values that hold a certain value

  • Ex. filter(str_detect(party_detailed, "SOCIALIST")) |> distinct(party_detailed)

str_trim()

removes whitespace at ends

• defaults to both sides

• can also indicate a side:

  • ex. str_trim(messy, side = "right") —> removes just white space on right side

str_squish()

removes white space at the ends AND collapses spaces in the middle

changing cases

• str_to_lower(): replaces all text with lowercase

• str_to_upper(): replaces all text with uppercase

• str_to_title(): capitalises 1st letter of every word after a space

str_lenght()

gives character length of each value within the string

• ex. str_length(messy)

  [1] 15 5 14 9

str_count()

can look for a single element within a strong

• str_count(messy, "s"): returns how many times “s” appears in each element of the vector messy

• str_count(c("NJ-11", "NJ-11 and NY-14"), "-"): counts the number of hyphens in the vectors “NJ-11" and "NJ-11 and NY-14"

• Can use WITHIN mutate

  • mutate(n_commas = str_count(candidate, ",")) |>  count(n_commas)

    • Make a column with how many rows have a comma, then count the results of that column

str_replace()

replace parts of a string w smth else

• str_replace([string name], “find this”, “replace with this”)

  • Ex. str_replace(messy, "Co\\.", "County")

• str_replace_all(): can replace smth in many rows at the same time

  • ex. str_replace_all(towns, "Co\\.", "County")

str_remove()

will remove 1st in each element

• str_remove(messy, " "): removes 1st space

can use str_remove_all(): will remove ALL of a certain element in each value

• can add _all to ANY stringr function (usually can be ur default)

• good way to remove commas in numbers: as.numeric(str_remove_all(votes_text, ","))

str_sub()

subtracts part(s) of a string

• str_sub([string name], number of characters)

  • Ex. str_sub(zips, 1, 5): first five characters

• use negative sign to start from the back

  • Ex. str_sub(zips, -4): last four characters

str_pad()

adds extra characters when full #s don’t reach length you want

• Add zero to the front of a 5 digit number (like a zip code)

  • Ex. str_pad(broken_fips, width = 5, side = "left", pad = "0")

str_c()

combine —> glues strings together

• Can combine ind elements

  • Ex. str_c("NJ", "-", "11")

    • NJ-11

  • Ex. str_c(c("NJ", "NY", "PA"), "-11")

    • NJ-11, NY-11, PA-11

  • Ex. str_c("NJ", "11", sep = "-")

    • NJ-11

• Can combine full strings

  • Ex. str_c(state_po, " ", year)

• Does the same thing as paste0()

  • but str_c() recognizes NA whereas paste0() will just see it as a character

Regular Expressions (Regex)

looking for patterns (“think like a computer”)

• pattern describes the SHAPE of the thing instead of its exact letters

• reads your string from left to right, one character at a time

• By default the engine finds a match ANYWHERE in the string, not just at the beginning

  • So "County" matches "County Line Road" if we're looking for County.

• It reports the FIRST match it finds, scanning left to right, not the best one or the longest one

  • str_extract() gives you that first match and stops

• can use in str_detect()

  • Ex. str_detect(“StXLouis”, “St.Louis”)

    • Will come back TRUE bc “.” is looking for ANY character

    • If you want punctuation to function as punctuation and not code: add escapes

      • Ex. str_detect("StXLouis", "St\\.Louis")

        • This will then come back FALSE

Regex examples

• .          any single character

  • (why you need the back slashes when period is a part of a name)

•  \\d        a digit

• \\w        a letter, digit, or underscore

• \\s        a space, tab, or newline

• [A-Z]      any character in the set

• [^0-9]     any character NOT in the set

• +          one or more of the thing before it

• *          zero or more

• ?          zero or one, which means optional

• {2}        exactly two       {2,4}  between two and four

• ^          the start of the string

• $          the end of the string

• |          or

• ( )        a group, which lets us pull out one piece

• hyphen on its own is a normal symbol, but in [ ] it becomes a range

str_detect() and regex

How to use regex in practice

• Ex. str_detect(“StXLouis”, “St.Louis”)

  • Will come back TRUE bc “.” is looking for ANY character

  • If you want punctuation to function as punctuation and not code: add escapes

    • Ex. str_detect("StXLouis", "St\\.Louis")

      • This will then come back FALSE

Regex match patterns

•    .+                    one or more, greedy

•   .+?                    one or more, lazy

•    .*                     zero or more, greedy

•    .*?                   zero or more, lazy

  • optional → doesn’t HAVE to be there, but okay if it is

• \\s*                     zero or more spaces

• (JR|SR|III|II)      one of these four

• \\.?                      an optional period

•  $                        the end of the string

EX: How would you read: str_detect(districts, "^[A-Z]{2}-[0-9]{1,2}$") ?

• ^ =start of the string

• {A-Z]{2} =exactly two capital letters

• - =a literal dash

• [0-9]{1,2} =one or two digits

• $ =end of the string

escapes

Used to tell R that certain symbols are a part of the character and NOT a part of a code

• ex. Co\\.

  • Tells R you are looking for Co. and stops it from interpreting the period as a code

• Needs double slashes (other languages use single slashes)

  • In regex, \d means a digit

[[:xxx:]]

means ANY of this thing

• [[:punct:]] any punctuation mark

• [[:alpha:]] any letter

• [[:digit:]] any digi

• use to strip all punctuation (helpful before joining smth)

  • ex. str_remove_all(c("BUSH, GEORGE H.W.", "O'BRIEN, MARY-JANE"), "[[:punct:]]")

anchor

a symbol that ties a pattern to a position

• matches a place in the string rather than a character → conditions about WHERE the match is allowed to happen

• ^ the start of the string

  • "^County" the string must START, and then the word County

• $ the end of the string

  • "County$" the word County, and then the string must END

• can also use alternatives: str_starts(counties, "County") and str_ends(counties, "County")

greedy quantifiers

take as much as they possibly can —> then give characters back one at a time, and only if that is the only way to make the rest of the pattern fit

• Quantifiers like + and *

• ex. str_extract(line, "\\(.+\\)")

  • asked for an open parenthesis, then one or more of any character, then a close parenthesis

  • this means that if the vector titled line is "PEREZ (D) defeated SMITH (R)", this command will return “(D) defeated SMITH (R)" rather than (D) bc it goes ALL THE WAY to the last ) it finds

Lazy quantifiers

take as little as possible, then add one character at a time only when forced

• put a ? after a quantifier

• ex. str_extract(line, "\\(.+?\\)")

  • now, .+? starts with one character, the engine checks for a ")", finds one right there, and stops. We get "(D)"

Why store dates as their own type?

• won’t sort chronologically if text

  • 1/1/2025 will come before 12/1/2020

• imputing as numbers looks like gibberish to human eyes

  • also! R will drop the 0 at the from of the 1st nine months

how to read in dates

always read in dates in format they APPEAR in your data set —> NOT the format you want them to be

• international standard: ISO 8601 (aka YYYY-MM-DD)

  • Unambiguous, zero-padded, biggest to smallest

  • sorting alphabetically also sorts chronologically

  • how R prints everything (no matter how you parse it)

  • if data comes in like this, R will automatically read it as dates (won’t do this for any other format!)

• seperators (- , /) DO NOT MATTER

• ymd("2024-11-05"): year, month, day

• mdy("11/05/2024"): month, day, year

• dmy("05-11-2024"): day, month, year

• CAN covert text to dates

  • ex. mdy("November 5, 2024")

range()

• check range of dates

• make sure there are no NAs

  • range(contribs$date, na.rm = TRUE)

if dates come is as numeric…

…pad the zeros first!

• have to convert to character, THEN dates

  • ex. as.character() |> str_pad(width = 8, side = "left", pad = "0") |> mdy()

extracting timeframes from dates data

• can create new columns of certain lengths of time when the data is stored as dates

• ex. store as weeks

  • mutate(week = floor_date(date, "week")

Pulling parts out

• pull out yr, month, or day from a vector

  • year(), month(), day()

• use label = TRUE to pull out with text

  • ex. month(election_day, label = TRUE) = Nov

how to get a length of time

• subtracting two dates

  • ex. election_day - ymd("2020-11-03") returns “Time difference of 1463 days”

    • this format will always give days

    • will NOT behave like a numeric value one its own —> have to use as.numeric(election_day - ymd("2020-11-03"))

  • between top and bottom of a column

    • tail(terms$termstart,1) - head(terms$termstart, 1)

How to add time

• election_day + days(30)

• election_day + weeks(2)

• election_day + years(1)

important to wrap in days() or weeks() so R knows what unit to use

floor_date()

rounds DOWN to the start of a period

• good for collapsing dates into a period

  • ex. floor_date(election_day, "month")

• also ceiling_date() and round_date() but we will only use floor_date()

• always round THEN group THEN summarize

3 TRAPS

1. TWO-DIGIT YEARS. Is 68 in 1968 or 2068?

   1. mdy("01/01/68") —> will see as 2068

      mdy("01/01/69") —> will see as 1969

2. TRAP 2: AMBIGUOUS FORMATS

   1. mdy("05/11/2024") —> May 11

      dmy("05/11/2024") —> Nov 5

3. TRAP 3: EXCEL

4. Spreadsheets convert anything that looks like a date into one, whether or not it is. A precinct code of "3-1" becomes March 1. Once that has happened in the file, the original value is gone and no amount of parsing recovers it.