ch.1how the internet works

How does the general tech savvy public think about web scrapers?

It's about retrieving HTML data from a domain page, parsing that data for target information, storing the target information, and Optionally moving to another page to repeat the process

What do most programmers actually know about how the internet works?

They only know abstractions that extends as far as what was required for them to solve a particularly tricky problem once in their career

What does this book teach about the internet?

Abstractions that describe the mechanics of the internet and web applications, to the extent needed to scrape the web( and then maybe a little more)

How is the internet governed?

There is no all-powerful governing body; different parts of the internet are governed by several different organizations

How did these internet-governing organizations develop?

Over time on a somewhat ad hoc(impromptu and for a specific purposse)and opt-in basis

What happens if you don't abide by these governing organization’s standards?

Your contributions to the internet may not work

What is web scraping?

Essentially, the practice of substituting a web browser for an application of your own design

Why is it important to understand the standards and protocols of browsers as a web scraper?

As a web scraper, you must both mimic and at times subvert the expected customs of the internet and its practices

In the early days, how were téléphone calls connected to each other?

Each téléphone was connected to a physical wire which was connected to a switchboard and switchboard operators had to create a line of wires between you and the recipient’s phone

In the modern day, how does data know where to go?

The data guides itself, in a process called packet switching

What technology is the one responsible for bring about the internet?

Packet switching

How does the process of packet switching work?

The message to be sent is divided into discrete ordered packets, each with its own sender and destination address

How do packets travel through networks?

Packets in the same message transmission might take different routes across the network and be reordered by the receiving computer when they arrive

Open Systems Interconnection(OSI) model

A way to describe a modern packet switching network

What are the 7 layers of the OSI model?

Physical, data link, network, transport, session, presentation, and application

In what layer of the OSI model do most web application developers spend their time?

Layer 7, the application layer

Which layer does this book focus most on?

Layer 7, application

TLS fingerprinting

A web scraping detection method that involves the transport layer

Why is it important to know about the other layers of the OSI model, not just application?

It can help you troubleshoot errors in your web applications and web scrapers

What does the physical layer of the OSI model do?

Specifies how information is physically transmitted with electricity over the ethernet wire in your house or any local network

What does the physical layer of the OSI model define?

Things like voltage levels that encode 1s and 0s and how fast those voltages can be pulsed; also how radio waves over bluetooth and wifi are interpreted

What does the physical layer of the OSI model not involve?

Programming or digital instructions; it is based purely on physics and electrical standards

What does the data link layer of the OSI model specify?

How information is transmitted between to nodes in a local network, for example, between your computer and a router

What does the data link layer define?

The beginning and ending of a single transmission and provides for error correction if the transmission is lost or garbled

What happens to packets at the data link layer?

They are wrapped in an additional “digital envelope” containing routing information

What are the packets in the data link layer referred to as?

Frames

What happens when the information in the frame is no longer needed?

It is unwrapped and sent across the network as a packet

What happens in the data link layer to all the devices on a network?

They are receiving the same data at all times and there's no actual switching or control over how the data is going

What do devices that don’t have data addressed to them do at the data link layer?

They will generally ignore the data and wait until they get something that's meant for them

What happens at the network layer?

Packet switching and therefore the internet

How are packets forwarded to their destinations in the network server?

by a router

What part of the Transmission Control Protocol/Internet Protocol(TCP/IP) is involved in the network layer?

Internet Protocol

What do IP addresses allow?

Any computer in the world can send data to an address

What is Layer 4?

Transport Layer

What is the transport layer for?

Connecting a specific service or application running on a computer, rather than connecting to the computers themselves; it is also responsible for any error connection or retrying needed in the stream of data

In what way is TCP picky?

It will keep requesting any missing packets until all of them are correctly received

What is TCP often used for?

File transfers

What must be done in TCP file transfer for it to work?

All packets must be correctly received in the right order

What does UDP stand for?

User Datagram Protocol

What will UDP do differently from TCP during file transfers?

It will happily skip over the missing packets in order to keep the data streaming in

In what situations is UDP often used?

Videoconferencing or audioconferencing

Why is UDP often used in video and audioconferencing?

A temporary drop in transmission quality is preferable to a lag in the conversation

Why are port numbers relevant to the transport layer?

Different applications on your computer can have different data reliability needs at the same time

What is an example of different data reliability needs?

Making a phone call while downloading a file

How is your operating system relevant to port numbers?

The OS assigns each application or service running on your computer to a specific port, from where it sends and receives data

Where does the OS write down the port number?

After the IP address, separated by a colon

Session layer

Responsible for opening and closing a session between two applications

What kind of information does the session layer contain?

What data has and hasn’t been sent, and who the computer is communicating with

How long does the session stay open?

Generally, for as long as it takes to complete the data request, and then closes

What does the session layer do if there’s a brief crash or disconnect?

It allows for retrying a transmission

How can the term “sessions” be used differently in computer science?

Sessions in the session layer of the OSI model are different from sessions and session data that the web developers usually talk about

Session variable

Part of a web application, they are a concept in the application layer that are implemented by the web browser software

How long do session variables stay in the browser?

For as long as they need to or until the user closes the browser window

How long do sessions usually last in the OSI model?

For as long as it takes to transmit a single file

Presentation layer

Transforms incoming data from character strings into a format that the application can understand and use; also responsible for character encoding and data compression

Example of what presentation layer does

Cares about whether incoming data received by the application represents a PNG file or an HTML file, and hands this file to the application layer accordingly