Web Search Fundamentals and How Search Engines Work

Web Search Fundamentals

  • A search engine is a sophisticated collection of computer programs designed to help users find information on the Web by indexing and organizing vast amounts of digital content. Key components typically include:

    • Crawlers (or spiders): These programs autonomously browse the Web, following links and collecting data from web pages.

    • Indexers: They process the data collected by crawlers, creating a searchable index of keywords and their locations within web pages, similar to an index in a book.

    • Query processors: These interpret user queries, search the index, and return relevant results, often ranking them by complex algorithms.

  • The necessity for search engines arose from the massive explosion of web-based digital information globally, making it virtually impossible for any single institution or individual to manually organize or navigate. Before their advent, finding specific information on the internet was a tedious and often fruitless task, usually relying on meticulously curated directories.

  • There are estimated to be over 4.5 billion4.5\text{ billion} web pages on the World Wide Web, and this number is constantly growing, underscoring the vital role search engines play in making this information accessible.

Origin of Google

  • The Google search engine was created by two Stanford graduate students, Larry Page and Sergey Brin, who met in 1995 while pursuing their Ph.D.s in computer science.

  • According to Google lore, they initially disagreed on many things during their first meeting when Brin was showing Page around Stanford, but they soon found common ground in their passion for extracting relevant information from the web.

  • Their first search engine was called BackRub, a name derived from the importance of "backlinks" or incoming links from other websites. Page and Brin theorized that the more websites that linked to a particular page, the more important or authoritative that page must be. This innovative approach to ranking web pages, known as PageRank, was a fundamental departure from existing search engines that primarily relied on keyword density.