* [The PageRank distribution for the web][12] (no video, short supplement extending the last post)
* [Using your laptop to compute PageRank for millions of webpages][13]
* [Write your first MapReduce program in 20 minutes][14]
* [Using MapReduce to compute PageRank][15]
* [Consistent Hashing][16]
* [A Number-Theoretic Approach to Consistent Hashing][17]
* **To be added:** We’ll put MapReduce on a cluster, use it to compute PageRank, and much more.
#### Data mining and machine learning
* [Introduction to statistical machine translation][18]. Good background is [Peter Norvig’s introduction to spelling correction][19].
* [Implementing statistical machine translation using MapReduce][20]
* **Much more to be added**
### About
Part of what makes Google such an amazing engine of innovation is their internal technology stack: a set of powerful proprietary technologies that makes it easy for Google developers to generate and process enormous quantities of data. According to a senior Microsoft developer who moved to Google, Googlers work and think at a higher level of abstraction than do developers at many other companies, including Microsoft: “Google uses Bayesian filtering the way Microsoft uses the if statement” ([Credit: Joel Spolsky][21]). This series of posts describes some of the technologies that make this high level of abstraction possible. The technologies I’ll describe include:
* **The Google File System:** a simple way of accessing enormous amounts of data spread across a large cluster of machines. Acts as a “virtual file system” that makes the cluster look to developers more like a single machine, and eliminates the need to think about details like what happens when a machine fails.
* **Bigtable:** Building on the Google File System, Bigtable provides a simple database model which can be run across large clusters. This allows developers to ignore many of the underlying details of the cluster, and concentrate on getting things done.
* **MapReduce:** A powerful programming model for processing and generating very large data sets on large clusters, MapReduce makes it easy to automatically parallelize many programming tasks. It is used internally by Google developers to process many petabytes of data every day, enabling developers to code and run simple but large parallel jobs in minutes, instead of days.
Together, these technologies make it easy to run large parallel jobs on very big data sets. Running on top of this technology stack are many powerful data mining and machine learning algorithms. I’ll describe at least two of these:
* Web Crawling
* PageRank
and may describe more, including statistical approaches to spell checking and machine translation, and recommendation algorithms.
In addition to understanding how these technologies work, we’ll also:
* Develop toy implementations of some of the technologies.
* Investigate some related open source technologies, such as Hadoop, CouchDB, Nutch, Lucene, and others.
I’ve never worked at Google. The posts are based on the published literature, especially some key technical publications released by Google describing the Google File System, Bigtable, MapReduce, and PageRank. I’m sure there are some significant differences between the material I will describe, and the current internal state-of-the-art. Still, we should be able to build up a pretty good picture of a data-intensive web company, even if it’s not accurate about Google’s particulars.
**Exercises and problems:** The posts will contain exercises for you to do to master the material, and also some more challenging and often open-ended problems. Note that while I’ve worked on many of the problems, I by no means have full solutions.
**FriendFeed room:** There is a [FriendFeed room][22] for the series.
**Accompanying lectures:** I’m basing a lecture series in Waterloo, Ontario, on the posts. Here’s some details about the lectures.
**Audience:** The audience will include both theoretical physicists and developers; the lectures should be accessible to both sets of people.
**Time:** Lectures will be held every Tuesday, 7:00-8:30pm. The exception is the first lecture, which will be Thursday, December 4, 2008, 7:00-8:30pm.
**Location:** [AideRSS][23] have kindly offered us a room for the course, at 505-180 King Street South, Waterloo – it’s opposite the Brick Brewery, and near the Hospital. _To get into the building, you need someone from AideRSS to let you in. If no one from AideRSS is by the entrance, call (519) 498 1476._
## Background reading
The lectures are based primarily on the following sources:
1.**Original PageRank paper:** “[The PageRank Citation Ranking: Bringing Order to the Web”][24], by Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd (1998).
2.**Textbook on PageRank and related ideas:** “[Google’s PageRank and Beyond: The Science of Search Engine Rankings”][25], by Amy N. Langville and Carl~D.~Meyer (2006).
3.**Overview of Google’s early architecture:** “[The Anatomy of a Large-Scale Hypertextual Web Search Engine”][26], by Sergey Brin and Lawrence Page (1998).
4.**More recent overview of Google’s architecture:** “[Web Search for a Planet: The Google Cluster Architecture”][27], by Luiz Barroso, Jeffrey Dean, and Urs Hoelze (2003).
5.**Original Google File System paper:** “[The Google File System”][28], by Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung (2003).
6.**Original Bigtable paper:** “[Bigtable: A distributed storage system for structured data”][29], by Fay Chang, Jeffrey Dean, Sanjay Ghemawat, Wilson C. Hsieh, Deborah A. Wallach, Mike Burrows, Tushar Chandra, Andrew Fikes, Robert E. Gruber (2006).
7.**Original MapReduce paper:** “[MapReduce: Simplified Data Processing on Large Clusters”][30], by Jeffrey Dean and Sanjay Ghemawat (2004).
Related resources include:
1. [First of five lectures on MapReduce][31].
2. Lecture course on [scalable systems][32] at the University of Washington, covering many similar topics. Videos of the lectures are included.
Many more resources will be posted in the [FriendFeed room][22] for the course.
14 Comments [Leave one →][33]
### Trackbacks and Pingbacks
1. [Michael Nielsen » Lectures on the Google Technology Stack 1: Introduction to PageRank][8]
2. [Michael Nielsen » Lectures on the Google Technology Stack 2: Building our PageRank Intuition][10]
3. [Michael Nielsen » Write your first MapReduce program in 20 minutes][14]
4. [Michael Nielsen » Using MapReduce to compute PageRank][15]
5. [map and reduce @ rvg][34]
6. [Michael Nielsen » Introduction to Statistical Machine Translation][18]
7. [Michael Nielsen » Implementing Statistical Machine Translation Using MapReduce][20]
8. [Michael Nielsen » Consistent hashing][16]
9. [Michael Nielsen » A number-theoretic approach to consistent hashing][35]
10. [Os segredos do Google « Charles Cadé Blog / Comunicação, tecnologia e cultura digital][36]
11. [Link list for october 2009 « Mixotricha][37]
12. [links for 2009-11-02 « Donghai Ma][38]
13. [Data-driven intelligence | Michael Nielsen][39]
14. [Tab sweep January 2014 | Colin Charles Agenda][40]
### Leave a Reply [Cancel reply][41]
Comment
**Note:** HTML is allowed. Your email address will not be published.
[Subscribe to this comment feed via RSS][42]
Name (required) Email (required, will not be published) Website
* ## Follow Michael
* ![][2][ Subscribe to this blog][43]
* ![][44][Subscribe to this blog by email][45]
* [Michael's blog on data-driven intelligence][46]
* ![][47][ Follow Michael on Twitter][48]
* ## Other places on the web
* [GitHub][49]
* [Delicious][50]
* [Delicious research links for book project][51]
* [Delicious research links for last book][52]
* [Google Plus][53]
* ## Books
![][54]![][55]
[Reinventing Discovery: The New Era of Networked Science][56]![][55] [(Errata)][57]
![][58]![][59]
[Quantum Computation and Quantum Information][60]![][59]
* ## Other projects
* [Essays][61]
* [TEDxWaterloo video about open science][62]
* [Quantum computing for the determined][63]
* [Polymath wiki][64]
* ## Recent Posts
* [Is there a tension between creativity and accuracy?][65]
* [Where will the key ideas shaping the future of scientific publishing come from?][66]
* [How the backpropagation algorithm works][67]
* [Reinventing Explanation][68]
* [Neural Networks and Deep Learning: first chapter goes live][69]
* ## Recent Comments
* [Siva][70] on [Is there a tension between creativity and accuracy?][71]
* [Reading into the landscape – drossbucket][72] on [Is there a tension between creativity and accuracy?][73]
* kBite on [Is there a tension between creativity and accuracy?][74]
* Siname on [Is there a tension between creativity and accuracy?][75]
* Florian Seidl-Schulz on [Is there a tension between creativity and accuracy?][76]