There’s a stage in a product cycle where you know it’s going to ship. Where you can see the end. It’s right there, sitting at the corner of Emerson St. and University Ave. Or maybe sipping coffee at Fraiche.

Oh, hello, it waves — there. In front of you. The End. (Or, An End.) And seeing this puts you in a special space that when you think about it — think about all the work that it took the team to get there, to bring that end so close — you are overwhelmed with a flood of emotions.

Some of our specific challenges of the denormalization process were:

1. Dozens of legacy data formats that evolved over years. Peter Ondruška, a Facebook summer intern, defined a custom language to concisely express our data format conversion rules and wrote a compiler to turn this into runnable PHP. Three “data archeologists” wrote the conversion rules.

2. Non-recent activity data had been moved to slow network storage. We hacked a read-only build of MySQL and deployed hundreds of servers to exert maximum IO pressure and copy this data out in weeks instead of months.

3. Massive join queries that did tons of random IO. We consolidated join tables into a tier of flash-only databases. Traditionally PHP can perform database queries on only one server at a time, so we wrote a parallelizing query proxy that allowed us to query the entire join tier in parallel.

In essence, personalization is a technology—it’s something that learns from you. So, for example, your technology section and someone else’s will look very different based on your behavior—rather than being the same generic thing that everyone else is seeing.

A Matter of Personas
With Circles, I believe that Google will attempt to accomplish something critics from the blogosphere, academia, SXSW 2010 keynoter danah boyd, privacy watchdogs and others have all called on the social networking world to do: to allow our online communication to respect the same boundaries that our offline social lives do. School and work, friends and family, the sacred and the profane; we’ve always been able to communicate different things to different people in different circumstances. Facebook, Twitter and other online social networks have collapsed all those contexts into one big bucket. We speak to our “friends” all at once, no matter what we might want to say to one group of people or another.