Omri Tintpulver

We all know that TV today — from a dollars standpoint in terms of advertisers — there’s a lot of dollars there,” David Aufhauser, the conference’s new vice president and general manager of digital media, said in an interview. “But fast-forward five to 10 years down the line, and digital is going to be what holds the cards.

Hitch-22 was a title born of the silly word games we played, one of which was Titles That Don’t Quite Make It, among which were A Farewell to Weapons, For Whom the Bell Rings, To Kill a Hummingbird, The Catcher in the Wheat, Mr. Zhivago, and Toby-Dick, a.k.a. Moby-Cock.

Some of our specific challenges of the denormalization process were:

1. Dozens of legacy data formats that evolved over years. Peter Ondruška, a Facebook summer intern, defined a custom language to concisely express our data format conversion rules and wrote a compiler to turn this into runnable PHP. Three “data archeologists” wrote the conversion rules.

2. Non-recent activity data had been moved to slow network storage. We hacked a read-only build of MySQL and deployed hundreds of servers to exert maximum IO pressure and copy this data out in weeks instead of months.

3. Massive join queries that did tons of random IO. We consolidated join tables into a tier of flash-only databases. Traditionally PHP can perform database queries on only one server at a time, so we wrote a parallelizing query proxy that allowed us to query the entire join tier in parallel.