Scalability at YouTube by J.J. Behrens and Mike Solomon of Google

Presenters: J.J. Behrens and Mike Solomon of Google

PyCon 2012 presentation page:

Slides: ???


Video running time: 38:43

First few minutes of the video are marketing fluff.

(10:16) The talk really starts.


  • scalability
  • efficiency
  • productivity

People often confuse scalabity and efficiency.


  • simple - “If I could give you one word about scalable systems, I think the word would be simple
  • solve the problem at hand
  • product of evolution

The Tao of Youtube

“choose the simplest solution with the loosest guarantees that are practical”



  • divide and conquer - e.g.: independent web servers, database sharding, partition the problem, simple and loose connections
  • approximate correctness - cheat and let people see their own comments rather than requiring immediate consistency
  • expert knob twiddling - consistency, durability
  • jitter - introduce more randomness, because surprisingly, things tend to stack up - example: cache expirations - if all caches expires at the same time => thundering herd. Add some random variance.
  • cheat

Jitter puts entropy back into the system.



  • well defined inputs
  • well defined dependencies
  • frequently end up behind an RPC
  • leverage local optimizations



  • communication through code/data/schema
  • RPC as a means of sanity



  • uncorrelated with scalability
  • focus on algorithms first
  • learn to measure - use representative samples

Efficient libraries


  • wiseguy - YouTube’s fast CGI container
  • pycurl
  • spitfire - YouTube’s templating system; similar to Cheetah
  • serialization formats* - Don’t use pickle

Efficient tools




  • philosophy vs. doctrine
  • more conventions - less documentation
  • more effective collaboration
  • more diffusion of responsibility

Zen proverb


“Do not seek to follow in the footsteps of masters; seek what they sought”

Code Bonsai


  • simple
  • pragmatic
  • elegant
  • orthogonal
  • composable