Scalability at YouTube by J.J. Behrens and Mike Solomon of Google¶
Presenters: J.J. Behrens and Mike Solomon of Google
PyCon 2012 presentation page: https://us.pycon.org/2012/schedule/presentation/315/
Video running time: 38:43
First few minutes of the video are marketing fluff.
(10:16) The talk really starts.
People often confuse scalabity and efficiency.
- simple - “If I could give you one word about scalable systems, I think the word would be simple“
- solve the problem at hand
- product of evolution
The Tao of Youtube¶
“choose the simplest solution with the loosest guarantees that are practical”
- divide and conquer - e.g.: independent web servers, database sharding, partition the problem, simple and loose connections
- approximate correctness - cheat and let people see their own comments rather than requiring immediate consistency
- expert knob twiddling - consistency, durability
- jitter - introduce more randomness, because surprisingly, things tend to stack up - example: cache expirations - if all caches expires at the same time => thundering herd. Add some random variance.
Jitter puts entropy back into the system.
- well defined inputs
- well defined dependencies
- frequently end up behind an RPC
- leverage local optimizations
- uncorrelated with scalability
- focus on algorithms first
- learn to measure - use representative samples
- philosophy vs. doctrine
- more conventions - less documentation
- more effective collaboration
- more diffusion of responsibility