Scalability at YouTube by J.J. Behrens and Mike Solomon of Google¶
Presenters: J.J. Behrens and Mike Solomon of Google
PyCon 2012 presentation page: https://us.pycon.org/2012/schedule/presentation/315/
Slides: ???
Video: https://www.youtube.com/watch?v=G-lGCC4KKok&list=PLBC82890EA0228306&index=22&feature=plpp_video
Video running time: 38:43
First few minutes of the video are marketing fluff.
(10:16) The talk really starts.
Scalable...systems¶
- simple - “If I could give you one word about scalable systems, I think the word would be simple“
- solve the problem at hand
- product of evolution
The Tao of Youtube¶
“choose the simplest solution with the loosest guarantees that are practical”
Scalable...techniques¶
- divide and conquer - e.g.: independent web servers, database sharding, partition the problem, simple and loose connections
- approximate correctness - cheat and let people see their own comments rather than requiring immediate consistency
- expert knob twiddling - consistency, durability
- jitter - introduce more randomness, because surprisingly, things tend to stack up - example: cache expirations - if all caches expires at the same time => thundering herd. Add some random variance.
- cheat
Jitter puts entropy back into the system.
Scalable...components¶
- well defined inputs
- well defined dependencies
- frequently end up behind an RPC
- leverage local optimizations
Efficiency¶
- uncorrelated with scalability
- focus on algorithms first
- learn to measure - use representative samples
Efficient libraries¶
Productivity¶
- philosophy vs. doctrine
- more conventions - less documentation
- more effective collaboration
- more diffusion of responsibility