Throwing Together Distributed Services With Gevent (Ginkgo) by Jeff Lindsay from Twilio¶
PyCon 2012 presentation page: https://us.pycon.org/2012/schedule/presentation/288/
In this talk we learn how to throw together a distributed system using gevent and a simple framework called Ginkgo (formerly gservice). We’ll go from nothing to a distributed messaging system ready for production deployment based on experiences building scalable, distributed systems at Twilio.
As some have found, gevent is one of the best kept secrets of Python. It gives you fast, evented network programming without messes of callbacks, code that is more Pythonic, and lets you use most regular Python networking libraries and protocol implementations. Now, let’s build on this.
In this talk we learn how to throw together distributed services using gevent and a simple framework called Ginkgo (formerly gservice). We’ll go from nothing to a distributed messaging system based on experiences building scalable, distributed systems at Twilio.
Jeff works at Twilio. Twilio uses a service oriented architecture. Twilio as a high level service is made up of many subservices - sms, voice, client. Each is made up of other services. Mostly written in tornado and gevent. Going to use a framework called Ginkgo.
- services are nested modules that can start, stop and reload
- going to build a scalable gateway around a self-organizing messaging system.
- first we’ll build a simple number client
- next a pubsub service
- then combine both into a gateway service
- and finally make it all distributed
- a basic gevent StreamServer
- for every connection generate a random number, sleep for 60 seconds
- configuration is a python file
- ginkgo has start / stop services
- start the number server in the background
- spawns a greenlet relative to your service
- makes services self-contained
- gets the numbers from the server and puts it in a queue
- uses httpstreamer
- subscription wraps a queue
- posts are publishes, gets are subscribes
- hub is now responsible for managing subscriptions
If you’re in a loop that doesn’t use IO, run gevent.sleep(0) to make sure it yields.
Code for the example is up here: https://github.com/progrium/ginkgotutorial
- Ginkgo - https://github.com/progrium/ginkgo
- Ginkgo tutorial - https://github.com/progrium/ginkgotutorial
- Notes from Andrew Schoen - http://readthedocs.org/docs/andrew-schoen-pycon-2012-notes/en/latest/friday/session_5.html