I have collected the examples we coded during the “Python 2.5 generators”
session in subversion on SourceForge. Specifically: the code of last
saturday is here:
http://weightless.svn.sourceforge.net/viewvc/weightless/trunk/examples/
I refactored/renamed some stuff as to make it easier to understand. Also, I
added two other versions of wordLength() (was lineLength, but it actually
worked on words, not on lines ;-) that use list comprehensions and generator
expressions. For if you are interested in more of the wonderful features of
Python!
There is more documentation on Weightless and a link to SF on
http://weightless.io/weightless
There are no file releases yet as this is a work in progress, although parts
of it are being used in production code. Parts that are usable are:
compose (demonstrated 4th october)
acceptor
server
observable (how to create dynamic pipelines, for next Python day?)
Experimental are:
gio (Generator I/O, this really is the goal!)
httpprotocol
I am working on them.
Best regards
Erik
Weightless I/O
General Info
Welcome to the world of asynchronous lightweight easy-to-program networking. Weightless presents a way for I/O driven applications to write them using generators or coroutines instead of with asynchronous callbacks or threads.
Weightless consists of:
* a Reactor/Acceptor implementation
* Gio: translates Reactor callbacks to generators/coroutines
* Compose: allows generators decomposition, just you do with functions
* Pipelines: Allows connecting generators as (Jackson) pipelines
* DNA: supports dynamically configured pipelines with composition
* HTTP Library: a HTTP implementation based on Gio
This site is dedicated to the Weightless I/O open source software package. It provides technical information on why Weightless I/O exists and how it works, and how it will evolve in the (near) future.
Weightless I/O is created by Erik J. Groeneveld, contact him at erik at cq2.org.
The HTTP library is in use by Meresco Core.
This site is last updated on October 2nd, 2008.
Sources can be found on SourceForge.net Logo
Weightless at Conferences
Software Practice Advancement Conference
Program Decomposition with Python Generators based on Weightless’ compose() has been presented and explored on the Software Practice Advancement 2008 conference. For the resuls, look here.
Agile Open conference
On Agile Open 2008 the notion on decomposition with generalized generators was introduced in an Open Space session.
Python Community Day
Many parts of Weightless are presented by Klaas and Erik on Python Community Day 2008
Theory
For the reader interested in theory, there are but a couple of papers describing some aspects of Weightless. They are mentioned below and shortly commented on for their relevance for Weightless.
1. JSP in perspective - A short and well readable 1975 paper by Jackson himself about structuring programs after the data structures they process. It’s relevance lies in the way it avoids intermediate buffering (tapes at that time, memory nowadays) by control flow inversion. This is exactly what Weightless does.
2. The Problem With Threads - About why threads seem attractive at first sight (ignorant sequential programming) and why and how it bites you hard sooner or later (pre-emptive concurrent threads, locking, …) and how must overhead there actually is involved in switching threads and do the blogkeeping, and about deadlock and…. so on. It just explains very well why Weightless is about avoiding threads.
Related Work
Before diving into Weightless, lets first review other alternatives, initiatives and approaches. Below is a list, with a few short comments on them.
1. Spasmoidal - a very abstract task-dispatcher based on Python 2.5 generators which features a Pollster (Reactor) and a Socket Acceptor. It clearly suffers lack of program decomposition with generators, as the Spasmodoidal Tasks are very long single generators. Doug’s approach is very generalized (it’s more than only I/O) and I think that Doug Fort and I can learn a lot from each other!
2. Twisted - a popular asynchronous networking library that contains many useful ideas. I have had the most experience with this library, and learned how not to solve common problems. It probably is the most popular one in the Python scene, but, as often happens with software once accepted by broader public, it suffers from an old and insufficient architecture, which hinders development. As an example, the Twisted folks seem to talk about but never implement sendfile funtionality. Pitty we had to abandon this one.
3. Asynchio created by Vladimir Sukhoy contains some very interesting idea’s. It implements the Asynchronous Completion Token pattern (Proactor) and it is truely asynchronous. It is a library that focusses on I/O and not on generators, and it is clear that the interface is a bit complicated to create networking programs every day. It might be more appropriate to (re)use parts of it under the Weightless cover.
4. Peers
5. Active Objects Only a pattern, not a real implementation. Interesting though.
6. Thread Safe MultiQueue
7. Continuations Continuations! Yes! Though this is very interesting, the main reason why Weightless explores generators, is the fact that generators can make event drivven programming as easy (in fact: easier, if you take the problems of threading that appear later into account) as writing a sequential piece of code in a threaded environment. Continuations, I believe - unless properly embedded in langauges such as Smalltalk - will not likely make it because it is too abstract for many human beings (at least me) to understand. But I can’t deny George Bauer’s work is cool!
8. Generator Coroutine Access to Its Own Handle (from Doug Fort’s Spadmoidal) describes a solution to a common problem you’ll hit when diving into this subject: a ’self’ or ‘this’ for generators. Only a pattern description, the implementation is in Spadmoidal. And there is a much better way to do it, btw: use reflection to get ‘this’ from the callstack with a short and efficient this() method.
9. MultiTask - This is the only other initiative I have found that really solves the decomposition problem and it combines generators with I/O; two problems Weightless addresses as well. Very neat solution and good implementation!
10. Cogen
11. Tramplining Seems to be gone.
Why Weightless?
You might wonder why there is yet another asynchronous networking package, this time called Weightless. Aren’t there enough initiatives already? Yes, there are, but Weightless is not about improving those, but about exploring the latest developments in Linux and Python, and to see where this gets us. It gets us… 35 years back in time. See the blog about Jackson Structured Programming (JSP). Looking at specific advancements in Linux and Python soon reveils that these call for a completely different software design: Python finally lets us write JSP programs, and Linux now lets us do I/O without copying and buffering all the time. Below, I will discuss the advancements in Linux and Python Weightless aims at.
Linux: Asynchronous I/O with splice and tee
Recently, the Linux kernel 2.6.17 got three very interesting new systems calls:
1. splice() - moves data from a pipe to a socket or v.v.
2. vmsplice() - move a memory page from userspace to kernel space without copying it.
3. tee() - copies data from one pipe to another without actually copying it.
Many network cards and their drivers are already capable of gathering data to be send from pages randomly scattered in memory. So the kernel can send data to network cards without the need for physically copying the data. Together with the three new system calls, it becomes possible to perform zero-copy networking. What that means is that your simple laptop computer becomes able to replace heavy weight network servers running hundreds of requests per second.
Python Generalized Generators or Co-routines
This piece has it’s own blog. It goes into details about JSP and the relation to Weightless. It describes co-routines and how typical problems are solved with them and how to do that in Python.
Weightless parts
Compose
The problems introduced in the blog about Python Co-routines are solved with a function called ‘compose’. It is described separately in this blog.
Asynchronous I/O
Reactor/Acceptor
Weightless features a Reactor and a Acceptor, just as the widely know patterns described them. There is not must special about them, except for the fact that they are carefully refactored (in fact redesigned a dozen times) to match lessons learned during its use. The most notable results of this process are:
1. The reactor uses synchronous sockets only. None of the calls to a socket will block however, because the reactor only calls a potentially blocking operation when the reactor knows the conditions are such that it won’t block.
2. The reactor is a normal object and an application program must nstantiate it and explicitely pass it to other object which need it.
3. All threads are abandoned from the reactor. Tests are application code can step through the reactor by calling step().
4. The reactor is not thread safe. It is not possible to modify the reactor’s state from another thread and then wake it up, or otherwise signal it.
5. All modifications done from within a callback called by the reactor are safe and behave sequentially, as expected.
6. The reactor includes a timer, since timing with sleep() is not possible of course.
7. The reactor does not interfere with signals and with other (synchronous) I/O since it is the applications programmer responsibility to choose whatever I/O he/she wants to use (or wants to live with).
8. The reactor allows the use of splice() or tee()
Gio
The Gio part of Weightless connects the Reactor and Acceptor to generators and it implements a simple open() call to open sockets as a client. This part is under development.
HTTP Library
There are two simple applications that use the reactor/acceptor: HTTP Server and HTTP Reader. The both work with python 2.4 generators - and will continue to do so. A python 2.5 variant with be based on Gio.
