The Python Object Model

At the core of the Python object model are two types: type and object. They have a curious relation: type is derived from  object, and object is an instance of type. In fact, type is also an instance of type. To look through the object model, we are going to walk through the process of creating a new class step by step, by invoking the type method (Python class definitions are just syntactic sugar for this):

• We’ll invoke T = type(“T”, (object,), {}), creating a new type named “T”, derived from object, and with no new methods
• type is an object, so we will invoke the __call__ operator. Operators are always looked up on the object’s class, which in this case is type itself.
• Therefore, we will be invoking __getattribute__ on type for __call__. type defines a __call__ method, so __getattribute__ doesn’t have to do any complex lookups.
• __call__ is a function, and all functions are descriptors, so it is time for an interlude…

Descriptors

Descriptors are also objects. The most well known type of descriptor is that returned by property (property is actually a class!), but classmethod and staticmethod also return descriptors. Descriptors come in two types:

• Non-data descriptors, which implement the __get__ method, and which are only invoked if a member of the object’s class, and are overriden by members on the object’s instance dictionary
• Data descriptors, which implement the __set__ method, override accesses to the instance dictionary

If a descriptor is looked up, then the method corresponding to the action will be invoked

• __getattribute__ -> __get__
• __setattr__ -> __set__
• __delattr__ -> __del__

So, a function is a descriptor. It’s irrelevant to this case, but it is actually a non-data descriptor. So…

• __get__returns a bound method (Ever wondered where those were created? Now we know!), which fills in the self parameter for calls
• Its now time to call the bound method. Now, as functions (and bound methods) are actually objects, we could follow the above again. However, it is easy to see how this could result in infinite recursion; fortunately, this doesn’t happen: Python recognizes that we are calling a function, and we don’t have to repeat the above
• Now we begin standard class construction: we will invoke type.__new__(type, (object,), {}) followed by a do-nothing type.__init__ method.

Constructing a new class is pretty simple: We create a new object with its instance dictionary set to the members dictionary passed as type’s third argument.

Making an instance of an object

Admittedly, we made an instance of an object above; but the case of creating a new type is rather different from the normal (because type is its own type). So, lets create a new instance of the type T we created above

• We’ll do O = T()
• T is again an object, so we will invoke __call__ again. Again, T is specifically a type, so we will be invoking type.__call__(T)
• type.__call__(T) will look up T.__new__. T doesn’t define a new method, so we will look in the next type in the method resolution order. Since T singly inherits (from object), we will be looking up __new__ in object.
• object.__new__(T) will create a new instance of T, and set it up as a normal Python object (as you might expect from object.__new__(T))
• type.__call__(T) will next invoke object.__init__(T), for the same reasons. Object’s __init__ method does nothing, so nothing of interest happens here
• type.__call__(T) will return the newly created object

Summarizing it all

We can see that, from a couple of rules, we can derive the Python object system:

• We can separate lookups  into two types: operator lookups and normal lookups
• Operator lookups always look up on the type of the object
• Normal lookups look up first on the object, and then on the type
• Lookups always involve invoking the __getattribute__ operator
• If a data descriptor is found during a lookup, it is always invoked
• If a non-data descriptor is found during a lookup, it is only invoked if it was found somewhere but the instance dictionary of the object being interrogated

At its core, the Python object model is incredibly simple (excluding some complexity around descriptors, which is, in my opinion, justified because it makes using it more inituitive). It is, however, somewhat hard to inspect from outside; the core workings of it are not particularly brilliantly documented and are implemented in C, and some of the interactions are hard to identify without re-implementing it yourself (In fact, much of this was discerned by reimplementing it in Lua)

Its not a simple process by a long shot. There is a long road ahead of us.

But we just moved one small – but oh so very important – step closer. EForge is moving to its first release, 0.5.

What is EForge today?

OK, firstly: EForge is not a complete replacement for any of those systems yet. So far, it is most of the way to replacing Trac or Redmine. EForge offers

• A project wiki, with history tracking (as all good wikis must have)
• A pretty competent issue tracker. Its still missing some features – most notably sorts – but its getting there.
• A source code browser for Git or Hg repositories. There are a few features missing, but the browser is already a good way of looking around your repository

Now, I should point something out: 0.5 is not really for production use. We expect that to happen more around the 0.6 or 0.7 time frame. 0.5 is just a preview release; a stabilised version for people to have a play around with.

Now, that said: EForge already works well enough for us to dogfood ourselves on it.

Where are we going?

There are a number of features we intend to add soon:

• Gitosis/Gitolite style user management. This is the “Common SSH account, various keys” system that you may be familiar with from services like Gitorious and GitHub
• Sorting in the bug tracker
• Subprojects
• Integration with Django’s sites framework (i.e. projects can be assigned to sites – so you can have eforge1.com and eforge2.com and they’ll be fully interlinked)
• User private clones of projects
• Gitorious style merge requests
• probably other things too

Helping out

If you have any features you want – please log them on our tracker. If you want the source – Git instructions are on the wiki. If you’d like to contribute – either post a patch to the tracker, or set up a Git repository somewhere and point us to it.

Its still early days, and there is a lot of work ahead of us.

But a massive milestone has been reached. The future is bright.

Its even better when they work everywhere at the same efficiency – that is, whether placed in the process’ local memory, or a shared memory segment. And its such a shame that nobody has implemented them outside of Linux.

So, what is a Futex?

A futex is actually surprisingly simple. Creating one doesn’t require any system calls, and is in fact most trivial.

A futex is just an aligned, normally 32-bit, integer

Its truly quite simple. Now, this isn’t the whole story: you can’t implement an efficient mutex on just an integer. Its just that the other involved structures (such as wait queues) only get involved when someone needs to wait for the futex. Until then, it is just a piece of memory that someone allocated

(In case you’re wondering how they work in shared memory: Futexes are keyed upon their physical address)

If its so simple, how do I use one?

A futex is just a counter. In fact, it can be said that a futex supports just two operations: “down” and “up”; which decrement and increment it respectively. To begin this illustration, we are going to begin with the semaphore, which is nothing more than a generalisation of a mutex.

To begin, allocate your futex somewhere, and set its value to the maximum value of the semaphore. In other words, if you want to be able to acquire the Semaphore 3 times, set it to 3. Obviously, for a mutex you would set it to 1.

Now, to acquire the semaphore: Perform an atomic decrement and fetch. Look at the value you just set the semaphore to; if this is positive or zero, you succeeded, if it is negative then ask the kernel to block you, and we will get to that in a moment.

How about releasing the semaphore? This is also simple; do the opposite: an atomic fetch and increment. Again look at the value returned; if it was positive or zero, you’re done; and otherwise you need set the counter to one, then ask the kernel to wake people up.

Getting the Kernel involved

OK, so far we have been working entirely in user space; but we can’t have an efficient mutex without the kernel. Lets get on to that.

Firstly, blocking; blocking is done by the FUTEX_WAIT method. You pass the address of the futex and the value the decrement operation returned to the kernel, and it atomically checks that the futex still has the value you passed and blocks you

This check is required in order to avoid a race condition: Another process ups the futex before you get into the kernel. In the case the value has changed, you must spin around and retry to acquire again.

Once you have been unblocked, you need to attempt to lock the futex again.

Now, unblocking: this is done by the FUTEX_WAKE method. You pass the futex address, and the number of threads you wish to wake; 1 is obviously a good value most of the time (and should also be most fair).

There, that was simple enough, wasn’t it?

Expanding: Condition Variables

OK, that was pretty simple; but what about condition variables? They can be implemented in a rather simple manner too:

• Start the Futex at 0
• Down it each time a thread waits on it
• Up it each time you want to wake a single thread
• Exchange the current value for 0, then wake up -the_previous_value processes to wake up everyone

See? Those are pretty simple too!

In closing: The issues, and how to make them better

As you can see, Futexes are both delightfully simple, and beautifully efficient. However, there is an area I can see for improvement. It also corrects what I call a hole in POSIX: Or, that mutexes cannot be converted to file descriptors (To wait on using mechanisms like select).

(In fact, it is interesting to note that there was in the past an attempt to add  this feature, FUTEX_FD, but it was racy).

I would add a single method, FUTEX_WAIT_FD. The behaviour of this is as follows:

FUTEX_WAIT_FD will atomically compare the value of the futex with the previous value (as passed in a parameter), and build a file descriptor. The returned file descriptor is read only; attempts to write to it will return an error. When another thread or process calls FUTEX_WAKE on the futex, this will cause the file descriptor to become readable; reading it will return the number of waiters that FUTEX_WAKE was asked to wake up. Once this file descriptor has become readable, the thread must re-attempt to lock the futex in the same way as is done when woken from FUTEX_WAIT. After the file descriptor has become readable, it will not be activated again and must be closed.

I can also see other possibilities for improving the futex system (and other locking systems in general); obviously, they should be adapted to the operating system that is using them.

Further Reading

• man futex(2)
• man futex(7)
• Fuss, futexes and furwocks: Fast Userlevel Locking in Linux (PDF, by Hubertus Franke, Rusty Russell, Matthew Kirkwood)
• Futexes are tricky (PDF, by Ulrich Drepper, going into how to use them, and how Glibc uses them, in depth)

And that is that, unfortunately, UDI has been almost completely ignored

Now, I’m going to admit something straight away: UDI is not perfect. UDI has its flaws; one that has been most commonly levelled at it is that it is complex, and indeed that is true. Complexity is never great, but it is acceptable if it brings features, and it does

However,  this is not the main reason people rejected it. They had other ones

“Why should I change my interface when it benefits my competitors?”

This one often came from the Unix companies (And Novell, with regards to Netware). And it is kind of true: If you support UDI, then your competitors will benefit from drivers written from you OS; but the effect is reciprocal: you will also benefit from drivers written from their OS.

It gets silly, however, when they claim “We have leading hardware vendor support! This will benefit my competitors more than me”, because there is only one company who can reasonably claim that – or who could reasonably claim it any time in the foreseeable past and present.

But then, how many of the Unix vendors are still standing today on the basis of their Unix offerings?

“UDI will be slower than my native interface”

OK, maybe. However, know this: UDI was designed to be fast and scalable. UDI pretty much forces you to write scalable drivers. Any benefit you can get over UDI will be small.

And, indeed, when SCO¹ implemented UDI, as a wrapper on top of their native interface, they found it was faster than their native drivers. This certainly sounds counter-intuitive: the thing to take away from this is that UDI has low overhead and produces fast drivers.

“UDI will not work with my operating system design”

Maybe, but unlikely. UDI was designed to operate on anything from a single tasking, single threaded single CPU system, to a multi core, multi machine, multi threaded distributed system. You would have to be developing something massively radical to design a system in which UDI could not work

“UDI doesn’t have many drivers”

This is a chicken and egg problem and you know it. However, for any operating system which wants to use UDI, there are UDI enthusiasts (of which I am one) which would be most willing to help anyone trying to implement it – and even contribute to the work.

Remember: Every operating system supporting UDI makes it more attractive to other operating systems and driver writers. Someone must just take the plunge and support it.

“UDI means closed source drivers”

This one comes primarily from the FSF, but then much of what the FSF says I disagree with. OK, I’ll admit: You would likely get closed source drivers – but then you already do. Have you ever seen the source of ATi or nVIDIA’s graphics drivers? And, of course, most devices have open datasheets anyway – so even if the vendor didn’t open source their drivers we would be stuck.

In fact, I see very few cases where the  current state of affairs have caused a driver’s source code to be released – most drivers are developed by unrelated developers, and the same benefits of delegating much of the ongoing maintenance work to the project’s volunteers would apply.

However, there is a very, very compelling reason to use UDI, which I feel outweighs this:

UDI gives us better quality systems

This applies from two points:

• Driver maintenance gets shared between bigger pools of people, increasing the overall quality of drivers
• Operating systems developers can focus more on the bits of their OS which differentiate it, since they are no longer spending as much time on maintaining drivers

UDI gives us more choice

At present, in descending order of hardware support, you have

• Windows
• Linux
• The BSDs, Solaris, etc

with enormous gulfs between them. This is constricting: It means that people are often forced to choose one because of their hardware, and denies users choice.

UDI makes closing these gulfs much, much easier, because we reduce duplication of effort.

UDI benefits everyone

Where can I get more information?

The UDI core specification can be found at projectudi.org. However, this website hasn’t been updated since 2001. You may think that this makes UDI obsolete, but you would be wrong: Software and good engineering doesn’t rust. Daily we use technologies, such as the SCSI command set and x86 and ARM processor architectures, and operating systems, which are much older. Many of these have been modified substantially, true; but the core design concepts remain mostly unchanged since their inception.

Deven Corzine said similar things about UDI as I just did a while ago.

A group of us interested in reviving UDI have setup a website at udi.io and associated mailing lists and other resources upon which to conduct the discussion with regards to taking UDI forwards. The effort is new, yes, but the discussion that has proceeded it has been going on for a year, and involved some of the specification developers.

The UDI project is dead; long live the UDI project.

Things were nice and simple; all characters were the same size, and that size was 8-bits. Things were easy to handle, for the most part. Much of C’s string handling is rooted in this era, with functions like isalpha inexorably tied to the English language.

Of course, since then the world has changed.

Things got ugly: Multi-byte character sets

Now, these aren’t all bad; many can be considered, for most things, like the one byte per character sets that preceded them. They maintain the invariant that the contents of a multi-byte character cannot be interpreted as a single byte character. This does of course make your coding take more space, but makes legacy programs work.

Some encodings, however, are not so kind. A good example of this is Shift-JIS. Shift-JIS reuses the single byte character space in the second bytes of multi-byte characters. This means that, if you do strchr(‘e’), you might not actually find an E.

Now things are getting really messy, because C doesn’t provide standard functions for dealing with these character sets. The other problem is that dealing with the  thousands of character sets in existence is rather complex.

The solution: Unicode

Unicode, as you probably know, provides an incredible repertoire of characters, and is capable of representing pretty much every character in common (and uncommon) use.

Unfortunately, there is a great big problem.

C’s Unicode support sucks

C99 doesn’t actually define any official support for Unicode; it defines support for “Wide Characters”, which every implementation defines to be Unicode because this is the only sane option.

The problem is that the C standard requires the wide character set to be a fixed width encoding – which requires UCS-4/UTF-32, but nobody else uses UTF-32.

Worse,  C’s character handling functions (quite rightfully) only provide a small set of features – and so everybody ignores them.

You see, everyone has their own opinion as to which Unicode encoding to use:

• Unix – UTF-8 for backwards compatibility with traditional APIs
• Windows – UTF-16 (Predates Unicode 2.0)*
• Java – UTF-16 (Predates Unicode 2.0)
• ICU – UTF-16

The end result is that the world’s most popular programming languages (C and C++) are left with Unicode support that is quite frankly useless.

The Future

C++0X doesn’t touch on the topic, but C1X saves our bacon. C1X adds the types char16_t and char32_t, which quite natively map on to UTF-16 and UTF-32. It also allows creating character constants in those encodings, which will vastly simplify use of them.

It will also make use of libraries like ICU feel a lot less like stabbing ones self in the eye.

In fact, there are only two major issues I see with C1X’ Unicode support: Firstly, there is no convertibility defined to wchar_t. Secondly, and probably as a consequence of the former, there is no support provided for outputting Unicode strings to the console. Perhaps with time this will change.

* Microsoft defines wchar_t to be 16-bits. This is definitely non-compliant, but wasn’t when they did it: at the time 16 bits covered the entirety of Unicode. Changing it now, of course, would be impossible, as pretty much the entire Windows API is built upon passing around wchar_t`s.

[Correction 2010-07-07: Mistake in my recollection of C++0x removed - C++0x does in fact add char16_t/char32_t.]