Faster ENUM
tl; dr
github.com/QratorLabs/fastenum
pip install fast-enumWhat are enums
(If you think you know that — scroll down to the «Enums in Standard Library» section).
Imagine that you need to describe a set of all possible states for the entities in your database model. You’ll probably use a bunch of constants defined as module-level attributes:
# /path/to/package/static.py:
INITIAL = 0
PROCESSING = 1
PROCESSED = 2
DECLINED = 3
RETURNED = 4
...
…or as class-level attributes defined in their own class:
class MyModelStates:
INITIAL = 0
PROCESSING = 1
PROCESSED = 2
DECLINED = 3
RETURNED = 4
That helps you refer to those states by their mnemonic names, while they persist in your storage as simple integers. By this, you get rid of magic numbers scattered through your code and make it more readable and self-descriptive.
But, both the module-level constant and the class with the static attributes suffer from the inherent nature of python objects: they are all mutable. You may accidentally assign a value to your constant at runtime, and that is a mess to debug and rollback your broken entities. So, you might want to make your set of constants immutable, which means both the number of constants declared and the values they are mapped to must not be modified at runtime.
For this purpose you could try to organize them into named tuples with namedtuple(), as an example:
MyModelStates = namedtuple('MyModelStates', ('INITIAL', 'PROCESSING', 'PROCESSED', 'DECLINED', 'RETURNED'))
EntityStates = MyModelStates(0, 1, 2, 3, 4)
However, this still doesn’t look too understandable: in addition to that, namedtuple objects aren’t really extensible. Let’s say you have a UI which displays all these states. You can then use your module-based constants, your class with the attributes, or named tuples to render them (the latter two are easier to render, while we’re at it). But your code doesn’t provide any opportunities to give the user an adequate description for each state you’ve defined. Furthermore, if you plan to implement multi-language support and i18n in your UI, you’ll find that filling in all the translations for these descriptions becomes an unbelievably tedious task. The matching state values might not necessarily have matching descriptions which means that you cannot just map all of your INITIAL states onto the same description in gettext. Instead, your constant becomes this:
INITIAL = (0, 'My_MODEL_INITIAL_STATE')
Your class then becomes this:
class MyModelStates:
INITIAL = (0, 'MY_MODEL_INITIAL_STATE')
And finally, your namedtuple becomes this:
EntityStates = MyModelStates((0, 'MY_MODEL_INITIAL_STATE'), ...)
Well, good enough, it now makes sure both the state value and the translation stub are mapped to the languages supported by your UI. But now you may notice that the code which uses those mappings has turned into a mess. Whenever you try to assign a value to your entity, you also need not forget to extract value at index 0 from the mapping you use:
my_entity.state = INITIAL[0]or
my_entity.state = MyModelStates.INITIAL[0]or
my_entity.state = EntityStates.INITIAL[0]
And so on. Keep in mind that the first two approaches using constants and class attributes, respectively, still suffer from mutability.
And then Enums come at the stage
class MyEntityStates(Enum):
def __init__(self, val, description):
self.val = val
self.description = description
INITIAL = (0, 'MY_MODEL_INITIAL_STATE')
PROCESSING = (1, 'MY_MODEL_BEING_PROCESSED_STATE')
PROCESSED = (2, 'MY_MODEL_PROCESSED_STATE')
DECLINED = (3, 'MY_MODEL_DECLINED_STATE')
RETURNED = (4, 'MY_MODEL_RETURNED_STATE')
That«s it. Now you could easily iterate the enum in your renderer (Jinja2 syntax):
{% for state in MyEntityState %}
{% endfor %}
Enum is immutable for both member set (you can«t define a new member at runtime, nor can you delete a member already defined) and those member values they keep (you can«t reassign any attribute values or delete an attribute).
In your code you just assign values to your entities like this:
my_entity.state = MyEntityStates.INITIAL.val
Well, clear enough. Self-descriptive. Fairly extensible. That«s what we use Enums for.
Why is it faster?
But the default ENUM is rather slow so we asked ourselves — could we make it faster?
As it turns out, we can. Namely, it is possible to make it:
- 3 times faster on member access
- ~8.5 times faster on attribute (
name,value) access - 3 times faster on enum access by value (call on enum’s class
MyEnum(value)) - 1.5 times faster on enum access by name (dict-like
MyEnum[name])
Types and objects are dynamic in Python. But Python has the tools to limit the dynamic nature of the objects. With their help one can gain significant performance boost using __slots__ as well as avoid using Data Descriptors where possible without significant complexity growth or if you can get benefit in speed.
Slots
For example, one could use a class declaration with __slots__ — in this case, class instances would have only a restricted set of attributes: attributes declared in __slots__ and all __slots__ of parent classes.
Descriptors
By default the Python interpreter returns an attribute value of an object directly:
value = my_obj.attribute # this is a direct access to the attribute value by the pointer that the object holds for that attribute
According to the Python data model, if the attribute value of an object is itself an object that implements the Data Descriptor Protocol, it means that when you try to get that value you first get the attribute as an object and then a special method __get__ is called on that attribute-object passing the keeper object itself as an argument:
obj_attribute = my_obj.attribute
obj_attribute_value = obj_attribute.__get__(my_obj)Enums in Standard Library
At least name and value attributes of standard Enum implementation are declared as types.DynamicClassAttribute. That means that when you try to get a member«s name (or value) the flow is following:
one_value = StdEnum.ONE.value # that is what you write in your code
one_value_attribute = StdEnum.ONE.value
one_value = one_value_attribute.__get__(StdEnum.ONE)# and this is what really __get__ does (python 3.7 implementation):
def __get__(self, instance, ownerclass=None):
if instance is None:
if self.__isabstractmethod__:
return self
raise AttributeError()
elif self.fget is None:
raise AttributeError("unreadable attribute")
return self.fget(instance)# since DynamicClassAttribute is a decorator on Enum methods `name` and `value` the final row of __get__() ends up with:
@DynamicClassAttribute
def name(self):
"""The name of the Enum member."""
return self._name_
@DynamicClassAttribute
def value(self):
"""The value of the Enum member."""
return self._value_
So, the complete flow could be represented as the following pseudo-code:
def get_func(enum_member, attrname):
# this is also a __dict__ lookup so hash + hashtable scan also occur
return getattr(enum_member, f'_{attrnme}_')
def get_name_value(enum_member):
name_descriptor = get_descriptor(enum_member, 'name')
if enum_member is None:
if name_descriptor.__isabstractmethod__:
return name_descriptor
raise AttributeError()
elif name_descriptor.fget is None:
raise AttributeError("unreadable attribute")
return get_func(enum_member, 'name')
We«ve made a simple script that demonstrates the above conclusion:
from enum import Enum
class StdEnum(Enum):
def __init__(self, value, description):
self.v = value
self.description = description
A = 1, 'One'
B = 2, 'Two'
def get_name():
return StdEnum.A.name
from pycallgraph import PyCallGraph
from pycallgraph.output import GraphvizOutput
graphviz = GraphvizOutput(output_file='stdenum.png')
with PyCallGraph(output=graphviz):
v = get_name()
And after we«ve run the script it created this picture for us: 
This proves that each time you access stdlib enum«s attributes name and value it calls a descriptor. That descriptor in turn ends up with a call to stdlib enum«s def name(self) property decorated with the descriptor.
Well, you can compare this to our FastEnum:
from fast_enum import FastEnum
class MyNewEnum(metaclass=FastEnum):
A = 1
B = 2
def get_name():
return MyNewEnum.A.name
from pycallgraph import PyCallGraph
from pycallgraph.output import GraphvizOutput
graphviz = GraphvizOutput(output_file='fastenum.png')
with PyCallGraph(output=graphviz):
v = get_name()
Which outputs this picture: 
That is what is really done inside standard Enum implementation each time you access name and value attributes of your Enum members. And that«s why our implementation is faster.
Python Standard Library«s implementation of Enum class uses tons of descriptor protocol calls. When we tried to use standard enum in our projects we«ve noticed how many descriptor protocol calls for name and value attributes of the Enum members were invoked. And because enumerations were used excessively throughout the code, the resulting performance was poor.
Furthermore, the standard enum class contains a couple of helper «protected» attributes:
_member_names_— a list that holds all the names of enum members;_member_map_— an OrderedDict that maps a name of an enum member to the member itself;_value2member_map_— a reverse dictionary that maps enum member values to corresponding enum members.
Dictionary lookups are slow since each one leads to a hash calculation and a hash table lookup, making those dictionaries non-optimal base structures for the enum class. Even the member retrieval itself (as in StdEnum.MEMBER) is a dictionary lookup.
Our way
While developing our Enum implementation, we kept in mind those pretty C-language enumerations and the beautiful extensible Java Enums. The main features we wanted in our implementation:
- an Enum must be as static as possible; by «static» we mean: If something could be calculated once and at declaration time, it should;
- an Enum can not be subclassed (must be a «final» class) if a subclass defines new enum members — this is true for standard library implementation, with the exception that subclassing is prohibited even if no new members defined;
- an Enum should have vast possibilities for extensions (additional attributes, methods and so on).
The only time we use dictionary lookups is in a reverse mapping value to Enum member. All other calculations are done just once during the class declaration (where metaclasses hooks used to customize type creation).
In contrast to the standard library implementation, we treat the first value after the = sign in the class declaration as the member value: A = 1, 'One' in standard library enum the whole tuple 1, "One" is treated as valueA: 'MyEnum' = 1, 'One' in our implementation only 1 is treated as value
Further speed-up is gained by using __slots__ whenever possible. In the Python data model classes declared with __slots__ do not have __dict__ attribute that holds instance attributes (so you can«t assign any attribute that is not mentioned in __slots__). Additionally, attributes defined in __slots__ accessed at constant offsets to the C-level object pointer. That is high-speed attribute access since it avoids hash calculations and hashtable scans.
What are the additional perks?
FastEnum is not compatible with any version of Python prior to 3.6, since it excessively uses typing module that was introduced in Python 3.6; One could assume that installing a backported typing module from PyPI would help. The answer is: no. The implementation uses PEP-484 for some functions and methods arguments and return value type hinting, so any version prior to Python 3.5 is not supported due to syntax incompatibility. But then again, the very first line of code in __new__ of the metaclass uses PEP-526 syntax for variable type hinting. So Python 3.5 won«t do either. It«s possible to port the implementation to older versions, though we in Qrator Labs tend to use type hinting whenever possible since it helps developing complex projects greatly. And, hey! You don’t want to stick to any python prior to 3.6 since there’s no backwards incompatibilities with your existing code (assuming you are not using Python 2) though a lot of work was done in asyncio compared to 3.5.
That, in turn, makes special imports like auto unnecessary, unlike in standard library. You type-hint all your Enum members with your Enum class name, providing no value at all — and the value would be generated for you automatically. Though python 3.6 is sufficient to work with FastEnum, be warned that the standard dictionary order of declaration guarantee was introduced only in python 3.7. We don«t know any useful appliances where auto-generated value order is important (since we assume the value generated itself is not the value a programmer does care about). Nevertheless, consider yourself warned if you still stick with python 3.6;
Those who need their enum start from 0 (zero) instead of default 1 can do this with a special enum declaration attribute _ZERO_VALUED, that attribute is «erased» from the resulting Enum class;
There are some limitations though: all enum member names must be CAPITALIZED or they won«t be picked up by the metaclass and won«t be treated as enum members;
However, you could declare a base class for your enums (keep in mind that base class can use enum metaclass itself, so you don’t need to provide metaclass to all subclasses): you may define common logic (attributes and methods) in this class, but may not define enum members (so that class won’t be «finalized»). You could then subclass that class in as many enum declarations as you want and that would provide you with all the common logic;
Aliases. We«ll explain them in a separate topic (implemented in 1.2.5)
Aliases and how they could help
Suppose you have code that uses:
package_a.some_lib_enum.MyEnum
And that MyEnum is declared like this:
class MyEnum(metaclass=FastEnum):
ONE: 'MyEnum'
TWO: 'MyEnum'
Now, you decided to make some refactoring and want to move your enum into another package. You create something like this:
package_b.some_lib_enum.MyMovedEnum
Where MyMovedEnum is declared like this:
class MyMovedEnum(MyEnum):
pass
Now. You are ready to begin the «deprecation» stage for all the code that uses your enums. You divert direct usages of MyEnum to use MyMovedEnum (the latter has all its members be proxied into MyEnum). You state within your project docs that MyEnum is deprecated and will be removed from the code at some point in the future. For example, in the next release. Consider your code saves your objects with enum attributes using pickle. At this point, you use MyMovedEnum in your code, but internally all your enum members are still the MyEnum instances. Your next step would be to swap declarations of MyEnum and MyMovedEnum so that MyMovedEnum will now not be a subclass of MyEnum and declare all its members itself; MyEnum, on the other hand, would not declare any members but become just an alias (subclass) of MyMovedEnum.
And that concludes it. On the restart of your runtimes on unpickle stage all your enum values will be re-routed into MyMovedEnum and become re-bound to that new class. The moment you are sure all your pickled objects have been un (re)pickled with this class organization structure, you are free to make a new release, where previously marked as deprecated your MyEnum can be declared obsolete and obliterated from your codebase.
We encourage you to try it! github.com/QratorLabs/fastenum, pypi.org/project/fast-enum. All credits go to the FastEnum author santjagocorkez.
