👉 Watch the video: https://www.youtube.com/watch?v=WiQqqB9MlkA 👈
Elegant Solutions for Everyday Python Problems - Pycon 2018
Are you an intermediate python developer looking to level up? Luckily, python provides us with a unique set of tools to make our code more elegant and readable by providing language features that make your code more intuitive and cut down on repetition. In this talk, I’ll share practical pythonic solutions for supercharging your code.
Specifically, I'll cover:
What magic methods are, and show you how to use them in your own code.
When and how to use partial methods.
An explanation of ContextManagers and Decorators, as well as multiple techniques for implementing them.
How to effectively use NamedTuples, and even subclass and extend them!
Lastly, I'll go over some example code that ties many of these techniques together in a cohesive way. You'll leave this talk feeling confident about using these tools and techniques in your next python project!
2. slides: h!p://bit.ly/elegant-python
This talk is for you if:
You're an intermediate python programmer
You're coming to python from another language
You want to learn about language features like: magic
methods, iterators, decorators, and context managers
@nnja
7. Magic methods start and end with a double underscore
(dunder)
By implementing a few straightforward magic methods,
you can make your objects behave like built-ins such as:
numbers
lists
dictionaries
and more...
@nnja
13. >>> soda_cost = Money('$', 5.25)
>>> pizza_cost = Money('€', 7.99)
>>> print(soda_cost + pizza_cost)
$14.33
More on Magic Methods: Dive into Python3 - Special Method
Names
14. >>> soda_cost = Money('$', 5.25)
>>> pizza_cost = Money('€', 7.99)
>>> print(soda_cost + pizza_cost)
$14.33
>>> print(pizza_cost + soda_cost)
€12.61
More on Magic Methods: Dive into Python3 - Special Method
Names
15. some magic methods map to symbols
>>> d = {'one': 1, 'two': 2}
>>> d['two']
2
>>> d.__getitem__('two')
2
@nnja
16. other magic methods map to built-in functions
class SquareShape:
def __len__(self):
""" Return the number of sides in our shape """
return 4
>>> my_square = SquareShape()
>>> len(my_square)
4
@nnja
18. Making classes iterable
In order to be iterable, a class needs to implement
__iter__()
__iter__() must return an iterator
In order to be an iterator a class needs to implement
__next__() which must raise StopIteration when
there are no more items to return
Great explanation of iterable vs. iterator vs. generator
19. Scenario..
We have a Server instance running services on
different ports.
Some services are active, some are inactive.
When we loop over our the Server instance, we only
want to loop over active services.
@nnja
21. class IterableServer:
def __init__(self):
self.current_pos = 0
def __iter__(self):
# can return self, because __next__ implemented
return self
def __next__(self):
while self.current_pos < len(self.services):
service = self.services[self.current_pos]
self.current_pos += 1
if service['active']:
return service['protocol'], service['port']
raise StopIteration
@nnja
22. class IterableServer:
def __init__(self):
self.current_pos = 0
def __iter__(self):
# can return self, because __next__ implemented
return self
def __next__(self):
while self.current_pos < len(self.services):
service = self.services[self.current_pos]
self.current_pos += 1
if service['active']:
return service['protocol'], service['port']
raise StopIteration
@nnja
23. >>> for protocol, port in IterableServer():
print('service %s on port %d' % (protocol, port))
service ssh on port 22
service http on port 80
loops over all active services ... not bad
@nnja
24. tip: use a generator
when your iterator doesn't need to
maintain a lot of state
(which is most of the time)
@nnja
25. class Server:
services = [
{'active': False, 'protocol': 'ftp', 'port': 21},
{'active': True, 'protocol': 'ssh', 'port': 22},
{'active': True, 'protocol': 'http', 'port': 21},
]
def __iter__(self):
for service in self.services:
if service['active']:
yield service['protocol'], service['port']
@nnja
26. class Server:
services = [
{'active': False, 'protocol': 'ftp', 'port': 21},
{'active': True, 'protocol': 'ssh', 'port': 22},
{'active': True, 'protocol': 'http', 'port': 21},
]
def __iter__(self):
for service in self.services:
if service['active']:
yield service['protocol'], service['port']
@nnja
27. Why does this work?
use single parenthesis ( ) to create a generator
comprehension
^ technically, a generator expression but I like this term better, and so does Ned
Batchelder
>>> my_gen = (num for num in range(1))
>>> my_gen
<generator object <genexpr> at 0x107581bf8>
@nnja
28. An iterator must implement __next__()
>>> next(my_gen) # __next__() maps to built-in next()
0
and raise StopIteration when there are no more elements
>>> next(my_gen)
... StopIteration Traceback (most recent call last)
see itertools for working with iterators
30. alias methods
class Word:
def __init__(self, word):
self.word = word
def __repr__(self):
return self.word
def __add__(self, other_word):
return Word('%s %s' % (self.word, other_word))
# Add an alias from method __add__ to the method concat
concat = __add__
@nnja
31. We can add an alias from __add__ to concat because
methods are just objects
>>> # remember, concat = __add__
>>> first_name = Word('Max')
>>> last_name = Word('Smith')
>>> first_name + last_name
Max Smith
>>> first_name.concat(last_name)
Max Smith
>>> Word.__add__ == Word.concat
True
@nnja
32. Dog class
>>> class Dog:
sound = 'Bark'
def speak(self):
print(self.sound + '!', self.sound + '!')
>>> my_dog = Dog()
>>> my_dog.speak()
Bark! Bark!
read the docs
35. Example: command line tool with dynamic commands
class Operations:
def say_hi(self, name):
print('Hello,', name)
def say_bye(self, name):
print('Goodbye,', name)
def default(self, arg):
print('This operation is not supported.')
if __name__ == '__main__':
operations = Operations()
# let's assume error handling
command, argument = input('> ').split()
getattr(operations, command, operations.default)(argument)
read the docs
36. Output
$ python demo.py
> say_hi Nina
Hello, Nina
> blah blah
This operation is not supported.
✨
additional reading - inverse of getattr() is setattr()
37. functools.partial(func, *args, **kwargs)
Return a new partial object which behaves like func
called with args & kwargs
if more arguments are passed in, they are appended
to args
if more keyword arguments are passed in, they extend
and override kwargs
read the docs on partials
38. functool.partial(func, *args, **kwargs)
# We want to be able to call this function on any int
# without having to specify the base.
>>> int('10010', base=2)
18
>>> from functools import partial
>>> basetwo = partial(int, base=2)
>>> basetwo
<functools.partial object at 0x1085a09f0>
>>> basetwo('10010')
18
read the docs on partials
39. functool.partial(func, *args, **kwargs)
# We want to be able to call this function on any int
# without having to specify the base.
>>> int('10010', base=2)
18
>>> from functools import partial
>>> basetwo = partial(int, base=2)
>>> basetwo
<functools.partial object at 0x1085a09f0>
>>> basetwo('10010')
18
read the docs on partials
40. functool.partial(func, *args, **kwargs)
# We want to be able to call this function on any int
# without having to specify the base.
>>> int('10010', base=2)
18
>>> from functools import partial
>>> basetwo = partial(int, base=2)
>>> basetwo
<functools.partial object at 0x1085a09f0>
>>> basetwo('10010')
18
read the docs on partials
42. library I
!
: github.com/mozilla/agithub
agithub is a (poorly named) REST API client with
transparent syntax which facilitates rapid prototyping
— on any REST API!
Implemented in ~400 lines.
Add support for any REST API in ~30 lines of code.
agithub knows everything it needs to about protocol
(REST, HTTP, etc), but assumes nothing about your
upstream API.
@nnja
44. then, start using the API!
>>> gh = GitHub('token')
>>> status, data = gh.user.repos.get()
>>> # ^ Maps to GET /user/repos
>>> data
... ['tweeter', 'snipey', '...']
github.com/mozilla/agithub
45. 404 if we provide a path that doesn't exist:
>>> gh = GitHub('token')
>>> status, data = gh.this.path.doesnt.exist.get()
>>> status
... 404
github.com/jpaugh/agithub
51. When should I use one?
Need to perform an action before and/or after an
operation.
Common scenarios:
Closing a resource after you're done with it (file,
network connection)
Perform cleanup before/after a function call
@nnja
52. Example Problem: Feature Flags
Turn features of your application on and off easily.
Uses of feature flags:
A/B Testing
Rolling Releases
Show Beta version to users opted-in to Beta Testing
Program
More on Feature Flags
53. class FeatureFlags:
SHOW_BETA = 'Show Beta version of Home Page'
flags = {
SHOW_BETA: True
}
@classmethod
def is_on(cls, name):
return cls.flags[name]
@classmethod
def toggle(cls, name, value):
cls.flags[name] = value
feature_flags = FeatureFlags()
@nnja
54. How do we temporarily turn features on and off when
testing flags?
Want:
with feature_flag(FeatureFlags.SHOW_BETA):
assert '/beta' == get_homepage_url()
@nnja
55. Using Magic Methods __enter__ and __exit__
class feature_flag:
""" Implementing a Context Manager using Magic Methods """
def __init__(self, name, on=True):
self.name = name
self.on = on
self.old_value = feature_flags.is_on(name)
def __enter__(self):
feature_flags.toggle(self.name, self.on)
def __exit__(self, *args):
feature_flags.toggle(self.name, self.old_value)
See: contextlib.contextmanager
56. The be!er way: using the contextmanager decorator
from contextlib import contextmanager
@contextmanager
def feature_flag(name, on=True):
old_value = feature_flags.is_on(name)
feature_flags.toggle(name, on)
yield
feature_flags.toggle(name, old_value)
See: contextlib.contextmanager
57. The be!er way: using the contextmanager decorator
from contextlib import contextmanager
@contextmanager
def feature_flag(name, on=True):
""" The easier way to create Context Managers """
old_value = feature_flags.is_on(name)
# behavior of __enter__()
feature_flags.toggle(name, on)
yield
# behavior of __exit__()
feature_flags.toggle(name, old_value)
See: contextlib.contextmanager
58. Note: yield?
from contextlib import contextmanager
@contextmanager
def feature_flag(name, on=True):
""" The easier way to create Context Managers """
old_value = feature_flags.is_on(name)
feature_flags.toggle(name, on) # behavior of __enter__()
yield
feature_flags.toggle(name, old_value) # behavior of __exit__()
See: contextlib.contextmanager
59. either implementation
def get_homepage_url():
""" Returns the path of the page to display """
if feature_flags.is_on(FeatureFlags.SHOW_BETA):
return '/beta'
else:
return '/homepage'
def test_homepage_url_with_context_manager():
with feature_flag(FeatureFlags.SHOW_BETA):
# saw the beta homepage...
assert get_homepage_url() == '/beta'
with feature_flag(FeatureFlags.SHOW_BETA, on=False):
# saw the standard homepage...
assert get_homepage_url() == '/homepage'
60. either implementation
def get_homepage_url():
""" Returns the path of the page to display """
if feature_flags.is_on(FeatureFlags.SHOW_BETA):
return '/beta'
else:
return '/homepage'
def test_homepage_url_with_context_manager():
with feature_flag(FeatureFlags.SHOW_BETA):
assert get_homepage_url() == '/beta'
print('seeing the beta homepage...')
with feature_flag(FeatureFlags.SHOW_BETA, on=False):
assert get_homepage_url() == '/homepage'
print('seeing the standard homepage...')
62. Decorators:
Wrap a function in another function.
Do something:
before the call
after the call
with provided arguments
modify the return value or arguments
@nnja
63. class User:
is_authenticated = False
def __init__(self, name):
self.name = name
Throw an exception if trying to access data only for
logged in users:
def display_profile_page(user):
""" Display profile page for logged in User """
if not user.is_authenticated:
raise Exception('User must login.')
print('Profile: %s' % user.name)
64. def enforce_authentication(func):
def wrapper(user):
if not user.is_authenticated:
raise Exception('User must login.')
return func(user)
return wrapper
the important logic:
def enforce_authentication(func):
def wrapper(user):
if not user.is_authenticated:
raise Exception('User must login.')
return func(user)
return wrapper
@nnja
65. Using enforce_authentication without a decorator:
enforce_authentication(display_profile_page)(some_user)
Or, as a decorator:
@enforce_authentication
def display_profile_page(user):
print('Profile: %s' % user.name)
Now this raises an Exception if unauthenticated:
user = User('nina')
display_profile_page(nina)
66. Problem: lost context using a decorator
>>> display_profile_page.__name__
'wrapper'
>>>> display_profile_page.__doc__
# ... empty
Solution: Use contextlib.wraps
from contextlib import wraps
def enforce_authentication(func):
@wraps(func)
def wrapper(user):
# ... rest of the code
69. By using ContextDecorator you can easily write
classes that can be used both as decorators with @
and context managers with the with statement.
ContextDecorator is used by contextmanager(),
so you get this functionality automatically .
Or, you can write a class that extends from
ContextDecorator or uses ContextDecorator as a
mixin, and implements __enter__, __exit__ and
__call__
@nnja
71. use it as a context manager
with feature_flag(FeatureFlags.SHOW_BETA):
assert get_homepage_url() == '/beta'
or use as a decorator
@feature_flag(FeatureFlags.SHOW_BETA, on=False)
def get_profile_page():
beta_flag_on = feature_flags.is_on(
FeatureFlags.SHOW_BETA)
if beta_flag_on:
return 'beta.html'
else:
return 'profile.html'
72. library I
!
: freezegun
lets your python tests ❇ travel through time! ❇
from freezegun import freeze_time
# use it as a Context Manager
def test():
with freeze_time("2012-01-14"):
assert datetime.datetime.now() == datetime.datetime(
2012, 1, 14)
assert datetime.datetime.now() != datetime.datetime(2012, 1, 14)
# or a decorator
@freeze_time("2012-01-14")
def test():
assert datetime.datetime.now() == datetime.datetime(2012, 1, 14)
read the source sometime, it's mind-bending!
73. NamedTuple
Useful when you need lightweight representations of
data.
Create tuple subclasses with named fields.
@nnja
74. Example
from collections import namedtuple
CacheInfo = namedtuple(
"CacheInfo", ["hits", "misses", "max_size", "curr_size"])
@nnja
75. Giving NamedTuples default values
RoutingRule = namedtuple(
'RoutingRule',
['prefix', 'queue_name', 'wait_time']
)
(1) By specifying defaults
RoutingRule.__new__.__defaults__ = (None, None, 20)
(2) or with _replace to customize a prototype instance
default_rule = RoutingRule(None, None, 20)
user_rule = default_rule._replace(
prefix='user', queue_name='user-queue')
76. NamedTuples can be subclassed and extended
class Person(namedtuple('Person', ['first_name', 'last_name'])):
""" Stores first and last name of a Person"""
def __str__(self):
return '%s %s' % (self.first_name, self.last_name)
>>> me = Person('nina', 'zakharenko')
>>> str(me)
'nina zakharenko'
>>> me
Person(first_name='nina', last_name='zakharenko')
77. New Tools
Magic Methods
make your objects behave like builtins (numbers,
list, dict, etc)
Method ❇Magic❇
alias methods
getattr
@nnja