1. Script to PyPi to Github
@__mharrison__
http://hairysun.com

2. About Me
● 12 years Python
● Worked in HA, Search, Open Source, BI
and Storage
● Author of multiple Python Books

3. Continuums
More like perl-TMTOWTDI

4. Agenda
● Project Development
– Versioning
– Configuration
– Logging
– File input
– Shell invocation
● Environment Layout * virtualenv * pip
● Project layout

5. Agenda (2)
● Documentation
● Automation/Makefile
● Packaging
– setup.py
– PyPi
● Testing
● Github
● Travis CI
● Poachplate

6. Begin

7. Warning
● Starting from basic Python knowledge
● Hands on
– (short) lecture
– (short) code
– repeat until time is gone

8. Project
Create pycat. Pythonic implementation of
cat

9. Scripting

10. hello world cat.py
import sys
for line in sys.stdin:
print line,

11. hello world - Python 3
import sys
for line in sys.stdin:
print (line, end='')

12. 2 or 3?

13. 2 or 3?
● Better legacy/library support for 2
● Possible to support both

14. hello world
import sys
for line in sys.stdin:
sys.stdout.write(line)

15. Assignment
create cat.py

16. Single File or Project?

17. Layout
Can depend on distribution mechanism:
● Single file
● Zip file
– Python entry point
– Splat/run
● System package
● PyPi/Distribute/pip package

18. Single File
● chmod and place in $PATH
● add #!/usr/bin/env python

19. Single File (2)
● No reuse

20. Zip File
● PYTHONPATH=cat.zip python -m
__main__
● tar -zxvf cat.zip; cd cat;
python cat.py

21. Zip File (2)
● No reuse
● No stacktrace

22. System Package
● emerge -av qtile (rpm|apt|brew)

23. System Package (2)
● Requires root
● At mercy of packager (maybe worse than
PyPi)
● Reuse
● Limited to single version
● python -m modulename

24. Pip Package
“Best practice” is combo of:
● Distribute
● Virtualenv
● Pip

25. Pip Package (2)
$ virtualenv catenv
$ source catenv/bin/activate
$ pip install pycat

26. Pip Package (3)
● Multiple versions
● Reuse
● Effort to create setup.py

27. Minimal Example Layout
Project/
README.txt
project/
__init__.py
other.py
...
setup.py

28. Better Example Layout
Project/
.gitignore
doc/
Makefile
index.rst
README.txt
Makefile
bin/
runfoo.py
project/
__init__.py
other.py
...
setup.py

29. Assignment
create layout for
cat.py

30. Semantic Versioning

31. Versioning
http://semver.org
Formal spec for versioning projects

32. Python Versioning
PEP 386
N.N[.n]+[{a|b|c}N[.N]+][.postN][.devN]
1.0a1 < 1.0a2 < 1.0b2.post345

33. setup.py
from distutils.core import setup
setup(name='PyCat',....
version='1.0')

34. Where to store version?
In module.__version__ (might cause
importing issues)

35. Version
If using Sphinx for docs, be sure to update:
docs/
conf.py

36. argparse
ap = argparse.ArgumentParser(version='1.0')
...
ap.print_version()

37. Assignment
Add version to project

38. Configuration

39. ● Python Configuration
– Django (settings.py)
– Python (site.py, setup.py)
● JSON/YAML
– Google App Engine
● Environment Variables
– Python (PYTHONPATH)
● .ini (ConfigParser, ConfigObj)
– matplotlib
– ipython
● Command line options (argparse)
● Sqlite blob
● Shelve/pickle blob

40. Configuration (2)
Are not mutually exclusive

41. Configuration (3)
(Unix) Hierarchy:
● System rc (run control) (/etc/conf.d)
● User rc (~/.config/app/...)
● Environment variables
● Command line options
http://www.faqs.org/docs/artu/ch10s02.html
Filesystem Hierarchy Standard:
http://www.pathname.com/fhs/

42. Configuration (4)
● Plain text config is easily approachable
● Careful with Python config on process
run by root

43. Assignment
Add configuration -n to
show line numbers

44. Logging

45. Logging
logging module provides feature-rich
logging

46. Logging (2)
import logging
logging.basicConfig(level=logging.ERROR,
filename='.log')
...
logging.error('Error encountered in...')

47. Assignment
Add configuration
--verbose to log file
being “catted”

48. Dealing with File Input

49. “Files”
Dealing with?
● Filename
● file object
● string data

50. Filename
Somewhat analogous to an Iterable.
● Can open/iterate many times
● Implementation depends on file
● Need to manage closing file

51. File Object
Somewhat analogous to an Iterator.
● Can iterate once (unless seeked)
● Can accept file, StringIO, socket,
generator, etc
● Memory friendly - scalable

52. String Data
No iterator analogy.
● Memory hog - less scalable

53. Stdlib Examples
Module String Data File Filename
json loads load
pickle loads load
xml.etree.Element fromstring parse parse
Tree
xml.dom.minidom parseString parse parse
ConfigParser cp.readfp cp.read(filenames
)
csv reader DictReader
pyyaml 3rd party load, safe_load load, safe_load

54. Stdlib Take-aways
● mostly functions
● file interface is required, others optional
● parse or load

55. Example
>>> import sys
>>> def parse(fin):
... for line in upper(fin):
... sys.stdout.write(line)
>>> def upper(iterable):
... for item in iterable:
... yield str(item).upper()

56. Create file to parse
>>> with open('/tmp/data', 'w')
as fout:
... fout.write('line1n ')
... fout.write('line2n ')

57. Filename to file
>>> filename = '/tmp/data'
>>> with open(filename) as fin:
... parse(fin)
LINE1
LINE2

58. String data to file
>>> data = "stringn datan "
>>> import StringIO
>>>
parse(StringIO.StringIO(data))
STRING
DATA

59. Parse Iterable
>>> data = ['foon ', 'barn ']
>>> parse(data)
FOO
BAR

60. More file benefits
● Combine with generators to filter, tweak
● Easier to test

61. Assignment
Add a parse function

62. Invoking Shell
Commands

63. Reading output
>>> import subprocess
>>> p = subprocess.Popen('id -u', shell=True,
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
>>> p.stdout.read()
'1000n'
>>> p.returncode # None means not done
>>> print p.wait()
0

64. Feeding stdin
Can use communicate or
p2.stdin.write w/ flush/close.
>>> p2 = subprocess.Popen('wc -l', shell=True,
stdout=subprocess.PIPE, stdin=subprocess.PIPE,
stderr=subprocess.PIPE)
>>> out, err = p2.communicate('foon barn ')
#p.stdin.flush()
>>> out
'2n'
>>> p2.returncode
0

65. Chaining scripts.
Chaining is pretty straightforward make sure to close stdin
http://stackoverflow.com/questions/1595492/blocks-send-input-to-python-subpr
ocess-pipeline
>>> p3 = subprocess.Popen('sort', shell=True,
... stdout=subprocess.PIPE,
... stdin=subprocess.PIPE)
>>> p4 = subprocess.Popen('uniq', shell=True,
... stdout=subprocess.PIPE,
... stdin=p3.stdout,
... close_fds=True) # hangs w/o close_fds
>>> p3.stdin.write('1n 2n 1n ')
>>> p3.stdin.flush(); p3.stdin.close();
>>> p4.stdout.read()
'1n2n'

66. Chaining scripts and python
cat 0-2, add 10 to them (in python) and wc
-l results.
>>> import os
>>> p5 = subprocess.Popen('cat', shell=True,
stdout=subprocess.PIPE, stdin=subprocess.PIPE, close_fds=True)
>>> def p6(input):
... ''' add 10 to line in input '''
... for line in input:
... yield '%d%s ' %(int(line.strip())+10, os.linesep)

67. Chaining scripts and python
(2)
>>> p7 = subprocess.Popen('wc -l', shell=True,
stdout=subprocess.PIPE, stdin=subprocess.PIPE, close_fds=True)
>>> [p5.stdin.write(str(x)+os.linesep) for x in xrange(3)]
>>> p5.stdin.close()
>>> [p7.stdin.write(x) for x in p6(p5.stdout.xreadlines())]
>>> p7.stdin.close()
>>> p7.stdout.read()
'3n'

68. Environment

69. Python Environment
● System Python or “Virtual” Python
● Installation method

70. Which Python
System Python
● Requires root
● Only one version of library
● System packages may be out of date

71. Which Python (2)
“Virtual” Python
● Runs as user
● Specify version
● Sandboxed from system
● Create multiple sandboxes

72. Which Python (3)
Install virtualenv

73. virtualenv
Installation:
● System package $ sudo apt-get install
python-virtualenv
● With pip $ pip install virtualenv
● $ easy_install virtualenv
● $ wget
https://raw.github.com/pypa/virtualenv/master/
virtualenv.py; python virtualenv.py

74. virtualenv (2)
Create virtual environment:
$ virtualenv env_name

75. Directory Structure
env_name/
bin/
activate
python
pip
lib/
python2.7/
site-packages/

76. virtualenv (3)
Activate virtual environment:
$ source env_name/bin/activate

77. virtualenv (4)
Windows activate virtual environment:
> pathtoenvScriptsactivate
May require PS C:>
Set-ExecutionPolicy AllSigned

78. virtualenv (5)
Comes with pip to install packages:
$ pip install sqlalchemy

79. virtualenv (6)
$ which python # not /usr/bin/python
/home/matt/work/courses/script-pypi-github/env/b
in/python

80. virtualenv (7)
>>> sys.path
['', ...,
'/home/matt/work/courses/script-py
pi-github/env/lib/python2.7/site-p
ackages']

81. virtualenv (8)
Use deactivate shell function to reset
PATH:
$ deactivate

82. Assignment
create virtualenv env

83. pip

84. pip
Recursive Acronym - Pip Installs Packages
● install
● upgrade
● uninstall
● “pin” versions
● requirements.txt

85. pip (2)
Install:
$ pip install sqlalchemy

86. pip (3)
Upgrade:
$ pip install --upgrade sqlalchemy

87. pip (4)
Uninstall:
$ pip uninstall sqlalchemy

88. pip (5)
“Pin” version:
$ pip install sqlalchemy==0.7

89. pip (6)
Requirements file:
$ pip install -r requirements.txt

90. pip (7)
Requirements file $ cat requirements.txt:
sqlalchemy==0.7
foobar
bazlib>=1.0

91. pip (8)
Create requirement file from env:
$ pip freeze > req.txt

92. pip (9)
● pip docs say to install from virtualenv
● virtualenv docs say to install from pip

93. pip (10)
Install from directory:
pip install --download packages -r
requirements.txt
pip install --no-index
--find-links=file://full/path/to/p
ackages -r requirements.txt

94. distribute, setuptools,
distutils
● pip wraps distribute adds uninstall,
req.txt
● distribute fork of setuptools
● setuptools unmaintained, adds eggs,
dependencies, easy_install
● distutils stdlib packaging library

95. Assignment
use pip to install
package from internet

96. More Layout

97. Better Example Layout
Project/
.gitignore
doc/
Makefile
index.rst
README.txt
Makefile
bin/
runfoo.py
project/
__init__.py
other.py
...
setup.py

98. .gitignore
*.py[cod]
# C extensions
*.so
# Packages
*.egg
*.egg-info
dist
build
eggs
parts
bin
var
sdist

99. develop-eggs
.installed.cfg
.gitignore
lib
lib64
__pycache__
# Installer logs
pip-log.txt
# Unit test / coverage reports
.coverage
.tox
nosetests.xml
# Translations
*.mo
https://github.com/github/gitignore

100. .gitignore (2)
See blaze-core for example of C and
Docs.
https://github.com/ContinuumIO/blaze-cor
e

101. From requests:
.coverage
.gitignore (3)
MANIFEST
coverage.xml
nosetests.xml
junit-report.xml
pylint.txt
toy.py
violations.pyflakes.txt
cover/
docs/_build
requests.egg-info/
*.pyc
*.swp
env/
.workon
t.py
t2.py
https://github.com/kennethreitz/requests

102. ●
Other(for django)
localsettings.py
tips
● *~ (emacs)
● Run git status and add outliers to
.gitignore
● Make settings global:
git config --global core.excludesfile
Python.gitignore
git config --global core.excludesfile
Python.gitignore

103. Assignment
add (premptive)
.gitignore

104. Documentation

105. Documentation
Two types:
● Developer docs (README, INSTALL,
HACKING, etc)
● End user

106. Developer
README - main entry point for project
● Brief intro
● Links/Contact
● License

107. README
For github integration name it README.rst
or README.md

108. LICENSE
Include text of license. Templates at
http://opensource.org/licenses/index.html

109. Licensing
Some include dunder meta in project docstring (requests
__init__.py):
:copyright: (c) 2013 by Kenneth Reitz.
:license: Apache 2.0, see LICENSE for more
details.
(note IANAL)

110. Licensing (2)
Some include dunder meta in project (requests
__init__.py):
__title__ = 'requests'
__version__ = '1.1.0'
__build__ = 0x010100
__author__ = 'Kenneth Reitz'
__license__ = 'Apache 2.0'
__copyright__ = 'Copyright 2013 Kenneth Reitz'
(note IANAL)

111. Other files
● AUTHORS
● HISTORY/CHANGELOG
● TODO

112. Assignment
create simple README

113. End User Docs
Sphinx is a tool that makes it easy to create
intelligent and beautiful documentation,
written by Georg Brandl and licensed
under the BSD license.
http://sphinx-doc.org

114. Suggested setup
Project/
doc/
# sphinx stuff
Makefile

115. Sphinx in 4 Lines
$ cd docs
$ sphinx-quickstart
$ $EDITOR index.rst
$ make html

116. Assignment
write docs at a later
time :)

117. Makefile

118. Running commands often:
● nosetests (plus options)
Motivation
● create sdist
● upload to PyPi
● create virtualenv
● install dependencies
● cleanup cruft
● create TAGS
● profile
● sdist
● PyPi - register and upload
● creating pngs from svgs
● docs
● Python 3 testing
● etc...

119. Makefile
● Knows about executing (build)
commands
● Knows about dependencies

120. Example
To test:
● Make virtualenv
● install code dependencies
● install nose (+coverage)
● run tests

121. Clean checkout
$ make test
vs
$ virtualenv env
$ env/bin/activate
$ pip install -r deps.txt
$ pip install nose coverage.py
$ nosestests

122. Enough make
knowledge to be
dangerous

123. Makefile (1)
Syntax of Makefile:
file: dependentfile
<TAB>Command1
...
<TAB>CommandN

124. Makefile (2)
Running (runs Makefile by default):
$ make file
# will build dependentfile if
necessary
# then build file

125. Makefile (3)
Example:
foo: foo.c foo.h
<TAB>cc -c foo.c
<TAB>cc -o foo foo.o

126. Makefile (4)
Running (echoes commands by default -s
for silent):
$ make
cc -c foo.c
cc -o foo foo.o

127. Makefile (5)
Subsequent runs do nothing:
$ make
make: `foo' is up to date.

128. Makefile (6)
Add a clean command:
.PHONY: clean
clean:
<TAB>rm foo.o foo

129. Makefile (7)
Since clean isn't a file, need to use .PHONY
to indicate that to make. (If you had a file
named clean it wouldn't try to build it).

130. Makefile (8)
(Simply Expanded) Variables (expanded when set):
BIN := env/bin
PY := $(BIN)/python
NOSE := $(BIN)/nosetests
.PHONY: build
build: env
<TAB>$(PY) setup.py sdist

131. Makefile (9)
(Recursively Expanded) Variables (expanded when used):
FILE = foo
DATA = $(FILE)
# If DATA expanded would be foo
FILE = bar
# If DATA expanded would be bar

132. Makefile (10)
Shell functions:
.PHONY: pwd
pwd:
<TAB>pushd /etc

133. Makefile (11)
Invoking:
$ make pwd
pushd /etc
make: pushd: Command not found
make: *** [pwd] Error 127
(pushd is a bash function)

134. Makefile (12)
Shell functions:
SHELL := /bin/bash
.PHONY: pwd
pwd:
<TAB>pushd /etc

135. Makefile (13)
Multiple commands:
SHELL := /bin/bash
.PHONY: pwd
pwd:
<TAB>pushd /etc
<TAB>pwd
<TAB>popd

136. Makefile (14)
Multiple commands:
$ make pwd
pushd /etc
/etc /tmp/foo
pwd
/tmp/foo
popd
/bin/bash: line 0: popd: directory stack empty

137. Makefile (15)
Each tab indented command runs in its
own process. Use ; and put in one line or
use for line continuation

138. Makefile (16)
Multiple commands (use line continuation ):
SHELL := /bin/bash
.PHONY: pwd2
pwd2:
<TAB>pushd /etc;
<TAB>pwd;
<TAB>popd

139. Makefile (17)
Shell variables:
.PHONY: path
path:
<TAB>echo $PATH

140. Makefile (18)
Make thinks they are make variables:
$ make path
echo ATH
ATH

141. Makefile (19)
$ needs to be escaped with $:
.PHONY: path
path:
<TAB>echo $$PATH

142. Makefile (18)
Now it works:
$ make path
echo $PATH
/tmp/maketest

143. Makefiles for Python
Projects

144. Inspired by Rick Harding's
Talk
http://pyvideo.org/video/1354/starting-you
r-project-right-setup-and-automation
https://github.com/mitechie/pyohio_2012

145. Makefile for Python
Make virtualenv:
env:
<TAB>virtualenv env

146. Makefile for Python (2)
Make dependencies:
.PHONY: deps
deps: env
<TAB>$(PIP) install -r
requirements.txt

147. Makefile for Python (3)
Testing with nose:
.PHONY: test
test: nose deps
<TAB>$(NOSE)
# nose depends on the nosetests binary
nose: $(NOSE)
$(NOSE): env
<TAB>$(PIP) install nose

148. Contrary Opinions
“Dynamic languages don't need anything
like make, unless they have some
compile-time interface dependencies
between modules”
http://stackoverflow.com/questions/758093
9/why-are-there-no-makefiles-for-automati
on-in-python-projects

149. Other options
● paver
● fabric
● buildout

150. Packaging

151. setup.py overloaded
● create sdist (source distribution)
● upload to pypi
● install package

152. setup.py wart
require keyword of distutils doesn't
download reqs only documents them. Use
requirements.txt in combo with pip.

153. setup.py example
From requests:
setup(
name='requests',
version=requests.__version__,
description='Python HTTP for Humans.',
long_description=open('README.rst').read() +
'nn' +
open('HISTORY.rst').read(),
author='Kenneth Reitz',
author_email='me@kennethreitz.com',
url='http://python-requests.org',

154. setup.py example (2)
packages=packages,
package_data={'': ['LICENSE', 'NOTICE'],
'requests': ['*.pem']},
package_dir={'requests': 'requests'},
include_package_data=True,
install_requires=requires,
license=open('LICENSE').read(),
zip_safe=False,

155. setup.py example (3)
classifiers=(
'Development Status :: 5 - Production/Stable',
'Intended Audience :: Developers',
'Natural Language :: English',
'License :: OSI Approved :: Apache Software License',
'Programming Language :: Python',
'Programming Language :: Python :: 2.6',
'Programming Language :: Python :: 2.7',
'Programming Language :: Python :: 3',
# 'Programming Language :: Python :: 3.0',
'Programming Language :: Python :: 3.1',
'Programming Language :: Python :: 3.2',
'Programming Language :: Python :: 3.3',
),
)

156. setup.py modules
If project consists of a few modules this
may be easiest

157. setup.py packages
Need to explicitly list all packages, not just
root

158. setup.py scripts
Add executable Python files here

159. setup.py non-Python files
● add files to MANIFEST.in (include in
package)
● add files to package_data in setup.py
(include in install) Not recursive

160. MANIFEST.in language
● include|exclude pat1 pat2 ...
● recursive-(include|exclude) dir pat1 pat2 ...
● global-(include|exclude) dir pat1 pat2 ...
● prune dir
● graft dir
http://docs.python.org/release/1.6/dist/sdist-cmd.html#sdist-c
md

161. setup.py classifiers
Almost 600 different classifiers.
Not used by pip to enforce versions. For UI
only

162. Create sdist
$ python setup.py sdist

163. PyPi
Validate setup.py:
$ python setup.py check

164. PyPi Register
Click on “Register” on right hand box
https://pypi.python.org/pypi?%3Aaction=re
gister_form

165. PyPi Upload
$ python setup.py sdist register
upload

166. PyPi Upload (2)
$ python setup.py sdist register upload
...
Creating tar archive
removing 'rst2odp-0.2.4' (and everything under it)
running register
running check
We need to know who you are, so please choose either:
1. use your existing login,
2. register as a new user,
3. have the server generate a new password for you (and email it to you),
or
4. quit
Your selection [default 1]:
1

167. PyPi Upload (3)
Username: mharrison
Password:
Registering rst2odp to http://pypi.python.org/pypi
Server response (200): OK
I can store your PyPI login so future submissions will be faster.
(the login will be stored in /home/matt/.pypirc)
Save your login (y/N)?y
running upload
Submitting dist/rst2odp-0.2.4.tar.gz to http://pypi.python.org/pypi
Server response (200): OK

168. PyPi Note
Though PyPi packages are signed there is
no verification step during package
installation

169. Non PyPi URL
$ pip install --no-index -f
http://dist.plone.org/thirdparty/
-U PIL

170. Personal PyPi
https://github.com/benliles/djangopypi

171. PyPi
Makefile integration:
# --------- PyPi ----------
.PHONY: build
build: env
<TAB>$(PY) setup.py sdist
.PHONY: upload
upload: env
<TAB>$(PY) setup.py sdist register upload

172. Testing

173. Testing
Add tests to your project

174. Testing (2)
● use doctest or unittest

175. Testing (3)
Use nose to run:
$ env/bin/nosetests
..
-------------
Ran 2 tests in 0.007s
OK

176. Testing (4)
Makefile integration:
NOSE := env/bin/nosetests
# --------- Testing ----------
.PHONY: test
test: nose deps
<TAB>$(NOSE)
# nose depends on the nosetests binary
nose: $(NOSE)
$(NOSE): env
<TAB>$(PIP) install nose

177. Github

178. Github
Just a satisfied user

179. Why Github?
Not bazaar nor mercurial

180. Why Github? (2)
Code is a first class object

181. Github
Don't check in keys/passwords!
http://ejohn.org/blog/keeping-passwords-i
n-source-control/#postcomment

182. A Branching Strategy
http://github.com/nvie/gitflow

183. Travis CI

184. Travis CI
CI (continuous integration) for Github

185. Travis CI (2)
Illustrates power of (web) hooks

186. 5 Steps
● Sign-in with github
● Sync repos on Profile page
● Enable repo
● Create a .travis.yml file on github
● Push a commit on github

187. travis.yml
language: python
python:
- "3.3"
- "2.7"
- "2.6"
- "pypy"
# command to run tests
script: make test

188. More Info
http://about.travis-ci.org/

189. Poachplate

190. poachplate
Example of most of what we have talked
about today
https://github.com/mattharrison/poachplat
e

191. That's all
Questions? Tweet or email me
matthewharrison@gmail.com
@__mharrison__
http://hairysun.com