How a Clojure pet project turned into a full-blown cloud-computing web-app

‘How a Clojure pet project turned into a
full-blown cloud computing web app.
Or: What are the differences between a
Clojure web app and one written in Java or
Common Lisp?’

Berlin, February 2, 2010

Presentation, Release 0.1
Hamburg Lispers @ Berlin Lispers Meetup

Thanks to Franz Lisp and the
newthinking store here in
Berlin for making this
meetup possible!

Berlin Lispers Meetup
© freiheit.com Presentation, Rel. 0.1 2

Hi, I am Stefan Richter, CTO
and founder of freiheit.com
technologies.


Why am I here?
- We are working on a Clojure web-app
- This app runs on Google App Engine
- We use this app in our own organization
- We want to provide the app as a software service
- Why did we choose Clojure for such a mission
critical project?
- We want to share our experiences with you:
- How Clojure made us more productive in
developing web applications
(and how much fun it is!)
- What are the differences to Java and Common
Lisp when we talk about web development?


But first things first:
What kind of app is this?


I have to tell a little story
first ...


My company freiheit.com has
been developing large
internet systems for more
than 10 years.


We are about 70 people.
Only 7 are non-hackers.
So 90% are hackers.


We have no project managers.
The programmers ‘manage’
their projects themselves.


We are very efficient.
We are very well organized.
How does this work?


We have strong principles
on how to plan and deliver
software. We do this in a
way which is natural to
hackers.
All principles are supported
by internal software
systems.

Note:
The way we work is ‘agile.’
But I would not call it that
because I hate all this
‘agile’ hype.
We have been working like
this since 1999.
We know what we are doing.

So our principles have been
forged not only into our
culture, but also into our
internal ‘project
management’ systems.


Our main system is ‘Captain
Feature,’ which provides
task-management and
measures for self- and team
management.


The ‘Captain,’ as we fondly
call him, is an intranet app,
written in Java, over 8 years
old.


In 2008 I started a pet
project to replace the
old ‘Captain’ with new
software written in
Common Lisp.


I was the only programmer
on this project.
But I got very far.
I built a nice web-app using
sbcl, hunchentoot,
cl-who, cl-sql, JavaScript
and CSS.


We thought:
Our tools are so useful to
us, why don’t we share them
with other programmers?


This would mean:
Provide our internal
systems as external
services.


We don‘t want to own
hardware.
And we don‘t want to
operate our own server
farm.
Not even a virtual one.


So we checked
Google App Engine
because it promised
zero maintenance and a
price model based on the
resources you actually use.


Problem: App Engine only
runs Python and Java.

But we wanted Lisp.


Clojure is a Lisp and it runs
on the JVM.
If it runs on the JVM, it runs
on App Engine, too!


First, we started to design
and implement a ground-
breaking to-do manager.
The plan was to later build
the new ‘Captain’on top of
this to do manager
infrastructure.


We call this to-do manager
‘TheDeadline’.


‘TheDeadline’ is an
intelligent to-do manager
that behaves like a real,
human personal assistant.
It supports you. I doesn’t
annoy you. It helps you
achieve your goals.


I showed a screenshot of an
early version of
‘TheDeadline’ in a lightning
talk at the European
Common Lisp Meeting 2009:


And this is how it looks
today. (We release a new
version every week. It is
changing fast! Maybe I can
give you a demo later):


Our vision? The dream of
every hacker on this planet!


‘We want to eliminate all
human project management
work on this planet.’


Don’t get me wrong:
We want to eliminate the
‘project management work’
not the ‘project managers.’
;)


‘Instead we provide AI-
based systems that help
people (especially hackers)
to become more productive
and help them to manage
themselves!’


We use AI for Intelligence
Amplification and not to
replace humans.


Sounds crazy?!


To say it in the
words of Mike
Muir of Suicidal
Tendencies:
‘I’m not crazy.
You’re the one
that’s crazy.’

With so many projects still
failing today, we urgently
need intelligent software
for project management ...


With an AI system doing 80%
of the work that a human
assistant would do for you,
you can focus on the things
that AI can’t do (today).
And: ‘TheDeadline’ never
sleeps and doesn’t need
feeding.

Here is why I don’t like the
popular to do manager on
the web: They are just list
management apps with no
intelligence at all. They
don’t help you. They just
store your to-dos.


‘TheDeadline’ is currently
in Closed Beta. Public Beta
starts soon.
(Sign-up link at the end of the presentation.)


Now that you understand
why we did what we did, we
can start with the technical
part ... :)


Most people praise Clojure
because of the concurrency
features. Lots of stuff has
been written about it.
So, I won’t talk about this.
(You can’t start a thread in App Engine anyway ...)


So how does Clojure
improve your life as a
programmer, especially in
web development?!


Three things that make your
life much easier:
1. Structs vs. objects
2. Lisp syntax (data = code)
3. Macros (extending the language)


Of course first-class/higher-
order functions, everything
being a sequence,
immutability, the java
virtual machine, etc., are
all important factors.
But let us take a closer look
at the data structures and
code generation features ...

1. Structs vs. objects.


I used CLOS - which is cool,
especially compared to
Java’s OO system.
But sooner or later you get
the feeling that you don’t
need much OO in a
functional language.


You don’t want that much
state anyway. You want to
fold, transform and
destructure your data in
your app all the time.
And: Not everything is an
object.


BTW: Putting objects into a
relational database was
called ‘impedance mismatch’
in the 1990’s. Today it is
called ‘persistence
mapping.’
But this doesn’t fix the
problems.

Did you ever ask yourself
why not all data structures
are represented in the same
style? Especially when you
use maps in your code, send
JSON to your client and
store data in a key-value
store?

This is what Clojure offers:


The following examples are taken from „Programming Clojure,“ Stuart Halloway, Pragmatic Programmers Series.
Cool book. Buy it! :)


You define a struct. You
instantiate structs.
Same structs share their
keys. They behave like hash
tables (maps). They are
immutable. They are human-
readable.


The Clojure structs
combined with multimethods
are a really smart idea. It’s
like in CLOS: Classes and
methods are separated. But
I think structs are a more
readable and more flexible
concept.

You can even specialize a
method on more than one
class/type.
(Which is unknown in the
Java world).


Important: You don’t have to
use multimethods. But
sometimes they help to
reduce the number of
different function names in
your namespace.


2. Lisp syntax (data = code)


Most people say they are
afraid of the parenthesis.
Maybe that is why 31% of
the JAX mag readers think
that Scala has the most
potential of the JVM
languages. Only 15% bet on
Clojure ...

Lisp syntax is easy to learn
because there are almost no
syntax rules. Your programs
are essentially expression
trees built from function
calls.
Code generation is an
integral part of the
language.

This is a function call:
(+ 3 2)
This is data:
‘(+ 3 2)
You can translate data into
a function call at runtime
and execute it!


So, functions generate
functions. You generalize
functions and parameterize
them by passing other
functions to them. There is
no boilerplate code. Just
small functional building
blocks put together.

3. Macros
(extending the language)


Macros are Lisp programs
that generate other Lisp
programs.


Note: This is not the same
as self-modifying assembler
code! ;)


The Lisp syntax is the
precondition to make
language extension very
easy!
(Yes, this parenthesis.)


BTW: Scala has ‘no
parenthesis’, which is why
it is so bloated with
syntactic sugar ...


Why would I want to extend
the language?


In other languages, you
define an API to implement
a subsystem for a specific
domain, like database
access or html templating.
This API is then
encapsulated in a library.


To use such an API, you
normally build an
integration layer between
your application code and
the API. You need this
abstraction, but it also
requires extra work.


Instead, by extending the
language with some kind of
domain-specific language
constructs, you don’t need
an integration layer
anymore. This layer still
exists, but it is generated
by the macro system.

Most of the time, you can
change the code generation
inside the macros without
having to change the code
that is using the macros.


First example: html
templating. The current
approach in other languages
is broken. Putting your
application code into the
html code is wrong. And
mixing html code with your
app code is wrong.

This is why there are so
many complicated
templating frameworks for
Java: You have to mix two
different languages - Java
and html.


But what if you could just
extend your language to
‘know’ how to generate
html?


If everything would ‘feel’
like one language?


This example is taken from the TheDeadline’s printer-
friendly tabular to do list.
You build the html by using language constructs that
‘know’ how to generate html. You put function calls
into the right spots to fill in the blanks. And you just call
this as a function if you want to display a to do.


Second example: Accessing
the Google App Engine
datastore. You can use a
high-level Java interface
providing Java objects or
you can use a low-level
interface to work directly
on the key-value structures.

The key-value structure of
the Google datastore
entities is almost an exact
fit to our Clojure structs! So
we decided to use the low-
level API to the datastore.


In addition to that, we
wanted to have an
abstraction layer to make
the application itself
independent from the
underlying Google platform.


This is how you would
define a book entity in
Python ...

The following examples are taken from „Programming Google App Engine,“ Dan Sanderson, O‘Reilly
Cool book. Buy it! :)

This is how it is done in
Java...

Not enough space to show all the code that is needed for this example ...

And this is how we do it in
Clojure:

This automatically generates the code to generate, store and retrieve entities of the
type book. It converts the types to and from Google App Engine types and you can also
add your own functions to validate and to convert every single property to and from a
custom type (not shown in this example).
We will write a blog post about it and post the code:
http://www.hackers-with-attitude.com)


Our Clojure data structure
looks almost like the low-
level key-value structure
from Google’s datastore.
We can work directly with it
in our app. But the user of
defentity doesn’t have to
know how to use the
datastore API at all.

The difference to Python:
We don’t declare types, but
you can define functions
for :pre-save and :after-
read execution for every
property and for the whole
entity. We think, this is
much more concise and much
more powerful.

One last example: We are
using several AI techniques.

We implemented a rule-
based expert system that
provides the functionality
of systems like CLIPS, LISA
or OPS5.
© freiheit.com
Presentation, Rel. 0.1 88

A snippet from the famous
Monkey & Banana problem:


We extended the language
to be able to do expert
system tasks.
So, we are not just creating
APIs. We create powerful
abstractions that generate
their own integration layers
on the fly.

So, what is our conclusion?
What made Clojure more fun
and more productive for us
in comparison with Java (or
Scala)?


This is where you spend lots
of your time in Java web
development. It’s all about
moving data around:
User Interface
Generating HTML, JSON from application objects
Folding/ Translating HTTP-Requests to application objects
Destructuring your
data in your app Application
code Translating application objects to SQL statements
Translating SQL queries to application objects

Database/Persistence


So you access a database,
generate objects, iterate
over objects, copy objects,
convert objects to other
objects, convert objects to
JSON, to XML, etc., etc., etc.
That is what you do in Java
all the time...

There are lots of Java
frameworks that help you to
generate the needed code
because it is almost
impossible to do this by
hand. (That is also why you
need Eclipse at all ...)


But these frameworks add a
lot of complexity. You need
XML configuration and
annotations.
Java programming becomes
more and more a
configuration task.


The edit-build-deploy-run
cycle is very slow. All the
code that is generated by
the frameworks, the
dependency injections, the
start of the app server ...
You go through this for
every change you make.

Our experience is that it is
much better to have a
simple unified data
structure (struct) that you
can pass around in your
system and build powerful
abstractions/mini-languages
(with macros) to work on
these data structures ...

... and to use a real dynamic
and interactive
programming environment
that lets you change and
test your code in a running
system.
Interactive programming even works with the Google App
Engine dev server. We posted an article in our blog explaning
how to setup your environment. After that, you need ‘ant’
only for the deployment on the App Engine production
servers.

One more thing: You don’t
need Eclipse anymore. So
you don’t have to reserve
1.0 Gigs on your machine
for your editor/compiler
anymore.


But what about
Common Lisp?


You can do the same in
Common Lisp!
You could implement the
same features that Clojure
has in Common Lisp and vice
versa.
That’s the power of the Lisp
programming languages!

But nobody is doing it.
The Common Lisp Standard
is hardly changing anymore.


Cool people are writing
world-class libraries for
Common Lisp.
But the platform itself is
not changing anymore.


Common Lisp is still cool,
but Clojure is improving
very fast. It’s a modern
Lisp.


For us, Clojure is the next
evolutionary step of
Common Lisp.


That’s why we helped fund
Rich Hickey’s development
efforts on Clojure with a
donation.


So if you want to learn
Clojure, you can learn lots
of advanced stuff from
Common Lisp books.
Especially from Paul
Graham’s ‘On Lisp.’


And it is much easier to
sneak some Clojure code
into the enterprise Java
project your 40-person team
is currently working on than
to sneak in some Common
Lisp code ... ;)


Start tomorrow!


Thank you.

TheDeadline Pre-Registration:
https://the-deadline.appspot.com

Our corporate Blog:
http://www.hackers-with-attitude.com

Looking for a job?
http://www.freiheit.com

Self:
http://www.smartrevolution.de
http://www.twitter.com/smartrevolution

How a Clojure pet project turned into a full-blown cloud-computing web-app

Recommandé

Recommandé

Contenu connexe

Dernier

Dernier (20)

En vedette

En vedette (20)

How a Clojure pet project turned into a full-blown cloud-computing web-app