A guide to how to help grow your new open source project at Apache.

1. RUNNING AN APACHE PROJECT:
10 TRAPS AND HOW TO AVOID
THEM
Owen O’Malley
omalley@apache.org
September 2019
@owen_omalley

2. © 2019 Cloudera, Inc. All rights reserved. 2
WHO AM I?
• First committer added to Hadoop
 Working at Yahoo in 2006
 Original VP when Hadoop became a TLP
• Committer & PMC member on
 Ambari
 Hadoop
 Hive
 ORC
• Mentor for: Giraph, Kafka, Knox, Kylin, Iceberg, Metron, Ranger, Reef, & Tez

3. GETTING STARTED

4. © 2019 Cloudera, Inc. All rights reserved. 4
MISTAKE 1. STARTING ON GITHUB WITHOUT AN IP AGREEMENT
• Starting a project as GitHub repositories is very easy
Start developing immediately!
Allows building community before entering Apache
• But… getting lawyers to sign a code grant takes time
Need legal sign offs from each contributors company
Apache requires code grants for large chunks of code

5. © 2019 Cloudera, Inc. All rights reserved. 5
MISTAKE 1. STARTING ON GITHUB WITHOUT AN IP AGREEMENT (CONT.)
• When Hortonworks & Vertica developed a C++ ORC reader
Always intended to move to Apache
Only two companies
Still took a couple months to get the code grant signed
• Get IP agreements before committing code!

6. © 2019 Cloudera, Inc. All rights reserved. 6
MISTAKE 2. KEEPING YOUR PROJECT SECRET
• Open source is a vibrant ecosystem
Projects fill niches in that ecosystem
Creates choices for users and developers
• Your project is competing with many others
Apache doesn’t pick winners and losers
Fighting for attention

7. © 2019 Cloudera, Inc. All rights reserved. 7
MISTAKE 2. KEEPING YOUR PROJECT SECRET (CONT.)
• Advertise your project!
• Make building a good project website a priority!
Take down the old site
• Give conference talks
Tell users about new features
• Write blogs
Use cases
Experience in production

8. © 2019 Cloudera, Inc. All rights reserved. 8
MISTAKE 3. NOT FOSTERING DIVERSITY
• Apache is of two minds with respect to employers
Apache doesn’t care who your employer is.
Projects should encourage a diverse set of employers
• Your karma is yours, not your employer’s
Expected to keep your “hats” separate
• Avoid group-think
More voices and viewpoints are very very good
Happy users make your project grow

9. © 2019 Cloudera, Inc. All rights reserved. 9
MISTAKE 3. NOT FOSTERING DIVERSITY (CONT.)
• Don’t assume all smart people work at your company
Innovation happens everywhere
• Separate the company’s goals from the project’s
Don’t shoot down proposals because
• They compete with your proprietary products
• Would create work for your proprietary products
• Don’t promise features in upcoming versions

10. © 2019 Cloudera, Inc. All rights reserved. 10
MISTAKE 4. HOLDING FACE TO FACE DEVELOPER MEETINGS
• Excludes remote people
• Even video meetings are hard for different time zones
• Holding roadmap meetings are particularly problematic
Need to ensure full access to the community
Bring discussion back to the email list before finalizing
• When writing to the lists, use “I” instead of “we”
You are presenting your opinion, not a group’s
• Make your project website welcoming & helpful

11. © 2019 Cloudera, Inc. All rights reserved. 11
MISTAKE 5. INCLUDING BINARY RELEASE ARTIFACTS
• Many projects make source and binary release artifacts
• Binary artifacts are hard to review and get right
Make reproducible builds
Licensing for binary artifacts is the transitive closure
Watch Docker file artifact versions
• Far better to make only source release artifacts
Can make convivence binary artifacts after release vote
Even better is to make downstream binary artifacts

12. KEEP GOING

13. © 2019 Cloudera, Inc. All rights reserved. 13
MISTAKE 6. HOLDING A HIGH BAR FOR COMMITTER AND PMC MEMBERS
• Some projects require a lot of patches to make committer
• Much worse if project has a large patch queue
Really hard if you don’t know or work with a committer
3.6k uncommitted patches on Hadoop’s Jiras since 2006
• A committer shortage makes the patch queue worse
• Most important for becoming a committer should be:
Good technical taste
Knowing their own limits

14. © 2019 Cloudera, Inc. All rights reserved. 14
MISTAKE 7. IGNORING TRADEMARKS
• Make sure that your project doesn’t use another trademark
Often comes down to a judgement call about risks
Changing early is much better than later
• Ensure people don’t abuse your trademark
Very hard if it is a user/non-project member
Project members need to fix their company’s behavior
• Board has removed PMC members
• Hold training classes for engineers and marketing

15. © 2019 Cloudera, Inc. All rights reserved. 15
MISTAKE 8. NOT REWARDING OPEN SOURCE WORK
• If employees work on open source projects:
• Make and measure the engineer’s objectives reflect this
Code contributions
Documentation contributions
Code reviews – include other companies’ patches
Conference presentations
• Make the managers’ goals also reflect the community time

16. © 2019 Cloudera, Inc. All rights reserved. 16
MISTAKE 9. STEALTH DEVELOPMENT
• Developing off-line breaks the community
Cuts community off from participation
Forward motion stops & release train stalls
• Yahoo developed Hadoop Security privately
0.18, 0.19, 0.20 were ~3-4 months
0.21 was 12 months, Facebook & LinkedIn forked
0.20.203 was 24 months from 0.20
1.0 was 8 more months

17. © 2019 Cloudera, Inc. All rights reserved. 17
MISTAKE 10. LICENSING PROBLEMS
• Your project should have source with permissive licenses
Eg. Apache, BSD, MIT
• Can have binary dependency on weak-copyleft licenses
Eg. Eclipse, Mozilla, Creative Commons Attribution
• Can only use Category X in very specific cases
Eg. GPL, LGPL, JSON, CC-BY-A
Must be an optional, build tool, or system provided
• Includes recursive dependencies!

18. © 2019 Cloudera, Inc. All rights reserved. 18
MISTAKE 10. LICENSING PROBLEMS (CONT.)
• Can sneak up on you
Updated aircompressor dependency from 0.10 to 0.15
Started using the new zstd codec
Previously excluded slice dependency was now required
Slice depends on jol-core, which is GPL.
• Fortunately, the use of slice was for one method
Worked with aircompressor to get a fix
• Copying code from stack overflow has the same problem!

19. © 2019 Cloudera, Inc. All rights reserved. 19
SUMMARY
• Build your community
Give talks
Make the website
Make the project easy to build
Make the community friendly – look at Beam talk
• Do your work in the open
• Train your employees in Apache & open source
Hortonworks training - https://s.apache.org/apache-training

20. THANK YOU
Owen O’Malley
omalley@apache.org
@owen_omalley