This document discusses using the iTest framework for integration testing of HBase. iTest provides abstractions for querying and driving OS packaging systems during full system tests. It allows installing packages, running tests, checking results, and removing packages. The document proposes using iTest to enable running HBase integration tests locally or on clusters. It suggests converting unit tests that use MiniClusters to integration tests that can run on real clusters. Adding more long running ingestion and "chaos monkey" tests is also proposed. Expanding the tests to include full system applications and adding Kerberos setup helpers to iTest is discussed.

1. On HBase Integration Testing
Andrew Purtell, Hadoop Group at Trend Micro
apurtell@apache.org

2. Introducing Apache Bigtop
(and other custom
distros)

3. Introducing Apache Bigtop
A new project in incubation with committers from
Cloudera, Ubuntu, and others
The goal is to do for Hadoop what Debian did for
Linux
Build and packaging infrastructure
Makefile based (yes!)
RPM and DEB packaging targets
Deployment infrastructure
Puppet
Integration test infrastructure
iTest

4. Introducing Apache Bigtop
A new project in incubation with committers from
Cloudera, Ubuntu, and others
The goal is to do for Hadoop what Debian did for
Linux
Build and packaging infrastructure
Makefile based (yes!)
RPM and DEB packaging targets
Deployment infrastructure
Puppet
Integration test infrastructure
iTest
This seems like it could be useful

5. iTest
A framework for integration testing
Provides a high-level abstraction for querying and
driving OS level packaging systems during full
system integration tests
Install package
Run tests
Check results
Remove package
Start/stop SysV services
Transparent shell-outs written in Java and Groovy
Convenient to write tests that are scripting (Groovy) layered
over command line invocations ("hadoop jar ...")
Helper functions for unpacking data from JAR files,
filesystem paths, classpath, etc.

6. HBASE-6201 (and friends)
From Enis Söztutar
Add ability to run all integration/system tests locally or
on a given cluster
With 'mvn verify' though Stack suggests this hasn't been
possible to date using Maven machinery alone
Or, by importing and using iTest (and exposing this also to
BigTop) – I personally think this is promising
MiniCluster
'mvn verify' iTest
Actual cluster

7. HBASE-6201 (and friends)
Convert candidate unit tests that run on a full
MiniCluster into integration tests that can run on a
real cluster
TestAcidGuarantees
TestAtomicOperation
TestRegionBalancing
TestFullLogReconstruction
TestMasterFailover
TestImportExport
TestRollingRestart
TestMultiVersions
TestKeepDeletes
TestFromClientSide
TestSplitTransactionOnCluster
... and more

8. HBASE-6201 (and friends)
Add more tests, especially, long running ingestion
tests and chaos monkey
LoadTestTool
YCSB
GoraCI
Resurrect Gremlins? https://github.com/toddlipcon/gremlins

9. Going beyond HBASE-6201
Add full system application test cases
I have a Heritrix / hbase-writer based webtable thingy ?
JGit backed by HBase https://github.com/apurtell/jgit-hbase ?
Canned queries for Titan (http://thinkaurelius.github.com/titan/ ) or
something that uses Apache Giraph's HBase storage
adapter ?
Canned data and scripts for Apache Pig and Hive ?
Add Kerberos setup and usage helper functions to
iTest
so then we can finesse integration testing with security
enabled at the Hadoop, HBase, and/or ZooKeeper layers

10. End. Discuss.