We've all been there: debugging problems in a test case and silently screaming into the dark. (Sometimes not even silently.) Poor test case design can cost you significant time and effort let alone impact the quality of your application or product. Testing is vitally important, but so is having a test suite you can use effectively and can rely on. This session will take you through the top ten rules for writing effective and reliable testcases.
The new kids on the block such as Cloud or Docker and general "Infrastructure as Code" style solutions may make you believe old rules are just old. This talk will make you think again. Knowledge gained from personal experience is always best. Learn from these old masters how to design great test cases and maybe you'll never have to visit the dark side again.
2. #DevoxxUS @spoole167 @stuartmarks
Steve Poole : IBM
JVM Developer
DevOps practitioner
Developer Advocate
Stuart Marks : Oracle
Principal MTS
Java / OpenJDK Core Libraries
3. #DevoxxUS @spoole167 @stuartmarks
“I get paid for code that works, not for tests”
How to maximize your effort
and protect your investment
in tests and testing now with
added
Cloud
4. #DevoxxUS @spoole167 @stuartmarks
1. Think before you act
2. Make your tests understandable
3. Keep your tests “small and simple”
4. Test one thing only
5. Fast tests only
6. Absolute repeatability
7. Independent tests only
8. Provide diagnostic data on failure
9. No hard-coding of your environment
10.No extraneous output
5. #DevoxxUS @spoole167 @stuartmarks
1. Think before you act
What are
you testing?
Why are you
testing?
Planhttps://www.flickr.com/photos/ayurvedicmedicines/
6. #DevoxxUS @spoole167 @stuartmarks
2. Make your tests
understandable
Comments
Expected behaviour
Aid debug
Diagnostics
https://www.flickr.com/photos/83633410@N07/
7. #DevoxxUS @spoole167 @stuartmarks
3. Keep your tests
“small and simple”
Separate test logic / setup
Much easier to debug
Use setup / teardown
https://www.flickr.com/photos/9266144@N02/
8. #DevoxxUS @spoole167 @stuartmarks
4. Test one thing only
One scenario per test
Enables fast debug
Obvious why test failed
https://www.flickr.com/photos/ryantron/
9. #DevoxxUS @spoole167 @stuartmarks
5. Fast tests only
Run unit tests often as possible
Maintain quality bar
Quick results
https://www.flickr.com/photos/mcleod/
10. #DevoxxUS @spoole167 @stuartmarks
6. Absolute repeatability
Non-deterministic
tests are a headache
Fix intermittent tests
immediately
No value, waste of resource
Must trust all tests
https://www.flickr.com/photos/fdecomite/
11. #DevoxxUS @spoole167 @stuartmarks
7. Independent tests only
Must run in any order
Run subset, faster results
No dependencies
https://www.flickr.com/photos/sheila_sund/
12. #DevoxxUS @spoole167 @stuartmarks
8. Provide diagnostic
data on failure
Use message in asserts
Make it simple to debug
Reference input data
Record test environment info
https://www.flickr.com/photos/davidbaker/
13. #DevoxxUS @spoole167 @stuartmarks
9. No hard-coding of
your environment
No ports, IP addresses,
data files, databases
Use config files, system
properties or mock objects
Portable tests
https://www.flickr.com/photos/pug50/
14. #DevoxxUS @spoole167 @stuartmarks
10. No extraneous output
A passing test is a silent test
Too much output = confusion
Use option, config file to
turn on debug, save output
https://www.flickr.com/photos/3-bs/
15. #DevoxxUS @spoole167 @stuartmarks
Pirate rules
Guidelines only
Some may seen obvious.
Some may seem pedantic
Until you inherit a testsuite…
Thank you.
https://www.flickr.com/photos/gapic/
Steve: To say why we are doing. Intro
Why do we write tests. I don’t need to my code is good enough. I only get paid for code.
Stuart: Platitudes -> things are ‘better’ if we write tests. With not much complexity there was more knowledge in the code and tests than in my head. 200 lines of code
Your brain is not big enough to keep even a simple code in your head.
Pirate Rules (guidelines only)
Can we agree on rules vs guidelines ?
This list is the top ish(10) from professional testers and developers. Most of our pain has come from from personal experience in breaking these rules..
Our experiences only.
Valueable. Its not exclusive, just our opnion. And we don’t have time for more..
Non of these are unimportant
Steve – intro and substance
---
Your first job is to capture the expected behaviour of your system so you can continue to show you have not broken bevaiour..
You cannot test quality in
But without tests you cannot assert ‘quality’
you only have some much time to capture your systems behavior in tests - don’t waste it
--
Stuart – your opinion “on testing in relation to quality”
How do you measure this?
Test coverage -> Don’t just test getters and setters
Its not valueless but unless you’re looking at conformance don’t get too bent out of shape about coverage.
Improving Test coveage without bugs is not necessary a good thing.
JDK has jtreg. SQE has separate test suite (closed for no good reason) too much effort to open source the framework and tests.
Is adding more tests to extend coverage a good idea? Not necessary - time spend on arguing how the code is supposed to work Do new tests have bugs as often as code has bugs?
Stuart: tests are an equally important part of a code base as the production code.
Tests need to be maintained and enhanced alongside the production code, not left as an afterthought.
Q: What happens if your tests are not understandable?
Should tests be easier to understand than the code their testing?
Tests should be coded and commented well. A test suite is the inverse of the code. Its codifing expected behaviour so write it down what your think the behavior is.
When and a test fails you don’t want to be spending time figuring out if it’s the test. How would you report a bug if you can’t work out what the test.
Steve: tests should have a well defined setup
Can you find the actual line of testing in a large test suite
Why are we saying this - how often have you had to step a 1000times
Or debug a nested loop inside a nested loop?
Or inserted code to allow you to set a break point.
If you have to hack the test its too big
clean separate between getting to to the actual cost.
Q: “you are in full agreement?”
Stuart: setup – operation – verification – teardown
Don’t do multiple operations; they can spoil the fixture for subsequent op/verify steps.
Separate into multiple tests.
Q: “do you agree?”
Steve: Testing is domain / time context
We’d all like all our tests to run all the time. We can’t do that so when backing off chose wisely.
Running the whole testsuite is useful. Long tests encourage you to not run tests. You let more bugs through
Even if part of a CI run..
Long running test suites are evil. They encourage corner cutting. The make you think about not running tests…
Q: “Stuart – what happens if your tests are not fast?”
War story about test suite run monthly
Stuart: An ideal to strive for but possibly impossible to achieve in practice. Nonetheless v important!
(ask for show of hands)
Noise level of intermittent test failures makes result analysis harder, masks real failures.
Q: “Steve how does this work in a cloud environment?”
Cloud: reliable system built from unreliable components. This manifests itself in intermittent test failures.
How to distinguish "valid" failure (failure that can be handled by the larger system) vs an "invalid" failure? Do we care?
"Offensive programming" – deliberate injection of failures. Useful for system tests.
Random without a seed. Sleeping to wait for stuff to happen.
Repeatability with timeouts is difficult.
“Chaos Monkeys”
Steve: War story - junit test ordering. Exposed unexpected test interdependences…
I want to one day run one of these tests singualry for debug purposes. I don’t want to run 50 others first!
Run your tests in random order?
Q: “ has this ever happened to you?
Stuart: Yes. Don't write tests that have persistent side effects. CORBA test war story.
Stuart: Reduce debugging time. Exception stack traces usually help here, but not always.
Consider: asserting that right exception was thrown at the right time.
Timeout war story.
Q: what’s your experience with this?
Steve : War story FFDC
works for me syndrome
+ Cloud: Remote tests are difficult remove env is always different and un reproducable locally..
Steve: War story “the App server test env and 1yr”
+ Cloud needs you not to do this.. Every instance could be on a different box, somewhere else in the world
Reduce opportunities to be effected by using mock objects.
Q: “can you think of any reasons why you would want to hard code your environment?”
Stuart: Barely. Hard to avoid hard-coding 100%. Example: test that the default port is 80.
Stuart: Essential if you have 10,000 unit tests. Even saying ”passed” can generate too much output.
But for complex system tests, esp with timeouts, sometimes verbose logging is necessary.
Q: You like verbose output don’t you Steve?
Steve: if you have too much output - > you invest in systems to analyse the log and look for the real ‘truth’
Steve: Just like the movie says – these are all guidelines. We think you should think before you ignore the rules. Wait until you inherit your first test suite..
Q: Stuart –any closing thoughts?