Agile performance testing

© Copyright GlobalLogic 2011 1
Connect. Collaborate. Innovate.
Performance testing in Agile development.
Automation challenges.

Performance critical application
Performance critical java enterprise
application
Operating
systems
Application servers Database
servers
9 priority-driven configurations
two performance engineers

Time estimates
5 core load testing scenarios
On average manual execution of a single test takes 4 hours
(setup environment, execute a test, basic results analysis)
Estimated time to complete load testing against 9 configurations:
9 configurations * 5 test scenarios * 4 hours = 180 hours
1+ man-months
two-week sprints?

Before any work is started
1. Identify Test Environment
2. Identify Performance
Acceptance Criteria
3. Plan and Design Tests
4. Implement Test Design
5. Configure Test Environment
7. Analyze Results, Report,
and Retest
Most time-consuming
activities
Solutions
6. Execute Tests
Core performance testing
activities
More performance engineers ?
Limit the number of tests and
configurations?
Automation ?

More performance engineers
Cons:
• Noticeably bigger budget of performance testing effort
• Additional activities to manage a team, usually with dedicated
lead/manager
• Staff trainings: methodology, tools
• Time consuming and boring tasks are performed by
performance engineers
• Test LAB investments
• Performance engineer positions are hard to fill
Pros:
• There is no timemoney investment in automation
• More work done in less time
• Can be used when automation is impossible due to product
complexity and architecture limitations
• Can be applicable to a short-term project

Limit the number of tests and configurations
Pros:
• There is no timemoney investment in automation
• Focus on most commonly used configurations
• There is no need to expand performance testing team
• Can be applicable to short-term project
• Can be used when risk to miss performance issue is
considered and acceptable
Cons:
• There is a high risk to miss environment specific performance
issue which could cause performance problems in production
• Very risky when an application is performance critical
• The approach doesn’t guarantee that a team will have enough
time to investigate problems deeper
• Time-consuming and boring tasks are performed by
performance engineers

Automation
Pros:
• Automated performance tests 24X7
• Continuous integration process
• Concurrent performance tests
• Early warning of performance bottleneck or performance
issue
• Performance testing with a small team
• More time for results analysis and investigations
• There is no need to expand a performance team
• Profitable for long running, performance critical projects
• Reusable by any person in a team
Cons:
• Timemoney investment to build automation
• Ongoing efforts to keep automation in a good state
• Usually not applicable for short-term projects
• Test LAB investments
• Commercial tools are expensive

Principles of performance testing in Agile
Make performance explicit
• Make it an explicit requirement upfront and throughout the process (non-functional
requirements can be dotted onto the corner of the story card for a functional
requirement)
Performance testing is a part of continuos integration process:
• Automated compilation
• Automated packaging
• Automated deployment
• Automated configuration
• Automated tests execution
• Automated analysis
• Automated result archiving - can check raw output if discover issues later
• Automated scheduling. Overnight runs.
Use concurrent performance testing
Iterate and increment
Get feedback

Ant, Jmeter, Scalable Vector Graphics case study

TeamCity

Sample SVG Report

Concurrent performance testing

Conclusion
Input data
• 9 priority-driven configurations
• 2 performance engineers
• Two-week sprints
Completed activities:
• Performance testing process was automated (automatic application deployment,
tests execution, results analysis)
• Test LAB upgraded to support 4 concurrent performance testing executions against
isolated servers (25 servers in the LAB)
Statistics
• 3 months spent on building Jmeter based automated performance testing
• Automated deployment and configuration process is ~25% faster than manual
• Performance testing cycle takes ~4 days (instead of 1+ man-months estimated)
• 50% of time: results analysis, troubleshooting
• 10% of time: keep automation in a good state
• 40% of time: new tests development, exploratory testing, cluster tests
• 17 performance issues were found
• Performance testing reports are iteratively used by PM’s to assess quality of the
product
• PMs are happy 

Q&A
Thank you
skype: andreymelnyk

Agile performance testing

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Agile performance testing

Similaire à Agile performance testing (20)

Dernier

Dernier (20)

Agile performance testing