Decrease Test Build Time, Not Test Quality with CA BlazeMeter

World®
’16
Decrease Test Build Time NOT
Test Quality
Refael Botbol - Services Client Manager - CA Technologies
DO5X53E
DEVOPS

2 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD
© 2016 CA. All rights reserved. All trademarks referenced herein belong to their respective companies.
The content provided in this CA World 2016 presentation is intended for informational purposes only and does not form any type of
warranty. The information provided by a CA partner and/or CA customer has not been reviewed for accuracy by CA.
For Informational Purposes Only
Terms of this Presentation

Abstract
Today, your build time consists of much more than just the time it takes Jenkins to build your
application. It also includes regression tests, deployment to different environments, and maybe
other application specific tasks you want to automate as part of your “build”.
Your build time can significantly influence how often and when you decide to run your builds
(i.e. whether you run your builds weekly, nightly, twice a day, or just per commit). All of this has
a direct impact on your productivity and efficiency.
But how can you decrease your test build time without triggering a drop in product quality - or
even the number of tests you run? Take a look at a new approach to test automation within the
CI process, and learn how to:
• Increase the quantity and type of tests run while reducing your build time
• Include your performance tests as part of your CI process to identify performance
degradation issues early in the process
• Shift left all the way to the developer. See how developers can create tests with DSL using
their existing JSON/YAML knowledge
This session will start off with an interactive discussion before turning into a demonstration/
hands-on workshop. Attendees wanting to participate in hands-on will require to bring their own
laptops to execute the exercises.
Refael Botbol
CA Technologies
Services Client
Manager

Agenda
SAVE TIME: BY PARALLELING YOUR TESTS
DELEGATE: ENABLE EACH PERSONA CONTRIBUTE TO THE EFFORT
FOCUS: CONCENTRATE ON BUILDING NOT ON MAINTAINING TESTS
SCALE: EXECUTE BUILDS AS MANY AS YOUR BUSINESS REQUIRES
1
2
3
4
REDUCE THE BUILD TESTING PHASE WHILE INCREASING TEST COVERAGE
DEMONSTRATION / WORKSHOP5

Which type of test should we run?
§ Functional (API / UI)
§ Regressions
§ Critical Business Processes
§ Performance
§ Smoke tests

Open Source Tools you probably use for Testing
§ Functional (API / UI) – Selenium / JMeter / PhantomJS
§ Regressions – Selenium / JMeter / PhantomJS
§ Critical Business Processes – Gatling / JMeter / Selenium
§ Performance – Gatling / JMeter / Locust / Tsung
§ Smoke tests – Selenium / JMeter / Gatling

Agenda
1
2
3
4

Testing Layers
• Monitoring
• Smoke tests
• Full scale performance testsProd
• Critical business processes
• Performance testsStaging
• Functional UI
• Functional backend
• Regression tests
• Critical business processes
Integration

Divide, Parallel & Conquer
§ Plan your tests per layer and parallel as much as possible
§ Stop the build once a designated critical test has failed and…
– Notify the developer regarding the issue
– Free resources for other builds
§ Build once, run many (even when adding new tests)

DSL based Test Creation using YAML in Taurus

Describe Test Execution Scenario

Set Pass/Fail Criteria

1
2
3
4

Who can contribute to the build’s tests ?
§ Coders – By providing low level tests for API / UI and
regressions
§ Testers – By providing system wide tests and regressions
§ Operations – By providing critical key points and creating tests
with pass/fail criteria for them

Who can contribute to the build’s tests ?
§ Product – By providing pass/fail criteria for business processes
§ Marketers – By completing the product view and providing
marketing scenarios
§ Performance – By implementing performance tests

This is how we (can) do it
§ Coders – Using code based engines or even YML/JSON DSL
§ Testers – They would probably tell you how J
§ Operations – Automating the scripts and providing pass/fail
criteria to make sure nothing breaks

The How – Surprising Part
§ Product – Providing pass/fail criteria for staging and
production environments (and after several runs, even for the
integration..)
§ Marketers – Launching a new event? - Why not test is by using
a DSL with these pages ?
§ Performance – These guys already have the scripts and the
production values. The only thing that is left, is to include it.

Coders / Testers Examples
§ A simple API example
§ A complex scenario
§ Java / Python selenium tests
§ JMeter / Gatling tests

Product / Operations Examples
§ Create tests and set criteria
§ Set pass / fail criteria per environment
§ Easily include it as part of your test execution

Marketing Example

Add Performance Tests to your build

Summary
§ First, take a deep breath… you have made it so far, you
deserve it J
§ We can use DSL based on YML / JSON to create tests
§ We can choose the flavor of tools which best suits us, and
reuse our existing scripts
§ We can also use the DSL to set pass/fail criteria and describe
the test execution

1
2
3
4

Build Once – Use Many
§ Option 1 : Separate everything
– Have your scripts represent one test at a time
– Have one or more DSL describing the execution per build per
environment
– Have one or more DSL describing the pass/fail criteria per environment
This gives you maximum agility on controlling your build tests
– Adding tests is fast
– Criteria are applied to tests automatically per environment

Build Once – Use Many
§ Option 2 : Have clusters of tests
– Group your tests together base on type, functionality or any
characteristics that fits you
– Attach the right execution and criteria per group
This is the best option for special projects such as
– A new campaign
– Satellite projects which aren’t part of every build
– Performance tests

Option 1 Execution Example

Option 2 Clustered Test Example

1
2
3
4

Use the Force Cloud to Scale
§ Some test types can consume all of your Jenkins Worker’s
resources
§ Sometimes, the amount of tests is simply too high for one
Worker
§ You shortened the build time and now everyone wants to
build…

One Worker to Test Them All
§ The Jenkins Worker holds the files and configurations
§ The Worker start the tests using cloud resources to scale it
§ Results are transmitted to back to the Worker
§ Based on the results the Worker decides:
– Lunching the next pack of tests
– Stopping the build due to bad results
– Mark the build as successful and continue to the next step

Use the Force Cloud to Scale

Selenium geo-location scale

One Small Thing Before We Go

1
2
3
4

Getting Taurus
§ Overall, we install Taurus by using pip.
§ Information per OS can be found here :
http://gettaurus.org/install/Installation/

Get your Taurus up and running
§ Plug your flash drive
§ Install VirtualBox
§ Install vagrant (.dmg file)
§ Open terminal and create a folder
§ Go to that folder and do vagrant box add my-Taurus-box
file:///d:/path/to/taurus/box.box

Get your Taurus up and running
§ On that folder do vagrant init my-Taurus-box
§ Vagrant up
§ Vagrant ssh
§ Once you’re inside the vagrant, do cd TaurusScript
§ Do: bzt TaurusScenario.yml

The Challenge
1. Run a JMeter script with Taurus
2. Create your own (simple) YML DSL file and run it with Taurus
3. Run a Gatling test with Taurus
4. Parallel them J
5. Run your YML DSL with 3 different executors (JMeter, Gatling
and Selenium) and see the difference between them
6. Use –report to see the BlazeMeter’s report

Questions?
DO5X53E

Recommended Sessions
SESSION # TITLE DATE/TIME
DO5T03P
Leadership Panel: Continuous Delivery in the Financial
Services Industry
11/16/2016 at 04:30 pm
DO5T14S
Analyst Keynote: Continuous Delivery: Making DevOps
Awesome
11/17/2016 at 10:30 am
DO5X56S
Next-Generation Performance Testing with BlazeMeter,
Service Virtualization and Test Data Management
11/17/2016 at 04:30 pm

Must See Demos
Modernize
App Delivery
Integrated CD
Theater 5 - DOV501P
Deliver Test
Data Faster
Test Data Manager
Theater 5 - DOV511P
Deliver
Better Apps
Service Virtualization
Theater 5 - DOV507P
Orchestrate
Your Release
Release Automation
Theater 5 - DOV513P

Decrease Test Build Time, Not Test Quality with CA BlazeMeter

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

En vedette

En vedette (11)

Similaire à Decrease Test Build Time, Not Test Quality with CA BlazeMeter

Similaire à Decrease Test Build Time, Not Test Quality with CA BlazeMeter (20)

Plus de CA Technologies

Plus de CA Technologies (20)

Dernier

Dernier (20)

Decrease Test Build Time, Not Test Quality with CA BlazeMeter