​Build the ‘Right’ Regression Suite using Behavior Driven Testing (BDT)

1. BUILD THE “RIGHT”
REGRESSION SUITE
USING BEHAVIOR
DRIVEN TESTING (BDT)
vodQA

2. ABOUT US
about.me/anand.bagmar priyankh@thoughtworks.com
@BagmarAnand
vodqancr@thoughtworks.com

3. WHAT DO YOU EXPECT FROM THIS SESSION?

4. CASE STUDY
• Multiple, long running projects
• Legacy applications
• Integrated

5. • Limited / flaky / long-running Automation
• Long regression cycle
• Huge cost of fixing defects
FUNCTIONAL TESTING CHALLENGES

6. • Brittle
• Little / less value
• No visibility into what is tested
• Maintenance nightmare
• Outdated
• Expensive
AS A RESULT

7. • Trust deficit on the team
• In existing Automation
• In manual testing
• Finger pointing / blame game
BIGGEST PROBLEM

8. My Case Study

10. Concept #1
Given -> When -> Then

11. Expected Functionality:
The customer should be able to
withdraw money from his account via
an ATM machine

12. BDD Example:
Given the account has balance of 5000
When the customer requests 1000
Then the account is debited by 1000

13. 13

14. Workshop Begins – Iteration 1
Case Study

16. PHASE 1 REQUIREMENTS
• Guest User is able to search for a flight for a
single traveller
• Select option from Search Results page
• Specify contact information for person
booking the flight

17. A Guest User can …
WORKFLOW
Search
Search
Results
Contact
Information

18. SEARCH MODULE

19. SEARCH RESULTS

20. CONTACT DETAILS

21. CONTACT DETAILS

22. PHASE 1 REQUIREMENTS
• Guest User is able to search for a flight for a
single traveller
• Select option from Search Results page
• Specify contact information for person
booking the flight

23. A Guest User can …
WORKFLOW
Search
Search
Results
Contact
Information

24. LETS GET STARTED … WRITE 2 TEST SCENARIOS

25. Iteration #1
Showcase

26. Concept #2
Test Specification Styles

27. • Imperative
• Declarative
TEST SPECIFICATION STYLES

28. A Guest User can …
AN EXAMPLE
Search
Search
Results
Contact
Information

29. IMPERATIVE STYLE
Given I am a guest user on the home page
And I choose “round” trip option
And I select “Chicago” from the origin dropdown
And I select “San Francisco” from the destination dropdown
And I select departure date as “5 December 2013”
And I select returning date as “25 December 2013”
When I click on Search
Then I should see the search results page
And I should see at least 1 option for my criteria
...

30. IMPERATIVE STYLE …
…
When I select the “first” option
Then I am on the Contact Information page
When I enter first name as “Anand”
And I enter last name as “Bagmar”
…
…
And I click the “Select and Continue” button
Then I should be on the next page

31. DECLARATIVE STYLE
Given I am a guest user
When I search for flight options for a “one-way” trip for “1”
“Adult” from “Chicago” to “San Francisco”
And I select the “first” flight
And I enter “valid” contact details for “traveller1”
Then I am able to Save and Continue

32. • Ben Mabey -
http://benmabey.com/2008/05/19/
imperative-vs-declarative-scenarios-in-
user-stories.html
• Alister Scott -
http://watirmelon.com/2010/12/10/
cucumber-imperative-or-declarative-that-
is-the-question/
REFERENCES

33. Workshop – Iteration 2
Case Study Continues

34. • User registration from Contact information
page
• Registered user can login from home page
(below search criteria)
• For logged-in user, when booking ticket,
contact information is pre-filled with
logged-in user information
PHASE 2 REQUIREMENTS

35. Login Search
Search
Results
Contact
Information
• Pre-filled
WORKFLOW … ENHANCED

37. REGISTERED USER

39. • User registration from Contact information
page
• Registered user can login from home page
(below search criteria)
• For logged-in user, when booking ticket,
contact information is pre-filled with
logged-in user information
PHASE 2 REQUIREMENTS

40. Login Search
Search
Results
Contact
Information
• Pre-filled
WORKFLOW … ENHANCED

41. LETS GET STARTED … WRITE 2 TEST SCENARIOS

42. Iteration #2
Showcase

43. WHY IS THIS IMPORTANT?

44. Concept #3
The Test Pyramid

45. AUTOMATION – A SAFETY NET

46. • “Succeeding with Agile” – Mike Cohn
• Martin Fowler – Test Pyramid
martinfowler.com/bliki/TestPyramid.html
THE TEST PYRAMID

47. Time
Value
Cost
IDEAL TEST AUTOMATION PYRAMID
Manual / Exploratory
UI
Web Service
View
JavaScript
Integration
Unit
Technology-
facing Tests
Business-facing
Tests

48. REALITY

49. UI
Web Service
View
JavaScript
Integration
Unit
Manual / Exploratory
ICE CREAM CONE ANTI PATTERN

50. Developers Test Pyramid QA Team Test Pyramid
Web Service
View
JavaScript
Integration
Unit
Manual / Exploratory
UI
Web Service
DUAL TEST PYRAMID ANTI-PATTERN

51. Concept #4
Behavior Driven Testing (BDT)

52. BEHAVIOR DRIVEN TESTING (BDT)

53. • How is this functionality going to be used
by the end-user?
• What is their thought process going to be
when using this product?
• What “core-business-value” should be
delivered to the end-user?
THE BIG PICTURE

54. Manual / Exploratory
UI
Web Service
View
JavaScript
Integration
Unit
Technology-
facing Tests
Business-facing
Tests
Identify the “right” type of tests that sit
on top of the Test Pyramid

55. My Case Study

57. HOW TO SOLVE THE PROBLEM?

58. • Remove the ambiguity
• Same visibility for all
SOLUTION

59. • Whiteboards
• Mind maps
• Flow charts
• Collaboration!!
TOOLS TO ENABLE BDT

60. HOW DID WE ACCOMPLISH THIS?

61. IDENTIFY BUSINESS FLOWS

62. • Identify business flows based on personas,
and how the personas use the business
functionality, or,
• Identify business flows, and how different
user personas can use the business flows
NOTE: Business flows are different from UI
flows
HOW TO IDENTIFY BUSINESS FLOWS ?

63. PRIORITIZE BUSINESS FLOWS

64. 1st candidate
• Automate!
• High Value, Low Cost
2nd candidate
• Automate!
• High Value, High Cost
Can Automate?
• Manual Regression?
• Low Value, Low Cost
Do Not Automate
• Manual Regression!
• Low Value, High Cost
High
Value
High
Cost
Low
Value
Low
Cost

65. AUTOMATE THE TOP LAYER USING
• BDD Tools, or,
• Directly in any appropriate programming
language / tools

66. • Identification of Tests
• Automate it!
WHAT’S NEXT?

67. AUTOMATE TESTS THAT ARE VALUABLE,
NOT BECAUSE YOU CAN OR LIKE TO!
CAUTION!

68. Who is doing the action?
What is the business
functionality?
Depicts a user flow / user
journey
Actions & Verifications
happen implicitly
Given auctioneer creates an auction
And 5 vehicles are added to the
auction
When auctioneer starts the auction
And auctioneer sets a starting bid
of 5000
And buyer_1 bids
And buyer_2 bids
Then auctioneer Sells the vehicle
And buyer_2 wins the vehicle
EXAMPLE

69. Delete
profile
UPDATE BUSINESS FLOWS

70. 1st candidate
• Automate!
• High Value, Low Cost
2nd candidate
• Automate!
• High Value, High Cost
Can Automate?
• Manual Regression?
• Low Value, Low Cost
Do Not Automate
• Manual Regression!
• Low Value, High Cost
High
Value
High
Cost
Low
Value
Low
Cost

71. Project Management tool Mingle
Test case repository (manual
+ automated)
Cucumber .feature files with
appropriate tagging
Feature files / Scenarios 33 / 65
# of Manual Tests 0
Tests running on CI (Jenkins) Since the 1st test was
automated
Smoke tests execution time 15 minutes
Regression tests execution
time
45 minutes
FINAL STATE … IN MY CASE STUDY

72. • Builds and deployments were automated
• Tests ran on demand, automatically!
• Smoke / Regression / Javascript / Unit
• Functional Performance
• Concurrent Testing
ADDITIONALLY …

73. How did this Technique (BDT) become
successful?

74. CONTEXT IS KING!
Domain
Distributed
Teams
Skills
Capabilities Product-under test
Tech Stack
Process
Tools

75. NOT work in isolation
BDT WILL …

76. BDT REQUIRES …
Lower layers of Test Pyramid to be well
automated!

77. BDT REQUIRES …
Dedicated and Focused testing (manual /
exploratory)

79. BDT helps identify the “right” type
of regression tests!

80. Executable
Specs
BDT
Scenarios
Business Idea
Features
Epics
Stories
Iterations
Releases
BDT VALUE PROPOSITION

81. BDT VALUE PROPOSITION …
Incrementally
build
Business Flows
Regress
Business Value
Test in-sync with
Product Functionality
Effective
Tests

82. BDT VALUE PROPOSITION …
Remove
ambiguity
Understand
new functionality
Living
Documentation
Onboarding

83. ABOUT US
about.me/anand.bagmar priyankh@thoughtworks.com
@BagmarAnand
vodqancr@thoughtworks.com