Automated UI test on mobile - with Cucumber/Calabash

Mobiltest automation
- experiences from the trenches
Let’s Talk Apple, 11. sep, 2014
Niels Frydenholm, ebay Classifieds

eBay Inc. Confidential
2
Agenda
• Intro
• Tools
• Examples
/
Demo
• Continuous
Integration
/
Follow
up
• Challenges
• Latest
improvements
• Questions?

eBay
Classifieds,
Denmark
• DBA,
BilBasen
og
Bilinfo
• DBA
(a
classifieds
site/app)
has
approx.
1
mill
downloads
• BilBasen
(cars)
has
approx.
450.000
downloads
• Both
brands
have
apps
for
iPhone,
iPad
and
Android
• Traffic
from
mobile
doubled
during
2013
• Strive
for
rapid
releases
eBay Inc. Confidential 3

How
we
are
organised
• Developers
• QA
• Product Owner
• UX
• Other stakeholders

Tools
Jenkins (CI)

Cucumber
-‐
more
than
a
test
tool

Cucumber
-‐
more
than
a
test
tool
• Behaviour-‐Driven
Development
• Adds
“power”
to
Gherkin
syntax
• A
great
way
to
communicate
and
cooperate
between
“the
business
people”,
developers
and
QA.
• Written
in
(your)
business/domain
language
• Easy
to
both
read
and
write

Cucumber
-‐
more
than
a
test
tool
Development
• Adds
“power”
to
Gherkin
syntax
• A
great
way
to
communicate
and
cooperate
between
“the
business
people”,
developers
and
QA.
• Written
in
(your)
business/domain
language
• Easy
to
both
read
and
write
Feature: Simple search
As a user
I want to be able to do simple searches

Cucumber
-‐
more
than
a
test
tool
Development
• Adds
“power”
to
Gherkin
syntax
• A
great
way
to
communicate
and
cooperate
between
“the
business
people”,
developers
and
QA.
• Written
in
(your)
business/domain
language
• Easy
to
both
read
and
write
As a user
Scenario: I can search without any parameters
Given I am on search
When I perform the search
Then I see the result

Cucumber
-‐
more
than
a
test
tool
Development
• Adds
“power”
to
Gherkin
syntax
• A
great
way
to
communicate
and
cooperate
between
“the
business
people”,
developers
and
QA.
• Written
in
(your)
business/domain
language
• Easy
to
both
read
and
write
As a user
Scenario: I can search without any parameters
Given I am on search
When I perform the search
Then I see the result
Step-definition! !
Then(/^I see the result$/) do!
unless query("view marked:'SearchResultCell'").count > 0!
screenshot_and_raise "No result was shown"!
end!
end! !

Calabash
• Enables
the
use
of
Cucumber
for
Native
apps
• iOS
• Android
!
• The
bits
and
pieces
that
interact
with
the
app
• touch,
scroll,
swipe
etc
• Query-‐syntax
(iOS
examples)
• query(“label
marked:’label
text’”)
• query(“button
marked:’Ok’”)

Calabash
-‐
architecture

Scenario
implementation
example

Scenario: Check that change classification clears
matrixdata with warning!
Given I am logged in as "Buyer"!
And I am on the SYI hub!
And I select classification "Hovedtelefoner"!
And I set "Type" to “Nakkekontur”!
And I set price to "250"! !
When I press "Kategori"!
Then I wait for "Du er ved at skifte kategori" to appear! !
When I touch "Fortryd"!
Then I see cells containing "Nakkekontur"! !
When I touch "Skift"!
Then I see the SYI classification search!
9
Scenario
implementation
example
SYI = Sell your item

matrixdata with warning!
Given I am logged in as "Buyer"!
And I am on the SYI hub!
And I select classification "Hovedtelefoner"!
And I set "Type" to “Nakkekontur”!
When I touch "Fortryd"!
Then I see cells containing "Nakkekontur"! !
When I touch "Skift"!
Then I see the SYI classification search!
9
matrixdata with warning
Given I am logged in as "Buyer"
And I am on the SYI hub
And I select classification "Hovedtelefoner"
And I set matrixdata "Type" to “Nakkekontur"
When I try to change classification
Then I see an alert with title "Du er ved at skifte kategori" !
When I cancel the alert
Then the classification is "Nakkekontur" !
When I try to change classification
Then I see an alert with title "Du er ved at skifte kategori" !
When I accept the change classification alert
Then I see the SYI classification search
Scenario
implementation
example

Scenario
implementation
example
Step from “sell your item feature”! !
And I set price to "250"

Scenario
implementation
example
!
And /^I set price to "(.*?)"$/ do |price|!
macro 'I swipe up'!
sleep(1)!
touch("view marked:'Price'")!
sleep(0.5)!
set_text("view marked:'Price'", price)!
sleep(0.5)!
touch("view marked:'OK'")!
sleep(1)!
end

Scenario
implementation
example
!
macro 'I swipe up'!
sleep(1)!
sleep(0.5)!
sleep(0.5)!
sleep(1)!
end
!
And /^I set price to "(.*?)"$/ do |price|
@page.write_price(price)
end

Scenario
implementation
example
!
macro 'I swipe up'!
sleep(1)!
sleep(0.5)!
sleep(0.5)!
sleep(1)!
end
!
And /^I set price to "(.*?)"$/ do |price|
@page.write_price(price)
end
SellYourItemPage (page object)
def write_price(price)
scroll_and_wait_for_row_with_mark("priceCell")
touch("view marked:'Price'")
keyboard_enter_text price
close_keyboard
end

PageObject
pattern
Feature
Scenario
Step

PageObject
pattern
Feature
Scenario
Step
Page object
iPad iPhone Android

PageObject
pattern
Feature
Scenario
Step
Page object
iPad iPhone Android
SyiHubPage (PageObject) !
• go_to_picture_center !
• go_to_classification_search !•
go_to_value_selection_for_matrix
data(label) !
• write_price(price)

DEMO

Continuous
Integration
• Jenkins
on
a
couple
of
Mac
Mini’s
• Runs
multiple
builds/tests
in
parallel
(Master/Slave
Jenkins
setup)
• The
tests
only
runs
in
the
simulator
• Unit
tests
—>
Integration
tests
—>
Accept/UI
tests
• A
subset
of
the
Accept
tests
run
after
each
commit
• Controlled
by
@tags
in
cucumber
• All
Accept
tests
runs
once
every
evening
(or
manually)

The
“control
room”

Cucumber
reports

When
it
fails…

Ensure
high
quality
tests
• Review
and/or
Pair
program
tests
with
QA
• (Only)
use
“Custom
steps”
• Test
code
is
(also)
production
code
• Let
the
quality
reflect
that!
• Refactor,
Refactor,
Refactor!
• Make
sure
the
tests
actually
covers
what
is
expected
• Do
it
with
facts
-‐
no
guessing/assumptions

Code
coverage
• Do
not
(necessarily)
focus
on
the
coverage
percentage
• Find
inspiration
to
new
test
scenarios
• Ensure
every
feature
in
the
app
is
well
covered

Keys
to
success
• Team
effort
• Definition
of
Done
(DoD)
includes
automated
tests
• Know
that
it
costs
time
-‐
it’s
not
a
silver
bullet
• Maintenance
of
setup
-‐
new
versions
of
iOS
&
Calabash
• Fix
broken
builds
ASAP!

Future
improvements
(as
seen
in
January)
Run
a
subset
of
tests
in
Xamarin
test
cloud
a
couple
of
times
each
week
Do
more
BDD/TDD
Automatic
feature
builds
on
Jenkins
Faster
feedback
cycle
Select
a
subset
(with
@tags)
to
run
after
each
commit
Use
“backdoor”
to
do
e.g.
login
Avoid
resetting
after
each
scenario
Make
all
tests
run
stable
all
the
time!

Xamarin
Cloud

Xamarin
Cloud
• Currently
only
as
proof
of
concept
on
DBA
•
5
scenarios
on
3
devices
• Waiting
for
a
proxy
solution
to
take
full
advantage
• Create
listings
(and
complete
payments)
• BilBasen
is
also
running
in
the
cloud
• Trying
out
“Series”
for
iPad
and
iPhone
and
other
cloud
experiments
to
find
out
what
works
best
for
us

Calabash
backdoor

Calabash
backdoor
• Invoke
logic
in
the
app
“behind
the
scenes”
• Reduce
scenario
test-‐time
• Avoid
testing
the
same
UI
over
and
over
in
each
scenario
• Stable
• Easy
to
use

Calabash
backdoor
• Invoke
logic
in
the
app
“behind
the
scenes”
• Reduce
scenario
test-‐time
• Avoid
testing
the
same
UI
over
and
over
in
each
scenario
• Stable
• Easy
to
use
• We
use
it
for
• Login
(and
create
user)
• Create
listings
• Clean
up
data
• (Code
coverage)

Unstable
tests

Unstable
tests
• UI
Automation
/
iOS
Simulator
often
fails
• I
thought
it
was
Calabash
being
unstable
:-‐)
• We
would
start
tests
over
and
over
and
over…
• …and
overlook
real
failures
• It
was
almost
impossible
to
get
our
“All
scenarios”
to
run
green

Stable
tests
• Cucumber
Rerun
formatter
to
the
rescue

Stable
tests
• Cucumber
Rerun
formatter
to
the
rescue
A simple script that will….

Stable
tests
• Cucumber
Rerun
formatter
to
the
rescue
Run tests - use rerun formatter

Stable
tests
• Cucumber
Rerun
formatter
to
the
rescue
If rerun.txt contains tests - run them

Stable
tests
• Cucumber
Rerun
formatter
to
the
rescue
Inform jenkins how it all went

Stable
tests
• Cucumber
Rerun
formatter
to
the
rescue
Inform jenkins how it all went
Replace rerun result in original result

Wrap
up!
• Test
scenarios
follows
the
code
=
the
truth
• Enables
rapid
releases
/
fast
time
to
market
• A
necessity
to
deliver
high
quality
• Spend
less
time
doing
(boring)
trivial
regression
tests
• Happy
QA´er
• Brave
developers
-‐
higher
code
quality
• Quality
product
• Happy
users

QUESTIONS?

Ressourcer
• The
Cucumber
Book
• Calabash
@github
(https://github.com/calabash/)
• Calabash
google
groups
• https://groups.google.com/forum/#!forum/calabash-‐ios
• https://groups.google.com/forum/#!forum/calabash-‐android
• Calabash
overview
(http://docs.xamarin.com/guides/cross-‐platform/test_cloud/calabash/intro_to_calabash/)
• Jenkins
(http://jenkins-‐ci.org/)
• plugins:
Cucumber-‐reports,
Cobertura
(code
coverage),
Build
monitor
view
• RubyMine
(http://www.jetbrains.com/ruby/)
• Uncle
Bobs
Clean
Code
• because
it’s
a
great
book
that
promotes
craftsmanship

Automated UI test on mobile - with Cucumber/Calabash

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Automated UI test on mobile - with Cucumber/Calabash

Similar to Automated UI test on mobile - with Cucumber/Calabash (20)

Recently uploaded

Recently uploaded (20)

Automated UI test on mobile - with Cucumber/Calabash