One of the core processes at the Ministry Of Justice is the identification of crime suspects and foreigners. For this purpose, all police offices are equipped with so called identification consoles, to scan id documents, take finger prints and photos. Based on these information, the system queries many national and European databases to verify if this person is the person he claims to be, check the residence status and if he was involved in any crime. This presentation shares the experiences of testing this entire chain of systems, end-to-end. Aspects covered are: - How to assemble a solid test team - How to specify test scenarios - How to manage test data - How to effectively report to stakeholders - Lessons learnt All is illustrated with real-world examples and anecdotes.

1. Testing the Ministry Of Justice
Biometric Identification System
Gerlof Hoekstra (NL)

2. | Gerlof Hoekstra| © Atos
Chain testing, Acceptance testing,
End-To-End Business Processes
▶ Multiple stakeholders / applications
▶ Complex
▶ Publicly visible
▶ Politically sensitive
▶ Previous attempt(s) failed
Introduction and context
Introducing myself…
Gerlof
Developer
Designer
Tester
2

3. | Gerlof Hoekstra| © Atos
▶ Context
▶ The Ministry Of Justice Intelligence Department
▶ The identification system
▶ The situation I faced when I joined the project
▶ What makes this chain test challenging
▶ Aspects:
▶ Team
▶ Test specification
▶ Formal / informal testing
▶ Test data
▶ From waterfall to iterative
▶ Findings
▶ Lessons learnt
3

4. | Gerlof Hoekstra| © Atos
Introduction and context
Ministry Of Justice Intelligence Department
4

5. | Gerlof Hoekstra| © Atos
▶ Police
▶ Prosecutors
▶ Judges
▶ Investigation services
▶ Prisons
▶ Lawyers
▶ Rehabilitation
▶ Customs offices
▶ Debt collection agency
▶ …
▶ …
The judicial chain
5

6. | Gerlof Hoekstra| © Atos
The Identification Console
Installed in:
- every Police Office
- central immigration offices
- prisons
- Schiphol airport
6

7. | Gerlof Hoekstra| © Atos
Hot topic !
7

8. | Gerlof Hoekstra| © Atos
The Identification Console
Functionality overview
8

9. | Gerlof Hoekstra| © Atos
The Identification System
Technology
▶ Biometric middleware
▶ Photo-software
▶ NIST
▶ ebXML
▶ Jubes
▶ ESB
▶ External Police Broker
▶ NORA
▶ JAB
▶ Ministry of Justice security standards
▶ Photo standards
▶ Fingerprint standards
▶ GBA character set
▶ 7x24
▶ 30.000 users
9

10. | Gerlof Hoekstra| © Atos
The Identification System
High level application architecture & test scope
Crime Immigration
Central message broker
10

11. | Gerlof Hoekstra| © Atos
Initial situation
When I joined the project
▶ Phase 1
▶ Goal: connect to Havank (crime evidence traces)
▶ Earlier attempt failed
▶ New project leader
▶ Well documented business process
▶ Integrated test environment available
▶ Phase 2:
▶ Build & implement a completely new front-end
▶ Connect to foreigner/immigration chain
▶ Validate ID-documents
▶ Optimize business process (scan once)
11

12. | Gerlof Hoekstra| © Atos
Challenges
What makes this chain special?
▶ Complicated business process (many legal rules, privacy)
▶ Many registrations: de facto inconsistent
▶ Most “customers” not willing to co-operate (try to fraude)
▶ Chain-consistent test data (esp. fingerprints)
▶ Many different stakeholders, physical distance, living on their own islands
▶ Uncertainty / problems in the Police organization
▶ Heavy message transfer, complex message broker functionality
12

13. | Gerlof Hoekstra| © Atos
Assembling a Test Team
Testers vs subject matter experts, formal vs informal
▶ Subject matter matters!
▶ Process knowledge
▶ Know the organization
▶ Middleware & message
transfer
▶ Involved in implementation
▶ Police officers
▶ Dactylocopists
▶ Application manager
▶ Justice chain expert
▶ Free thinkers
▶ Not afraid to experiment
▶ No dogmas
▶ Excellent observers
▶ Easy making contact
13

14. | Gerlof Hoekstra| © Atos
Test specification: formal part
Keep it 'simple' and compact; connect to the world of the stakeholders
▶ Process description(test basis)
▶ Classification tree (test overview)
▶ User scenarios (details)
14

15. | Gerlof Hoekstra| © Atos
Test specification: informal part
Make optimum usage of subject matter experts: test charters
Theme based:
▶ Chain components not available
▶ Attribute validation
▶ Unstructured addresses
▶ (Partly) unknown birth dates
▶ Message transfer anomalies
▶ False ID-documents
▶ Bad quality fingerprints
▶ Residential address unknown
▶ Inconsistencies between registrations
Role based:
▶ Test with customs employee
▶ Test met prison employee
15

16. | Gerlof Hoekstra| © Atos
Formal vs informal
Dealing with the quality 'watchdog'
Dilemma:
 versus 
“How Dare You Apply Exploratory Testing”
Solution:
▶ Have a small, but well thought-out predefined test set
▶ Identify exploratory test charters in advance
▶ Use simple check lists
▶ Have a good relation with QA management
▶ Explain why you are doing it like this
▶ Report what you have done
16

17. | Gerlof Hoekstra| © Atos
Test data
Accepting & dealing with restrictions
▶ Which fingerprints do I
use?
▶ How to load these into
a far far away foreign
test database?
▶ How to clean up test
data?
▶ How to get ID
documents for test?
▶ Solution: puzzling,
know what you want,
preparation, know the
right people!
Direct
database
inserts
Insert data
with
application
Use
available
test data
Cleanup
possible?
17

18. | Gerlof Hoekstra| © Atos
From waterfall to iterative
Initial plan phase 2
▶ Newly build front-end
▶ Monthly iterations
▶ Back-end system changes
▶ Every party has his own process & implementation schedule
▶ E2E test starts when all components are ready
▶ mitigate late integration bugs by:
▶ Gerlof reviews/monitors all supplier tests
▶ 1:1 interface tests (peer-to-peer)
▶ stub usage
18

19. | Gerlof Hoekstra| © Atos
From waterfall to iterative
Project in trouble!
▶ Front-end
▶ SCRUM in name only
▶ No business value delivered yet
▶ Back-end system changes
▶ DONE ! (?)
▶ Some even deployed in production
▶ Reviews, peer-to-peer testing, stubs:
▶ I did what I could ....
▶ No time, complex, error prone, ...
19

20. | Gerlof Hoekstra| © Atos
From waterfall to iterative
Re-organizing the project
▶ Front-end
▶ Monthly sprints, now with business focus
▶ My struggle to realize an early & iterative E2E test
▶ 'No time/resources for linking to integrated test environment'
▶ 'Trust the stubs'
▶ 'You disturb the developers'
▶ 'In the end, Iterative E2E testing takes more time'
▶ 'No need, it should work, we reviewed everything’
20

21. | Gerlof Hoekstra| © Atos
From waterfall to iterative
Finally, from sprint 4 we were able to E2E test, AND HECK WE DID !!!!!
21

22. | Gerlof Hoekstra| © Atos
Some interesting findings
Typical integration problems
▶ Simple 'No Hits' leading to fatal errors at the front-end application
▶ Legacy systems not following agreed standards
▶ Un- or incorrectly documented interfaces
22

23. | Gerlof Hoekstra| © Atos
Getting defects fixed
Many independent parties, legacy systems
▶ Sometimes very hard & frustrating
▶ “Not our fault”
▶ “On our side, everything works fine!”
▶ “Yeah, that’s exactly how it is supposed to work”
▶ “Please call our service desk, they will help you”
▶ “Next release we will fix the problem (in 6 months)”
▶ Process
▶ Responsibility for defect fixing !!
▶ Diagnose (zie processchema)
▶ Solution alternatives
▶ Decision making
▶ Fixing & retesting
▶ Follow-up
23

24. | Gerlof Hoekstra| © Atos
What did we learn
Co-operation
▶ Getting into contact / building relations / acquire business chain
knowledge is crucial
▶ (From time to time) working from 1 location adds tremendous value
▶ End users / monitoring / troubleshooting / architect
24

25. | Gerlof Hoekstra| © Atos
What did we learn
Early integration
▶ Perform E2E chain testing during the
sprints
▶ Do not postpone difficult challenges;
if something seems difficult, do it as
early as possible!
▶ If you really want something, you can
arrange it!
▶ Stubs can be useful, but “Nothing
beats the real thing”
25

26. | Gerlof Hoekstra| © Atos
What did we learn
Test specification
▶ Some carefully specified test scenarios are
useful, but do not completely rely on pre-
defined test cases
▶ Make a mix between formal techniques
and exploratory testing and let coincident
happen.
▶ Do not underestimate the value of human
observation (we revealed many bugs by
observing a bit deeper than the test script)
▶ Don't trust application designs, even if
reviewed 100 times;
▶ for E2E testing, a application design is
no more than a supporting document
▶ Business process = test basis
?
A collection of applications that
all work as designed does NOT
automatically mean a working
chain !
26

27. | Gerlof Hoekstra| © Atos
What did we learn
Others
▶ Effective reporting
▶ Frequent
▶ Visual
▶ In the stakeholders' language
▶ Forecast
▶ Directly linked to the implementation scenario
▶ Implement testability
▶ Log’s and traces
▶ Alternative input (esp. for fingerprints)
▶ Police officers are great testers!
27

28. | Gerlof Hoekstra| © Atos
Result
Status summer 2016
▶ 2.5 years extensive use
▶ Much more asylum seekers than planned
▶ More locations than planned
▶ Users are happy – less work – quicker business process
▶ No messing with ink fingerprints anymore
▶ More crimes solved
▶ Improved maintainability
▶ No more dependency on 1 external supplier
▶ Various enhancements realized
▶ Finally, the maintenance contract is signed
28

29. Atos, the Atos logo, Atos Consulting, Atos Worldgrid, Worldline,
BlueKiwi, Canopy the Open Cloud Company, Yunano, Zero Email, Zero
Email Certified and The Zero Email Company are registered
trademarks of Atos. January 2015. © 2015 Atos. Confidential
information owned by Atos, to be used by the recipient only. This
document, or any part of it, may not be reproduced, copied, circulated
and/or distributed nor quoted without prior written approval from
Atos.
Thank You !
Contact:
M+ 31 6 512 88 478
gerlof.hoekstra@atos.net