Kristin Wisnewski: “Measuring What Matters” DesignOps Summit 2019 • October 23-24, 2019 • New York, NY, USA http://www.designopssummit.com

1. CIO Design
Team
IT
Department
(CIO)

3. Did you know?
Cleveland, Ohio
USA*And +25 other countries!
Iceland

4. 51%30%

6. The
CIO Mission

7. The state of IT is a
daily reflection of what
IBM
about its employees
thinks & feels

8. Any given day, hundreds of thousands of IBMers around the
globe…
Book and
expenses
travel
Seek help with a
device
Upgrade a
mobile
device
Attend
meetings
Share or
collaborate on a
document
Search for an
answer
?
Respond to
emails

9. IT pyramid of
pain
Tasks
Applications
End-to-End Processes
Technical Underpinning
The user experience for
any task is impacted by
layers of technology
and decisions.

10. Direct to
the CIO,
with the
mandate
to grow
• Small team
• Narrow tool set
• Assigning work was
simple
• Communication was
frequent and easy
• Measurement was high-
level
• Reach was limited

11. What we do
User Experience
Design
Visual Design User Research &
Analytics
Multimedia design Agile Coaching &
Facilitation
Content &
Communication
Design & Business
Strategy
Design Operations

12. CIO Design
impacts every
aspect of the
work
experience of
IBMers Physical
environment
Tone &
Culture
Tools &
Applications

13. What do we
measure?

14. Monthly metrics
at a glance

15. Year-to-date
research metrics

16. 1H Mailer metrics

17. Reactive
Example application measures
Proactive
• Net Promoter Score (NPS)
• Satisfaction
• Ease & Capabilities (UMUX-Lite)
• Goal completion
• # of help tickets
• Employee workplace preferences
Example experiential measures
Across end-to-end processes that:
• are founded by years of findings
• are cross-domain, cross-application
• incorporate visionary ideas from our team
• provide huge time and cost savings

18. Happiness
User
Goal completion
Productivity
Retention
Engagement
Research operations model

19. Voice of the
Employee
program (VotE)

21. Library of User
Experience (LUX)

22. The primary issue is that we usually have very tight deadlines
to get something done. If we expand the schedule to include
more user research, there will be pushback.
ProductOwner, IBM
”
”Before LUX

23. LUX metrics:
NPS

24. LUX metrics:
Goal
completion

25. LUX metrics:
Ease &
capabilities

26. LUX metrics:
Scorecard

27. LUX metrics:
Comments

28. LUX metrics:
Detailed
comments

29. You've helped us understand that our visitors are a rich
source of information, and that they can also provide the
kinds of insight you simply cannot get anywhere else. We've
learned to always be testing versus always be guessing.
ProductOwner, IBM
”
”After LUX

30. LUX Research

31. “Every minute spent struggling with an
IT system is a minute of time not spent
on delivering value for IBM and our
clients.
Fletcher Previn
CIO, IBM
”

32. Some examples
of research-
driven
improvements
to the IBMer
experience

35. Employee directory
After
Before

36. Employee directory
After
Before
I love BluePages! It makes it easy to
see contacts and also reporting
chains. It's the go-to tool for
employee information.
”
”
IBMer

37. Help system
After
Before

38. Help system
After
Before
I love this site. The information,
format, and delivery is exponentially
better in the last year or two. I use it
constantly to find answers for
myself, my team members, and my
peers.
”
”
IBMer

39. Corporate intranet
After
Before

40. Corporate intranet
After
Before
It looks so fresh, so organic, so
intuitive. It’s the best version of [our
intranet] I have seen so far.
”
”
IBMer

41. Workstation
refresh program
New workstation
every 4 years!
All employees eligible to upgrade
But, only 30% accepted…

42. Different needs,
different system
status
Azin | Mobile Developer
Azin is frustrated that she has to use “old” technology. A new
machine every 2 years would enhance her productivity and
engagement.
Claude | HR Partner
Claude doesn’t need the latest and greatest – his 5 year old
MacBook is working just fine for him and he’d wait another few
years to upgrade.
Soo Yun | Client Rep
Soo Yun is a road warrior who puts many miles on her machine –
she’s in no hurry to upgrade, but has no idea that her 3 year old
PC is about to fail....

44. Measurement is
not without its
challenges.

45. Don't try to control it, or force it to say what you
want. The real goal is for data to help you make
the right decision; not to support the decision
you’ve already made. Let the insights help guide
you to your next set of questions.
Mark Malatesta
DataAnalytics Strategist,CIO Design
”
”

48. NPS for
altered
version
+25.5
NPS for
standard
version
-9.3

49. We are the…
Arbiters of truth
Uncoverers of scandal
Validators of fact
Defenders of the user
experience
Unbiased third party

50. Our DesignOps journey
Phase 1 Phase 2 Phase 3 Phase 4
2015
The need for
DesignOps
2016-2018
The start of
DesignOps
Now
Maturing
DesignOps
2020+
Mastering
DesignOps