Quo vadis 2018 Presentation about Ui Development

1. “How hard can it be”
10 years of Ui development at keen games
Julien Koenen - Technical Director keen core GmbH
This is a presentation I gave at the Quo Vadis 2018 conference in Berlin (24.04.2018)

2. Agenda
● Introduction
● History
● Portal Knights
● Lessons learned
● Q&A
● Bonus
Short overview of the Content:

3. Keen?
Independent game development studio from frankfurt
- Founded 2005
- Before: NEON Studios (1992-2005) - Sold to Jowood
- Most of the founders of ‘92 are still active in the company
- Split up into 3 companies:
- keen flare (Part of flaregames) - Mobile F2P Games (Royal
Revolt 1+2, Olympus Rising, Monsters with Attitude)
- keen games (productions) - PC/Console games - currently:
Portal Knights
- keen core - Engine + Tools + Middleware
- Currently around 50 people
- Keen games: 20
- Keen core: 5
- Keen flare: 26

4. Keen!
- More than 30 games over the last 25 years
- Tunnel B1 (DOS,PS1,Saturn)
- Legend of Kay (PS2) -> one of the few big PS2 games produced in
germany
- Anno: Dawn of Discovery (WiiU/NDS) - one of the few games with > 80
metacritic from germany
- TNT Racers -> the Only Wii-Game with DLC Support in Europe
- Sacred 3 (PS3,Xb360,PC)
- Royal Revolt 1+2 -> f2p games
- Portal Knights -> Early access / shipped on 6 platforms / continuous
development (> 1MPlayers)
- Now mostly working on Games-as-a-Service and F2P Mobile games

5. Keen!
- Shipped games on more than 20 platforms
- PS1, Saturn, PC/DOS, GameBoy Color, GBA, PS2, Wii, PSP,
PC/Windows, PS3, Xbox, Xb360, NDS, WiiU, iOS, Android, WP8,
UWP, PS4, Xb1, …
- Custom tech all the time
- One short adventure trying to use RenderWare on PS2

6. Game UI is...
● The interface to between player and game code
● Design and Logic
● Often underestimated
● Changing all the time
● Lots of Approaches
● Reinvented all the time
- Ui is the way for the player(s) to interact with the gameplay code
○ That means it always has to share / interpret data from the game
○ Tight coupling to the gameplay features (which changes all the time)
- Traditionally two separate roles:
○ Ui Artist / Designer - responsible for the ‘look’
○ Ui Coder/Scripter - responsible for connecting the Ui Art with the gameplay
code
- Look&Feel / Design of the UI
○ What is placed where / how do thinks look
○ How do elements move / animate over time
○ Each game usually has its own unique design
○ Lots of iterations required for good results
- Logic / Functionality
○ How do the elements react to input
○ How is the game data transformed into changes in the Ui
- “It’s just Ui Code” - usually the complexity of the Ui Code was wildly underestimated
○ Therefore the UI Code is done by the least experienced programmer in the
team
- No common solution for the last games at keen. Some historic examples:...

7. Dance Dance Revolution
Disney Channel Edition
Platform: PS2
- New Version of the Popular DDR IP
- Complete rewrite (used the source of other versions as reference)
- Except for some background video and 1-2 Dancing characters everything
visible is Ui
- Very flashy/animated -> everything is moving all the time
- Single Platform

8. Dance Dance Revolution
Disney Channel Edition
Platform: PS2
● Design in 3DS Max
● Code in C++
● Strict separation
● Pro: Familiar Tools
● Pro: Animations
● Pro: Iteration speed on some things
● Con: Dependency on Coder
● Con: Iteration speed for most things
● Con: Ui ‘Feel’ in ‘wrong’ hands
- The UI Artist (who was doing the Character Models as well) was pretty
familiar with 3D Studio Max.. so we created a workflow based on MAX
- Based on ‘dialogs’ and ‘elements’
- Whole dialog layouts where defined in MAX (with baked fade in /
fade out / idle animations)
- Connection of ui code with ui layout was done through name
lookup
- Strict separation of UI Code and UI Design
- Ui Code was C++ written by the gameplay programmer
- Ui Design was done in 3D Studio Max
- Pro:
- Each party used familiar tools
- Animations where no problem at all
- Ui Designer could quickly iterate on design-only changes
- Cons:
- Ui Designer had to wait for the Programmer to see anything new
in the game
- Iterations involving code-changes were slow
- All of the Ui “Feel” was developed by the ‘wrong’ person

9. Sacred 3
Platforms: PS3, XB360, PC/Steam
- Action RPG for Consoles
- “Gauntlet” Style Combat
- Lots of iterations because the game changed a whole lot during development

10. Sacred 3
Platforms: PS3, XB360, PC/Steam
● Ui Code in Script
● Ui Layout in Data
● Pro: Ui Code separate from GP Code
● Pro: Lots of flexibility
● Con: Buggy Tools
● Con: No Experts
● Con: Iteration Times
- UI Code written scripting language
○ Idea: “easy enough” for the Ui designer
○ Separate Tool to define the ‘Layout’ of Ui Screens
- Pro
○ Both Layout and Ui-Code should be written by the same person
(In Theory)
○ Clear separation between Ui Code and gameplay code
○ Standalone tool to test / develop the Ui Requests without the
actual game (input/output was defined)
- Con
○ Buggy Tools (Debugger for the Scripting language came too late)
○ Most of the Ui-Code was written by someone else in the end
anyway (wasn’t easy enough)
- Goal was clear separation between UI Code and gameplay code (don’t
mix
- KScript: simple functional language .. debugger (late)
- ‘Request’ style ui: the gameplay code started async ui requests and
interacted with them (send data in / out)
- Each request corresponded to one ui script

11. Olympus Rising
Platforms: iOS, Android, UWP
- Mobile F2P Title (still running)
- “Game as a Service” -> Looong lifetimes
- Lots of changes over the Lifetime of the title (especially in Ui)
- The whole “meta-game” required for F2P mobile titles is mostly implemented
through Ui

12. ● Ui Code in C++
● Layout + Logic in Code
● Mockups by Ui Designer
● Pro: Simple Solution
● Pro: Shared Code
● Con: Ui Designers had to wait
● Con: Coder “Art”
● Con: Iteration times
Olympus Rising
Platforms: iOS, Android, UWP
- Ui Code written in C++ by gameplay coders
- Layout + Logic defined in Code
- Ui Designer created ‘mockups’ in Photoshop and Unity for new dialogs
- ‘Retained’ mode Ui style (MVC)
- Pro
○ ‘Simpler’ Solution .. only one place+language for everything (no
converter/exporter)
○ Shared Code -> the Ui Code was just like ‘normal’ GP Code
(Code reviews, coding standards, …)
- Cons
○ Ui Designers had to wait for coders (for everything!)
○ Ui was pretty static / fixed (partially due to the restrictions of the
mockup tools)
○ Iteration times were not great.. Build+Restart the whole game for
every change

13. Portal Knights
Platforms: PC/Steam, PS4, XB1,
Switch, iOS, Android
- Early Access title -> very tight schedule until first shipped version
- PC only in the beginning
- Big push to switch the ui system after Sacred 3
- Everyone else is doing it…

14. Portal Knights
Platforms: PC/Steam, PS4, XB1,
Switch, iOS, Android
Hopes:
- Ui Designers can do everything
- Ui decoupled from Gameplay code
- Stable / familiar tools
- Easier to hire new UI Designers
- “Visual” workflow would lead to better UI Designs
- The ‘real’ programmers don’t need to deal with UI Code anymore

15. Scaleform - The good
● We shipped
● Porting to XB1/PS4 was (pretty) easy
● Happy Ui Designer
● Ui Code isolated
- We actually managed to ship the first version of Portal Knights into
Early Access using Scaleform -> it worked!
- We managed to port the scaleform UI to PS4+XB1 with manageable
effort
- The UI Designer was pretty happy in general
- The UI Code was completely isolated from the gameplay code and all
data was moved through a narrow interface

16. Scaleform - The bad
● Performance
● Bugs
● Complexity
● Tools
● Iteration Times
● Hiring
- The Performance turned out to be pretty bad
○ Especially when porting to Consoles and Mobile later (PC was
less of a problem)
■ We had to run Scaleform in a separate thread in parallel
to the client on console (took more time than the game)
○ Mostly hidden performance costs in value-copies, cache
invalidation
○ Pretty easy to hit a ‘slow path’
○ Surprising cost for some render effects (scissor using stencil
buffer was pretty slow on mobile)
○ No (easy) way for us to optimize
■ Problems were detected rather late..
- Bugs:
○ Several memory leaks
○ The Console backend of Scaleform had a couple interesting
bugs
○ At least we had the source code .. so it was possible to fix these
- Complexity
○ Huge C++ Codebase with no experts around.. Wasn’t really fun.

17. ■ Around 800k LOC
○ Lots of dependencies to third party libraries (pcre, curl, …) we
didn’t want / need but where used in several places
○ Version conflicts with some of the third party libraries
- Tools
○ No debugger for the Ui Code
- Iteration Times weren’t great as well..
- Hiring
○ Wasn’t really easier to find Ui Designers with Flash + Game
background

18. Scaleform - The ugly
● No Switch support
● Performance on Android + iOS was *really* bad
Then after the PS4/XB1 port was done we got asked if we could do a Switch version
of the game..
- No Switch Support in Scaleform….
And while we are at it: What about Mobile?
- Around 50-60 ms on Mid Level Android Devices

19. Scaleform - The ugly
● No Switch support
● Performance on Android + iOS was *really* bad
Then Autodesk killed Scaleform
And finally: Scaleform gets canceled
- The biggest problem with third party middleware: No control..
- Usually you want to wrap it to be safe..
- Not possible in this case (The interface is basically the whole Ui Code)

20. Scaleform - What now?
● Option 1: Fix it
● Option 2: Split it
● Option 3: Get rid of it
- Option 1: Write our own scaleform backend + optimize performance
○ We had the source code .. so at least we had this option
- Option 2: Don’t use scaleform on switch+mobile
○ Would be pretty painful to have multiple implementations of the
Ui
- Option 3: Don’t use scaleform
○ Buy something else?
■ Nothing on the market (Iggy1 has no Android support)
○ So there is only one Way: get rid of Scaleform

21. Scaleform - The way out: Ui2
Rewrite all Ui from scratch:
● Step 1: Hide Scaleform
● Step 2: Implement new Backend
● Step 3: ???
● Step 4: Profit!!
In 2017 we decided to abandon Scaleform and switch over to a new system
Rewrite all UI from scratch:
- Step 1: Abstract all Scaleform code from the rest of the codebase
- Step 2: Implement new ui2 backend on the side (because we could still
ship with scaleform on other platforms)
- Step 3: Switch over to the Ui2 Backend on all platforms
- Step 4: profit!! (Delete all Scaleform code)
It wasn’t really clear how long it would take
- 62k LOC Actionscript3 code + 15k LOC of C++ Code

22. Decision Time: What language for Ui2?
● Option 1: Declarative language
● Option 2: Scripting language
● Option 3: Use C++ ?
We still had to decide what the alternative (Ui2) would look like:
- Option 1: Use a (custom) declarative language
○ Not flexible enough.. Would stil need some kind of ‘real’
programming language
- Option 2: Use a scripting language
○ We tried that before on Sacred3 + TNT Racers -> “Friction Costs
tend to be high”
- Option 3: Use C++ for everything
○ Only used C++ for the ‘Code’ part before
○ Unclear how to define the layout/look efficiently in C++ code

23. C++ as Ui Language - Pros
● Support
● Performance
● Tools
● ‘Native’ data model
● All Platforms
● We can do everything
● Compatibility
- Better support from our team (all of them are C++ experts, none of them
knows Actionscript/Javascript/…)
- Better baseline performance
- An actual Debugger, Profiler and we can use all the in-house tools we
developed for C++
- ‘Native’ data model binding -> no data model transfer overhead
- Support for all platforms
- We can do everything / unbounded flexibility
○ Not restricted by a Scripting Language API / third-party SDK /
- Compatibility with the rest of the code base

24. C++ as Ui Language - (Potential) Cons
● C++ is ‘hard’
● Memory management
● Iteration times
● Productivity
● Crashbugs
We had to find solutions to some potential problems to make this work
C++ has an image of a ‘hard’ language.. This is mostly due to the insane
complexity of the language combined with
- Manual memory management is really hard
- Iteration times are bad (rebuild the game for each change?)
- Rewriting all the code will be a nightmare / take too long
- Ui Code Bugs will crash the whole Game
So we tried to tackle these issues:

25. C++ as Ui Language: Make it simpler + faster
● Make it simpler:
○ Immediate UI
○ No explicit memory management
○ Lots of helpers
● Faster:
○ Hot reloading
○ Fast build times
● Handle Crashbugs
- Simpler:
○ Use Immediate UI Method to simplify the code
○ Remove all explicit memory management needed in UI Code
○ Remove need for pointer management in UI Code
○ Add helper / wrapper for common things
- Faster iteration times:
○ Hot reloading
○ Fast build times (Separate .DLL project with minimal
dependencies + restructuring of the physical layout to optimize
build times)
- Sensible handling of Crashbugs in UI Code

26. Immediate Ui - Super Short Intro
vs.
- Short introduction into immediate mode ui concept
- On the fly creation of widgets
- No explicit create/destroy
- No callbacks for event handling
- Term coined by Casey Muratori (public video presentation in 2005)
- DearImGui/Nuklear are popular imgui implementations
- Allegedly used in Unity3d (I wouldn’t know;)

27. Immediate Ui - Advantages
● Dynamic
● Simpler
● Hot reloading is easy
● Some experience
● Composable
- Much more ‘dynamic’ binding to game state
- Layout is implicit / code driven -> so we can do lots of dynamic tweaks
depending on the state / platform / timing
- if( platform == ps4 ) { if( button( “ps4 specific thing” ) {
doPs4Thing() ) };
- No callbacks
- Memory Management is a lot easier (and faster!)
- State is built up each frame .. so it’s pretty easy to support hot-reloading
- Our programmers used this for debug ui for a couple years already
- Composable
- doComplexButton() { … doButton() … };

28. Ui2: Multi Pass Immediate Ui
● Linear List of UiFrames
● Additional Stack for Hierarchy
● Three passes:
○ Layout
○ Render
○ Input
● Same code executed each time
● Different state
Three Passes
- Layout - collects layout information from the widgets
- Input - reacts on input events
- Render - records render commands
- Potential problem: the ui code can pretty easily break when it behaves
differently in the different passes

29. ● Collect all UiFrames with a unique id
● Store layout parameters
● Layout Frames
● Flexbox and Grid layout
● Optional scrolling of content
● Store Result
Ui2: Layout
- The first pass is just collecting all the Ui Frames and assigning them a
unique id
- During this pass we store all the input parameters for the layout
algorithm (min/max/preferred size, layout of child elements, padding,
…)
- After that pass is done we run over the frame hierarchy bottom up to
propagate the size constraints upwards and after that we traverse from
the top down to actually place the frames in their final position
- We support a simple ‘flexbox’ style and a grid layout method for child
frames
- Optional Scrolling is implemented through a simple ‘content offset’ in
each frame and a scissor rect during rendering
- The Final result of the layout pass (basically a rectangle for each frame)
is stored for the later passes

30. Ui2: Rendering
● Simple API
● Use layout data
● Collect Draw Commands
● Scissor Rects
● Transforms
● Layers
● Compositing
- Simple drawing API for lines/rects/(textured)quads/shapes
- Sits on top of our cross-platform Graphics Api abstraction
- The Ui code will call drawXXX functions which create
UiDrawCommands
- All draw commands are gathered and sorted according to sort key
- Support for Scissor Rects (Push/Pop Style)
- Support for Transforms (Scale/Offset only)
- Support for overlapped ‘layers’ within the same Window
- Simple ‘compositing’ API

31. Ui2: Composite Rendering
Renders to dynamic offscreen targets
Support for a couple blend modes
Easily extensible when required
Used a lot for transition animations (fading/scaling/moving/rotation of whole windows /
hierarchies)

32. Ui2: Input Handling
● Input Events
● Use Layout Data
● Focus handling
● Local Multiplayer (Window based)
● Text Input
- Each input event coming in will create a Ui Pass
○ We merge adjacent mouse move events
- Player id attached to Input event (for local multiplayer)
- Input Device Id attached to Input event
- Frames can filter input based on player id / device type
- Full support for keyboard/mouse/gamepad/touch
- Each frame can mark itself as ‘focusable’
- On each level of the tree a focus frame is maintained
- The focus frame gets the keyboard / gamepad input
- Support for virtual and real keyboard text input

33. Ui2: Wrapper: Windows
● Special Frames
● Virtual Size
● Scope Style API
Wrapper for Window - Frames
- Windows can overlap / move
- Windows have a screen transform (aligned/scaled on the screen)
- Each contains a tree of UiFrames
- Constructor ‘pushes’ the window and destructor pops it again

34. Ui2: Wrapper: Layouts
● Wrapper for Layout Frames
● Scope API
Another wrapper for ‘layout’ frames
- Internally just push_frame / pop_frame
- Can’t forget the pop_frame

35. Ui2: Wrapper: Widgets
● doXXX calls for widgets
● Wrap frame hierarchy
● Complete input and output as parameters
● Composable (>100 Widget Functions)
Wrappers for widgets (mostly functions.. Some classes)
- Each doFloatValue() creates the UiFrames necessary for the Widget
and adds them as child frames to the currently open Layout frame
- Pretty minimal information
- The hasMouse is a good example for the simplicity of dynamic /
platform-specific UI adjustments

36. Frame “State” Memory
● ‘Persistent’ State API
State memory is bound to the frame .. will be removed automatically
- Use explicit “States” for data that should persist across frame
boundaries:
- Uses a simple hashmap using the frame-id as key
- Two lifetimes: “Temporary” and “Persistent”
- Temporary State will be removed when the Host-Frame is not
present anymore (so when during a ui pass you don’t call
newState<> on a specific frame-id - this is mostly useful for
animation-states (like the example) and other widget-local states
- Persistent states will live throughout the whole game -> they are
never removed
- Useful to remember the last focus element in a window so
on reopen it will focus the same element again

37. Ui2: Results
● Performance
● Amount of Code
● Iteration times
● Support from Other Coders
● Simple C++
● Ui Designer (pretty) Happy
● Lots of ‘Magic’ Numbers
- Performance
○ Android: > 50 ms -> 4ms / Frame
- Amount of Code
○ 46k LOC C++ Code vs 62k LOC As3 Code + 14k C++
- Iteration times
○ 1-3 seconds after any change of the code the results are visible
○ In-place live editing of values through tweakable variable
interface
- Gameplay programmers can (and do) help with UI Tasks / Bugs
- Engine coders can (and do) help with performance / new features
- The concrete Ui Code contains lots of ‘magic numbers’ for sizes /
positions.. But that is fine it’s the ‘design’ of the ui and the whole idea is
to have it inline in the code

38. ● No silver bullet
● Work with your team
● Ui Code is Code
● DIY!
● Iteration times!
● Wrap your middleware!
Ui2 Conclusion
- Very happy with the results -> but no silver bullet..
○ Depends on your team structure and the game
- Our way forward for the next projects
- You should use whatever works for you
- We still could improve the density of the UI code and try to share more
between projects
- We plan to experiment with a serialized / data representation of the
Frame Data to allow a more traditional “Ui Editor” for simple cases
- We actually decided to replace QT with the immediate Ui Framework for
our (new) Tools

39. Questions?
Contact: julien@keencore.com

40. Idea: Convert the ActionScript to C++
● Parse Actionscript
● Convert to C++
● Compile C++
● Profit ?
- The idea was to parse the ActionScript code and create equivalent C++
code. Our thought was that having an actual optimizing compiler +
profiling tools would be enough to get the performance up.
- We actually implemented an ActionScript 3 parser and were able to
create the AST of the Ui Code (62k Lines of code)
- Lots of memory allocations (Action Script uses a GC -> no such thing in
C++)
- Big Flash interfaces we would have to reimplement
- Actionscript was not the main problem.. It was the concepts / the used
API was too abstract / disconnected

41. Bonus: Hot reloading of C++ code
- It’s awesome!
- Pretty easy
- 1-3s
- Can change everything
- Lots of different ways
- We decided to just export the main entry point (doPkUi) from a .dll and
have a separate .dll project
- The game executable still contains the complete ui code as well as a
fallback / for shipping
- Build times are pretty good.. Around 1-3 seconds.
- Instantly replaces *all* ui code -> no restriction on which part can be
hot-reloaded
- No state kept.. All Frame-States are ‘new’
- Structured exception handling around the Ui code -> fallback to the .exe
code

42. Bonus: Text Rendering
● Trade-Offs
● SDF Glyph Renderer
● Parse .TTF / .OTF at runtime
● Emojis
● Simple Layout
- A *very* complex topic
- We implemented a Signed distance field (SDF) font renderer
- Directly parse .TTF files, extract the outlines and create the SDF at
runtime
- We support .TTF and .OTF files and can use the system fonts on
Android and iOS directly (as a fallback) (mostly to reduce size)
- Partial support for Emojis
- Text Layout is pretty simple / incomplete but enough for our games -
We don’t write Text Editors
- Thought about buying a third-party library.. But decided against it
○ We want to control the code of critical components
○ We have to support mobile
○ We wanted to support the system fonts to reduce download size)
○ We want to make the trade-offs ourselves

43. Bonus: Stepping through the Code
A (very) shallow step through the main Ui code..
The main entry point for the Ui code is called doPkUi()
PkUiContext contains references to all data we can reference
This function is called for each Ui pass (for each input event, layout+render)

44. Ui2: Entry Point: multiple players
Split into ‘player ui’ which is player specific (we support local-coop) and ‘system’ ui
which is global

45. Ui2: Entry Point: multiple players
Very easy to read at a top-level .. simple to understand / modify / extend

46. Ui2: Widget implementation
Just a simple real world code snippet for a (Portal Knights specific) widget

47. Ui2: Widget implementation

48. Bonus: Debug Visualization of Frames
We implemented a simple debug ui to visualize the Ui frame Tree at runtime and
highlight the frames when hovering over them
This is pretty nice already but we are thinking about even better tooling to inspect /
browse the dynamic part of the ui state