SLIDES WITH NOTES: http://bitly.com/rej-practical-ai
This talk is an introductory material for students and programmers aspiring for developing AI for games.
Talk is split into 2 parts - first part provides an overview on popular AI approaches in games and second part gives an outlook on technology that might be relevant for games AI in the future.
I gave this talk at Vilnius University as a guest speaker in the late October 2016.
1. COPYRIGHT 2016 @ UNITY TECHNOLOGIES
PRACTICAL AI
IN GAMES
Renaldas Zioma
Unity Technologies
(INTRO LEVEL)
2. COPYRIGHT 2016 @ UNITY TECHNOLOGIES
INTRO
• Why graphics programmer is talking about AI?
3. COPYRIGHT 2016 @ UNITY TECHNOLOGIES
NO AI IN GAMES!
• No “intelligence”
4. COPYRIGHT 2016 @ UNITY TECHNOLOGIES
NO AI IN GAMES!
• No “intelligence”
• This statement is 99% true
5. COPYRIGHT 2016 @ UNITY TECHNOLOGIES
NO AI IN GAMES!
• No “intelligence”
• This statement is 99% true
• Behaviors are predefined - game does not learn!
6. COPYRIGHT 2016 @ UNITY TECHNOLOGIES
NO AI IN GAMES!
• No “intelligence”
• This statement is 99% true
• Behaviors are predefined - game does not learn!
• … but going to change soon-ish
7. COPYRIGHT 2016 @ UNITY TECHNOLOGIES
WHY?
• Hard!
Chess
• Average branching factor = 35
• Average depth = 80
• Search space = 1080.
Go
• Branching factor = 250
• Average depth = 150
• Search space = 10360
• Branching factor = 3050 to 30200 !!!
• Average Depth = ???
• Search space =
Starcraft
💣 💣 💣
8. COPYRIGHT 2016 @ UNITY TECHNOLOGIES
AND NOT REALLY THE POINT!
• AI is not to “solve” your game
• Entertain player
11. COPYRIGHT 2016 @ UNITY TECHNOLOGIES
DESIGN PRINCIPLES
• Relatable - if player does not understand that NPC is “smart”, you just
wasted your time
12. COPYRIGHT 2016 @ UNITY TECHNOLOGIES
DESIGN PRINCIPLES
• Fun - instead of “just hard”
• Responsive - slow feels “stupid”
• Relatable - if player does not understand that NPC is “smart”, you just
wasted your time
• Player should feel “smart” - not AI or you!
13. COPYRIGHT 2016 @ UNITY TECHNOLOGIES
DESIGN PRINCIPLES
• Fun, Responsive, Relatable
• Player should feel “smart”
• AI is a tool for game designer!
14. COPYRIGHT 2016 @ UNITY TECHNOLOGIES
PERFORMANCE
• You have 3 ms for AI !!!
• Graphics, physics, audio - first, because crucial for immersion
• GPU does NOT do all graphics work!
• AI usually is time sliced - executed over several frames
1 second = 1000 ms
30 FPS = 1000 / 30 = ~33 ms
60 FPS = 1000 / 60 = ~16 ms
15. COPYRIGHT 2016 @ UNITY TECHNOLOGIES
AI “STACK”
• Example from Robotics
• Autonomous Car Architecture
Navigation
Behavior Planner
Driver
Motion Planner
Trajectory Planner
2Hz
60Hz
10Hz
5Hz
1Hz
FinerscaleCoarserscale
Nicolas Meuleau
Unity Technologies
exApple, exNASA
Director of AI Research
16. COPYRIGHT 2016 @ UNITY TECHNOLOGIES
AI “STACK”
• Games
Navigation
Animation controller
Trajectory, Steering
Behavior
2Hz
60Hz
10Hz
24. COPYRIGHT 2016 @ UNITY TECHNOLOGIES
COMMON PROBLEMS
• Static world
• dynamic obstacles are easier
to support with grid
• Actor size is predefined
• Fast moving actors with steering constraints
• wheeled vehicles
25. COPYRIGHT 2016 @ UNITY TECHNOLOGIES
COMMON SOLUTIONS
• Delegate solution to lower-level systems
• steering for dynamic obstacle avoidance
• Several data structures
• different data structures can co-exist
• several copies with different parameters
40. COPYRIGHT 2016 @ UNITY TECHNOLOGIES
PACMAN
• I felt it would be too stressful for a human […] to be continually surrounded and
hunted down. So I created the monsters’ invasions to come in waves. They’d attack
and then they’d retreat. As time went by they would regroup, attack, and disperse
again. It seemed more natural than having constant attack.
- Toru Iwatani, Pac-Man creator
• Just before ghosts turn a corner, their eyes move to point in the new direction of
movement
- Ghost Pshychology, www.webpacman.com
48. COPYRIGHT 2016 @ UNITY TECHNOLOGIES
BEHAVIOR TREE
• Pros
• better at complexity than FSM
• used in many games: Halo, Bioshock, Crysis, Spore
• Cons
• large BT can get slow
• hardcoded decisions
53. COPYRIGHT 2016 @ UNITY TECHNOLOGIES
UTILITY AI
• Pros
• can achieve “smarter” behavior than BT
• Killzone, Sims, Civilization
• Cons
• can be hard to design scorers and debug
• some loss of designer control
• still pretty hardcoded logic
54. COPYRIGHT 2016 @ UNITY TECHNOLOGIES
FSM / BT / UTILITY
HOW TO CHOOSE?
• Pick approach you are most comfortable with
• Pick the simplest tool that will make job done
55. COPYRIGHT 2016 @ UNITY TECHNOLOGIES
PLAN AHEAD
• FSM, BT, Utility AI - reactive
• only reacts to current situation
• no planning
• Deliberate AI
• plans ahead
56. COPYRIGHT 2016 @ UNITY TECHNOLOGIES
INFLUENCE MAPS
• Add Behavior related
information to path planning
• Simple way to plan for
actions ahead
• piggy-back pathfinding code
• Better decisions
67. COPYRIGHT 2016 @ UNITY TECHNOLOGIES
INFLUENCE MAPS
• Types of data:
• covers
• distance to enemies
• distance to power-ups
• heatmaps
• etc…
71. COPYRIGHT 2016 @ UNITY TECHNOLOGIES
PLANNING IN GAME AI
• Goal Oriented Action Planning
72. COPYRIGHT 2016 @ UNITY TECHNOLOGIES
GOAP
• Games: F.E.A.R, S.T.A.L.K.E.R, Fallout 3, Just Cause 2
73. COPYRIGHT 2016 @ UNITY TECHNOLOGIES
GOAP
• Pros
• better decisions
• emergent gameplay
• Cons
• performance! worse than NP-hard!
• loss of designer control
• so far didn’t became common approach
• maybe needs better tools
74. COPYRIGHT 2016 @ UNITY TECHNOLOGIES
BIOLOGICAL BRAIN
• Consists of Neurons
• Ultra low power consumption
• Both DIGITAL and ANALOG!
75. COPYRIGHT 2016 @ UNITY TECHNOLOGIES
NEURON
• Can compute
• Can learn
• Can generate patterns
• Form networks - highly interconnected
• Very parallel - but single one is relatively slow
78. COPYRIGHT 2016 @ UNITY TECHNOLOGIES
NEURON
• Both digital and analog!
• Working principle includes 3 things at once:
• Electrical
• Bio-mechanical
• Chemical
• IMHO in the 21st century you must study some neuroscience!
DO IT!
79. COPYRIGHT 2016 @ UNITY TECHNOLOGIES
NEURON, APPROXIMATELY
• Sums up weighted inputs
• Makes binary decision
• Sends to connected neurons
92. COPYRIGHT 2016 @ UNITY TECHNOLOGIES
CEREBRAL ORGANOIDS
• Easier way to study and learn from real “intelligence”
• Eventually will be connected to computers
• biological AI co-processor
94. COPYRIGHT 2016 @ UNITY TECHNOLOGIES
BACK TO GAME AI
• Machine Learning is more like Intuition
• Planning is more like Reason
• Need both
Notes de l'éditeur
“Practical AI in Games” talk is an introductory material for students and programmers aspiring for developing AI for games.
Talk is split into 2 parts - first part provides an overview on popular AI approaches in games and second part gives an outlook on technology that might be relevant for games AI in the future.
I gave this talk at Vilnius University as a guest speaker in the late October 2016. Thanks Žilvinas Ledas for invitation.
Lingo: NPC is a Non-Player Character. Synonymous to Actor.
Of course it is a controversial statement and does not apply to all games. It is here to grab your attention…
… and to draw line between “scientific AI”, human intelligence and what we actually use in games.
This talk is split into 2 parts.
First part is more practical and covers set of approaches that are most used in today games.
Second part aims to give an outlook about future of AI and what young professionals should keep an eye.
Traditional games like Chess and Go have well defined discreet states. In theory computer AI could beat the game by looking through all possible states and finding the best solution. In practice however it is extremely computationally expensive and memory hungry task. Search space is number of all possible moves (actions) in such game.
However state in most modern games is far from discreet (unit can move just “a little bit”) and have complex rules - dramatically increasing search space.
Although task at hand is extremely complex - complete and optimal “solutions” belong more to the domain of “scientific AI”.
Instead as game developers we need to entertain the player!
“Glued to Games: How Video Games Draw Us In and Hold Us Spellbound” is a great book disassembling what does fun actually mean.
https://www.amazon.com/Glued-Games-Video-Spellbound-Directions/dp/0313362246
Take a look at the image - the player have thrown grenade just a couple of frames before, however NPC (Non-Player Characters) are reacting already - jumping aside or hiding! This is a good example of responsive NPC behavior.
Rather simple, but well timed behavior is better than smart, but late one.
Players must relate to and understand behavior of NPCs to fully appreciate it.
Animations, audio and visual clues are at least as important as AI code behind NPC decisions.
In the end it is all about making good game.
If your game designer (1) does not understand what AI code can achieve, (2) thinks AI is not predictable enough, (3) has insufficient tools or (4) code is not solving tasks at hand - he will not be able to tap into the full potential of AI. You wasted your time!
ms - millisecond, is one thousandth of a second.
You can realistically expect to spend on AI somewhere between 1 and 5 milliseconds during a single frame. Not more than that! In other words, if you would turn off everything and just keep AI code running - your game should be running between 1000 and 200 frames per second. Unless you are making turn-based strategy game of course :)
Why? Keeping constant framerate is very important for player immersion, reaction time and for perceived quality of the game. And it is even worse with VR - people will literally puke when framerate drops. Therefore tasks that are necessary to run and complete during each frame are of highest priority, such are - graphics, physics, animations, audio, input processing, network, etc. Note that in order to keep Graphics Processors (GPU) busy lots of work needs to be executed on the main CPU first (even with modern graphics APIs like DirectX12, Vulkan or Metal). Modern engines will strive to distribute the work across all cores too. In other words CPU with all its cores is quite busy in games.
Good news for AI some decisions can be spread over several frames to execute. Slow human brains often won’t notice.
For the sake of broadening our horizon - let’s look at AI “stack” in real-life scenarios from Robitics and self-driving Cars.
Different layers are executed at different frequencies (for performance reasons). Lower layers that require immediate reaction and are critical for survival are executed more often. Higher layers are responsible for long term planning.
If condition changes - elk jumps in front of the car - lower layers will react immediately and might trigger higher layers to re-evaluate situation / re-plan further actions.
Navigation - responsible for finding efficient route to the gaol - decides on which road intersection to turn. Does not concern itself with details like driving on a straight road however.
Behavior - responsible for optimizing driving style like gas consumption vs speed.
Driver - execute physical actions.
Motion Planner - plans ahead how to drive in the next intersection, when to change lanes, etc.
Trajectory Planner - calculates actual trajectory given car speed, physical parameters. Reacts to sudden changes like appearance of unpredicted obstacles.
AI “stack” in games is somewhat similar.
Behavior - gameplay level decisions: attack, seek cover, reload, etc.
Navigation - finds route to the next location as commanded by Behavior.
Animation - play animations accordingly to NPC actions and intentions.
Trajectory - moves NPC from frame to frame as commanded by Navigation and in accordance with Animations layer. Reacts to dynamic obstacles.
Next we are going to look at each layer separately.
Navigation aka Pathfinding - finds route between two locations.
There are different ways how to subdivide world into set of discrete interconnected locations.
Read more: https://en.wikipedia.org/wiki/Pathfinding
Grid is a 2D array covering the world. Grid locations can be blocked or open for navigation.
NavMesh or Navigation Mesh is a simplified geometrical representation of the world. Polygons represent locations accessible for navigation. Actor can move on top of NavMesh polygon freely without encountering any obstacles.
Read more: https://en.wikipedia.org/wiki/Navigation_mesh
Waypoint graph - nodes represent locations accessible for navigation. Edges connecting nodes represent paths between locations that are obstacle free.
Read more: https://en.wikipedia.org/wiki/Waypoint
Grid naturally suit 2D or 2.5D worlds. Dynamic obstacles can be represented relatively easy by changing state of grid elements as occupied by NPC (Non-Player Character).
Bigger NPC units usually will cover several elements of the grid.
NavMesh naturally suit 3D worlds. Notice on the image that NavMesh (blue color) is narrower than the actual passages - size of the actor has to be implicitly “baked” into the NavMesh, otherwise actor would stick through the walls while walking.
Producing NavMesh from dense world geometry can be a complex task, luckily there are robust libraries like: http://masagroup.github.io/recastdetour/
Waypoints naturally suit 3D worlds. Waypoints provide easy control for designers to define accessible areas for NPC, but that of course means more manual work to setup.
Nowadays waypoints are falling out of favor and are usually replaced with NavMesh due to automated process.
All described approaches work best for (1) static world, (2) where all NPCs are roughly the same size and (3) can turn on a spot easily.
It is quite common to combine several data structures in one game.
For instance in RPG game NavMesh approach can be used for navigation inside the houses, but Waypoints could be used to represent the outside world. For Strategy game multiple grids of different resolution can overlap in order to handle dynamic obstacles where it matters more, etc.
Read more: http://www.red3d.com/cwr/steer/gdc99/
Navigation layer alone provides us with very rough path (red line). If actor would follow it blindly it would look very unnatural and “robotic”.
One task for Steering layer is to take information from Navigation and create more believable path (green line).
Another task is to avoid dynamic obstacles that Navigation does not want to deal with. It is important to remember that while steering can avoid dynamic obstacles it can lead NPC far away from the path planned by Navigation. When that happens Steering will usually trigger Navigation to reevaluate the path starting from the current location.
For both Path simplification and Dynamic obstacle avoidance engine will fire additional (short) ray-queries to check, if path is blocked.
Example of Steering for NPC character following the player (“Buddy follow”) while staying behind him, but visible through the camera. Yellow spheres depict possible locations, red arrows point to locations that are blocked by the rock.
Worth reading: http://allenchou.net/2016/05/a-brain-dump-of-what-i-worked-on-for-uncharted-4/
Read: https://en.wikipedia.org/wiki/Finite-state_machine
FSM is a graph where:
nodes are states of NPC “brain”
edges are conditions that will make “brain” jump (transition) from one state to another
NPC actions are associated with both states and transitions
FSM is a simple “map of the brain”. What makes NPC act - actions assigned to every state or transition. As AI designer you would layout FSM, implement various actions with script and assign those actions to FSM states and transitions.
Example of script code that is attached to FSM and drives actual behavior - in this case spawning new particle and pushing it away with force.
OnStateEnter, OnStateExit - functions that execute once when FSM enters/exits particular state
OnStateUpdate - function is executed every frame while FSM is in a particular state.
Example of FSM from a real game.
FSM are great as a starting point in games AI.
Hierarchical FSM: http://aigamedev.com/open/article/hfsm-gist/
Lets take classic Pacman game as an example and see how AI is implemented there.
Watch Pacman gameplay: https://www.youtube.com/watch?v=uswzriFIf_k
More about Pacman gamedesign, ghost behavior, etc:
http://gameinternals.com/post/2072558330/understanding-pac-man-ghost-behavior
http://www.gamasutra.com/view/feature/3938/the_pacman_dossier.php
https://www.khanacademy.org/computer-programming/pac-man-ghost-ai-visualisation/6163767378051072
Although Pacman is small and old game - it is a great example of the good design principles for AI.
Keep close attention to ghosts in Pacman:
just couple of frames before making turn, they will turn their eyes towards intended direction - that gives a glimpse of intent
ghosts reverse direction when important state of the “brain” changes - that gives opportunity for player to know of change in behavior
they change speed of movement and color when they become “eatable” - player knows whenever ghosts are dangerous or not
Ghosts in Pacman have 4 states of the “brain”. In “Scatter” state they go towards predefined corners and stay there for a short period of time.
In “Attack” state every ghost has slightly different behavior that affects Navigation. Navigation in Pacman is extremely simple, just a single goal that ghost is trying to reach. In case of a red ghost Navigation goal is player itself. In case of a pink ghost - goal is always slightly ahead of the player, etc.
Steering in Pacman is extremely simple too - at every intersection ghosts turn in the direction that would bring them closer to the goal commanded by Navigation.
Read: http://gameinternals.com/post/2072558330/understanding-pac-man-ghost-behavior
Visualization of FSM behind ghost “brain”.
Read: https://en.wikipedia.org/wiki/Behavior_tree
BT is a tree where:
branches are conditions
leaves are actions.
With BT we no longer operate with predefined “brain” states like we did with FSM. Instead BT “brain” will pick the best possible action to execute based on the hierarchy of conditions. This hierarchy of conditions are parent nodes of any particular action.
BT execution is started from the root node and descends down the tree. Depending on the type of the node - either (Selector) only one of its children or (Sequence) all children will be executed. Leaf nodes (Action) will make NPC act.
In the image below grey nodes are conditions that were false, therefore actions (“Seek” action in this case) will not be executed. Green nodes are under conditions that are true during this frame, therefore actions below them will be executed. NPC is going to “Flee”.
This example will execute sequence of actions starting with selecting random point to travel and then wait until the movement succeeds.
BT is a very common approach in games today.
Read: http://www.gameaipro.com/GameAIPro/GameAIPro_Chapter09_An_Introduction_to_Utility_Theory.pdf
Utility is a list of actions and associated utility scores. Utility scorer simply tells us how useful particular action is at the given moment. “Brain” will pick and execute the best possible action with highest utility score.
To simplify Utility AI is a more generic and flexible representation than BT. BT can be transformed into Utility AI representation.
This example shows how leafs of BT become list of actions in Utility AI representation…
… and conditions become utility scorers.
Scorers are implemented as small code snippets than evaluate state of our NPC and world around and return single value - score.
Read more: http://www.gamasutra.com/blogs/JakobRasmussen/20160427/271188/Are_Behavior_Trees_a_Thing_of_the_Past.php
Utility AI are gaining popularity in the game development.
Utility AI allows to create more modular and emergent behavior than FSM or BT.
For more detail comparison: http://intrinsicalgorithm.com/IAonAI/2012/11/ai-architectures-a-culinary-guide-gdmag-article/
All discussed approaches purely react to current situation. Can we make AI to anticipate future and plan for it? It is called “deliberate” approach.
More on planning: http://aigamedev.com/open/review/planning-in-games/
Influence Maps can be seen as a simple way to add some planning ahead into otherwise reactive AI.
Influence Maps modifies how Navigation is searching for the best path. The best path becomes not the shortest path, but path which satisfies specific NPC goal better.
Excerpt from the great presentation about Killzone AI: https://www.guerrilla-games.com/read/killzones-ai-dynamic-procedural-tactics
Killzone talk perfectly explains the concept - I decided to “borrow” their slides instead of making poor copy of this great example.
Make a plan of actions towards desired goal. Execute and re-plan when necessary. Idea should remind you pathfinding, except looking for path in action space instead of geometrical space.
Read more: http://alumni.media.mit.edu/~jorkin/goap.html
Approach that GOAP (Goal Oriented Action Planning) is based upon is called STRIPS in scientific literature: https://en.wikipedia.org/wiki/STRIPS. Slightly more advanced version of STRIPS is called SHOP: https://www.cs.umd.edu/projects/shop/description.html
It is not easy however to evaluate how particular action will affect the state of the world - search space can “explode” (see early slide on Chess/Go/Starcraft).
This is somewhat less explored area in games - we need to do more work here.
So far we have looked at “engineer” approaches to AI - rational, step-by-step, hardcoded. They do not learn anything or even try to analyze opponents play style.
Unsurprisingly that is NOT how our brain works :) Our brains can learn, adapt to situations and find new ingenious solutions to problems. Can we learn from brain to build better machines and game AI?
Read more: https://en.wikipedia.org/wiki/Cerebral_cortex
Read more: https://en.wikipedia.org/wiki/Neuron
Magnified and colored slice of human brain - ~2 mm deep slice.
Fat dots are “bodies” of neuron cells, lines are connections between neurons.
Computer generated reconstruction of connections between neurons in cerebral cortex.
Why study neuroscience? It will teach you about different type of computations - highly parallel, highly scalable with emphasize on learning instead of hardcoding.
Meanwhile modern day sequential code that executes just on handful of cores is going to be less and less relevant. In the future hardcoding or as we call it today “programming” for chips based on classical Von Neumann architectures will become as prized and eventful as soldiering radio sets today!
Lets look at development of human built systems that take some inspiration from brains.
Photo - Facebook GPU based computer purposed for Machine Learning tasks. More: https://code.facebook.com/posts/1687861518126048/facebook-to-open-source-ai-hardware-design/
Google took GPU design and removed “parts” unnecessary for Machine Learning.
More: https://en.wikipedia.org/wiki/Tensor_processing_unit
FPGA - not new hardware design, but recently gaining more interest from Machine Learning. Hard to program though.
More: https://en.wikipedia.org/wiki/Field-programmable_gate_array
Currently implemented on top of FPGA. One step towards turning FPGA type of design into Machine Learning friendly.
More: http://www.clement.farabet.net/research.html#neuflow
Neuromorphic chips build neurons directly in hardware. Building neurons in hardware is not hard, however interconnecting lots of them is the main challenge. Sometimes called - “network on chip”.
Read more: https://en.wikipedia.org/wiki/TrueNorth
Neuromorphic approach in general: https://en.wikipedia.org/wiki/Neuromorphic_engineering
Single chip has 4096 cores. Each core simulates one neuron, stores information about its connection and has network router to communicate with other neurons.
TEDxCERN talk by Madeline Lancaster: https://www.youtube.com/watch?v=EjiWRINEatQ
Cerebral Organoid is a tiny brain grown in the lab. Neurons in cerebral organoids seems to be functional, however connection and activity seems to be chaotic comparing with “real” brains. Some promising breakthroughs are happening in the last 2 years in this area!
So far cerebral organoids are not fully developed brains comparing to alive organisms. Nor they are connected to any body - do not get any meaningful information and do not do any meaningful computations.
I would guess that in the future (10 years?) we will learn how to augment cerebral organoids with computers - closing feedback loop and training them. Biological AI co-processor, if you wish.
Most likely in the process we will discover more about how biological brain learn. And improve our Machine Learning approaches.
2016… but really shift was happening in the last 4-5 years.