The document provides an overview of iOS visual effects capabilities and limitations. It discusses Cocoa Touch's reliance on CALayers for 3D capabilities despite UIViews living in 2D space. It introduces OpenGL and the power of shaders for advanced visual effects. Code examples are provided of various GLSL shaders for effects like color remapping, hue shifting, and edge detection. The document emphasizes the parallel processing capabilities of shaders for complex effects.
25. The Shader Backstory
• Pixar creates Reyes (Render Everything You Ever Saw)
• Pixar creates RenderMan: C-like language for describing a
3D look at the sample (pixel) level.
• Boom! Everything changes.
• GLSL created in the spirit of RenderMan.
26. Shaders allow us to ignore the rest of a
software system, freeing us to focus on
achieving a specific look.
44. The GPU is a parallel processing beast. It uses a SIMD
architecture (Single Instruction Multiple Data) to achieve
massive processing power.
A GLSL shader is a SIMD program. The GPU takes a shader
and evaluates it simultaneously - but with different data - at
every sample (pixel) in parallel:
On iPad that is:
2048 x 1536 * 30 fps = 94,371,840 shader evaluations per sec.
45. What is exciting about this level of performance on an iOS
device is the ability to take tasks previously thought of as
desktop tasks done with Photoshop, Final Cut, etc. and do
them live in a handheld device equipped with camera, mic,
and other sensors.
47. Shader Idioms
To fully “get” the power of shaders and their style of use it helps
know the idioms guiding their use:
• Multiple Passes
• Iteration aka Ping/Pong
• Buffers/Channels
• Indirection/Remapping
48. Shader Idioms
These idioms derive directly from Hollywood film production
workflows and practices that enable:
• complex problem decomposition. Tractability.
• rapid turnaround.
• maximum flexibility and tweek-ability.
• responsiveness to director’s whims
50. ElasticImage highlights.
• Rapid shader creation & deployment
• Shaders and gesture declared in plist
• Cocoa Touch for gestures & the usual.
• C++ for 3D glue code.
55. I selected interesting palettes and created a texture.
Horizontal axis is the color palette.
Vertical axis selects between color palettes.
At runtime this palette texture is loaded and attached to
the ColourLoverShader.
56. EISColourLoversShader.fsh
varying!
mediump vec2 v_st;
// Palette selector
uniform float paletteDial;
// Contribution of remapped color
uniform float strengthDial;
// Palettes are ganged together into a single texture.
// A specific palette is selected with the paletteDial
uniform sampler2D colourLoversPalettes;
// hero
uniform int heroChannels;
uniform sampler2D hero;
void main() {
!
! vec4 raw = (heroChannels == 1) ? vec4(texture2D(hero, v_st).a) : texture2D(hero, v_st);
!
! vec3 cooked;
! cooked.r = texture2D(colourLoversPalettes, vec2(raw.r, paletteDial)).r;
! cooked.g = texture2D(colourLoversPalettes, vec2(raw.g, paletteDial)).g;
! cooked.b = texture2D(colourLoversPalettes, vec2(raw.b, paletteDial)).b;
!
! gl_FragColor = vec4(mix(raw.rgb, cooked, strengthDial), raw.a);
}
57. EISColourLoversShader.fsh
varying!
mediump vec2 v_st;
// Palette selector
uniform float paletteDial;
// Contribution of remapped color
uniform float strengthDial;
// Palettes are ganged together into a single texture.
// A specific palette is selected with the paletteDial
uniform sampler2D colourLoversPalettes;
// hero
uniform int heroChannels;
uniform sampler2D hero;
void main() {
!
! vec4 raw = (heroChannels == 1) ? vec4(texture2D(hero, v_st).a) : texture2D(hero, v_st);
!
! vec3 cooked;
! cooked.r = texture2D(colourLoversPalettes, vec2(raw.r, paletteDial)).r;
! cooked.g = texture2D(colourLoversPalettes, vec2(raw.g, paletteDial)).g;
! cooked.b = texture2D(colourLoversPalettes, vec2(raw.b, paletteDial)).b;
!
! gl_FragColor = vec4(mix(raw.rgb, cooked, strengthDial), raw.a);
}