Writing shaders with Metal for user interface

updated 10 months ago; latest suggestion 9 months ago

This proposal has been withdrawn...

Have you ever been interested in Metal but don’t know where to start or where to use in your application? Shader programming with Metal would be a good starting point for learning the framework and understating the basic concept of parallel computing. In this presentation, I would like to introduce shader programming with Metal and how fun and easy it is.

Shaders are used widely in computer graphics field and Metal enables them to be integrated easily to the UIKit based application. Shaders can enable animations which would be very complicated or difficult to draw with the existing animation frameworks such as UIView animations or CALayer animation, and, I believe, it brings more visually appealing user interface design.

In this presentation, I will cover the following points:

Brief introduction to Metal

Probably one of the difficulties of learning Metal is the number of APIs to use. In my experience, writing shaders on Playground, reflecting the change and playing around with MTKView in realtime was a very fast way to understanding how the framework works. In this section, I will explain the basics of Metal and the shortest path to play with it.

  • What is Metal, and advantages over OpenGL
  • Requirements for Metal
  • Main APIs of Metal
  • Basic idea of Shader
  • Metal on Playground

Drawing UI parts with fragment shader

Even the way shaders draw lines or curves is completely different from the one of UIKit. You probably would use UIBezierPath or other vector based APIs for UIKit. Shaders define functions for each paths and decide the color of every pixels. I will show examples that combines those functions and draw complicated figures such as downloading indicator with an animating texture or pop ups with wavy outlines like below:

downloading indicator pop up

Fluid simulation with Metal

The examples above use only the graphics pipeline shaders such as vertex shader or fragment shader. Basically, they use only one MTLTexture and animate the figures changing the color of pixels with the time input.

In this second example, I will show a fluid animation modeled with Navier-Stokes equations as below and explain the concept of GPGPU. The image is converted from UIImage bitmap to MTLTexture and animated disappearing like smoke. The fluid simulation model requires several MTLTextures that might not be directly rendered on the screen but be used to preserve scalar or vector fields such as advection or pressure and the fluid model computes the next state based on the previous state using kernel functions. I will show one of the advantages of Metal, the integrated API for graphics rendering and data-parallel computation through this example.

fluid simulation

Suggestions

  • The proposal author responds 9 months ago

    Thanks for your comment! I am inspired this idea from some shader-ish 3D-ish userinerfade of Apple’s official applications such as Digital touch, Siri’s waveform animation or Animoji. Although there are some difficulties to use the framework in a production app (I would love to mention it in the presentation), I think it definitely can give some hints for more advanced UI/UX and broaden design options for designers.

  • 8b2f2d3e6553c2b4048b93300959895ab76b4154?size=100x100 8b2f2d3e6553c2b4048b93300959895ab76b4154 suggests 10 months ago

    OK. This is seriously cool. I don't have any idea how I would use this in my app, but I could see finding a place for it where I could never justify using Metal to build a game.