Mocha

trackFinger recording touch on an iPad

Screen Replacement for iOS and Beyond

I missed this massive post on Pro Video Coalition earlier this year, but it talks about a ton of techniques for screen replacement, including an iOS app called trackFinger that you can use to put a green- or bluescreen on your iPad or iPhone, and also have it track any touches on the screen for import into After Effects. With this, you can animate based on the touch motion, and composite the video into the greenscreen on the video.

I’ve tried it myself, it doesn’t really lend itself to doing demos of real iOS apps — it’s a little difficult to match up where the real tap targets are when you can’t see them. However, the color keys out just fine for compositing animated and futuristic interfaces for your next sci-fi short. Just be sure to turn the screen brightness down to match the lighting in your scene, because that greenscreen is going to spill like crazy on your talent’s hand if the brightness is up too high.

The PVC article goes into more depth on screen replacement techniques, many of which I use, and embeds several how-to videos on the subject. If you want to play around with Trackfinger, there’s also now a free “lite” version of the iOS app.

Motion Tracking for Screen Replacement

is a great introduction to tracking screen replacements. Imagineer Systems makes Mocha, a planar tracker that will allow you to get excellent tracks of a flat (or flat-ish) surface, often without tracking markers. A lite version, Mocha AE comes bundled with After Effects.

Obviously, this isn’t a complete solution for iOS developers looking to demo their apps with live hand-held iPhone and iPad shots, but it’s an important piece of the puzzle. Imagineer has that shows more of what goes into a screen replacement on a touch screen. Even though it demonstrates the final compositing steps with an outdated version of Final Cut Pro, it gives you a good idea of the steps that go into this kind of shot.