Back to Apping: Creepy Hands, Part 2

So, my task was to code a hand: one that could point and tap the index finger.

I suspect that, when large companies are given this task, they just send it over to the 3D graphics departments, whose cheerful minions happily generate a fifty thousand triangle mesh, manipulate it in cutting-edge software, then spit out some beautifully-rendered video files. Which is all well and good if you have that option.

But video files won’t cut it for me. If I’m future-proofing (like I always aim to do), then the number of video files needed would be ridiculous. The digit combinations alone must be 120 or so – and that’s just for opening and closing; still not enough to greet Mr Spock.

So: a 3D mesh. Is it doable? Well, yes, but 3D object construction is not exactly my forte, and it does seem a bit like overkill to add all those triangles for one little hand.

Hello, Hack-imation

Time is always an issue so I decided to do what I do best when faced with thorny coding problems: I cheat. Realizing that I only needed to see the back of the hand, I decided to go 2D. Yes, some of the movements are 3D, but smart animation can compensate for that.

First step: trace my hand on squared paper.

I traced the open hand, and also a a fist, and then tried to work out where the joints were. I like coding in circles, so each point of interest is a circle. I chose a scale such that I could type in ratios. The y-axis goes to 100, which is obviously 1.0f as a ratio, and makes scaling that much easier.

Yes, this will be a glove, which means I don’t have to worry about drawing fingernails!

If I had had the time, I might have chosen to render the hand in Metal. But I picked plain old Quartz for convenience and expediency.

I originally wanted to do the entire outline as one shape, but quickly decided against it, due to fingers and thumb sometimes hiding behind other digits (and the palm). So I draw the thumb, then little, ring, index and middle fingers in that sequence, because that’s usually the z-order.

The main part of the hand has to be drawn twice. Once at the back, and once more without borders, to erase the lines at the base of all the fingers. This is the “hack” part of the hack-imation, along with the right-hand-side, which I decided was most easily simulated with a Bézier curve.

How to draw a finger? Well, if you can work out where the outer tangent lines meet for two circles, that’s most of the job done. After that it’s just straight lines and arcs.

I treated the thumb like a finger, with the same number of joints. Yes, the base joint is just above the wrist!

One funny thing: if (as my picture shows above) you have the wrist curve dipping down, then your hand will quickly resemble a foot with spidery finger-toes! I strongly advise to flip that smile upside-down to avoid that!

Animating in Code

The maths required for animating a finger or thumb is refreshingly simple. Each joint moves from angle A (zero) to angle B in a linear fashion. You can eyeball the end angles. Just a few cos and sin functions required.

There are one or two sticking points, for example when a fingertip passes below a knuckle. That case is simple enough to simulate from a top-down perspective, however: simply, by drawing to the knuckle and no further.

There are always the odd graphical glitches that crop up, but these can be dealt with – either cleanly or with special-case hacks.

Using as a Cursor

When the hand is finished, the next step is to use it with the cursor. This is easy enough to do with your input code in OSX. The trouble (as usual) lies with iOS: there’s still no way of working out where your finger-tip is until it presses the screen.

What you can do is show the cursor when you tap the screen. You can add a small delay to have the hand fade in and then, when you tap-up, the hand can glide serenely away, fading out as it does so. But this is not ideal. I needed the hand to know where I was going to press for the product video.

I had a brainwave to connect a mouse to my iPhone and have a cursor show up at all times. Well, this works nicely on Android but not on iOS. On iOS, you can’t even connect an Apple mouse or trackpad to an iPhone! You can connect a third-party mouse, but nothing further: there’s no cursor.

Yes, I could have simply cheated and used OSX for all my product videos. But I didn’t want to do that. So I had to think again.

Automating

I thought, what if the hand did know where I was going to press? Rather: the hand could be coded to move, click, rotate, fade in and out, and so on. What I needed was a class to automate this.

Then it occurred to me: this wouldn’t just be useful for product videos, but for inline help.

The automation class here goes hand-in-hand with the calling class. At each indexed stage, the calling class should update such details as duration-of-action, hand-state, action-type, alpha, angle etc, and, as necessary, supply destination co-ordinates relative to a view it also supplies.

Finally you get something like this. Still not perfect, but it’ll do. 🙂

Back to Apping: Creepy Hands, Part 1

Once your masterpiece app is finished, there remain some irritating stumbling blocks on your way to the App Store. One of these is the product video.

A screenshot may tell a thousand words, but a good video can replace a fair few screenshots, and won’t require your potential beloved customers to scroll. Did I mention I hate scrolling?

You might suppose that the only challenge in the creation of a product video is to find software to record your screen. Unfortunately, this is the easy part. The difficult part is finding software that will also record your hand movements while your finger is still hovering above the screen.

Solution 1: use another device to take a video of yourself actually using the app!

This is a messy solution at best. Good luck to you if you want to do this. If you don’t have creepy hands, if you can eliminate reflections, sort out the lighting and background, and if you have good video after-effects software then this might work well enough.

Solution 2: Add a virtual finger (or hand) afterwards with after-effects software!

This may be a bit easier than Solution 1, and it should look neater. I almost researched this idea but decided that, if I were to make dozens of videos, it would take too long. In addition to the potentially prohibitive cost of after-effects software.

Solution 3: Draw the actual finger (or hand) in-app!

This was my favoured solution, because it’s more of a coding challenge. It does not require learning any other skills, or purchasing any other fancy software.

I tried to remember how I solved this problem before, years ago, for my previous apps (which are now lost in time). I had a vague memory of taking a photo of my hand and employing that as a makeshift cursor. Then using the simulator to record everything.

Why didn’t I take a look at my old code? Well, because coders like me really don’t like doing that. Nobody should want to peruse the horrible code of our less-experienced selves. We much prefer writing shiny new code.

But anyway, the first step was to photograph my hand. I wanted two top-down shots of a pointing index finger, with one of them simulating the tap motion.

You won’t believe how wretchedly difficult this is to do. Because, the thing is, I needed to have a background which I could magic away quickly in software. Yet no matter what colour card I had underneath my hand, masking this out in one go proved extremely difficult, due to shadows and skin reflections and weird optical illusions like this one.

In the end, I thought back twenty-five years. My preference back then was never to use photo-manipulation software. It was to use vector software and manually mask out the image, using straight lines and Bézier curves. This is what I did (thanks, Inkscape).

So, coding a hand for a cursor is trivial. It’s not a proper cursor, of course, rather it’s an image that has its origin at the pad of the index finger and follows the mouse around. But here was my new problem:

My hand looked extremely creepy as a cursor.

Yeesh. I’ve spared you the two other colours I used as backgrounds. Notice the shadows above the knuckles? That’s where bitmap software fails. Vector software meanwhile loses hairs, but that’s the least of things, as far as creepiness is concerned.

I had to ask myself: was my hand always this creepy? I know I’m no spring chicken but that thing was like something out of Tales of the Crypt. Does my hand really look like that in real life?!

Well, it wouldn’t do at all, and I didn’t want to go through that whole process again with someone else’s hand. What if their hand turned out to be just as creepy as mine? On the other hand (ahem), what if it looked great and I ended up with a complex about my own general hideousness.

So, my next thought was to draw a cartoon hand. Well, actually, not a hand: a glove. If it’s good enough for Bugs Bunny then it’s good enough for me. Now, I could draw this with pencil on paper and then scan it in and that would surely have been the best solution.

But then (unfortunately) I had another idea, and it was this: I should really draw this hand in code…

The Problem With Buttons

I have a problem with buttons (small b, it’s not about the pantomime character, so sorry if you thought it was). To “click” (or, as the youngsters among us say, “tap”) on a standard button, you have to be accurate. If three buttons are side-by-side and you want the middle one, and you hit one of the buttons on the side, you’ve failed. Go straight to Jail. Or, rather: you now have to slide your finger away and try again.

Using a mouse, this isn’t a problem. Mice are accurate beasties. With a touch-screen, though, especially when your screen is small, it becomes an issue.

The standard solution for an iPhone? Make your buttons much bigger.

Needless to say, I hate this solution:

  • the app now looks less like Technic Lego, more like Duplo;
  • it takes valuable screen-space away from other things.

So, how do I deal with it?

I allow the user to slide from one button to the next. If buttons are adjoining, that is then one control, not many. In the input_down code, work out which button you’ve hit. In the dragging code, calculate again. The input_up code is what matters for the “click”.

Additionally, if there’s a natural margin around the buttons, register a hit on this margin as a hit on the button (the margin also becomes a part of your control).

If you want to keep your buttons small, there’s one more problem to deal with: the user’s fingers. How do they know which button they’re over if their whopping-great sausage finger is obscuring their view of the screen? Well, standard keyboard displays do this with a pop-up key display. This works fine for a keyboard but won’t be much good it your button is at the top of the screen.

Once upon a time, a status bar at the bottom of the screen would be the solution. 9 times out of 10 this is all that’s needed. But, um, what if the keyboard is being displayed?

Well, in addition to a status bar (which I still think is very useful) alongside a replacement (or reframed standard) keyboard, the solution I favour is to use the middle of the screen. OSX often uses what I imaginatively call a “grey square control”, which is semi-transparent, non-interactive and fades away by itself before it wears out its welcome.

Is there a problem with using the middle of the screen? What if I were to have a button there? Well, no, that’s actually a silly place to put a cluster of small buttons!

So, to sum up: for frustration-free small buttons, you might feel you need to:

  1. code a multi-button control & handle the input and drawing yourself (the easy part);
  2. code a status bar;
  3. deal with the on-screen keyboard (or code your own) to accommodate said status bar;
  4. code a grey square control (because, if nothing else, they look cool).

Alternatively:

  1. be lazy, save yourself a fair bit of time, and use big buttons.

I don’t blame you if you opt for big buttons. Because it’s not just buttons, it’s lots of controls. They’re simply not designed to work well with one another when they’re in proximity. As soon as there are multiple views involved, the standard input process fails. To fix this one problem, you’ll soon find yourself fixing similarly-irritating other ones. It’s less of a rabbit-hole, more like those caves in The Descent. And that’s where I’ve been, thankfully with the American ending and not the UK one. 😉

Keyboard Worrier

When the iPhone first arrived, I didn’t get one. I wanted an iPod Touch instead, which was great for music, not so good for calls.

As amazing as it was at the time, the iPhone software was missing something, and it’s something that’s been bugging me ever since. Here’s the on-screen keyboard, as it appears now on my iPhone:

And here’s the landscape variant:

What’s wrong? Well, mainly the absence of beloved keys. Where’s my tab key? Where’s escape? Forward delete? The cursor keys?! Dare I even ask about Command, Option and Control?

Why cannot such buttons at least be put on the landscape mode view, instead of elongating the portrait layout like silly putty?

There is no getting around it: I often feel hamstrung having to use this simple keyboard.

Yes, I do understand the thinking that such keys would just clutter and confuse and that, with a touch-screen, they might also be superfluous.

Well, boo to that with knobs on!

  • Placing the typing caret down between the two letters I want is an exercise in frustration;
  • Tabbing now requires tapping – plus sometimes scrolling – different areas of the screen instead of one key;
  • As for dumbing down for the masses, why not supply two modes with a keyboard, the same as for a calculator: basic and advanced?

For the app developer wanting to reimplement what was lost, the documentation suggests that such missing buttons should be placed (awkwardly) in a row above the other keys. It’s a bit of a hack in my humble opinion, and can look pretty nasty.

So, one of the things I wanted to work on for my software library was a decent keyboard control. If I needed to fit extra keys to my keyboard in portrait mode, I’d clearly need some sort of zoom (I also do for the iPhone keyboard, as I’m always hitting the wrong key at the edges).

Here’s my SuperKeyboard, as it appears on my iPhone:

(the bezel shape was added afterwards by me, since it doesn’t appear in the screenshot!)

Coding your own keyboard has its small challenges, including:

  • dealing with the peculiar way iOS handles rotation (i.e. when it’s rotated landscape but the frame still thinks it’s in portrait mode);
  • the carriage-return key, which has to be drawn separately due to its odd shape;
  • dealing with bezels and safe areas (I would love to be able to access the shape of the screen programmatically; instead I am provided with four margin values).

My layouts are all very simple text files, which are then read into the SuperKeyboard, with the code working out the relative sizes for the keys depending on their types. As such it is now relatively trivial to modify any layout, or create a new one as desired.

Oh, and one last thing I wanted: a smiley keyboard with a spacebar, arrow keys etc.

Shifting does this:

That’s one more control down. 😉

Back to Apping, Part III

Now, where was I? Ah, yes: the software library. What’s wrong with the tools we’re given? Well, nothing, really… Well, actually…

I love Apple’s controls. In no time you can cobble something together that looks pretty cool… but, then, so can the next programmer. Worse, the next programmer probably has much more money and resources than you do. And even if they don’t, then there are a thousand more that do.

I need my work to stand out.

If I can create something cooler than the norm, then great. For instance, here’s an Apple slider.

Simple, elegant, does what it says on the tin. But here are my problems with it:

  1. My app might require sub-ticks as well as ticks;
  2. I cannot control a range of values; I’d need multiple sliders… = more scrolling;
  3. The blue is a system-set colour – and would probably look better as a horizontal gradient;
  4. If I have multiple sliders, and one is more important than the others, how can I make it stand out?

In summary, the standard control is too simple, and does not give me enough fine control. Which is absolutely fine for 95% of uses.

Now, here’s my control, in a basic configuration:

Apple’s bar has a subtle 3D-gradient fill; mine has a less subtle horizontal gradient. My drop shadow is also a little different.

Here’s my control again, with some more options added:

Sliders can also be vertical; I’m too lazy to screenshot that but tilt your head to one side and you’ll get the idea!

The coding of such a control (I imaginatively call it a PBGUI_Super_Slider!) does not take long. Maybe a week or two with my time limitations. But I felt I needed to do it to minimize scrolling (did I mention I hate scrolling?) while hopefully not making everything look too cluttered.

One thing that annoys me about Slider controls in general (I don’t think it’s just Apple who are guilty of doing this) is the property numberOfTickMarks. For a scale of 0 to 10 I have to think a beat about whether this should be set to 10 or 11. If I then change the min/max value I also have to change the number of tick marks again. I appreciate that this might seem like a trivial issue but when my tiny brain’s capacity is full, trivial problems lead to easy errors. So my “Super” Slider has an alternative property for setting the value interval. In the above example, I can simply set it to 1 and forget about it.

My Super Slider is not perfect. If I had more time I’d add an extra 3D gradient and some animation. If I had coded it this year not last I’d be using Metal for that little extra sparkle. But, for now, it’ll do for my purposes.

Anyway, that’s one control out of many. I might talk about some of the others next…

The Case of the Useless Valve Adaptor

Warning: the following article is even more boring than this picture

I’ve been rewatching Babylon 5 after 25+ years with my kids. I’m chuffed that they’re enjoying it, that they love G’Kar and also find Marcus intensely irritating. We’re in mid-season 3 where things are getting really interesting… But then arrives an episode like “A Late Delivery From Avalon“, which is needless filler and annoys us for ruining the flow. This post will be a bit like that, so feel free to skip ahead to Back to Apping, Part III, if you can. If you can’t, unfortunately it would appear that you live too much in the present, sorry!

You might recall that, in my previous post, I talked about things that needlessly suck away my time. A good example arrived promptly after I’d written that, and I thought I’d share.

I had to collect my son from school, so I wheeled out my bike and its trailer and, because I was a few minutes early, I foolishly decided to pump up one of the trailer’s tyres before I left. It didn’t really need doing.

(Now, my daughter had “borrowed” my bike pump in the summer, in an attempt to inflate a paddling pool that was never used, because the instant the pool was inflated, it rained for a week and then stayed relatively cold. Somehow she had managed to break one part of the pump, the connector for the very tyre valve that I now needed.)

Instead of adding a bit more pressure to the tyre, all I succeeded in doing was to expel the remaining air from the inner-tube. That’s why my son had to walk home with me.

The pump has two connectors for it, you see, the normal “Presta” one, which every other bike pump in the world has, and one for a “Schrader” valve. It’s the Schrader part I needed, and (of course) that was the one which was broken. I found three other old bike pumps in the shed, and none of them had this connector.

So I spent a while trying (and failing) to fix the broken connector. It wouldn’t lock. I suspect a vital piece was missing, a treat for future archaeologists to gush over when they dig up my lawn in they year 5000.

A normal person wouldn’t have bothered any more with this. They’d buy a new pump. If they were really flash they’d buy a new wheel. And, if they were filthy rich, I don’t think they’d have a bike trailer in the first place; they’d have a solar-powered helicopter or a Hyperloop direct to the school.

But, as others will gladly and maliciously attest, I’m not normal. When faced with a problem, I can’t rest until I’ve solved it – as cheaply as I possibly can. It occurred to me that I could buy an adaptor to turn the Schrader connection to a normal connection, and then use the working connector of my pump with it.

So, off I cycled to my local Hagebaumarkt (which is German for B&Q), and bought an overpriced small bag labelled “bike pump adaptor”, containing three different adaptors, one of which I knew should do the trick. And I cycled back, quietly confident it would work.

It didn’t.

Inflating a Schrader valve is as simple as it can be. There’s no chance to do anything wrong here.

bicycle-guider.com

The adaptor screwed on perfectly, the seal was good to my pump. But I couldn’t pump air into the inner-tube. Because, to do this with a Schrader valve, the inner-valve pin thingummy needs to be pushed down in order to let the air in, and the adaptor did nothing to push the inner pin. It merely entombed it.

No, the inner valve would not screw outwards. The adaptor was useless. Like a travel power adaptor with the right holes for your appliances’ plugs but none of the internal copper to conduct electricity from one end to the other.

But I wasn’t finished just yet. If the adaptor wouldn’t push down the inner valve pin, I’d need something else to do it for me. So I rummaged around in the dirt of a nearby flowerbed and found a tiny pebble, and I placed that in the adaptor and screwed it in. The pebble wasn’t so perfectly formed that it would block the air, but it did push down the pin like a charm, and I was finally able to pump up the tyre.

Life’s like that, sometimes. You try to fix something that’s not really broken, and you break it. And then you waste time getting back to Square One. Like your day, for example. I bet it was going just great until you wasted your time here. You didn’t need to read this twaddle. But now you did and you’re going to have to work hard to find a more interesting article to erase this experience from your head… Good luck! 😀

Back to Apping, Part II

When people find out (I try not to mention) that I’m a programmer, invariably they chirp in with, “Oh! I’ve got a great idea for an app! It’s a doozy!” And (unprompted) they go on to tell me how, for example,

  • it calculates sums of numbers (App already included with every phone), or
  • it makes amusing noises (Apps done to death), or
  • it uses the FBI facial-recognition database to spot disguised celebrities and/or escaped convicts in your local Lidl (App requires discreetly accessing the FBI facial-recognition database), or
  • it’s like Minecraft/Fortnite/GTA X but with way better graphics/gameplay.

And then they suggest I code that for them. “We can be partners!”

Well, “app ideas” was never the problem. I already have a list as long as my arm for app ideas. The problem is always Time. And lack thereof. Yesterday, I read through Apple’s app review guidelines, which is enough to make anyone’s eyes bleed. And that’s just reading it.

I need to introduce you to one of my laws. I call it “Philip’s First Law of Coding” because I’m Mr Vain. Here it is:

Coding will always take at least three times as long to do as you say it will, even (or especially) when you try to take into account Philip’s First Law of Coding.

Philip’s First Law of Coding, (c) PBG circa 2015

Star Trek’s Scotty knew just how long it would take to repair something, multiplied that time by a factor of four in his estimations and then everyone was mightily impressed when he managed it in a quarter of the time he said it would. Scotty’s a git! If I think something will take a month, I might decide to say that it’ll take three, and then it’ll actually take nine. Yes, it can be extraordinarily tricky to guesstimate how long even a simple project can take, especially when I am confined to two-to-three hours a day to work. There are all sorts of things which crop up:

  • unexpected and mysterious memory bugs that the debugger can’t handle;
  • new OSX/iOS/XCode (old code suddenly stops working, or works differently);
  • new code A requires recoding old code B, and then B affects C, D & E, which need adapting and testing, and then D dominoes into the rest of the Roman alphabet and threatens the Cyrillic;
  • hardware problems (failing hard disks is #1);
  • problems due to stupid initial decisions (mostly from severely underestimating complexities);
  • real-life emergencies, illnesses of all flavours, mood-swings and depressions, weather too hot or too cold, rain-patter too loud against the ceiling window, and so on.

This isn’t just a problem for coders. Please do take a moment to read up on the history of the upcoming Game of Thrones book, The Winds of Winter. In case you didn‘t just do that, here’s that link again. I did say please! I am pretty sure that Mr Martin is not a slacker and that he was being entirely honest with all of his completion estimations. Finishing even a simple project is tricky and takes times to do right. I know, because I’ve been spending many, many years writing my own software library, in order to code an app that will change the world.

More about that in Part III. 🙂

Back to Apping (Was Napping), Part I

Portrait of a Coding Dinosaur

I’m a coding dinosaur. I grew up with writing databases in BASIC on the ZX Spectrum 48K. I’m so old, computing was an optional afternoon hobby at my secondary school. There were no laptops there, just a dozen big beige boxes in one room hidden away from all the others. Cutting edge tech was a VGA monitor and a greyscale scanner you had to roll over a picture at just the right snail speed for it to work. My mates & I were wowed by Windows 3.0 and spent our days wrangling with autoexec.bat and config.sys in order to free enough DOS memory to play Origin games. (No, new kids, not the same Origin you know).

So I stumbled into software engineering after university, doing my best to muck up their statistics by not landing my first job till a year after my 1st-class-honours Maths degree. I went to work in an international bank making graph software for Value Added Risk (which is as much as you would probably ever want to know about that).

Then, at the turn of the new millennium, I tried to ride the wave of the Dot Com revolution. I journeyed to America, helped make a website for buying/selling sports cards. It was called thePit and, amazingly enough, I see that it’s still going. But I learnt then that you can’t ride a wave that’s actually a bubble that had already burst.

After that I learnt C++ in order to code my first computer game. It was a vector-based space shooter called Surrounded! and it was awesome, but I can’t market and can’t afford marketing and consequently nobody ever really played it. Except me and a marvellous chap called Saturn.

What was the next “Big Thing”? Well, it was the iPhone so that seemed like a good opportunity for me to find a bigger audience with its built-in App Store. I enjoyed extremely modest success there. Looking back I joined a bit too late, the same as with the Dot Com thing. Matters were further muddied by my move to Germany and having to juggle my new family life, an impossible new language, six years of lost sleep and an idiopathic chronic illness. I can cobble together more silly excuses but the truth of it probably was that my apps simply weren’t good enough to pass serious muster. When the whole tax situation became too thorny (as my dev account fed my UK bank account for my UK company, but I was in Germany), I decided it was simpler to take my apps off the store.

(And I’m breaking up this story because you surely have better things to do than read articles which are too long. Another bite-sized piece of my story coming soon!)