Coasters, Consumers and A/B Testing

So Space Miner is going on a coaster this weekend. So what's a coaster?

Basically, a coaster is a term I'm coining to describe bargain pricing your app to take a ride up the charts. If things go well, then a day by day plot should show your app climbing up the charts very quickly (like the ascent on a roller coaster). Eventually it will peak, and hopefully maintain it's position for a few days before starting to fall again. Once the fall starts, the usual response is to return to normal pricing, which accelerates the decline, at least on the top download charts.

A number of companies do this to stimulate sales. You'll see Gameloft do this for brief stretches, and a recent example was the first week of April sale where EA discounted all their games (not all to $.99, but it is EA, not Joe Schmoe after all). However one game, namely Simpson's Arcade, went all the way to the top of the chart. At the end of the week, guess what? They left it there.

Why do this? Well there are a number of reasons, but the basic one is obvious - if it goes right, you'll make more money, even though your margin is reduced. Sales on the chart are in no way linear, and the top games sell A LOT more than games not in the top chart. I believe Lima Sky announced something like 4 million sales recently, so that tells you something.

However it may not go right, and there are consequences.

The obvious one is that your sales don't increase enough to make up for the difference in price. In Space Miner's case, we are going to 1/5th our normal price, and will need an increase of at least 5x to break even. And really something more like 8x-10x to make it seem worthwhile. Short of that means we're probably wasting our time.

Another consequence is eroding consumer confidence in the price of your games, and pissing people off. For whatever reason, $1 on the App Store is like $10 in any other marketplace, and you'll get more grief for that $1 than you can imagine. I do believe it's the vocal minority though - so time will tell how much stock to place in those complaints.

And finally, once you place your app at the hands of the $1 masses, expect to get a lot of customers who are not necessarily interested in your product for the right reasons. They do less research, spend less time on the App, and are more likely to hit you with the 1-star Rate on Delete. In general, the lower your price point, the lower your average ratings. There are certainly exceptions, but it follows normal consumer patterns. A great book that covers this type of behavior is Predictably Irrational by Dan Arielly, and it's a good read if you're into that stuff.

So why are we doing this with Space Miner? The official reason right now is marketing - we want more customers because we're launching a new game soon. With visibility being such a huge factor in App Store success, you could argue very easily that for a given daily profit, it's better to pick the price that gives you more customers, since if they like your App, they'll be likely to look at your other products favorably. We've also added a More Games type section to allow easier discovery from within our games. So hopefully we can get some sales synergies happening.

The second reason is probably, in reality, the real reason. And that is that I'm just damn curious as to what's going to happen. It is a compelling sale, since the game is normally $4.99, so if nothing else people should be interested in that. But do we have enough visibility that there will be 5x the number of purchases? I really have no idea. And what will be our peak? Can we actually break the top 25 with a game like this? I've certainly seen much worse games grace the top of the charts. We did a press release to lay the groundwork, but until we see what happens this weekend it's all speculation. It's pretty exciting to watch though, so it should be a fun weekend.

Our plan is to run it for three days, but like EA, I'll reserve the right to change my mind if we make a lot more money this way :)

Following up on last week, we did run an experiment with AdMob doing some A/B icon testing. We tested about six different icons, and the results were, unfortunately, not beneficial. Let me explain.

We had some technical difficulties just getting the test running. It was pretty difficult to get an even number of ads served in a single campaign, so I had to do a lot of manual turning on and off of different ads to get a high enough volume that each one had statistically significant results. But because of the different times of day ads were running, and since some ads just got served a lot more than others, it was hard to tell what was causing the deviations in ad performance outside of just the icons.

Be that as it may, at the end of the experiment the best performing icon was - the icon we're using. Go us! So when I say not beneficial, I mostly mean that there is no clear "better icon", at least superficially, for us to be using and hence no panacea to App Store success.

The sale is on, so it's time to sit back on the couch with the iPad and keep hitting refresh on the hourly rankings. I don't know if this will be profitable or not, but we'll have a bunch of new customers at least, and hopefully fans. Next week I'll update with some statistics from the sale, and any lessons learned.

Three Months on the App Store

We launched Space Miner on February 5th, with no clear idea of what was going to happen. We'd been developing games for almost eight years as a company, but this is really the first time we exposed ourselves directly to our consumers besides a couple of minor Windows Mobile projects. Being primarily a third party developer for companies like EA and Sega meant that we never had to deal with getting a product launched. To say that we were unprepared would be a woeful understatement.

This was not due to ignorance. We knew about the importance of marketing, buzz, etc. but at the end of the day, we didn't have time. The game was way over budget and way late, and we just needed to get the thing done. Furthermore we also questioned how much value there was in doing this for a couple of different reasons. One was that we didn't have a clear message for what the game was - marketing the next great Asteroids RPG didn't seem like it would be a compelling message. The other was that it seemed like the App Store audience was completely fickle, and quite dispersed, meaning there was no clear avenue to really spread the word short of spending a lot of money. In the end, these may have just been justifications to make myself feel better at night because we hadn't done these things I knew we should do.

We did what we could though. We tried emailing some journalists, and ran a short promotion on TouchArcade the week before, but I think at launch maybe 100 people in the world knew that there was a game called Space Miner. Including us and our families. Not the best job there, by any means.

What happened afterwards is pure luck. Our little promotion on TA meant we had some people waiting for the game, and as soon as it launched, they got it, and loved it. They literally evangelized the product so much that our thread exploded and the editors of Touch Arcade got the game and ran a glowing review two days later. It was also a slow weekend so it was the top article on their site for nearly 48 hours. Our sales shot up, the buzz continued, and great reviews kept rolling in. A couple of weeks later we got featured by Apple, cut the price, and got into the high 20s for overall App Sales.

Saying this is pure luck isn't really true, since we did do one very important thing right - we made a good game. For the most part, hype can give you a bump, but if you don't deliver the goods, it isn't going to do too well.

But we did kind of hit this frustration point - we couldn't get onto the top of the App Store. I was seeing all these games ranked higher than us that weren't reviewed nearly as well, by the press or their users, and some of them, frankly, sucked. Perhaps if we cut the price to $.99 at that point we may have been able to breakthrough, but I felt it was a disservice to take a game at this production level and price it like Doodle Jump. So we stuck to our guns, and began our gradual descent back down the charts, finally going back to our $4.99 price point.

Thus I learned my first lesson - chart ranks don't really help sales, unless you are near the top. For the most part, our ranking always reflected how well we were doing, and never served to increase our sales. When we shot up, it was because of being featured by Apple, or it was because we cut our price, and it happened within a day of the event. But then it would always hold steady or taper. We never experienced a rise, followed by a rise, which would indicate that getting up the chart was helping us sell more product. Perhaps this visibility would have been helpful if we'd had a Lite version available at the time, but we messed that one up and didn't get it out until we were off of Apple's featured lists. So we'll never know really.

Things have been tapering slowly since then, not doing horribly by any stretch, but it's a tenth or less of what it was at its peak, and seems to keep tapering. And we're a long way from break-even.

So we're now in a new territory for us, yet familiar to many hardened App Store veterans, where we try to figure out ways to extend the life of Space Miner. We've done a couple of things so far, including a couple of feature updates and getting out a Lite version. These have given us slight bumps, but haven't made a material difference in our sales. So we need to look at other methods.

I ran across a really good blog post on A/B testing, which I was aware of, but in terms of applying it to your icon, which I had never thought of. Here is a link to it since it's a good read: http://www.markj.net/ab-testing-iphone-app-names-360idev/. It looks like a relatively cheap method to perhaps get some better response to your app listing.

Another thing we're looking to do is get some cross-promotion going between our products. Since we only have one now, that's not really possible, but we'll be releasing our new game, Ninjatown: Trees Of Doom! in a couple of weeks, so we can see if that helps get more visibility (which will of course depend highly on how well it does.) We're also working on an arcade version of Space Miner that will be a pure free-to-play with ads.

Then of course there is the traditional medium of just buying ads. We'll probably dip our toes in this a bit to see how well it works, but you can sink a lot of money here fast, and the economics may be questionable if you don't do enough to drive yourself to the top of the charts.

The final thing I'm considering is to take the game on a "coaster". I don't know if that's a real term or not, but if it isn't, I call dibs! As for what a coaster is, I think it's best described by what EA did with the Simpsons or what Gameloft does occasionally. I'll expound on it more in another post.

Anyhow, the honeymoon is over, and we're settling in and getting comfortable now. The business challenges remain, but I look forward to finding solutions and learning new ways to be successful in this marketplace. I think a lot of people give up and move on at this point, which is a mistake. I think there is a lot of life left in Space Miner, and if nothing else, we'll expect to see a bump when we finish Space Miner 2 ;)

And one last thing - we're going to actually try and make this blog worth checking occasionally, so look for more frequent updates.