The Campfire is a blog at the intersection of nature & technology brought to you by the team behind the award winning Portal App.

Why You Might Not Need In-App Analytics

We've always believed in our customers' need for privacy when using our App but Apple's recent announcement of App privacy details on the App Store prompted us to think even more deeply about the data we do collect and why. 

What started as the question "what data do we capture?", quickly turned into the question "what data do we need to capture?" This article explores our journey to zero customer data.

We're a small bootstrapped team of 4 with a minimal budget. Our approach to growth has always been to craft the best product we can, delight customers, charge a fair price and rely on word of mouth for organic growth. Short term growth hacks have never held much appeal to us, we're in this for the long term.

The Portal App has never had any 3rd party ad trackers, data-brokers or resold any data, we value our customers a lot more than that. Until recently we did, however, include the Google Firebase SDK which we used internally for 3 things...

  1. Crashlytics Crash Reports
  2. Google Analytics
  3. Firebase Remote Config Experiments - A/B tests

As of this week, we've removed Firebase and all three of the above are no longer in use. This was the result of several internal discussions around our use of data, prompted by Apple's App privacy questionnaire and our long standing concerns around 3rd party data processing.

The rest of this article will discuss our thought process behind removing all product analytics and what approaches we are trying instead.

Privacy as a Competitive Advantage

From meditation, to sleep, to focus; our customers have many different reasons for reaching for Portal but a common thread is the need for escape in its many forms, this is a deeply personal activity. Given this, anything we can do to reduce our customers’ anxiety whilst using the App must be a good thing right? 

The last thing you want when trying to escape with nature is the worry that someone is spying on you or the fear that your data is feeding an algorithm. This isn't a concern in the wilderness and we never want to be a concern in our App either.

As we've mentioned, we don't run any paid advertising but a lot of our competitors do. They've taken VC cash and need to show exponential growth. They need to advertise and growth hack their way to scale or die trying. This mindset necessitates tracking users.

Can our weakness of not having deep advertising pockets and large growth targets be our competitive advantage? It’s a long term bet but we believe it can.

Scale: small vs large, needle moving impact

Vanity Metrics vs The Environment

To be honest, one of our favourite uses of analytics data was to check in on the real time data and see how many hundreds of customers were actively using the App right now. These vanity metrics can be a motivator for a product team but have little practical benefit. 

You have to wonder if it's worth all those additional CPU cycles, battery drain, network traffic and data creep to provide developers with an ego boost. Just think of the worldwide carbon footprint of showing pretty realtime counters. And that's before we consider the user creepiness factor. We just can’t justify that any longer.

Looking back the primary situation where this data was genuinely useful to us was noticing sudden traffic changes as an indication of possible App Store features or 3rd party news articles. But we can replace this with:

  • App Follow - for tracking App Store features
  • CDN bandwidth stats for alerts on general traffic spikes or abuse
  • Google Search alerts, social search alerts and outreach for press mentions

Optimising For Customer Satisfaction Over Engagement

Common developer wisdom focuses on optimising products to maximise user engagement. How much can we get User X to open App Y and for how long. This path leads to invasive tracking, push notifications masked as user benefit and sub-optimal user experiences.

You don't buy a football and expect a reminder from it to go for a kick-about every morning, why should downloading an App be any different?

Our approach here has always been clear. We are delighted when our customers choose to use Portal but we understand that they all have different needs, goals and lives far beyond us. Using Portal (or any technology) should be a free and conscious choice made to enhance your life, not a socially engineered marketing optimisation.

How then can we ever trust engagement metrics? If we make a change and some measure of engagement goes up, how do we ever truly know whether to attribute that to a better product market fit or a mind hack? The developer’s intentions would be a driver here but isn't it easy to convince oneself any change is for the greater good if it helps your cause?

What's the alternative? We are trying to take a customer first approach by:

  1. Carrying out 1-to-1 customer interviews with anyone who is willing to chat with us. These don't scale but that's part of the beauty of it. The insights learned from a dozen 20 minute video calls is worth a lot more to us than any summary stats we were combing through. A nice side benefit is that you can help customers with support issues and prove you care deeply about their experience at the same time.
  2. Carefully reading, noting and replying to every single customer review, combing for common issues, niggles and feature requests. Replying to reviews gives us an opportunity to open dialogues with customers and get right to the heart of issues. Beyond this we produce weekly summaries of how reviews are trending and what the common themes are.
  3. Keeping a tight feedback loop between support and product development. Everyone on the team pitches in with all aspects to try and keep things as customer focussed as possible.

A/B Testing & Feature Rollout

By now you might have guessed that we aren't in the business of A/B testing lots of small colour and copy changes. We know we could likely optimise these things to increase conversion but that's not our style. We are craftsmen and would much prefer to create a product with heart that helps customers achieve their goals rather than one that was micro optimised against internal metrics.

Inevitably sometimes you need to make product changes that are high risk. Examples for us might include different audio visual technology choices or rolling out new experimental features. In the past we have relied on Firebase Remote Config for this.

Looking back we have run a handful of multivariate tests and in almost all cases they have given the result we likely would have chosen internally. It turns out if you use your product and are optimising for the customer (e.g. you) then your judgement calls are normally correct. By staying humble, listening intently to customers and letting your instincts guide you, you'll be able to get close enough and iterate from there.

There’s some additional technical benefits of avoiding remote configuration that are often overlooked:

  • Faster answers: At our scale it takes a couple of weeks to get meaningful experiment data which can impact the speed of shipping features. Sometimes shipping software, gathering feedback and iterating is faster than running the experiment!
  • Less bugs: Adding experiments to an App undoubtedly leads to code bloat and more corner cases to be tested, each experiment leads to exponentially more issues here.
  • Simpler support: When everyone is using a different version of your App supporting customers becomes problematic and remote debugging becomes a minefield. With everyone seeing the same code it's easier to streamline support and reason about errors.

Testing Pricing and Conversion

This is a big one, but in truth when we take a step back Firebase data isn't a huge help for us here. The simple fact is that testing pricing is hard and testing pricing on the App Store is near impossible.

Over recent years the funnel data in App Store Connect has improved to the point that it is now useful enough for our purposes and tracking general trends. But the App Store does not make testing pricing easy and changes can lead to a lot of unnecessary customer confusion, which leads us to make changes rarely, if at all.

More data isn't the fix here. Better Apple tooling is.

Tracking Software Quality

If an App crashes but no one hears it did it even happen?

Honestly, this was the biggest concern for our small team. As discussed the customer comes first, so it pains us every time there is a crash report. We think we have a good track record here though, before turning off crash reporting our "Crash Free Users" number was pretty stable at 99.93%. Still, it's concerning to think about customers hitting issues and us not knowing about them until they complain.

We've made peace with it and decided to try and minimise the uncertainty.

  • App Store Connect data gives us basic visibility into App stability via Apple's opt-in crash reports. (Albeit aggregated and very delayed)
  • We will be looking to focus on improving internal testing and QA processes over the next few months.
  • We work in cycles internally but try to avoid the bi-weekly App Store release cadence of certain other Apps, this gives us freedom to ship features when they are ready.

Given all this, we do have anxieties about how this change might impact our ability to ship a high quality product and that when issues do occur customer support may suffer due to less data to debug issues. Only time will tell.

When Zero Isn't Quite Zero

By now I hope to have persuaded you that zero data is a good and worthwhile goal that can even have commercial advantages. Can you ever truly get to zero though? 

We've removed all analytics style data but even then we do have data leakage, we've been careful to make sure it's not identifiable info and that we are using it for the right reasons but the App still sends:

  • Requests to our API to fetch the content needed.
  • App version, OS version and device type in a user-agent header so we can serve the correctly formatted content from our Content Delivery Network.
  • App Store purchase receipts so that we can validate purchases are genuine and unlock access to content.
  • API server logs which can include IP addresses for reasons of security and fraud prevention.

We've tried to minimise this data as much as possible and it's never used for tracking users or identifying them but it's worth acknowledging that even without explicit tracking any App that talks to a server on the internet at any point is leaking data in some form or another.

Inspired by the physical world

As an App developer, it's easy to take having data for granted in modern times but we should remember that this isn't the norm elsewhere.

If you go for a hike in the woods no one is hiding in the woods watching how long you walk for or how big your strides are (except maybe you but that's a different story). The landowner is presumably just happy you came by.

If you buy your child a new bike for her birthday, the shopkeeper doesn't keep tabs on how many miles she’s ridden. He's just happy you brought a bike and hopeful that he gave good enough service that you'll come back again next time.

That's the kind of world we want to live in and the kind of App we hope to build.