Putting Instabug to the Test

Our software engineers spend a lot of time analyzing customer feedback, and yours probably do, too. Is Instabug our remedy?

Customer feedback encompasses everything. From whether customers like an app’s user interface (UI) and user experience (UX), through to knowing what steps they take to perform an action, we can determine what they did immediately before an app crash or bug. Acquiring maximum customer feedback is critical to all phases of app development, especially with Minimum Viable Products. In a quest to improve the quantity and quality of the data our engineers collect, they decided to put Instabug to the test.

1. What problem/s were we trying to solve?

We were looking for a multi-functional tool that would give us the ability to:

1. a) capture the steps users took that resulted in crashes of applications
2. b) collect feedback directly from users.

We also wanted to reduce the time it took for users to report crashes, while collecting feedback from all participants.

2. Why Instabug?

Instabug is an established software company. Founded in 2012, it specializes in bug and crash reports, alongside user surveys. It gets very good reviews, and has been used successfully by the likes of Paypal, Samsung, Lyft, Scribd, and Mashable – among many others. Its feature list matched what we were looking for:

• Instabug’s “instant bug and crash” reporting was very attractive, especially when coupled with physical manipulations like shaking the device.
• Their paid version enables us to see which steps a user took, before submitting a report – e.g. which elements they interacted with.
• Instabug’s session profiler provides information about the user’s device – battery, RAM, connectivity, etc.
• Their app provides network logging – it captures network requests, then attaches them to the ticket, providing more information on bug or crash reports.
• Only a few steps are needed to integrated Instabug into a project.
• They have a professional (and very responsive) support team.

3. More Instabug Details

Instabug has one free and four paid packages: $124, $249, $499 and the enterprise-level “Let’s Talk $” monthly subscriptions, each with a 2-week free trial. We tested the $249 Team Tier for four members on up to two apps with its offer of in-app chats and surveys, alongside analytics and advanced integrations. The Enterprise level provides dedicated hosting, white labeling, SDK customizations, and no cap on members, apps or data retention. Instabug also integrates easily with Slack, Zapier, Jira, Trello, Basecamp and other project management tools.

4. Results and Observation

After we had everything set up, we put Instabug to the test. There are a lot of things that we like about Instabug. But again, our objective was to capture the steps users took that resulted in bugs or crashes, thereby collecting feedback directly from users faster.

Instabug Pros:

1. Instabug conveniently displayed the sequence of transitions between screens.
2. The session profiler captured RAM, storage, connectivity, battery life, and screen orientation info for the 60 seconds preceding a bug or crash.
3. It allowed us to record transitions between screens, with names of views (for iOS) or activity/fragment names (for Android), automatically creating screenshots of this when reporting a bug/crash.

Instabug Cons:

1. Screenshots are made right after transitions to the next screen – so most of the screenshots of user steps included transition screens.
2. Since Instabug listens to all in-app activities, the load on CPU and memory is high. We detected memory leaks on Android, and a major CPU load on iOS.
3. It caused a significant decrease in application performance resulting in a slower transition between application tabs.

Our Conclusion:

We ultimately concluded that Instabug is a good and useful tool. Nevertheless, it cannot substitute for a QA engineer’s effort when a complex issue must be investigated. It addressed our objectives in part, with faster user feedback, but lacking some details. With user steps, it was capturing the wrong screenshots (transitions). The slower application performance, memory leaks and CPU-load could also cause issues during application development and in long-term support.

We frequently use transition screens and animations as a UI and UX design element to let users know their latest action is being processed. Instabug’s present handling of transitions, however works well for apps that don’t make use of transitions. Similarly, apps with a narrow range of activities and low memory requirements could ease or negate the hit on performance. However, that does not eliminate the potential impact it could have with continuous product development. It may have nominal impact today, but add a feature or two and performance could become an issue later.

Some project owners may look at solutions like Instabug and think they can get away by not having an engineer on their team. Engineers in mobile app development have the ongoing task of tracking and improving app performance. They also fix bugs, perform compatibility tests, maintain your network’s infrastructure and evaluate tools. Instabug is a very good tool by a good company with reliable support, that has worked for a lot of great apps. Even so, it’s not a perfect fit for every mobile app, or for that matter, every developer.