Debugging: symbolicating crash reports manually (stack trace, backtrace)

Today, we’ll talk about manually symbolicating iOS and OS X application “crash reports.” Why? When you hear about a crash in one of your apps from a customer, the first thing you should do is try to get a copy of the crash report. But there are times when you get crash reports that aren’t automatically symbolicated, or that you can’t symbolicate by dragging into Xcode, or are partially symbolicated. When not symbolicated, you’re reading numeric addresses when you want to be reading code, like your function/class names. There are workarounds and we’ll discuss one today. Download the sample Xcode 9 project written in Objective-C to follow along. What’s a crash report, anyway? According to Apple:

When an application crashes, a crash report is created and stored on the device. Crash reports describe the conditions under which the application terminated, in most cases including a complete backtrace for each executing thread, and are typically very useful for debugging issues in the application. You should look at these crash reports to understand what crashes your application is having, and then try to fix them.

Crash reports with backtraces need to be symbolicated before they can be analyzed. Symbolication replaces memory addresses with human-readable function names and line numbers. If you get crash logs off a device through Xcode’s Devices window, then they will be symbolicated for you automatically after a few seconds. Otherwise you will need to symbolicate the .crash file yourself by importing it to the Xcode Devices window. …

Er, ah, when it comes to believing the statement, “they will be symbolicated for you automatically after a few seconds. Otherwise you will need to symbolicate the .crash file yourself by importing it to the Xcode Devices window,” well, don’t bet your life on either automatic symbolication or symbolication upon dragging into Xcode. Often, dragging into Xcode does absolutely nothing.

Introduction

Suppose that someone sends you a copy of crash report through email or on a USB memory stick — a text file with .txt or .crash extension. It happens quite often, especially when you have an installed base of hundreds of thousands or millions. Some people don’t want Apple collecting all sorts of information about them on their Macs, MacBook Pros, iPhones, iPads, watches, etc., so they turn off the “Share analytics, diagnostics, and usage information with Apple” option. There are also people out there using OS X applications developed by third-party software boutiques and distributed outside of The Mac App Store.

Yes, Apple claims that “Xcode will automatically symbolicate [a] crash report and display the results” if you drag the report onto Xcode, and say you’ve kept copies of your “crashing application’s binary and dSYM file.” But guess what? I don’t know how many times I’ve seen this “automatic” symbolication fail. Have you ever tried to examine an unsymbolicated crash report? It’s often useless, especially when none of your code has been symbolicated. You’re just looking at a big list of numbers — memory addresses. That’s no help. What do you do if you’ve just received a crash report and it looks like this?

Look at lines 7,8,9 in the code view above (lines 4,5,6 in the crash report’s backtrace, respectively). Those are just numbers (addresses). They’re not symbolicated. I want to see my code. This crash report was the result of a crash in a release version of an app, one in which the symbols (human-readable code), have been stripped out. What’s a release version? I’m not going to explain everything, but will give you clues.

One of the purposes of this blog is to inculcate developers with the skills, behaviors, and knowledge needed to be the best of the best. Doing research is one of those skills. Follow the links I provide and either discover what the terms I’ve highlighted mean or at least reinforce your understanding of those terms. Here’s a snippet from Microsoft on Visual Studio, but the same concepts apply to Xcode (my edits in strikeout and in brackets, like [ and ]):

Visual Studio [Xcode] projects have separate release and debug configurations for your program. As the names imply, you build the debug version for debugging and the release version for the final release distribution.

The debug configuration of your program is compiled with full symbolic debug information and no optimization. Optimization complicates debugging, because the relationship between source code and generated instructions is more complex.

The release configuration of your program contains no symbolic debug information and is fully optimized. Debug information can be generated in .pdb [.dSYM] files, depending on the compiler options that are used. Creating .pdb [.dSYM] files can be very useful if you later have to debug your release version. …

For Xcode, note how I substituted “.dSYM” for “.pdb” in the previous quote.

Definitions

Crash report
A “crash report” or “crash log” contains a wealth of information to help you debug a released app: the app name, the operating system version, the date/time of the crash, the threads running at the time of the crash, the stack trace, the libraries your app is dependent upon (e.g., .dylib), information on allocated memory, details about the device on which the crash occurred, the target machine architecture (i.e., x86_64 for a MacBook or arm64 for an iPhone), etc. Make use of crash reports. If someone reports a crash in one of your apps, the first thing you should do is try to get a copy of the crash report.

A crash report for a release version of an app is useful only as long as you keep the current app version’s .dSYM or “debug symbol file.” .dSYM files store the original symbols you used when writing your app in the Xcode IDE. Your pseudo-English source code files cannot be run on a device’s processor. When building (compiling and linking), your source code’s object names, variable names, function names, etc., are converted into a binary format that is executable on a device. But those names — symbols — are converted to addresses when Xcode produces a binary.

Call stack/stack trace/backtrace
Generally, the “call stack,” “backtrace,” or “stack trace” is what most developers want to look at to find out where in their code the bug/problem/exception occurred that caused their app to crash. According to Wikipedia:

A stack trace allows tracking the sequence of nested functions called – up to the point where the stack trace is generated. In a post-mortem scenario this extends up to the function where the failure occurred (but was not necessarily caused). Sibling calls do not appear in a stack trace.

If you’ve just started software development or are a seasoned developer, by now you should understand terms like “stack” and “stack trace.” You need to know what a “stack” is most definitely. If you don’t, you’re in big trouble.

Apple has a good article on “Examining the Call Stack.” Read it.

Symbolicating a crash report
After everything I’ve said so far, do you understand what “symbolicating a crash report” means? Think about it first before reading further. When I was a student, I learned that effective studying meant trying to rebuild the information conveyed to me in class in my mind before looking at my notes. To reinforce my discussion, read Apple’s definition (my emphasis added):

Symbolication is the process of resolving backtrace addresses to source code method or function names, known as symbols. Without first symbolicating a crash report it is difficult to determine where the crash occurred.

Manually symbolicating a crash report

If you find that you can’t symbolicate a crash report automatically, all is not lost. You can manually symbolicate each line in the backtrace. You’re going to use the OS X atos command in Terminal.

The man page for the atos command on OS X states:

The atos command converts numeric addresses to their symbolic equivalents. If full debug symbol information is available, for example in a .app.dSYM sitting beside a .app, then the output of atos will include file name and source line number information.

Note that other Unix operating systems support atos.

Apple even admits that you may have to manually symbolicate crash reports, and provides guidance for atos usage:

Here are the steps (we’ll perform a real symbolication below):

1. Get the .dSYM for the currently released version of your app;
2. Copy the .dSYM into a folder in which you stored things like your .app bundle, DistributionSummary.plist, ExportOptions.plist, and Packaging.log (you can copy the .dSYM anywhere, but I like to stay organized);
3. Open Terminal and cd to the folder containing your .dSYM;
4. Note the hardware platform on which the crash occurred (Apple crash reports say things like “X86-64,” but atos wants “x86_64”);
5. Open your unsymbolicated crash report in a good text editor like TextWrangler;
6. Run the atos command in Terminal for each line of unsymbolicated code, noting the corresponding crash report line number, and atos will produce the fully symbolicated line of code; and,
7. Copy each symbolicated line of code, noting the line number in the backtrace, and paste into the corresponding unsymbolicated crash report, overwriting the corresponding unsymbolicated line in the backtrace.

Example
Refer to the excerpt of the unsymbolicated crash report at the beginning of this article. We’re going to symbolicate it now:

Line 4

Line 5

Line 6

Now let’s copy the symbolicated lines into the unsymbolicated crash report and — voila! — Shama Lama Ding Dong! (see video below). Look at code view lines 7,8,9 (corresponding crash report lines 4,5,6, respectively). Thar’s code in that thar crash report!

Wrapping up

Take a look at the sample Xcode 9 project to see how I forced a crash. Look at the function calls that lead to the crash — the stack trace! I hope this helps you smash some ugly bugs.

Enjoy — and above all, celebrate with Otis Day And The Knights: