.png)
1 day ago
2
Much like having thoughts in the shower and working on trains, I often find my best thinking happens while walking my dog in the morning. To not lose track of all this good thinking, I’ve started recording voice notes on my iPhone using Apple’s Voice Memos app.
After recording, the app automatically generates transcriptions, allowing me to easily search, copy, and edit my memos. Unfortunately—Apple being Apple—you can’t access these transcripts from anywhere outside the Voice Memos app.
Using Voice Memos ticks a lot of boxes for me:
- I can be a better dog dad: I used to bring a small notebook on our morning walks, but I felt rude asking my dog to wait while I scribbled something every few meters.
- I get to think out loud: I’ve found that verbalizing my ideas helps me organize, prioritize, and clarify them.
- I actually use my notes: Writing emails, drafting blog posts, making todo lists; automatic transcriptions are queryable and ready for editing.
This is what’s called a “win-win-win”—and I love it when it happens—but that last “win” doesn’t come as easily as the others.
Step 0: The Problem
Unfortunately, the only way to get your transcripts out of Voice Memos and into somewhere useful, is to clumsily copy-and-paste them.
This is an example of the worst kind of problem.
Copying-and-pasting isn’t a big deal, I know. But it is just enough of a deal to stop me from using the transcripts like I want to—which have become more valuable to me than the audio itself.
Plus, if there’s anything we programmers hate most, it’s copying-and-pasting… isn’t that what we have computers for?
Step 1: Find the Transcripts
Thinking of how I might solve this, I was reminded of a time when I tried exporting text from Apple Notes. When synced via iCloud, I learned that Notes uses SQLite and that it was accessible on Mac.
Thankfully, the authors of tools like apple-notes-to-sqlite had already deciphered and documented the complicated schemas and built open source tools to solve my problem.
I dove in hoping the same would be true for Voice Memos.
However, in the far reaches of Apple StackExchange, I came across this:
Q. Where are the transcripts that are generated from recordings in the Voice Memos app stored in the file system?
A. There is no documentation from Apple that I know of, but the MPEG-4 standard provides for encoding a text stream in the container itself; i.e., the *.m4a file. That must be what Voice Memos is doing. I don’t see any separate file resulting from transciption [sic]. See: ISO/IEC 14496-17:2006(en) Information technology — Coding of audio-visual objects — Part 17: Streaming text format (iso.org)
— Location of Voice Memos transcripts in the file system?, Apple StackExchange
The good news was that, when synced via iCloud, the Voice Memos audio could be accessed as .m4a files directly.
The bad news was that I didn’t know anything about audio file formats or what “…encoding a text stream in the container itself…” meant.
Step 2: Explore the Files
To postpone having to open up any part of ISO/IEC 144961, I thought I’d just poke around the .m4a files a bit.
If you try opening one of those files in a text editor, you’ll see that it is not meant to be read by humans. But, useful plaintext strings can often be found buried within various binary formats.
Finding these hidden strings is something reverse engineers do when they otherwise don’t know how to decompile or deconstruct a binary file. (.m4a is a well established format, but we haven’t read the documentation yet, remember?)
If Apple is encoding transcripts directly within the .m4a file, as the StackExchange answer suggested, we might be able to see them using the command line tool, strings.
Sure enough, strings reveals that, starting at byte 7438979, the .m4a file contains a JSON-esque string which has some of my Voice Memos transcript in it.
If it really was JSON (spoiler: it was, except for the “tsrp” part before the opening brace), then we should be able to extract and parse it.
Sticking to the commandline, let’s use a few more tools for wrangling this text.
- strings, as before, will output printable strings,
- rg, will filter for that tsrp prefix,
- sed, will remove the prefix and leave just the JSON, and finally,
- jq, will let us explore and manipulate the payload.
Author’s note: this was a very exploratory and iterative process, especially the jq code. I’ve written a tool called ijq to help with this. You can read about it here: Interactive jq
The first thing I’m interested in is the structure of the JSON.
The transcript data is stored in an attributedString object, which has an attributeTable (an array of timeRange objects) and a runs array. The locale object contains the language identifier and current locale.
The actual text of the transcript is inside the runs array. Each word is followed by an index.
The index maps to a timeRange in the attributeTable array.
Each timeRange is an array itself, containing two numbers: the start and end time offset of each transcribed word, in seconds.
Let’s just take a moment to pause and appreciate that, despite not knowing anything about audio formats or how text streaming works in ISO/IEC 14496-17:2006(en), we were able to find the raw transcript embedded in the .m4a file!
From here, we could continue wrangling and reconstruct the text.
But, since the dog has already gone for a walk, I guess there’s no better time than now to crack open ISO/IEC 14496, and really try to understand what’s going on.
Containers
.m4a files are a type of MPEG-4 (.mp4), specifically an audio-only variant. The MPEG-4 Part 14 (.mp4) standard is an implementation of the ISO Base Media File Format (Part 12 of ISO/IEC 14496).
These files are called “container formats” because they bundle together multiple different types of data.
The “A” in .m4a stands for “audio”, but these file types can also hold things like images, video, artist metadata, chapter markers, and yes, even transcriptions.
Atoms
The “container” is organized in a hierarchical structure of “boxes” (which Apple calls “atoms”). Each atom contains a particular piece of data related to the media being stored; like metadata, configuration, the media itself, or other nested atoms.
We can use the mp4dump command-line tool to look at the atom structure of our .m4a file.
View full mp4dump outputEach atom begins with an 8-byte header (typically). The first 4 bytes specify the atom size (a 32-bit integer indicating the total size of the atom, including the header). The next 4 bytes specify the type (e.g., ftyp, mdat, moov), a four-character code identifying how to interpret the atom’s payload.2
Note: mp4dump outputs the size as "#{header_size}+#{payload_size}".
The mp4dump output shows us the headers of each atom. Here, we can see one is of type "tsrp", which matches the string we found earlier with strings.
tsrp is a “leaf” atom since it contains no child atoms in the hierarchy. Leaf atoms are the ones that usually contain media data, and this particular tsrp atom takes up 168861 bytes.
Apple defines what each of these atoms are for and how they’re structured in their Quicktime File Format documentation.3 The documentation provides a list of the most common atoms, their types, and their purposes.
| ftyp | An atom that identifies the file type specifications with which the file is compatible. |
| moov | An atom that specifies the information that defines a movie. |
| trak | An atom that defines a single track of a movie. |
| tkhd | An atom that specifies the characteristics of a single track within a movie. (Required by trak) |
| mdia | An atom that describes and defines a track’s media type and sample data. (Required by trak) |
| mdhd | An atom that specifies the characteristics of a media, including time scale and duration. (Required by mdia) |
| udta | An atom where you define and store data associated with a QuickTime object, e.g. copyright. |
| tsrp | NULL?? |
As you might have noticed, I didn’t include a definition for the tsrp atom in the table above.
This is because Apple doesn’t document it, and being a custom atom, it’s not part of the ISO/IEC 14496 standard itself.
All we know is that tsrp is a custom atom type used to store transcriptions of audio recordings in JSON.
I’m very curious about this:
Why doesn’t Apple document their custom tsrp atom, similar to how they document other custom atoms?
Why have a custom atom at all? Surely there might be a suitable type already in the standards?4
If not, why use a JSON string, instead of nesting text atoms within the tsrp atom?5
And, what are the trade-offs to storing the transcript in SQLite, similar to Notes?
One benefit of storing the transcript directly in the audio file is its portability. Saving the file means you also save the transcript, even outside of Apple’s ecosystem.
Then again, Apple has this to say about undocumented atom types:
If your application encounters an atom of an unknown type, it should not attempt to interpret the atom’s data. Use the atom’s size field to skip this atom and all of its contents. This allows a degree of forward compatibility with extensions to the QuickTime file format.
—QuickTime File Format/Storing and sharing media with QuickTime files/Atoms
Accessing the Transcript Atom
Ignoring Apple’s advice, I’ve written a Ruby script to attempt to interpret this atom’s data.
Using what we’ve discovered, the script parses atom headers, recursively traverses the hierarchy to find the tsrp atom, and then extracts and reconstructs the JSON payload within.
It was a great learning experience to read through the documentation and write the Ruby script. But, to make the job even easier, we can use purpose-built tools and libraries.
mp4extract (which comes bundled with the same tools as mp4dump), takes an atom path, like “moov/trak/udta/tsrp”, and outputs its payload (using the --payload-only option avoids also outputting the header).
Once we have the payload, we can use jq to not only parse the JSON, but also extract the text from the runs array and concatenate it into a single string.
This now means that we only need to copy-and-paste a few lines of code to get the transcripts out of Voice Memos and into somewhere useful!
- Better dog dad? Win.
- Thinking out loud? Win.
No copy-and-paste?Improved workflow? Win.
Postscript
First, I hope you enjoyed this journey into reverse engineering Apple’s Voice Memos format. I had fun and I hope you did too.
If you have any questions, comments, or actually know something about this stuff, please feel free to reach out!
Oh, and if we actually want to avoid having to copy-and-paste, we can, of course, write a bash script :)
June 8, 2025 · ruby, reverse-engineering
