Analysing FIT data with Perl: interactive data analysis

14 minute read

Printing statistics to the terminal or plotting data extracted from FIT files is all well and good. One problem is that the feedback loops are long. Sometimes questions are better answered by playing with the data directly. Enter the Perl Data Language.

Interactively analysing bike ride FIT data with Perl.
Image credits: Wikimedia Commons. Perl onion, Perl camel, Cyclist emoji.
Yes, I know the axes aren’t labelled.

Interactive data analysis

For more fine-grained analysis of our FIT file data, it’d be great to be able to investigate it interactively. Other languages such as Ruby, Raku and Python have a built-in REPL.¹ Yet Perl doesn’t.² But help is at hand! PDL (the Perl Data Language) is designed to be used interactively and thus has a REPL we can use to manipulate and investigate our activity data.³

Getting set up

Before we can use PDL, we’ll have to install it:

After it has finished installing (this can take a while), you’ll be able to start the perlDL shell with the pdl command:

To exit the pdl shell, enter Ctrl-D at the prompt and you’ll be returned to your terminal.

Cleaning up to continue

To manipulate the data in the pdl shell, we want to be able to call individual routines from the geo-fit-plot-data.pl script. This way we can use the arrays that some of the routines return to initialise PDL data objects.

It’s easier to manipulate the data if we get ourselves a bit more organised first.⁴ In other words, we need to extract the routines into a module, which will make calling the code we created earlier from within pdl much easier.

Before we create a module, we need to do some refactoring. One thing that’s been bothering me is the way the plot_activity_data() subroutine also parses and manipulates date/time data. This routine should be focused on plotting data, not on massaging its requirements into the correct form. Munging the date/time data is something that should happen in its own routine. This way we encapsulate the concepts and abstract away the details. Another way of saying this is that the plotting routine shouldn’t “know” how to manipulate date/time information to do its job.

To this end, I’ve moved the time extraction code into a routine called get_time_data():

The main change here in comparison to the previous version of the code is that we pass the activity data as an argument to get_time_data(), returning the @times array to the calling code.

The code creating the date string used in the plot title now also resides in its own function:

Where again, we’re passing the @activity_data array to the function. It then returns the $date string that we use in the plot title.

Both of these routines use the $date_parser object, which I’ve extracted into a constant in the main script scope:

Making a mini-module

It’s time to make our module. I’m not going to create the full Perl module infrastructure here, as it’s not necessary for our current goal. I want to import a module called Geo::FIT::Utils and then access the functions that it provides.⁵ Thus–in an appropriate project directory–we need to create a file called lib/Geo/FIT/Utils.pm as well as its associated path:

Opening the file in an editor and entering this stub module code:

we now have the scaffolding of a module that (at least, theoretically) exports the functionality we need.

Line 1 specifies the name of the module. Note that the module’s name must match its path on the filesystem, hence why we created the file Geo/FIT/Utils.pm.

We import the Exporter module (line 3) so that we can specify the functions to export. This is the @EXPORT_OK array’s purpose (lines 6-12).

Finally, we end the file on line 14 with the code 1;. This line is necessary so that importing the package (which in this case is also a module) returns a true value. The value 1 is synonymous with Boolean true in Perl, hence why it’s best practice to end module files with 1;.

Copying all the code except the main() routine from geo-fit-plot-data.pl into Utils.pm, we end up with this:

… which is what we had before, but put into a nice package for easier use.

One upside to having put all this code into a module is that the geo-fit-plot-data.pl script is now much simpler:

Poking and prodding

We’re now ready to investigate our power and heart rate data interactively!

Start pdl and enter use lib 'lib' at the pdl> prompt so that it can find our new module:⁶

Now import the functions we wish to use:

Since we need the activity data from the FIT file to pass to the other routines, we grab it and put it into a variable:

We also need to load the time data:

which we can then read into a PDL array:

With the time data in a PDL array, we can manipulate it more easily. For instance, we can display elements of the array with the PDL print statement in combination with the splice() method. The following code shows the last five elements of the $time array:

Loading power output and heart rate data into PDL arrays works similarly:

In the previous article, we wanted to know what the maximum power was for the second sprint. Here’s the graph again for context:

Eyeballing the graph from above, we can see that the second sprint occurred between approximately 47 and 48 minutes elapsed time. We know that the arrays of time and power data all have the same length. Thus, if we find out the indices of the $time array between these times, we can use them to select the corresponding power data. To get array indices for known data values, we use the PDL which command:

We can check that we’ve got the correct range of time values by passing the $indices array as a slice of the $time array:

The time values lie between 47 and 48, so we can conclude that we’ve selected the correct indices.

Note that we have to use the bitwise logical AND operator here because it operates on an element-by-element basis across the array.

Selecting $power array values at these indices is as simple as passing the $indices array as a slice:

Using the max() method on this output gives us the maximum power:

In other words, the maximum power for the second sprint was 932 W. Not as good as the first sprint (which achieved 1023 W), but I was getting tired by this stage.

The same procedure allows us to find the maximum power for the first sprint with PDL. Again, eyeballing the graph above, we can see that the peak for the first sprint occurred between 24 and 26 minutes. Constructing the query in PDL, we have

which gives the maximum power value we expect.

We can also find out the maximum heart rate values around these times. E.g. for the first sprint:

in other words, 157 bpm. For the second sprint, we have:

i.e. 165 bpm, which matches the value that we found earlier. Note that I broadened the range of times to search over heart rate data here because its peak occurred a bit after the power peak for the second sprint.

Looking forward

Where to from here? Well, we could extend this code to handle processing multiple FIT files. This would allow us to find trends over many activities and longer periods. Perhaps there are other data sources that one could combine with longer trends. For instance, if one has access to weight data over time, then it’d be possible to work out things like power-to-weight ratios. Maybe looking at power and heart rate trends over a longer time can identify things such as overtraining. I’m not a sport scientist, so I don’t know how to go about that, yet it’s a possibility. Since we’ve got fine-grained, per-ride data, if we can combine this with longer-term analysis, there are probably many more interesting tidbits hiding in there that we can look at and think about.

Open question

One thing I haven’t been able to work out is where the calorie information is. As far as I can tell, Zwift calculates how many calories were burned during a given ride. Also, if one uploads the FIT file to a service such as Strava, then it too shows calories burned and the value is the same. This would imply that Strava is only displaying a value stored in the FIT file. So where is the calorie value in the FIT data? I’ve not been able to find it in the data messages that Geo::FIT reads, so I’ve no idea what’s going on there.

Conclusion

What have we learned? We’ve found out how to read, analyse and plot data from Garmin FIT files all by using Perl modules. Also, we’ve learned how to investigate the data interactively by using the PDL shell. Cool!

One main takeaway that might not be obvious is that you don’t really need online services such as Strava. You should now have the tools to process, analyse and visualise data from your own FIT files. With Geo::FIT, Chart::Gnuplot and a bit of programming, you can glue together the components to provide much of the same (and in some cases, more) functionality yourself.

I wish you lots of fun when playing around with FIT data!

Support

If you liked this post and want to see more, please buy me a coffee!