dev

Improving C++ Code Coverage with Gcov, Gcovr and Doxide

Doxide can now produce line data on function templates, even when not instantiated, to mix with execution data for more accurate test coverage reports. This is especially useful when testing header-only libraries.

2024-11-13 (updated 2024-11-27)

Providing accurate code coverage reports for C++ is difficult, perhaps more so than other programming languages, because of function templates. Coverage tools such as GCC’s gcov and LLVM’s Source-Based Code Coverage instrument a program binary to count which lines, branches and functions are executed when that program is run, but function templates are compile-time entities that do not translate into a program binary at all if they are never instantiated. When computing code coverage, these missing function templates can mean under-counting the number of lines in a C++ program, and so over-reporting code coverage. This problem is particularly acute for test coverage of header-only libraries, where just about everything is a template.

I have been working on some tooling to improve this, and have added an experimental cover command to Doxide. That may seem a little out of place—Doxide is a documentation tool for C++ source code, not a test code coverage tool (not yet?)—but to extract that documentation it uses a C++ parser provided by Tree-sitter, and that same parser provides all the information necessary on function templates to make them visible to code coverage tools that otherwise rely only on binaries. The basic idea of this new cover command is to output that information to JSON with baseline zero execution counts, mix it with data from gcov on nonzero execution counts, then produce an HTML report of the two together using gcovr. In this way line data from source code supplements execution data from the binary program for more accurate code coverage reports.

This post provides a step by step guide to the whole process of using gcov, gcovr and doxide to produce such a report.

Update 17 November 2024

After further developing this idea, Doxide can now produce its own HTML code coverage reports too! Check it out.

Overview of workflow
Preliminaries
Working example
Building and running with coverage enabled
Demonstrating the problem
Generating a JSON report with Gcovr
Generating a JSON report with Doxide
Merging the JSON reports of Gcovr and Doxide
Summary and further work
Related links

Overview of workflow

The basic workflow is as follows:

Build tests with code coverage enabled.
Run tests.
Use gcovr to create a code coverage report from that execution in a JSON file.
Use doxide to create a line data report from the original source files in a JSON file.
Use gcovr again to create a final code coverage report in HTML, merging the two JSON files created in steps 3 and 4.

As we shall see, the lines of function templates that have not been instantiated will be reported as uncovered in the final report.

Preliminaries

You will need to install Gcovr and Doxide for this to work. The former is readily available from Linux distributions and as a Python package installable via pip; see the installation instructions. The latter you are unlikely to find in your Linux distribution just yet, but there are Linux, Mac and Windows packages available; see the installation instructions—you will need the absolute latest version at time of writing, which is 0.7.0.

Working example

We will consider the following simple program:

#include <iostream>

/**
 * Overload of f() with concrete type.
 */
double twice(double x) {
  return 2*x;
}

/**
 * Function template.
 */
template<class T>
T twice(T x) {
  return 3*x;  // deliberate error
}

int main() {
  double x = 5.0;
  double y = twice(x);
  std::cout << y << std::endl;
  return 0;
}

This has a function twice() that multiplies its argument by two. There is an overload of this function for arguments of type double, as well as a function template for generic types. The main() program, which we can think of as a test program for twice(), calls that function with argument x and prints out the result. Because x is of type double, the first overload of the function is called and the test looks successful. The function template is not instantiated during the compile, and not executed during the run—for good measure, we’ve snuck a silly error in there that would not be caught if this was our test suite!

Unfortunately, as we shall see, code coverage is reported at 100%, and the busy developer may overlook this problem.

Building and running with coverage enabled

To build a program with gcov coverage enabled, using either g++ or clang++, simply add the --coverage command-line option. It is a good idea to disable optimization and inlining as well. Using g++, if the above code is in a file called test.cpp, we can build with:

g++ --coverage -O0 -fno-inline -o test test.cpp

As well as the usual compile, this creates a file test.gcno in the same directory, with some basic block and source line mapping information necessary for collecting coverage data.

In addition to gcov-compatible coverage data, clang++ can provide alternative Source-Based Code Coverage data, enabled with different command-line options. However, gcovr and doxide require gcov-compatible data.

Now run the program as normal:

./test

This runs the program and outputs the result as normal, but also creates a second code coverage file test.gcda in the same directory, with data on how many times each line, branch, and function of the program was executed.

For more information on the test.gcno and test.gcda files, see Brief Description of gcov Data Files in the GCC documentation.

Demonstrating the problem

To see the problem with function templates, we can produce an HTML report using gcovr:

gcovr --html-details gcovr.html \
    --exclude-function-lines \
    --exclude-noncode-lines

This command already enables a few options to make the line count more accurate, excluding some lines that do not really reflect executable code: function signatures (--exclude-function-lines) and end-of-function closing braces (--exclude-noncode-lines).

Now open gcovr.html in your browser (created by the last command). It reports 100% code coverage. Clicking on test.cpp for further details, we see that, as expected, all of the main() function is executed along with the first definition of twice(), but that the generic version of twice() is unmarked. This means that it is simply not included in the coverage report at all; if it was included but uncovered it would be marked in red instead, and total coverage would be less than 100%. We may consider this a problem.

Screen grab of gcovr detailed output for test.cpp reporting 100% code coverage, but missing the body of the uninstantiated function template, which is marked in neither green as executed, nor red as unexecuted, it is simply missed

A fix is to mix line data from the source code with the count data from the execution. We can get that from Doxide. But first, we need to produce JSON output from Gcovr to mix it into.

Generating a JSON report with Gcovr

We previously ran gcovr to produce an HTML report. A simple change to the command allows us to produce a JSON report instead:

gcovr --json gcovr.json \
    --exclude-function-lines \
    --exclude-noncode-lines

An alternative to gcovr is lcov. We do not use it here as it does not have the JSON support that makes merging reports from gcovr and doxide straightforward. But for completeness, to use lcov, first collate coverage information in an lcov.info file:

lcov --directory . --no-external --capture --output lcov.info

then produce an HTML report with the accompanying tool genhtml:

genhtml lcov.info -o lcov

and open lcov/index.html in your browser. The report looks similar to gcovr.

Generating a JSON report with Doxide

Doxide must be first set up for documentation, although that is easy. Just run, in your souce directory:

doxide init

Inspect the doxide.yaml configuration file that is created and make sure that the files section includes your source files. By default all *.hpp and *.h header files are included, recursively. If you have function templates in *.cpp source files then add those source files here as well. For the working example we have the test.cpp source file to add at least. A minimal example of a working doxide.yaml file for code coverage is something like:

title: Untitled
path: /blog/cplusplus-code-coverage-with-gcov-gcovr-doxide/
category: dev
description: 
files:
  - "*.hpp"
  - "**/*.hpp"
  - "*.cpp"
  - "**/*.cpp"
  - "*.h"
  - "**/*.h"

Once Doxide is configured, produce a line data report by running, from the same directory as the doxide.yaml file:

doxide cover > doxide.json

The doxide.json file now contains the report. It lists files and line numbers for all functions and operators found in the source code, with zero execution counts.

The remainder of the doxide.yaml configuration file and the other files generated by doxide init are not important, unless you are also interested in using Doxide for documentation—they can be deleted if not of interest. The only exception is if Doxide fails to parse your source files correctly, often due to macro use, in which case further tweaking of the configuration file may be necessary. See the Doxide website on Parsing Considerations if that seems to be affecting you.

Merging the JSON reports of Gcovr and Doxide

In the previous two sections we produced a gcovr.json file with coverage data from executing the program, and a doxide.json file with line data from the source code. The next step is to combine them into a single HTML report with gcovr:

gcovr --html-details gcovr.html \
    --exclude-function-lines \
    --exclude-noncode-lines \
    --json-add-tracefile gcovr.json \
    --json-add-tracefile doxide.json

The two --json-add-tracefile options at the end of the command add the two JSON files that we have produced.

We can now inspect gcovr.html again. We see that coverage is now reported at 83.3%, with the function template marked as uncovered when the program was run. We may consider this a more accurate reflection of the code coverage of a test suite, which will be particularly useful for large header-only libraries, where a lack of data on uninstantiated function templates is easily missed.

Screen grab of gcovr detailed output for test.cpp, marking the body of the uninstantiated function template as uncovered, in red, and reporting less than 100% code coverage

Summary and further work

This post has proposed an approach to including uninstantiated function templates in C++ code coverage reports by mixing line data from source code with execution data from gcov. It uses a new feature in Doxide—the cover command—to provide the supplementary line data from source code.

This is still early work. While use of line data from source code can provide better recall of all executable lines, it may lack precision and include lines that are not, in fact, executable code, or that are executed at compile time rather than runtime (e.g. if constexpr lines). Refinements are needed in this regard.

Force-cover, which uses Clang tooling to preprocess C++ source code with comments to identify templates, then a Python program to postprocess code coverage output to ensure that those templates are counted as uncovered code.