Co-authored with Amit Dharmapurikar and Atri Sarkar.

Introduction

Code coverage is a crucial metric which can provide good insight about software quality. Code coverage is typically expressed in percentage and provides information about how much of your software and which parts of your code have been tested by a set of unit tests or a test suite. Imagine it as a text highlighting tool which shows which parts of code are tested and which parts of the code can be considered for further testing in order to improve the quality of your code.

Higher code coverage increases confidence in your code’s functionality and reduces the risk of bugs. Most code coverage tools can be integrated into a CI pipeline so that code coverage is continuously monitored as your code grows. In several projects, code coverage requirements are included in merge policies to ensure software quality. As code coverage tools also report the areas of code which are not tested by the test suite, it helps to identify focus areas for testing. In certain regulated domains like automotive, aerospace, and healthcare where safety is crucial, code coverage requirements are part of the qualification and certification process.

There are various methods available for estimating code coverage by a test suite:

1. Statement coverage: Although a more popular term is line coverage, in many languages there could be multiple statements on a single line, so purists prefer statement coverage. If a statement is executed at least once when running the test suite, it is considered covered.
2. Region coverage: The percentage of code regions that have been executed. A code region may span multiple lines consisting of multiple statements, such as a function body.
3. Branch coverage: A metric used to assess how many branches or decision points in a program’s control flow have been executed by test cases (e.g. if statements or match arms).
4. MC/DC coverage: Modified Condition/Decision Coverage is a rigorous testing technique used in safety-critical systems. It ensures every decision has been tested under all possible outcomes, and each condition within a decision can independently affect the outcome while requiring fewer test cases than full condition coverage.

In this blog, we will discuss how to generate code coverage information for Rust applications. We will also discuss the current state and features expected in future versions of the Rust compiler related to code coverage. Rust uses LLVM as its compilation backend and leverages instrumentation coverage features provided by LLVM.

Enabling code coverage in Rust

Code coverage data in Rust can be collected with two methods:

1. Using -Z profile flag: Uses GCC-compatible gcov-based coverage based on DebugInfo. This provides a clear roadmap of code execution.

2. Using -C instrument-coverage flag: Enables LLVM’s native coverage instrumentation, known for high efficiency and precision.

In this blog, we focus on -C instrument-coverage in detail as it is widely used for modern source-based code coverage.

The following simple example shows how this works:

pub fn is_even(n: u32) -> bool {
    if n % 2 == 0 {
        true
    } else {
        false
    }
}

and its corresponding test:

#[test]
pub fn test_even() {
    assert_eq!(is_even(2), true);
}

To enable code coverage, we pass the -C instrument-coverage flag while compiling with the Rust compiler.

When using Cargo, we enable this flag using the RUSTFLAGS environment variable:

RUSTFLAGS="-C instrument-coverage" cargo test

This produces an “instrumented” binary with extra code injected to count lines, regions, and branches.

Effectively, the compiler injects counter increments like shown below:

pub fn is_even(n: u32) -> bool {
    is_even_counter += 1;
    if n % 2 == 0 {
        is_even_true_branch_counter += 1;
        true
    } else {
        is_even_false_branch_counter += 1;
        false
    }
}

Whenever a specific function, region, or branch is entered, an associated counter is incremented.

Apart from this, a runtime is also injected into the binary which initializes and manages these counters throughout program execution (the LLVM profiler runtime):

// Implicit main of test binary
pub fn main() {
    __initialize_counters();
    test_even();
    // Other tests...
    __write_counter_data_to_file();
}

When the program exits, raw counter data is written to a .profraw file which is then used to create coverage reports.

Note: This is an intentional simplification. The compiler doesn’t actually inject counters at the source level. During compilation, source code is lowered through intermediate representations (MIR), where counter injection actually takes place.

Code coverage tools in Rust

Popular coverage tools in the Rust ecosystem:

1. Tarpaulin: A tool providing source code line coverage tailored specifically for Cargo projects.
2. grcov: An aggregation tool that collects coverage data from test suites using llvm-cov to create HTML/lcov reports.
3. cargo-llvm-cov: A modern Cargo subcommand that wraps LLVM source-based code coverage.

Using cargo-tarpaulin

# Install cargo-tarpaulin
cargo install cargo-tarpaulin

# Run cargo-tarpaulin to collect coverage data
cargo tarpaulin

Using grcov

# Install grcov
cargo install grcov

# Run your tests with coverage instrumentation
RUSTFLAGS="-C instrument-coverage" cargo test

# Generate HTML coverage report
grcov . -s . --binary-path ./target/debug/ -t html --ignore tests/ -o ./target/debug/coverage/

# Open the report in your browser
open target/debug/coverage/index.html

Using cargo-llvm-cov

# Install cargo-llvm-cov
cargo install cargo-llvm-cov

# Run tests with coverage and print report to console
cargo llvm-cov

Conclusion

Code coverage provides highly valuable insights for assessing and improving the quality of software. Rust provides strong support for line and region coverage with ongoing advancements in LLVM for branch and MC/DC coverage.