Code Coverage

Code coverage is a metric that measures which parts of the tested code have been executed and which parts have not. By analyzing code coverage, the effectiveness and thoroughness of testing can be evaluated.

Code coverage includes:

• Line Coverage: The number of lines executed in the tested code. This is the simplest metric, and the goal is usually 100%.
• Branch Coverage: Whether each branch of every control structure has been executed. For example, in an if statement, have both the true and false branches been executed?
• FSM Coverage: Whether all states of a finite state machine have been reached.
• Toggle Coverage: Tracks the toggling of signals in the tested code, ensuring that every circuit node has both 0 -> 1 and 1 -> 0 transitions.
• Path Coverage: Examines the coverage of paths. In always or initial blocks, if ... else and case statements can create various data paths in the circuit structure.

* The primary simulator used in this project is Verilator, with a focus on line coverage. Verilator supports coverage statistics, so when building the DUT, the -c option must be added to the compilation options to enable coverage statistics.

Relevant Locations in This Project

To enable coverage, the -c option must be added during compilation (when using the picker command). Refer to the Picker Parameter Explanation. Additionally, the line coverage function must be implemented and enabled in the test files to generate coverage statistics during Toffee testing.

In conjunction with the above description, code coverage will be involved when compiling, writing and enabling line coverage functions and tests in this project:

Adding Compilation Scripts

Write the build(cfg) -> bool Function

# Omitted earlier code
    if not os.path.exists(get_root_dir("dut/RVCExpander")):
        info("Exporting RVCExpander.sv")
        s, out, err = exe_cmd(f'picker export --cp_lib false {get_rtl_dir("rtl/RVCExpander.sv", cfg=cfg)
                                                              } --lang python --tdir {get_root_dir("dut")}/ -w rvc.fst -c')
        assert s, "Failed to export RVCExpander.sv: %s\n%s" % (out, err)
# Omitted later code

In the line s, out, err=..., the picker command is used with the -c option to enable code coverage.

Set Target Coverage Files (line_coverage_files Function)

Write the line_coverage_files(cfg) -> list[str] function as needed, and enable test result processing (doc_result.disable = False) to ensure it is invoked.

Building the Test Environment

Define Necessary Fixtures

set_line_coverage(request, coverage_file)                          # Pass the generated code coverage file to toffee-report

Use the toffee-test.set_line_coverage function to pass the coverage file to Toffee-Test, enabling it to collect data for generating reports with line coverage.

Ignoring Specific Statistics

Sometimes, certain parts of the code may need to be excluded from coverage statistics. For example, some parts may not need to be tested, or it may be normal for certain parts to remain uncovered. Ignoring these parts can help optimize coverage reports or assist in debugging. Our framework supports two methods for ignoring coverage:

1. Using Verilator to Specify Ignored Sections

Using verilator_coverage_off/on Directives

Verilator supports ignoring specific code sections from coverage statistics using comment directives. For example:

// *verilator coverage_off*
// Code section to ignore
...
// *verilator coverage_on*

Example:

module example;
    always @(posedge clk) begin
        // *verilator coverage_off*
        if (debug_signal) begin
            $display("This is for debugging only");
        end
        // *verilator coverage_on*
        if (enable) begin
            do_something();
        end
    end
endmodule

In the above example, the debug_signal section will not be included in coverage statistics, while the enable section will still be counted.

For more ways to ignore coverage in Verilator, refer to the Verilator Documentation.

2. Using Toffee to Specify Filters

def set_line_coverage(request, datfile, ignore=[]):
    """Pass

    Args:
        request (pytest.Request): Pytest's default fixture.
        datfile (string): The coverage file generated by the DUT.
        ignore (list[str]): Coverage filter files or directories.
    """

The ignore parameter can specify content to be filtered out from the coverage file. For example:

...
set_line_coverage(request, coverage_file,
                  get_root_dir("scripts/frontend_ifu_rvc_expander"))

During coverage statistics, the line_coverage.ignore file in the scripts/frontend_ifu_rvc_expander directory will be searched, and its wildcard patterns will be used for filtering.

# Line coverage ignore file
# Ignore Top file
*/RVCExpander_top*%

The above file indicates that files containing the keyword RVCExpander_top will be ignored during coverage statistics (the corresponding data is collected but excluded from the final report).

Viewing Statistics Results

After completing all the steps, including preparing the test environment (Download RTL Code, Compile DUT, Edit Configuration), and adding tests (Add Compilation Scripts, Build Test Environment, Add Test Cases):

Now, Run Tests. Afterward, an HTML version of the test report will be generated in the out/report directory by default.

You can also view the statistics results by selecting the corresponding test report (named by test time) under “Current Version” in the Progress Overview section and clicking the link on the right.