Add Test

• 1: Add Compilation Script
• 2: Build Test Environment
• 3: Add Test Cases
• 4: Code Coverage
• 5: Functional Coverage

1 - Add Compilation Script

Script Target

Write a compilation file for the corresponding RTL in the scripts directory using Python (e.g., build_ut_frontend_ifu_rvc_expander.py).
The goal of this script is to provide RTL-to-Python DUT compilation, target coverage files, and custom functionality.

Creation Process

Determine File Name

Select the UT to be verified in XiangShan Kunming Lake DUT Verification Progress. If it is not available or needs further refinement, you can manually add it by editing configs/dutree/xiangshan-kmh.yaml.
For example, if we want to verify the rvc_expander module under the ifu module in the frontend, we need to add the corresponding part to configs/dutree/xiangshan-kmh.yaml (this module already exists in the YAML file; this is just an example):

name: "kmh_dut"
desc: "All Kunming Lake DUTs"
children:
  - name: "frontend"
    desc: "Frontend Module"
    children:
      - name: "ifu"
        desc: "Instruction Fetch Unit"
        children:
          - name: "rvc_expander"
            desc: "RVC Instruction Expander"

The naming format for the script file is as follows:

scripts/build_<top_module>_<sub_module>_..._<target_module>.py

Currently, the project includes four top-level modules:

1. ut_frontend (Frontend)
2. ut_backend (Backend)
3. ut_mem_block (Memory Access)
4. ut_misc (Miscellaneous)

Submodules do not have the ut_ prefix (the top-level directories have this prefix to distinguish them from other directories).

For example, if the target DUT to be verified is the rvc_expander module:
This module belongs to the frontend, so the top-level module is ut_frontend. Its submodule is ifu, and the target module is rvc_expander.
From the previously opened yaml file, we can also see that the children of frontend is ifu, and the children of ifu is rvc_expander.
Thus, the script name to be created is build_ut_frontend_ifu_rvc_expander.py.

Write the build(cfg) -> bool Function

The build function is defined as follows:

def build(cfg) -> bool:
    """Compile DUT
    Args:
        cfg: Runtime configuration, which can be used to access configuration items, e.g., cfg.rtl.version
    Return:
        Returns True or False, indicating whether the function achieved its intended goal
    """

The build function is called during make dut. Its main purpose is to convert the target RTL into a Python module. Other necessary processes, such as compiling dependencies, can also be added. For example, in build_ut_frontend_ifu_rvc_expander.py, the function primarily performs RTL checks, DUT checks, RTL compilation, and disasm dependency compilation:

import os
from comm import warning, info


def build(cfg):
    # Import related dependencies
    from toffee_test.markers import match_version
    from comm import is_all_file_exist, get_rtl_dir, exe_cmd, get_root_dir
    # Check RTL version (an empty version parameter means all versions are supported)
    if not match_version(cfg.rtl.version, "openxiangshan-kmh-*"):
        warning("ifu frontend rvc expander: %s" % f"Unsupported RTL version {cfg.rtl.version}")
        return False
    # Check if the target file exists in the current RTL
    f = is_all_file_exist(["rtl/RVCExpander.sv"], get_rtl_dir(cfg=cfg))
    assert f is True, f"File {f} not found"
    # If the DUT does not contain RVCExpander, use picker to package it into Python
    if not os.path.exists(get_root_dir("dut/RVCExpander")):
        info("Exporting RVCExpander.sv")
        s, out, err = exe_cmd(f'picker export --cp_lib false {get_rtl_dir("rtl/RVCExpander.sv", cfg=cfg)} --lang python --tdir {get_root_dir("dut")}/ -w rvc.fst -c')
        assert s, "Failed to export RVCExpander.sv: %s\n%s" % (out, err)
    # If disasm/build does not exist in tools, compile disasm
    if not os.path.exists(get_root_dir("tools/disasm/build")):
        info("Building disasm")
        s, _, _ = exe_cmd("make -C %s" % get_root_dir("tools/disasm"))
        assert s, "Failed to build disasm"
    # Compilation successful
    return True

def line_coverage_files(cfg):
    return ["RVCExpander.v"]

For details on how to use picker, refer to its documentation and usage guide.

In the scripts directory, you can create subdirectories to store files needed for UT verification. For example, the rvc_expander module creates a scripts/frontend_ifu_rvc_expander directory, where rtl_file.f specifies the input RTL file, and line_coverage.ignore stores lines of code to be ignored in coverage statistics. Custom directory names should be reasonable and should indicate the module and file they belong to.

Write the line_coverage_files(cfg) -> list[str] Function

The line_coverage_files function is defined as follows:

def line_coverage_files(cfg) -> list[str]:
    """Specify files to be covered
    Args:
        cfg: Runtime configuration, which can be used to access configuration items, e.g., cfg.rtl.version
    Return:
        Returns the names of RTL files targeted for line coverage statistics
    """

In the build_ut_frontend_ifu_rvc_expander.py file, the line_coverage_files function is defined as follows:

def line_coverage_files(cfg):
    return ["RVCExpander.v"]

This indicates that the module focuses on coverage for the RVCExpander.v file. If you want to enable test result processing, set disable=False under doc-result in configs/_default.yaml (the default parameter is False, meaning it is enabled). If you do not enable test result processing (disable=True), the above function will not be called.

2 - Build Test Environment

Determine Directory Structure

The directory structure of the Unit Test (UT) should match its naming convention. For example, frontend.ifu.rvc_expander should be located in the ut_frontend/ifu/rvc_expander directory, and each directory level must include an __init__.py file to enable Python imports.

The file for this chapter is your_module_wrapper.py (if your module is rvc_expander, the file would be rvc_expander_wrapper.py).

A wrapper is essentially a layer of abstraction that encapsulates the methods needed for testing into APIs decoupled from the DUT. These APIs are then used in test cases.

*Note: Decoupling ensures that test cases are independent of the DUT, allowing them to be written and debugged without needing to know the DUT’s implementation details. For more information, refer to Decoupling Verification Code from the DUT.

This file should be placed in the ut_frontend_or_backend/top_module/your_module/env directory. For example, if rvc_expander belongs to the frontend, its top-level directory should be ut_frontend. The next-level directory would be ifu, followed by rvc_expander. Since we are building the test environment, an additional env directory is created. The full path would be: ut_frontend_or_backend/top_module/your_module/env.

ut_frontend/ifu/rvc_expander
├── classical_version
│   ├── env
│   │   ├── __init__.py
│   │   └── rvc_expander_wrapper.py
│   ├── __init__.py
│   └── test_rvc_expander.py
├── __init__.py
├── README.md
└── toffee_version
    ├── agent
    │   └── __init__.py
    ├── bundle
    │   └── __init__.py
    ├── env
    │   ├── __init__.py
    │   └── ref_rvc_expand.py
    ├── __init__.py
    └── test
        ├── __init__.py
        ├── rvc_expander_fixture.py
        └── test_rvc.py

In the rvc_expander directory, there are two versions: classical_version (traditional) and toffee_version (using Toffee).
The traditional version uses the pytest framework for testing, while the Toffee version leverages more features of the Toffee framework.
In general, the traditional version is sufficient for most cases, and the Toffee version is only needed when the traditional version cannot meet the requirements.
When building the test environment, choose one version.
The directory structure within a module (e.g., rvc_expander) is determined by the contributor. You do not need to create additional classical_version or toffee_version directories, but the structure must comply with Python standards and be logically and consistently named.

Env Requirements

• Perform RTL version checks.
• The APIs provided by Env must be independent of pins and timing.
• The APIs provided by Env must be stable and should not undergo arbitrary changes in interfaces or return values.
• Define necessary fixtures.
• Initialize functional checkpoints (functional checkpoints can be independent modules).
• Perform coverage statistics.
• Include documentation.

Building the Test Environment: Traditional Version

In the test environment for the UT verification module, the goal is to accomplish the following:

1. Encapsulate DUT functionality to provide stable APIs for testing.
2. Define functional coverage.
3. Define necessary fixtures for test cases.
4. Collect coverage statistics at appropriate times.

Taking the RVCExpander in the IFU environment as an example (ut_frontend/ifu/rvc_expander/classical_version/env/rvc_expander_wrapper.py):

1. DUT Encapsulation

The following content is located in ut_frontend/ifu/rvc_expander/classical_version/env/rvc_expander_wrapper.py.

class RVCExpander(toffee.Bundle):
    def __init__(self, cover_group, **kwargs):
        super().__init__()
        self.cover_group = cover_group
        self.dut = DUTRVCExpander(**kwargs)                     # Create DUT
        self.io = toffee.Bundle.from_prefix("io_", self.dut)    # Bind pins using Bundle and prefix
        self.bind(self.dut)                                     # Bind Bundle to DUT

    def expand(self, instr, fsIsOff):
        self.io["in"].value = instr                                 # Assign value to DUT pin
        self.io["fsIsOff"].value = fsIsOff                          # Assign value to DUT pin
        self.dut.RefreshComb()                                      # Trigger combinational logic
        self.cover_group.sample()                                   # Collect functional coverage statistics
        return self.io["out_bits"].value, self.io["ill"].value      # Return result and illegal instruction flag

    def stat(self):  # Get current state
        return {
            "instr": self.io["in"].value,           # Input instruction
            "decode": self.io["out_bits"].value,    # Decoded result
            "illegal": self.io["ill"].value != 0,   # Whether the input is illegal
        }

In the example above, class RVCExpander encapsulates DUTRVCExpander and provides two APIs:

• expand(instr: int, fsIsOff: bool) -> (int, int): Accepts an input instruction instr for decoding and returns (result, illegal instruction flag). If the illegal instruction flag is non-zero, the input instruction is illegal.
• stat() -> dict(instr, decode, illegal): Returns the current state, including the input instruction, decoded result, and illegal instruction flag.

These APIs abstract away the DUT’s pins, exposing only general functionality to external programs.

2. Define Functional Coverage

Define functional coverage in the environment whenever possible. If necessary, coverage can also be defined in test cases. For details on defining functional coverage with Toffee, refer to What is Functional Coverage. To establish a clear relationship between functional checkpoints and test cases, functional coverage definitions should be linked to test cases (reverse marking).

The following content is located in ut_frontend/ifu/rvc_expander/classical_version/env/rvc_expander_wrapper.py.

import toffee.funcov as fc
# Create a functional coverage group
g = fc.CovGroup(UT_FCOV("../../../CLASSIC"))

def init_rvc_expander_funcov(expander, g: fc.CovGroup):
    """Add watch points to the RVCExpander module to collect functional coverage information"""

    # 1. Add point RVC_EXPAND_RET to check expander return value:
    #    - bin ERROR: The instruction is not illegal
    #    - bin SUCCE: The instruction is not expanded
    g.add_watch_point(expander, {
                                "ERROR": lambda x: x.stat()["illegal"] == False,
                                "SUCCE": lambda x: x.stat()["illegal"] != False,
                          }, name="RVC_EXPAND_RET")
    ...
    # 5. Reverse mark functional coverage to the checkpoint
    def _M(name):
        # Get the module name
        return module_name_with(name, "../../test_rv_decode")

    #  - Mark RVC_EXPAND_RET
    g.mark_function("RVC_EXPAND_RET", _M(["test_rvc_expand_16bit_full",
                                          "test_rvc_expand_32bit_full",
                                          "test_rvc_expand_32bit_randomN"]), bin_name=["ERROR", "SUCCE"])
    ...

In the code above, a functional checkpoint named RVC_EXPAND_RET is added to check whether the RVCExpander module can return illegal instructions. The checkpoint requires both ERROR and SUCCE conditions to be met, meaning the illegal field in stat() must have both True and False values. After defining the checkpoint, the mark_function method is used to link it to the relevant test cases.

3. Define Necessary Fixtures

The following content is located in ut_frontend/ifu/rvc_expander/classical_version/env/rvc_expander_wrapper.py.

version_check = get_version_checker("openxiangshan-kmh-*")                  # Specify the required RTL version
@pytest.fixture()
def rvc_expander(request):
    version_check()                                                         # Perform version check
    fname = request.node.name                                               # Get the name of the test case using this fixture
    wave_file = get_out_dir("decoder/rvc_expander_%s.fst" % fname)          # Set waveform file path
    coverage_file = get_out_dir("decoder/rvc_expander_%s.dat" % fname)      # Set code coverage file path
    coverage_dir = os.path.dirname(coverage_file)
    os.makedirs(coverage_dir, exist_ok=True)                                # Create directory if it doesn't exist
    expander = RVCExpander(g, coverage_filename=coverage_file, waveform_filename=wave_file)
                                                                            # Create RVCExpander
    expander.dut.io_in.AsImmWrite()                                         # Set immediate write timing for io_in pin
    expander.dut.io_fsIsOff.AsImmWrite()                                    # Set immediate write timing for io_fsIsOff pin
    init_rvc_expander_funcov(expander, g)                                   # Initialize functional checkpoints
    yield expander                                                          # Return the created RVCExpander to the test case
    expander.dut.Finish()                                                   # End DUT after the test case is executed
    set_line_coverage(request, coverage_file)                               # Report code coverage file to toffee-report
    set_func_coverage(request, g)                                           # Report functional coverage data to toffee-report
    g.clear()                                                               # Clear functional coverage statistics

This fixture accomplishes the following:

1. Performs RTL version checks. If the version does not meet the "openxiangshan-kmh-*" requirement, the test case using this fixture is skipped.
2. Creates the DUT and specifies the paths for waveform and code coverage files (the paths include the name of the test case using the fixture: fname).
3. Calls init_rvc_expander_funcov to add functional coverage points.
4. Ends the DUT and processes code and functional coverage (sending them to toffee-report for processing).
5. Clears functional coverage statistics.

*Note: In PyTest, before executing a test case like test_A(rvc_expander, ...), (rvc_expander is the method name we defined when we used the fixure decorator), the part of rvc_expander(request) before the yield keyword will be automatically called and executed (which is equivalent to initialization). and then rvc_expander will be returned to call the test_A case via yield (the object returned by yield is the method name we defined in our fixture of the test case). After the execution of the case is completed, then continue to execute the part of the fixture after the field keyword. For example: refer to the following code of statistical coverage, the penultimate line of rvc_expand(rvc_expander, generate_rvc_instructions(start, end)), where rvc_expander is the name of the method that we defined in the fixture, that is, the yield return object.

4. Collect Coverage Statistics

The following content is located in ut_frontend/ifu/rvc_expander/classical_version/test_rvc_expander.py.

N = 10
T = 1 << 16
@pytest.mark.toffee_tags(TAG_LONG_TIME_RUN)
@pytest.mark.parametrize("start,end",
                         [(r * (T // N), (r + 1) * (T // N) if r < N - 1 else T) for r in range(N)])
def test_rvc_expand_16bit_full(rvc_expander, start, end):
    """Test the RVC expand function with a full compressed instruction set

    Description:
        Perform an expand check on 16-bit compressed instructions within the range from 'start' to 'end'.
    """
    # Add checkpoint: RVC_EXPAND_RANGE to check expander input range.
    #   When run to here, the range[start, end] is covered
    covered = -1
    g.add_watch_point(rvc_expander, {
                                "RANGE[%d-%d]" % (start, end): lambda _: covered == end
                          }, name="RVC_EXPAND_ALL_16B", dynamic_bin=True)
    # Reverse mark function to the checkpoint
    g.mark_function("RVC_EXPAND_ALL_16B", test_rvc_expand_16bit_full, bin_name="RANGE[%d-%d]" % (start, end))
    # Drive the expander and check the result
    rvc_expand(rvc_expander, generate_rvc_instructions(start, end))
    # When go to here, the range[start, end] is covered
    covered = end
    g.sample()  # Sample coverage

After defining coverage, it must be collected in the test cases. In the code above, a functional checkpoint rvc_expander is added in the test case using add_watch_point. The checkpoint is then marked and sampled. Coverage sampling triggers a callback function to evaluate the bins defined in add_watch_point. If any bins’s condition evaluates to True, it is counted as a pass.

Building the Test Environment: Toffee Version

Testing with Python can be enhanced by using our open-source testing framework Toffee.

The official Toffee tutorial can be found here.

Bundle: Quick DUT Encapsulation

Toffee uses Bundles to bind to DUTs. It provides multiple methods for establishing Bundle-to-DUT bindings. Relevant code can be found in ut_frontend/ifu/rvc_expander/toffee_version/bundle.

Manual Binding

In the Toffee framework, the lowest-level class supporting pin binding is Signal, which binds to DUT pins using name matching. For example, consider the simplest RVCExpander with the following I/O pins:

module RVCExpander(
  input  [31:0] io_in,
  input         io_fsIsOff,
  output [31:0] io_out_bits,
  output        io_ill
);

There are four signals: io_in, io_fsIsOff, io_out_bits, and io_ill. A common prefix, such as io_, can be extracted (note that in cannot be used directly as a variable name in Python). The remaining parts can be defined as pin names in the corresponding Bundle class:

class RVCExpanderIOBundle(Bundle):
    _in, _fsIsOff, _out_bits, _ill = Signals(4)

In a higher-level Env or Bundle, the from_prefix method can be used to complete the prefix binding:

self.agent = RVCExpanderAgent(RVCExpanderIOBundle.from_prefix("io").bind(dut))

Automatic Bundle Definition

The Bundle class definition can also be omitted by using prefix binding:

self.io = toffee.Bundle.from_prefix("io_", self.dut)  # Bind pins using Bundle and prefix
self.bind(self.dut)

If the from_prefix method is passed a DUT, it automatically generates pin definitions based on the prefix and DUT pin names. Accessing the pins can then be done using a dictionary-like approach:

self.io["in"].value = instr
self.io["fsIsOff"].value = False

Bundle Code Generation

The Toffee framework’s scripts provide two scripts.

The bundle_code_gen.py script offers three methods:

def gen_bundle_code_from_dict(bundle_name: str, dut, dict: dict, max_width: int = 120)
def gen_bundle_code_from_prefix(bundle_name: str, dut, prefix: str = "", max_width: int = 120)
def gen_bundle_code_from_regex(bundle_name: str, dut, regex: str, max_width: int = 120)

These methods generate Bundle code by passing in a DUT and generation rules (dict, prefix, or regex).

The bundle_code_intel_gen.py script parses the signals.json file generated by Picker to automatically generate hierarchical Bundle code. It can be invoked from the command line:

python bundle_code_intel_gen.py [signal] [target]

If you encounter bugs in the auto-generation scripts, feel free to submit an issue for us to fix.

Agent: Driving Methods

If Bundles abstract the data responsibilities of a DUT, Agents encapsulate its behavioral responsibilities into interfaces. Simply put, an Agent provides multiple methods that abstract groups of I/O operations into specific behaviors:

class RVCExpanderAgent(Agent):
    def __init__(self, bundle: RVCExpanderIOBundle):
        super().__init__(bundle)
        self.bundle = bundle

    @driver_method()
    async def expand(self, instr, fsIsOff):             # Accepts RVC instruction and fs.status enable flag
        self.bundle._in.value = instr                   # Assign value to pin
        self.bundle._fsIsOff.value = fsIsOff            # Assign value to pin

        await self.bundle.step()                        # Trigger clock
        return self.bundle._out_bits.value,             # Return expanded instruction
               self.bundle._ill.value                   # Return legality check

For example, the RVCExpander’s instruction expansion function accepts an input instruction (which could be an RVI or RVC instruction) and the CSR’s enable flag for fs.status. This functionality is abstracted into the expand method, which takes two parameters in addition to self. The method returns the corresponding RVI instruction and a legality check for the input instruction.

Env: Test Environment

class RVCExpanderEnv(Env):
    def __init__(self, dut: DUTRVCExpander):
        super().__init__()
        dut.io_in.xdata.AsImmWrite()
        dut.io_fsIsOff.xdata.AsImmWrite()  # Set pin write timing
        self.agent = RVCExpanderAgent(RVCExpanderIOBundle.from_prefix("io").bind(dut))  # Complete prefix and bind DUT

Coverage Definition

The method for defining coverage groups is similar to the one described earlier and will not be repeated here.

Test Suite Definition

The definition of test suites differs slightly:

@toffee_test.fixture
async def rvc_expander(toffee_request: toffee_test.ToffeeRequest):
    import asyncio
    version_check()
    dut = toffee_request.create_dut(DUTRVCExpander)
    start_clock(dut)
    init_rvc_expander_funcov(dut, gr)

    toffee_request.add_cov_groups([gr])
    expander = RVCExpanderEnv(dut)
    yield expander

    cur_loop = asyncio.get_event_loop()
    for task in asyncio.all_tasks(cur_loop):
        if task.get_name() == "__clock_loop":
            task.cancel()
            try:
                await task
            except asyncio.CancelledError:
                break

Due to Toffee’s more powerful coverage management features, manual line coverage settings are not needed. Additionally, because of Toffee’s clock mechanism, it is recommended to check if all tasks have ended at the end of the suite code.

3 - Add Test Cases

Naming Requirements

All test case files should be named in the format test_*.py, where * is replaced with the test target (e.g., test_rvc_expander.py). All test cases should also start with the test_ prefix. The test case names must have clear and meaningful descriptions.

Examples of naming:

def test_a():  # Not acceptable, as "a" does not indicate the test target
    pass

def test_rvc_expand_16bit_full():  # Acceptable, as the name indicates the test content
    pass

Using Assert

Each test case must use assert to determine whether the test passes.
pytest relies on the results of assert statements, so these statements must ensure correctness.

The following content is located in ut_frontend/ifu/rvc_expander/classical_version/test_rvc_expander.py:

def rvc_expand(rvc_expander, ref_insts, is_32bit=False, fsIsOff=False):
    """Compare the RVC expand result with the reference

    Args:
        rvc_expander (wrapper): the fixture of the RVC expander
        ref_insts (list[int]): the reference instruction list
    """
    find_error = 0
    for insn in ref_insts:
        insn_disasm = disasmbly(insn)
        value, instr_ex = rvc_expander.expand(insn, fsIsOff)
        if is_32bit:
            assert value == insn, "RVC expand error, 32-bit instruction must remain unchanged"
        if (insn_disasm == "unknown") and (instr_ex == 0):
            debug(f"Found bad instruction: {insn}, ref: 1, dut: 0")
            find_error += 1
        elif (insn_disasm != "unknown") and (instr_ex == 1):
            if (instr_filter(insn_disasm) != 1):
                debug(f"Found bad instruction: {insn}, disasm: {insn_disasm}, ref: 0, dut: 1")
                find_error += 1
    assert find_error == 0, f"RVC expand error ({find_error} errors)"

Writing Comments

Each test case must include necessary explanations and comments, adhering to the Python Docstring Conventions.

Example format for test case documentation:

def test_<name>(a: type_a, b: type_b):
    """Test abstract

    Args:
        a (type_a): Description of argument a.
        b (type_b): Description of argument b.

    Detailed test description here (if needed).
    """
    ...

Test Case Management

To facilitate test case management, use the @pytest.mark.toffee_tags tag feature provided by toffee-test. Refer to the Other section of this site and the toffee-test documentation.

Reference Test Cases

If many test cases share the same operations, the common parts can be extracted into a utility function. For example, in RVCExpander verification, the comparison of compressed instruction expansion with the reference model (disasm) can be encapsulated into the following function:

The following content is located in ut_frontend/ifu/rvc_expander/classical_version/test_rvc_expander.py:

def rvc_expand(rvc_expander, ref_insts, is_32bit=False, fsIsOff=False):
    """Compare the RVC expand result with the reference

    Args:
        rvc_expander (wrapper): the fixture of the RVC expander
        ref_insts (list[int]): the reference instruction list
    """
    find_error = 0
    for insn in ref_insts:
        insn_disasm = disasmbly(insn)
        value, instr_ex = rvc_expander.expand(insn, fsIsOff)
        if is_32bit:
            assert value == insn, "RVC expand error, 32-bit instruction must remain unchanged"
        if (insn_disasm == "unknown") and (instr_ex == 0):
            debug(f"Found bad instruction: {insn}, ref: 1, dut: 0")
            find_error += 1
        elif (insn_disasm != "unknown") and (instr_ex == 1):
            if (instr_filter(insn_disasm) != 1):
                debug(f"Found bad instruction: {insn}, disasm: {insn_disasm}, ref: 0, dut: 1")
                find_error += 1
    assert find_error == 0, f"RVC expand error ({find_error} errors)"

The above utility function includes assert statements, so the test cases calling this function can also rely on these assertions to determine the results.

During test case development, debugging is often required. To quickly set up the verification environment, “smoke tests” can be written for debugging. For example, a smoke test for expanding 16-bit compressed instructions in RVCExpander is as follows:

@pytest.mark.toffee_tags(TAG_SMOKE)
def test_rvc_expand_16bit_smoke(rvc_expander):
    """Test the RVC expand function with 1 compressed instruction"""
    rvc_expand(rvc_expander, generate_rvc_instructions(start=100, end=101))

For easier management, the above test case is tagged with the SMOKE label using toffee_tags. Its input parameter is rvc_expander, which will automatically invoke the corresponding fixture with the same name during runtime.

The goal of testing 16-bit compressed instructions in RVCExpander is to traverse all 2^16 compressed instructions and verify that all cases match the reference model (disasm). If a single test is used for traversal, it would take a significant amount of time. To address this, we can use pytest’s parametrize feature to configure test parameters and execute them in parallel using the pytest-xdist plugin:

The following content is located in ut_frontend/ifu/rvc_expander/classical_version/test_rvc_expander.py:

N = 10
T = 1 << 16
@pytest.mark.toffee_tags(TAG_LONG_TIME_RUN)
@pytest.mark.parametrize("start,end",
                         [(r * (T // N), (r + 1) * (T // N) if r < N - 1 else T) for r in range(N)])
def test_rvc_expand_16bit_full(rvc_expander, start, end):
    """Test the RVC expand function with a full compressed instruction set

    Description:
        Perform an expand check on 16-bit compressed instructions within the range from 'start' to 'end'.
    """
    # Add checkpoint: RVC_EXPAND_RANGE to check expander input range.
    # When run to here, the range [start, end] is covered
    g.add_watch_point(rvc_expander, {
                                "RANGE[%d-%d]" % (start, end): lambda _: True
                          }, name="RVC_EXPAND_ALL_16B").sample()

    # Reverse mark function to the checkpoint
    g.mark_function("RVC_EXPAND_ALL_16B", test_rvc_expand_16bit_full, bin_name="RANGE[%d-%d]" % (start, end))

    # Drive the expander and check the result
    rvc_expand(rvc_expander, generate_rvc_instructions(start, end))

In the above test case, the parameters start and end are defined to specify the range of compressed instructions. These parameters are grouped and assigned using the @pytest.mark.parametrize decorator. The variable N specifies the number of groups for the target data, with a default of 10 groups. During runtime, the test case test_rvc_expand_16bit_full will expand into 10 test cases, such as test_rvc_expand_16bit_full[0-6553] to test_rvc_expand_16bit_full[58977-65536].

4 - Code Coverage

Code coverage is a metric that measures which parts of the tested code have been executed and which parts have not. By analyzing code coverage, the effectiveness and thoroughness of testing can be evaluated.

Code coverage includes:

• Line Coverage: The number of lines executed in the tested code. This is the simplest metric, and the goal is usually 100%.
• Branch Coverage: Whether each branch of every control structure has been executed. For example, in an if statement, have both the true and false branches been executed?
• FSM Coverage: Whether all states of a finite state machine have been reached.
• Toggle Coverage: Tracks the toggling of signals in the tested code, ensuring that every circuit node has both 0 -> 1 and 1 -> 0 transitions.
• Path Coverage: Examines the coverage of paths. In always or initial blocks, if ... else and case statements can create various data paths in the circuit structure.

* The primary simulator used in this project is Verilator, with a focus on line coverage. Verilator supports coverage statistics, so when building the DUT, the -c option must be added to the compilation options to enable coverage statistics.

Relevant Locations in This Project

To enable coverage, the -c option must be added during compilation (when using the picker command). Refer to the Picker Parameter Explanation. Additionally, the line coverage function must be implemented and enabled in the test files to generate coverage statistics during Toffee testing.

In conjunction with the above description, code coverage will be involved when compiling, writing and enabling line coverage functions and tests in this project:

Adding Compilation Scripts

Write the build(cfg) -> bool Function

# Omitted earlier code
    if not os.path.exists(get_root_dir("dut/RVCExpander")):
        info("Exporting RVCExpander.sv")
        s, out, err = exe_cmd(f'picker export --cp_lib false {get_rtl_dir("rtl/RVCExpander.sv", cfg=cfg)
                                                              } --lang python --tdir {get_root_dir("dut")}/ -w rvc.fst -c')
        assert s, "Failed to export RVCExpander.sv: %s\n%s" % (out, err)
# Omitted later code

In the line s, out, err=..., the picker command is used with the -c option to enable code coverage.

Set Target Coverage Files (line_coverage_files Function)

Write the line_coverage_files(cfg) -> list[str] function as needed, and enable test result processing (doc_result.disable = False) to ensure it is invoked.

Building the Test Environment

Define Necessary Fixtures

set_line_coverage(request, coverage_file)                          # Pass the generated code coverage file to toffee-report

Use the toffee-test.set_line_coverage function to pass the coverage file to Toffee-Test, enabling it to collect data for generating reports with line coverage.

Ignoring Specific Statistics

Sometimes, certain parts of the code may need to be excluded from coverage statistics. For example, some parts may not need to be tested, or it may be normal for certain parts to remain uncovered. Ignoring these parts can help optimize coverage reports or assist in debugging. Our framework supports two methods for ignoring coverage:

1. Using Verilator to Specify Ignored Sections

Using verilator_coverage_off/on Directives

Verilator supports ignoring specific code sections from coverage statistics using comment directives. For example:

// *verilator coverage_off*
// Code section to ignore
...
// *verilator coverage_on*

Example:

module example;
    always @(posedge clk) begin
        // *verilator coverage_off*
        if (debug_signal) begin
            $display("This is for debugging only");
        end
        // *verilator coverage_on*
        if (enable) begin
            do_something();
        end
    end
endmodule

In the above example, the debug_signal section will not be included in coverage statistics, while the enable section will still be counted.

For more ways to ignore coverage in Verilator, refer to the Verilator Documentation.

2. Using Toffee to Specify Filters

def set_line_coverage(request, datfile, ignore=[]):
    """Pass

    Args:
        request (pytest.Request): Pytest's default fixture.
        datfile (string): The coverage file generated by the DUT.
        ignore (list[str]): Coverage filter files or directories.
    """

The ignore parameter can specify content to be filtered out from the coverage file. For example:

...
set_line_coverage(request, coverage_file,
                  get_root_dir("scripts/frontend_ifu_rvc_expander"))

During coverage statistics, the line_coverage.ignore file in the scripts/frontend_ifu_rvc_expander directory will be searched, and its wildcard patterns will be used for filtering.

# Line coverage ignore file
# Ignore Top file
*/RVCExpander_top*%

The above file indicates that files containing the keyword RVCExpander_top will be ignored during coverage statistics (the corresponding data is collected but excluded from the final report).

Viewing Statistics Results

After completing all the steps, including preparing the test environment (Download RTL Code, Compile DUT, Edit Configuration), and adding tests (Add Compilation Scripts, Build Test Environment, Add Test Cases):

Now, Run Tests. Afterward, an HTML version of the test report will be generated in the out/report directory by default.

You can also view the statistics results by selecting the corresponding test report (named by test time) under “Current Version” in the Progress Overview section and clicking the link on the right.

5 - Functional Coverage

Functional Coverage is a user-defined metric used to measure the proportion of design specifications executed during verification. Functional coverage focuses on whether the features and functionalities of the design have been covered by the test cases.

Mapping refers to associating functional points with test cases. This allows you to see which test cases correspond to each functional point during statistics, making it easier to identify which functional points have more test cases and which have fewer. This helps optimize test cases in the later stages.

Relevant Locations in This Project

Functional coverage must be defined before it can be collected, primarily during the process of building the test environment.

In Building the Test Environment:

• Define Functional Coverage: Create functional coverage groups, add watch points, and map them.
• Define Necessary Fixtures: Pass the collected results to toffee-report.
• Collect Coverage: Add watch points and mappings.

Other:

• Functional points can also be written in each test case for use in test cases.

Functional Coverage Workflow

Specify Group Name

The test report matches the Group name with the DUT name. Use comm.UT_FCOV to obtain the DUT prefix. For example, in the Python module ut_frontend/ifu/rvc_expander/classical_version/env/rvc_expander_wrapper.py, the following call is made:

from comm import UT_FCOV
# Module name: ut_frontend.ifu.rvc_expander.classical_version.env.rvc_expander_wrapper
# Remove classical_version and the parent module env, rvc_expander_wrapper using ../../../
# UT_FCOV will automatically remove the prefix ut_
g = fc.CovGroup(UT_FCOV("../../../CLASSIC"))
# name = UT_FCOV("../../../CLASSIC")

The value of name is frontend.ifu.rvc_expander.CLASSIC. When collecting the final results, the longest prefix will be matched to the target UT (i.e., matched to the frontend.ifu.rvc_expander module).

Create Coverage Group

Use toffee’s funcov to create a coverage group.

import toffee.funcov as fc
# Use the GROUP name specified above
g = fc.CovGroup(name)

These two steps can also be combined into one: g = fc.CovGroup(UT_FCOV("../../../CLASSIC")).
The created g object represents a functional coverage group, which can be used to provide watch points and mappings.

Add Watch Points and Mappings

Inside each test case, you can use add_watch_point (or its alias add_cover_point, which is identical) to add watch points and mark_function to add mappings.
A watch point is triggered when the signal meets the conditions defined in the watch point, and its name (i.e., the functional point) will be recorded in the functional coverage.
A mapping associates functional points with test cases, allowing you to see which test cases correspond to each functional point during statistics.

The location of the watch point depends on the actual situation. Generally, adding watch points outside the test case is acceptable. However, sometimes more flexibility is required.

1. Outside the test case (in decode_wrapper.py):

def init_rvc_expander_funcov(expander, g: fc.CovGroup):
    """Add watch points to the RVCExpander module to collect functional coverage information"""
    # 1. Add point RVC_EXPAND_RET to check expander return value:
    #    - bin ERROR: The instruction is not illegal
    #    - bin SUCCE: The instruction is not expanded
    g.add_watch_point(expander, {
                                "ERROR": lambda x: x.stat()["ilegal"] == False,
                                "SUCCE": lambda x: x.stat()["ilegal"] != False,
                          }, name="RVC_EXPAND_RET")
    # 5. Reverse mark function coverage to the check point
    def _M(name):
        # Get the module name
        return module_name_with(name, "../../test_rv_decode")

    #  - mark RVC_EXPAND_RET
    g.mark_function("RVC_EXPAND_RET", _M(["test_rvc_expand_16bit_full",
                                          "test_rvc_expand_32bit_full",
                                          "test_rvc_expand_32bit_randomN"]), bin_name=["ERROR", "SUCCE"])

    # The End
    return None

In this example, the first g.add_watch_point is placed outside the test case because it is not directly related to the existing test cases. Placing it outside the test case is more convenient. Once the conditions in the bins of the add_watch_point method are triggered, the toffee-test framework will collect the corresponding functional points.

2. Inside the test case (in test_rvc_expander.py):

N = 10
T = 1 << 32
@pytest.mark.toffee_tags([TAG_LONG_TIME_RUN, TAG_RARELY_USED])
@pytest.mark.parametrize("start,end",
                         [(r * (T // N), (r + 1) * (T // N) if r < N - 1 else T) for r in range(N)])
def test_rvc_expand_32bit_full(rvc_expander, start, end):
    """Test the RVC expand function with a full 32-bit instruction set

    Description:
        Randomly generate N 32-bit instructions for each check, and repeat the process K times.
    """
    # Add check point: RVC_EXPAND_ALL_32B to check instr bits.
    covered = -1
    g.add_watch_point(rvc_expander, {"RANGE[%d-%d]" % (start, end): lambda _: covered == end},
                      name="RVC_EXPAND_ALL_32B", dynamic_bin=True)
    # Reverse mark function to the check point
    g.mark_function("RVC_EXPAND_ALL_32B", test_rvc_expand_32bit_full)
    # Drive the expander and check the result
    rvc_expand(rvc_expander, list([_ for _ in range(start, end)]))
    # When reaching here, the range [start, end] is covered
    covered = end
    g.sample()

In this example, the watch point is inside the test case because start and end are determined by pytest.mark.parametrize. Since the values are not fixed, the watch point needs to be added inside the test case.

Sampling

At the end of the previous example, we called g.sample(). This function notifies toffee-test that the bins in add_watch_point have been executed. If the conditions are met, the watch point is recorded as a pass.

There is also an automatic sampling option. During the test environment setup, you can add StepRis(lambda x: g.sample()) in the fixture definition. This will automatically sample at the rising edge of each clock cycle.

The following content is from ut_backend/ctrl_block/decode/env/decode_wrapper.py:

@pytest.fixture()
def decoder(request):
    # Before test
    init_rv_decoder_funcov(g)
    func_name = request.node.name
    # If the output directory does not exist, create it
    output_dir_path = get_out_dir("decoder/log")
    os.makedirs(output_dir_path, exist_ok=True)
    decoder = Decode(DUTDecodeStage(
        waveform_filename=get_out_dir("decoder/decode_%s.fst" % func_name),
        coverage_filename=get_out_dir("decoder/decode_%s.dat" % func_name),
    ))
    decoder.dut.InitClock("clock")
    decoder.dut.StepRis(lambda x: g.sample())
    yield decoder
    # After test
    decoder.dut.Finish()
    coverage_file = get_out_dir("decoder/decode_%s.dat" % func_name)
    if not os.path.exists(coverage_file):
        raise FileNotFoundError(f"File not found: {coverage_file}")
    set_line_coverage(request, coverage_file, get_root_dir("scripts/backend_ctrlblock_decode"))
    set_func_coverage(request, g)
    g.clear()

As shown above, we call g.sample() before yield, enabling automatic sampling at the rising edge of each clock cycle.

The StepRis function executes the passed function at the rising edge of each clock cycle. For more details, refer to the Picker Usage Guide.