Boost C++ SDK Quality: Implement Code Coverage

by Editorial Team 47 views
Iklan Headers

Hey everyone! 👋 Let's dive into a crucial aspect of software development: code coverage for the C++ SDK. Ensuring that our code is well-tested is super important for maintaining quality, preventing bugs, and making sure everything works as expected. This guide is designed for experienced contributors, so if you're comfortable navigating unfamiliar systems, get ready to roll up your sleeves and improve our C++ SDK! We'll be using tools like gcov, lcov, and services like Codecov to get the job done.

🐞 The Problem: Untested Code Paths

Currently, the C++ SDK lacks code coverage statistics. This absence makes it difficult to pinpoint untested code paths. Without this information, we risk releasing code that hasn't been adequately tested, which can lead to:

  • Increased Bugs: Untested code is a breeding ground for bugs. Without tests to catch errors, bugs can slip through and cause problems for users.
  • Difficulty in Tracking Test Coverage Trends: It's hard to tell if your test coverage is improving or getting worse over time.
  • Reduced Code Quality: Lack of coverage metrics makes it challenging to maintain high code quality standards. It's difficult to identify which parts of the code need more testing or refactoring.

Current State of the SDK

  • Tests Run Via CTest: Our tests are currently executed using CTest in our CI pipeline.
  • No Coverage Instrumentation: We're not currently generating coverage data during test runs.
  • No Coverage Reporting: There are no reports generated to show test coverage.
  • No Visibility into Metrics: We lack visibility into test coverage metrics. We don't know how much of our code is actually covered by tests.

💡 Expected Outcome: Integrated Code Coverage

Our goal is to integrate code coverage into the C++ SDK's build and CI pipeline. This will provide us with the tools to track and improve our test coverage.

Goals of this Integration:

  1. Generate Coverage Data: Generate code coverage data during test execution. This data will tell us which parts of the code are being executed by our tests.
  2. Upload Reports: Upload coverage reports to a reporting service like Codecov or Coveralls. These services provide a user-friendly way to view and analyze our coverage data.
  3. Display Metrics in PR Checks: Show coverage metrics in PR checks. This will help us ensure that new code is adequately tested before it's merged.
  4. Track Trends: Track coverage trends over time. This will help us monitor the effectiveness of our testing efforts and identify areas where we need to improve.

What we are NOT doing (Non-goals):

  • Enforcing Thresholds: We're not immediately enforcing minimum coverage thresholds. This can be added later as part of a more comprehensive testing strategy.
  • Coverage for all Platforms: We'll initially focus on Linux coverage, as it's sufficient for our immediate needs.

🧠 Design Considerations: Key Decisions

Implementing code coverage involves several architectural choices. Let's explore these design considerations to ensure a successful integration.

1. Choosing the Right Coverage Tool

We have a few options for C++ coverage tooling. Let's explore each one.

  • gcov + lcov: This combination is the most common for GCC-based builds. gcov is GCC's native coverage tool, and lcov generates HTML reports. This is a solid and reliable choice.
  • llvm-cov: LLVM/Clang's coverage tool offers similar functionality. It's a good option if you're using Clang as your compiler.
  • gcovr: A Python-based tool that can generate coverage reports from gcov data. It offers more advanced reporting features.

Recommendation: For this project, we recommend sticking with gcov + lcov. It's well-established, works seamlessly with GCC, and will get the job done efficiently.

2. CMake Integration: Building with Coverage

We'll need to add a coverage build option to CMakeLists.txt. This will allow us to build the SDK with the necessary instrumentation for code coverage. Here's how we can do it:

option(CODE_COVERAGE "Enable code coverage instrumentation" OFF)

if(CODE_COVERAGE AND CMAKE_CXX_COMPILER_ID MATCHES "GNU|Clang")
  add_compile_options(--coverage -O0 -g)
  add_link_options(--coverage)
endif()

Things to consider:

  • Debug Builds: Coverage should be enabled only for Debug builds. This is because coverage instrumentation can impact performance.
  • Performance Impact: Coverage instrumentation can slow down the build process and test execution. We need to be mindful of this impact and make sure it doesn't significantly affect our CI pipeline.
  • Separate CMake Preset: We might need a separate CMake preset for coverage builds to keep things organized.

3. CI Integration Strategy: Integrating Coverage into Our Pipeline

We have a few options for integrating code coverage into our CI (Continuous Integration) pipeline. Let's consider the pros and cons of each one:

  • Separate Coverage Job: A dedicated workflow for coverage generation and reporting. This keeps the coverage steps separate from the main build and test process.
  • Extend Existing Build: Add a coverage step to the current CI workflow. This integrates coverage directly into the existing build process.
  • Coverage-Only on Merge: Run coverage checks only when merging pull requests. This reduces the overhead on pull requests but might delay feedback.

Our Current CI Workflow:

Currently, our CI pipeline (zxc-build-library.yaml) performs these steps:

  1. CMake configure/build
  2. Solo network setup
  3. CTest execution

Coverage Steps:

We will need to add the following steps to the current workflow to integrate code coverage:

  1. Build with coverage flags enabled.
  2. Run tests. This generates .gcda files containing coverage data.
  3. Process the data using lcov to generate a coverage report.
  4. Upload the report to a service like Codecov or Coveralls.

4. Reporting Service: Choosing a Platform

We need a reporting service to visualize and track our code coverage. Here are a few popular options:

  • Codecov: A widely used service with excellent GitHub integration. It's free for open-source projects and provides detailed coverage reports.
  • Coveralls: Similar to Codecov, Coveralls offers comparable features and is another viable option.
  • SonarCloud: A more comprehensive analysis platform that includes code coverage, static analysis, and other quality metrics.

Recommendation: Codecov is a great choice due to its popularity, ease of use, and integration with GitHub. It's a solid choice for our needs.

5. Defining Coverage Scope: What to Cover?

We need to decide which parts of our codebase to include in the coverage analysis. Here are some key decisions:

  • Unit Tests vs. Integration Tests: Should we include both unit and integration tests, or focus on unit tests initially?
  • Excluding Dependencies: Should we exclude third-party code, such as vcpkg dependencies, to avoid skewing our coverage metrics?
  • Generated Code: Should we exclude generated code, such as protobuf code, as it's often not directly written by developers?

📂 Relevant Files: Where the Magic Happens

Here's a look at the files you'll be working with:

  • CMakeLists.txt: The main CMake configuration file. This is where we'll add the coverage build option and configure the build process.
  • CMakePresets.json: This file contains build presets, which help streamline the build configuration process.
  • .github/workflows/flow-pull-request-checks.yaml: This workflow handles PR checks. We'll need to modify this to include coverage reporting.
  • .github/workflows/zxc-build-library.yaml: This file defines the build and test pipeline. We'll add the coverage steps here.
  • src/sdk/tests/unit/: This directory contains the unit tests.
  • src/sdk/tests/integration/: This directory contains the integration tests.

✅ Acceptance Criteria: What Success Looks Like

To consider this task complete, we need to meet the following criteria:

  • CMake Option: A CMake option should be added to enable coverage instrumentation.
  • Coverage Data: Coverage data should be generated during test execution.
  • Reports Uploaded: Coverage reports should be uploaded to a reporting service.
  • Metrics in PRs: Coverage badges and metrics should be visible in pull requests.
  • Exclusions: Generated and third-party code should be excluded from coverage reports.
  • Documentation: Documentation should be updated with coverage information.
  • CI Performance: The execution time of the CI pipeline should remain reasonable.

📋 Contribution Guide: How to Contribute

To make your contribution as smooth as possible, follow these steps:

  1. Comment /assign: Comment /assign to request the issue.
  2. Wait for Assignment: Wait for assignment before you start working.
  3. Fork and Branch: Fork the repository and create a branch for your changes.
  4. Propose a Design: Before you start implementing, propose a design in the PR description.
  5. Implement Changes: Implement the CMake changes and CI workflow updates.
  6. Test Locally: Test coverage generation locally to ensure everything works.
  7. Sign Commits: Sign each commit using -s -S to ensure commit integrity.
  8. Open a Pull Request: Push your branch and open a pull request.

Important: Pull requests cannot be merged without signed commits (-S and -s).

For more detailed guidance, check out these resources:

  • Workflow Guide: Check out the Workflow Guide for step-by-step guidance.
  • Setup Instructions: Find setup instructions in the README.md.
  • Signing Guide: Consult the Signing Guide for details on signing your commits.

📚 Additional Context and Resources: Get More Info

Here are some helpful resources to get you started:

# Coverage configuration
option(CODE_COVERAGE "Enable code coverage" OFF)

if(CODE_COVERAGE)
  if(CMAKE_CXX_COMPILER_ID MATCHES "GNU|Clang")
    message(STATUS "Code coverage enabled")
    add_compile_options(--coverage -O0 -g -fprofile-arcs -ftest-coverage)
    add_link_options(--coverage)
  else()
    message(WARNING "Code coverage requires GCC or Clang")
  endif()
endif()
  • Example CI step:
- name: Generate Coverage Report
  run: |
    lcov --capture --directory . --output-file coverage.info
    lcov --remove coverage.info '/usr/*' '*/vcpkg/*' '*.pb.*' --output-file coverage.info

- name: Upload to Codecov
  uses: codecov/codecov-action@v4
  with:
    files: coverage.info
    fail_ci_if_error: true

If you have any questions, don't hesitate to reach out! The community is always happy to help. You can find us here:

Let's work together to boost the quality of our C++ SDK! 🚀