CTBase.jl

Appearance

Test Runner Guide ​

This guide explains how to set up a modular testing infrastructure for Julia packages using the TestRunner extension of CTBase.jl. The entry point is CTBase.DevTools.run_tests, activated by loading the Test weak dependency. This setup enables granular test execution and is friendly both for human developers and AI agents.

Architecture Overview ​

A robust testing architecture typically involves:

  • Test Runner: A runtests.jl file that allows running specific test groups via command-line arguments.

  • Test Suite Structure: Modular test files, each containing a main entry point function.

We recommend placing your tests in a suite subdirectory to keep the top-level test/ folder clean.

text
MyPackage.jl/
├── src/
│   └── ...
├── test/
│   ├── runtests.jl          # Main test runner
│   └── suite/               # Directory containing test files
│       ├── test_utils.jl
│       ├── test_core.jl
│       └── ...
└── ...

Restriction

The test/ directory must not contain a subdirectory named test. This would conflict with the automatic test/ prefix stripping (see Path prefix stripping).

Setting up runtests.jl ​

The runtests.jl file is the entry point for your test suite. By using CTBase.DevTools.run_tests, you enable a powerful mechanism to filter and execute specific tests using command-line arguments. This is crucial for fast iteration cycles.

Example test/runtests.jl ​

julia
# Load dependencies
using Test: Test
using CTBase: CTBase # Provides run_tests
using MyPackage: MyPackage # Your package

# Define where your tests are located
const TEST_DIR = @__DIR__

# Run tests using the CTBase test runner
CTBase.DevTools.run_tests(;
    args=String.(ARGS),                 # Pass command line arguments
    testset_name="MyPackage Tests",     # Name of the main testset
    available_tests=[                   # Available test groups
        "suite/*"                       # Glob: all tests in suite/
    ],
    excluded_tests=[                    # Tests removed from the available groups
        "suite/slow/*"
    ],
    filename_builder = name -> "test_$(name).jl",
    funcname_builder = name -> "test_$(name)",
    test_dir=TEST_DIR,                  # Directory containing test files
)

Keyword Arguments ​

ArgumentDefaultDescription
args::AbstractVector{<:AbstractString}String[]Command-line arguments (typically String.(ARGS))
testset_name::String"Tests"Name of the main @testset
available_tests::VectorSymbol[]Allowed tests (Symbols, Strings, or glob patterns). Empty = auto-discovery
excluded_tests::VectorSymbol[]Tests removed from the available tests (Symbols, Strings, or glob patterns)
filename_builder::Functionidentityname → filename mapping
funcname_builder::Functionidentityname → function_name mapping (return nothing to skip eval)
eval_mode::BooltrueWhether to call the function after include
verbose::BooltrueVerbose @testset output
showtiming::BooltrueShow timing in @testset output
test_dir::Stringjoinpath(pwd(), "test")Root directory for test files
on_test_start::Function or NothingnothingCallback before eval (see Callbacks)
on_test_done::Function or NothingnothingCallback after eval (see Callbacks)
show_progress_line::BooltrueShow progress line with symbol, index, spec, and time. Ignored when on_test_done is provided
show_progress_bar::BooltrueShow graphical progress bar [█░░░...] within the line
progress_bar_threshold::Int100Maximum tests for full-resolution progress bar (see Progress Bar)

Writing Test Files ​

To support the modular execution model, each test file wraps its tests in a module (so that helpers and fake types never leak between files) and exposes one entry-point function at the top level, named as funcname_builder expects. run_tests includes the file and then calls that function.

Example test/suite/test_utils.jl ​

julia
module TestUtils

using Test: Test
using MyPackage: MyPackage

function test_utils()
    Test.@testset "Utilities" begin
        Test.@test MyPackage.add(1, 1) == 2
        Test.@test MyPackage.sub(2, 1) == 1
    end
end

end # module

# Entry point called by run_tests (matches `funcname_builder` in runtests.jl)
test_utils() = TestUtils.test_utils()

Use a unique module name per file (e.g. derived from the file name): including two files that define the same module triggers WARNING: replacing module.

Running Tests ​

Default (all available tests) ​

bash
julia --project -e 'using Pkg: Pkg; Pkg.test("MyPackage")'

Run a specific test group ​

bash
julia --project -e 'using Pkg: Pkg; Pkg.test("MyPackage"; test_args=["utils"])'
julia --project -e 'using Pkg: Pkg; Pkg.test("MyPackage";
    test_args=["core", "utils"])'

Exclude tests ​

excluded_tests is applied after available_tests (or auto-discovery) and before command-line selections. It accepts symbols, paths, and glob patterns:

julia
CTBase.DevTools.run_tests(;
    available_tests=["suite/*"],
    excluded_tests=["suite/slow/*"],
    args=["--all"],
)

When available_tests is empty, tests are auto-discovered first and then filtered by excluded_tests.

Path prefix stripping ​

Selection arguments starting with test/ are automatically stripped, so the following are equivalent:

bash
# These two commands run the same tests:
julia --project -e 'using Pkg: Pkg; Pkg.test("MyPackage"; test_args=["suite"])'
julia --project -e 'using Pkg: Pkg; Pkg.test("MyPackage";
    test_args=["test/suite"])'

This is convenient when tab-completing paths from the project root.

CLI flags ​

  • -a or --all — run all available tests (same as no arguments).

  • -n or --dryrun — print the list of tests that would run, without executing them.

  • Coverage flags (coverage=true, --coverage, etc.) are silently filtered out.

Directory selection ​

Bare directory names are automatically expanded. For example, suite/exceptions is treated as suite/exceptions/*, selecting all test files in that directory.

Progress Bar ​

By default, run_tests displays a progress bar after each test completes. The bar uses a cursor-style display: only the current test position is highlighted, past successes are cleared:

text
[░░░░░░░█░░░░░░░░░░░] ✓ [08/19] suite/exceptions/test_display.jl (2.5s)
[░░░░░░░░█░░░░░░░░░░] ✓ [09/19] suite/exceptions/test_exceptions.jl (0.6s)
[░░░░░░░░░▚░░░░░░░░░] ✗ [10/19] suite/exceptions/test_types.jl FAILED, (0.9s)

To hide the graphical bar while keeping the progress line, set show_progress_bar=false:

julia
CTBase.DevTools.run_tests(; args=String.(ARGS), show_progress_bar=false)
text
✓ [08/19] suite/exceptions/test_display.jl (2.5s)
✓ [09/19] suite/exceptions/test_exceptions.jl (0.6s)
✗ [10/19] suite/exceptions/test_types.jl FAILED, (0.9s)

Visual elements ​

  • Bar: █ (success), ▚ (failure), ┆ (skip), and ░ (empty), enclosed in […]

  • Status symbol: ✓ (green, success), ✗ (red, failure/error), ○ (yellow, skipped)

  • Index: zero-padded [01/19] for alignment

  • Spec: test identifier (relative path or symbol name)

  • Time: wall-clock elapsed time for the eval phase

  • FAILED: appended in bold red when the test failed

Adaptive bar width ​

The bar width adapts to the number of tests:

  • ≤ 100 tests (default): width equals the total number of tests (one block per test).

  • > 100 tests (default): fixed width of 100 characters. Some tests will not visually advance the cursor (position computed as round(Int, index / total * progress_bar_threshold)).

The threshold can be customized via the progress_bar_threshold parameter:

julia
# Use a smaller threshold for narrow terminals
CTBase.DevTools.run_tests(; args=String.(ARGS), progress_bar_threshold=30)

# Use a larger threshold for wide displays
CTBase.DevTools.run_tests(; args=String.(ARGS), progress_bar_threshold=100)

Cursor-style display ​

  • Full-resolution mode (total ≤ threshold): Only the current test position is filled for successes. Failures and skips persist at their original positions. This creates a cursor-like effect where past successes are ephemeral but errors remain visible.

Example with 6 tests (2 success, 2 failure, 2 success):

text
[█░░░░░] ✓ [01/06] suite/test1.jl (0.2s)
[░█░░░░] ✓ [02/06] suite/test2.jl (0.7s)
[░░▚░░░] ✗ [03/06] suite/test3.jl FAILED, (1.2s)
[░░▚▚░░] ✗ [04/06] suite/test4.jl FAILED, (0.5s)
[░░▚▚█░] ✓ [05/06] suite/test5.jl (0.3s)
[░░▚▚░█] ✓ [06/06] suite/test6.jl (0.4s)
  • Compressed mode (total > threshold): A single cursor block moves across the bar as tests complete. The cursor shows the current test's status (█ for success, ▚ for failure, ┆ for skip). Failures do not persist. When total greatly exceeds threshold, multiple consecutive tests may share the same cursor position.

Example with 11 tests and threshold 10:

text
[█░░░░░░░░░] ✓ [01/11] suite/test1.jl (0.2s)
[░█░░░░░░░░] ✓ [02/11] suite/test2.jl (0.7s)
[░░▚░░░░░░░] ✗ [03/11] suite/test3.jl FAILED, (1.2s)
[░░░▚░░░░░░] ✗ [04/11] suite/test4.jl FAILED, (0.5s)
[░░░░█░░░░░] ✓ [05/11] suite/test5.jl (0.3s)
[░░░░█░░░░░] ✓ [06/11] suite/test6.jl (0.4s)

Failure detection ​

The progress bar correctly detects both types of failures:

  • Exceptions: errors thrown during test execution (caught via try/catch).

  • @test assertion failures: detected by scanning the enclosing DefaultTestSet results before and after eval. This is more reliable than checking anynonpass, which is only updated when a testset finishes.

Disabling the progress display ​

Set show_progress_line=false to disable the entire progress line:

julia
CTBase.DevTools.run_tests(; args=String.(ARGS), show_progress_line=false)

To display minimal output without the graphical bar, set show_progress_bar=false:

julia
CTBase.DevTools.run_tests(; args=String.(ARGS), show_progress_bar=false)
# Output: ✓ [01/76] suite/test.jl (0.2s)

The progress line is also automatically disabled when a custom on_test_done callback is provided.

Callbacks ​

The on_test_start and on_test_done callbacks allow custom actions during the test lifecycle. Both receive a TestRunInfo struct with the following fields:

TestRunInfo ​

julia
struct TestRunInfo
    spec::Union{Symbol,String}           # Test identifier
    filename::String                     # Absolute path of the test file
    func_symbol::Union{Symbol,Nothing}   # Function to call
                                          # (nothing if eval_mode=false)
    index::Int                           # 1-based index
    total::Int                           # Total number of selected tests
    status::Symbol                       # See below
    error::Union{Exception,Nothing}      # Captured exception
                                          # (only when status == :error)
    elapsed::Union{Float64,Nothing}      # Wall-clock seconds
                                          # (only in on_test_done)
end

Status values ​

StatusWhenCallback
:pre_evalAfter include, before evalon_test_start
:post_evalAfter successful evalon_test_done
:test_failedAfter eval with @test failures (no exception)on_test_done
:errorAfter eval raised an exceptionon_test_done
:skippedWhen eval_mode=false or on_test_start returned falseon_test_done

on_test_start ​

Called after the test file is included but before the function is evaluated. Must return a Bool:

  • true — proceed with eval.

  • false — skip eval (triggers on_test_done with :skipped).

on_test_done ​

Called after eval completes (or after skip/error). The built-in progress bar is implemented as a default on_test_done callback.

Example: custom callbacks ​

julia
CTBase.DevTools.run_tests(;
    args=String.(ARGS),
    on_test_start = info -> begin
        print("  [$(info.index)/$(info.total)] $(info.spec)...")
        return true  # proceed with eval
    end,
    on_test_done = info -> begin
        if info.status == :post_eval
            println(" ✓ ($(round(info.elapsed; digits=1))s)")
        elseif info.status == :error || info.status == :test_failed
            println(" ✗ FAILED")
        elseif info.status == :skipped
            println(" ○ skipped")
        end
    end,
)

Advanced Usage ​

Filtering Tests with Glob Patterns ​

You can use glob patterns to organize tests hierarchically:

julia
CTBase.DevTools.run_tests(;
    args=String.(ARGS),
    testset_name="MyPackage Tests",
    available_tests=[
        "suite/core/*",       # All core tests
        "suite/utils/*",      # All utility tests
        "suite/integration/*" # All integration tests
    ],
    # ...
)

Selection arguments are matched against multiple representations of each candidate:

  • The candidate name (e.g. :utils)

  • The full relative path (e.g. suite/test_utils.jl)

  • The path without .jl extension

  • The basename (e.g. test_utils.jl)

  • The basename without extension

  • The basename without test_ prefix (e.g. utils)

Shared Test Settings ​

To share settings such as verbose/showtiming with every test file, define them in a small module of runtests.jl and read them from the test modules — this is what CTBase's own test suite does:

julia
# In runtests.jl
module TestData
const VERBOSE = true
const SHOWTIMING = true
end

# In a test module
const VERBOSE = isdefined(Main, :TestData) ? Main.TestData.VERBOSE : true
const SHOWTIMING = isdefined(Main, :TestData) ? Main.TestData.SHOWTIMING : true

function test_utils()
    Test.@testset verbose = VERBOSE showtiming = SHOWTIMING "Utilities" begin
        # tests here
    end
end

The fallback values keep a test file runnable on its own, outside run_tests.

Debugging Test Failures ​

Common Issues and Solutions ​

Issue: Test function not found ​

Error: UndefVarError: test_utils not defined

Solution: Ensure your test file exposes the entry point outside its module:

julia
# At the end of test/suite/test_utils.jl
test_utils() = TestUtils.test_utils()

Issue: Module conflicts ​

Error: WARNING: replacing module TestUtils

Solution: Two test files define the same module name — give each file its own (see Writing Test Files).

Issue: Tests not discovered ​

Error: No tests run when specifying a test name

Solution: Check that your filename_builder and funcname_builder match your file structure:

julia
# If your files are named "utils_test.jl"
filename_builder = name -> "$(name)_test.jl"

# If your functions are named "run_utils_tests"
funcname_builder = name -> "run_$(name)_tests"

Issue: @test failure shown as success ​

The progress bar detects @test failures by scanning the enclosing testset results before and after eval. If you see a ✓ for a test that should have failed, make sure the test function is wrapped in a @testset block so that failures are recorded in the results.

Listing the selected tests ​

To check which files a selection resolves to without running them, use the dry-run flag:

bash
julia --project -e 'using Pkg: Pkg; Pkg.test("MyPackage"; test_args=["utils", "--dryrun"])'

Best Practices ​

  • One test function per file: Keep test files focused and easy to navigate

  • Use descriptive names: Name test files and functions clearly (e.g., test_optimization.jl, test_optimization())

  • Organize by feature: Group related tests in subdirectories

  • Fast tests first: Place quick unit tests before slow integration tests

  • Isolate test state: Each test should be independent and not rely on execution order

  • Use test fixtures: Create helper functions for common test setup

  • Document test requirements: Note any special dependencies or setup needed

  • No test/ subdirectory in test/: Avoid naming a subdirectory test inside your test directory

Integration with CI/CD ​

CI simply calls Pkg.test(): runtests.jl receives no arguments and runs every available test. In the control-toolbox ecosystem this is done by the centralized reusable workflows — see the CTActions workflows guide.

See Also ​