Private API

struct TestRunInfo

Context information passed to test callbacks (on_test_start, on_test_done).

Provides details about the current test being executed, including progress information (index, total) and execution results (status, error, elapsed).

Fields

• spec::TestSpec: test identifier (Symbol or relative path String)
• filename::String: absolute path of the included test file
• func_symbol::Union{Symbol,Nothing}: function to call (nothing if eval_mode=false)
• index::Int: 1-based index of the current test in the selected list
• total::Int: total number of selected tests
• status::Symbol: one of :pre_eval, :post_eval, :skipped, :error, :test_failed
• error::Union{Exception,Nothing}: captured exception when status == :error
• elapsed::Union{Float64,Nothing}: wall-clock seconds for the eval phase (only in on_test_done)

Example

julia> using CTBase.TestRunner

julia> info = TestRunner.TestRunInfo(
           :utils, 
           "/path/to/test_utils.jl", 
           :test_utils, 
           3, 10, 
           :post_eval, 
           nothing, 
           1.23
       )
TestRunner.TestRunInfo(:utils, "/path/to/test_utils.jl", :test_utils, 3, 10, :post_eval, nothing, 1.23)

julia> info.status
:post_eval

julia> info.elapsed
1.23

Union type representing a test specification.

A test spec can be either:

• Symbol: A logical test name (e.g., :utils, :core)
• String: A relative file path or glob pattern (e.g., "suite/test_utils.jl", "suite/core/*")

This type is used throughout TestRunner to represent both user-provided selections and internal test identifiers.

Notes

• Symbol specs are resolved via filename_builder and funcname_builder
• String specs are treated as relative paths from test_dir
• Glob patterns are supported for String specs

See also: CTBase.run_tests, TestRunner._select_tests

_FULL_BAR_THRESHOLD

Internal constant defining the maximum number of tests for full-resolution progress bars.

When the total number of tests is ≤ _FULL_BAR_THRESHOLD (50), the progress bar displays one character per test with cumulative coloring (each test gets its own colored block). Beyond this threshold, the bar switches to compressed mode with uniform coloring.

This threshold balances visual clarity with terminal width constraints.

_bar_width(total::Int64) -> Int64

Compute the progress bar character width based on the number of tests.

• total ≤ 50: width equals total (one block per test).
• total > 50: fixed width of 50 (some tests skip a block advance).

Arguments

• total::Int: Total number of tests

Returns

• Int: Character width for the progress bar (0 if total ≤ 0)

Example

julia> using CTBase.TestRunner

julia> TestRunner._bar_width(10)
10

julia> TestRunner._bar_width(25)
25

julia> TestRunner._bar_width(0)
0

_block_char_for_severity(sev::Int) -> String

Internal helper to map severity level to block character for colorblind-friendly display.

Uses distinct glyphs to ensure progress bars are readable without color:

• Success: █ (solid block)
• Skipped: ┆ (thin vertical line)
• Failure: ▚ (diagonal pattern)

Arguments

• sev::Int: Severity level (3=failure, 2=skipped, 1=success)

Returns

• String: Unicode block character representing the severity

_builder_to_string(x) -> String

Convert a Symbol or String to String.

This helper function ensures that builder function outputs are always converted to strings for consistent handling.

Arguments

• x: Symbol or String to convert

Returns

• String: The string representation of x

Example

julia> TestRunner._builder_to_string(:utils)
"utils"

julia> TestRunner._builder_to_string("utils")
"utils"

_collect_test_files_recursive(test_dir::AbstractString) -> Vector{String}

Recursively collect all .jl files in test_dir (excluding runtests.jl).

Returns relative paths from test_dir, sorted alphabetically.

Arguments

• test_dir::AbstractString: Root directory to search

Returns

• Vector{String}: Relative paths to all .jl files (excluding runtests.jl)

Example

# Assuming test_dir contains:
# - test/utils.jl
# - test/core/test_core.jl
# - test/runtests.jl

julia> TestRunner._collect_test_files_recursive("test")
2-element Vector{String}:
 "test/core/test_core.jl"
 "test/utils.jl"

_color_for_severity(sev::Int) -> String

Internal helper to map severity level to ANSI color code.

Arguments

• sev::Int: Severity level (3=failure, 2=skipped, 1=success)

Returns

• String: ANSI color escape code (red for failure, yellow for skipped, green for success)

_default_on_test_done(info::TestRunInfo)

Backward compatibility shim for the default test completion callback.

Creates a fresh stateful callback via _make_default_on_test_done and invokes it with the given info. This function exists for compatibility with existing code/tests that expect a stateless callback signature.

For new code, prefer using _make_default_on_test_done directly to create a persistent callback that maintains test history across multiple invocations.

Arguments

• info::TestRunInfo: Test execution information

See also: _make_default_on_test_done

_ensure_jl(filename::AbstractString) -> String

Ensure that a filename ends with .jl extension.

If the filename already ends with .jl, returns it unchanged. Otherwise, appends .jl to the filename.

Arguments

• filename::AbstractString: Base filename with or without .jl extension

Returns

• String: Filename guaranteed to end with .jl

Example

julia> TestRunner._ensure_jl("test_utils")
"test_utils.jl"

julia> TestRunner._ensure_jl("test_utils.jl")
"test_utils.jl"

_find_symbol_test_file_rel(name::Symbol, filename_builder::Function; test_dir::AbstractString) -> Union{String,Nothing}

Find the relative path to a test file for a given symbol name.

Uses the filename_builder to construct the expected filename, then searches for files matching that basename. If multiple matches exist (e.g., files in different subdirectories), prefers the shallowest path.

Arguments

• name::Symbol: Test name to resolve
• filename_builder::Function: Function that maps test names to filenames
• test_dir::AbstractString: Root directory containing test files

Returns

• String: Relative path to the matching test file
• nothing: If no matching file is found

Notes

• Searches recursively in test_dir
• Excludes runtests.jl from consideration
• Prefers shallower paths when multiple matches exist
• Returns the exact relative path if found

See also: TestRunner._collect_test_files_recursive, TestRunner._ensure_jl

_format_progress_line(
    io::IO,
    info::TestRunner.TestRunInfo;
    history,
    cumulative_severity
)

Write a styled progress line for a completed test to io.

Uses ANSI colors: green for success, red for errors, yellow for skipped.

Arguments

• io::IO: Output stream to write to
• info::TestRunInfo: Test execution information

Notes

• Format: [progress_bar] symbol [index/total] spec (time) status
• Colors: green for success, red for errors, yellow for skipped
• Time is displayed with one decimal place when available

Example

julia> using CTBase.TestRunner, IOBuffer

julia> info = TestRunner.TestRunInfo(
           :test_example, 
           "/path/to/test.jl", 
           :test_example, 
           5, 10, 
           :post_eval, 
           nothing, 
           1.23
       );

julia> buf = IOBuffer();
julia> TestRunner._format_progress_line(buf, info);
julia> String(take!(buf))
"[█████░░░░░░░░░░░] ✓ [05/10] test_example (1.2s)"

_glob_to_regex(pattern::AbstractString) -> Regex

Convert a glob pattern (using * and ?) into a regular expression.

The returned regex is anchored (matches the full string).

Arguments

• pattern::AbstractString: Glob pattern to convert

Returns

• Regex: Anchored regular expression equivalent to the glob pattern

Example

julia> using CTBase.TestRunner

julia> TestRunner._glob_to_regex("test_*.jl")
r"^test_.*\.jl$"

julia> TestRunner._glob_to_regex("suite/test_?.jl")
r"^suite/test.\.jl$"

_has_failures_in_results(ts::Test.DefaultTestSet) -> Bool
_has_failures_in_results(
    ts::Test.DefaultTestSet,
    from::Int64
) -> Bool

Recursively scan a DefaultTestSet results for Test.Fail or Test.Error entries, starting at index from.

This is used to detect @test failures that occurred during a specific eval by comparing the results count before and after the eval. The anynonpass field is unreliable because it is only updated when a testset finishes (in Test.finish).

Arguments

• ts::Test.DefaultTestSet: TestSet to scan
• from::Int: Starting index for scanning (default: 1)

Returns

• Bool: true if any failures are found, false otherwise

Example

julia> using CTBase.TestRunner, Test

julia> ts = Test.DefaultTestSet("test", [])
julia> Test.@testset "example" begin
           Test.@test 1 == 1
           Test.@test 2 == 0  # This will fail
       end
Test.DefaultTestSet("example", Any[Test.Pass(1), Test.Fail("false")])

julia> TestRunner._has_failures_in_results(ts)
true

_make_default_on_test_done(
    io::IO,
    total::Int64
) -> TestRunner.var"#update#_make_default_on_test_done##0"{<:IO, Base.RefValue{Int64}, Vector{Int64}}

Default progress callback for on_test_done. Prints to stdout.

Arguments

• info::TestRunInfo: Test execution information to display

Notes

• This is the default callback used when progress=true and no custom on_test_done is provided
• Outputs a formatted progress line to stdout with colors and timing information

Example

julia> using CTBase.TestRunner

julia> info = TestRunner.TestRunInfo(
           :test_example, 
           "/path/to/test.jl", 
           :test_example, 
           5, 10, 
           :post_eval, 
           nothing, 
           1.23
       );

julia> TestRunner._default_on_test_done(info)
[█████░░░░░░░░░░░] ✓ [05/10] test_example (1.2s)

_normalize_available_tests(available_tests) -> Vector{TestSpec}

Normalize and validate the available_tests argument.

Converts the input to a Vector{TestSpec} and validates that all entries are either Symbol or String. Returns an empty vector if available_tests is nothing.

Arguments

• available_tests: nothing, Vector, or Tuple containing Symbol or String entries

Returns

• Vector{TestSpec}: Normalized vector of test specifications

Throws

• ArgumentError: If available_tests is not a Vector/Tuple or contains invalid entries

Example

julia> TestRunner._normalize_available_tests([:utils, "suite/*"])
2-element Vector{Union{Symbol, String}}:
 :utils
 "suite/*"

julia> TestRunner._normalize_available_tests(nothing)
Union{Symbol, String}[]

_normalize_selections(
    selections::Vector{String},
    candidates::Vector{<:Union{String, Symbol}}
) -> Vector{String}

Normalize user-provided selection patterns before glob matching.

Applied transformations:

• Strip trailing / (e.g. "suite/exceptions/" → "suite/exceptions")
• If a selection contains no glob wildcard (* or ?) and matches a directory prefix of at least one candidate, expand it to "selection/*" so that all files under that directory are selected.

The original selection is always kept so that exact-name matches still work.

Arguments

• selections::Vector{String}: User-provided selection patterns
• candidates::Vector{<:TestSpec}: Available test candidates

Returns

• Vector{String}: Normalized selection patterns

Example

julia> using CTBase.TestRunner

julia> TestRunner._normalize_selections(
           ["suite/"], 
           ["suite/test_a.jl", "suite/test_b.jl"]
       )
2-element Vector{String}:
 "suite"
 "suite/*"

_parse_test_args(
    args::Vector{String}
) -> Tuple{Vector{String}, Bool, Bool}

Parse command-line test arguments, filtering out coverage-related flags.

Arguments

• args::Vector{String}: Raw command-line arguments

Returns

• Tuple{Vector{String}, Bool, Bool}: (selections, run_all, dry_run) where:
  • selections: selection patterns provided by the user (as strings)
  • run_all: whether -a / --all was present
  • dry_run: whether -n / --dryrun was present

Notes

• Coverage flags (coverage=true, --coverage, etc.) are automatically filtered out
• Selection patterns starting with test/ or test\ are automatically stripped so that users can write test/suite/foo or suite/foo interchangeably

Example

julia> using CTBase.TestRunner

julia> TestRunner._parse_test_args(["utils", "-a", "--dryrun"])
(["utils"], true, true)

julia> TestRunner._parse_test_args(["test/suite", "coverage=true"])
(["suite"], false, false)

_progress_bar(index::Int64, total::Int64; width) -> String

Render a progress bar string like [████████░░░░░░░░░░░].

When width is nothing (default), the width is computed automatically via _bar_width(total). Returns an empty string when the bar is hidden.

Arguments

• index::Int: current progress (1-based)
• total::Int: total number of items
• width::Union{Int,Nothing}: character width of the bar (default: auto)

Returns

• String: Progress bar string, or empty string if hidden

Example

julia> using CTBase.TestRunner

julia> TestRunner._progress_bar(5, 10)
"[█████░░░░░]"

julia> TestRunner._progress_bar(5, 10; width=20)
"[████████████████████░░░░░░]"

julia> TestRunner._progress_bar(0, 10; width=5)
"[░░░░░]"

_resolve_test(
    spec::Union{String, Symbol};
    available_tests,
    filename_builder,
    funcname_builder,
    eval_mode,
    test_dir
)

Resolve a test spec into an absolute filename and function symbol.

Handles both String specs (relative paths) and Symbol specs (logical names). Raises errors if the file is not found or if eval_mode=true but no function can be determined.

Arguments

• spec::TestSpec: Test specification to resolve
• available_tests::AbstractVector{<:TestSpec}: Available tests for validation
• filename_builder::Function: Function to map test names to filenames
• funcname_builder::Function: Function to map test names to function names
• eval_mode::Bool: Whether to resolve a function name
• test_dir::String: Root directory containing test files

Returns

• Tuple{String, Union{Symbol,Nothing}}: (filename, func_symbol) where:
  • filename: Absolute path to the test file
  • func_symbol: Function symbol to call (or nothing if eval_mode=false)

Throws

• ErrorException: If the test file is not found
• ErrorException: If eval_mode=true but no function can be determined

Notes

• This function is not part of the public API
• Use run_tests for running tests with proper error handling

_run_single_test(
    spec::Union{String, Symbol};
    available_tests,
    filename_builder,
    funcname_builder,
    eval_mode,
    test_dir,
    index,
    total,
    on_test_start,
    on_test_done
)

Run a single selected test.

This helper:

• Resolves a test filename via filename_builder
• Includes the file into Main
• Calls on_test_start (if provided) after include, before eval
• Optionally evaluates a function (via funcname_builder) when eval_mode=true
• Calls on_test_done (if provided) after eval, skip, or error

Arguments

• spec::TestSpec: Test specification to run
• available_tests::AbstractVector{<:TestSpec}: Available tests for validation
• filename_builder::Function: Function to map test names to filenames
• funcname_builder::Function: Function to map test names to function names
• eval_mode::Bool: Whether to evaluate the function after include
• test_dir::String: Root directory containing test files
• index::Int: 1-based index in the selected list (default: 1)
• total::Int: Total number of selected tests (default: 1)
• on_test_start::Union{Function,Nothing}: Callback before eval (default: nothing)
• on_test_done::Union{Function,Nothing}: Callback after eval (default: nothing)

Notes

• This function is not part of the public API
• Use run_tests for running multiple tests with proper orchestration

_select_tests(
    selections::Vector{String},
    available_tests::AbstractVector{<:Union{String, Symbol}},
    run_all::Bool,
    filename_builder::Function;
    test_dir
) -> Vector{Union{String, Symbol}}

Determine which tests to run based on selections, available_tests filter, and file globbing.

1. Identify potential test files in test_dir (default: test/).
2. Filter by available_tests if provided.
3. Filter by selections (interpreted as globs) if present.

Arguments

• selections::Vector{String}: User-provided selection patterns
• available_tests::AbstractVector{<:TestSpec}: Allowed tests (empty = auto-discovery)
• run_all::Bool: Whether to run all available tests
• filename_builder::Function: Function to map test names to filenames
• test_dir::String: Root directory containing test files

Returns

• Vector{TestSpec}: Selected test specifications

Notes

• If available_tests is empty, this function falls back to an auto-discovery heuristic using the filename stem as the candidate test name
• Selection arguments are matched against multiple representations of each candidate

_severity(status::Symbol) -> Int

Internal helper to map test status to severity level for display formatting.

Arguments

• status::Symbol: Test status (:error, :test_failed, :skipped, or success)

Returns

• Int: Severity level (3=failure, 2=skipped, 1=success)

_strip_test_prefix(s::AbstractString) -> Any

Strip a leading test/ or test\ prefix from a selection pattern.

This allows users to type test/suite/foo instead of suite/foo since the test directory is already the root for pattern matching.

Arguments

• s::AbstractString: Selection pattern to process

Returns

• String: Pattern with test/ or test\ prefix stripped (if present)

Example

julia> using CTBase.TestRunner

julia> TestRunner._strip_test_prefix("test/suite/foo")
"suite/foo"

julia> TestRunner._strip_test_prefix("suite/foo")
"suite/foo"

julia> TestRunner._strip_test_prefix("test\windows\path")
"windows\path"