Public API

PerformanceProfileConfig{M}

Configuration describing how to build a performance profile from a benchmark results table.

Fields

• instance_cols::Vector{Symbol}: Columns defining an instance (e.g., [:problem, :grid_size]).
• solver_cols::Vector{Symbol}: Columns defining a solver/model (e.g., [:model, :solver]).
• criterion::ProfileCriterion{M}: Metric extraction and comparison rule.
• is_success::Function: row::DataFrameRow -> Bool, selects successful runs.
• row_filter::Function: row::DataFrameRow -> Bool, additional filtering.
• aggregate::Function: Aggregation xs::AbstractVector{M} -> M when multiple runs exist for the same instance/solver.

Example

config = PerformanceProfileConfig{Float64}(
    [:problem, :grid_size],
    [:model, :solver],
    cpu_criterion,
    row -> row.success == true && row.benchmark !== nothing,
    row -> true,
    xs -> mean(skipmissing(xs))
)

PerformanceProfileRegistry

A container for named performance profile configurations.

Example

registry = PerformanceProfileRegistry()
register!(registry, "default_cpu", cpu_config)
config = get_config(registry, "default_cpu")

ProfileCriterion{M}

Criterion used to extract and compare a scalar metric from benchmark runs.

Fields

• name::String: Human-readable name of the criterion (e.g., "CPU time (s)").
• value::Function: Function row::DataFrameRow -> M extracting the metric.
• better::Function: Function (a::M, b::M) -> Bool returning true if a is strictly better than b according to the criterion.

Example

cpu_criterion = ProfileCriterion{Float64}(
    "CPU time (s)",
    row -> get(row.benchmark, "time", NaN),
    (a, b) -> a <= b
)

analyze_performance_profile(pp::PerformanceProfile) -> String

Generate a detailed textual analysis of a performance profile.

This is a convenience function that combines compute_profile_stats and _format_analysis_markdown. For programmatic access to analysis data, use compute_profile_stats directly.

Arguments

• pp::PerformanceProfile: Pre-computed performance profile data

Returns

• String: Markdown-formatted analysis report

Details

This function extracts key metrics from the performance profile:

• Robustness: proportion of instances successfully solved by each combo
• Efficiency: proportion of instances where each combo was fastest

benchmark(
;
    problems,
    solver_models,
    grid_sizes,
    disc_methods,
    tol,
    ipopt_mu_strategy,
    print_trace,
    max_iter,
    max_wall_time
)

Run benchmarks on optimal control problems and build a JSON-ready payload.

This function performs the following steps:

1. Detects CUDA availability and filters out :exa_gpu if CUDA is not functional
2. Runs benchmarks using benchmark_data() to generate a DataFrame of results
3. Collects environment metadata (Julia version, OS, machine, timestamp)
4. Builds a JSON-friendly payload combining results and metadata
5. Returns the payload as a Dict

The JSON file can be easily loaded and converted back to a DataFrame using:

using JSON, DataFrames
data = JSON.parsefile("path/to/data.json")
df = DataFrame(data["results"])

Arguments

• problems: Vector of problem names (Symbols)
• solver_models: Vector of Pairs mapping solver => models (e.g., [:ipopt => [:jump, :adnlp], :madnlp => [:exa, :exa_gpu]])
• grid_sizes: Vector of grid sizes (Int)
• disc_methods: Vector of discretization methods (Symbols)
• tol: Solver tolerance (Float64)
• ipopt_mu_strategy: Mu strategy for Ipopt (String)
• print_trace: Boolean - whether to print solver output (for debugging)
• max_iter: Maximum number of iterations (Int)
• max_wall_time: Maximum wall time in seconds (Float64)

Returns

• Dict

Example

julia> using CTBenchmarks

julia> payload = CTBenchmarks.benchmark(
           problems = [:beam],
           solver_models = [:ipopt => [:jump]],
           grid_sizes = [100],
           disc_methods = [:trapeze],
           tol = 1e-6,
           ipopt_mu_strategy = "adaptive",
           print_trace = false,
           max_iter = 1000,
           max_wall_time = 60.0,
       )
Dict{String, Any} with 3 entries:
  "metadata"  => Dict{String, Any}(...)
  "results"   => Vector{Dict}(...)
  "solutions" => Any[...]

build_profile_from_df(
    df::DataFrame,
    bench_id::AbstractString,
    cfg::PerformanceProfileConfig{M};
    allowed_combos::Union{Nothing, Vector{Tuple{String,String}}}=nothing,
) where {M}

Build a PerformanceProfile{M} from a benchmark results table.

This function takes a DataFrame of benchmark rows, applies the PerformanceProfileConfig{M} (instance/solver columns, criterion, filters and aggregation), and computes Dolan–Moré ratios and solver–model labels.

Arguments

• df::DataFrame: raw benchmark results loaded from JSON.
• bench_id::AbstractString: benchmark identifier.
• cfg::PerformanceProfileConfig{M}: configuration describing how to extract and aggregate the metric.
• allowed_combos::Union{Nothing, Vector{<:Tuple}}: optional list of solver combination tuples to keep (must match cfg.solver_cols dimension); nothing uses all available combinations.

Returns

• PerformanceProfile{M} if at least one valid metric is available; nothing if no instances or valid metrics are found.

load_benchmark_df(source::AbstractString) -> DataFrame

Load benchmark data from a JSON file path.

Arguments

• source: Path to a JSON file containing benchmark results.

Returns

• DataFrame with the "results" array from the JSON, or an empty DataFrame if not found.

load_benchmark_df(source::Dict) -> DataFrame

Load benchmark data from a parsed JSON dictionary.

Arguments

• source: Dictionary containing benchmark data with a "results" key.

Returns

• DataFrame with the "results" array, or an empty DataFrame if not found.

load_benchmark_df(source::DataFrame) -> DataFrame

Pass-through for already-loaded DataFrame.

plot_performance_profile(pp::PerformanceProfile; plot_config=nothing) -> Plots.Plot

Generate a Dolan–Moré performance profile plot from a PerformanceProfile struct.

Arguments

• pp::PerformanceProfile: Pre-computed performance profile data
• plot_config::Union{Nothing, PerformanceProfilePlotConfig}: Optional styling configuration

Returns

• Plots.Plot: Performance profile visualization

plot_solutions(payload::Dict, output_dir::AbstractString)

Generate PDF plots comparing solutions for each (problem, grid_size) pair.

This is the main entry point for visualizing benchmark results. It creates comprehensive comparison plots where all solver-model combinations for a given problem and grid size are overlaid on the same figure, enabling easy visual comparison of solution quality and convergence behavior.

Arguments

• payload::Dict: Benchmark results dictionary containing:
  • "results": Vector of result dictionaries with fields: problem, grid_size, model, solver, etc.
  • "solutions": Vector of solution objects (OptimalControl.Solution or JuMP.Model)
• output_dir::AbstractString: Directory where PDF files will be saved (created if not exists)

Output

Generates one PDF file per (problem, gridsize) combination with filename format: `<problem>N<grid_size>.pdf`

Each plot displays:

• State and costate trajectories (2 columns)
• Control trajectories (full width below)
• All solver-model combinations overlaid with consistent colors and markers
• Success/failure indicators (✓/✗) in legend

Details

• OptimalControl solutions are plotted first (simple overlay)
• JuMP solutions are plotted last (for proper subplot layout)
• Uses consistent color and marker schemes via get_color and get_marker_style
• Handles missing or failed solutions gracefully

Example

julia> using CTBenchmarks

julia> payload = Dict(
           "results" => [...],  # benchmark results
           "solutions" => [...]  # solution objects
       )

julia> CTBenchmarks.plot_solutions(payload, "plots/")
📊 Generating solution plots...
  - Plotting beam with N=100 (4 solutions)
    ✓ Saved: beam_N100.pdf
✅ All solution plots generated in plots/

run(; ...) -> Dict
run(version::Symbol; filepath, print_trace) -> Dict

Run comprehensive benchmarks on optimal control problems with various solvers and discretization methods.

This function executes a predefined benchmark suite that evaluates the performance of different optimal control solvers (Ipopt, MadNLP) across multiple models (JuMP, ADNLP, Exa, Exa-GPU) and problems. Results are collected in a structured dictionary and optionally saved to JSON.

Arguments

• version::Symbol: Benchmark suite version to run (default: :complete)
  • :complete: Full suite with 14 problems, multiple grid sizes (100, 200, 500), and two discretization methods
  • :minimal: Quick suite with only the beam problem and grid size 100 (useful for testing)
• filepath::Union{AbstractString, Nothing}: Optional path to save results as JSON file (must end with .json). If nothing, results are only returned in memory.
• print_trace::Bool: Whether to print solver trace information during execution (default: false)

Returns

• Dict: Benchmark results containing timing data, solver statistics, and metadata for each problem-solver-model combination

Throws

• CTBase.IncorrectArgument: If filepath is provided but does not end with .json
• ErrorException: If version is neither :complete nor :minimal

Example

julia> using CTBenchmarks

julia> # Run minimal benchmark and save results
julia> results = run(:minimal; filepath="results.json")

julia> # Run complete benchmark without saving
julia> results = run(:complete)

julia> # Run with solver trace output
julia> results = run(:minimal; print_trace=true)

See Also

• benchmark: Core benchmarking function with full customization