Gas Categorized Benchmarks

This guide explains how to run gas-categorized profiling using the zkGas profiling framework. Gas-categorized profiling executes resource analysis tests across different gas limit categories, providing comprehensive analysis of the resources needed for proving different OPCODEs across various computational workloads.

Overview

The gas-categorized profiling system runs resource analysis tests on pre-generated fixture files organized by gas categories. It supports multiple zkVM implementations, execution clients, and profiling actions, allowing for comprehensive comparison of opcode resource requirements across different zk environments.

Script: run-gas-categorized-benchmarks.sh

Basic Usage

# Run all gas categories with default settings
./scripts/run-gas-categorized-benchmarks.sh
 
# Run with specific zkVM and execution client
./scripts/run-gas-categorized-benchmarks.sh --zkvm sp1 --execution-client reth
 
# Run on a single gas category
./scripts/run-gas-categorized-benchmarks.sh --gas-category 10M

Command Line Options

Supported zkVM Implementations

Supported Execution Clients

Supported Gas Categories

Usage Examples

Basic Benchmark Execution

Run All Gas Categories

# Run all gas categories with default settings (RISC0, Reth, GPU)
./scripts/run-gas-categorized-benchmarks.sh

Run with Specific zkVM

# Run with SP1 zkVM
./scripts/run-gas-categorized-benchmarks.sh --zkvm sp1
 
# Run with OpenVM zkVM
./scripts/run-gas-categorized-benchmarks.sh --zkvm openvm

Run with Different Execution Client

# Run with Ethrex execution client
./scripts/run-gas-categorized-benchmarks.sh --execution-client ethrex
 
# Run with Reth execution client (default)
./scripts/run-gas-categorized-benchmarks.sh --execution-client reth

Advanced Configuration

Custom Resource and Action

# Run with CPU resource and execute action
./scripts/run-gas-categorized-benchmarks.sh --action execute --resource cpu
 
# Run with GPU resource and prove action (default)
./scripts/run-gas-categorized-benchmarks.sh --action prove --resource gpu

Memory Tracking

# Enable memory tracking for detailed memory analysis
./scripts/run-gas-categorized-benchmarks.sh --memory-tracking true
 
# Run with memory tracking on specific gas category
./scripts/run-gas-categorized-benchmarks.sh --memory-tracking true --gas-category 10M
 
# Run with memory tracking and specific zkVM
./scripts/run-gas-categorized-benchmarks.sh --memory-tracking true --zkvm sp1

Single Gas Category

# Run only on 10M gas category
./scripts/run-gas-categorized-benchmarks.sh --gas-category 10M
 
# Run only on 100M gas category with SP1
./scripts/run-gas-categorized-benchmarks.sh --gas-category 100M --zkvm sp1

Custom Input Directory

# Run with custom input directory
./scripts/run-gas-categorized-benchmarks.sh --input-dir ./my-custom-fixtures

Preview and Validation

Dry Run Mode

# Preview what would be executed
./scripts/run-gas-categorized-benchmarks.sh --dry-run
 
# Preview with specific configuration
./scripts/run-gas-categorized-benchmarks.sh --dry-run --zkvm sp1 --gas-category 10M
 
# Preview with memory tracking enabled
./scripts/run-gas-categorized-benchmarks.sh --dry-run --memory-tracking true --zkvm sp1

Output Structure

The benchmark script generates the following output structure:

zkevm-metrics-risc0-1M/      # RISC0 results for 1M gas
zkevm-metrics-risc0-10M/     # RISC0 results for 10M gas
zkevm-metrics-sp1-1M/        # SP1 results for 1M gas
zkevm-metrics-sp1-10M/       # SP1 results for 10M gas
...

Each metrics directory contains:

• JSON files with detailed benchmark results
• Execution metrics (cycles, duration, region breakdowns)
• Proving metrics (proof size, proving time, memory usage)
• Memory tracking data (when enabled): detailed memory allocation patterns, peak memory usage, memory efficiency metrics
• Metadata about the benchmark run

Workflow Details

1. Pre-flight Checks

The script performs several validation steps:

• Verifies cargo is available
• Confirms the script is run from the project root directory
• Validates the project structure (presence of Cargo.toml and crates/ere-hosts)
• Validates gas category if specified

2. Input Fixture Validation

The script checks for required input fixture directories:

• Scans for directories matching the pattern {BASE_INPUT_DIR}-{GAS_VALUE}
• Warns about missing fixture directories
• Continues with available directories

3. Project Build

The script automatically builds the ere-hosts binary with the specified zkVM feature:

cargo build --release --bin ere-hosts

4. Benchmark Execution

For each gas category, the script:

• Creates the metrics output directory
• Runs the benchmark with the specified configuration
• Collects execution and proving metrics
• Saves results in JSON format

5. Summary Report

After completion, the script provides:

• Summary of benchmark results for each gas category
• File counts for each metrics directory
• Location of all generated metrics

Performance Considerations

Resource Requirements

GPU Resources

• Recommended: NVIDIA GPU with CUDA support
• Memory: At least 8GB VRAM for larger gas categories
• Performance: Significantly faster for proving operations

CPU Resources

• Recommended: Multi-core CPU with at least 16GB RAM
• Performance: Slower but more accessible for development

Memory Tracking

When memory tracking is enabled (--memory-tracking true), the benchmark system provides detailed memory analysis:

Memory Metrics Collected

• Peak Memory Usage: Maximum memory consumption during execution
• Memory Allocation Patterns: Detailed tracking of memory allocations and deallocations
• Memory Efficiency: Analysis of memory utilization vs. waste
• Memory Growth Trends: How memory usage scales with gas categories
• Memory Hotspots: Identification of memory-intensive operations

Use Cases for Memory Tracking

• Performance Optimization: Identify memory bottlenecks in zkVM implementations
• Resource Planning: Determine optimal memory requirements for different gas categories
• Memory Leak Detection: Monitor for potential memory leaks during long-running benchmarks
• Comparative Analysis: Compare memory efficiency between different zkVM implementations

Performance Impact

• Overhead: Memory tracking adds approximately 5-10% execution time overhead
• Storage: Generates additional JSON files with memory metrics
• Recommended: Enable for detailed analysis, disable for maximum performance

Execution Time Estimates

Times are estimates and may vary based on hardware and zkVM implementation

Troubleshooting

Common Issues

Missing Input Fixtures

# Check if fixture directories exist
ls -la zkevm-fixtures-input-*
 
# Generate fixtures if missing
./scripts/generate-gas-categorized-fixtures.sh

Build Failures

# Ensure you're in the project root directory
pwd  # Should show: .../zkevm-benchmark-workload
 
# Check if Cargo.toml exists
ls Cargo.toml
 
# Verify ere-hosts crate exists
ls crates/ere-hosts/

Permission Issues

# Make the script executable
chmod +x scripts/run-gas-categorized-benchmarks.sh

Resource Issues

# Check available disk space
df -h
 
# Check GPU availability (if using GPU)
nvidia-smi
 
# Check system memory
free -h

Error Messages

Integration with Analysis

The generated benchmark results are designed to work with the analysis system:

1. Markdown Tables: Generate formatted tables from results

   ./scripts/generate_results.sh --compare --statistics

2. Performance Comparison: Compare results between different runs

   python3 scripts/compare_executions.py baseline-folder optimized-folder
   python3 scripts/compare_provings.py baseline-folder optimized-folder

3. Statistical Analysis: Include statistical analysis in reports

   ./scripts/generate_results.sh --statistics --output detailed-results.md

Best Practices

Benchmark Execution

1. Start Small: Begin with smaller gas categories (1M, 10M) to validate setup
2. Use Dry Run: Always preview commands with --dry-run before execution
3. Monitor Resources: Keep an eye on disk space and memory usage
4. Document Configuration: Record the exact command used for reproducibility

Performance Optimization

1. Use GPU: GPU resources provide significant speedup for proving operations
2. Parallel Execution: Run different zkVM implementations in parallel
3. Resource Management: Ensure sufficient disk space for results
4. Network Considerations: Some zkVM implementations may require network access

Result Management

1. Organize Results: Use descriptive directory names for different benchmark runs
2. Backup Important Results: Archive significant benchmark results
3. Version Control: Consider versioning benchmark configurations
4. Documentation: Document the purpose and configuration of each benchmark run

Next Steps

After running gas-categorized benchmarks, you can:

1. Generate Analysis Reports: Create markdown tables and statistical analysis

   ./scripts/generate_results.sh --compare --statistics

2. Compare Performance: Analyze performance differences between zkVM implementations

   python3 scripts/compare_executions.py zkevm-metrics-risc0-10M zkevm-metrics-sp1-10M

3. Optimize Configuration: Use results to optimize benchmark parameters for your use case