utils.benchmark.benchmarker module

class Benchmarker[source]

Bases: abc.ABC

Benchmarker class used to run benchmarks of our integrator comparing to others and perform parameter scans

abstract benchmark_method(d, integrand, integrator_config=None, integrand_params=None, n_batch=100000, keep_history=False, device=device(type='cpu'))[source]

Run a single benchmark evaluation of a given integrand

abstract generate_config_samples(dimensions, integrator_grid, integrand_grid)[source]

Sample over dimensions, integrator and integrand configurations from lists of possible option values.

Parameters

  • dimensions (List[int]) – list of dimensions to sample from

  • integrator_grid (Dict[str, List[Any]]) – mapping (option name) -> (list of values) for the integrator

  • integrand_grid (Dict[str, List[Any]]) – mapping (option name) -> (list of values) for the integrand

Yields

Tuple[int, Dict[str, Any], Dict[str, Any]] – triplets (d, integrator_config, integrand_params) that can be used to sample configurations

run(dimensions, integrand, *, base_integrand_params, base_integrator_config=None, integrand_params_grid=None, integrator_config_grid=None, n_batch=100000, debug=True, cuda=0, sql_dtypes=None, dbname=None, experiment_name='benchmark', keep_history=False)[source]

Run benchmarks over a grid of parameters for the integrator and the integrand.

set_benchmark_grid_config(config=None, dimensions=None, n_batch=None, keep_history=None, dbname=None, experiment_name=None, cuda=None, debug=None, default_dimension=(2,), base_integrand_params=())[source]

Prepare standard arguments for run_benchmark_grid The parameter importance hierarchy is:

  1. CLI arguments (direct arguments to this function)

  2. config file (filepath given as config)

  3. default values provided as argument

static set_benchmark_grid_config_param(benchmark_grid_config, param, param_value, config)[source]

Set an argument in the configuration dictionary to be provided to run_benchmark_grid according to the established hierarchy:

  1. direct argument value (typically from CLI)

  2. config file value

  3. default value - what is already in the config dictionary when entering this function

Parameters

  • benchmark_grid_config (dict) – configuration dictionary being constructed. Keys should match the arguments of run_benchmark_grid

  • param (str) – name of the parameter to be set

  • param_value (optional) – direct argument value for the parameter. Use the configuration dictionary if not provided

  • config (Configuration, optional) – Configuration nested dictionary for the benchmarking run at hand. If it is not provided or does not specify a value of param, use the default value instead.

class GridBenchmarker(n_repeat=1)[source]

Bases: utils.benchmark.benchmarker.Benchmarker

Benchmark by sampling configurations like a grid. Can repeat each configurations multiple times

Parameters

n (int) – Optional: How often the grid is sampled.

generate_config_samples(dimensions, integrator_grid, integrand_grid)[source]

Sample over dimensions, integrator and integrand configurations from lists of possible option values.

Parameters

  • dimensions (List[int]) – list of dimensions to sample from

  • integrator_grid (Dict[str, List[Any]]) – mapping (option name) -> (list of values) for the integrator

  • integrand_grid (Dict[str, List[Any]]) – mapping (option name) -> (list of values) for the integrand

Yields

Tuple[int, Dict[str, Any], Dict[str, Any]] – triplets (d, integrator_config, integrand_params) that can be used to sample configurations

class RandomHyperparameterBenchmarker(n_samples=5, n_repeat=1)[source]

Bases: utils.benchmark.benchmarker.Benchmarker

Benchmark by sampling integrator configurations randomly, a fixed number of times. Can repeat each configurations multiple times

Parameters

  • n_samples (int) – Number of random integrator configurations to draw

  • n_repeat (int) – Optional: How often the grid is sampled.

generate_config_samples(dimensions, integrator_grid, integrand_grid)[source]

Sample over dimensions, integrator and integrand configurations from lists of possible option values.

Parameters

  • dimensions (List[int]) – list of dimensions to sample from

  • integrator_grid (Dict[str, List[Any]]) – mapping (option name) -> (list of values) for the integrator

  • integrand_grid (Dict[str, List[Any]]) – mapping (option name) -> (list of values) for the integrand

Yields

Tuple[int, Dict[str, Any], Dict[str, Any]] – triplets (d, integrator_config, integrand_params) that can be used to sample configurations

class SequentialBenchmarker(n_repeat=1)[source]

Bases: utils.benchmark.benchmarker.Benchmarker

Benchmark by going through a list of integrands and matching integrator configurations (as opposed to scanning through parameters for either).

This is intended for benchmarking integrands on “optimal configurations” found through previous hyper parameter searching. Can repeat each configurations multiple times

Parameters

n (int) – Optional: How often the grid is sampled.

generate_config_samples(dimensions, integrator_grid, integrand_grid)[source]

Sample over dimensions, integrator and integrand configurations from lists of possible option values.

Parameters

  • dimensions (List[int]) – list of dimensions to sample from

  • integrator_grid (Dict[str, List[Any]]) – mapping (option name) -> (list of values) for the integrator

  • integrand_grid (Dict[str, List[Any]]) – mapping (option name) -> (list of values) for the integrand

Yields

Tuple[int, Dict[str, Any], Dict[str, Any]] – triplets (d, integrator_config, integrand_params) that can be used to sample configurations

Notes

The integrator grid is interpreted as a sequence:

integrator_grid = {
  param1: [v1, v2, ..., vn]
  param2: [w1, w2, ..., wn]
}

is scanned as the list of configurations

config1 = {param1: v1, param2: w1}
config2 = {param1: v2, param2: w2}
...
class SequentialIntegratorBenchmarker(n_repeat=1)[source]

Bases: utils.benchmark.benchmarker.Benchmarker

Benchmark by going through a list of full integrator configurations (as opposed to going through parameters in the configuration independently) and scan over a grid of possible integrands. Can repeat each configurations multiple times

Parameters

n (int) – Optional: How often the grid is sampled.

generate_config_samples(dimensions, integrator_grid, integrand_grid)[source]

Sample over dimensions, integrator and integrand configurations from lists of possible option values.

Parameters

  • dimensions (List[int]) – list of dimensions to sample from

  • integrator_grid (Dict[str, List[Any]]) – mapping (option name) -> (list of values) for the integrator

  • integrand_grid (Dict[str, List[Any]]) – mapping (option name) -> (list of values) for the integrand

Yields

Tuple[int, Dict[str, Any], Dict[str, Any]] – triplets (d, integrator_config, integrand_params) that can be used to sample configurations

Notes

The integrator grid is interpreted as a sequence:

integrator_grid = {
  param1: [v1, v2, ..., vn]
  param2: [w1, w2, ..., wn]
}

is scanned as the list of configurations

config1 = {param1: v1, param2: w1}
config2 = {param1: v2, param2: w2}