benchmark.utils

setup_seed(seed: int | None = None, cuda: bool = True) → int

setup_argparse()

setup_args(parser: ArgumentParser) → Namespace

save_args(logpath: Path, args: dict)

dict_to_json(dictionary) → dict

setup_logger(logpath: Path | str = PosixPath('../log'), level_console: int = 10, level_file: int = 15, quiet: bool = True, fmt='{message}')

clear_logger(logger: Logger)

setup_logpath(dir: Path | str = PosixPath('../log'), folder_args: Tuple | None = None, quiet: bool = True)

Resolve log path for saving.

Parameters:

• dir (Path | str, default: PosixPath('../log')) – Base directory for saving logs.

• folder_args (Tuple | None, default: None) – Subfolder names.

• quiet (bool, default: True) – Quiet run without creating directories.

Returns:

logpath (Path) – Path for log directory.

class ResLogger(logpath: Path | str = PosixPath('../log'), prefix: str = 'summary', suffix: str | None = None, quiet: bool = True)

Bases: object

Logger for formatting result to strings by wrapping pd.DataFrame table.

Parameters:

• logpath (Path | str, default: PosixPath('../log')) – Path to CSV file saving directory.

• quiet (bool, default: True) – Quiet run without saving file.

static guess_fmt(key: str, val) → Callable

Guesses the string format function based on its name.

property nrows

property ncols

_set(data: DataFrame, fmt: Series)

Sets the data from input DataFrame.

Parameters:

• data (DataFrame) – Concat on columns, inner join on index.

• fmt (Series) – Inner join on columns.

concat(vals: List[Tuple[str, Any, Callable]] | Dict, row: int = 0, suffix: str | None = None)

Concatenate data entries of a single row to data.

Parameters:

• vals (List[Tuple[str, Any, Callable]] | Dict) – list of entries (key, value, formatter).

• row (int, default: 0) – New index in self dataframe for vals to be logged.

• suffix (str | None, default: None) – Suffix string for input keys. Default is None.

Returns:

self (ResLogger)

merge(logger: ResLogger, rows: List[int] | None = None, suffix: str | None = None)

Merge from another logger.

Parameters:

• logger (ResLogger) – Logger to merge.

• row – New index in self dataframe.

• suffix (str | None, default: None) – Suffix string for input keys. Default is None.

del_col(col: List | str) → ResLogger

Delete columns from data.

Parameters:

col (List | str) – Column(s) to delete.

_get(col: List | str | None = None, row: List | str | None = None) → DataFrame | Series | str

Retrieve one or sliced data and apply string format.

Parameters:

• col (List | str | None, default: None) – Column(s) to retrieve. Defaults to all.

• row (List | str | None, default: None) – Row(s) to retrieve. Defaults to all.

Returns:

val – Formatted data. - type: follows the return type of DataFrame.loc[row, col]. - value: formatted string in each entry.

save()

Saves table data to CSV file.

get_str(col: List | str | None = None, row: List | int | None = None, maxlen: int = -1) → str

Get formatted long string for printing of the specified columns and rows.

Parameters:

• col (List | str | None, default: None) – Column(s) to retrieve. Defaults to all.

• row (List | int | None, default: None) – Row(s) to retrieve. Defaults to all.

• maxlen (int, default: -1) – Max line length of the resulting string.

Returns:

s (str) – Formatted string representation.

flt_str(metric) → str

Remove all substring start with s but not contain metric.

Parameters:

metric (str) – Metric to keep.

class CkptLogger(logpath: Path | str, patience: int = -1, period: int = 0, prefix: str = 'model', storage: str = 'state_gpu', metric_cmp: Callable[[float, float], bool] | str = 'max')

Bases: object

Checkpoint Logger for saving and loading models and managing early stopping during training.

Parameters:

• logpath (Path | str) – Path to checkpoints saving directory.

• patience (int, default: -1) – Patience for early stopping. Defaults no early stopping.

• period (int, default: 0) – Periodic saving interval. Defaults to no periodic saving.

• prefix (str, default: 'model') – Prefix for the checkpoint file names.

• storage (str, default: 'state_gpu') – Storage scheme for saving the checkpoints. * ‘model’ vs ‘state’: Save model object or state_dict. * ‘_file’, ‘_ram’, ‘_gpu’: Save as file, RAM, or GPU memory.

• metric_cmp (Callable[[float, float], bool] | str, default: 'max') – Comparison function for the metric. Can be ‘max’ or ‘min’.

set_epoch(epoch: int = 0)

_get_model_path(*suffix) → Path

get_last_epoch() → int

Get last saved model epoch. Useful for deciding load model path.

save(*suffix, model: Module)

Save the model according to storage scheme.

Parameters:

• suffix – Variable length argument for suffix in the model file name.

• model (nn.Module) – The model to be saved.

load(*suffix, model: Module, map_location='cpu') → Module

Load the model from the storage.

Parameters:

• suffix – Variable length argument for suffix in the model file name.

• model (Module) – The model structure to load.

• map_location (default: 'cpu') – map_location argument for torch.load.

Returns:

model (nn.Module) – The loaded model.

clear()

property is_early_stop: bool

Whether current epoch satisfies early stopping criteria.

property is_period: bool

Whether current epoch should do periodic saving.

_is_improved(metric) → bool

Whether the metric is better than previous best.

step(metric: float, model: Module | None = None) → bool

Step one epoch with periodic saving and early stopping.

Parameters:

• metric (float) – Metric value for the current step.

• model (Module | None, default: None) – Model for the current step. Defaults to None.

Returns:

early_stop (bool) – True if early stopping criteria is met.

set_at_best(**kwargs)

Save given args to model attributes if is the best epoch.

get_at_best() → list

Get saved model attributes from the best epoch.

benchmark.utils.config

config.force_list_str(x)
config.force_list_int(x)
config.list_str(x)
config.list_int(x)
config.list_float(x)