neuraxle.metaopt.trial

Module-level documentation for neuraxle.metaopt.trial. Here is an inheritance diagram, including dependencies to other base modules of Neuraxle:

Inheritance diagram of neuraxle.metaopt.trial

Neuraxle’s Trial Classes

Trial objects used by AutoML algorithm classes.

Classes

TRIAL_STATUS

Trial status.

Trial(trial_number, hyperparams, …)

Trial data container for AutoML.

TrialSplit(trial, split_number, …)

One split of a trial.

Trials(trials)

Data object containing auto ml trials.


class neuraxle.metaopt.trial.Trial(trial_number: int, hyperparams: neuraxle.hyperparams.space.HyperparameterSamples, main_metric_name: str, save_trial_function: Callable, status: Optional[neuraxle.metaopt.trial.TRIAL_STATUS] = None, pipeline: neuraxle.base.BaseStep = None, validation_splits: List[TrialSplit] = None, cache_folder: str = None, error: str = None, error_traceback: str = None, start_time: datetime.datetime = None, end_time: datetime.datetime = None, logger: logging.Logger = None)[source]

Bases: object

Trial data container for AutoML. A Trial contains the results for each validation split. Each trial split contains both the training set results, and the validation set results.

See also

AutoML, TrialSplit, HyperparamsRepository, BaseHyperparameterSelectionStrategy, RandomSearchHyperparameterSelectionStrategy, DataContainer

__init__(trial_number: int, hyperparams: neuraxle.hyperparams.space.HyperparameterSamples, main_metric_name: str, save_trial_function: Callable, status: Optional[neuraxle.metaopt.trial.TRIAL_STATUS] = None, pipeline: neuraxle.base.BaseStep = None, validation_splits: List[TrialSplit] = None, cache_folder: str = None, error: str = None, error_traceback: str = None, start_time: datetime.datetime = None, end_time: datetime.datetime = None, logger: logging.Logger = None)[source]

Initialize self. See help(type(self)) for accurate signature.

save_trial() → neuraxle.metaopt.trial.Trial[source]

Update trial with the hyperparams repository.

Returns

new_validation_split(pipeline: neuraxle.base.BaseStep, delete_pipeline_on_completion: bool = True) → neuraxle.metaopt.trial.TrialSplit[source]

Create a new trial split. A trial has one split when the validation splitter function is validation split. A trial has one or many split when the validation splitter function is kfold_cross_validation_split.

Parameters

delete_pipeline_on_completion (bool) – bool to delete pipeline on completion or not

Returns

one trial split

save_model(label: str)[source]

Save fitted model in the trial hash folder.

_save_model(pipeline: neuraxle.base.BaseStep, label: str)[source]
get_model(label: str) → neuraxle.base.BaseStep[source]

Load model in the trial hash folder.

set_main_metric_name(name: str) → neuraxle.metaopt.trial.Trial[source]

Set trial main metric name.

Returns

self

set_hyperparams(hyperparams: neuraxle.hyperparams.space.HyperparameterSamples) → neuraxle.metaopt.trial.Trial[source]

Set trial hyperparams.

Parameters

hyperparams (HyperparameterSamples) – trial hyperparams

Returns

is_higher_score_better() → bool[source]

Return True if higher scores are better for the main metric.

Returns

if higher score is better

get_validation_score() → float[source]

Return the best validation score for the main scoring metric. Returns the average score for all validation splits.

Returns

validation score

get_n_epoch_to_best_validation_score() → float[source]

Return the best validation score for the main scoring metric. Returns the average score for all validation splits.

Returns

validation score

set_success() → neuraxle.metaopt.trial.Trial[source]

Set trial status to success.

Returns

self

update_final_trial_status()[source]

Set trial status to success.

set_failed(error: Exception) → neuraxle.metaopt.trial.Trial[source]

Set failed trial with exception.

Parameters

error (Exception) – catched exception

Returns

self

get_trained_pipeline(split_number: int = 0)[source]

Get trained pipeline inside the validation splits.

Parameters

split_number (int) – split number to get trained pipeline from

Returns

_get_trial_hash(hp_dict: Dict[KT, VT])[source]

Hash hyperparams with md5 to create a trial hash.

Parameters

hp_dict – hyperparams dict

Returns

to_json()[source]
static from_json(update_trial_function: Callable, trial_json: Dict[KT, VT], cache_folder: str = None) → neuraxle.metaopt.trial.Trial[source]
_initialize_logger_with_file() → logging.Logger[source]
_free_logger_file()[source]

Remove file handlers from logger to free file lock on Windows.

class neuraxle.metaopt.trial.TrialSplit(trial: neuraxle.metaopt.trial.Trial, split_number: int, main_metric_name: str, status: Optional[neuraxle.metaopt.trial.TRIAL_STATUS] = None, error: Exception = None, error_traceback: str = None, metrics_results: Dict[KT, VT] = None, start_time: datetime.datetime = None, end_time: datetime.datetime = None, pipeline: neuraxle.base.BaseStep = None, delete_pipeline_on_completion: bool = True)[source]

Bases: object

One split of a trial.

See also

AutoML, HyperparamsRepository, BaseHyperparameterSelectionStrategy, RandomSearchHyperparameterSelectionStrategy, DataContainer

__init__(trial: neuraxle.metaopt.trial.Trial, split_number: int, main_metric_name: str, status: Optional[neuraxle.metaopt.trial.TRIAL_STATUS] = None, error: Exception = None, error_traceback: str = None, metrics_results: Dict[KT, VT] = None, start_time: datetime.datetime = None, end_time: datetime.datetime = None, pipeline: neuraxle.base.BaseStep = None, delete_pipeline_on_completion: bool = True)[source]

Initialize self. See help(type(self)) for accurate signature.

get_metric_names() → List[str][source]
save_parent_trial() → neuraxle.metaopt.trial.TrialSplit[source]

Save parent trial.

Returns

self

save_model(label: str)[source]

Saves the pipeline instance the same way a Trial instance would. Will overwrite :return:

fit_trial_split(train_data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext) → neuraxle.metaopt.trial.TrialSplit[source]

Fit the trial split pipeline with the training data container.

Parameters
Returns

trial split with its fitted pipeline.

predict_with_pipeline(data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext) → neuraxle.data_container.DataContainer[source]

Predict data with the fitted trial split pipeline.

Parameters
Returns

predicted data container

set_main_metric_name(name: str) → neuraxle.metaopt.trial.TrialSplit[source]

Set main metric name.

Parameters

name (str) – main metric name.

Returns

self

add_metric_results_train(name: str, score: float, higher_score_is_better: bool, log_metric: bool = False)[source]

Add a train metric result in the metric results dictionary.

Parameters
  • name (str) – name of the metric

  • score (float) – score

  • higher_score_is_better (bool) – if higher score is better or not for this metric

Returns

add_metric_results_validation(name: str, score: float, higher_score_is_better: bool, log_metric: bool = False)[source]

Add a validation metric result in the metric results dictionary.

Parameters
  • name (str) – name of the metric

  • score (float) – score

  • higher_score_is_better (bool) – if higher score is better or not for this metric

Returns

get_validation_scores()[source]

Return the validation scores for the main scoring metric.

get_final_validation_score()[source]

Return the latest validation score for the main scoring metric.

get_best_validation_score()[source]

Return the best validation score for the main scoring metric.

get_n_epochs_to_best_validation_score()[source]

Return the number of epochs

get_pipeline()[source]

Return the trained pipeline.

is_higher_score_better() → bool[source]

Return True if higher scores are better for the main metric.

Returns

is_new_best_score()[source]

Return True if the latest validation score is the new best score.

Returns

get_metric_validation_results(metric_name)[source]
get_metric_train_results(metric_name)[source]
to_json() → dict[source]

Return the trial in a json format.

Returns

static from_json(trial: neuraxle.metaopt.trial.Trial, trial_split_json: Dict[KT, VT]) → neuraxle.metaopt.trial.TrialSplit[source]

Create a trial split object from json.

Parameters
  • trial – parent trial

  • trial_split_json – trial json

Returns

set_success() → neuraxle.metaopt.trial.TrialSplit[source]

Set trial status to success.

Returns

self

is_success()[source]

Set trial status to success.

set_failed(error: Exception) → neuraxle.metaopt.trial.TrialSplit[source]

Set failed trial with exception.

Parameters

error (Exception) – catched exception

Returns

self

class neuraxle.metaopt.trial.TRIAL_STATUS[source]

Bases: enum.Enum

Trial status.

FAILED = 'failed'[source]
STARTED = 'started'[source]
SUCCESS = 'success'[source]
PLANNED = 'planned'[source]
class neuraxle.metaopt.trial.Trials(trials: List[neuraxle.metaopt.trial.Trial] = None)[source]

Bases: object

Data object containing auto ml trials.

See also

AutoMLSequentialWrapper, RandomSearch, HyperparamsRepository, MetaStepMixin, BaseStep

__init__(trials: List[neuraxle.metaopt.trial.Trial] = None)[source]

Initialize self. See help(type(self)) for accurate signature.

get_best_hyperparams() → neuraxle.hyperparams.space.HyperparameterSamples[source]

Get best hyperparams from all trials.

Returns

get_best_trial() → neuraxle.metaopt.trial.Trial[source]
Returns

trial with best score from all trials

split_good_and_bad_trials(quantile_threshold: float, number_of_good_trials_max_cap: int) → Tuple[neuraxle.metaopt.trial.Trials, neuraxle.metaopt.trial.Trials][source]
is_higher_score_better() → bool[source]

Return true if higher score is better.

:return

append(trial: neuraxle.metaopt.trial.Trial)[source]

Add a new trial.

Parameters

trial (Trial) – new trial

Returns

filter(status: neuraxle.metaopt.trial.TRIAL_STATUS) → neuraxle.metaopt.trial.Trials[source]

Get all the trials with the given trial status.

Parameters

status – trial status

Returns

get_number_of_split()[source]
get_metric_names() → List[str][source]