neuraxle.metaopt.data.reporting

Module-level documentation for neuraxle.metaopt.data.reporting. Here is an inheritance diagram, including dependencies to other base modules of Neuraxle:


Neuraxle’s AutoML Metric Reporting classes.

Classes are splitted like this for the metric analysis:

  • ProjectReport

  • ClientReport

  • RoundReport

  • TrialReport

  • TrialSplitReport

These AutoML reports are used to get information from the nested dataclasses, such as to create visuals.

Classes

BaseReport(dc)

A report contains a dataclass of the same subclass-level of itself, so as to be able to dig into the dataclass so as to observe it, such as to generate statistics and query its information.

ClientReport(dc)

MetricResultsReport(dc)

ProjectReport(dc)

RootReport(dc)

RoundReport(dc)

TrialReport(dc)

TrialSplitReport(dc)


neuraxle.metaopt.data.reporting._filter_df_hps(wildcarded_hps: OrderedDict[str, Any], wildcards_to_keep: List[str] = None) → OrderedDict[str, Any][source]

Filters the hyperparameters so as to keep only the indicated wildcards to keep. Also parses the hyperparameters to strings if not numeric.

class neuraxle.metaopt.data.reporting.BaseReport(dc: SubDataclassT)[source]

Bases: typing.Generic

A report contains a dataclass of the same subclass-level of itself, so as to be able to dig into the dataclass so as to observe it, such as to generate statistics and query its information. The dataclasses represent the results of an AutoML optimization round, even multiple rounds. These AutoML reports are used to get information from the nested dataclasses, such as to create visuals.

See also

neuraxle.metaopt.data.vanilla.BaseDataclass neuraxle.metaopt.data.reporting.BaseAggregate

__init__(dc: SubDataclassT)[source]

Initialize self. See help(type(self)) for accurate signature.

static from_dc(dc: SubDataclassT) → SubReportT[source]
static from_json(json_dc_data: str) → SubReportT[source]
get_id() → Union[str, int][source]
info_df() → pandas.core.frame.DataFrame[source]
subreport(_dataclass: SubDataclassT) → SubReportT[source]
class neuraxle.metaopt.data.reporting.RootReport(dc: SubDataclassT)[source]

Bases: neuraxle.metaopt.data.reporting.BaseReport

class neuraxle.metaopt.data.reporting.ProjectReport(dc: SubDataclassT)[source]

Bases: neuraxle.metaopt.data.reporting.BaseReport

to_clients_with_best_scores_df() → pandas.core.frame.DataFrame[source]

Get a dataframe with the info of each last round for each client.

class neuraxle.metaopt.data.reporting.ClientReport(dc: SubDataclassT)[source]

Bases: neuraxle.metaopt.data.reporting.BaseReport

CLIENT_ID_COLUMN_NAME = 'client_name'
to_rounds_with_best_scores_df() → pandas.core.frame.DataFrame[source]

Get a dataframe with the best scores for each round as well as the metric used for that round.

class neuraxle.metaopt.data.reporting.RoundReport(dc: SubDataclassT)[source]

Bases: neuraxle.metaopt.data.reporting.BaseReport

ROUND_ID_COLUMN_NAME = 'round_number'
SUMMARY_STATUS_COLUMNS_NAME = 'status'
main_metric_name
get_best_trial(metric_name: str = None) → Optional[neuraxle.metaopt.data.reporting.TrialReport][source]

Return trial report with best score from all trials, provided that this trial has a score and was successful.

get_best_trial_id(metric_name: str = None) → Union[str, int, None][source]

Get best trial id from all trials. Will return None if there are no successful trial with such score.

get_best_hyperparams(metric_name: str = None) → neuraxle.hyperparams.space.HyperparameterSamples[source]

Get best hyperparams from all trials.

is_higher_score_better(metric_name: str = None) → bool[source]

Return true if higher score is better. If metric_name is None, the optimizer’s metric is taken.

get_n_val_splits()[source]

Finds the number of validation splits on record in this round’s first trial.

get_metric_names() → List[str][source]

Get the name of all metrics on record.

best_result_summary(metric_name: str = None, use_wildcards: bool = False) → Tuple[float, int, neuraxle.base.TrialStatus, OrderedDict[str, Any]][source]

Return the best result summary for the given metric, as the [score, trial_number, hyperparams_flat_dict].

summary(metric_name: str = None, use_wildcards: bool = False) → List[Tuple[float, int, OrderedDict[str, Any]]][source]

Get a summary of the round. Best score is first. Values in the returned triplet tuples are: (score, trial_number, hyperparams), sorted by score such that the best score is first.

get_all_hyperparams(as_flat: bool = True, use_wildcards: bool = False) → List[OrderedDict[str, Any]][source]

Get the list of hyperparams for all trials.

list_hyperparameters_wildcards(discard_singles=False) → List[str][source]

Returns a list of all the hyperparameters wildcards used in the round. Discarding singles would prune out the hyperparameters with values that never vary.

list_successful_avg_validation_scores() → List[float][source]

Returns a list of all the average validation scores on record, only for succesful trials.

successful_trials

Returns a list of all the succesful trials on record.

to_round_scatterplot_df(metric_name: str = None, wildcards_to_keep: List[str] = None) → Tuple[pandas.core.frame.DataFrame, pandas.core.frame.DataFrame][source]

Returns a dataframe with trial ids, the selected metric, and the wildcarded hyperparameters to keep.

to_scores_over_time_df(metric_name: str = None, wildcards_to_keep: List[str] = None) → pandas.core.frame.DataFrame[source]

Returns a dataframe with trial ids, epochs, the selected metric, and the wildcarded hyperparameters to keep.

class neuraxle.metaopt.data.reporting.TrialReport(dc: SubDataclassT)[source]

Bases: neuraxle.metaopt.data.reporting.BaseReport

TRIAL_ID_COLUMN_NAME = 'trial_number'
get_metric_names() → List[str][source]

Get the name of all metrics on record.

get_hyperparams() → neuraxle.hyperparams.space.RecursiveDict[source]

Get the hyperparameters of the trial.

is_success()[source]

Checks if the trial is successful from its dataclass record.

get_status() → neuraxle.base.TrialStatus[source]

Get the status of the trial.

are_all_splits_successful() → bool[source]

Return true if all splits are successful.

are_all_splits_failures() → bool[source]

Return true if all splits are failed.

get_avg_validation_score(metric_name: str, over_time=False) → Union[float, List[float], None][source]

Returns the average score for all validation splits’s best validation score for the specified scoring metric.

: param metric_name: The name of the metric to use. : param over_time: If true, return all the avg scores over time instead of the best avg score. : return: validation score

get_avg_n_epoch_to_best_validation_score(metric_name: str) → Optional[float][source]
is_higher_score_better(metric_name: str) → bool[source]
to_scores_over_time_df(metric_name: str) → pandas.core.frame.DataFrame[source]

Returns a dataframe with the trial id, the epoch, and the selected metric.

class neuraxle.metaopt.data.reporting.TrialSplitReport(dc: SubDataclassT)[source]

Bases: neuraxle.metaopt.data.reporting.BaseReport

TRIAL_SPLIT_ID_COLUMN_NAME = 'split_number'
get_hyperparams() → neuraxle.hyperparams.space.RecursiveDict[source]

Get the hyperparameters of the trial.

get_metric_names() → List[str][source]

List metric names that are the subdataclass’ keys.

is_success()[source]

Set trial status to success.

is_higher_score_better(metric_name: str) → bool[source]
to_scores_over_time_df(metric_name: str) → pandas.core.frame.DataFrame[source]

Returns a dataframe with the trial id, the epoch, and the selected metric.

_get_scores_over_time_train_val_df_list(metric_name: str) → List[Dict[str, Any]][source]
class neuraxle.metaopt.data.reporting.MetricResultsReport(dc: SubDataclassT)[source]

Bases: neuraxle.metaopt.data.reporting.BaseReport

METRIC_COLUMN_NAME = 'metric'
EPOCH_COLUMN_NAME = 'epoch'
TRAIN_VAL_COLUMN_NAME = 'phase'
metric_name
get_train_scores() → List[float][source]

Return the train scores’ values.

get_valid_scores() → List[float][source]

Return the validation scores for the given scoring metric.

get_final_validation_score() → float[source]

Return the latest validation score for the given scoring metric.

get_best_validation_score() → Optional[float][source]

Return the best validation score for the given scoring metric.

get_n_epochs_to_best_validation_score() → Optional[int][source]

Return the number of epochs

is_higher_score_better() → bool[source]

Return True if higher scores are better for the main metric.

is_new_best_score() → bool[source]

Return True if the latest validation score is the new best score.