neuraxle.steps.flow¶

Module-level documentation for neuraxle.steps.flow. Here is an inheritance diagram, including dependencies to other base modules of Neuraxle:

Neuraxle’s Flow Steps¶

Wrapper steps that apply some effect to the flow of data in the pipeline.

For instance, the following steps are wrappers of other steps and apply some control logic to the flow of data into the steps:

While,
ForEach,
BreakIf,
ContinueIf,
Try,
Catch (Except),
RaiseIf (Throw),
If & ElseIf & Else,
Switch

Classes

`ChooseOneOrManyStepsOf`(steps, joiner)	A pipeline to allow choosing many steps using an hyperparameter.
`ChooseOneStepOf`(steps[, hyperparams])
`ChooseStepElseIdentity`(wrapped, enabled[, …])
`ExecuteIf`(condition_function, wrapped)
`ExecutionPhaseSwitch`(phase_to_callable, …)
`ExpandDim`(wrapped)	Similar to numpys expand_dim function, ExpandDim step expands the dimension of all the data inside the data container.
`IfExecutionPhaseIsThen`(phase, wrapped, …)	If, at runtime, the execution phase is the same as the one given to the constructor, then execute wrapped step.
`OptionalStep`(wrapped, enabled[, …])	A wrapper to nullify a step: nullify its hyperparams, and also nullify all of his behavior.
`ReversiblePreprocessingWrapper`(…)	TruncableSteps with a preprocessing step(1), and a postprocessing step(2) that inverse transforms with the preprocessing step at the end (1, 2, reversed(1)).
`SelectNonEmptyDataContainer`()	A step that selects non empty data containers.
`SelectNonEmptyDataInputs`()	A step that selects non empty data inputs.
`TestOnlyWrapper`(wrapped)	A wrapper to run wrapped step only in test mode
`TrainOnlyWrapper`(wrapped)	A wrapper to run wrapped step only in train mode
`TrainOrTestOnlyWrapper`(wrapped[, …])	A wrapper to run wrapped step only in test mode, or only in train mode.

Examples using `neuraxle.steps.flow.ChooseOneStepOf`¶

class neuraxle.steps.flow.TrainOrTestOnlyWrapper(wrapped: neuraxle.base.BaseStep, is_train_only=True, cache_folder_when_no_handle=None)[source]¶

Bases: neuraxle.base.ForceHandleOnlyMixin, neuraxle.base.MetaStep

A wrapper to run wrapped step only in test mode, or only in train mode.

Execute only in test mode:

p = TrainOrTestOnlyWrapper(Identity(), is_train_only=True)

Execute only in train mode:

p = TrainOnlyWrapper(Identity(), test_only=False)

__init__(wrapped: neuraxle.base.BaseStep, is_train_only=True, cache_folder_when_no_handle=None)[source]¶: Initialize self. See help(type(self)) for accurate signature.

_fit_data_container(data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext) → neuraxle.base.BaseStep[source]¶

Return type

BaseStep

Parameters

data_container (DataContainer) – data container
context (ExecutionContext) – execution context

Returns

step, data_container

_fit_transform_data_container(data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext) → Tuple[neuraxle.base.BaseStep, neuraxle.data_container.DataContainer][source]¶

Parameters

data_container (DataContainer) – data container
context (ExecutionContext) – execution context

Returns

step, data_container

_transform_data_container(data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext) → neuraxle.data_container.DataContainer[source]¶

Return type

DataContainer

Parameters

data_container (DataContainer) – data container
context (ExecutionContext) – execution context

Returns

step, data_container

_should_execute_wrapped_step()[source]¶

_abc_impl = <_abc_data object>¶

class neuraxle.steps.flow.TrainOnlyWrapper(wrapped: neuraxle.base.BaseStep)[source]¶

Bases: neuraxle.steps.flow.TrainOrTestOnlyWrapper

A wrapper to run wrapped step only in train mode

Execute only in train mode:

p = Pipeline([
    TrainOnlyWrapper(Identity())
])

__init__(wrapped: neuraxle.base.BaseStep)[source]¶: Initialize self. See help(type(self)) for accurate signature.

_abc_impl = <_abc_data object>¶

class neuraxle.steps.flow.TestOnlyWrapper(wrapped: neuraxle.base.BaseStep)[source]¶

Bases: neuraxle.steps.flow.TrainOrTestOnlyWrapper

A wrapper to run wrapped step only in test mode

Execute only in train mode:

p = Pipeline([
    TestOnlyWrapper(Identity())
])

__init__(wrapped: neuraxle.base.BaseStep)[source]¶: Initialize self. See help(type(self)) for accurate signature.

_abc_impl = <_abc_data object>¶

class neuraxle.steps.flow.ExecuteIf(condition_function: Callable, wrapped: neuraxle.base.BaseStep)[source]¶

Bases: neuraxle.base.HandleOnlyMixin, neuraxle.base.MetaStep

__init__(condition_function: Callable, wrapped: neuraxle.base.BaseStep)[source]¶: Initialize self. See help(type(self)) for accurate signature.

_fit_data_container(data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext)[source]¶

Fit data container.

Parameters

data_container (DataContainer) – data container
context (ExecutionContext) – execution context

Returns

(fitted self, data container)

_fit_transform_data_container(data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext)[source]¶

Fit and transform data container with the given execution context. Will do:

data_container, context = self._fit_data_container(data_container, context)
data_container = self._transform_data_container(data_container, context)
return self, data_container

Parameters

data_container (DataContainer) – data container
context (ExecutionContext) – execution context

Returns

transformed data container

_transform_data_container(data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext)[source]¶

Transform data container.

Parameters

data_container (DataContainer) – data container
context (ExecutionContext) – execution context

Returns

data container

_abc_impl = <_abc_data object>¶

class neuraxle.steps.flow.IfExecutionPhaseIsThen(phase: neuraxle.base.ExecutionPhase, wrapped: neuraxle.base.BaseTransformer, raise_if_phase_unspecified: bool = True)[source]¶

Bases: neuraxle.steps.flow.ExecuteIf

If, at runtime, the execution phase is the same as the one given to the constructor, then execute wrapped step.

By default, will raise an error if the execution phase is not specified in the context. Steps which implement ForceHandleMixin create context with unspecified phase on fit, fit_transform and transform call.

__init__(phase: neuraxle.base.ExecutionPhase, wrapped: neuraxle.base.BaseTransformer, raise_if_phase_unspecified: bool = True)[source]¶: Initialize self. See help(type(self)) for accurate signature.

check_context(step, data_container, context: neuraxle.base.ExecutionContext)[source]¶

_abc_impl = <_abc_data object>¶

class neuraxle.steps.flow.ExecutionPhaseSwitch(phase_to_callable: Dict[neuraxle.base.ExecutionPhase, neuraxle.base.BaseTransformer], default: Optional[neuraxle.base.BaseTransformer] = None)[source]¶

Bases: neuraxle.base.HandleOnlyMixin, neuraxle.base.TruncableSteps

__init__(phase_to_callable: Dict[neuraxle.base.ExecutionPhase, neuraxle.base.BaseTransformer], default: Optional[neuraxle.base.BaseTransformer] = None)[source]¶: Initialize self. See help(type(self)) for accurate signature.

_get_step(context)[source]¶

_set_step(context, step)[source]¶

_fit_data_container(data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext) → neuraxle.base.BaseStep[source]¶

Fit data container.

Parameters

data_container (DataContainer) – data container
context (ExecutionContext) – execution context

Returns

(fitted self, data container)

_fit_transform_data_container(data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext) → Tuple[neuraxle.base.BaseTransformer, neuraxle.data_container.DataContainer][source]¶

Fit and transform data container with the given execution context. Will do:

data_container, context = self._fit_data_container(data_container, context)
data_container = self._transform_data_container(data_container, context)
return self, data_container

Parameters

data_container (DataContainer) – data container
context (ExecutionContext) – execution context

Returns

transformed data container

_transform_data_container(data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext) → Tuple[neuraxle.base.BaseTransformer, neuraxle.data_container.DataContainer][source]¶

Transform data container.

Parameters

data_container (DataContainer) – data container
context (ExecutionContext) – execution context

Returns

data container

_abc_impl = <_abc_data object>¶

class neuraxle.steps.flow.OptionalStep(wrapped: neuraxle.base.BaseTransformer, enabled: bool = True, nullified_return_value=None, cache_folder_when_no_handle=None, use_hyperparameter_space=True, nullify_hyperparams=True)[source]¶

Bases: neuraxle.base.ForceHandleOnlyMixin, neuraxle.base.MetaStep

A wrapper to nullify a step: nullify its hyperparams, and also nullify all of his behavior. Example usage:

p = Pipeline([
    OptionalStep(Identity(), enabled=True)
])

See also

MetaStep, AutoML,

__init__(wrapped: neuraxle.base.BaseTransformer, enabled: bool = True, nullified_return_value=None, cache_folder_when_no_handle=None, use_hyperparameter_space=True, nullify_hyperparams=True)[source]¶: Initialize self. See help(type(self)) for accurate signature.

_fit_data_container(data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext) → Tuple[neuraxle.base.BaseTransformer, neuraxle.data_container.DataContainer][source]¶

Nullify wrapped step hyperparams, and don’t fit the wrapped step.

Parameters

data_container (DataContainer) – data container
context (ExecutionContext) – execution context

Returns

step, data_container

_fit_transform_data_container(data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext) → Tuple[neuraxle.base.BaseTransformer, neuraxle.data_container.DataContainer][source]¶

Nullify wrapped step hyperparams, and don’t fit_transform the wrapped step.

Parameters

data_container (DataContainer) – data container
context (ExecutionContext) – execution context

Returns

step, data_container

_transform_data_container(data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext) → neuraxle.data_container.DataContainer[source]¶

Nullify wrapped step hyperparams, and don’t transform the wrapped step.

Return type

DataContainer

Parameters

data_container (DataContainer) – data container
context (ExecutionContext) – execution context

Returns

step, data_container

_nullify_hyperparams()[source]¶: Nullify wrapped step hyperparams using hyperparams_space.nullify().

_passtrough_dact(data_container)[source]¶

_abc_impl = <_abc_data object>¶

class neuraxle.steps.flow.ChooseStepElseIdentity(wrapped: neuraxle.base.BaseTransformer, enabled: bool = True, nullify_hyperparams=True)[source]¶

Bases: neuraxle.steps.flow.OptionalStep

__init__(wrapped: neuraxle.base.BaseTransformer, enabled: bool = True, nullify_hyperparams=True)[source]¶: Initialize self. See help(type(self)) for accurate signature.

_passtrough_dact(data_container)[source]¶

_abc_impl = <_abc_data object>¶

class neuraxle.steps.flow.ChooseOneStepOf(steps, hyperparams=None)[source]¶

Bases: neuraxle.union.FeatureUnion

CHOICE_HYPERPARAM = 'choice'¶

A pipeline to allow choosing one step using an hyperparameter.

Example usage :

p = Pipeline([
    ChooseOneStepOf([
        ('a', Identity()),
        ('b', Identity())
    ])
])
p.set_hyperparams({
    'ChooseOneOrManyStepsOf__choice': 'a',
})
# or
p.set_hyperparams({
    'ChooseOneStepOf': {
        'a': { 'enabled': True },
        'b': { 'enabled': False }
    }
})

See also

Pipeline Optional

__init__(steps, hyperparams=None)[source]¶: Create a feature union. :param steps_as_tuple: the NamedStepsList of steps to process in parallel and to join. :param joiner: What will be used to join the features. NumpyConcatenateInnerFeatures() is used by default. :param n_jobs: The number of jobs for the parallelized joblib.Parallel loop in fit and in transform. :param backend: The type of parallelization to do with joblib.Parallel. Possible values: “loky”, “multiprocessing”, “threading”, “dask” if you use dask, and more.

set_hyperparams(hyperparams: Union[neuraxle.hyperparams.space.HyperparameterSamples, dict])[source]¶

Set the step hyperparameters.

Example :

step.set_hyperparams(HyperparameterSamples({
    'learning_rate': 0.10
}))

Parameters: hyperparams – hyperparameters
Returns: self

Note

This is a recursive method that will call _set_hyperparams().

See also

HyperparameterSamples, :class:̀_HasChildrenMixin`, BaseStep.apply(), _HasChildrenMixin._apply(), _HasChildrenMixin._set_train()

_set_hyperparams(hyperparams: neuraxle.hyperparams.space.HyperparameterSamples) → neuraxle.hyperparams.space.HyperparameterSamples[source]¶

update_hyperparams(hyperparams: Union[Dict[KT, VT], neuraxle.hyperparams.space.HyperparameterSamples]) → neuraxle.base.BaseTransformer[source]¶

Update the step hyperparameters without removing the already-set hyperparameters. This can be useful to add more hyperparameters to the existing ones without flushing the ones that were already set.

Example :

step.set_hyperparams(HyperparameterSamples({
    'learning_rate': 0.10
    'weight_decay': 0.001
}))

step.update_hyperparams(HyperparameterSamples({
    'learning_rate': 0.01
}))

assert step.get_hyperparams()['learning_rate'] == 0.01
assert step.get_hyperparams()['weight_decay'] == 0.001

Parameters: hyperparams – hyperparameters
Returns: self

See also

HyperparameterSamples, :class:̀_HasChildrenMixin`, BaseStep.apply(), _HasChildrenMixin._apply(), _HasChildrenMixin._update_hyperparams()

_update_hyperparams(hyperparams: Union[neuraxle.hyperparams.space.HyperparameterSamples, dict])[source]¶

Set chosen step hyperparams.

Parameters: hyperparams (HyperparameterSamples) – hyperparams
Returns

_update_optional_hyperparams()[source]¶

_make_all_steps_optional()[source]¶: Wrap all steps with Optional wrapper.

_abc_impl = <_abc_data object>¶

class neuraxle.steps.flow.ChooseOneOrManyStepsOf(steps, joiner: neuraxle.base.NonFittableMixin = None)[source]¶

Bases: neuraxle.union.FeatureUnion

A pipeline to allow choosing many steps using an hyperparameter.

Example usage :

p = Pipeline([
    ChooseOneOrManyStepsOf([
        ('a', Identity()),
        ('b', Identity())
    ])
])
p.set_hyperparams({
    'ChooseOneOrManyStepsOf__a__enabled': True,
    'ChooseOneOrManyStepsOf__b__enabled': False
})
# or
p.set_hyperparams({
    'ChooseOneOrManyStepsOf': {
        'a': { 'enabled': True },
        'b': { 'enabled': False }
    }
})

See also

Pipeline Optional

__init__(steps, joiner: neuraxle.base.NonFittableMixin = None)[source]¶: Create a feature union. :type joiner: NonFittableMixin :param steps_as_tuple: the NamedStepsList of steps to process in parallel and to join. :param joiner: What will be used to join the features. NumpyConcatenateInnerFeatures() is used by default. :param n_jobs: The number of jobs for the parallelized joblib.Parallel loop in fit and in transform. :param backend: The type of parallelization to do with joblib.Parallel. Possible values: “loky”, “multiprocessing”, “threading”, “dask” if you use dask, and more.

_make_all_steps_optional()[source]¶: Wrap all steps with Optional wrapper.

_abc_impl = <_abc_data object>¶

class neuraxle.steps.flow.SelectNonEmptyDataInputs[source]¶

Bases: neuraxle.base.TransformHandlerOnlyMixin, neuraxle.base.BaseTransformer

A step that selects non empty data inputs.

See also

TransformHandlerOnlyMixin, BaseStep

__init__()[source]¶: Initialize self. See help(type(self)) for accurate signature.

_transform_data_container(data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext)[source]¶

Handle transform.

Parameters

data_container (DataContainer) – the data container to join
context (ExecutionContext) – execution context

Returns

transformed data container

_abc_impl = <_abc_data object>¶

class neuraxle.steps.flow.SelectNonEmptyDataContainer[source]¶

Bases: neuraxle.base.TransformHandlerOnlyMixin, neuraxle.base.BaseTransformer

A step that selects non empty data containers. Assumes that the given DataContainer contains a list of DataContainer as data_inputs.

See also

TransformHandlerOnlyMixin, BaseStep

__init__()[source]¶: Initialize self. See help(type(self)) for accurate signature.

_transform_data_container(data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext)[source]¶

Transform data container with the given execution context.

Parameters

data_container (DataContainer) – data container
context (ExecutionContext) – execution context

Returns

transformed data container

_abc_impl = <_abc_data object>¶

class neuraxle.steps.flow.ExpandDim(wrapped: neuraxle.base.BaseTransformer)[source]¶

Bases: neuraxle.base.MetaStep

Similar to numpys expand_dim function, ExpandDim step expands the dimension of all the data inside the data container. ExpandDim sends the expanded data container to the wrapped step. ExpandDim returns the transformed expanded dim reduced to its original shape (see reduce_dim()).

The wrapped step will receive a single current_id, data_input, and expected output:

The ids is a list of one element that contains a single summary id created from all of the current ids.
The data_inputs is a list of one element that contains the original expected outputs list.
The expected_outputs is a list of one element that contains the original expected outputs list.

See also

ForceAlwaysHandleMixin, MetaStepMixin, BaseStep BaseHasher ExpandedDataContainer

__init__(wrapped: neuraxle.base.BaseTransformer)[source]¶: Initialize self. See help(type(self)) for accurate signature.

_will_process(data_container, context)[source]¶: Apply side effects before any step method. :param data_container: data container :param context: execution context :return: (data container, execution context)

_did_process(data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext)[source]¶

Apply side effects after any step method.

Parameters

data_container (DataContainer) – data container
context (ExecutionContext) – execution context

Returns

(data container, execution context)

_abc_impl = <_abc_data object>¶

class neuraxle.steps.flow.ReversiblePreprocessingWrapper(preprocessing_step, postprocessing_step)[source]¶

Bases: neuraxle.base.HandleOnlyMixin, neuraxle.base.TruncableSteps

TruncableSteps with a preprocessing step(1), and a postprocessing step(2) that inverse transforms with the preprocessing step at the end (1, 2, reversed(1)).

Example usage :

step = ReversiblePreprocessingWrapper(
    preprocessing_step=MultiplyBy2(),
    postprocessing_step=Add10()
)

outputs = step.transform(np.array(range(5)))

assert np.array_equal(outputs, np.array([5, 6, 7, 8, 9]))

__init__(preprocessing_step, postprocessing_step)[source]¶: Initialize self. See help(type(self)) for accurate signature.

_fit_data_container(data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext) → neuraxle.steps.flow.ReversiblePreprocessingWrapper[source]¶

Handle fit by fitting preprocessing step, and postprocessing step.

Parameters

data_container (DataContainer) – data container to fit on
context (ExecutionContext) – execution context

Returns

self, data_container

Return type

(ReversiblePreprocessingWrapper, DataContainer)

_transform_data_container(data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext) → neuraxle.data_container.DataContainer[source]¶

According to the idiom of (1, 2, reversed(1)), we do this, in order:

1. Transform preprocessing step

2. Transform postprocessing step

reversed(1). Inverse transform preprocessing step

Return type

DataContainer

Parameters

data_container (DataContainer) – data container to transform
context (ExecutionContext) – execution context

Returns

data_container

_abc_impl = <_abc_data object>¶

_fit_transform_data_container(data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext) → Tuple[neuraxle.base.BaseStep, neuraxle.data_container.DataContainer][source]¶

According to the idiom of (1, 2, reversed(1)), we do this, in order:

1. Fit Transform preprocessing step

2. Fit Transform postprocessing step

reversed(1). Inverse transform preprocessing step

Parameters

data_container (DataContainer) – data container to transform
context (ExecutionContext) – execution context

Returns

(self, data_container)

neuraxle.steps.flow¶

Neuraxle’s Flow Steps¶

Examples using neuraxle.steps.flow.ChooseOneStepOf¶

Examples using `neuraxle.steps.flow.ChooseOneStepOf`¶