neuraxle.steps.flow

Module-level documentation for neuraxle.steps.flow. Here is an inheritance diagram, including dependencies to other base modules of Neuraxle:

Inheritance diagram of neuraxle.steps.flow

Neuraxle’s Flow Steps

Pipeline wrapper steps that only implement the handle methods, and don’t apply any transformation to the data.

Classes

ChooseOneOrManyStepsOf(steps, joiner)

A pipeline to allow choosing many steps using an hyperparameter.

ChooseOneStepOf(steps[, hyperparams])

ExecuteIf(condition_function, wrapped)

ExecutionPhaseSwitch(phase_to_callable, …)

ExpandDim(wrapped)

Similar to numpys expand_dim function, ExpandDim step expands the dimension of all the data inside the data container.

IfExecutionPhaseIsThen(phase, wrapped, …)

If, at runtime, the execution phase is the same as the one given to the constructor, then execute wrapped step.

Optional(wrapped, enabled[, …])

A wrapper to nullify a step : nullify its hyperparams, and also nullify all of his behavior.

ReversiblePreprocessingWrapper(…)

TruncableSteps with a preprocessing step(1), and a postprocessing step(2) that inverse transforms with the preprocessing step at the end (1, 2, reversed(1)).

SelectNonEmptyDataContainer()

A step that selects non empty data containers.

SelectNonEmptyDataInputs()

A step that selects non empty data inputs.

TestOnlyWrapper(wrapped)

A wrapper to run wrapped step only in test mode

TrainOnlyWrapper(wrapped)

A wrapper to run wrapped step only in train mode

TrainOrTestOnlyWrapper(wrapped[, …])

A wrapper to run wrapped step only in test mode, or only in train mode.

Examples using neuraxle.steps.flow.ChooseOneStepOf

Examples using neuraxle.steps.flow.ExpandDim


class neuraxle.steps.flow.TrainOrTestOnlyWrapper(wrapped: neuraxle.base.BaseStep, is_train_only=True, cache_folder_when_no_handle=None)[source]

Bases: neuraxle.base.ForceHandleOnlyMixin, neuraxle.base.MetaStep

A wrapper to run wrapped step only in test mode, or only in train mode.

Execute only in test mode:

p = TrainOrTestOnlyWrapper(Identity(), is_train_only=True)

Execute only in train mode:

p = TrainOnlyWrapper(Identity(), test_only=False)
__init__(wrapped: neuraxle.base.BaseStep, is_train_only=True, cache_folder_when_no_handle=None)[source]

Initialize self. See help(type(self)) for accurate signature.

_fit_data_container(data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext) → neuraxle.base.BaseStep[source]
Return type

BaseStep

Parameters
Returns

step, data_container

_fit_transform_data_container(data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext) -> ('BaseStep', <class 'neuraxle.data_container.DataContainer'>)[source]
Parameters
Returns

step, data_container

_transform_data_container(data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext) → neuraxle.data_container.DataContainer[source]
Return type

DataContainer

Parameters
Returns

step, data_container

should_resume(data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext) → bool[source]
_should_execute_wrapped_step()[source]
_abc_impl = <_abc_data object>
class neuraxle.steps.flow.TrainOnlyWrapper(wrapped: neuraxle.base.BaseStep)[source]

Bases: neuraxle.steps.flow.TrainOrTestOnlyWrapper

A wrapper to run wrapped step only in train mode

Execute only in train mode:

p = Pipeline([
    TrainOnlyWrapper(Identity())
])
__init__(wrapped: neuraxle.base.BaseStep)[source]

Initialize self. See help(type(self)) for accurate signature.

_abc_impl = <_abc_data object>
class neuraxle.steps.flow.TestOnlyWrapper(wrapped: neuraxle.base.BaseStep)[source]

Bases: neuraxle.steps.flow.TrainOrTestOnlyWrapper

A wrapper to run wrapped step only in test mode

Execute only in train mode:

p = Pipeline([
    TestOnlyWrapper(Identity())
])
__init__(wrapped: neuraxle.base.BaseStep)[source]

Initialize self. See help(type(self)) for accurate signature.

_abc_impl = <_abc_data object>
class neuraxle.steps.flow.ExecuteIf(condition_function: Callable, wrapped: neuraxle.base.BaseStep)[source]

Bases: neuraxle.base.HandleOnlyMixin, neuraxle.base.MetaStep

__init__(condition_function: Callable, wrapped: neuraxle.base.BaseStep)[source]

Initialize self. See help(type(self)) for accurate signature.

_fit_data_container(data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext)[source]

Fit data container.

Parameters
Returns

(fitted self, data container)

_fit_transform_data_container(data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext)[source]

Fit transform data container.

Parameters
Returns

(fitted self, data container)

_transform_data_container(data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext)[source]

Transform data container.

Parameters
Returns

data container

_abc_impl = <_abc_data object>
class neuraxle.steps.flow.IfExecutionPhaseIsThen(phase: neuraxle.base.ExecutionPhase, wrapped: neuraxle.base.BaseTransformer, raise_if_phase_unspecified: bool = True)[source]

Bases: neuraxle.steps.flow.ExecuteIf

If, at runtime, the execution phase is the same as the one given to the constructor, then execute wrapped step.

By default, will raise an error if the execution phase is not specified in the context. Steps which implement ForceHandleMixin create context with unspecified phase on fit, fit_transform and transform call.

__init__(phase: neuraxle.base.ExecutionPhase, wrapped: neuraxle.base.BaseTransformer, raise_if_phase_unspecified: bool = True)[source]

Initialize self. See help(type(self)) for accurate signature.

check_context(step, data_container, context: neuraxle.base.ExecutionContext)[source]
_abc_impl = <_abc_data object>
class neuraxle.steps.flow.ExecutionPhaseSwitch(phase_to_callable: Dict[neuraxle.base.ExecutionPhase, neuraxle.base.BaseTransformer], default: Optional[neuraxle.base.BaseTransformer] = None)[source]

Bases: neuraxle.base.HandleOnlyMixin, neuraxle.base.TruncableSteps

__init__(phase_to_callable: Dict[neuraxle.base.ExecutionPhase, neuraxle.base.BaseTransformer], default: Optional[neuraxle.base.BaseTransformer] = None)[source]

Initialize self. See help(type(self)) for accurate signature.

_get_step(context)[source]
_set_step(context, step)[source]
_fit_data_container(data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext) → neuraxle.base.BaseStep[source]

Fit data container.

Parameters
Returns

(fitted self, data container)

_fit_transform_data_container(data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext) -> ('BaseTransformer', <class 'neuraxle.data_container.DataContainer'>)[source]

Fit transform data container.

Parameters
Returns

(fitted self, data container)

_transform_data_container(data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext) -> ('BaseTransformer', <class 'neuraxle.data_container.DataContainer'>)[source]

Transform data container.

Parameters
Returns

data container

_abc_impl = <_abc_data object>
class neuraxle.steps.flow.Optional(wrapped: neuraxle.base.BaseTransformer, enabled: bool = True, nullified_return_value=None, cache_folder_when_no_handle=None, use_hyperparameter_space=True, nullify_hyperparams=True)[source]

Bases: neuraxle.base.ForceHandleOnlyMixin, neuraxle.base.MetaStep

A wrapper to nullify a step : nullify its hyperparams, and also nullify all of his behavior.

Example usage :

p = Pipeline([
    Optional(Identity(), enabled=True)
])
__init__(wrapped: neuraxle.base.BaseTransformer, enabled: bool = True, nullified_return_value=None, cache_folder_when_no_handle=None, use_hyperparameter_space=True, nullify_hyperparams=True)[source]

Initialize self. See help(type(self)) for accurate signature.

_fit_data_container(data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext) -> ('BaseStep', <class 'neuraxle.data_container.DataContainer'>)[source]

Nullify wrapped step hyperparams, and don’t fit the wrapped step.

Parameters
Returns

step, data_container

_fit_transform_data_container(data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext) -> ('BaseStep', <class 'neuraxle.data_container.DataContainer'>)[source]

Nullify wrapped step hyperparams, and don’t fit_transform the wrapped step.

Parameters
Returns

step, data_container

_transform_data_container(data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext) → neuraxle.data_container.DataContainer[source]

Nullify wrapped step hyperparams, and don’t transform the wrapped step.

Return type

DataContainer

Parameters
Returns

step, data_container

_nullify_hyperparams()[source]

Nullify wrapped step hyperparams using hyperparams_space.nullify().

_abc_impl = <_abc_data object>
class neuraxle.steps.flow.ChooseOneStepOf(steps, hyperparams=None)[source]

Bases: neuraxle.union.FeatureUnion

CHOICE_HYPERPARAM = 'choice'

A pipeline to allow choosing one step using an hyperparameter.

Example usage :

p = Pipeline([
    ChooseOneStepOf([
        ('a', Identity()),
        ('b', Identity())
    ])
])
p.set_hyperparams({
    'ChooseOneOrManyStepsOf__choice': 'a',
})
# or
p.set_hyperparams({
    'ChooseOneStepOf': {
        'a': { 'enabled': True },
        'b': { 'enabled': False }
    }
})

See also

Pipeline Optional

__init__(steps, hyperparams=None)[source]

Create a feature union. :param steps_as_tuple: the NamedTupleList of steps to process in parallel and to join. :param joiner: What will be used to join the features. NumpyConcatenateInnerFeatures() is used by default. :param n_jobs: The number of jobs for the parallelized joblib.Parallel loop in fit and in transform. :param backend: The type of parallelization to do with joblib.Parallel. Possible values: “loky”, “multiprocessing”, “threading”, “dask” if you use dask, and more.

set_hyperparams(hyperparams: Union[neuraxle.hyperparams.space.HyperparameterSamples, dict])[source]

Set chosen step hyperparams.

Parameters

hyperparams (HyperparameterSamples) – hyperparams

Returns

_update_hyperparams(hyperparams: Union[neuraxle.hyperparams.space.HyperparameterSamples, dict])[source]

Set chosen step hyperparams.

Parameters

hyperparams (HyperparameterSamples) – hyperparams

Returns

_update_optional_hyperparams()[source]
_make_all_steps_optional()[source]

Wrap all steps with Optional wrapper.

_abc_impl = <_abc_data object>
class neuraxle.steps.flow.ChooseOneOrManyStepsOf(steps, joiner: neuraxle.base.NonFittableMixin = None)[source]

Bases: neuraxle.union.FeatureUnion

A pipeline to allow choosing many steps using an hyperparameter.

Example usage :

p = Pipeline([
    ChooseOneOrManyStepsOf([
        ('a', Identity()),
        ('b', Identity())
    ])
])
p.set_hyperparams({
    'ChooseOneOrManyStepsOf__a__enabled': True,
    'ChooseOneOrManyStepsOf__b__enabled': False
})
# or
p.set_hyperparams({
    'ChooseOneOrManyStepsOf': {
        'a': { 'enabled': True },
        'b': { 'enabled': False }
    }
})

See also

Pipeline Optional

__init__(steps, joiner: neuraxle.base.NonFittableMixin = None)[source]

Create a feature union. :type joiner: NonFittableMixin :param steps_as_tuple: the NamedTupleList of steps to process in parallel and to join. :param joiner: What will be used to join the features. NumpyConcatenateInnerFeatures() is used by default. :param n_jobs: The number of jobs for the parallelized joblib.Parallel loop in fit and in transform. :param backend: The type of parallelization to do with joblib.Parallel. Possible values: “loky”, “multiprocessing”, “threading”, “dask” if you use dask, and more.

_make_all_steps_optional()[source]

Wrap all steps with Optional wrapper.

_abc_impl = <_abc_data object>
class neuraxle.steps.flow.SelectNonEmptyDataInputs[source]

Bases: neuraxle.base.TransformHandlerOnlyMixin, neuraxle.base.BaseTransformer

A step that selects non empty data inputs.

__init__()[source]

Initialize self. See help(type(self)) for accurate signature.

_transform_data_container(data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext)[source]

Handle transform.

Parameters
Returns

transformed data container

_abc_impl = <_abc_data object>
class neuraxle.steps.flow.SelectNonEmptyDataContainer[source]

Bases: neuraxle.base.TransformHandlerOnlyMixin, neuraxle.base.BaseTransformer

A step that selects non empty data containers. Assumes that the given DataContainer contains a list of DataContainer as data_inputs.

__init__()[source]

Initialize self. See help(type(self)) for accurate signature.

_transform_data_container(data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext)[source]

Transform data container with the given execution context.

Parameters
Returns

transformed data container

_abc_impl = <_abc_data object>
class neuraxle.steps.flow.ExpandDim(wrapped: neuraxle.base.BaseTransformer)[source]

Bases: neuraxle.base.ResumableStepMixin, neuraxle.base.MetaStep

Similar to numpys expand_dim function, ExpandDim step expands the dimension of all the data inside the data container. ExpandDim sends the expanded data container to the wrapped step. ExpandDim returns the transformed expanded dim reduced to its original shape (see reduce_dim()).

The wrapped step will receive a single current_id, data_input, and expected output:
  • The current_id is a list of one element that contains a single summary hash for all of the current ids.

  • The data_inputs is a list of one element that contains the original expected outputs list.

  • The expected_outputs is a list of one element that contains the original expected outputs list.

See also

ForceAlwaysHandleMixin, MetaStepMixin, BaseStep BaseHasher ExpandedDataContainer

__init__(wrapped: neuraxle.base.BaseTransformer)[source]

Initialize self. See help(type(self)) for accurate signature.

_will_process(data_container, context)[source]

Apply side effects before any step method. :param data_container: data container :param context: execution context :return: (data container, execution context)

_did_process(data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext)[source]

Apply side effects after any step method.

Parameters
Returns

(data container, execution context)

resume(data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext)[source]
should_resume(data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext) → bool[source]

Returns True if a step can be resumed with the given the data container and execution context. See Checkpoint class documentation for more details on how a resumable checkpoint works.

Return type

bool

Parameters
Returns

if we can resume

_abc_impl = <_abc_data object>
class neuraxle.steps.flow.ReversiblePreprocessingWrapper(preprocessing_step, postprocessing_step)[source]

Bases: neuraxle.base.HandleOnlyMixin, neuraxle.base.TruncableSteps

TruncableSteps with a preprocessing step(1), and a postprocessing step(2) that inverse transforms with the preprocessing step at the end (1, 2, reversed(1)).

Example usage :

step = ReversiblePreprocessingWrapper(
    preprocessing_step=MultiplyBy2(),
    postprocessing_step=Add10()
)

outputs = step.transform(np.array(range(5)))

assert np.array_equal(outputs, np.array([5, 6, 7, 8, 9]))
__init__(preprocessing_step, postprocessing_step)[source]

Initialize self. See help(type(self)) for accurate signature.

_fit_data_container(data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext) → neuraxle.steps.flow.ReversiblePreprocessingWrapper[source]

Handle fit by fitting preprocessing step, and postprocessing step.

Parameters
Returns

self, data_container

Return type

(ReversiblePreprocessingWrapper, DataContainer)

_transform_data_container(data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext) → neuraxle.data_container.DataContainer[source]

According to the idiom of (1, 2, reversed(1)), we do this, in order:

  • 1. Transform preprocessing step

  • 2. Transform postprocessing step

  • reversed(1). Inverse transform preprocessing step

Return type

DataContainer

Parameters
Returns

data_container

_abc_impl = <_abc_data object>
_fit_transform_data_container(data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext) -> ('BaseStep', <class 'neuraxle.data_container.DataContainer'>)[source]

According to the idiom of (1, 2, reversed(1)), we do this, in order:

  • 1. Fit Transform preprocessing step

  • 2. Fit Transform postprocessing step

  • reversed(1). Inverse transform preprocessing step

Parameters
Returns

(self, data_container)