neuraxle.pipeline

Neuraxle’s Pipeline Classes

This is the core of Neuraxle’s pipelines. You can chain steps to call them one after an other.

Classes

Barrier(hyperparams, hyperparams_space, …)

A Barrier step to be used in a minibatch sequential pipeline.

BasePipeline(steps, BaseStep], BaseStep]])

CustomHandleMethodsMixin

CustomPipelineMixin

Boilerplate code for custom pipelines that only implements handle methods.

Joiner(batch_size)

A Special Barrier step that joins the transformed mini batches together with list.extend method.

MiniBatchSequentialPipeline(steps, …[, …])

Mini Batch Sequential Pipeline class to create a pipeline processing data inputs in batch.

Pipeline(steps, BaseStep], BaseStep]][, …])

Fits and transform steps

ResumablePipeline(steps, BaseStep], BaseStep]])

Fits and transform steps after latest checkpoint

class neuraxle.pipeline.Barrier(hyperparams: neuraxle.hyperparams.space.HyperparameterSamples = None, hyperparams_space: neuraxle.hyperparams.space.HyperparameterSpace = None, name: str = None, savers: List[neuraxle.base.BaseSaver] = None, hashers: List[neuraxle.base.BaseHasher] = None)[source]

A Barrier step to be used in a minibatch sequential pipeline. It forces all the data inputs to get to the barrier in a sub pipeline before going through to the next sub-pipeline.

p = MiniBatchSequentialPipeline([
    SomeStep(),
    SomeStep(),
    Barrier(), # must be a concrete Barrier ex: Joiner()
    SomeStep(),
    SomeStep(),
    Barrier(), # must be a concrete Barrier ex: Joiner()
], batch_size=10)
join_fit_transform(step: neuraxle.pipeline.Pipeline, data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext) → Tuple[Any, neuraxle.data_container.DataContainer][source]

Execute the given pipeline fit_transform() with the given data container, and execution context. :param step: truncable steps to execute :param data_container: data container :param context: execution context :return: (fitted step, transformed data container)

join_transform(step: neuraxle.base.TruncableSteps, data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext) → neuraxle.data_container.DataContainer[source]

Execute the given pipeline transform() with the given data container, and execution context. :param step: truncable steps to execute :type step: TruncableSteps :param data_container: data container :type data_container: DataContainer :param context: execution context :type context: ExecutionContext :return: transformed data container :rtype: DataContainer

class neuraxle.pipeline.BasePipeline(steps: List[Union[Tuple[str, BaseStep], BaseStep]])[source]
fit(data_inputs, expected_outputs=None) → neuraxle.pipeline.BasePipeline[source]

Fit step with the given data inputs, and expected outputs.

Parameters
  • data_inputs – data inputs

  • expected_outputs – expected outputs to fit on

Returns

fitted self

fit_transform(data_inputs, expected_outputs=None) -> ('BasePipeline', typing.Any)[source]

Fit, and transform step with the given data inputs, and expected outputs.

Parameters
  • data_inputs – data inputs

  • expected_outputs – expected outputs to fit on

Returns

(fitted self, tranformed data inputs)

transform(data_inputs)[source]

Transform given data inputs.

Parameters

data_inputs – data inputs

Returns

transformed data inputs

class neuraxle.pipeline.CustomHandleMethodsMixin[source]
fit_data_container(data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext) → neuraxle.base.BaseStep[source]
fit_transform_data_container(data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext) → Tuple[neuraxle.base.BaseStep, neuraxle.data_container.DataContainer][source]
handle_fit(data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext) → neuraxle.base.BaseStep[source]
handle_fit_transform(data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext) -> ('BaseStep', <class 'neuraxle.data_container.DataContainer'>)[source]
handle_transform(data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext) → neuraxle.data_container.DataContainer[source]
transform_data_container(data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext) → neuraxle.data_container.DataContainer[source]
class neuraxle.pipeline.CustomPipelineMixin[source]

Boilerplate code for custom pipelines that only implements handle methods.

fit(data_inputs, expected_outputs=None) → neuraxle.pipeline.Pipeline[source]
Parameters
  • data_inputs – the data input to fit on

  • expected_outputs – the expected data output to fit on

Returns

the pipeline itself

fit_transform(data_inputs, expected_outputs=None) -> ('Pipeline', typing.Any)[source]
Parameters
  • data_inputs – the data input to fit on

  • expected_outputs – the expected data output to fit on

Returns

the pipeline itself

transform(data_inputs: Any)[source]
Parameters

data_inputs – the data input to transform

Returns

transformed data inputs

class neuraxle.pipeline.Joiner(batch_size)[source]

A Special Barrier step that joins the transformed mini batches together with list.extend method.

join_fit_transform(step: neuraxle.pipeline.Pipeline, data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext) → Tuple[Any, neuraxle.data_container.DataContainer][source]

Concatenate the pipeline fit transform output of each batch of self.batch_size together. :param step: pipeline to fit transform on :type step: Pipeline :param data_container: data container to fit transform on :type data_container: DataContainer :param context: execution context :return: fitted self, transformed data inputs :rtype: Tuple[Any, DataContainer]

join_transform(step: neuraxle.pipeline.Pipeline, data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext) → neuraxle.data_container.DataContainer[source]

Concatenate the pipeline transform output of each batch of self.batch_size together. :param step: pipeline to transform on :type step: Pipeline :param data_container: data container to transform :type data_container: DataContainer :param context: execution context :return: transformed data container :rtype: DataContainer

class neuraxle.pipeline.MiniBatchSequentialPipeline(steps: List[Union[Tuple[str, BaseStep], BaseStep]], batch_size=None, cache_folder=None)[source]

Mini Batch Sequential Pipeline class to create a pipeline processing data inputs in batch.

Provide a default batch size :

sub_pipelines = [SomeStep()]
pipeline = MiniBatchSequentialPipeline(sub_pipelines, batch_size=32)

Or manually add a :class`Barrier` step to the mini batch sequential pipeline :

sub_pipelines = [SomeStep(), Joiner(32)]
pipeline = MiniBatchSequentialPipeline(sub_pipelines)
fit_data_container(data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext) → neuraxle.base.BaseStep[source]

Fit all sub pipelines splitted by the Barrier steps. :param data_container: data container to transform. :param context: execution context :return: data container

fit_transform_data_container(data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext) → Tuple[neuraxle.base.BaseStep, neuraxle.data_container.DataContainer][source]

Transform all sub pipelines splitted by the Barrier steps. :param data_container: data container to transform. :param context: execution context :return: data container

transform_data_container(data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext) → neuraxle.data_container.DataContainer[source]

Transform all sub pipelines splitted by the Barrier steps. :param data_container: data container to transform. :param context: execution context :return: data container

class neuraxle.pipeline.Pipeline(steps: List[Union[Tuple[str, BaseStep], BaseStep]], cache_folder='cache')[source]

Fits and transform steps

fit(data_inputs, expected_outputs=None) → neuraxle.pipeline.Pipeline[source]

After loading the last checkpoint, fit each pipeline steps

Parameters
  • data_inputs – the data input to fit on

  • expected_outputs – the expected data output to fit on

Returns

the pipeline itself

fit_data_container(data_container)[source]
fit_transform(data_inputs, expected_outputs=None) -> ('Pipeline', typing.Any)[source]

After loading the last checkpoint, fit transform each pipeline steps

Parameters
  • data_inputs – the data input to fit on

  • expected_outputs – the expected data output to fit on

Returns

the pipeline itself

fit_transform_data_container(data_container)[source]
flush_all_cache()[source]
inverse_transform(processed_outputs) → Any[source]

After transforming all data inputs, and obtaining a prediction, we can inverse transform the processed outputs

Parameters

processed_outputs – the forward transformed data input

Returns

backward transformed processed outputs

transform(data_inputs: Any)[source]

After loading the last checkpoint, transform each pipeline steps

Parameters

data_inputs – the data input to transform

Returns

transformed data inputs

transform_data_container(data_container: neuraxle.data_container.DataContainer)[source]
class neuraxle.pipeline.ResumablePipeline(steps: List[Union[Tuple[str, BaseStep], BaseStep]], cache_folder='cache')[source]

Fits and transform steps after latest checkpoint

should_resume(data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext) → bool[source]

Return True if the pipeline has a saved checkpoint that it can resume from

Parameters
  • context – execution context

  • data_container – the data container to resume

Returns

bool