neuraxle.pipeline

Neuraxle’s Pipeline Classes

This is the core of Neuraxle’s pipelines. You can chain steps to call them one after an other.

Classes

Barrier(hyperparams, hyperparams_space, …)

A Barrier step to be used in a minibatch sequential pipeline.

BasePipeline(steps, BaseStep], BaseStep]])

Joiner(batch_size)

A Special Barrier step that joins the transformed mini batches together with list.extend method.

MiniBatchSequentialPipeline(steps, …)

Mini Batch Sequential Pipeline class to create a pipeline processing data inputs in batch.

Pipeline(steps, BaseStep], BaseStep]][, …])

Fits and transform steps

ResumablePipeline(steps, BaseStep], BaseStep]])

Fits and transform steps after latest checkpoint

class neuraxle.pipeline.Barrier(hyperparams: neuraxle.hyperparams.space.HyperparameterSamples = None, hyperparams_space: neuraxle.hyperparams.space.HyperparameterSpace = None, name: str = None, savers: List[neuraxle.base.BaseSaver] = None, hashers: List[neuraxle.base.BaseHasher] = None)[source]

A Barrier step to be used in a minibatch sequential pipeline. It forces all the data inputs to get to the barrier in a sub pipeline before going through to the next sub-pipeline.

``` p = MiniBatchSequentialPipeline([

SomeStep(), SomeStep(), Barrier(), # must be a concrete Barrier ex: Joiner() SomeStep(), SomeStep(), Barrier(), # must be a concrete Barrier ex: Joiner()

], batch_size=10) ```

join_fit_transform(step: neuraxle.pipeline.Pipeline, data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext) → Tuple[Any, neuraxle.data_container.DataContainer][source]

Execute the given pipeline fit_transform() with the given data container, and execution context.

Parameters
Returns

(fitted step, transformed data container)

Return type

Tuple[‘Any’, DataContainer]

join_transform(step: neuraxle.base.TruncableSteps, data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext) → neuraxle.data_container.DataContainer[source]

Execute the given pipeline transform() with the given data container, and execution context.

Parameters
Returns

transformed data container

Return type

DataContainer

class neuraxle.pipeline.BasePipeline(steps: List[Union[Tuple[str, BaseStep], BaseStep]])[source]
fit(data_inputs, expected_outputs=None) → neuraxle.pipeline.BasePipeline[source]

Fit step with the given data inputs, and expected outputs.

Parameters
  • data_inputs – data inputs

  • expected_outputs – expected outputs to fit on

Returns

fitted self

Return type

BaseStep

fit_transform(data_inputs, expected_outputs=None) -> ('BasePipeline', typing.Any)[source]

Fit, and transform step with the given data inputs, and expected outputs.

Parameters
  • data_inputs – data inputs

  • expected_outputs – expected outputs to fit on

Returns

(fitted self, tranformed data inputs)

Return type

Tuple[BaseStep, Any]

transform(data_inputs)[source]

Transform given data inputs.

Parameters

data_inputs – data inputs

Returns

transformed data inputs

Return type

Any

class neuraxle.pipeline.Joiner(batch_size)[source]

A Special Barrier step that joins the transformed mini batches together with list.extend method.

join_fit_transform(step: neuraxle.pipeline.Pipeline, data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext) → Tuple[Any, neuraxle.data_container.DataContainer][source]

Concatenate the pipeline fit transform output of each batch of self.batch_size together.

Parameters
  • step (Pipeline) – pipeline to fit transform on

  • data_container (DataContainer) – data container to fit transform on

  • context – execution context

Returns

fitted self, transformed data inputs

Return type

Tuple[Any, DataContainer]

join_transform(step: neuraxle.pipeline.Pipeline, data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext) → neuraxle.data_container.DataContainer[source]

Concatenate the pipeline transform output of each batch of self.batch_size together.

Parameters
  • step (Pipeline) – pipeline to transform on

  • data_container (DataContainer) – data container to transform

  • context – execution context

Returns

transformed data container

Return type

DataContainer

class neuraxle.pipeline.MiniBatchSequentialPipeline(steps: List[Union[Tuple[str, BaseStep], BaseStep]])[source]

Mini Batch Sequential Pipeline class to create a pipeline processing data inputs in batch.

fit(data_inputs, expected_outputs=None) → neuraxle.pipeline.Pipeline[source]
Parameters
  • data_inputs – the data input to fit on

  • expected_outputs – the expected data output to fit on

Returns

the pipeline itself

fit_transform(data_inputs, expected_outputs=None) -> ('Pipeline', typing.Any)[source]
Parameters
  • data_inputs – the data input to fit on

  • expected_outputs – the expected data output to fit on

Returns

the pipeline itself

handle_fit(data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext) → Tuple[neuraxle.pipeline.MiniBatchSequentialPipeline, neuraxle.data_container.DataContainer][source]

Fit all sub pipelines splitted by the Barrier steps.

Parameters
  • data_container – data container to transform.

  • context – execution context

Returns

data container

handle_fit_transform(data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext) → Tuple[neuraxle.pipeline.MiniBatchSequentialPipeline, neuraxle.data_container.DataContainer][source]

Transform all sub pipelines splitted by the Barrier steps.

Parameters
  • data_container – data container to transform.

  • context – execution context

Returns

data container

handle_transform(data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext) → neuraxle.data_container.DataContainer[source]

Transform all sub pipelines splitted by the Barrier steps.

Parameters
  • data_container – data container to transform.

  • context – execution context

Returns

data container

transform(data_inputs: Any)[source]
Parameters

data_inputs – the data input to transform

Returns

transformed data inputs

class neuraxle.pipeline.Pipeline(steps: List[Union[Tuple[str, BaseStep], BaseStep]], cache_folder='cache')[source]

Fits and transform steps

fit(data_inputs, expected_outputs=None) → neuraxle.pipeline.Pipeline[source]

After loading the last checkpoint, fit each pipeline steps

Parameters
  • data_inputs – the data input to fit on

  • expected_outputs – the expected data output to fit on

Returns

the pipeline itself

fit_transform(data_inputs, expected_outputs=None) -> ('Pipeline', typing.Any)[source]

After loading the last checkpoint, fit transform each pipeline steps

Parameters
  • data_inputs – the data input to fit on

  • expected_outputs – the expected data output to fit on

Returns

the pipeline itself

flush_all_cache()[source]
inverse_transform(processed_outputs) → Any[source]

After transforming all data inputs, and obtaining a prediction, we can inverse transform the processed outputs

Parameters

processed_outputs – the forward transformed data input

Returns

backward transformed processed outputs

transform(data_inputs: Any)[source]

After loading the last checkpoint, transform each pipeline steps

Parameters

data_inputs – the data input to transform

Returns

transformed data inputs

class neuraxle.pipeline.ResumablePipeline(steps: List[Union[Tuple[str, BaseStep], BaseStep]], cache_folder='cache')[source]

Fits and transform steps after latest checkpoint

should_resume(data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext) → bool[source]

Return True if the pipeline has a saved checkpoint that it can resume from

Parameters
  • context – execution context

  • data_container – the data container to resume

Returns

bool