neuraxle.base

Module-level documentation for neuraxle.base. Here is an inheritance diagram, including dependencies to other base modules of Neuraxle:

Inheritance diagram of neuraxle.base

Neuraxle’s Base Classes

This is the core of Neuraxle. They are worth noticing. Most classes inherit from these classes, composing them differently.

This package ensures proper respect of the Interface Segregation Principle (ISP), That is a SOLID principle of OOP programming suggesting to segregate interfaces. This is what is done here as the project gained in abstraction, and that the base classes needed to compose other base classes.

Functions

synchroneous_flow_method(log_any_func, Any])

Locking the lock of the flow to prevent race conditions in accessing the flow.

Classes

AssertExpectedOutputIsNone()

AssertExpectedOutputIsNoneMixin()

AssertExpectedOutputIsNoneStep()

AssertExpectedOutputIsNotNoneMixin()

AssertionMixin()

BaseSaver

Any saver must inherit from this one.

BaseService(config, VT], …)

Base class for all services registred into the ExecutionContext.

BaseStep(hyperparams, hyperparams_space, …)

Base class for a transformer step that can also be fitted.

BaseTransformer(hyperparams, …)

Base class for a pipeline step that can only be transformed.

CX

alias of neuraxle.base.ExecutionContext

ContextLock(lock)

ContextLock is a service that is used to lock the flow and repos when doing parallel processing.

DidProcessAssertionMixin()

EvaluableStepMixin()

A step that can be evaluated with the scoring functions.

ExecutionContext(root, flow, …)

Execution context object containing all of the pipeline hierarchy steps.

ExecutionMode

This enum defines the execution mode of a BaseStep.

ExecutionPhase

This enum defines the execution phase of a BaseStep.

Flow(context)

This is like a news feed for pipelines where you post (log) info.

ForceHandleIdentity()

An identity step which forces usage of handler methods.

ForceHandleMixin([cache_folder])

A step that automatically calls handle methods in the transform, fit, and fit_transform methods.

ForceHandleOnlyMixin([cache_folder])

A step that automatically calls handle methods in the transform, fit, and fit_transform methods.

FullDumpLoader(name[, stripped_saver])

Identity step that can load the full dump of a pipeline step.

GlobalServiceAssertionExecutorMixin()

Any step which inherit of this class will test globaly retrievable service assertion of itself and all its children on a will_process call.

GlobalyRetrievableServiceAssertionWrapper(…)

Is used to assert the presence of service at the start of the pipeline AND at execution time for a given step.

HandleOnlyMixin()

A pipeline step that only requires the implementation of handler methods :

Identity([name, savers])

A pipeline step that has no effect at all but to return the same data without changes.

IdentityHandlerMethodsMixin([cache_folder])

A step that has a default implementation for all handler methods.

JoblibStepSaver

Saver that can save, or load a step with joblib.load, and joblib.dump.

LocalServiceAssertionWrapper(wrapped, …)

Is used to assert the presence of service at execution time for a given step

MetaService(wrapped, config, name)

A service containing other services.

MetaServiceMixin(wrapped)

A mixin for services containing other services

MetaStep(wrapped, hyperparams, …)

MetaStepJoblibStepSaver()

Custom saver for meta step mixin.

MetaStepMixin(wrapped, savers)

A class to represent a step that wraps another step.

MixinForBaseService()

Any steps/transformers within a pipeline that inherits of this class should implement BaseStep/BaseTransformer and initialize it before any mixin.

MixinForBaseTransformer()

Any steps/transformers within a pipeline that inherits of this class should implement BaseStep/BaseTransformer and initialize it before any mixin.

NonFittableMixin()

A pipeline step that requires no fitting: fitting just returns self when called to do no action.

NonTransformableMixin()

A pipeline step that has no effect at all but to return the same data without changes.

StepWithContext(wrapped, context, …)

A step with context is a step that has an ExecutionContext as a pre-registered dependency.

TransformHandlerOnlyMixin()

A pipeline step that only requires the implementation of _transform_data_container.

TrialStatus

Enum of the possible states of a trial.

TruncableJoblibStepSaver()

Step saver for a TruncableSteps.

TruncableService(services, BaseServiceT] = None)

TruncableServiceMixin(services, …)

TruncableSteps(steps_as_tuple, …)

Step that contains multiple steps.

TruncableStepsMixin(steps_as_tuple, …)

A mixin for services that can be truncated.

WillProcessAssertionMixin()

Examples using neuraxle.base.BaseStep

Examples using neuraxle.base.ExecutionContext

Examples using neuraxle.base.ForceHandleMixin

Examples using neuraxle.base.Identity

Examples using neuraxle.base.MetaStep

Examples using neuraxle.base.NonFittableMixin

Examples using neuraxle.base.NonTransformableMixin


class neuraxle.base.BaseSaver[source]

Bases: abc.ABC

Any saver must inherit from this one. Some savers just save parts of objects, some save it all or what remains. Each :class`BaseStep` can potentially have multiple savers to make serialization possible.

See also

save(), load()

save_step(step: neuraxle.base.BaseTransformer, context: neuraxle.base.ExecutionContext) → neuraxle.base.BaseTransformer[source]

Save a step or a step’s parts using the execution context.

Parameters
  • step – step to save

  • context – execution context

  • save_savers

Returns

can_load(step: neuraxle.base.BaseTransformer, context: neuraxle.base.ExecutionContext)[source]

Returns true if we can load the given step with the given execution context.

Parameters
  • step – step to load

  • context – execution context to load from

Returns

load_step(step: neuraxle.base.BaseTransformer, context: neuraxle.base.ExecutionContext) → neuraxle.base.BaseTransformer[source]

Load step with execution context.

Parameters
  • step – step to load

  • context – execution context to load from

Returns

loaded base step

_abc_impl = <_abc_data object>
class neuraxle.base.JoblibStepSaver[source]

Bases: neuraxle.base.BaseSaver

Saver that can save, or load a step with joblib.load, and joblib.dump.

This saver is a good default saver when the object is already stripped out of things that would make it unserializable.

It is the default stripped_saver for the ExecutionContext. The stripped saver is the first to load the step, and the last to save the step. The saver receives a stripped version of the step so that it can be saved by joblib.

can_load(step: neuraxle.base.BaseTransformer, context: neuraxle.base.ExecutionContext) → bool[source]

Returns true if the given step has been saved with the given execution context.

Return type

bool

Parameters
  • step – step that might have been saved

  • context – execution context

Returns

if we can load the step with the given context

_get_step_path(context, step)[source]

Create step path for the given context.

Parameters
  • context – execution context

  • step – step to save, or load

Returns

path

save_step(step: neuraxle.base.BaseTransformer, context: neuraxle.base.ExecutionContext) → neuraxle.base.BaseTransformer[source]

Saved step stripped out of things that would make it unserializable.

Parameters
  • step – stripped step to save

  • context – execution context to save from

Returns

load_step(step: neuraxle.base.BaseTransformer, context: neuraxle.base.ExecutionContext) → neuraxle.base.BaseTransformer[source]

Load stripped step.

Parameters
  • step – stripped step to load

  • context – execution context to load from

Returns

_abc_impl = <_abc_data object>
class neuraxle.base.ExecutionMode[source]

Bases: enum.Enum

This enum defines the execution mode of a BaseStep. It is available in the ExecutionContext as execution_mode.

FIT_OR_FIT_TRANSFORM_OR_TRANSFORM = 'fit_or_fit_transform_or_transform'[source]
FIT_OR_FIT_TRANSFORM = 'fit_or_fit_transform'[source]
TRANSFORM = 'transform'[source]
FIT = 'fit'[source]
FIT_TRANSFORM = 'fit_transform'[source]
INVERSE_TRANSFORM = 'inverse_transform'[source]
class neuraxle.base.ExecutionPhase[source]

Bases: enum.Enum

This enum defines the execution phase of a BaseStep. It is available in the ExecutionContext as execution_mode.

UNSPECIFIED = None[source]
PRETRAIN = 'pretraining'[source]
TRAIN = 'training'[source]
VALIDATION = 'validation'[source]
TEST = 'test'[source]
PROD = 'production'[source]
class neuraxle.base.MixinForBaseService[source]

Bases: object

Any steps/transformers within a pipeline that inherits of this class should implement BaseStep/BaseTransformer and initialize it before any mixin. This class checks that its the case at initialization.

__init__()[source]

Initialize self. See help(type(self)) for accurate signature.

_ensure_baseservice_init_called()[source]

Assert that BaseTransformer’s init method has been called.

class neuraxle.base._RecursiveArguments(ra=None, args: Union[List[Any], List[neuraxle.hyperparams.space.RecursiveDict]] = None, kwargs: Union[Dict[str, Any], neuraxle.hyperparams.space.RecursiveDict] = None, current_level: int = 0)[source]

Bases: object

This class is used by apply(), and _HasChildrenMixin to pass the right arguments to steps with children.

Two types of arguments: - args: arguments that are not named - kwargs: arguments that are named

For the values of both args and kwargs, we use either values or recursive values: - value is not RecursiveDict: the value is replicated and passed to each sub step. - value is RecursiveDict: the value is sliced accordingly and decomposed into the next levels.

As a shorthand, if another _RecursiveArguments (ra) is passed as an argument, it is used almost as is to merge different ways of using ra: using a past ra, or else some args.

__init__(ra=None, args: Union[List[Any], List[neuraxle.hyperparams.space.RecursiveDict]] = None, kwargs: Union[Dict[str, Any], neuraxle.hyperparams.space.RecursiveDict] = None, current_level: int = 0)[source]

Initialize self. See help(type(self)) for accurate signature.

children_names() → List[str][source]

Return the names of the children steps.

Returns

list of children step names

class neuraxle.base._HasRecursiveMethods(name: str = None)[source]

Bases: object

An internal class to represent a step that has recursive methods. The apply apply() function is used to apply a method to a step and its children.

Example usage :

class _HasHyperparams:
    # ...
    def set_hyperparams(self, hyperparams: Union[HyperparameterSamples, Dict]) -> HyperparameterSamples:
        self.apply(method='_set_hyperparams', hyperparams=HyperparameterSamples(hyperparams))
        return self

    def _set_hyperparams(self, hyperparams: Union[HyperparameterSamples, Dict]) -> HyperparameterSamples:
        self._invalidate()
        hyperparams = HyperparameterSamples(hyperparams)
        self.hyperparams = hyperparams if len(hyperparams) > 0 else self.hyperparams
        return self.hyperparams

pipeline = Pipeline([
    SomeStep()
])

pipeline.set_hyperparams(HyperparameterSamples({
    'learning_rate': 0.1,
    'SomeStep__learning_rate': 0.05
}))
__init__(name: str = None)[source]

Initialize self. See help(type(self)) for accurate signature.

set_name(name: str) → neuraxle.base.BaseService[source]

Set the name of the service.

Parameters

name (str) – a string.

Returns

self

Note

A step name is in the keys of steps_as_tuple

get_name() → str[source]

Get the name of the service.

Returns

the name, a string.

Note

A step name is the same value as the one in the keys of Pipeline.steps_as_tuple

getattr(attr_name: str) → neuraxle.hyperparams.space.RecursiveDict[source]

Get an attribute of the service or step, if it exists, returned as a RecursiveDict.

Return type

RecursiveDict

Parameters

attr_name (str) – the name of the attribute to get in each step or service.

Returns

A RecursiveDict with terminal leafs like RecursiveDict({attr_name: getattr(self, attr_name)}).

_getattr(attr_name: str) → RecursiveDict[str, str][source]

Get an attribute if it exists, as a RecursiveDict({attr_name: getattr(self, attr_name)}).

get_step_by_name(name: str) → Optional[BaseServiceT][source]
apply(method: Union[str, Callable], ra: neuraxle.base._RecursiveArguments = None, *args, **kwargs) → neuraxle.hyperparams.space.RecursiveDict[source]

Apply a method to a step and its children.

Here is an apply usage example to invalidate each steps. This example comes from the saving logic:

# preparing to save steps and its nested children:
if full_dump:
    # initialize and invalidate steps to make sure that all steps will be saved

    def _initialize_if_needed(step):
        if not step.is_initialized:
            step._setup(context=context)
        if not step.is_initialized:
            raise NotImplementedError(f"The `setup` method of the following class "
                                    f"failed to set `self.is_initialized` to True: {step.__class__.__name__}.")
        return RecursiveDict()

    def _invalidate(step):
        step._invalidate()
        return RecursiveDict()

    self.apply(method=_initialize_if_needed)
    self.apply(method=_invalidate)

# save steps:
...

Here is another example. For instance, when setting the hyperparams space of a step, we use _set_hyperparams_space() to set the hyperparams of the step. The trick is that the space argument HyperparameterSpace is a recursive dict. The implementation is the same for setting the hyperparams and config of the step and its children, not only its space. The cool thing is that such hyperparameter spaces are recursive, inheriting from RecursiveDict. and applying recursive arguments to the step and its children with the _HasChildrenMixin.apply() of _HasChildrenMixin. Here is the implementation, using apply:

def set_hyperparams_space(self, hyperparams_space: HyperparameterSpace) -> 'BaseTransformer':
    self.apply(method='_set_hyperparams_space', hyperparams_space=HyperparameterSpace(hyperparams_space))
    return self

def _set_hyperparams_space(self, hyperparams_space: Union[Dict, HyperparameterSpace]) -> HyperparameterSpace:
    self._invalidate()
    self.hyperparams_space = HyperparameterSpace(hyperparams_space)
    return self.hyperparams_space
Return type

RecursiveDict

Parameters
  • method – method name that need to be called on all steps

  • ra (_RecursiveArguments) – recursive arguments

  • args – any additional arguments to be passed to the method

  • kwargs – any additional positional arguments to be passed to the method

Returns

method outputs, or None if no method has been applied

class neuraxle.base._HasConfig(config: Union[Dict[KT, VT], neuraxle.hyperparams.space.RecursiveDict] = None)[source]

Bases: abc.ABC

An internal class to represent a step that has config params. This is useful to store the config of a step.

A config RecursiveDict config attribute is used when you don’t want to use a HyperparameterSamples attribute. The reason sometimes is that you don’t want to tune your config, whereas hyperparameters are used to tune your step in the AutoML from hyperparameter spaces, such as using hyperopt.

A good example of a config parameter would be the number of threads, or an API key loaded from the OS’ environment variables, since they won’t be tuned but are changeable from the outside.

Thus, this class looks a lot like _HasHyperparams and HyperparameterSpace.

__init__(config: Union[Dict[KT, VT], neuraxle.hyperparams.space.RecursiveDict] = None)[source]

Initialize self. See help(type(self)) for accurate signature.

set_config(config: Union[Dict[KT, VT], neuraxle.hyperparams.space.RecursiveDict]) → neuraxle.base.BaseTransformer[source]

Set step config. See set_hyperparams() for more usage examples and documentation, it works the same way.

_set_config(config: neuraxle.hyperparams.space.RecursiveDict) → neuraxle.hyperparams.space.RecursiveDict[source]
update_config(config: Union[Dict[KT, VT], neuraxle.hyperparams.space.RecursiveDict]) → neuraxle.base.BaseTransformer[source]

Update the step config variables without removing the already-set config variables. This method is similar to update_hyperparams(). Refer to it for more documentation and usage examples, it works the same way.

_update_config(config: neuraxle.hyperparams.space.RecursiveDict) → neuraxle.hyperparams.space.RecursiveDict[source]
get_config() → neuraxle.hyperparams.space.RecursiveDict[source]

Get step config. Refer to get_hyperparams() for more documentation and usage examples, it works the same way.

_get_config() → neuraxle.hyperparams.space.RecursiveDict[source]
_abc_impl = <_abc_data object>
class neuraxle.base.BaseService(config: Union[Dict[KT, VT], neuraxle.hyperparams.space.RecursiveDict] = None, name: str = None)[source]

Bases: neuraxle.base._HasConfig, neuraxle.base._HasRecursiveMethods, abc.ABC

Base class for all services registred into the ExecutionContext.

__init__(config: Union[Dict[KT, VT], neuraxle.hyperparams.space.RecursiveDict] = None, name: str = None)[source]

Initialize self. See help(type(self)) for accurate signature.

_abc_impl = <_abc_data object>
class neuraxle.base._HasChildrenMixin[source]

Bases: neuraxle.base.MixinForBaseService, typing.Generic

Mixin to add behavior to the steps that have children (sub steps).

apply(method: Union[str, Callable], ra: neuraxle.base._RecursiveArguments = None, *args, **kwargs) → neuraxle.hyperparams.space.RecursiveDict[source]

Apply method to root, and children steps. Split the root, and children values inside the arguments of type RecursiveDict.

This method overrides the apply() method of _HasRecursiveMethods. Read the documentation of the original method to learn more.

Read more: Steps containing other steps.

Return type

RecursiveDict

Parameters
  • method – str or callable function to apply

  • ra (_RecursiveArguments) – recursive arguments

Returns

_apply_self(method: Union[str, Callable], ra: neuraxle.base._RecursiveArguments) → neuraxle.hyperparams.space.RecursiveDict[source]
_apply_childrens(results: neuraxle.hyperparams.space.RecursiveDict, method: Union[str, Callable], ra: neuraxle.base._RecursiveArguments) → neuraxle.hyperparams.space.RecursiveDict[source]
_validate_children_exists(ra)[source]

Validate that the provided childrens are in self, and if not, raise an error.

get_children() → List[BaseServiceT][source]

Get the list of all the childs for that step or service.

Returns

every child steps

get_children_names() → List[str][source]

Get the list of all the childs names for that step.

Returns

every child steps’ names

get_step_by_name(name: str) → Optional[BaseServiceT][source]
class neuraxle.base.MetaServiceMixin(wrapped: BaseServiceT)[source]

Bases: neuraxle.base._HasChildrenMixin

A mixin for services containing other services

__init__(wrapped: BaseServiceT)[source]

Initialize self. See help(type(self)) for accurate signature.

set_step(step: BaseServiceT) → BaseServiceT[source]

Set wrapped step to the given step.

Parameters

step – new wrapped step

Returns

self

get_step() → BaseServiceT[source]

Get wrapped step

Returns

self.wrapped

get_children() → List[BaseServiceT][source]

Get the list of all the childs for that step. _HasChildrenMixin calls this method to apply methods to all of the childs for that step.

Returns

list of child steps

_repr(level=0, verbose=False) → str[source]
class neuraxle.base.MetaService(wrapped: BaseServiceT = None, config: neuraxle.hyperparams.space.RecursiveDict = None, name: str = None)[source]

Bases: neuraxle.base.MetaServiceMixin, neuraxle.base.BaseService

A service containing other services.

__init__(wrapped: BaseServiceT = None, config: neuraxle.hyperparams.space.RecursiveDict = None, name: str = None)[source]

Initialize self. See help(type(self)) for accurate signature.

_abc_impl = <_abc_data object>
class neuraxle.base.TruncableServiceMixin(services: Dict[Union[str, Type[BaseServiceT]], BaseServiceT])[source]

Bases: neuraxle.base._HasChildrenMixin

__init__(services: Dict[Union[str, Type[BaseServiceT]], BaseServiceT])[source]

Initialize self. See help(type(self)) for accurate signature.

set_services(services: Dict[Union[str, Type[BaseServiceT]], BaseServiceT])[source]
_sanitize_service_name(service_name: Union[str, Type[BaseServiceT]]) → str[source]
register_service(service_type: Union[str, Type[BaseServiceT]], service_instance: BaseServiceT) → neuraxle.base.ExecutionContext[source]

Register base class instance inside the services. This is useful to register services. Make sure the service is an instance of the class BaseService.

Parameters
  • service_type – base type

  • service_instance – instance

Returns

self

get_services() → Dict[Union[str, Type[BaseServiceT]], BaseServiceT][source]

Get the registered instances in the services.

Returns

self

get_service(service_type: Union[str, Type[BaseServiceT]]) → object[source]

Get the registered instance for the given abstract class BaseService type. It is common to use service types as keys in the services dictionary.

Return type

object

Parameters

service_type – service type

Returns

self

has_service(service_type: Union[str, Type[BaseServiceT]]) → bool[source]

Return a bool indicating if the service has been registered.

Return type

bool

Parameters

service_type – base type

Returns

if the service registered or not

get_children() → List[BaseServiceT][source]

Get the list of all the childs for that step. _HasChildrenMixin calls this method to apply methods to all of the childs for that step.

Returns

list of child steps

class neuraxle.base.TruncableService(services: Dict[Type[BaseServiceT], BaseServiceT] = None)[source]

Bases: neuraxle.base.TruncableServiceMixin, neuraxle.base.BaseService

__init__(services: Dict[Type[BaseServiceT], BaseServiceT] = None)[source]

Initialize self. See help(type(self)) for accurate signature.

_abc_impl = <_abc_data object>
neuraxle.base.synchroneous_flow_method(log_any_func: Callable[[...], Any]) → Callable[source]

Locking the lock of the flow to prevent race conditions in accessing the flow.

class neuraxle.base.TrialStatus[source]

Bases: enum.Enum

Enum of the possible states of a trial.

PLANNED = 'PLANNED'[source]
RUNNING = 'RUNNING'[source]
ABORTED = 'ABORTED'[source]
FAILED = 'FAILED'[source]
SUCCESS = 'SUCCESS'[source]
class neuraxle.base.Flow(context: Optional[neuraxle.base.ExecutionContext] = None)[source]

Bases: neuraxle.base.BaseService

This is like a news feed for pipelines where you post (log) info. Flow is a step that can be used to store the status, metrics, logs, and other information of the execution of the current run.

Concrete implementations of this object may interact with repositories.

__init__(context: Optional[neuraxle.base.ExecutionContext] = None)[source]

Initialize self. See help(type(self)) for accurate signature.

Link the context to the flow with a weak ref.

Unlink the context from the flow.

logger
copy() → neuraxle.base.Flow[source]

Copy the flow.

log(message: str, level: int = 20, stacklevel=4)[source]
log_train_metric(metric_name: str, metric_value: float)[source]

Log training metric

log_valid_metric(metric_name: str, metric_value: float)[source]

Log validation metric

log_model(model: neuraxle.base.BaseStep)[source]
log_status(status: neuraxle.base.TrialStatus)[source]
log_planned(hps: neuraxle.hyperparams.space.HyperparameterSamples)[source]
log_hps(hps: neuraxle.hyperparams.space.HyperparameterSamples)[source]
log_start()[source]
log_epoch(epoch: int, n_epochs: int)[source]
log_end(status: neuraxle.base.TrialStatus = <TrialStatus.SUCCESS: 'SUCCESS'>)[source]
log_success()[source]
log_best_hps(main_metric_name, best_hps: neuraxle.hyperparams.space.HyperparameterSamples)[source]
log_failure(exception: Exception)[source]
log_error(exception: Exception)[source]
log_aborted(exception: Exception)[source]

Probably aborted with CTRL+C

_abc_impl = <_abc_data object>
class neuraxle.base.ContextLock(lock: multiprocessing.context.BaseContext.RLock = None)[source]

Bases: neuraxle.base.BaseService

ContextLock is a service that is used to lock the flow and repos when doing parallel processing.

__init__(lock: multiprocessing.context.BaseContext.RLock = None)[source]

Initialize self. See help(type(self)) for accurate signature.

synchroneous()[source]
copy()[source]
lock
_abc_impl = <_abc_data object>
class neuraxle.base.ExecutionContext(root: str = None, flow: neuraxle.base.Flow = None, execution_phase: neuraxle.base.ExecutionPhase = <ExecutionPhase.UNSPECIFIED: None>, execution_mode: neuraxle.base.ExecutionMode = <ExecutionMode.FIT_OR_FIT_TRANSFORM_OR_TRANSFORM: 'fit_or_fit_transform_or_transform'>, stripped_saver: neuraxle.base.BaseSaver = None, parents: List[BaseStep] = None, services: Dict[Union[str, Type[BaseServiceT]], BaseServiceT] = None)[source]

Bases: neuraxle.base.TruncableService

Execution context object containing all of the pipeline hierarchy steps. First item in execution context parents is root, second is nested, and so on. This is like a stack. It tracks the current step, the current phase, the current execution mode, and the current saver, as well as other information such as the current path and other caching information.

For instance, it is used in the handle_fit and handle_transform methods of the BaseStep as follows: handle_fit(), handle_transform().

This class can save and load steps using BaseSaver objects and the given root saving path.

Like a service locator, it is used to access some registered services to be made available to the pipeline at every step when they process data. If pipeline steps are composed like a tree, the execution context is used to pass information between steps. Thus, some domain services can be registered in the execution context, and then used by the pipeline steps.

One could design a lazy data loader that loads data only when needed, and have only the data IDs pass into the pipeline steps prior to hitting a step that needs the data and loads it when needed.

This way, a cache and several other contextual services can be used to store the data IDs and the data.

The AutoML class is an example of a step that uses this execution context extensively.

The execution context is used for fitted step saving:

See also

BaseStep, AutoML

__init__(root: str = None, flow: neuraxle.base.Flow = None, execution_phase: neuraxle.base.ExecutionPhase = <ExecutionPhase.UNSPECIFIED: None>, execution_mode: neuraxle.base.ExecutionMode = <ExecutionMode.FIT_OR_FIT_TRANSFORM_OR_TRANSFORM: 'fit_or_fit_transform_or_transform'>, stripped_saver: neuraxle.base.BaseSaver = None, parents: List[BaseStep] = None, services: Dict[Union[str, Type[BaseServiceT]], BaseServiceT] = None)[source]

Initialize self. See help(type(self)) for accurate signature.

logger
flow

Flow is a service that is used to log information about the execution of the pipeline.

static get_new_cache_folder() → str[source]
set_execution_phase(phase: neuraxle.base.ExecutionPhase) → neuraxle.base.ExecutionContext[source]

Set the instance’s execution phase to given phase from the enum ExecutionPhase.

Parameters

phase (ExecutionPhase) – execution phase

Returns

self

set_service_locator(services: Dict[Union[str, Type[BaseServiceT]], BaseServiceT]) → neuraxle.base.ExecutionContext[source]

Register abstract class type instances that inherit and implement the class BaseService.

Parameters

services – A dictionary of concrete services to register.

Returns

self

get_execution_mode() → neuraxle.base.ExecutionMode[source]

Get the instance’s execution mode from the enum ExecutionMode.

save(full_dump=True)[source]

Save all unsaved steps in the parents of the execution context using save(). This method is called from a step checkpointer inside a Checkpoint.

Parameters

full_dump (bool) – save full pipeline dump to be able to load everything without source code (false by default).

Returns

See also

BaseStep, save()

save_last()[source]

Save only the last step in the execution context.

See also

save()

should_save_last_step() → bool[source]

Returns True if the last step should be saved.

Returns

if the last step should be saved

pop_item() → neuraxle.base.BaseTransformer[source]

Change the execution context to be the same as the latest parent context.

Returns

pop() → bool[source]

Pop the context. Returns True if it successfully popped an item from the parents list.

Returns

if an item has been popped

push(step: neuraxle.base.BaseTransformer) → neuraxle.base.ExecutionContext[source]

Pushes a step in the parents of the execution context.

Parameters

step – step to add to the execution context

Returns

self

copy()[source]
_get_copy_kwargs()[source]
train() → neuraxle.base.ExecutionContext[source]

Set the context’s execution phase to train.

validation() → neuraxle.base.ExecutionContext[source]

Set the context’s execution phase to validation.

synchroneous() → multiprocessing.context.BaseContext.RLock[source]

Synchronous context: Create a managed reentrant lock (mutex).

thread_safe() → Tuple[multiprocessing.context.BaseContext.RLock, neuraxle.base.ExecutionContext][source]

Prepare the context and its services to be thread safe and reduce the current context for parallelization.

Returns

a tuple of the recursive dict to apply within thread, and the thread safe context

restore_lock(managed_thread_safe_lock: multiprocessing.context.BaseContext.RLock = None) → neuraxle.base.ExecutionContext[source]

Restore the lock from what was returned by self.thread_safe()[0].

Parameters

thread_safe_lock – lock to restore

Returns

self

peek() → neuraxle.base.BaseTransformer[source]

Get last parent.

Returns

the last parent base step

mkdir()[source]

Creates the directory to save the last parent step.

Returns

get_path(is_absolute: bool = True)[source]

Creates the directory path for the current execution context.

The returned context path is the concatenation of: - the root path, - the AutoML trial subpath info, - the parent steps subpath info.

Parameters

is_absolute (bool) – bool to say if we want to add root to the path or not

Returns

current context path

get_identifier(include_step_names: bool = True) → str[source]

Get an identifier depending on the ScopedLocation of the current context.

Useful for logging. Example:

Example: “neuraxle.default_project.default_client.0” + “.”.join(self.get_names())

See also

ScopedLocation

get_names() → List[str][source]

Returns a list of the parent names.

Returns

list of parents step names

empty()[source]

Return True if the context doesn’t have parent steps.

Returns

if parents len is 0

load(path: str) → neuraxle.base.BaseTransformer[source]

Load full dump at the given path.

Parameters

path (str) – pipeline step path

Returns

loaded step

to_identity() → neuraxle.base.ExecutionContext[source]

Create a fake execution context containing only identity steps. Create the parents by using the path of the current execution context.

Returns

fake identity execution context

flush_cache_root()[source]
flush_cache_local()[source]
_abc_impl = <_abc_data object>
neuraxle.base.CX[source]

alias of neuraxle.base.ExecutionContext

class neuraxle.base._HasSetupTeardownLifecycle[source]

Bases: neuraxle.base.MixinForBaseService

Step that has a setup and a teardown lifecycle methods.

Note

All heavy initialization logic should be done inside the setup method (e.g.: things inside GPU), and NOT in the constructor of your steps.

__init__()[source]

Initialize self. See help(type(self)) for accurate signature.

copy(context: neuraxle.base.ExecutionContext = None, deep=True) → neuraxle.base._HasSavers[source]

Copy the step.

Parameters

deep (bool) – if True, copy the savers as well

Returns

a copy of the step

setup(context: neuraxle.base.ExecutionContext) → neuraxle.base.BaseTransformer[source]

Initialize the step before it runs. Only from here and not before that heavy things should be created (e.g.: things inside GPU), and NOT in the constructor.

The _setup method is executed only if is self.is_initialized is False A setup function should set the self.is_initialized to True when called.

Warning

This setup method sets up the whole hierarchy of nested steps with children. If you want to setup progressively, use only self._setup() instead. The _setup method is called once for each step when handle_fit, handle_fit_transform or handle_transform is called.

Parameters

context – execution context

Returns

self

_setup(context: Optional[neuraxle.base.ExecutionContext] = None) → neuraxle.base.BaseTransformer[source]

Internal method to setup the step. May be used by Pipeline to setup the pipeline progressively instead of all at once.

teardown() → neuraxle.base.BaseTransformer[source]

Applies _teardown on the step and, if applicable, its children. :return: self

_teardown() → neuraxle.base.BaseTransformer[source]

Teardown step after program execution. Inverse of setup, and it should clear memory. Override this method if you need to clear memory.

Returns

self

class neuraxle.base._TransformerStep[source]

Bases: neuraxle.base.MixinForBaseService

An internal class to represent a step that can be transformed, or inverse transformed. See BaseTransformer, for the complete transformer step that can be used inside a neuraxle.pipeline.Pipeline. See BaseStep, for a step that can also be fitted inside a neuraxle.pipeline.Pipeline.

Every step must implement transform(). If a step is not transformable, you can inherit from NonTransformableMixin.

Every transformer step has handle methods that can be overridden to add side effects or change the execution flow based on the execution context, and the data container :
_will_process(data_container: neuraxle.data_container.DataContainer[~IDT, ~DIT, ~EOT][IDT, DIT, EOT], context: neuraxle.base.ExecutionContext) → Tuple[neuraxle.data_container.DataContainer[~IDT, ~DIT, ~EOT][IDT, DIT, EOT], neuraxle.base.ExecutionContext][source]

Apply side effects before any step method. :type context: ExecutionContext :param data_container: data container :param context: execution context :return: (data container, execution context)

handle_transform(data_container: neuraxle.data_container.DataContainer[~IDT, ~DIT, ~EOT][IDT, DIT, EOT], context: neuraxle.base.ExecutionContext) → neuraxle.data_container.DataContainer[~IDT, ~DIT, typing.Union[~EOT, NoneType]][IDT, DIT, Optional[EOT]][source]

Override this to add side effects or change the execution flow before (or after) calling * transform().

Parameters
  • data_container – the data container to transform

  • context (ExecutionContext) – execution context

Returns

transformed data container

_will_transform_data_container(data_container: neuraxle.data_container.DataContainer[~IDT, ~DIT, ~EOT][IDT, DIT, EOT], context: neuraxle.base.ExecutionContext) → Tuple[neuraxle.data_container.DataContainer[~IDT, ~DIT, ~EOT][IDT, DIT, EOT], neuraxle.base.ExecutionContext][source]

Apply side effects before transform.

Parameters
  • data_container – data container

  • context (ExecutionContext) – execution context

Returns

(data container, execution context)

_transform_data_container(data_container: neuraxle.data_container.DataContainer[~IDT, ~DIT, ~EOT][IDT, DIT, EOT], context: neuraxle.base.ExecutionContext) → neuraxle.data_container.DataContainer[~IDT, ~DIT, typing.Union[~EOT, NoneType]][IDT, DIT, Optional[EOT]][source]

Transform data container.

Parameters
  • data_container – data container

  • context (ExecutionContext) – execution context

Returns

data container

transform(data_inputs: DIT) → DIT[source]

Transform given data inputs.

Parameters

data_inputs – data inputs

Returns

transformed data inputs

_did_transform(data_container: neuraxle.data_container.DataContainer[~IDT, ~DIT, typing.Union[~EOT, NoneType]][IDT, DIT, Optional[EOT]], context: neuraxle.base.ExecutionContext) → neuraxle.data_container.DataContainer[~IDT, ~DIT, typing.Union[~EOT, NoneType]][IDT, DIT, Optional[EOT]][source]

Apply side effects after transform.

Parameters
  • data_container – data container

  • context (ExecutionContext) – execution context

Returns

data container

_did_process(data_container: neuraxle.data_container.DataContainer[~IDT, ~DIT, typing.Union[~EOT, NoneType]][IDT, DIT, Optional[EOT]], context: neuraxle.base.ExecutionContext) → neuraxle.data_container.DataContainer[~IDT, ~DIT, typing.Union[~EOT, NoneType]][IDT, DIT, Optional[EOT]][source]

Apply side effects after any step method.

Parameters
  • data_container – data container

  • context (ExecutionContext) – execution context

Returns

(data container, execution context)

handle_fit(data_container: neuraxle.data_container.DataContainer[~IDT, ~DIT, ~EOT][IDT, DIT, EOT], context: neuraxle.base.ExecutionContext) → neuraxle.base.BaseTransformer[source]

Override this to add side effects or change the execution flow before (or after) calling fit().

Parameters
  • data_container – the data container to transform

  • context (ExecutionContext) – execution context

Returns

tuple(fitted pipeline, data_container)

handle_fit_transform(data_container: neuraxle.data_container.DataContainer[~IDT, ~DIT, ~EOT][IDT, DIT, EOT], context: neuraxle.base.ExecutionContext) → Tuple[neuraxle.base.BaseTransformer, neuraxle.data_container.DataContainer[~IDT, ~DIT, typing.Union[~EOT, NoneType]][IDT, DIT, Optional[EOT]]][source]

Override this to add side effects or change the execution flow before (or after) calling * fit_transform().

Parameters
  • data_container – the data container to transform

  • context (ExecutionContext) – execution context

Returns

tuple(fitted pipeline, data_container)

fit(data_inputs: DIT, expected_outputs: EOT) → neuraxle.base._TransformerStep[source]

Fit given data inputs. By default, a step only transforms in the fit transform method. To add fitting to your step, see class:_FittableStep for more info. :param data_inputs: data inputs :param expected_outputs: expected outputs to fit on :return: transformed data inputs

fit_transform(data_inputs: DIT, expected_outputs: EOT = None) → Tuple[neuraxle.base._TransformerStep, DIT][source]

Fit transform given data inputs. By default, a step only transforms in the fit transform method. To add fitting to your step, see class:_FittableStep for more info.

Parameters
  • data_inputs – data inputs

  • expected_outputs – expected outputs to fit on

Returns

transformed data inputs

handle_predict(data_container: neuraxle.data_container.DataContainer[~IDT, ~DIT, ~EOT][IDT, DIT, EOT], context: neuraxle.base.ExecutionContext) → neuraxle.data_container.DataContainer[~IDT, ~DIT, typing.Union[~EOT, NoneType]][IDT, DIT, Optional[EOT]][source]

Handle_transform in test mode.

Parameters
  • data_container – the data container to transform

  • context (ExecutionContext) – execution context

Returns

transformed data container

predict(data_input: DIT) → DIT[source]

Predict the expected output in test mode using func:~neuraxle.base._TransformerStep.transform, but by setting self to test mode first and then reverting the mode.

Parameters

data_input – data input to predict

Returns

prediction

handle_inverse_transform(data_container: neuraxle.data_container.DataContainer[~IDT, ~DIT, typing.Union[~EOT, NoneType]][IDT, DIT, Optional[EOT]], context: neuraxle.base.ExecutionContext) → neuraxle.data_container.DataContainer[~IDT, ~DIT, ~EOT][IDT, DIT, EOT][source]

Override this to add side effects or change the execution flow before (or after) calling inverse_transform().

Parameters
  • data_container – the data container to inverse transform

  • context (ExecutionContext) – execution context

Returns

data_container

_inverse_transform_data_container(data_container: neuraxle.data_container.DataContainer[~IDT, ~DIT, typing.Union[~EOT, NoneType]][IDT, DIT, Optional[EOT]], context: neuraxle.base.ExecutionContext) → neuraxle.data_container.DataContainer[~IDT, ~DIT, ~EOT][IDT, DIT, EOT][source]
inverse_transform(processed_outputs: DIT) → DIT[source]

Inverse Transform the given transformed data inputs.

p = Pipeline([MultiplyByN(2)])
_in = np.array([1, 2])
_out = p.transform(_in)
_regenerated_in = p.inverse_transform(_out)
assert np.array_equal(_regenerated_in, _in)
assert np.array_equal(_out, _in * 2)
Parameters

processed_outputs – processed data inputs

Returns

inverse transformed processed outputs

class neuraxle.base._FittableStep[source]

Bases: neuraxle.base.MixinForBaseService

An internal class to represent a step that can be fitted. See BaseStep, for a complete step that can be transformed, and fitted inside a neuraxle.pipeline.Pipeline.

handle_fit(data_container: neuraxle.data_container.DataContainer[~IDT, ~DIT, ~EOT][IDT, DIT, EOT], context: neuraxle.base.ExecutionContext) → neuraxle.base.BaseStep[source]

Override this to add side effects or change the execution flow before (or after) calling fit().

Parameters
  • data_container – the data container to transform

  • context (ExecutionContext) – execution context

Returns

tuple(fitted pipeline, data_container)

_will_fit(data_container: neuraxle.data_container.DataContainer[~IDT, ~DIT, ~EOT][IDT, DIT, EOT], context: neuraxle.base.ExecutionContext) → Tuple[neuraxle.data_container.DataContainer[~IDT, ~DIT, ~EOT][IDT, DIT, EOT], neuraxle.base.ExecutionContext][source]

Before fit is called, apply side effects on the step, the data container, or the execution context.

Parameters
  • data_container – data container

  • context (ExecutionContext) – execution context

Returns

(data container, execution context)

_fit_data_container(data_container: neuraxle.data_container.DataContainer[~IDT, ~DIT, ~EOT][IDT, DIT, EOT], context: neuraxle.base.ExecutionContext) → neuraxle.base._FittableStep[source]

Fit data container.

Parameters
  • data_container – data container

  • context (ExecutionContext) – execution context

Returns

(fitted self, data container)

_did_fit(data_container: neuraxle.data_container.DataContainer[~IDT, ~DIT, ~EOT][IDT, DIT, EOT], context: neuraxle.base.ExecutionContext) → neuraxle.data_container.DataContainer[~IDT, ~DIT, ~EOT][IDT, DIT, EOT][source]

Apply side effects before fit is called.

Parameters
  • data_container – data container

  • context (ExecutionContext) – execution context

Returns

(data container, execution context)

fit(data_inputs: DIT, expected_outputs: EOT) → neuraxle.base._FittableStep[source]

Fit data inputs on the given expected outputs.

Parameters
  • data_inputs – data inputs

  • expected_outputs – expected outputs to fit on.

Returns

self

handle_fit_transform(data_container: neuraxle.data_container.DataContainer[~IDT, ~DIT, ~EOT][IDT, DIT, EOT], context: neuraxle.base.ExecutionContext) → Tuple[neuraxle.base.BaseStep, neuraxle.data_container.DataContainer[~IDT, ~DIT, typing.Union[~EOT, NoneType]][IDT, DIT, Optional[EOT]]][source]

Override this to add side effects or change the execution flow before (or after) calling * fit_transform().

Parameters
  • data_container – the data container to transform

  • context (ExecutionContext) – execution context

Returns

tuple(fitted pipeline, data_container)

_will_fit_transform(data_container: neuraxle.data_container.DataContainer[~IDT, ~DIT, ~EOT][IDT, DIT, EOT], context: neuraxle.base.ExecutionContext) → Tuple[neuraxle.data_container.DataContainer[~IDT, ~DIT, ~EOT][IDT, DIT, EOT], neuraxle.base.ExecutionContext][source]

Apply side effects before fit_transform is called.

Parameters
  • data_container – data container

  • context (ExecutionContext) – execution context

Returns

(data container, execution context)

_fit_transform_data_container(data_container: neuraxle.data_container.DataContainer[~IDT, ~DIT, ~EOT][IDT, DIT, EOT], context: neuraxle.base.ExecutionContext) → Tuple[neuraxle.base.BaseStep, neuraxle.data_container.DataContainer[~IDT, ~DIT, typing.Union[~EOT, NoneType]][IDT, DIT, Optional[EOT]]][source]

Fit transform data container.

Parameters
  • data_container – data container

  • context (ExecutionContext) – execution context

Returns

(fitted self, data container)

fit_transform(data_inputs, expected_outputs=None) → Tuple[neuraxle.base.BaseStep, Any][source]

Fit, and transform step with the given data inputs, and expected outputs.

Parameters
  • data_inputs – data inputs

  • expected_outputs – expected outputs to fit on

Returns

(fitted self, tranformed data inputs)

_did_fit_transform(data_container: neuraxle.data_container.DataContainer[~IDT, ~DIT, typing.Union[~EOT, NoneType]][IDT, DIT, Optional[EOT]], context: neuraxle.base.ExecutionContext) → neuraxle.data_container.DataContainer[~IDT, ~DIT, typing.Union[~EOT, NoneType]][IDT, DIT, Optional[EOT]][source]

Apply side effects after fit transform.

Parameters
  • data_container – data container

  • context (ExecutionContext) – execution context

Returns

(fitted self, data container)

class neuraxle.base._CustomHandlerMethods[source]

Bases: neuraxle.base.MixinForBaseService

A class to represent a step that needs to add special behavior on top of the normal handler methods. It allows the step to apply side effects before calling the real handler method.

Apply additional behavior (mini-batching, parallel processing, etc.) before calling the internal handler methods :
handle_fit(data_container: neuraxle.data_container.DataContainer[~IDT, ~DIT, ~EOT][IDT, DIT, EOT], context: neuraxle.base.ExecutionContext) → neuraxle.base.BaseStep[source]

Handle fit with a custom handler method for fitting the data container. The custom method to override is fit_data_container. The custom method fit_data_container replaces _fit_data_container.

Parameters
  • data_container – the data container to transform

  • context (ExecutionContext) – execution context

Returns

tuple(fitted pipeline, data_container)

See also

DataContainer, ExecutionContext

handle_fit_transform(data_container: neuraxle.data_container.DataContainer[~IDT, ~DIT, ~EOT][IDT, DIT, EOT], context: neuraxle.base.ExecutionContext) → Tuple[neuraxle.base.BaseStep, neuraxle.data_container.DataContainer[~IDT, ~DIT, typing.Union[~EOT, NoneType]][IDT, DIT, Optional[EOT]]][source]

Handle fit_transform with a custom handler method for fitting, and transforming the data container. The custom method to override is fit_transform_data_container. The custom method fit_transform_data_container replaces _fit_transform_data_container().

Parameters
  • data_container – the data container to transform

  • context (ExecutionContext) – execution context

Returns

tuple(fitted pipeline, data_container)

See also

DataContainer, ExecutionContext

handle_transform(data_container: neuraxle.data_container.DataContainer[~IDT, ~DIT, ~EOT][IDT, DIT, EOT], context: neuraxle.base.ExecutionContext) → neuraxle.data_container.DataContainer[~IDT, ~DIT, typing.Union[~EOT, NoneType]][IDT, DIT, Optional[EOT]][source]

Handle transform with a custom handler method for transforming the data container. The custom method to override is transform_data_container. The custom method transform_data_container replaces _transform_data_container().

Parameters
  • data_container – the data container to transform

  • context (ExecutionContext) – execution context

Returns

transformed data container

See also

DataContainer, ExecutionContext

fit_data_container(data_container: neuraxle.data_container.DataContainer[~IDT, ~DIT, ~EOT][IDT, DIT, EOT], context: neuraxle.base.ExecutionContext)[source]

Custom fit data container method.

Parameters
  • data_container – data container to fit on

  • context (ExecutionContext) – execution context

Returns

fitted self

fit_transform_data_container(data_container: neuraxle.data_container.DataContainer[~IDT, ~DIT, ~EOT][IDT, DIT, EOT], context: neuraxle.base.ExecutionContext)[source]

Custom fit transform data container method.

Parameters
  • data_container – data container to fit on

  • context (ExecutionContext) – execution context

Returns

fitted self, transformed data container

transform_data_container(data_container: neuraxle.data_container.DataContainer[~IDT, ~DIT, ~EOT][IDT, DIT, EOT], context: neuraxle.base.ExecutionContext)[source]

Custom transform data container method.

Parameters
  • data_container – data container to transform

  • context (ExecutionContext) – execution context

Returns

transformed data container

class neuraxle.base._HasHyperparamsSpace(hyperparams_space: neuraxle.hyperparams.space.HyperparameterSpace = None)[source]

Bases: neuraxle.base.MixinForBaseService

An internal class to represent a step that has hyperparameter spaces of type HyperparameterSpace. See BaseStep, for a complete step that can be transformed, and fitted inside a neuraxle.pipeline.Pipeline.

__init__(hyperparams_space: neuraxle.hyperparams.space.HyperparameterSpace = None)[source]

Initialize self. See help(type(self)) for accurate signature.

set_hyperparams_space(hyperparams_space: Union[Dict[KT, VT], neuraxle.hyperparams.space.HyperparameterSpace]) → neuraxle.base.BaseTransformer[source]

Set step hyperparameters space.

Example :

step.set_hyperparams_space(HyperparameterSpace({
    'hp': RandInt(0, 10)
}))
Parameters

hyperparams_space – hyperparameters space

Returns

self

Note

This is a recursive method that will call BaseStep._set_hyperparams_space() in the end.

See also

HyperparameterSamples, HyperparameterSpace, HyperparameterDistribution :class:̀_HasChildrenMixin`, BaseStep.apply(), _HasChildrenMixin._apply(), _HasChildrenMixin._get_params()

_set_hyperparams_space(hyperparams_space: neuraxle.hyperparams.space.HyperparameterSpace) → neuraxle.hyperparams.space.HyperparameterSpace[source]
update_hyperparams_space(hyperparams_space: Union[Dict[KT, VT], neuraxle.hyperparams.space.HyperparameterSpace]) → neuraxle.base.BaseTransformer[source]

Update the step hyperparameter spaces without removing the already-set hyperparameters. This can be useful to add more hyperparameter spaces to the existing ones without flushing the ones that were already set.

Example :

step.set_hyperparams_space(HyperparameterSpace({
    'learning_rate': LogNormal(0.5, 0.5)
    'weight_decay': LogNormal(0.001, 0.0005)
}))

step.update_hyperparams_space(HyperparameterSpace({
    'learning_rate': LogNormal(0.5, 0.1)
}))

assert step.get_hyperparams_space()['learning_rate'] == LogNormal(0.5, 0.1)
assert step.get_hyperparams_space()['weight_decay'] == LogNormal(0.001, 0.0005)
Parameters

hyperparams_space – hyperparameters space

Returns

self

Note

This is a recursive method that will call BaseStep._update_hyperparams_space() in the end.

See also

update_hyperparams(), HyperparameterSpace

_update_hyperparams_space(hyperparams_space: neuraxle.hyperparams.space.HyperparameterSpace) → neuraxle.hyperparams.space.HyperparameterSpace[source]
get_hyperparams_space() → neuraxle.hyperparams.space.HyperparameterSpace[source]

Get step hyperparameters space.

Example :

step.get_hyperparams_space()
Returns

step hyperparams space

Note

This is a recursive method that will call BaseStep._get_hyperparams_space() in the end.

_get_hyperparams_space() → neuraxle.hyperparams.space.HyperparameterSpace[source]
class neuraxle.base._HasHyperparams(hyperparams: neuraxle.hyperparams.space.HyperparameterSamples = None)[source]

Bases: neuraxle.base.MixinForBaseService

An internal class to represent a step that has hyperparameters of type HyperparameterSamples. See BaseStep, for a complete step that can be transformed, and fitted inside a neuraxle.pipeline.Pipeline.

Every step has hyperparemeters, and hyperparameters spaces that can be set before the learning process begins. Hyperparameters can not only be passed in the constructor, but also be set by the pipeline that contains all of the steps :

pipeline = Pipeline([
    SomeStep()
])

pipeline.set_hyperparams(HyperparameterSamples({
    'learning_rate': 0.1,
    'SomeStep__learning_rate': 0.05
}))
__init__(hyperparams: neuraxle.hyperparams.space.HyperparameterSamples = None)[source]

Initialize self. See help(type(self)) for accurate signature.

set_hyperparams(hyperparams: Union[Dict[KT, VT], neuraxle.hyperparams.space.HyperparameterSamples]) → neuraxle.base.BaseTransformer[source]

Set the step hyperparameters.

Example :

step.set_hyperparams(HyperparameterSamples({
    'learning_rate': 0.10
}))
Parameters

hyperparams – hyperparameters

Returns

self

Note

This is a recursive method that will call _set_hyperparams().

See also

HyperparameterSamples, :class:̀_HasChildrenMixin`, BaseStep.apply(), _HasChildrenMixin._apply(), _HasChildrenMixin._set_train()

_set_hyperparams(hyperparams: neuraxle.hyperparams.space.HyperparameterSamples) → neuraxle.hyperparams.space.HyperparameterSamples[source]
update_hyperparams(hyperparams: Union[Dict[KT, VT], neuraxle.hyperparams.space.HyperparameterSamples]) → neuraxle.base.BaseTransformer[source]

Update the step hyperparameters without removing the already-set hyperparameters. This can be useful to add more hyperparameters to the existing ones without flushing the ones that were already set.

Example :

step.set_hyperparams(HyperparameterSamples({
    'learning_rate': 0.10
    'weight_decay': 0.001
}))

step.update_hyperparams(HyperparameterSamples({
    'learning_rate': 0.01
}))

assert step.get_hyperparams()['learning_rate'] == 0.01
assert step.get_hyperparams()['weight_decay'] == 0.001
Parameters

hyperparams – hyperparameters

Returns

self

See also

HyperparameterSamples, :class:̀_HasChildrenMixin`, BaseStep.apply(), _HasChildrenMixin._apply(), _HasChildrenMixin._update_hyperparams()

_update_hyperparams(hyperparams: neuraxle.hyperparams.space.HyperparameterSamples) → neuraxle.hyperparams.space.HyperparameterSamples[source]
get_hyperparams() → neuraxle.hyperparams.space.HyperparameterSamples[source]

Get step hyperparameters as HyperparameterSamples.

Returns

step hyperparameters

Note

This is a recursive method that will call _get_hyperparams().

See also

HyperparameterSamples, :class:̀_HasChildrenMixin`, BaseStep.apply(), _HasChildrenMixin._apply(), _HasChildrenMixin._get_hyperparams()

_get_hyperparams() → neuraxle.hyperparams.space.HyperparameterSamples[source]
set_params(**params) → neuraxle.base.BaseTransformer[source]

Set step hyperparameters with a dictionary.

Example :

s.set_params(learning_rate=0.1)
hyperparams = s.get_params()
assert hyperparams == {"learning_rate": 0.1}

:param arbitrary number of arguments for hyperparameters

Note

This is a recursive method that will call _set_params() in the end.

See also

HyperparameterSamples, :class:̀_HasChildrenMixin`, apply(), _apply(), _set_params()

get_params(deep=False) → dict[source]

Get step hyperparameters as a flat primitive dict. The “deep” parameter is ignored.

Example :

s.set_params(learning_rate=0.1)
hyperparams = s.get_params()
assert hyperparams == {"learning_rate": 0.1}
Returns

hyperparameters

See also

HyperparameterSamples :class:̀_HasChildrenMixin`, apply(), apply(), _get_params()

class neuraxle.base._HasSavers(savers: List[neuraxle.base.BaseSaver] = None)[source]

Bases: neuraxle.base.MixinForBaseService

An internal class to represent a step that can be saved. A step with savers is saved using its list of savers. Each saver saves some parts of the step.

A pipeline can save the step that need to be saved (see save()) can be saved :

step = Pipeline([
    Identity()
])
step.save()
step = step.load()

Or, it can also save a full dump that can be reloaded without any source code :

step = Identity().set_name('step_name')
step.save(full_dump=True)
step = ExecutionContext().load('step_name')
__init__(savers: List[neuraxle.base.BaseSaver] = None)[source]

Initialize self. See help(type(self)) for accurate signature.

invalidate() → neuraxle.base.BaseTransformer[source]

Invalidate a step, and all of its children. Invalidating a step makes it eligible to be saved again.

A step is invalidated when any of the following things happen :
  • an hyperparameter has changed func: ~neuraxle.base._HasHyperparams.set_hyperparams

  • an hyperparameter space has changed func: ~neuraxle.base._HasHyperparamsSpace.set_hyperparams_space

  • a call to the fit method func:~neuraxle.base._FittableStep.handle_fit

  • a call to the fit_transform method func:~neuraxle.base._FittableStep.handle_fit_transform

  • the step name has changed func:~neuraxle.base.BaseStep.set_name

Returns

self

Note

This is a recursive method used in :class:̀_HasChildrenMixin`.

See also

apply(), _apply()

_invalidate()[source]
get_savers() → List[neuraxle.base.BaseSaver][source]

Get the step savers of a pipeline step.

Returns

step savers

See also

BaseSaver

set_savers(savers: List[neuraxle.base.BaseSaver]) → neuraxle.base.BaseTransformer[source]

Set the step savers of a pipeline step.

Returns

self

See also

BaseSaver

add_saver(saver: neuraxle.base.BaseSaver) → neuraxle.base.BaseTransformer[source]

Add a step saver of a pipeline step.

Returns

self

See also

BaseSaver

should_save() → bool[source]

Returns true if the step should be saved. If the step has been initialized and invalidated, then it must be saved.

A step is invalidated when any of the following things happen :

  • a mutation has been performed on the step mutate()

  • an hyperparameter has changed func:~neuraxle.base._HasHyperparams.set_hyperparams

  • an hyperparameter space has changed func:~neuraxle.base._HasHyperparamsSpace.set_hyperparams_space

  • a call to the fit method func:~neuraxle.base._FittableStep.handle_fit

  • a call to the fit_transform method func:~neuraxle.base._FittableStep.handle_fit_transform

  • the step name has changed func:~neuraxle.base.BaseStep.set_name

Returns

if the step should be saved

save(context: neuraxle.base.ExecutionContext, full_dump=True) → neuraxle.base.BaseTransformer[source]

Save step using the execution context to create the directory to save the step into. The saving happens by looping through all of the step savers in the reversed order.

Some savers just save parts of objects, some save it all or what remains. The ExecutionContext.stripped_saver has to be called last because it needs a stripped version of the step.

Parameters
  • context (ExecutionContext) – context to save from

  • full_dump (bool) – save full pipeline dump to be able to load everything without source code (false by default).

Returns

self

load(context: neuraxle.base.ExecutionContext, full_dump=True) → neuraxle.base.BaseTransformer[source]

Load step using the execution context to create the directory of the saved step. Warning:

Parameters
  • context (ExecutionContext) – execution context to load step from

  • full_dump (bool) – save full dump bool

Returns

loaded step

Warning

Please do not override this method because on loading it is an identity step that will load whatever step you coded.

_load_step(context, savers)[source]
class neuraxle.base._CouldHaveContext[source]

Bases: neuraxle.base.MixinForBaseService

Step that can have a context. It has “has service assertions” to ensure that the context has registered all the necessary services.

A context can be injected with the with_context method:

context = ExecutionContext(root=tmpdir)
service = SomeService()
context.set_service_locator({SomeBaseService: service})

p = Pipeline([
    SomeStep().assert_has_services(SomeBaseService)
]).with_context(context=context)

Or alternatively,

p = Pipeline([
    RegisterSomeService(),
    SomeStep().assert_has_services_at_execution(SomeBaseService)
])

Context services can be used inside any step with handler methods:

class SomeStep(ForceHandleMixin, Identity):
    def __init__(self):
        Identity.__init__(self)
        ForceHandleMixin.__init__(self)

    def _transform_data_container(self, data_container: DataContainer, context: ExecutionContext):
        service: SomeBaseService = context.get_service(SomeBaseService)
        service.service_method(data_container.data_inputs)
        return data_container
with_context(context: neuraxle.base.ExecutionContext) → neuraxle.base.StepWithContext[source]

An higher order step to inject a context inside a step. A step with a context forces the pipeline to use that context through handler methods. This is useful for dependency injection because you can register services inside the ExecutionContext. It also ensures that the context has registered all the necessary services.

context = ExecutionContext(tmpdir)
context.set_service_locator(ServiceLocator().services)
# where services is of type Dict[Type['BaseService'], 'BaseService']

p = WithContext(Pipeline([
    # When the context will be processing the SomeStep,
    # it will be asserted that the context will be able to access the SomeBaseService
    SomeStep().with_assertion_has_services(SomeBaseService)
]), context)
assert_has_services(*service_assertions) → neuraxle.base.GlobalyRetrievableServiceAssertionWrapper[source]

Set all service assertions to be made at the root of the pipeline and before processing the step.

Parameters

service_assertions (List[Type]) – base types that need to be available in the execution context at the root of the pipeline

assert_has_services_at_execution(*service_assertions) → neuraxle.base.LocalServiceAssertionWrapper[source]

Set all service assertions to be made before processing the step.

Parameters

service_assertions (List[Type]) – base types that need to be available in the execution context before the execution of the step

_assert_at_lifecycle(context: neuraxle.base.ExecutionContext)[source]

Assert that the context has all the services required to process the step. This method will be registred within a handler method’s _will_process or _did_process, or other lifecycle methods like these.

_assert(condition: bool, err_message: str, context: neuraxle.base.ExecutionContext = None)[source]

Assert that the condition is true. If not, raise an exception with the message. The exception will be logged with the logger in the context. If the context is in ExecutionPhase .PROD, the exception will not be raised and only logged.

it is good to call assertions here in a context-dependent way. For more information on contextual validation, read Martin Fowler’s article on Contextual Validation.

Parameters
  • condition (bool) – condition to assert

  • err_message (str) – message to log and raise if the condition is false

  • context (ExecutionContext) – execution context to log the exception, and not raise it if in PROD mode.

_assert_equals(a: Any, b: Any, err_message: str, context: neuraxle.base.ExecutionContext)[source]

Assert that the condition is true. If not, raise an exception with the message. The exception will be logged with the logger in the context. If the context is in ExecutionPhase .PROD, the exception will not be raised and only logged.

Parameters
  • a – element to compare to b with ==

  • b – element to compare to a with ==

  • err_message (str) – message to log and raise if the condition is false

  • context (ExecutionContext) – execution context to log the exception, and not raise it if in PROD mode.

class neuraxle.base.BaseTransformer(hyperparams: neuraxle.hyperparams.space.HyperparameterSamples = None, hyperparams_space: neuraxle.hyperparams.space.HyperparameterSpace = None, config: Union[Dict[KT, VT], neuraxle.hyperparams.space.RecursiveDict] = None, name: str = None, savers: List[neuraxle.base.BaseSaver] = None)[source]

Bases: neuraxle.base._CouldHaveContext, neuraxle.base._HasSavers, neuraxle.base._TransformerStep, neuraxle.base._HasHyperparamsSpace, neuraxle.base._HasHyperparams, neuraxle.base._HasSetupTeardownLifecycle, neuraxle.base.BaseService, abc.ABC

Base class for a pipeline step that can only be transformed.

Every step has hyperparemeters, and hyperparameters spaces that can be set before the learning process begins (see _HasHyperparams, and _HasHyperparamsSpace for more info).

Example usage :

class AddN(BaseTransformer):
    def __init__(self, add=1):
        super().__init__(hyperparams=HyperparameterSamples({'add': add}))

    def transform(self, data_inputs):
        if not isinstance(data_inputs, np.ndarray):
            data_inputs = np.array(data_inputs)

        return data_inputs + self.hyperparams['add']

    def inverse_transform(self, processed_outputs):
        if not isinstance(data_inputs, np.ndarray):
            data_inputs = np.array(data_inputs)

        return data_inputs - self.hyperparams['add']

Note

All heavy initialization logic should be done inside the setup method (e.g.: things inside GPU), and NOT in the constructor.

__init__(hyperparams: neuraxle.hyperparams.space.HyperparameterSamples = None, hyperparams_space: neuraxle.hyperparams.space.HyperparameterSpace = None, config: Union[Dict[KT, VT], neuraxle.hyperparams.space.RecursiveDict] = None, name: str = None, savers: List[neuraxle.base.BaseSaver] = None)[source]

Initialize self. See help(type(self)) for accurate signature.

set_train(is_train: bool = True)[source]

This method overrides the method of BaseStep to also consider the wrapped step as well as self. Set pipeline step mode to train or test.

Note

This is a recursive method used in :class:̀_HasChildrenMixin`.

See also

apply(), _apply() _set_train()

_set_train(is_train) → neuraxle.hyperparams.space.RecursiveDict[source]
_repr(level=0, verbose=False) → str[source]
_abc_impl = <_abc_data object>
neuraxle.base._sklearn_to_neuraxle_step(step) → neuraxle.base.BaseTransformer[source]
class neuraxle.base.BaseStep(hyperparams: neuraxle.hyperparams.space.HyperparameterSamples = None, hyperparams_space: neuraxle.hyperparams.space.HyperparameterSpace = None, config: Union[Dict[KT, VT], neuraxle.hyperparams.space.RecursiveDict] = None, name: str = None, savers: List[neuraxle.base.BaseSaver] = None)[source]

Bases: neuraxle.base._FittableStep, neuraxle.base.BaseTransformer, abc.ABC

Base class for a transformer step that can also be fitted.

If a step is not fittable, you can inherit from BaseTransformer instead. If a step is not transformable, you can inherit from NonTransformableMixin. A step should only change its state inside fit() or fit_transform() (see _FittableStep for more info). Every step has hyperparemeters, and hyperparameters spaces that can be set before the learning process begins (see _HasHyperparams, and _HasHyperparamsSpace for more info).

Example usage :

class Normalize(BaseStep):
    def __init__(self):
        BaseStep.__init__(self)
        self.mean = None
        self.std = None

    def fit(
        self, data_inputs: ARG_X_INPUTTED, expected_outputs: ARG_Y_EXPECTED = None
    ) -> 'BaseStep':
        self._calculate_mean_std(data_inputs)
        return self

    def _calculate_mean_std(self, data_inputs: ARG_X_INPUTTED):
        self.mean = np.array(data_inputs).mean(axis=0)
        self.std = np.array(data_inputs).std(axis=0)

    def fit_transform(
        self, data_inputs: ARG_X_INPUTTED, expected_outputs: ARG_Y_EXPECTED = None
    ) -> Tuple['BaseStep', ARG_Y_PREDICTD]:
        self.fit(data_inputs, expected_outputs)
        return self, (np.array(data_inputs) - self.mean) / self.std

    def transform(self, data_inputs: ARG_X_INPUTTED) -> ARG_Y_PREDICTD:
        if self.mean is None or self.std is None:
            self._calculate_mean_std(data_inputs)
        return (np.array(data_inputs) - self.mean) / self.std

p = Pipeline([
    Normalize()
])

p, outputs = p.fit_transform(data_inputs, expected_outputs)
_abc_impl = <_abc_data object>
class neuraxle.base.MixinForBaseTransformer[source]

Bases: object

Any steps/transformers within a pipeline that inherits of this class should implement BaseStep/BaseTransformer and initialize it before any mixin. This class checks that its the case at initialization.

__init__()[source]

Initialize self. See help(type(self)) for accurate signature.

_ensure_basetransformer_init_called()[source]

Assert that BaseTransformer’s init method has been called.

class neuraxle.base.MetaStepMixin(wrapped: BaseServiceT = None, savers: List[neuraxle.base.BaseSaver] = None)[source]

Bases: neuraxle.base.MixinForBaseTransformer, neuraxle.base.MetaServiceMixin

A class to represent a step that wraps another step. It can be used for many things. For example, ForEachDataInput adds a loop before any calls to the wrapped step :

class ForEachDataInput(MetaStepMixin, BaseStep):
    def __init__(
        self,
        wrapped: BaseStep
    ):
        BaseStep.__init__(self)
        MetaStepMixin.__init__(self, wrapped)

    def fit(self, data_inputs: ARG_X_INPUTTED, expected_outputs: ARG_Y_EXPECTED = None) -> 'BaseStep':
        if expected_outputs is None:
            expected_outputs = [None] * len(data_inputs)
        for di, eo in zip(data_inputs, expected_outputs):
            self.wrapped = self.wrapped.fit(di, eo)
        return self

    def transform(self, data_inputs: ARG_X_INPUTTED) -> ARG_Y_PREDICTD:
        outputs = []
        for di in data_inputs:
            output = self.wrapped.transform(di)
            outputs.append(output)
        return outputs

    def fit_transform(
        self, data_inputs: ARG_X_INPUTTED, expected_outputs: ARG_Y_EXPECTED = None
    ) -> Tuple['BaseStep', ARG_Y_PREDICTD]:
        if expected_outputs is None:
            expected_outputs = [None] * len(data_inputs)
        outputs = []
        for di, eo in zip(data_inputs, expected_outputs):
            self.wrapped, output = self.wrapped.fit_transform(di, eo)
        outputs.append(output)
        return self, outputs

See also

ForEachDataInput, StepClonerForEachDataInput

__init__(wrapped: BaseServiceT = None, savers: List[neuraxle.base.BaseSaver] = None)[source]

Initialize self. See help(type(self)) for accurate signature.

handle_fit_transform(data_container: neuraxle.data_container.DataContainer[~IDT, ~DIT, ~EOT][IDT, DIT, EOT], context: neuraxle.base.ExecutionContext) → Tuple[neuraxle.base.BaseStep, neuraxle.data_container.DataContainer[~IDT, ~DIT, typing.Union[~EOT, NoneType]][IDT, DIT, Optional[EOT]]][source]
handle_transform(data_container: neuraxle.data_container.DataContainer[~IDT, ~DIT, ~EOT][IDT, DIT, EOT], context: neuraxle.base.ExecutionContext) → neuraxle.data_container.DataContainer[~IDT, ~DIT, typing.Union[~EOT, NoneType]][IDT, DIT, Optional[EOT]][source]
_fit_transform_data_container(data_container: neuraxle.data_container.DataContainer[~IDT, ~DIT, ~EOT][IDT, DIT, EOT], context: neuraxle.base.ExecutionContext) → Tuple[neuraxle.base.BaseStep, neuraxle.data_container.DataContainer[~IDT, ~DIT, typing.Union[~EOT, NoneType]][IDT, DIT, Optional[EOT]]][source]
_fit_data_container(data_container: neuraxle.data_container.DataContainer[~IDT, ~DIT, ~EOT][IDT, DIT, EOT], context: neuraxle.base.ExecutionContext) → neuraxle.base.BaseStep[source]
_transform_data_container(data_container: neuraxle.data_container.DataContainer[~IDT, ~DIT, ~EOT][IDT, DIT, EOT], context: neuraxle.base.ExecutionContext) → neuraxle.data_container.DataContainer[~IDT, ~DIT, typing.Union[~EOT, NoneType]][IDT, DIT, Optional[EOT]][source]
_inverse_transform_data_container(data_container: neuraxle.data_container.DataContainer[~IDT, ~DIT, typing.Union[~EOT, NoneType]][IDT, DIT, Optional[EOT]], context: neuraxle.base.ExecutionContext) → neuraxle.data_container.DataContainer[~IDT, ~DIT, ~EOT][IDT, DIT, EOT][source]
fit_transform(data_inputs: DIT, expected_outputs: EOT = None) → Tuple[neuraxle.base.BaseStep, DIT][source]
fit(data_inputs: DIT, expected_outputs: EOT = None) → neuraxle.base.BaseStep[source]
transform(data_inputs: DIT) → DIT[source]
inverse_transform(processed_outputs: DIT) → DIT[source]
class neuraxle.base.MetaStep(wrapped: BaseServiceT = None, hyperparams: neuraxle.hyperparams.space.HyperparameterSamples = None, hyperparams_space: neuraxle.hyperparams.space.HyperparameterSpace = None, name: str = None, savers: List[neuraxle.base.BaseSaver] = None)[source]

Bases: neuraxle.base.MetaStepMixin, neuraxle.base.BaseStep

__init__(wrapped: BaseServiceT = None, hyperparams: neuraxle.hyperparams.space.HyperparameterSamples = None, hyperparams_space: neuraxle.hyperparams.space.HyperparameterSpace = None, name: str = None, savers: List[neuraxle.base.BaseSaver] = None)[source]

Initialize self. See help(type(self)) for accurate signature.

_abc_impl = <_abc_data object>
class neuraxle.base.MetaStepJoblibStepSaver[source]

Bases: neuraxle.base.JoblibStepSaver

Custom saver for meta step mixin.

__init__()[source]

Initialize self. See help(type(self)) for accurate signature.

save_step(step: neuraxle.base.MetaStep, context: neuraxle.base.ExecutionContext) → neuraxle.base.MetaStep[source]

Save MetaStepMixin.

# . Save wrapped step. # . Strip wrapped step form the meta step mixin. # . Save meta step with wrapped step savers.

Return type

MetaStep

Parameters
Returns

load_step(step: neuraxle.base.MetaStep, context: neuraxle.base.ExecutionContext) → neuraxle.base.MetaStep[source]

Load MetaStepMixin.

# . Loop through all of the sub steps savers, and only load the sub steps that have been saved. # . Refresh steps

Parameters
Returns

loaded truncable steps

_abc_impl = <_abc_data object>
class neuraxle.base.NonFittableMixin[source]

Bases: neuraxle.base.MixinForBaseTransformer

A pipeline step that requires no fitting: fitting just returns self when called to do no action. Note: fit methods are not implemented

_fit_data_container(data_container: neuraxle.data_container.DataContainer[~IDT, ~DIT, ~EOT][IDT, DIT, EOT], context: neuraxle.base.ExecutionContext) → neuraxle.base.BaseStep[source]
_fit_transform_data_container(data_container: neuraxle.data_container.DataContainer[~IDT, ~DIT, ~EOT][IDT, DIT, EOT], context: neuraxle.base.ExecutionContext) → Tuple[neuraxle.base.BaseStep, neuraxle.data_container.DataContainer[~IDT, ~DIT, typing.Union[~EOT, NoneType]][IDT, DIT, Optional[EOT]]][source]
fit(data_inputs: DIT, expected_outputs: EOT = None) → neuraxle.base.BaseStep[source]

Don’t fit.

Return type

BaseStep

Parameters
  • data_inputs – the data that would normally be fitted on.

  • expected_outputs – the data that would normally be fitted on.

Returns

self

class neuraxle.base.NonTransformableMixin[source]

Bases: neuraxle.base.MixinForBaseTransformer

A pipeline step that has no effect at all but to return the same data without changes. Transform method is automatically implemented as changing nothing.

Example :

class PrintOnFit(NonTransformableMixin, BaseStep):
    def __init__(self):
        BaseStep.__init__(self)

    def fit(self, data_inputs: ARG_X_INPUTTED, expected_outputs: ARG_Y_EXPECTED = None) -> BaseStep:
        print((data_inputs, expected_outputs))
        return self

Note

Fit methods are not implemented.

_fit_transform_data_container(data_container: neuraxle.data_container.DataContainer[~IDT, ~DIT, ~EOT][IDT, DIT, EOT], context: neuraxle.base.ExecutionContext)[source]
_transform_data_container(data_container: neuraxle.data_container.DataContainer[~IDT, ~DIT, ~EOT][IDT, DIT, EOT], context: neuraxle.base.ExecutionContext) → neuraxle.data_container.DataContainer[~IDT, ~DIT, typing.Union[~EOT, NoneType]][IDT, DIT, Optional[EOT]][source]

Do nothing - return the same data.

Parameters
  • data_container – data container

  • context (ExecutionContext) – execution context

Returns

data container

transform(data_inputs: DIT) → DIT[source]

Do nothing - return the same data.

Parameters

data_inputs – the data to process

Returns

the data_inputs, unchanged.

inverse_transform(processed_outputs: DIT) → DIT[source]

Do nothing - passthrough to return the same data.

Parameters

processed_outputs – the data to process

Returns

the processed_outputs, unchanged.

class neuraxle.base.TruncableJoblibStepSaver[source]

Bases: neuraxle.base.JoblibStepSaver

Step saver for a TruncableSteps. TruncableJoblibStepSaver saves, and loads all of the sub steps using their savers.

__init__()[source]

Initialize self. See help(type(self)) for accurate signature.

save_step(step: neuraxle.base.TruncableSteps, context: neuraxle.base.ExecutionContext)[source]

# . Loop through all the steps, and save the ones that need to be saved. # . Add a new property called sub step savers inside truncable steps to be able to load sub steps when loading. # . Strip steps from truncable steps at the end.

Parameters
Returns

load_step(step: neuraxle.base.TruncableSteps, context: neuraxle.base.ExecutionContext) → neuraxle.base.TruncableSteps[source]

# . Loop through all of the sub steps savers, and only load the sub steps that have been saved. # . Refresh steps

Parameters
Returns

loaded truncable steps

_abc_impl = <_abc_data object>
class neuraxle.base.TruncableStepsMixin(steps_as_tuple: List[Union[Tuple[str, BaseTransformerT], BaseTransformerT]], mute_step_renaming_warning: bool = True)[source]

Bases: neuraxle.base._HasChildrenMixin

A mixin for services that can be truncated.

__init__(steps_as_tuple: List[Union[Tuple[str, BaseTransformerT], BaseTransformerT]], mute_step_renaming_warning: bool = True)[source]

Initialize self. See help(type(self)) for accurate signature.

set_steps(steps_as_tuple: List[Union[Tuple[str, BaseTransformerT], BaseTransformerT]], invalidate=True) → neuraxle.base.TruncableStepsMixin[source]

Set steps as tuple.

Parameters

steps_as_tuple – list of tuple containing step name and step

Returns

get_children() → List[BaseServiceT][source]

Get the list of sub step inside the step with children.

Returns

children steps

_wrap_non_base_steps(steps_as_tuple: List[Union[Tuple[str, BaseTransformerT], BaseTransformerT]]) → List[Union[Tuple[str, BaseTransformerT], BaseTransformerT]][source]

If some steps are not of type BaseStep, we’ll try to make them of this type. For instance, sklearn objects will be wrapped by a SKLearnWrapper here.

Parameters

steps_as_tuple – a list of steps or of named tuples of steps (e.g.: NamedStepsList)

Returns

a NamedStepsList

_patch_missing_names(steps_as_tuple: List[Union[Tuple[str, BaseTransformerT], BaseTransformerT]]) → List[Union[Tuple[str, BaseTransformerT], BaseTransformerT]][source]

Make sure that each sub step has a unique name, and add a name to the sub steps that don’t have one already.

Parameters

steps_as_tuple – a NamedStepsList

Returns

a NamedStepsList with fixed names

_rename_step(step_name, class_name, names_yet: set)[source]

Rename step by adding a number suffix after the class name. Ensure uniqueness with the names yet parameter.

Parameters
  • step_name – step name

  • class_name – class name

  • names_yet (set) – names already taken

Returns

new step name

_refresh_steps(invalidate=True)[source]

Private method to refresh inner state after having edited self.steps_as_tuple (recreate self.steps from self.steps_as_tuple).

_step_index_to_name(step_index)[source]
items() → ItemsView[KT, VT_co][source]

Returns all of the steps as tuples items (step_name, step).

Returns

step items tuple : (step name, step)

keys() → KeysView[KT][source]

Returns the step names.

Returns

list of step names

values() → ValuesView[VT_co][source]

Get step values.

Returns

all of the steps

class neuraxle.base.TruncableSteps(steps_as_tuple: List[Union[Tuple[str, BaseTransformerT], BaseTransformerT]], hyperparams: neuraxle.hyperparams.space.HyperparameterSamples = {}, hyperparams_space: neuraxle.hyperparams.space.HyperparameterSpace = {}, mute_step_renaming_warning: bool = True)[source]

Bases: neuraxle.base.TruncableStepsMixin, neuraxle.base.BaseStep, abc.ABC

Step that contains multiple steps. Pipeline inherits form this class. It is possible to truncate this step * __getitem__()

  • self.steps contains the actual steps

  • self.steps_as_tuple contains a list of tuple of step name, and step

__init__(steps_as_tuple: List[Union[Tuple[str, BaseTransformerT], BaseTransformerT]], hyperparams: neuraxle.hyperparams.space.HyperparameterSamples = {}, hyperparams_space: neuraxle.hyperparams.space.HyperparameterSpace = {}, mute_step_renaming_warning: bool = True)[source]

Initialize self. See help(type(self)) for accurate signature.

append(item: Tuple[str, BaseTransformer]) → neuraxle.base.TruncableSteps[source]

Add an item to steps as tuple.

Parameters

item – item tuple (step name, step)

Returns

self

pop() → neuraxle.base.BaseTransformer[source]

Pop the last step.

Returns

last step

popitem(key=None) → Tuple[str, neuraxle.base.BaseTransformer][source]

Pop the last step, or the step with the given key

Parameters

key – step name to pop, or None

Returns

last step item

popfront() → neuraxle.base.BaseTransformer[source]

Pop the first step.

Returns

first step

popfrontitem() → Tuple[str, neuraxle.base.BaseTransformer][source]

Pop the first step.

Returns

first step item

split(step_type: type) → List[neuraxle.base.TruncableSteps][source]

Split truncable steps by a step class (type).

Parameters

step_type (type) – step class type to split on.

Returns

list of truncable steps containing the splitted steps

ends_with(step_type: Type[CT_co]) → bool[source]

Returns true if truncable steps end with a step of the given type.

Return type

bool

Parameters

step_type – step type

Returns

if truncable steps ends with the given step type

should_save()[source]

Returns if the step needs to be saved or not.

Returns

If self or any of his sub steps should be saved, returns True.

_repr(level=0, verbose=False) → str[source]
_abc_impl = <_abc_data object>
class neuraxle.base.Identity(name=None, savers=None)[source]

Bases: neuraxle.base.NonTransformableMixin, neuraxle.base.NonFittableMixin, neuraxle.base.BaseStep

A pipeline step that has no effect at all but to return the same data without changes.

This can be useful to concatenate new features to existing features, such as what AddFeatures do.

Identity inherits from NonTransformableMixin and from NonFittableMixin which makes it a class that has no effect in the pipeline: it doesn’t require fitting, and at transform-time, it returns the same data it received.

__init__(name=None, savers=None)[source]

Initialize self. See help(type(self)) for accurate signature.

_abc_impl = <_abc_data object>
class neuraxle.base.TransformHandlerOnlyMixin[source]

Bases: neuraxle.base.MixinForBaseTransformer

A pipeline step that only requires the implementation of _transform_data_container.

_transform_data_container(data_container: neuraxle.data_container.DataContainer[~IDT, ~DIT, ~EOT][IDT, DIT, EOT], context: neuraxle.base.ExecutionContext) → neuraxle.data_container.DataContainer[~IDT, ~DIT, typing.Union[~EOT, NoneType]][IDT, DIT, Optional[EOT]][source]

Transform data container with the given execution context.

Parameters
  • data_container – data container

  • context (ExecutionContext) – execution context

Returns

transformed data container

transform(data_inputs) → neuraxle.base.HandleOnlyMixin[source]
class neuraxle.base.HandleOnlyMixin[source]

Bases: neuraxle.base.MixinForBaseTransformer

A pipeline step that only requires the implementation of handler methods :

  • _fit_data_container

  • _transform_data_container

  • _fit_transform_data_container (by default, will call the above fit and then transform)

If forbids only implementing fit or transform or fit_transform without the handles. So it forces the handles.

_fit_data_container(data_container: neuraxle.data_container.DataContainer[~IDT, ~DIT, ~EOT][IDT, DIT, EOT], context: neuraxle.base.ExecutionContext) → neuraxle.base.BaseTransformer[source]
_transform_data_container(data_container: neuraxle.data_container.DataContainer[~IDT, ~DIT, ~EOT][IDT, DIT, EOT], context: neuraxle.base.ExecutionContext) → neuraxle.data_container.DataContainer[~IDT, ~DIT, typing.Union[~EOT, NoneType]][IDT, DIT, Optional[EOT]][source]
_fit_transform_data_container(data_container: neuraxle.data_container.DataContainer[~IDT, ~DIT, ~EOT][IDT, DIT, EOT], context: neuraxle.base.ExecutionContext) → Tuple[neuraxle.base.BaseTransformer, neuraxle.data_container.DataContainer[~IDT, ~DIT, typing.Union[~EOT, NoneType]][IDT, DIT, Optional[EOT]]][source]

Fit and transform data container with the given execution context. Will do:

data_container, context = self._fit_data_container(data_container, context)
data_container = self._transform_data_container(data_container, context)
return self, data_container
Parameters
  • data_container – data container

  • context (ExecutionContext) – execution context

Returns

transformed data container

transform(data_inputs) → neuraxle.base.HandleOnlyMixin[source]
fit(data_inputs: DIT, expected_outputs: EOT = None) → neuraxle.base.HandleOnlyMixin[source]
fit_transform(data_inputs: DIT, expected_outputs: EOT = None) → Tuple[neuraxle.base.HandleOnlyMixin, DIT][source]
class neuraxle.base.ForceHandleMixin(cache_folder=None)[source]

Bases: neuraxle.base.MixinForBaseTransformer

A step that automatically calls handle methods in the transform, fit, and fit_transform methods. A class which inherits from ForceHandleMixin can’t use BaseStep’s _fit_data_container, _fit_transform_data_container and _transform_data_container. They must be redefined; failure to do so will trigger an Exception on initialisation (and would create infinite loop if these checks were not there).

__init__(cache_folder=None)[source]

Initialize self. See help(type(self)) for accurate signature.

_ensure_method_overriden(method_name, original_cls)[source]

Asserts that a given method of current instance overrides the default one defined in a given class. We assume that current instance inherits from the given class but do not test it (MixinForBaseTransformer test inheritance to BaseTransfromer already).

Parameters
  • method_name

  • original_cls

Returns

transform(data_inputs: DIT) → DIT[source]

Using handle_transform(), transform data inputs.

Parameters

data_inputs – data inputs

Returns

outputs

fit(data_inputs: DIT, expected_outputs: EOT = None) → neuraxle.base.ForceHandleMixin[source]

Using handle_fit(), fit step with the given data inputs, and expected outputs.

Parameters

data_inputs – data inputs

Returns

fitted self

fit_transform(data_inputs: DIT, expected_outputs: EOT = None) → Tuple[neuraxle.base.ForceHandleMixin, DIT][source]

Using handle_fit_transform(), fit and transform step with the given data inputs, and expected outputs.

Parameters

data_inputs – data inputs

Returns

fitted self, outputs

_encapsulate_data(data_inputs: DIT, expected_outputs: EOT, execution_mode: neuraxle.base.ExecutionMode) → Tuple[neuraxle.base.ExecutionContext, neuraxle.data_container.DataContainer[~IDT, ~DIT, ~EOT][IDT, DIT, EOT]][source]

Encapsulate data with DataContainer.

Parameters
  • data_inputs – data inputs

  • expected_outputs – expected outputs

  • execution_mode (ExecutionMode) – execution mode

Returns

execution context, data container

class neuraxle.base.ForceHandleIdentity[source]

Bases: neuraxle.base.ForceHandleMixin, neuraxle.base.Identity

An identity step which forces usage of handler methods. Mostly used in unit tests. Useful when you only want to define few methods such as _will_process or _did_process.

__init__()[source]

Initialize self. See help(type(self)) for accurate signature.

_abc_impl = <_abc_data object>
class neuraxle.base.ForceHandleOnlyMixin(cache_folder=None)[source]

Bases: neuraxle.base.ForceHandleMixin, neuraxle.base.HandleOnlyMixin

A step that automatically calls handle methods in the transform, fit, and fit_transform methods.

It also requires the implementation of handler methods :
  • _transform_data_container

  • _fit_transform_data_container

  • _fit_data_container

__init__(cache_folder=None)[source]

Initialize self. See help(type(self)) for accurate signature.

class neuraxle.base.IdentityHandlerMethodsMixin(cache_folder=None)[source]

Bases: neuraxle.base.ForceHandleOnlyMixin

A step that has a default implementation for all handler methods.

It is useful for steps that only the following methods :
  • will_fit()

  • will_transform()

  • will_fit_transform()

  • will_process()

  • did_process()

  • did_fit()

  • did_transform()

  • did_fit_transform()

_fit_data_container(data_container: neuraxle.data_container.DataContainer[~IDT, ~DIT, ~EOT][IDT, DIT, EOT], context: neuraxle.base.ExecutionContext) → Tuple[neuraxle.base.BaseTransformer, neuraxle.data_container.DataContainer][source]
_transform_data_container(data_container: neuraxle.data_container.DataContainer[~IDT, ~DIT, ~EOT][IDT, DIT, EOT], context: neuraxle.base.ExecutionContext) → neuraxle.data_container.DataContainer[~IDT, ~DIT, typing.Union[~EOT, NoneType]][IDT, DIT, Optional[EOT]][source]
_fit_transform_data_container(data_container: neuraxle.data_container.DataContainer[~IDT, ~DIT, ~EOT][IDT, DIT, EOT], context: neuraxle.base.ExecutionContext) → Tuple[neuraxle.base.BaseTransformer, neuraxle.data_container.DataContainer[~IDT, ~DIT, typing.Union[~EOT, NoneType]][IDT, DIT, Optional[EOT]]][source]

Fit and transform data container with the given execution context. Will do:

data_container, context = self._fit_data_container(data_container, context)
data_container = self._transform_data_container(data_container, context)
return self, data_container
Parameters
  • data_container – data container

  • context (ExecutionContext) – execution context

Returns

transformed data container

class neuraxle.base.EvaluableStepMixin[source]

Bases: neuraxle.base.MixinForBaseTransformer

A step that can be evaluated with the scoring functions.

See also

BaseStep :class:’MixinForBaseTransformer’

get_score()[source]
class neuraxle.base.FullDumpLoader(name, stripped_saver=None)[source]

Bases: neuraxle.base.Identity

Identity step that can load the full dump of a pipeline step. Used by load().

Usage example:

saved_step = FullDumpLoader(
    name=path,
    stripped_saver=self.stripped_saver
).load(context_for_loading, True)
__init__(name, stripped_saver=None)[source]

Initialize self. See help(type(self)) for accurate signature.

load(context: neuraxle.base.ExecutionContext, full_dump=True) → neuraxle.base.BaseStep[source]

Load the full dump of a pipeline step.

Return type

BaseStep

Parameters
  • context (ExecutionContext) – execution context

  • full_dump (bool) – load full dump or not (always true, inherited from BaseStep

Returns

loaded step

_abc_impl = <_abc_data object>
class neuraxle.base.AssertionMixin[source]

Bases: neuraxle.base.ForceHandleMixin

__init__()[source]

Initialize self. See help(type(self)) for accurate signature.

_assert_at_lifecycle(data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext)[source]
class neuraxle.base.WillProcessAssertionMixin[source]

Bases: neuraxle.base.AssertionMixin

_will_process(data_container: neuraxle.data_container.DataContainer[~IDT, ~DIT, ~EOT][IDT, DIT, EOT], context: neuraxle.base.ExecutionContext) → Tuple[neuraxle.data_container.DataContainer[~IDT, ~DIT, ~EOT][IDT, DIT, EOT], neuraxle.base.ExecutionContext][source]

Calls self._assert_at_lifecycle(data_container, context).

class neuraxle.base.DidProcessAssertionMixin[source]

Bases: neuraxle.base.AssertionMixin

_did_process(data_container: neuraxle.data_container.DataContainer[~IDT, ~DIT, typing.Union[~EOT, NoneType]][IDT, DIT, Optional[EOT]], context: neuraxle.base.ExecutionContext) → Tuple[neuraxle.data_container.DataContainer[~IDT, ~DIT, typing.Union[~EOT, NoneType]][IDT, DIT, Optional[EOT]], neuraxle.base.ExecutionContext][source]

Calls self._assert_at_lifecycle(data_container,context)

class neuraxle.base.AssertExpectedOutputIsNoneMixin[source]

Bases: neuraxle.base.WillProcessAssertionMixin

_assert_at_lifecycle(data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext)[source]
class neuraxle.base.AssertExpectedOutputIsNotNoneMixin[source]

Bases: neuraxle.base.WillProcessAssertionMixin

_assert_at_lifecycle(data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext)[source]
class neuraxle.base.AssertExpectedOutputIsNoneStep[source]

Bases: neuraxle.base.AssertExpectedOutputIsNoneMixin, neuraxle.base.Identity

__init__()[source]

Initialize self. See help(type(self)) for accurate signature.

_abc_impl = <_abc_data object>
class neuraxle.base.AssertExpectedOutputIsNone[source]

Bases: neuraxle.base.AssertExpectedOutputIsNoneMixin, neuraxle.base.Identity

__init__()[source]

Initialize self. See help(type(self)) for accurate signature.

_abc_impl = <_abc_data object>
class neuraxle.base.LocalServiceAssertionWrapper(wrapped: neuraxle.base.BaseTransformer = None, service_assertions: List[Type[BaseService]] = None)[source]

Bases: neuraxle.base.WillProcessAssertionMixin, neuraxle.base.MetaStep

Is used to assert the presence of service at execution time for a given step

__init__(wrapped: neuraxle.base.BaseTransformer = None, service_assertions: List[Type[BaseService]] = None)[source]

Initialize self. See help(type(self)) for accurate signature.

_assert_at_lifecycle(data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext)[source]

Assert self.local_service_assertions are present in the context.

do_assert_has_services(context: neuraxle.base.ExecutionContext)[source]

Assert that all the necessary services are provided in the execution context.

Parameters

context (ExecutionContext) – The ExecutionContext for which we test the presence of service.

_abc_impl = <_abc_data object>
class neuraxle.base.GlobalyRetrievableServiceAssertionWrapper(wrapped: neuraxle.base.BaseTransformer = None, service_assertions: List[Type[BaseService]] = None)[source]

Bases: neuraxle.base.LocalServiceAssertionWrapper

Is used to assert the presence of service at the start of the pipeline AND at execution time for a given step.

_global_assert_has_services(context: neuraxle.base.ExecutionContext) → neuraxle.hyperparams.space.RecursiveDict[source]

Intended to be used in a .apply(‘_global_assert_has_services’) call from the outside. Is used to test the presence of services at the root of the pipeline.

See also GlobalServiceAssertionExecutorMixin._apply_service_assertions :rtype: RecursiveDict :type context: ExecutionContext :params context : the execution context

_abc_impl = <_abc_data object>
class neuraxle.base.GlobalServiceAssertionExecutorMixin[source]

Bases: neuraxle.base.WillProcessAssertionMixin

Any step which inherit of this class will test globaly retrievable service assertion of itself and all its children on a will_process call.

_assert_at_lifecycle(data_container: neuraxle.data_container.DataContainer, context: neuraxle.base.ExecutionContext)[source]

Calls _global_assert_has_services on GlobalyRetrievableServiceAssertionWrapper instances that are (recursively) children of this node.

Parameters
  • data_container (DataContainer) – The DataContainer, probably unused.

  • context (ExecutionContext) – The ExecutionContext for which we test the presence of service.

class neuraxle.base._WithContextStepSaver[source]

Bases: neuraxle.base.BaseSaver

Custom saver for steps that have an ExecutionContext. Loading will inject the saved dependencies inside the :class`ExecutionContext`.

See also

_HasContext, BaseSaver, ExecutionContext

load_step(step: neuraxle.base.StepWithContext, context: neuraxle.base.ExecutionContext) → neuraxle.base.StepWithContext[source]

Load a step with a context by setting the context as the loading context.

Parameters
  • step – step with context

  • context (ExecutionContext) – execution context to load from

Returns

loaded step with context

save_step(step: neuraxle.base.StepWithContext, context: neuraxle.base.ExecutionContext) → neuraxle.base.StepWithContext[source]

If needed, remove parents of a step with context before saving.

Parameters
  • step – step with context

  • context (ExecutionContext) – execution context to load from

Returns

saved step with context

can_load(step: neuraxle.base.StepWithContext, context: neuraxle.base.ExecutionContext)[source]

Returns true if we can load the given step with the given execution context.

Parameters
  • step – step to load

  • context – execution context to load from

Returns

_abc_impl = <_abc_data object>
class neuraxle.base.StepWithContext(wrapped: neuraxle.base.BaseTransformer, context: neuraxle.base.ExecutionContext, raise_if_not_root: bool = True)[source]

Bases: neuraxle.base.GlobalServiceAssertionExecutorMixin, neuraxle.base.MetaStep

A step with context is a step that has an ExecutionContext as a pre-registered dependency. This way, it’s possible to call the vanilla “fit”, “transform”, “predict”, and other vanilla methods on a step with context, and the handle will be made automatically with the provided context, also wrapping the data into a data container.

__init__(wrapped: neuraxle.base.BaseTransformer, context: neuraxle.base.ExecutionContext, raise_if_not_root: bool = True)[source]

Initialize self. See help(type(self)) for accurate signature.

_will_process(data_container: neuraxle.data_container.DataContainer[~IDT, ~DIT, ~EOT][IDT, DIT, EOT], context: neuraxle.base.ExecutionContext) → Tuple[neuraxle.data_container.DataContainer[~IDT, ~DIT, ~EOT][IDT, DIT, EOT], neuraxle.base.ExecutionContext][source]

Inject the given context and test service assertions (if any are appliable) before processing the wrapped step.

Parameters

data_container – data container to process

Returns

data container, execution context

save(context: neuraxle.base.ExecutionContext = None, full_dump=True) → neuraxle.base.BaseTransformer[source]

Save the wrapped step with its context with the provided context location.

Parameters

context (ExecutionContext) – Context to save that would override the registred one in self.context. Optional.

Returns

saved step with context

_abc_impl = <_abc_data object>