neuraxle.steps.caching

Pipeline Steps For Caching

Classes

BaseValueHasher

Md5Hasher

PickleValueCachingWrapper(wrapped, …)

Value Caching Wrapper class that caches the wrapped step transformed data inputs using python pickle library.

ValueCachingWrapper(wrapped, cache_folder, …)

Value caching wrapper wraps a step to cache the values.

class neuraxle.steps.caching.BaseValueHasher[source]
hash(data_input)[source]
class neuraxle.steps.caching.Md5Hasher[source]
hash(data_input)[source]
class neuraxle.steps.caching.PickleValueCachingWrapper(wrapped: neuraxle.base.BaseStep, cache_folder: str = 'cache', value_hasher: Optional[neuraxle.steps.caching.BaseValueHasher] = None)[source]

Value Caching Wrapper class that caches the wrapped step transformed data inputs using python pickle library.

contains_cache_for(data_input) → bool[source]

Returns true if the data input transform output is cached.

Parameters

data_input – to get cache from

Returns

boolean to indicate if a cache is present for the given data input

create_checkpoint_path(step_path: str) → str[source]

Create checkpoint path.

Parameters

step_path (str) – step path inside pipeline ex: Pipeline/step_name/

Returns

checkpoint path

flush_cache()[source]

Flush all cached values :return:

get_cache_path_for(data_input)[source]

Get the cache path for the given data input.

Parameters

data_input – data input to get cache path for

Returns

str for cache path

read_cache(data_input)[source]

Read cache for a given data input.

Parameters

data_input (Any) – data input to get cache for

Returns

transform(data_inputs)[source]

Do nothing - return the same data.

Parameters

data_inputs – the data to process

Returns

the data_inputs, unchanged.

write_cache(data_input, output)[source]

Write cache for a given data input and output.

Parameters
  • data_input (Any) – data input to write cache for

  • output (Any) – output to write cache for

Returns

class neuraxle.steps.caching.ValueCachingWrapper(wrapped: neuraxle.base.BaseStep, cache_folder: str = 'cache', value_hasher: Optional[neuraxle.steps.caching.BaseValueHasher] = None)[source]

Value caching wrapper wraps a step to cache the values.

contains_cache_for(data_input) → bool[source]

Returns true if the data input transform output is cached.

Parameters

data_input – to get cache from

Returns

boolean to indicate if a cache is present for the given data input

create_checkpoint_path(step_path: str) → str[source]

Create checkpoint path.

Parameters

step_path (str) – step path inside pipeline ex: Pipeline/step_name/

Returns

checkpoint path

flush_cache()[source]

Flush all cached values :return:

get_cache_path_for(data_input) → str[source]

Get the cache path for the given data input.

Parameters

data_input – data input to get cache path for

Returns

str for cache path

read_cache(data_input) → Any[source]

Read cache for a given data input.

Parameters

data_input (Any) – data input to get cache for

Returns

write_cache(data_input, output)[source]

Write cache for a given data input and output.

Parameters
  • data_input (Any) – data input to write cache for

  • output (Any) – output to write cache for

Returns