com.hof.mi.etl.step

Class AbstractETLCachedStep

java.lang.Object

com.hof.mi.etl.step.AbstractETLStep

com.hof.mi.etl.step.AbstractETLCachedStep

All Implemented Interfaces:

ETLStep, java.io.Serializable

public abstract class AbstractETLCachedStep
extends AbstractETLStep

This is a partial implementation AbstractETLStep. Step implementations which extend this should can process accumulated data.

See Also:

Serialized Form

Field Summary

Fields inherited from class com.hof.mi.etl.step.AbstractETLStep

commonDeps, dataPackets, dependencies

Constructor Summary

Constructors

Constructor and Description
AbstractETLCachedStep()
AbstractETLCachedStep(com.hof.mi.etl.ETLDependencies dependencies, com.hof.util.dependency.CommonDependencies commonDeps)

Method Summary

Methods

Modifier and Type	Method and Description
protected void	cleanup() All cleanup and releasing resources to happen here.
void	endRows(java.lang.String inFlowUuid) Signal end of data from an input step
ETLDataCache	getDataCache(java.lang.String inFlowUuid) Return the data cache for an input step.
protected void	postEndRows() Any post-processing after executing the core processing logic of this step.
protected void	preEndRows() Any pre-processing before executing the core processing logic of this step.
protected abstract void	processEndRows() This executes the core processing logic of the step implementation.
void	row(java.lang.Object[] data, java.lang.String inFlowUuid) Cache a row of data from an input step.

Methods inherited from class com.hof.mi.etl.step.AbstractETLStep

addDefaultMetadataField, addError, addGeneratedField, addInputStep, addNewGeneratedField, addOutputStep, addStepOption, beginInternalTransmission, checkErrorThreshold, clearAllWires, disable, emitData, emitData, endInternalTransmission, endInternalTransmission, excludeDefaultField, flowFinished, generatePanelCollection, getContext, getDefaultInternalOptions, getDefaultMetadataFields, getDefaultMetadataFieldsMap, getDefaultMetadataFieldUuids, getDefaultMetadataFieldUuidsList, getDefaultMetadataMap, getDefaultToInputFieldMap, getErrorFlowUuid, getErrorStep, getEtlDataTypeUtil, getETLException, getETLStepBean, getFile, getFile, getFileName, getFileName, getFirstInputFlow, getFirstOutputFlow, getFreshDataPacket, getInputFlowForDefaultMetadataGroup, getInputFlowUuids, getInputStep, getInputSteps, getInputToDefaultFieldMap, getInvalidConfigETLException, getMaxInputSteps, getMaxOutputSteps, getMetadataFields, getMinInputSteps, getMinOutputSteps, getOffImage, getOffImageMIMEType, getOffImagePath, getOnImage, getOnImageMIMEType, getOnImagePath, getOutputFlowUuids, getOutputFlowUuidsExceededRowLimit, getOutputStep, getOutputSteps, getPanelCollection, getRowLimit, getStatus, getStepBooleanValue, getStepClassification, getStepIntegerValue, getStepOption, getStepOptions, getStepOptionValueJSONArray, getStepOptionValueJSONObject, getStepSubCategory, getStringUtil, getText, getText, getUnhandledETLException, getUuid, getValidatedStepOptions, getWireForField, hasPendingDependency, includeDefaultField, isEnabled, isGeneratedField, isMainInputFlow, isReplacementField, onCopy, onDelete, parseData, removeDefaultMetadataField, replaceDefaultField, restoreReplacedField, setContext, setErrorStep, setETLStepBean, setMetadataFields, setRowLimit, setStatus, setStepOptions, setupDefaultFields, setupGeneratedFields, setupOptions, shouldOutputField, stepCanRun, throwInvalidConfigETLException, throwUnhandledETLException, updateFieldSortOrder, waitForDependencies

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.hof.mi.etl.step.ETLStep

getAPIVersion, getDefaultDescription, getDefaultName, getStepCategory, validate

Constructor Detail

AbstractETLCachedStep

public AbstractETLCachedStep()

AbstractETLCachedStep

public AbstractETLCachedStep(com.hof.mi.etl.ETLDependencies dependencies,
                     com.hof.util.dependency.CommonDependencies commonDeps)

Method Detail

row

public void row(java.lang.Object[] data,
       java.lang.String inFlowUuid)
         throws ETLException

Cache a row of data from an input step.

Parameters:

data - - A row of data

inFlowUuid - - Input flow UUID on which this data arrived

Throws:

ETLException

endRows

public void endRows(java.lang.String inFlowUuid)
             throws ETLException

Signal end of data from an input step

Parameters:

inFlowUuid - - Flow along which we will not receive any more data.

Throws:

ETLException

cleanup

protected void cleanup()

All cleanup and releasing resources to happen here. This method is always called before a step finishes. The method in AbstractETLCachedStep does nothing, but a step may override it if it needs to.

preEndRows

protected void preEndRows()

Any pre-processing before executing the core processing logic of this step. This implementation does nothing. Override to implement in a subclass.

postEndRows

protected void postEndRows()

Any post-processing after executing the core processing logic of this step. This implementation does nothing. Override to implement in a subclass.

getDataCache

public ETLDataCache getDataCache(java.lang.String inFlowUuid)

Return the data cache for an input step.

Parameters:

inFlowUuid - - input flowUuid whose data cache needs to be returned.

processEndRows

protected abstract void processEndRows()
                                throws ETLException,
                                       java.lang.InterruptedException

This executes the core processing logic of the step implementation. Data is available in data caches for each input, and may be retrieved using getDataCache(String).

This method should not handle exceptions, as errors during a Step's processing should be logged/handled by the framework

This method must propagate processed data downstream in the following manner:

• First get an empty data packet using #getFreshDataPacket getFreshDataPacket(String outFlowUuid)
• Call #beginInternalTransmission beginInternalTransmission(Object[] data, List fields) to put data on Wires, which will map input data to the correct positions in the output. Call this for every input.
• Process data by using Wire's methods to get and set data.
• Populate the data packet using #endInternalTransmission endInternalTransmission(ETLStepResult dataPacket)
• Finally, emit data using #emitData(com.hof.mi.etl.runner.ETLStepResult) emitData(ETLStepResult dataPacket)

Throws:

ETLException - when there's an error.

java.lang.InterruptedException - when execution is cancelled