com.hof.mi.etl.step

Class AbstractETLStep

java.lang.Object

com.hof.mi.etl.step.AbstractETLStep

All Implemented Interfaces:

ETLStep, java.io.Serializable

Direct Known Subclasses:

AbstractETLCachedStep, AbstractETLRowStep

public abstract class AbstractETLStep
extends java.lang.Object
implements ETLStep

This is a partial implementaton of the ETLStep interface.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected com.hof.util.dependency.CommonDependencies	commonDeps
protected ETLStepResults	dataPackets
protected com.hof.mi.etl.ETLDependencies	dependencies

Constructor Summary

Constructors

Constructor and Description
AbstractETLStep()
AbstractETLStep(com.hof.mi.etl.ETLDependencies dependencies, com.hof.util.dependency.CommonDependencies commonDeps)

Method Summary

Methods

Modifier and Type	Method and Description
void	addDefaultMetadataField(ETLStepMetadataFieldBean field) Add a Default Metadata Field for this step.
protected void	addError(ETLException e) Use this method to record errors during execution of the step.
ETLStepMetadataFieldBean	addGeneratedField(ETLStepMetadataFieldBean field, ETLFieldLinkType linkType, java.lang.String linkFieldUUID) Generated Metadata Fields are new fields which the step creates.
void	addInputStep(java.lang.String inFlowUuid, ETLStep inputStep) Register an input step.
ETLStepMetadataFieldBean	addNewGeneratedField(ETLStepMetadataFieldBean field, java.lang.String optionKey) Passes through to addGeneratedField(ETLStepMetadataFieldBean, ETLFieldLinkType, String) with the ETLFieldLinkType.NEWFIELD and if optionKey is not null, adds a step option containing the new field's UUID.
void	addOutputStep(java.lang.String outFlowUuid, ETLStep outputStep) Register an output step.
void	addStepOption(java.lang.String optionKey, java.lang.String optionValue) Add a config option for this step.
protected void	beginInternalTransmission(java.lang.Object[] data, java.util.List<java.lang.String> fields) Transmit a row of data from the Input, by placing data on internal wires.
protected void	checkErrorThreshold() Check if the step has exceeded this step's error threshold.
void	clearAllWires() Clear all wires
void	disable(boolean disabled) Disable the step
protected void	emitData(ETLStepResult dataPacket) Send a data packet to its output step
protected void	emitData(ETLStepResults data) Send data packets to all output steps.
protected ETLStepResult	endInternalTransmission(ETLStepResult outputData) Finish transmission by removing data from internal wires and putting them into an array in the output data packet (represented by ETLStepResult).
protected ETLStepResult	endInternalTransmission(ETLStepResult outputData, boolean clearWires) Finish transmission by getting data from internal wires and putting them into an array in the output data packet (represented by ETLStepResult).
protected void	excludeDefaultField(java.lang.String fieldUUID) Sets stepIncludeField to false on the given field and moves it's sort order to the end of the list.
protected void	flowFinished(java.lang.String flowUuid) Record that an input flow has finished sending data to this step.
protected abstract ParameterPanelCollection	generatePanelCollection() Method to generate the ParameterPanelCollection object which represents the step's configuration UI elements.
ETLRunContext	getContext() Get information about the ETL Runner running this step.
java.util.Map<java.lang.String,java.lang.String>	getDefaultInternalOptions() Return step optionKeys and optionValues for options internal to the step.
java.util.List<ETLStepMetadataFieldBean>	getDefaultMetadataFields() Get Default Metadata of this step.
java.util.Map<java.lang.String,ETLStepMetadataFieldBean>	getDefaultMetadataFieldsMap() Return the Default Metadata Fields keyed by field UUID.
protected java.util.Set<java.lang.String>	getDefaultMetadataFieldUuids() Get a copy of Default Field UUIDs in a Set.
protected java.util.List<java.lang.String>	getDefaultMetadataFieldUuidsList() Get a copy of Default Field UUIDs in a List.
java.util.Map<java.lang.String,ETLStepMetadataFieldBean>	getDefaultMetadataMap() Deprecated.
java.util.Map<java.lang.String,java.lang.String>	getDefaultToInputFieldMap() Get the mapping from default fields to input fields.
java.lang.String	getErrorFlowUuid() FlowUUID connecting this step to an error step.
ETLStep	getErrorStep() The error step.
protected com.hof.mi.etl.step.ETLDataTypeUtil	getEtlDataTypeUtil()
ETLException	getETLException(java.lang.String messageKey) Get an ETLException for the step with the message key that is passed in.
ETLStepBean	getETLStepBean() Get the ETLStepBean backing this step.
byte[]	getFile(java.lang.Integer fileId) Get a file that is associated with the passed fileId
byte[]	getFile(java.lang.String property) Get a file that is associated with the passed property
java.lang.String	getFileName(java.lang.Integer fileId) Helper method to get the file name of the selected passed in fileId
java.lang.String	getFileName(java.lang.String property) Helper method to get the file name of the selected passed in property
java.lang.String	getFirstInputFlow() Return the first input flowUUID.
java.lang.String	getFirstOutputFlow() Return the first output flowUUID.
ETLStepResult	getFreshDataPacket(java.lang.String outFlowUuid) Get an empty data packet for a specified output flowUUID.
java.lang.String	getInputFlowForDefaultMetadataGroup(java.lang.String groupUuid) Group ID is used to represent which default metadata fields come from the same source.
java.util.Set<java.lang.String>	getInputFlowUuids() Get all input flowUUIDs.
ETLStep	getInputStep(java.lang.String inFlowUuid) Get the input step connected to this step by the specified input flowUUID.
java.util.Collection<ETLStep>	getInputSteps() Get all Input Steps as a Collection
java.util.Map<java.lang.String,java.lang.String>	getInputToDefaultFieldMap() Get the mapping from input fields to default fields.
ETLException	getInvalidConfigETLException() Get a generic ETLException when there is a problem in the step's config.
java.lang.Integer	getMaxInputSteps() Maximum number of inputs this step supports
java.lang.Integer	getMaxOutputSteps() Maximum number of outputs this step supports
java.util.List<ETLStepMetadataFieldBean>	getMetadataFields(java.lang.String outFlowUuid) Get Output Metadata Fields for a specified output flowUUID.
java.lang.Integer	getMinInputSteps() Minimum number of inputs required for this step to work
java.lang.Integer	getMinOutputSteps() Minimum number of outputs required for this step to work
java.lang.String	getOffImage() Returns a byte array for a base 64 image
java.lang.String	getOffImageMIMEType() Returns a MIMEType for a file path string
protected java.lang.String	getOffImagePath() The "off" state image, relative to Yellowfin's root directory.
java.lang.String	getOnImage() Returns a byte array for a base 64 image
java.lang.String	getOnImageMIMEType() Returns a MIMEType for a file path string
protected java.lang.String	getOnImagePath() The "on" state image, relative to Yellowfin's root directory.
java.util.Set<java.lang.String>	getOutputFlowUuids() Get all output flowUUIDs.
protected java.util.Set<java.lang.String>	getOutputFlowUuidsExceededRowLimit() Return a set of output flow UUIDs which have exceeded the step's row limit
ETLStep	getOutputStep(java.lang.String outFlowUuid) Get the output step connected to this step by the specified output flowUUID.
java.util.Collection<ETLStep>	getOutputSteps() Get all Output Steps as a Collection.
ParameterPanelCollection	getPanelCollection() Get the ParameterPanelCollection object which represents the step's configuration UI elements.
long	getRowLimit() The maximum number of rows which will be output by this step.
ETLStepStatus	getStatus() Get the status of the step as an instance of ETLStepStatus.
boolean	getStepBooleanValue(java.lang.String property)
ETLStepClassification	getStepClassification() Returns a specific classification for an ETLStep
java.lang.Integer	getStepIntegerValue(java.lang.String property)
java.lang.String	getStepOption(java.lang.String optionKey) Get config option value for this step, corresponding to the specified option key.
java.util.Map<java.lang.String,java.lang.String>	getStepOptions() Get all config options for this step as a map of Option Key and Option Value.
com.google.gson.JsonArray	getStepOptionValueJSONArray(java.lang.String property)
com.google.gson.JsonObject	getStepOptionValueJSONObject(java.lang.String property)
java.lang.String	getStepSubCategory() Set up a default stepSubCategory
protected com.hof.util.UtilStringService	getStringUtil()
java.lang.String	getText(java.lang.Integer textId) Get the text associated with the passed text id
java.lang.String	getText(java.lang.String property) Get the text associated with the passed property
ETLException	getUnhandledETLException(java.lang.Throwable e) Get an ETLException when the error cannot be handled.
java.lang.String	getUuid() Get the step's UUID
abstract java.util.Map<java.lang.String,java.lang.String>	getValidatedStepOptions() Validate options returned by getStepOptions() and return a map of optionKeys and optionValues.
com.hof.mi.data.Wire<java.lang.Object,java.lang.String>	getWireForField(java.lang.String fieldUuid) Get the Wire corresponding to an ETLStepMetadataFieldUUID.
boolean	hasPendingDependency() Are there any dependent input steps which have not finished executing?
protected void	includeDefaultField(java.lang.String fieldUUID, java.lang.Integer newSortOrder) Sets stepIncludeField to true on the given field.
boolean	isEnabled() A step implementation may override to enable/disable itself.
protected boolean	isGeneratedField(ETLStepMetadataFieldBean field) Is the field generated by the step?
boolean	isMainInputFlow(java.lang.String inFlowUuid) Is this the "main" input flow?
protected boolean	isReplacementField(ETLStepMetadataFieldBean defField) Is this field being used as a 'replacement' for another field?
java.util.Map<java.lang.String,java.lang.String>	onCopy(java.util.Map<java.lang.String,java.lang.String> uuidMap) This method is called while the step is being copied and is intended to be used as a way for steps to copy their external dependencies.
void	onDelete() This method is called just before the step is deleted and is intended to be used as a way for steps to cleanup external dependencies.
void	parseData(java.util.Map<java.lang.String,java.lang.Object> data) Parse the data of panel collection from the front end to the correct format that should be saved into the database and return the updated step options
void	removeDefaultMetadataField(java.lang.String fieldUuid) Remove a Default Metadata Field from this step.
protected ETLStepMetadataFieldBean	replaceDefaultField(ETLStepMetadataFieldBean fieldToReplace) This method 'replaces' a default field with another one.
protected ETLStepMetadataFieldBean	restoreReplacedField(ETLStepMetadataFieldBean replacementField) This method restores the original replaced field back into it's place and removes the replacement field.
void	setContext(ETLRunContext context) Set the step's ETLRunContext.
void	setErrorStep(java.lang.String errorFlowUuid, ETLStep errorStep) Register the error step.
void	setETLStepBean(ETLStepBean stepBean) Set the ETLStepBean backing this step.
void	setMetadataFields(java.lang.String outFlowUuid, ETLFlowType flowType, java.util.List<ETLStepMetadataFieldBean> metadata) Set Output Metadata Fields for a specified output flowUUID.
void	setRowLimit(long rowLimit) Set the step's rowLimit.
void	setStatus(ETLStepStatus status) Set the status object for this step.
void	setStepOptions(java.util.Map<java.lang.String,java.lang.String> options) Reset the step options with the provided values.
void	setupDefaultFields() Set up the default metadata fields for this step.
abstract void	setupGeneratedFields() Set up the default metadata fields for fields generated by this step.
void	setupOptions() Set up step options.
boolean	shouldOutputField(java.lang.String flowUuid, ETLStepMetadataFieldBean field) Determine if a metadata field should be sent to the next step.
protected boolean	stepCanRun() Can this step run? Find out based on the "disabled" flag and whether the step implementation has enabled itself.
void	throwInvalidConfigETLException() Throw an ETLException when there is a problem in the step's config.
void	throwUnhandledETLException(java.lang.Throwable e) Throw an ETLException when the error cannot be handled.
protected void	updateFieldSortOrder(java.lang.String fieldUUID, java.lang.Integer sortOrder) Updates the sort order for a given existing default field.
void	waitForDependencies(long checkAfterMillis) Block the step's execution till all dependent input steps finish execution

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.hof.mi.etl.step.ETLStep

endRows, getAPIVersion, getDefaultDescription, getDefaultName, getStepCategory, row, validate

Field Detail

dataPackets

protected ETLStepResults dataPackets

dependencies

protected com.hof.mi.etl.ETLDependencies dependencies

commonDeps

protected com.hof.util.dependency.CommonDependencies commonDeps

Constructor Detail

AbstractETLStep

public AbstractETLStep()

AbstractETLStep

public AbstractETLStep(com.hof.mi.etl.ETLDependencies dependencies,
               com.hof.util.dependency.CommonDependencies commonDeps)

Method Detail

getUuid

public java.lang.String getUuid()

Description copied from interface: ETLStep

Get the step's UUID

Specified by:

getUuid in interface ETLStep

getETLStepBean

public ETLStepBean getETLStepBean()

Description copied from interface: ETLStep

Get the ETLStepBean backing this step.

Specified by:

getETLStepBean in interface ETLStep

setETLStepBean

public void setETLStepBean(ETLStepBean stepBean)

Description copied from interface: ETLStep

Set the ETLStepBean backing this step.

Specified by:

setETLStepBean in interface ETLStep

getInputFlowUuids

public java.util.Set<java.lang.String> getInputFlowUuids()

Description copied from interface: ETLStep

Get all input flowUUIDs.

Specified by:

getInputFlowUuids in interface ETLStep

getInputStep

public ETLStep getInputStep(java.lang.String inFlowUuid)

Description copied from interface: ETLStep

Get the input step connected to this step by the specified input flowUUID.

Specified by:

getInputStep in interface ETLStep

addInputStep

public void addInputStep(java.lang.String inFlowUuid,
                ETLStep inputStep)

Description copied from interface: ETLStep

Register an input step.

Specified by:

addInputStep in interface ETLStep

Parameters:

inFlowUuid - - FlowUUID connecting this step to the input step

inputStep - - The input step

getOutputFlowUuids

public java.util.Set<java.lang.String> getOutputFlowUuids()

Description copied from interface: ETLStep

Get all output flowUUIDs.

Specified by:

getOutputFlowUuids in interface ETLStep

getOutputStep

public ETLStep getOutputStep(java.lang.String outFlowUuid)

Description copied from interface: ETLStep

Get the output step connected to this step by the specified output flowUUID.

Specified by:

getOutputStep in interface ETLStep

addOutputStep

public void addOutputStep(java.lang.String outFlowUuid,
                 ETLStep outputStep)

Description copied from interface: ETLStep

Register an output step.

Specified by:

addOutputStep in interface ETLStep

Parameters:

outFlowUuid - - FlowUUID connecting this step to the output step

outputStep - - The output step

getDefaultMetadataFields

public java.util.List<ETLStepMetadataFieldBean> getDefaultMetadataFields()

Description copied from interface: ETLStep

Get Default Metadata of this step. Default Metadata is the collection of all fields from all inputs to the step, and all fields generated internally.

Specified by:

getDefaultMetadataFields in interface ETLStep

getDefaultMetadataMap

@Deprecated
public java.util.Map<java.lang.String,ETLStepMetadataFieldBean> getDefaultMetadataMap()

Deprecated.

Use getDefaultMetadataFieldsMap instead.

getDefaultMetadataFieldsMap

public java.util.Map<java.lang.String,ETLStepMetadataFieldBean> getDefaultMetadataFieldsMap()

Return the Default Metadata Fields keyed by field UUID. Take care while using this as changes will affect the underlying data structure.

addDefaultMetadataField

public void addDefaultMetadataField(ETLStepMetadataFieldBean field)

Description copied from interface: ETLStep

Add a Default Metadata Field for this step. It inserts it in the correct position, as per the field's sortOrder field. If the sort order is null, negative or larger than the current highest sort order, the field is added at the end.

Specified by:

addDefaultMetadataField in interface ETLStep

See Also:

ETLStep.getDefaultMetadataFields()

removeDefaultMetadataField

public void removeDefaultMetadataField(java.lang.String fieldUuid)

Description copied from interface: ETLStep

Remove a Default Metadata Field from this step.

Specified by:

removeDefaultMetadataField in interface ETLStep

See Also:

ETLStep.getDefaultMetadataFields()

getDefaultMetadataFieldUuids

protected java.util.Set<java.lang.String> getDefaultMetadataFieldUuids()

Get a copy of Default Field UUIDs in a Set. Modifying this will not affect the underlying data structure.

getDefaultMetadataFieldUuidsList

protected java.util.List<java.lang.String> getDefaultMetadataFieldUuidsList()

Get a copy of Default Field UUIDs in a List. Modifying this will not affect the underlying data structure.

getMetadataFields

public java.util.List<ETLStepMetadataFieldBean> getMetadataFields(java.lang.String outFlowUuid)

Description copied from interface: ETLStep

Get Output Metadata Fields for a specified output flowUUID. Output Metadata is a subset of the Default Metadata. This is the structure of data which will be sent to the output step, along the output flow.

Specified by:

getMetadataFields in interface ETLStep

See Also:

ETLStep.getDefaultMetadataFields()

setMetadataFields

public void setMetadataFields(java.lang.String outFlowUuid,
                     ETLFlowType flowType,
                     java.util.List<ETLStepMetadataFieldBean> metadata)

Description copied from interface: ETLStep

Set Output Metadata Fields for a specified output flowUUID. The flow type specified in ETLFlowType must be specified as it determines which fields to filter out. Passed-in fields which should not be output will be removed.

Specified by:

setMetadataFields in interface ETLStep

See Also:

ETLStep.getMetadataFields(String)

getMinInputSteps

public java.lang.Integer getMinInputSteps()

Description copied from interface: ETLStep

Minimum number of inputs required for this step to work

Specified by:

getMinInputSteps in interface ETLStep

getMaxInputSteps

public java.lang.Integer getMaxInputSteps()

Description copied from interface: ETLStep

Maximum number of inputs this step supports

Specified by:

getMaxInputSteps in interface ETLStep

getMinOutputSteps

public java.lang.Integer getMinOutputSteps()

Description copied from interface: ETLStep

Minimum number of outputs required for this step to work

Specified by:

getMinOutputSteps in interface ETLStep

getMaxOutputSteps

public java.lang.Integer getMaxOutputSteps()

Description copied from interface: ETLStep

Maximum number of outputs this step supports

Specified by:

getMaxOutputSteps in interface ETLStep

getContext

public ETLRunContext getContext()

Description copied from interface: ETLStep

Get information about the ETL Runner running this step.

Specified by:

getContext in interface ETLStep

setContext

public void setContext(ETLRunContext context)

Description copied from interface: ETLStep

Set the step's ETLRunContext.

Specified by:

setContext in interface ETLStep

getStatus

public ETLStepStatus getStatus()

Description copied from interface: ETLStep

Get the status of the step as an instance of ETLStepStatus.

Specified by:

getStatus in interface ETLStep

setStatus

public void setStatus(ETLStepStatus status)

Description copied from interface: ETLStep

Set the status object for this step.

Specified by:

setStatus in interface ETLStep

getRowLimit

public long getRowLimit()

Description copied from interface: ETLStep

The maximum number of rows which will be output by this step.

Specified by:

getRowLimit in interface ETLStep

setRowLimit

public void setRowLimit(long rowLimit)

Description copied from interface: ETLStep

Set the step's rowLimit.

Specified by:

setRowLimit in interface ETLStep

See Also:

ETLStep.getRowLimit()

checkErrorThreshold

protected void checkErrorThreshold()
                            throws ETLStepErrorThresholdExceededException

Check if the step has exceeded this step's error threshold. When this happens, set the step's status to FAILED and throw an ETLStepErrorThresholdExceededException.

Throws:

ETLStepErrorThresholdExceededException

flowFinished

protected final void flowFinished(java.lang.String flowUuid)

Record that an input flow has finished sending data to this step.

isMainInputFlow

public boolean isMainInputFlow(java.lang.String inFlowUuid)

Is this the "main" input flow?

getErrorFlowUuid

public java.lang.String getErrorFlowUuid()

Description copied from interface: ETLStep

FlowUUID connecting this step to an error step.

Specified by:

getErrorFlowUuid in interface ETLStep

See Also:

ETLStep.getErrorStep()

getErrorStep

public ETLStep getErrorStep()

Description copied from interface: ETLStep

The error step. Errors which can be handled are sent to this step.

Specified by:

getErrorStep in interface ETLStep

See Also:

ETLStep.getErrorFlowUuid()

setErrorStep

public void setErrorStep(java.lang.String errorFlowUuid,
                ETLStep errorStep)

Description copied from interface: ETLStep

Register the error step.

Specified by:

setErrorStep in interface ETLStep

Parameters:

errorFlowUuid - - FlowUUID connecting this step to the error step

errorStep - - The error step

See Also:

ETLStep.getErrorStep(), ETLStep.getErrorFlowUuid()

getInputSteps

public java.util.Collection<ETLStep> getInputSteps()

Get all Input Steps as a Collection

getOutputSteps

public java.util.Collection<ETLStep> getOutputSteps()

Get all Output Steps as a Collection. This excludes the error step, although technically it is an output step.

hasPendingDependency

public boolean hasPendingDependency()

Are there any dependent input steps which have not finished executing?

waitForDependencies

public void waitForDependencies(long checkAfterMillis)
                         throws ETLException

Block the step's execution till all dependent input steps finish execution

Throws:

ETLException

throwUnhandledETLException

public void throwUnhandledETLException(java.lang.Throwable e)
                                throws ETLException

Throw an ETLException when the error cannot be handled.

Throws:

ETLException

getUnhandledETLException

public ETLException getUnhandledETLException(java.lang.Throwable e)

Get an ETLException when the error cannot be handled.

getInvalidConfigETLException

public ETLException getInvalidConfigETLException()

Get a generic ETLException when there is a problem in the step's config.

getETLException

public ETLException getETLException(java.lang.String messageKey)

Get an ETLException for the step with the message key that is passed in.

throwInvalidConfigETLException

public void throwInvalidConfigETLException()
                                    throws ETLException

Throw an ETLException when there is a problem in the step's config.

Throws:

ETLException

emitData

protected void emitData(ETLStepResults data)
                 throws ETLException,
                        java.lang.InterruptedException

Send data packets to all output steps.

Parameters:

data - - Packets of data to be sent to all outputs

Throws:

java.lang.InterruptedException - if the step was cancelled

ETLException

emitData

protected void emitData(ETLStepResult dataPacket)
                 throws ETLException,
                        java.lang.InterruptedException

Send a data packet to its output step

Parameters:

dataPacket - - Packet of data to be sent to an output

Throws:

java.lang.InterruptedException - if the step was cancelled

ETLException

getFreshDataPacket

public ETLStepResult getFreshDataPacket(java.lang.String outFlowUuid)

Get an empty data packet for a specified output flowUUID.

getFirstInputFlow

public java.lang.String getFirstInputFlow()

Return the first input flowUUID. This is useful in steps which supports max one input.

getFirstOutputFlow

public java.lang.String getFirstOutputFlow()

Return the first output flowUUID. This is useful in steps which supports max one output excluding the error flow.

getWireForField

public com.hof.mi.data.Wire<java.lang.Object,java.lang.String> getWireForField(java.lang.String fieldUuid)

Get the Wire corresponding to an ETLStepMetadataFieldUUID. The field may belong to input, default or output metadata.

clearAllWires

public void clearAllWires()

Clear all wires

getDefaultToInputFieldMap

public java.util.Map<java.lang.String,java.lang.String> getDefaultToInputFieldMap()

Get the mapping from default fields to input fields.

getInputToDefaultFieldMap

public java.util.Map<java.lang.String,java.lang.String> getInputToDefaultFieldMap()

Get the mapping from input fields to default fields.

shouldOutputField

public boolean shouldOutputField(java.lang.String flowUuid,
                        ETLStepMetadataFieldBean field)

Determine if a metadata field should be sent to the next step. This is required because fields may be output to the next "inline" step, even if a user decided not to output it. However, it shouldn't be output to the next "main" step.

Parameters:

flowUuid - - flow UUID leading to the next step

field - - The field being inspected

getPanelCollection

public ParameterPanelCollection getPanelCollection()

Get the ParameterPanelCollection object which represents the step's configuration UI elements. Calls generatePanelCollection if the step's panelCollection has not been setup yet.

Specified by:

getPanelCollection in interface ETLStep

generatePanelCollection

protected abstract ParameterPanelCollection generatePanelCollection()

Method to generate the ParameterPanelCollection object which represents the step's configuration UI elements.

beginInternalTransmission

protected void beginInternalTransmission(java.lang.Object[] data,
                             java.util.List<java.lang.String> fields)

Transmit a row of data from the Input, by placing data on internal wires.

Parameters:

data - - A row of data

fields - - FieldUUIDs for every index in the data array.

endInternalTransmission

protected ETLStepResult endInternalTransmission(ETLStepResult outputData)

Finish transmission by removing data from internal wires and putting them into an array in the output data packet (represented by ETLStepResult).

Parameters:

outputData - - The ETLStepResult object to be populated at the end of this transmission.

endInternalTransmission

protected ETLStepResult endInternalTransmission(ETLStepResult outputData,
                                    boolean clearWires)

Finish transmission by getting data from internal wires and putting them into an array in the output data packet (represented by ETLStepResult). Data is either retained or removed depending on the boolean parameter passed in.

Parameters:

outputData - - The ETLStepResult object to be populated at the end of this transmission.

clearWires - - true, if wires should be cleared.

setupDefaultFields

public void setupDefaultFields()
                        throws ETLException

Set up the default metadata fields for this step. Includes generated fields and input(s).

Specified by:

setupDefaultFields in interface ETLStep

Throws:

ETLException

See Also:

ETLStep.getDefaultMetadataFields()

getStepOptions

public java.util.Map<java.lang.String,java.lang.String> getStepOptions()

Description copied from interface: ETLStep

Get all config options for this step as a map of Option Key and Option Value. One cannot add/remove options by manipulating the returned map.

Specified by:

getStepOptions in interface ETLStep

See Also:

ETLStep.addStepOption(String, String), ETLStep.setStepOptions(Map)

getStepOption

public java.lang.String getStepOption(java.lang.String optionKey)

Description copied from interface: ETLStep

Get config option value for this step, corresponding to the specified option key.

Specified by:

getStepOption in interface ETLStep

addStepOption

public void addStepOption(java.lang.String optionKey,
                 java.lang.String optionValue)

Description copied from interface: ETLStep

Add a config option for this step.

Specified by:

addStepOption in interface ETLStep

setStepOptions

public void setStepOptions(java.util.Map<java.lang.String,java.lang.String> options)

Reset the step options with the provided values. Protected options (options which have been setup automatically by the process) are maintained and not overwritten.

Specified by:

setStepOptions in interface ETLStep

getInputFlowForDefaultMetadataGroup

public java.lang.String getInputFlowForDefaultMetadataGroup(java.lang.String groupUuid)

Group ID is used to represent which default metadata fields come from the same source. Given a groupId, this method identifies the flow UUID corresponding to its source.

setupOptions

public void setupOptions()

Description copied from interface: ETLStep

Set up step options. It creates "internal" step options such as those identifying generated fields. It also validates and fixes up step options. After this method runs, the step will have valid options for its present configuration.

Specified by:

setupOptions in interface ETLStep

setupGeneratedFields

public abstract void setupGeneratedFields()
                                   throws ETLException

Set up the default metadata fields for fields generated by this step.

Throws:

ETLException

getDefaultInternalOptions

public java.util.Map<java.lang.String,java.lang.String> getDefaultInternalOptions()

Return step optionKeys and optionValues for options internal to the step. These options are not visible to the user. The step uses them internally for its processing. The default implementation returns null.

getValidatedStepOptions

public abstract java.util.Map<java.lang.String,java.lang.String> getValidatedStepOptions()

Validate options returned by getStepOptions() and return a map of optionKeys and optionValues. This map represents the current validated configuration of the step.

getOnImage

public java.lang.String getOnImage()

Returns a byte array for a base 64 image

Specified by:

getOnImage in interface ETLStep

getOffImage

public java.lang.String getOffImage()

Returns a byte array for a base 64 image

Specified by:

getOffImage in interface ETLStep

getOnImageMIMEType

public java.lang.String getOnImageMIMEType()

Returns a MIMEType for a file path string

Specified by:

getOnImageMIMEType in interface ETLStep

getOffImageMIMEType

public java.lang.String getOffImageMIMEType()

Returns a MIMEType for a file path string

Specified by:

getOffImageMIMEType in interface ETLStep

getOnImagePath

protected java.lang.String getOnImagePath()

The "on" state image, relative to Yellowfin's root directory. For ROOT/images/etl/step_on.png, this method returns images/etl/step_on.png. The method in AbstractETLStep returns null. Override to implement.

getOffImagePath

protected java.lang.String getOffImagePath()

The "off" state image, relative to Yellowfin's root directory. For ROOT/images/etl/step_off.png, this method returns images/etl/step_off.png The method in AbstractETLStep returns null. Override to implement.

getStepSubCategory

public java.lang.String getStepSubCategory()

Set up a default stepSubCategory

Specified by:

getStepSubCategory in interface ETLStep

parseData

public void parseData(java.util.Map<java.lang.String,java.lang.Object> data)

Description copied from interface: ETLStep

Parse the data of panel collection from the front end to the correct format that should be saved into the database and return the updated step options

Specified by:

parseData in interface ETLStep

getStepClassification

public ETLStepClassification getStepClassification()

Description copied from interface: ETLStep

Returns a specific classification for an ETLStep

Specified by:

getStepClassification in interface ETLStep

getStepOptionValueJSONObject

public com.google.gson.JsonObject getStepOptionValueJSONObject(java.lang.String property)

getStepOptionValueJSONArray

public com.google.gson.JsonArray getStepOptionValueJSONArray(java.lang.String property)

getStepBooleanValue

public boolean getStepBooleanValue(java.lang.String property)

getStepIntegerValue

public java.lang.Integer getStepIntegerValue(java.lang.String property)

addNewGeneratedField

public ETLStepMetadataFieldBean addNewGeneratedField(ETLStepMetadataFieldBean field,
                                            java.lang.String optionKey)

Passes through to addGeneratedField(ETLStepMetadataFieldBean, ETLFieldLinkType, String) with the ETLFieldLinkType.NEWFIELD and if optionKey is not null, adds a step option containing the new field's UUID.

addGeneratedField

public ETLStepMetadataFieldBean addGeneratedField(ETLStepMetadataFieldBean field,
                                         ETLFieldLinkType linkType,
                                         java.lang.String linkFieldUUID)

Generated Metadata Fields are new fields which the step creates. This method adds a new Generated Field to the Default Metadata collection, setting attributes required for the field to be recognised as a generated field. NOTE: This method does not create a step option

Parameters:

field - - The ETLStepMetadataFieldBean which will be saved as a "generated" metadata field and have required properties set on it.

linkType - - The type of link which is represented by linkField, see ETLFieldLinkType for more info on values. INPUT is not an accepted value for this param. See ETLFieldLinkType}.

linkFieldUUID - - UUID of another default field which this new field will be linked to. Can be null in the case where linkType is NEWFIELD.

Returns:

The newly generated Default Metadata field.

getOutputFlowUuidsExceededRowLimit

protected java.util.Set<java.lang.String> getOutputFlowUuidsExceededRowLimit()

Return a set of output flow UUIDs which have exceeded the step's row limit

onCopy

public java.util.Map<java.lang.String,java.lang.String> onCopy(java.util.Map<java.lang.String,java.lang.String> uuidMap)

Description copied from interface: ETLStep

This method is called while the step is being copied and is intended to be used as a way for steps to copy their external dependencies. The step can also choose to return a map of options which need to be updated - this is used for steps which have complex options which are related to the copied entities. Any option keys which are returned with a null value will be deleted. NOTE: When this method is called, the entities which are setup on the step will be the old entities, not the newly copied ones.

Specified by:

onCopy in interface ETLStep

Parameters:

uuidMap - mapping of oldUUID to newUUID for any UUID's inside the step including stepUUID, fieldUUIDs, flowUUIDs, metadataUUIDs, groupUUIDs, etc. which have been copied.

Returns:

map of optionKey -> optionValue for any options which should be updated after copying.

onDelete

public void onDelete()

Description copied from interface: ETLStep

This method is called just before the step is deleted and is intended to be used as a way for steps to cleanup external dependencies.

Specified by:

onDelete in interface ETLStep

excludeDefaultField

protected void excludeDefaultField(java.lang.String fieldUUID)

Sets stepIncludeField to false on the given field and moves it's sort order to the end of the list. This updates the sort orders on any fields which were after the original position of the field.

Parameters:

fieldUUID - UUID of the default field to update

includeDefaultField

protected void includeDefaultField(java.lang.String fieldUUID,
                       java.lang.Integer newSortOrder)

Sets stepIncludeField to true on the given field. Optionally pass newSortOrder to update the field's position after including the field.

Parameters:

fieldUUID - UUID of the default field to update

newSortOrder - If not invalid, the new sort position to put the field at.

See Also:

updateFieldSortOrder(String, Integer)

replaceDefaultField

protected ETLStepMetadataFieldBean replaceDefaultField(ETLStepMetadataFieldBean fieldToReplace)

This method 'replaces' a default field with another one. Use this when adding a generated field which is supposed to act as a 'replacement' for the original field eg. Type conversion where the original field must exist in order to recieve the input data, but the output field needs to be of a different data type. The fieldToReplace will be excluded and pushed to the bottom of the fields list, and the new field will take it's place.

Parameters:

fieldToReplace - this field will be excluded

Returns:

new generated field which sits in the same sort position as the original and has been set up as a replacement for the original field.

restoreReplacedField

protected ETLStepMetadataFieldBean restoreReplacedField(ETLStepMetadataFieldBean replacementField)

This method restores the original replaced field back into it's place and removes the replacement field.

Parameters:

replacementField - the field which was acting as a replacement for another field, this will be removed.

Returns:

the original field, now restored.

isReplacementField

protected boolean isReplacementField(ETLStepMetadataFieldBean defField)

Is this field being used as a 'replacement' for another field?

updateFieldSortOrder

protected void updateFieldSortOrder(java.lang.String fieldUUID,
                        java.lang.Integer sortOrder)

Updates the sort order for a given existing default field. Does nothing if sortOrder is invalid or if the field does not exist.

isEnabled

public boolean isEnabled()

A step implementation may override to enable/disable itself. This may be based on a configuration parameter or some other logic internal to the step.

Specified by:

isEnabled in interface ETLStep

disable

public final void disable(boolean disabled)

Description copied from interface: ETLStep

Disable the step

Specified by:

disable in interface ETLStep

stepCanRun

protected final boolean stepCanRun()

Can this step run? Find out based on the "disabled" flag and whether the step implementation has enabled itself.

getFile

public byte[] getFile(java.lang.Integer fileId)

Get a file that is associated with the passed fileId

Parameters:

fileId -

getFile

public byte[] getFile(java.lang.String property)

Get a file that is associated with the passed property

Parameters:

property -

getFileName

public java.lang.String getFileName(java.lang.Integer fileId)

Helper method to get the file name of the selected passed in fileId

Parameters:

fileId -

getFileName

public java.lang.String getFileName(java.lang.String property)

Helper method to get the file name of the selected passed in property

Parameters:

property -

getText

public java.lang.String getText(java.lang.Integer textId)

Get the text associated with the passed text id

Parameters:

textId -

getText

public java.lang.String getText(java.lang.String property)

Get the text associated with the passed property

Parameters:

property -

addError

protected final void addError(ETLException e)
                       throws ETLStepErrorThresholdExceededException

Use this method to record errors during execution of the step. After adding the error, it checks if the row threshold was exceeded and throws an ETLStepErrorThresholdExceededException if it has.

Throws:

ETLStepErrorThresholdExceededException - if the error threshold is breached

isGeneratedField

protected final boolean isGeneratedField(ETLStepMetadataFieldBean field)

Is the field generated by the step?

getEtlDataTypeUtil

protected com.hof.mi.etl.step.ETLDataTypeUtil getEtlDataTypeUtil()

getStringUtil

protected com.hof.util.UtilStringService getStringUtil()