Modifier and Type | Class and Description |
---|---|
protected class |
Bundle.BundleValue |
protected class |
Bundle.PoolTransfer
A container to store the transfers that need to be done on a single pool.
|
protected class |
Bundle.TransferContainer
A container class for storing the name of the transfer job, the list of
file transfers that the job is responsible for.
|
Modifier and Type | Field and Description |
---|---|
static java.lang.String |
DEFAULT_LOCAL_STAGE_IN_BUNDLE_FACTOR
The default bundling factor that identifies the number of transfer jobs
that are being created per execution pool for stageing in data for
the workflow.
|
static java.lang.String |
DEFAULT_LOCAL_STAGE_OUT_BUNDLE_FACTOR
The default bundling factor that identifies the number of transfer jobs
that are being created per execution pool for stageing out data for
the workflow.
|
static java.lang.String |
DEFAULT_REMOTE_STAGE_IN_BUNDLE_FACTOR
The default bundling factor that identifies the number of transfer jobs
that are being created per execution pool for stageing in data for
the workflow.
|
static java.lang.String |
DEFAULT_REMOTE_STAGE_OUT_BUNDLE_FACTOR
The default bundling factor that identifies the number of transfer jobs
that are being created per execution pool for stageing out data for
the workflow.
|
static java.lang.String |
DESCRIPTION
A short description of the transfer refinement.
|
protected boolean |
mAddNodesForSettingXBit
A boolean indicating whether chmod jobs should be created that set the
xbit in case of executable staging.
|
private int |
mCurrentSOLevel
The current level of the jobs being traversed.
|
protected java.lang.String |
mJobPrefix
The job prefix that needs to be applied to the job file basenames.
|
protected Pegasus |
mPegasusProfilesInProperties
Pegasus Profiles specified in the properties.
|
private ReplicaCatalogBridge |
mRCB
The handle to the replica catalog bridge.
|
private java.util.Map |
mRelationsParentMap
The map indexed by compute jobnames that contains the list of stagin job
names that are being added during the traversal of the workflow.
|
protected java.util.Map |
mSetupMap
The map indexed by staged executable logical name.
|
protected SiteStore |
mSiteStore
Handle to the SiteStore
|
protected Bundle.BundleValue |
mStageinLocalBundleValue
The BundleValue that evaluates for local stage in jobs.
|
private java.util.Map |
mStageInLocalMap
The map containing the list of stage in transfer jobs that are being
created for the workflow indexed by the execution poolname.
|
protected Bundle.BundleValue |
mStageInRemoteBundleValue
The BundleValue that evaluates for remote stage-in jobs.
|
private java.util.Map |
mStageInRemoteMap
The map containing the list of stage in transfer jobs that are being
created for the workflow indexed by the execution poolname.
|
protected Bundle.BundleValue |
mStageOutLocalBundleValue
The BundleValue that evaluates for local stage out jobs.
|
private java.util.Map<java.lang.String,Bundle.PoolTransfer> |
mStageOutLocalMapPerLevel
A map indexed by site name, that contains the pointer to the local stage out
PoolTransfer objects for that site.
|
protected Bundle.BundleValue |
mStageOutRemoteBundleValue
The BundleValue that evaluates for remote stage out jobs.
|
private java.util.Map<java.lang.String,Bundle.PoolTransfer> |
mStageOutRemoteMapPerLevel
A map indexed by site name, that contains the pointer to the remote stage out
PoolTransfer objects for that site.
|
mCreateRegistrationJobs, mFileTable, mLogMsg, mPPS, mRelationsMap
mDAG, mLogger, mPOptions, mProps, mRemoteTransfers, mTPT, mTXInterImplementation, mTXStageInImplementation, mTXStageOutImplementation, mTXSymbolicLinkImplementation, mXMLStore
INTER_POOL_PREFIX, LOCAL_PREFIX, REGISTER_PREFIX, REMOTE_PREFIX, STAGE_IN_PREFIX, STAGE_OUT_PREFIX
Constructor and Description |
---|
Bundle(ADag dag,
PegasusBag bag)
The overloaded constructor.
|
Modifier and Type | Method and Description |
---|---|
void |
addStageInXFERNodes(Job job,
boolean localTransfer,
java.util.Collection files,
int type,
java.util.Map<java.lang.String,Bundle.PoolTransfer> stageInMap,
Bundle.BundleValue bundleValue,
Implementation implementation)
Adds the stage in transfer nodes which transfer the input files for a job,
from the location returned from the replica catalog to the job's execution
pool.
|
void |
addStageInXFERNodes(Job job,
java.util.Collection<FileTransfer> files,
java.util.Collection<FileTransfer> symlinkFiles)
Adds the stage in transfer nodes which transfer the input files for a job,
from the location returned from the replica catalog to the job's execution
pool.
|
void |
addStageOutXFERNodes(Job job,
java.util.Collection files,
ReplicaCatalogBridge rcb,
boolean localTransfer,
boolean deletedLeaf)
Adds the stageout transfer nodes, that stage data to an output site
specified by the user.
|
void |
done()
Signals that the traversal of the workflow is done.
|
void |
doneStageIn(java.util.Map<java.lang.String,Bundle.PoolTransfer> stageInMap,
Implementation implementation,
int stageInJobType,
boolean localTransfer)
Signals that the traversal of the workflow is done.
|
protected java.lang.String |
getComputeJobBundleValue(Job job)
Returns the bundle value associated with a compute job as a String.
|
protected java.lang.String |
getDefaultBundleValueFromProperties(java.lang.String key,
java.lang.String defaultKey,
java.lang.String defaultValue)
Returns the default value for the clustering/bundling of jobs to be used.
|
java.lang.String |
getDescription()
Returns a textual description of the transfer mode.
|
Bundle.PoolTransfer |
getStageOutPoolTransfer(java.lang.String site,
boolean localTransfer,
int num)
Returns the appropriate pool transfer for a particular site.
|
protected void |
initializeBundleValues()
Initializes the bundle value variables, that are responsible determining
the bundle values.
|
protected java.util.Map |
resetStageOutMap(java.util.Map<java.lang.String,Bundle.PoolTransfer> map,
boolean localTransfer)
Resets a single map
|
protected void |
resetStageOutMaps()
Resets the local and remote stage out maps.
|
addInterSiteTXNodes, addJob, addRelation, addRelation, addStageInXFERNodes, addStageOutXFERNodes, appendAttribute, constructFileKey, createRegistrationJob, getJobPriority, logRefinerAction
loadImplementations
addStageInXFERNodes, getWorkflow, getXMLProducer, isSiteThirdParty, logConfigMessages, refinerPreferenceForLocalTransferJobs, refinerPreferenceForTransferJobLocation, runTPTOnRemoteSite, runTransferRemotely
public static final java.lang.String DESCRIPTION
public static final java.lang.String DEFAULT_LOCAL_STAGE_IN_BUNDLE_FACTOR
public static final java.lang.String DEFAULT_REMOTE_STAGE_IN_BUNDLE_FACTOR
public static final java.lang.String DEFAULT_LOCAL_STAGE_OUT_BUNDLE_FACTOR
public static final java.lang.String DEFAULT_REMOTE_STAGE_OUT_BUNDLE_FACTOR
private java.util.Map mStageInLocalMap
private java.util.Map mStageInRemoteMap
private java.util.Map mRelationsParentMap
protected Bundle.BundleValue mStageinLocalBundleValue
protected Bundle.BundleValue mStageInRemoteBundleValue
protected Bundle.BundleValue mStageOutLocalBundleValue
protected Bundle.BundleValue mStageOutRemoteBundleValue
protected java.util.Map mSetupMap
private java.util.Map<java.lang.String,Bundle.PoolTransfer> mStageOutLocalMapPerLevel
private java.util.Map<java.lang.String,Bundle.PoolTransfer> mStageOutRemoteMapPerLevel
private int mCurrentSOLevel
private ReplicaCatalogBridge mRCB
protected java.lang.String mJobPrefix
protected Pegasus mPegasusProfilesInProperties
protected SiteStore mSiteStore
protected boolean mAddNodesForSettingXBit
public Bundle(ADag dag, PegasusBag bag)
dag
- the workflow to which transfer nodes need to be added.bag
- the bag of initialization objectsprotected void initializeBundleValues()
protected java.lang.String getDefaultBundleValueFromProperties(java.lang.String key, java.lang.String defaultKey, java.lang.String defaultValue)
return value of pegasus profile key if it exists, else return value of pegasus profile defaultKey if it exists, else the defaultValue
key
- the pegasus profile keydefaultKey
- the default pegasus profile keydefaultValue
- the default value.public void addStageInXFERNodes(Job job, java.util.Collection<FileTransfer> files, java.util.Collection<FileTransfer> symlinkFiles)
addStageInXFERNodes
in class Basic
job
- Job
object corresponding to the node to
which the files are to be transferred to.files
- Collection of FileTransfer
objects containing the
information about source and destURL's.symlinkFiles
- Collection of FileTransfer
objects containing
source and destination file url's for symbolic linking
on compute site.public void addStageInXFERNodes(Job job, boolean localTransfer, java.util.Collection files, int type, java.util.Map<java.lang.String,Bundle.PoolTransfer> stageInMap, Bundle.BundleValue bundleValue, Implementation implementation)
job
- Job
object corresponding to the node to
which the files are to be transferred to.localTransfer
- boolean indicating whether transfer has to happen on local site.files
- Collection of FileTransfer
objects containing the
information about source and destURL's.type
- the type of transfer job being createdstageInMap
- Map indexed by site name that gives all the transfers for
that site.bundleValue
- used to determine the bundling factor to employ for a job.implementation
- the transfer implementation to use.public void addStageOutXFERNodes(Job job, java.util.Collection files, ReplicaCatalogBridge rcb, boolean localTransfer, boolean deletedLeaf)
addStageOutXFERNodes
in class Basic
job
- Job
object corresponding to the node to
which the files are to be transferred to.files
- Collection of FileTransfer
objects containing the
information about source and destURL's.rcb
- bridge to the Replica Catalog. Used for creating registration
nodes in the workflow.localTransfer
- whether the transfer should be on local site or not.deletedLeaf
- to specify whether the node is being added for
a deleted node by the reduction engine or not.
default: falseprotected java.lang.String getComputeJobBundleValue(Job job)
job
- public void done()
public void doneStageIn(java.util.Map<java.lang.String,Bundle.PoolTransfer> stageInMap, Implementation implementation, int stageInJobType, boolean localTransfer)
stageInMap
- maps site names to PoolTransferimplementation
- the transfer implementation to usestageInJobType
- whether a stagein or symlink stagein joblocalTransfer
- indicates whether transfer job needs to run on
local site or not.public java.lang.String getDescription()
getDescription
in class Basic
public Bundle.PoolTransfer getStageOutPoolTransfer(java.lang.String site, boolean localTransfer, int num)
site
- the site for which the PT is reqd.localTransfer
- whethe the associated transfer job runs on local site or remote.num
- the number of Stageout jobs required for that Pool.protected void resetStageOutMaps()
protected java.util.Map resetStageOutMap(java.util.Map<java.lang.String,Bundle.PoolTransfer> map, boolean localTransfer)
map
- the map to be resetlocalTransfer
- whether the transfer jobs need to run on local site or not