python.trfExe.transformExecutor Class Reference

Executors always only even execute a single step, as seen by the transform. More...

Inheritance diagram for python.trfExe.transformExecutor:

Collaboration diagram for python.trfExe.transformExecutor:

Public Member Functions
def	__init__ (self, name='Dummy', trf=None, conf=None, inData=set(), outData=set())
	Base class initaliser for transform executors. More...
def	myMerger (self)
	Now define properties for these data members. More...
def	name (self)
def	name (self, value)
def	substep (self)
def	trf (self)
def	trf (self, value)
def	inData (self)
def	inData (self, value)
def	inDataUpdate (self, value)
def	outData (self)
def	outData (self, value)
def	outDataUpdate (self, value)
def	input (self)
def	output (self)
def	extraMetadata (self)
def	hasExecuted (self)
def	rc (self)
def	errMsg (self)
def	validation (self)
def	validation (self, value)
def	hasValidated (self)
def	isValidated (self)
def	first (self)
def	preExeStartTimes (self)
def	exeStartTimes (self)
def	exeStopTimes (self)
def	valStartTimes (self)
def	valStopTimes (self)
def	preExeCpuTime (self)
def	preExeWallTime (self)
def	cpuTime (self)
def	usrTime (self)
def	sysTime (self)
def	wallTime (self)
def	memStats (self)
def	memAnalysis (self)
def	postExeCpuTime (self)
def	postExeWallTime (self)
def	validationCpuTime (self)
def	validationWallTime (self)
def	cpuTimeTotal (self)
def	wallTimeTotal (self)
def	eventCount (self)
def	reSimEvent (self)
def	athenaMP (self)
def	dbMonitor (self)
def	setPreExeStart (self)
def	setValStart (self)
def	preExecute (self, input=set(), output=set())
def	execute (self)
def	postExecute (self)
def	validate (self)
def	doAll (self, input=set(), output=set())
	Convenience function. More...

Public Attributes
	conf
	Executor configuration: More...
	inData
	outData

Private Attributes
	_name
	_inData
	_outData
	_hasExecuted
	_rc
	_errMsg
	_hasValidated
	_isValidated
	_extraMetadata
	_preExeStart
	_exeStart
	_exeStop
	_valStart
	_valStop
	_memStats
	_memLeakResult
	_memFullFile
	_eventCount
	_athenaMP
	_athenaMT
	_athenaConcurrentEvents
	_dbMonitor
	_resimevents
	_alreadyInContainer
	_containerSetup
	_myMerger
	_trf
	_validation

Detailed Description

Executors always only even execute a single step, as seen by the transform.

Definition at line 127 of file trfExe.py.

Constructor & Destructor Documentation

__init__()

def python.trfExe.transformExecutor.__init__	(		self,
			name = 'Dummy',
			trf = None,
			conf = None,
			inData = set(),
			outData = set()
	)

Base class initaliser for transform executors.

Parameters

name	Transform name
trf	Parent transform
conf	executorConfig object (if None then set from the trf directly)
inData	Data inputs this transform can start from. This should be a list, tuple or set consisting of each input data type. If a tuple (or list) is passed as a set member then this is interpreted as meaning that all of the data members in that tuple are necessary as an input.

Note

Curiously, sets are not allowed to be members of sets (they are not hashable, so no sub-sets)

Parameters

outData

List of outputs this transform can produce (list, tuple or set can be used)

Reimplemented in python.trfExe.dummyExecutor.

Definition at line 138 of file trfExe.py.

    def __init__(self, name = 'Dummy', trf = None, conf = None, inData = set(), outData = set()):
        # Some information to produce helpful log messages
        
        self._name = forceToAlphaNum(name)
        # Data this executor can start from and produce
        # Note we transform NULL to inNULL and outNULL as a convenience
        self._inData = set(inData)
        self._outData = set(outData)
        if 'NULL' in self._inData:
            self._inData.remove('NULL')
            self._inData.add('inNULL')
        if 'NULL' in self._outData:
            self._outData.remove('NULL')
            self._outData.add('outNULL')
            
        # It's forbidden for an executor to consume and produce the same datatype
        dataOverlap = self._inData & self._outData
        if len(dataOverlap) > 0:
            raise trfExceptions.TransformSetupException(trfExit.nameToCode('TRF_GRAPH_ERROR'), 
                                                        'Executor definition error, executor {0} is not allowed to produce and consume the same datatypes. Duplicated input/output types {1}'.format(self._name, ' '.join(dataOverlap)))
        
        
        if conf is not None:
            self.conf = conf
        else:
            self.conf = executorConfig()
            if trf is not None:
                self.conf.setFromTransform(trf)
        
        # Execution status
        self._hasExecuted = False
        self._rc = -1
        self._errMsg = None
        
        # Validation status
        self._hasValidated = False
        self._isValidated = False
        
        # Extra metadata
        # This dictionary holds extra metadata for this executor which will be 
        # provided in job reports
        self._extraMetadata = {}
        
        
        self._preExeStart = None
        self._exeStart = self._exeStop = None
        self._valStart = self._valStop = None
        self._memStats = {}
        self._memLeakResult = {}
        self._memFullFile = None
        self._eventCount = None
        self._athenaMP = 0
        self._athenaMT = 0
        self._athenaConcurrentEvents = 0
        self._dbMonitor = None
        self._resimevents = None
        self._alreadyInContainer = None
        self._containerSetup = None
 
        # Holder for execution information about any merges done by this executor in MP mode
        self._myMerger = []
 
        

Member Function Documentation

athenaMP()

def python.trfExe.transformExecutor.athenaMP

(

self

)

Definition at line 452 of file trfExe.py.

    def athenaMP(self):
        return self._athenaMP
 

cpuTime()

def python.trfExe.transformExecutor.cpuTime

(

self

)

Definition at line 366 of file trfExe.py.

    def cpuTime(self):
        if self._exeStart and self._exeStop:
            return calcCpuTime(self._exeStart, self._exeStop)
        else:
            return None
 

cpuTimeTotal()

def python.trfExe.transformExecutor.cpuTimeTotal

(

self

)

Definition at line 430 of file trfExe.py.

    def cpuTimeTotal(self):
        if self._preExeStart and self._valStop:
            return calcCpuTime(self._preExeStart, self._valStop)
        else:
            return None
 

dbMonitor()

def python.trfExe.transformExecutor.dbMonitor

(

self

)

Definition at line 456 of file trfExe.py.

    def dbMonitor(self):
        return self._dbMonitor
 
 

doAll()

def python.trfExe.transformExecutor.doAll	(		self,
			input = set(),
			output = set()
	)

Convenience function.

Definition at line 499 of file trfExe.py.

    def doAll(self, input=set(), output=set()):
        self.preExecute(input, output)
        self.execute()
        self.postExecute()
        self.validate()
 

errMsg()

def python.trfExe.transformExecutor.errMsg

(

self

)

Definition at line 304 of file trfExe.py.

    def errMsg(self):
        return self._errMsg
    

eventCount()

def python.trfExe.transformExecutor.eventCount

(

self

)

Definition at line 444 of file trfExe.py.

    def eventCount(self):
        return self._eventCount
 

execute()

def python.trfExe.transformExecutor.execute

(

self

)

Reimplemented in python.trfExe.bsMergeExecutor, python.trfExe.POOLMergeExecutor, python.trfExe.scriptExecutor, python.trfExe.dummyExecutor, and python.trfExe.echoExecutor.

Definition at line 475 of file trfExe.py.

    def execute(self):
        self._exeStart = os.times()
        msg.debug('exeStart time is {0}'.format(self._exeStart))
        msg.info('Starting execution of %s', self._name)
        self._hasExecuted = True
        self._rc = 0
        self._errMsg = ''
        msg.info('%s executor returns %d', self._name, self._rc)
        self._exeStop = os.times()
        msg.debug('preExeStop time is {0}'.format(self._exeStop))
        

exeStartTimes()

def python.trfExe.transformExecutor.exeStartTimes

(

self

)

Definition at line 336 of file trfExe.py.

    def exeStartTimes(self):
        return self._exeStart
 

exeStopTimes()

def python.trfExe.transformExecutor.exeStopTimes

(

self

)

Definition at line 340 of file trfExe.py.

    def exeStopTimes(self):
        return self._exeStop
 

extraMetadata()

def python.trfExe.transformExecutor.extraMetadata

(

self

)

Definition at line 292 of file trfExe.py.

    def extraMetadata(self):
        return self._extraMetadata
    

first()

def python.trfExe.transformExecutor.first

(

self

)

Note

At the moment only athenaExecutor sets this property, but that might be changed...

Definition at line 325 of file trfExe.py.

    def first(self):
        if hasattr(self, '_first'):
            return self._first
        else:
            return None
 

hasExecuted()

def python.trfExe.transformExecutor.hasExecuted

(

self

)

Definition at line 296 of file trfExe.py.

    def hasExecuted(self):
        return self._hasExecuted
    

hasValidated()

def python.trfExe.transformExecutor.hasValidated

(

self

)

Definition at line 316 of file trfExe.py.

    def hasValidated(self):
        return self._hasValidated
    

inData() [1/2]

def python.trfExe.transformExecutor.inData

(

self

)

Definition at line 235 of file trfExe.py.

    def inData(self):
        
        if '_inData' in dir(self):
            return self._inData
        return None
    

inData() [2/2]

def python.trfExe.transformExecutor.inData	(		self,
			value
	)

Definition at line 242 of file trfExe.py.

    def inData(self, value):
        self._inData = set(value)
        

inDataUpdate()

def python.trfExe.transformExecutor.inDataUpdate	(		self,
			value
	)

Definition at line 245 of file trfExe.py.

    def inDataUpdate(self, value):
        
        if '_inData' in dir(self):
            self._inData.update(value)
        else:
            
            self.inData = value
 
 

input()

def python.trfExe.transformExecutor.input

(

self

)

Note

This returns the actual input data with which this executor ran (c.f. inData which returns all the possible data types this executor could run with)

Definition at line 276 of file trfExe.py.

    def input(self):
        
        if '_input' in dir(self):
            return self._input
        return None
    

isValidated()

def python.trfExe.transformExecutor.isValidated

(

self

)

Definition at line 320 of file trfExe.py.

    def isValidated(self):
        return self._isValidated
    

memAnalysis()

def python.trfExe.transformExecutor.memAnalysis

(

self

)

Definition at line 398 of file trfExe.py.

    def memAnalysis(self):
        return self._memLeakResult
 

memStats()

def python.trfExe.transformExecutor.memStats

(

self

)

Definition at line 394 of file trfExe.py.

    def memStats(self):
        return self._memStats
 

myMerger()

def python.trfExe.transformExecutor.myMerger

(

self

)

Now define properties for these data members.

Definition at line 207 of file trfExe.py.

    def myMerger(self):
        return self._myMerger
 

name() [1/2]

def python.trfExe.transformExecutor.name

(

self

)

Definition at line 211 of file trfExe.py.

    def name(self):
        return self._name
 

name() [2/2]

def python.trfExe.transformExecutor.name	(		self,
			value
	)

Definition at line 215 of file trfExe.py.

    def name(self, value):
        self._name = value
 

outData() [1/2]

def python.trfExe.transformExecutor.outData

(

self

)

Definition at line 255 of file trfExe.py.

    def outData(self):
        
        if '_outData' in dir(self):
            return self._outData
        return None
 

outData() [2/2]

def python.trfExe.transformExecutor.outData	(		self,
			value
	)

Definition at line 262 of file trfExe.py.

    def outData(self, value):
        self._outData = set(value)
    

outDataUpdate()

def python.trfExe.transformExecutor.outDataUpdate	(		self,
			value
	)

Definition at line 265 of file trfExe.py.

    def outDataUpdate(self, value):
        
        if '_outData' in dir(self):
            self._outData.update(value)
        else:
            
            self.outData = value
    

output()

def python.trfExe.transformExecutor.output

(

self

)

Note

This returns the actual output data with which this executor ran (c.f. outData which returns all the possible data types this executor could run with)

Definition at line 285 of file trfExe.py.

    def output(self):
        
        if '_output' in dir(self):
            return self._output
        return None
    

postExeCpuTime()

def python.trfExe.transformExecutor.postExeCpuTime

(

self

)

Definition at line 402 of file trfExe.py.

    def postExeCpuTime(self):
        if self._exeStop and self._valStart:
            return calcCpuTime(self._exeStop, self._valStart)
        else:
            return None
 

postExecute()

def python.trfExe.transformExecutor.postExecute

(

self

)

Reimplemented in python.trfExe.bsMergeExecutor, python.trfExe.athenaExecutor, and python.trfExe.scriptExecutor.

Definition at line 486 of file trfExe.py.

    def postExecute(self):
        msg.info('Postexecute for %s', self._name)
        

postExeWallTime()

def python.trfExe.transformExecutor.postExeWallTime

(

self

)

Definition at line 409 of file trfExe.py.

    def postExeWallTime(self):
        if self._exeStop and self._valStart:
            return calcWallTime(self._exeStop, self._valStart)
        else:
            return None
 

preExeCpuTime()

def python.trfExe.transformExecutor.preExeCpuTime

(

self

)

Definition at line 352 of file trfExe.py.

    def preExeCpuTime(self):
        if self._preExeStart and self._exeStart:
            return calcCpuTime(self._preExeStart, self._exeStart)
        else:
            return None
 

preExecute()

def python.trfExe.transformExecutor.preExecute	(		self,
			input = set(),
			output = set()
	)

Reimplemented in python.trfExe.archiveExecutor, python.trfExe.bsMergeExecutor, python.trfExe.NtupPhysValPostProcessingExecutor, python.trfExe.NTUPMergeExecutor, python.trfExe.DQMPostProcessExecutor, python.trfExe.DQMergeExecutor, python.trfExe.reductionFrameworkExecutor, python.trfExe.POOLMergeExecutor, python.trfExe.athenaExecutor, python.trfExe.scriptExecutor, and python.trfExe.logscanExecutor.

Definition at line 471 of file trfExe.py.

    def preExecute(self, input = set(), output = set()):
        self.setPreExeStart()
        msg.info('Preexecute for %s', self._name)
        

preExeStartTimes()

def python.trfExe.transformExecutor.preExeStartTimes

(

self

)

Definition at line 332 of file trfExe.py.

    def preExeStartTimes(self):
        return self._preExeStart
    

preExeWallTime()

def python.trfExe.transformExecutor.preExeWallTime

(

self

)

Definition at line 359 of file trfExe.py.

    def preExeWallTime(self):
        if self._preExeStart and self._exeStart:
            return calcWallTime(self._preExeStart, self._exeStart)
        else:
            return None
 

rc()

def python.trfExe.transformExecutor.rc

(

self

)

Definition at line 300 of file trfExe.py.

    def rc(self):
        return self._rc
    

reSimEvent()

def python.trfExe.transformExecutor.reSimEvent

(

self

)

Definition at line 448 of file trfExe.py.

    def reSimEvent(self):
        return self._resimevents
 

setPreExeStart()

def python.trfExe.transformExecutor.setPreExeStart

(

self

)

Definition at line 461 of file trfExe.py.

    def setPreExeStart(self):
        if self._preExeStart is None:
            self._preExeStart = os.times()
            msg.debug('preExeStart time is {0}'.format(self._preExeStart))
 

setValStart()

def python.trfExe.transformExecutor.setValStart

(

self

)

Definition at line 466 of file trfExe.py.

    def setValStart(self):
        if self._valStart is None:
            self._valStart = os.times()
            msg.debug('valStart time is {0}'.format(self._valStart))
 

substep()

def python.trfExe.transformExecutor.substep

(

self

)

Reimplemented in python.trfExe.athenaExecutor.

Definition at line 219 of file trfExe.py.

    def substep(self):
        if '_substep' in dir(self):
            return self._substep
        return None
    

sysTime()

def python.trfExe.transformExecutor.sysTime

(

self

)

Definition at line 380 of file trfExe.py.

    def sysTime(self):
        if self._exeStart and self._exeStop:
            return self._exeStop[3] - self._exeStart[3]
        else:
            return None
 

trf() [1/2]

def python.trfExe.transformExecutor.trf

(

self

)

Definition at line 225 of file trfExe.py.

    def trf(self):
        if '_trf' in dir(self):
            return self._trf
        return None
    

trf() [2/2]

def python.trfExe.transformExecutor.trf	(		self,
			value
	)

Definition at line 231 of file trfExe.py.

    def trf(self, value):
        self._trf = value
        

usrTime()

def python.trfExe.transformExecutor.usrTime

(

self

)

Definition at line 373 of file trfExe.py.

    def usrTime(self):
        if self._exeStart and self._exeStop:
            return self._exeStop[2] - self._exeStart[2]
        else:
            return None
        

validate()

def python.trfExe.transformExecutor.validate

(

self

)

Reimplemented in python.trfExe.DQMPostProcessExecutor, python.trfExe.DQMergeExecutor, python.trfExe.optionalAthenaExecutor, python.trfExe.athenaExecutor, python.trfExe.scriptExecutor, and python.trfExe.logscanExecutor.

Definition at line 489 of file trfExe.py.

    def validate(self):
        self.setValStart()
        self._hasValidated = True        
        msg.info('Executor %s has no validation function - assuming all ok', self._name)
        self._isValidated = True
        self._errMsg = ''
        self._valStop = os.times()
        msg.debug('valStop time is {0}'.format(self._valStop))
    

validation() [1/2]

def python.trfExe.transformExecutor.validation

(

self

)

Definition at line 308 of file trfExe.py.

    def validation(self):
        return self._validation
    

validation() [2/2]

def python.trfExe.transformExecutor.validation	(		self,
			value
	)

Definition at line 312 of file trfExe.py.

    def validation(self, value):
        self._validation = value
        

validationCpuTime()

def python.trfExe.transformExecutor.validationCpuTime

(

self

)

Definition at line 416 of file trfExe.py.

    def validationCpuTime(self):
        if self._valStart and self._valStop:
            return calcCpuTime(self._valStart, self._valStop)
        else:
            return None
    

validationWallTime()

def python.trfExe.transformExecutor.validationWallTime

(

self

)

Definition at line 423 of file trfExe.py.

    def validationWallTime(self):
        if self._valStart and self._valStop:
            return calcWallTime(self._valStart, self._valStop)
        else:
            return None
 

valStartTimes()

def python.trfExe.transformExecutor.valStartTimes

(

self

)

Definition at line 344 of file trfExe.py.

    def valStartTimes(self):
        return self._valStart
 

valStopTimes()

def python.trfExe.transformExecutor.valStopTimes

(

self

)

Definition at line 348 of file trfExe.py.

    def valStopTimes(self):
        return self._valStop
    

wallTime()

def python.trfExe.transformExecutor.wallTime

(

self

)

Definition at line 387 of file trfExe.py.

    def wallTime(self):
        if self._exeStart and self._exeStop:
            return calcWallTime(self._exeStart, self._exeStop)
        else:
            return None
        

wallTimeTotal()

def python.trfExe.transformExecutor.wallTimeTotal

(

self

)

Definition at line 437 of file trfExe.py.

    def wallTimeTotal(self):
        if self._preExeStart and self._valStop:
            return calcWallTime(self._preExeStart, self._valStop)
        else:
            return None
        

Member Data Documentation

_alreadyInContainer

python.trfExe.transformExecutor._alreadyInContainer

private

Definition at line 198 of file trfExe.py.

_athenaConcurrentEvents

python.trfExe.transformExecutor._athenaConcurrentEvents

private

Definition at line 195 of file trfExe.py.

_athenaMP

python.trfExe.transformExecutor._athenaMP

private

Definition at line 193 of file trfExe.py.

_athenaMT

python.trfExe.transformExecutor._athenaMT

private

Definition at line 194 of file trfExe.py.

_containerSetup

python.trfExe.transformExecutor._containerSetup

private

Definition at line 199 of file trfExe.py.

_dbMonitor

python.trfExe.transformExecutor._dbMonitor

private

Definition at line 196 of file trfExe.py.

_errMsg

python.trfExe.transformExecutor._errMsg

private

Definition at line 172 of file trfExe.py.

_eventCount

python.trfExe.transformExecutor._eventCount

private

Definition at line 192 of file trfExe.py.

_exeStart

python.trfExe.transformExecutor._exeStart

private

Definition at line 187 of file trfExe.py.

_exeStop

python.trfExe.transformExecutor._exeStop

private

Definition at line 187 of file trfExe.py.

_extraMetadata

python.trfExe.transformExecutor._extraMetadata

private

Definition at line 181 of file trfExe.py.

_hasExecuted

python.trfExe.transformExecutor._hasExecuted

private

Definition at line 170 of file trfExe.py.

_hasValidated

python.trfExe.transformExecutor._hasValidated

private

Definition at line 175 of file trfExe.py.

_inData

python.trfExe.transformExecutor._inData

private

Definition at line 144 of file trfExe.py.

_isValidated

python.trfExe.transformExecutor._isValidated

private

Definition at line 176 of file trfExe.py.

_memFullFile

python.trfExe.transformExecutor._memFullFile

private

Definition at line 191 of file trfExe.py.

_memLeakResult

python.trfExe.transformExecutor._memLeakResult

private

Definition at line 190 of file trfExe.py.

_memStats

python.trfExe.transformExecutor._memStats

private

Definition at line 189 of file trfExe.py.

_myMerger

python.trfExe.transformExecutor._myMerger

private

Definition at line 202 of file trfExe.py.

_name

python.trfExe.transformExecutor._name

private

Definition at line 141 of file trfExe.py.

_outData

python.trfExe.transformExecutor._outData

private

Definition at line 145 of file trfExe.py.

_preExeStart

python.trfExe.transformExecutor._preExeStart

private

Note

Place holders for resource consumption. CPU and walltime are available for all executors but currently only athena is instrumented to fill in memory stats (and then only if PerfMonSD is enabled).

Definition at line 186 of file trfExe.py.

_rc

python.trfExe.transformExecutor._rc

private

Definition at line 171 of file trfExe.py.

_resimevents

python.trfExe.transformExecutor._resimevents

private

Definition at line 197 of file trfExe.py.

_trf

python.trfExe.transformExecutor._trf

private

Definition at line 232 of file trfExe.py.

_validation

python.trfExe.transformExecutor._validation

private

Definition at line 313 of file trfExe.py.

_valStart

python.trfExe.transformExecutor._valStart

private

Definition at line 188 of file trfExe.py.

_valStop

python.trfExe.transformExecutor._valStop

private

Definition at line 188 of file trfExe.py.

conf

python.trfExe.transformExecutor.conf

Executor configuration:

Note

that if conf and trf are None then we'll probably set the conf up later (this is allowed and expected to be done once the master transform has figured out what it's doing for this job)

Definition at line 163 of file trfExe.py.

inData

python.trfExe.transformExecutor.inData

Note

Protect against _inData not yet being defined

Use normal setter

Definition at line 251 of file trfExe.py.

outData

python.trfExe.transformExecutor.outData

Note

Protect against _outData not yet being defined

Use normal setter

Definition at line 271 of file trfExe.py.

---

The documentation for this class was generated from the following file:

• trfExe.py