Specalise the script executor to deal with the BS merge oddity of excluding empty DRAWs. More...

Inheritance diagram for python.trfExe.bsMergeExecutor:

Collaboration diagram for python.trfExe.bsMergeExecutor:

Public Types
typedef HLT::TypeInformation::for_each_type_c< typenameEDMLIST::map, my_functor, my_result<>, my_arg< HLT::TypeInformation::get_cont, CONTAINER > >::type	result

Public Member Functions
	preExecute (self, input=set(), output=set())
	execute (self)
	postExecute (self)
	exeArgs (self)
	exeArgs (self, value)
	exe (self)
	exe (self, value)
	validate (self)
	inData (self)
	inData (self, value)
	outData (self)
	outData (self, value)
	myMerger (self)
	Now define properties for these data members.
	name (self)
	name (self, value)
	substep (self)
	trf (self)
	trf (self, value)
	inDataUpdate (self, value)
	outDataUpdate (self, value)
	input (self)
	output (self)
	extraMetadata (self)
	hasExecuted (self)
	rc (self)
	errMsg (self)
	validation (self)
	validation (self, value)
	hasValidated (self)
	isValidated (self)
	first (self)
	preExeStartTimes (self)
	exeStartTimes (self)
	exeStopTimes (self)
	valStartTimes (self)
	valStopTimes (self)
	preExeCpuTime (self)
	preExeWallTime (self)
	cpuTime (self)
	usrTime (self)
	sysTime (self)
	wallTime (self)
	memStats (self)
	memAnalysis (self)
	postExeCpuTime (self)
	postExeWallTime (self)
	validationCpuTime (self)
	validationWallTime (self)
	cpuTimeTotal (self)
	wallTimeTotal (self)
	eventCount (self)
	reSimEvent (self)
	athenaMP (self)
	dbMonitor (self)
	setPreExeStart (self)
	setValStart (self)
	doAll (self, input=set(), output=set())
	Convenience function.

Public Attributes
	exeArgs
	conf = conf
	Executor configuration:
	inData = value
	outData = value

Protected Member Functions
	_buildStandardCommand (self)

Protected Attributes
	_inputBS = list(input)[0]
	_outputBS = list(output)[0]
list	_maskedFiles = []
bool	_useStubFile = False
str	_mergeBSFileList = '{0}.list'.format(self._name)
str	_mergeBSLogfile = '{0}.out'.format(self._name)
	_outputFilename = self.conf.dataDictionary[self._outputBS].value[0]
bool	_doRename = False
str	_expectedOutput = self._outputFilename + '._0001.data'
	_exe = exe
	_exeArgs = exeArgs
bool	_echoOutput = False
list	_cmd = None
bool	_memMonitor = memMonitor
	_input = input
	_output = output
str	_logFileName = "log.{0}".format(self._name)
	_echologger = logging.getLogger(self._name)
	_exeLogFile = logging.FileHandler(self._logFileName, mode='w', **encargs)
	_echostream = logging.StreamHandler(_encoding_stream(sys.stdout))
	_workdir
str	_memSummaryFile = 'prmon.summary.' + self._name + '.json'
	_name = forceToAlphaNum(name)
	_inData = set(inData)
	_outData = set(outData)
bool	_hasExecuted = False
int	_rc = -1
str	_errMsg = None
bool	_hasValidated = False
bool	_isValidated = False
dict	_extraMetadata = {}
	_preExeStart = None
	_exeStart = None
	_valStart = None
dict	_memStats = {}
dict	_memLeakResult = {}
	_memFullFile = None
	_eventCount = None
int	_athenaMP = 0
int	_athenaMT = 0
int	_athenaConcurrentEvents = 0
	_dbMonitor = None
	_resimevents = None
	_alreadyInContainer = None
	_containerSetup = None
list	_myMerger = []
	_trf = value
	_validation = value
	_exeStop
	_valStop

Detailed Description

Specalise the script executor to deal with the BS merge oddity of excluding empty DRAWs.

Definition at line 2184 of file trfExe.py.

Member Typedef Documentation

◆ result

typedef HLT::TypeInformation::for_each_type_c<typenameEDMLIST::map,my_functor,my_result<>,my_arg<HLT::TypeInformation::get_cont,CONTAINER>>::type master_search< typenameEDMLIST::map, HLT::TypeInformation::get_cont, CONTAINER >::result

inherited

Definition at line 90 of file EDM_MasterSearch.h.

Member Function Documentation

◆ _buildStandardCommand()

python.trfExe.scriptExecutor._buildStandardCommand ( self )

protectedinherited

Definition at line 706 of file trfExe.py.

    def _buildStandardCommand(self):
        if self._exe:
            self._cmd = [self.exe, ]
        else:
            raise trfExceptions.TransformExecutionException(trfExit.nameToCode('TRF_EXEC_SETUP_FAIL'), 
                                                            'No executor set in {0}'.format(self.__class__.__name__))
        for arg in self.exeArgs:
            if arg in self.conf.argdict:
                # If we have a list then add each element to our list, else just str() the argument value
                # Note if there are arguments which need more complex transformations then
                # consider introducing a special toExeArg() method.
                if isinstance(self.conf.argdict[arg].value, list):
                    self._cmd.extend([ str(v) for v in self.conf.argdict[arg].value])
                else:
                    self._cmd.append(str(self.conf.argdict[arg].value))
 
 

◆ athenaMP()

python.trfExe.transformExecutor.athenaMP ( self )

inherited

Definition at line 452 of file trfExe.py.

    def athenaMP(self):
        return self._athenaMP
 

◆ cpuTime()

python.trfExe.transformExecutor.cpuTime ( self )

inherited

Definition at line 366 of file trfExe.py.

    def cpuTime(self):
        if self._exeStart and self._exeStop:
            return calcCpuTime(self._exeStart, self._exeStop)
        else:
            return None
 

◆ cpuTimeTotal()

python.trfExe.transformExecutor.cpuTimeTotal ( self )

inherited

Definition at line 430 of file trfExe.py.

    def cpuTimeTotal(self):
        if self._preExeStart and self._valStop:
            return calcCpuTime(self._preExeStart, self._valStop)
        else:
            return None
 

◆ dbMonitor()

python.trfExe.transformExecutor.dbMonitor ( self )

inherited

Definition at line 456 of file trfExe.py.

    def dbMonitor(self):
        return self._dbMonitor

◆ doAll()

python.trfExe.transformExecutor.doAll	(	self,
		input = set(),
		output = set() )

inherited

Convenience function.

Definition at line 499 of file trfExe.py.

    def doAll(self, input=set(), output=set()):
        self.preExecute(input, output)
        self.execute()
        self.postExecute()
        self.validate()
 

◆ errMsg()

python.trfExe.transformExecutor.errMsg ( self )

inherited

Definition at line 304 of file trfExe.py.

    def errMsg(self):
        return self._errMsg
    

◆ eventCount()

python.trfExe.transformExecutor.eventCount ( self )

inherited

Definition at line 444 of file trfExe.py.

    def eventCount(self):
        return self._eventCount
 

◆ exe() [1/2]

python.trfExe.scriptExecutor.exe ( self )

inherited

Definition at line 641 of file trfExe.py.

    def exe(self):
        return self._exe
    

◆ exe() [2/2]

python.trfExe.scriptExecutor.exe	(		self,
			value )

inherited

Definition at line 645 of file trfExe.py.

    def exe(self, value):
        self._exe = value
        self._extraMetadata['script'] = value
 

◆ exeArgs() [1/2]

python.trfExe.scriptExecutor.exeArgs ( self )

inherited

Definition at line 650 of file trfExe.py.

    def exeArgs(self):
        return self._exeArgs
    

◆ exeArgs() [2/2]

python.trfExe.scriptExecutor.exeArgs	(		self,
			value )

inherited

Definition at line 654 of file trfExe.py.

    def exeArgs(self, value):
        self._exeArgs = value
#        self._extraMetadata['scriptArgs'] = value
 

◆ execute()

python.trfExe.bsMergeExecutor.execute ( self )

Reimplemented from python.trfExe.scriptExecutor.

Definition at line 2242 of file trfExe.py.

    def execute(self):
        if self._useStubFile:
            # Need to fake execution!
            self._exeStart = os.times()
            msg.debug('exeStart time is {0}'.format(self._exeStart))
            msg.info("Using stub file for empty BS output - execution is fake")
            if self._outputFilename != self.conf.argdict['emptyStubFile'].value:
                os.rename(self.conf.argdict['emptyStubFile'].value, self._outputFilename)            
            self._memMonitor = False
            self._hasExecuted = True
            self._rc = 0
            self._exeStop = os.times()
            msg.debug('exeStop time is {0}'.format(self._exeStop))
        else:
            super(bsMergeExecutor, self).execute()
        

◆ exeStartTimes()

python.trfExe.transformExecutor.exeStartTimes ( self )

inherited

Definition at line 336 of file trfExe.py.

    def exeStartTimes(self):
        return self._exeStart
 

◆ exeStopTimes()

python.trfExe.transformExecutor.exeStopTimes ( self )

inherited

Definition at line 340 of file trfExe.py.

    def exeStopTimes(self):
        return self._exeStop
 

◆ extraMetadata()

python.trfExe.transformExecutor.extraMetadata ( self )

inherited

Definition at line 292 of file trfExe.py.

    def extraMetadata(self):
        return self._extraMetadata
    

◆ first()

python.trfExe.transformExecutor.first ( self )

inherited

Note: At the moment only athenaExecutor sets this property, but that might be changed...

Definition at line 325 of file trfExe.py.

    def first(self):
        if hasattr(self, '_first'):
            return self._first
        else:
            return None
 

◆ hasExecuted()

python.trfExe.transformExecutor.hasExecuted ( self )

inherited

Definition at line 296 of file trfExe.py.

    def hasExecuted(self):
        return self._hasExecuted
    

◆ hasValidated()

python.trfExe.transformExecutor.hasValidated ( self )

inherited

Definition at line 316 of file trfExe.py.

    def hasValidated(self):
        return self._hasValidated
    

◆ inData() [1/2]

python.trfExe.transformExecutor.inData ( self )

inherited

Definition at line 235 of file trfExe.py.

    def inData(self):
        
        if '_inData' in dir(self):
            return self._inData
        return None
    

◆ inData() [2/2]

python.trfExe.transformExecutor.inData	(		self,
			value )

inherited

Definition at line 242 of file trfExe.py.

    def inData(self, value):
        self._inData = set(value)
        

◆ inDataUpdate()

python.trfExe.transformExecutor.inDataUpdate	(		self,
			value )

inherited

Definition at line 245 of file trfExe.py.

    def inDataUpdate(self, value):
        
        if '_inData' in dir(self):
            self._inData.update(value)
        else:
            
            self.inData = value

◆ input()

python.trfExe.transformExecutor.input ( self )

inherited

Note: This returns the actual input data with which this executor ran (c.f. inData which returns all the possible data types this executor could run with)

Definition at line 276 of file trfExe.py.

    def input(self):
        
        if '_input' in dir(self):
            return self._input
        return None
    

◆ isValidated()

python.trfExe.transformExecutor.isValidated ( self )

inherited

Definition at line 320 of file trfExe.py.

    def isValidated(self):
        return self._isValidated
    

◆ memAnalysis()

python.trfExe.transformExecutor.memAnalysis ( self )

inherited

Definition at line 398 of file trfExe.py.

    def memAnalysis(self):
        return self._memLeakResult
 

◆ memStats()

python.trfExe.transformExecutor.memStats ( self )

inherited

Definition at line 394 of file trfExe.py.

    def memStats(self):
        return self._memStats
 

◆ myMerger()

python.trfExe.transformExecutor.myMerger ( self )

inherited

Now define properties for these data members.

Definition at line 207 of file trfExe.py.

    def myMerger(self):
        return self._myMerger
 

◆ name() [1/2]

python.trfExe.transformExecutor.name ( self )

inherited

Definition at line 211 of file trfExe.py.

    def name(self):
        return self._name
 

◆ name() [2/2]

python.trfExe.transformExecutor.name	(		self,
			value )

inherited

Definition at line 215 of file trfExe.py.

    def name(self, value):
        self._name = value
 

◆ outData() [1/2]

python.trfExe.transformExecutor.outData ( self )

inherited

Definition at line 255 of file trfExe.py.

    def outData(self):
        
        if '_outData' in dir(self):
            return self._outData
        return None
 

◆ outData() [2/2]

python.trfExe.transformExecutor.outData	(		self,
			value )

inherited

Definition at line 262 of file trfExe.py.

    def outData(self, value):
        self._outData = set(value)
    

◆ outDataUpdate()

python.trfExe.transformExecutor.outDataUpdate	(		self,
			value )

inherited

Definition at line 265 of file trfExe.py.

    def outDataUpdate(self, value):
        
        if '_outData' in dir(self):
            self._outData.update(value)
        else:
            
            self.outData = value
    

◆ output()

python.trfExe.transformExecutor.output ( self )

inherited

Note: This returns the actual output data with which this executor ran (c.f. outData which returns all the possible data types this executor could run with)

Definition at line 285 of file trfExe.py.

    def output(self):
        
        if '_output' in dir(self):
            return self._output
        return None
    

◆ postExeCpuTime()

python.trfExe.transformExecutor.postExeCpuTime ( self )

inherited

Definition at line 402 of file trfExe.py.

    def postExeCpuTime(self):
        if self._exeStop and self._valStart:
            return calcCpuTime(self._exeStop, self._valStart)
        else:
            return None
 

◆ postExecute()

python.trfExe.bsMergeExecutor.postExecute ( self )

Reimplemented from python.trfExe.scriptExecutor.

Definition at line 2258 of file trfExe.py.

    def postExecute(self):
        if self._useStubFile:
            pass
        elif self._doRename:
            self._expectedOutput = self._outputFilename + '._0001.data'
            msg.info('Renaming {0} to {1}'.format(self._expectedOutput, self.conf.dataDictionary[self._outputBS].value[0]))
            try:
                os.rename(self._outputFilename + '._0001.data', self.conf.dataDictionary[self._outputBS].value[0])
            except OSError as e:
                raise trfExceptions.TransformExecutionException(trfExit.nameToCode('TRF_OUTPUT_FILE_ERROR'), 
                                                                'Exception raised when renaming {0} to {1}: {2}'.format(self._outputFilename, self.conf.dataDictionary[self._outputBS].value[0], e))
        super(bsMergeExecutor, self).postExecute()
 
 

◆ postExeWallTime()

python.trfExe.transformExecutor.postExeWallTime ( self )

inherited

Definition at line 409 of file trfExe.py.

    def postExeWallTime(self):
        if self._exeStop and self._valStart:
            return calcWallTime(self._exeStop, self._valStart)
        else:
            return None
 

◆ preExeCpuTime()

python.trfExe.transformExecutor.preExeCpuTime ( self )

inherited

Definition at line 352 of file trfExe.py.

    def preExeCpuTime(self):
        if self._preExeStart and self._exeStart:
            return calcCpuTime(self._preExeStart, self._exeStart)
        else:
            return None
 

◆ preExecute()

python.trfExe.bsMergeExecutor.preExecute	(	self,
		input = set(),
		output = set() )

Reimplemented from python.trfExe.scriptExecutor.

Definition at line 2186 of file trfExe.py.

    def preExecute(self, input = set(), output = set()):
        self.setPreExeStart()
        self._inputBS = list(input)[0]
        self._outputBS = list(output)[0]
        self._maskedFiles = []
        self._useStubFile = False
        if 'maskEmptyInputs' in self.conf.argdict and self.conf.argdict['maskEmptyInputs'].value is True:
            eventfullFiles = []
            for fname in self.conf.dataDictionary[self._inputBS].value:
                nEvents = self.conf.dataDictionary[self._inputBS].getSingleMetadata(fname, 'nentries')
                msg.debug('Found {0} events in file {1}'.format(nEvents, fname))
                if isinstance(nEvents, int) and nEvents > 0:
                    eventfullFiles.append(fname)
            self._maskedFiles = list(set(self.conf.dataDictionary[self._inputBS].value) - set(eventfullFiles))
            if len(self._maskedFiles) > 0:
                msg.info('The following input files are masked because they have 0 events: {0}'.format(' '.join(self._maskedFiles)))
                if len(eventfullFiles) == 0:
                    if 'emptyStubFile' in self.conf.argdict and path.exists(self.conf.argdict['emptyStubFile'].value):
                        self._useStubFile = True
                        msg.info("All input files are empty - will use stub file {0} as output".format(self.conf.argdict['emptyStubFile'].value))
                    else:
                        raise trfExceptions.TransformExecutionException(trfExit.nameToCode('TRF_INPUT_FILE_ERROR'), 
                                                                        'All input files had zero events - aborting BS merge')            
        
        # Write the list of input files to a text file, so that testMergedFiles can swallow it
        self._mergeBSFileList = '{0}.list'.format(self._name)
        self._mergeBSLogfile = '{0}.out'.format(self._name)
        try:
            with open(self._mergeBSFileList, 'w') as BSFileList:
                for fname in self.conf.dataDictionary[self._inputBS].value:
                    if fname not in self._maskedFiles:
                        print(fname, file=BSFileList)
        except OSError as e:
            errMsg = 'Got an error when writing list of BS files to {0}: {1}'.format(self._mergeBSFileList, e)
            msg.error(errMsg)
            raise trfExceptions.TransformExecutionException(trfExit.nameToCode('TRF_EXEC_SETUP_WRAPPER'), errMsg)
        
        # Hope that we were given a correct filename...
        self._outputFilename = self.conf.dataDictionary[self._outputBS].value[0]
        if self._outputFilename.endswith('._0001.data'):
            self._doRename = False
            self._outputFilename = self._outputFilename.split('._0001.data')[0]    
        elif self.conf.argdict['allowRename'].value is True:
            # OK, non-fatal, we go for a renaming
            msg.info('Output filename does not end in "._0001.data" will proceed, but be aware that the internal filename metadata will be wrong')
            self._doRename = True
        else:
            # No rename allowed, so we are dead...
            errmsg = 'Output filename for outputBS_MRGFile must end in "._0001.data" or infile metadata will be wrong'
            raise trfExceptions.TransformExecutionException(trfExit.nameToCode('TRF_OUTPUT_FILE_ERROR'), errmsg)
        
        # Set the correct command for execution
        self._cmd = [self._exe, self._mergeBSFileList, '0', self._outputFilename] 
        
        super(bsMergeExecutor, self).preExecute(input=input, output=output)
    

◆ preExeStartTimes()

python.trfExe.transformExecutor.preExeStartTimes ( self )

inherited

Definition at line 332 of file trfExe.py.

    def preExeStartTimes(self):
        return self._preExeStart
    

◆ preExeWallTime()

python.trfExe.transformExecutor.preExeWallTime ( self )

inherited

Definition at line 359 of file trfExe.py.

    def preExeWallTime(self):
        if self._preExeStart and self._exeStart:
            return calcWallTime(self._preExeStart, self._exeStart)
        else:
            return None
 

◆ rc()

python.trfExe.transformExecutor.rc ( self )

inherited

Definition at line 300 of file trfExe.py.

    def rc(self):
        return self._rc
    

◆ reSimEvent()

python.trfExe.transformExecutor.reSimEvent ( self )

inherited

Definition at line 448 of file trfExe.py.

    def reSimEvent(self):
        return self._resimevents
 

◆ setPreExeStart()

python.trfExe.transformExecutor.setPreExeStart ( self )

inherited

Definition at line 461 of file trfExe.py.

    def setPreExeStart(self):
        if self._preExeStart is None:
            self._preExeStart = os.times()
            msg.debug('preExeStart time is {0}'.format(self._preExeStart))
 

◆ setValStart()

python.trfExe.transformExecutor.setValStart ( self )

inherited

Definition at line 466 of file trfExe.py.

    def setValStart(self):
        if self._valStart is None:
            self._valStart = os.times()
            msg.debug('valStart time is {0}'.format(self._valStart))
 

◆ substep()

python.trfExe.transformExecutor.substep ( self )

inherited

Reimplemented in python.trfExe.athenaExecutor.

Definition at line 219 of file trfExe.py.

    def substep(self):
        if '_substep' in dir(self):
            return self._substep
        return None
    

◆ sysTime()

python.trfExe.transformExecutor.sysTime ( self )

inherited

Definition at line 380 of file trfExe.py.

    def sysTime(self):
        if self._exeStart and self._exeStop:
            return self._exeStop[3] - self._exeStart[3]
        else:
            return None
 

◆ trf() [1/2]

python.trfExe.transformExecutor.trf ( self )

inherited

Definition at line 225 of file trfExe.py.

    def trf(self):
        if '_trf' in dir(self):
            return self._trf
        return None
    

◆ trf() [2/2]

python.trfExe.transformExecutor.trf	(		self,
			value )

inherited

Definition at line 231 of file trfExe.py.

    def trf(self, value):
        self._trf = value
        

◆ usrTime()

python.trfExe.transformExecutor.usrTime ( self )

inherited

Definition at line 373 of file trfExe.py.

    def usrTime(self):
        if self._exeStart and self._exeStop:
            return self._exeStop[2] - self._exeStart[2]
        else:
            return None
        

◆ validate()

python.trfExe.scriptExecutor.validate ( self )

inherited

Reimplemented from python.trfExe.transformExecutor.

Reimplemented in python.trfExe.athenaExecutor, python.trfExe.DQMergeExecutor, python.trfExe.DQMPostProcessExecutor, and python.trfExe.optionalAthenaExecutor.

Definition at line 801 of file trfExe.py.

    def validate(self):
        if self._valStart is None:
            self._valStart = os.times()
            msg.debug('valStart time is {0}'.format(self._valStart))
        self._hasValidated = True
        
        
        if self._rc == 0:
            msg.info('Executor {0} validated successfully (return code {1})'.format(self._name, self._rc))
            self._isValidated = True
            self._errMsg = ''
        else:
            # Want to learn as much as possible from the non-zero code
            # this is a bit hard in general, although one can do signals.
            # Probably need to be more specific per exe, i.e., athena non-zero codes
            self._isValidated = False
            if self._rc < 0:
                # Map return codes to what the shell gives (128 + SIGNUM)
                self._rc = 128 - self._rc
            if trfExit.codeToSignalname(self._rc) != "":
                self._errMsg = '{0} got a {1} signal (exit code {2})'.format(self._name, trfExit.codeToSignalname(self._rc), self._rc)
            else:
                self._errMsg = 'Non-zero return code from %s (%d)' % (self._name, self._rc)
            raise trfExceptions.TransformValidationException(trfExit.nameToCode('TRF_EXEC_FAIL'), self._errMsg)
 
        
        if 'checkEventCount' in self.conf.argdict and self.conf.argdict['checkEventCount'].returnMyValue(exe=self) is False:
            msg.info('Event counting for substep {0} is skipped'.format(self.name))
        else:
            if 'mpi' in self.conf.argdict and not mpi.mpiShouldValidate():
                msg.info('MPI mode -- skipping output event count check')
            else:
                checkcount=trfValidation.eventMatch(self)
                checkcount.decide()
                self._eventCount = checkcount.eventCount
                msg.info('Event counting for substep {0} passed'.format(self.name))
 
        self._valStop = os.times()
        msg.debug('valStop time is {0}'.format(self._valStop))
 
 
 

◆ validation() [1/2]

python.trfExe.transformExecutor.validation ( self )

inherited

Definition at line 308 of file trfExe.py.

    def validation(self):
        return self._validation
    

◆ validation() [2/2]

python.trfExe.transformExecutor.validation	(		self,
			value )

inherited

Definition at line 312 of file trfExe.py.

    def validation(self, value):
        self._validation = value
        

◆ validationCpuTime()

python.trfExe.transformExecutor.validationCpuTime ( self )

inherited

Definition at line 416 of file trfExe.py.

    def validationCpuTime(self):
        if self._valStart and self._valStop:
            return calcCpuTime(self._valStart, self._valStop)
        else:
            return None
    

◆ validationWallTime()

python.trfExe.transformExecutor.validationWallTime ( self )

inherited

Definition at line 423 of file trfExe.py.

    def validationWallTime(self):
        if self._valStart and self._valStop:
            return calcWallTime(self._valStart, self._valStop)
        else:
            return None
 

◆ valStartTimes()

python.trfExe.transformExecutor.valStartTimes ( self )

inherited

Definition at line 344 of file trfExe.py.

    def valStartTimes(self):
        return self._valStart
 

◆ valStopTimes()

python.trfExe.transformExecutor.valStopTimes ( self )

inherited

Definition at line 348 of file trfExe.py.

    def valStopTimes(self):
        return self._valStop
    

◆ wallTime()

python.trfExe.transformExecutor.wallTime ( self )

inherited

Definition at line 387 of file trfExe.py.

    def wallTime(self):
        if self._exeStart and self._exeStop:
            return calcWallTime(self._exeStart, self._exeStop)
        else:
            return None
        

◆ wallTimeTotal()

python.trfExe.transformExecutor.wallTimeTotal ( self )

inherited

Definition at line 437 of file trfExe.py.

    def wallTimeTotal(self):
        if self._preExeStart and self._valStop:
            return calcWallTime(self._preExeStart, self._valStop)
        else:
            return None
        

Member Data Documentation

◆ _alreadyInContainer

python.trfExe.transformExecutor._alreadyInContainer = None

protectedinherited

Definition at line 198 of file trfExe.py.

◆ _athenaConcurrentEvents

int python.trfExe.transformExecutor._athenaConcurrentEvents = 0

protectedinherited

Definition at line 195 of file trfExe.py.

◆ _athenaMP

int python.trfExe.transformExecutor._athenaMP = 0

protectedinherited

Definition at line 193 of file trfExe.py.

◆ _athenaMT

int python.trfExe.transformExecutor._athenaMT = 0

protectedinherited

Definition at line 194 of file trfExe.py.

◆ _cmd

python.trfExe.scriptExecutor._cmd = None

protectedinherited

Note: If an inherited class has set self._cmd leave it alone

Definition at line 636 of file trfExe.py.

◆ _containerSetup

python.trfExe.transformExecutor._containerSetup = None

protectedinherited

Definition at line 199 of file trfExe.py.

◆ _dbMonitor

python.trfExe.transformExecutor._dbMonitor = None

protectedinherited

Definition at line 196 of file trfExe.py.

◆ _doRename

bool python.trfExe.bsMergeExecutor._doRename = False

protected

Definition at line 2226 of file trfExe.py.

◆ _echologger

python.trfExe.scriptExecutor._echologger = logging.getLogger(self._name)

protectedinherited

Note: Query the environment for echo configuration Let the manual envars always win over auto-detected settings

Definition at line 692 of file trfExe.py.

◆ _echoOutput

bool python.trfExe.scriptExecutor._echoOutput = False

protectedinherited

Note: Query the environment for echo configuration Let the manual envars always win over auto-detected settings

Definition at line 633 of file trfExe.py.

◆ _echostream

python.trfExe.scriptExecutor._echostream = logging.StreamHandler(_encoding_stream(sys.stdout))

protectedinherited

Definition at line 702 of file trfExe.py.

◆ _errMsg

str python.trfExe.transformExecutor._errMsg = None

protectedinherited

Definition at line 172 of file trfExe.py.

◆ _eventCount

python.trfExe.transformExecutor._eventCount = None

protectedinherited

Definition at line 192 of file trfExe.py.

◆ _exe

python.trfExe.scriptExecutor._exe = exe

protectedinherited

Definition at line 623 of file trfExe.py.

◆ _exeArgs

python.trfExe.scriptExecutor._exeArgs = exeArgs

protectedinherited

Definition at line 626 of file trfExe.py.

◆ _exeLogFile

python.trfExe.scriptExecutor._exeLogFile = logging.FileHandler(self._logFileName, mode='w', **encargs)

protectedinherited

Definition at line 697 of file trfExe.py.

◆ _exeStart

python.trfExe.transformExecutor._exeStart = None

protectedinherited

Definition at line 187 of file trfExe.py.

◆ _exeStop

python.trfExe.transformExecutor._exeStop

protectedinherited

Definition at line 367 of file trfExe.py.

◆ _expectedOutput

python.trfExe.bsMergeExecutor._expectedOutput = self._outputFilename + '._0001.data'

protected

Definition at line 2262 of file trfExe.py.

◆ _extraMetadata

dict python.trfExe.transformExecutor._extraMetadata = {}

protectedinherited

Definition at line 181 of file trfExe.py.

◆ _hasExecuted

bool python.trfExe.transformExecutor._hasExecuted = False

protectedinherited

Definition at line 170 of file trfExe.py.

◆ _hasValidated

bool python.trfExe.transformExecutor._hasValidated = False

protectedinherited

Definition at line 175 of file trfExe.py.

◆ _inData

python.trfExe.transformExecutor._inData = set(inData)

protectedinherited

Definition at line 144 of file trfExe.py.

◆ _input

python.trfExe.scriptExecutor._input = input

protectedinherited

Definition at line 662 of file trfExe.py.

◆ _inputBS

python.trfExe.bsMergeExecutor._inputBS = list(input)[0]

protected

Definition at line 2188 of file trfExe.py.

◆ _isValidated

bool python.trfExe.transformExecutor._isValidated = False

protectedinherited

Definition at line 176 of file trfExe.py.

◆ _logFileName

python.trfExe.scriptExecutor._logFileName = "log.{0}".format(self._name)

protectedinherited

Note: If an inherited class has set self._cmd leave it alone; Query the environment for echo configuration Let the manual envars always win over auto-detected settings

Definition at line 671 of file trfExe.py.

◆ _maskedFiles

python.trfExe.bsMergeExecutor._maskedFiles = []

protected

Definition at line 2190 of file trfExe.py.

◆ _memFullFile

python.trfExe.transformExecutor._memFullFile = None

protectedinherited

Definition at line 191 of file trfExe.py.

◆ _memLeakResult

dict python.trfExe.transformExecutor._memLeakResult = {}

protectedinherited

Definition at line 190 of file trfExe.py.

◆ _memMonitor

bool python.trfExe.scriptExecutor._memMonitor = memMonitor

protectedinherited

Definition at line 638 of file trfExe.py.

◆ _memStats

dict python.trfExe.transformExecutor._memStats = {}

protectedinherited

Definition at line 189 of file trfExe.py.

◆ _memSummaryFile

python.trfExe.scriptExecutor._memSummaryFile = 'prmon.summary.' + self._name + '.json'

protectedinherited

Definition at line 749 of file trfExe.py.

◆ _mergeBSFileList

python.trfExe.bsMergeExecutor._mergeBSFileList = '{0}.list'.format(self._name)

protected

Definition at line 2211 of file trfExe.py.

◆ _mergeBSLogfile

str python.trfExe.bsMergeExecutor._mergeBSLogfile = '{0}.out'.format(self._name)

protected

Definition at line 2212 of file trfExe.py.

◆ _myMerger

list python.trfExe.transformExecutor._myMerger = []

protectedinherited

Definition at line 202 of file trfExe.py.

◆ _name

python.trfExe.transformExecutor._name = forceToAlphaNum(name)

protectedinherited

Definition at line 141 of file trfExe.py.

◆ _outData

python.trfExe.transformExecutor._outData = set(outData)

protectedinherited

Definition at line 145 of file trfExe.py.

◆ _output

python.trfExe.scriptExecutor._output = output

protectedinherited

Definition at line 663 of file trfExe.py.

◆ _outputBS

python.trfExe.bsMergeExecutor._outputBS = list(output)[0]

protected

Definition at line 2189 of file trfExe.py.

◆ _outputFilename

python.trfExe.bsMergeExecutor._outputFilename = self.conf.dataDictionary[self._outputBS].value[0]

protected

Definition at line 2224 of file trfExe.py.

◆ _preExeStart

python.trfExe.transformExecutor._preExeStart = None

protectedinherited

Note: Place holders for resource consumption. CPU and walltime are available for all executors but currently only athena is instrumented to fill in memory stats (and then only if PerfMonSD is enabled).

Definition at line 186 of file trfExe.py.

◆ _rc

python.trfExe.transformExecutor._rc = -1

protectedinherited

Definition at line 171 of file trfExe.py.

◆ _resimevents

python.trfExe.transformExecutor._resimevents = None

protectedinherited

Definition at line 197 of file trfExe.py.

◆ _trf

python.trfExe.transformExecutor._trf = value

protectedinherited

Definition at line 232 of file trfExe.py.

◆ _useStubFile

bool python.trfExe.bsMergeExecutor._useStubFile = False

protected

Definition at line 2191 of file trfExe.py.

◆ _validation

python.trfExe.transformExecutor._validation = value

protectedinherited

Definition at line 313 of file trfExe.py.

◆ _valStart

python.trfExe.transformExecutor._valStart = None

protectedinherited

Definition at line 188 of file trfExe.py.

◆ _valStop

python.trfExe.transformExecutor._valStop

protectedinherited

Definition at line 417 of file trfExe.py.

◆ _workdir

python.trfExe.scriptExecutor._workdir

protectedinherited

Definition at line 744 of file trfExe.py.

◆ conf

python.trfExe.transformExecutor.conf = conf

inherited

Executor configuration:

Note: that if conf and trf are None then we'll probably set the conf up later (this is allowed and expected to be done once the master transform has figured out what it's doing for this job)

Definition at line 163 of file trfExe.py.

◆ exeArgs

python.trfExe.scriptExecutor.exeArgs

inherited

Definition at line 712 of file trfExe.py.

◆ inData

python.trfExe.transformExecutor.inData = value

inherited

Note: Protect against _inData not yet being defined; Use normal setter

Definition at line 251 of file trfExe.py.

◆ outData

python.trfExe.transformExecutor.outData = value

inherited

Note: Protect against _outData not yet being defined; Use normal setter

Definition at line 271 of file trfExe.py.

The documentation for this class was generated from the following file:

trfExe.py

Public Types

Public Member Functions

Public Attributes

Protected Member Functions

Protected Attributes

Detailed Description

Member Typedef Documentation

◆ result

Member Function Documentation

◆ _buildStandardCommand()

◆ athenaMP()

◆ cpuTime()

◆ cpuTimeTotal()

◆ dbMonitor()

◆ doAll()

◆ errMsg()

◆ eventCount()

◆ exe() [1/2]

◆ exe() [2/2]

◆ exeArgs() [1/2]

◆ exeArgs() [2/2]

◆ execute()

◆ exeStartTimes()

◆ exeStopTimes()

◆ extraMetadata()

◆ first()

◆ hasExecuted()

◆ hasValidated()

◆ inData() [1/2]

◆ inData() [2/2]

◆ inDataUpdate()

◆ input()

◆ isValidated()

◆ memAnalysis()

◆ memStats()

◆ myMerger()

◆ name() [1/2]

◆ name() [2/2]

◆ outData() [1/2]

◆ outData() [2/2]

◆ outDataUpdate()

◆ output()

◆ postExeCpuTime()

◆ postExecute()

◆ postExeWallTime()

◆ preExeCpuTime()

◆ preExecute()

◆ preExeStartTimes()

◆ preExeWallTime()

◆ rc()

◆ reSimEvent()

◆ setPreExeStart()

◆ setValStart()

◆ substep()

◆ sysTime()

◆ trf() [1/2]

◆ trf() [2/2]

◆ usrTime()

◆ validate()

◆ validation() [1/2]

◆ validation() [2/2]

◆ validationCpuTime()

◆ validationWallTime()

◆ valStartTimes()

◆ valStopTimes()

◆ wallTime()

◆ wallTimeTotal()

Member Data Documentation

◆ _alreadyInContainer

◆ _athenaConcurrentEvents

◆ _athenaMP

◆ _athenaMT

◆ _cmd

◆ _containerSetup

◆ _dbMonitor

◆ _doRename

◆ _echologger

◆ _echoOutput

◆ _echostream

◆ _errMsg