python.trfExe.archiveExecutor Class Reference

Archive transform. More...

Inheritance diagram for python.trfExe.archiveExecutor:

Collaboration diagram for python.trfExe.archiveExecutor:

Public Member Functions
def	preExecute (self, input=set(), output=set())
def	exe (self)
def	exe (self, value)
def	exeArgs (self)
def	exeArgs (self, value)
def	execute (self)
def	postExecute (self)
def	validate (self)
def	inData (self)
def	inData (self, value)
def	outData (self)
def	outData (self, value)
def	myMerger (self)
	Now define properties for these data members. More...
def	name (self)
def	name (self, value)
def	substep (self)
def	trf (self)
def	trf (self, value)
def	inDataUpdate (self, value)
def	outDataUpdate (self, value)
def	input (self)
def	output (self)
def	extraMetadata (self)
def	hasExecuted (self)
def	rc (self)
def	errMsg (self)
def	validation (self)
def	validation (self, value)
def	hasValidated (self)
def	isValidated (self)
def	first (self)
def	preExeStartTimes (self)
def	exeStartTimes (self)
def	exeStopTimes (self)
def	valStartTimes (self)
def	valStopTimes (self)
def	preExeCpuTime (self)
def	preExeWallTime (self)
def	cpuTime (self)
def	usrTime (self)
def	sysTime (self)
def	wallTime (self)
def	memStats (self)
def	memAnalysis (self)
def	postExeCpuTime (self)
def	postExeWallTime (self)
def	validationCpuTime (self)
def	validationWallTime (self)
def	cpuTimeTotal (self)
def	wallTimeTotal (self)
def	eventCount (self)
def	reSimEvent (self)
def	athenaMP (self)
def	dbMonitor (self)
def	setPreExeStart (self)
def	setValStart (self)
def	doAll (self, input=set(), output=set())
	Convenience function. More...

Public Attributes
	conf
	Executor configuration: More...
	inData
	outData

Private Member Functions
def	_buildStandardCommand (self)

Private Attributes
	_memMonitor
	_exe
	_cmd
	_exeArgs
	_echoOutput
	_input
	_output
	_logFileName
	_echologger
	_exeLogFile
	_echostream
	_hasExecuted
	_exeStart
	_memSummaryFile
	_memFullFile
	_rc
	Check rc. More...
	_exeStop
	_memStats
	_valStart
	_hasValidated
	_isValidated
	_errMsg
	_eventCount
	Check event counts (always do this by default) Do this here so that all script executors have this by default (covers most use cases with events) More...
	_valStop
	_name
	_inData
	_outData
	_extraMetadata
	_preExeStart
	_memLeakResult
	_athenaMP
	_athenaMT
	_athenaConcurrentEvents
	_dbMonitor
	_resimevents
	_alreadyInContainer
	_containerSetup
	_myMerger
	_trf
	_validation

Detailed Description

Archive transform.

Definition at line 2114 of file trfExe.py.

Member Function Documentation

_buildStandardCommand()

def python.trfExe.scriptExecutor._buildStandardCommand (   self )

privateinherited

Definition at line 705 of file trfExe.py.

    def _buildStandardCommand(self):
        if self._exe:
            self._cmd = [self.exe, ]
        else:
            raise trfExceptions.TransformExecutionException(trfExit.nameToCode('TRF_EXEC_SETUP_FAIL'), 
                                                            'No executor set in {0}'.format(self.__class__.__name__))
        for arg in self.exeArgs:
            if arg in self.conf.argdict:
                # If we have a list then add each element to our list, else just str() the argument value
                # Note if there are arguments which need more complex transformations then
                # consider introducing a special toExeArg() method.
                if isinstance(self.conf.argdict[arg].value, list):
                    self._cmd.extend([ str(v) for v in self.conf.argdict[arg].value])
                else:
                    self._cmd.append(str(self.conf.argdict[arg].value))
 
 

athenaMP()

def python.trfExe.transformExecutor.athenaMP (   self )

inherited

Definition at line 451 of file trfExe.py.

    def athenaMP(self):
        return self._athenaMP
 

cpuTime()

def python.trfExe.transformExecutor.cpuTime (   self )

inherited

Definition at line 365 of file trfExe.py.

    def cpuTime(self):
        if self._exeStart and self._exeStop:
            return calcCpuTime(self._exeStart, self._exeStop)
        else:
            return None
 

cpuTimeTotal()

def python.trfExe.transformExecutor.cpuTimeTotal (   self )

inherited

Definition at line 429 of file trfExe.py.

    def cpuTimeTotal(self):
        if self._preExeStart and self._valStop:
            return calcCpuTime(self._preExeStart, self._valStop)
        else:
            return None
 

dbMonitor()

def python.trfExe.transformExecutor.dbMonitor (   self )

inherited

Definition at line 455 of file trfExe.py.

    def dbMonitor(self):
        return self._dbMonitor
 
 

doAll()

def python.trfExe.transformExecutor.doAll (   self,   input = set(),   output = set()  )

inherited

Convenience function.

Definition at line 498 of file trfExe.py.

    def doAll(self, input=set(), output=set()):
        self.preExecute(input, output)
        self.execute()
        self.postExecute()
        self.validate()
 

errMsg()

def python.trfExe.transformExecutor.errMsg (   self )

inherited

Definition at line 303 of file trfExe.py.

    def errMsg(self):
        return self._errMsg
    

eventCount()

def python.trfExe.transformExecutor.eventCount (   self )

inherited

Definition at line 443 of file trfExe.py.

    def eventCount(self):
        return self._eventCount
 

exe() [1/2]

def python.trfExe.scriptExecutor.exe (   self )

inherited

Definition at line 640 of file trfExe.py.

    def exe(self):
        return self._exe
    

exe() [2/2]

def python.trfExe.scriptExecutor.exe (   self,   value  )

inherited

Definition at line 644 of file trfExe.py.

    def exe(self, value):
        self._exe = value
        self._extraMetadata['script'] = value
 

exeArgs() [1/2]

def python.trfExe.scriptExecutor.exeArgs (   self )

inherited

Definition at line 649 of file trfExe.py.

    def exeArgs(self):
        return self._exeArgs
    

exeArgs() [2/2]

def python.trfExe.scriptExecutor.exeArgs (   self,   value  )

inherited

Definition at line 653 of file trfExe.py.

    def exeArgs(self, value):
        self._exeArgs = value
#        self._extraMetadata['scriptArgs'] = value
 

execute()

def python.trfExe.scriptExecutor.execute (   self )

inherited

Reimplemented from python.trfExe.transformExecutor.

Reimplemented in python.trfExe.bsMergeExecutor, and python.trfExe.POOLMergeExecutor.

Definition at line 722 of file trfExe.py.

    def execute(self):
        self._hasExecuted = True
        msg.info('Starting execution of {0} ({1})'.format(self._name, self._cmd))
        
        self._exeStart = os.times()
        msg.debug('exeStart time is {0}'.format(self._exeStart))
        if ('execOnly' in self.conf.argdict and self.conf.argdict['execOnly'] is True):
            msg.info('execOnly flag is set - execution will now switch, replacing the transform')
            os.execvp(self._cmd[0], self._cmd)
 
        encargs = {'encoding' : 'utf8'}
        try:
            # if we are already inside a container, then
            # must map /srv of the nested container to /srv of the parent container:
            # https://twiki.atlas-canada.ca/bin/view/AtlasCanada/Containers#Mount_point_summary
            if self._alreadyInContainer and self._containerSetup is not None:
                msg.info("chdir /srv to launch a nested container for the substep")
                os.chdir("/srv")
            p = subprocess.Popen(self._cmd, shell = False, stdout = subprocess.PIPE, stderr = subprocess.STDOUT, bufsize = 1, **encargs)
            # go back to the original working directory immediately after to store prmon in it
            if self._alreadyInContainer and self._containerSetup is not None:
                msg.info("chdir {} after launching the nested container".format(self._workdir))
                os.chdir(self._workdir)
 
            if self._memMonitor:
                try:
                    self._memSummaryFile = 'prmon.summary.' + self._name + '.json'
                    self._memFullFile = 'prmon.full.' + self._name
                    memMonitorCommand = ['prmon', '--pid', str(p.pid), '--filename', 'prmon.full.' + self._name, 
                                         '--json-summary', self._memSummaryFile, '--log-filename', 'prmon.' + self._name + '.log',
                                         '--interval', '30']
                    mem_proc = subprocess.Popen(memMonitorCommand, shell = False, close_fds=True, **encargs)
                    # TODO - link mem.full.current to mem.full.SUBSTEP
                except Exception as e:
                    msg.warning('Failed to spawn memory monitor for {0}: {1}'.format(self._name, e))
                    self._memMonitor = False
            
            while p.poll() is None:
                line = p.stdout.readline()
                if line:
                    self._echologger.info(line.rstrip())
            # Hoover up remaining buffered output lines
            for line in p.stdout:
                self._echologger.info(line.rstrip())
    
            self._rc = p.returncode
            msg.info('%s executor returns %d', self._name, self._rc)
            self._exeStop = os.times()
            msg.debug('exeStop time is {0}'.format(self._exeStop))
        except OSError as e:
            errMsg = 'Execution of {0} failed and raised OSError: {1}'.format(self._cmd[0], e)
            msg.error(errMsg)
            raise trfExceptions.TransformExecutionException(trfExit.nameToCode('TRF_EXEC'), errMsg)
        finally:
            if self._memMonitor:
                try:
                    mem_proc.send_signal(signal.SIGUSR1)
                    countWait = 0
                    while (not mem_proc.poll()) and countWait < 10:
                        time.sleep(0.1)
                        countWait += 1
                except OSError:
                    pass
        
        

exeStartTimes()

def python.trfExe.transformExecutor.exeStartTimes (   self )

inherited

Definition at line 335 of file trfExe.py.

    def exeStartTimes(self):
        return self._exeStart
 

exeStopTimes()

def python.trfExe.transformExecutor.exeStopTimes (   self )

inherited

Definition at line 339 of file trfExe.py.

    def exeStopTimes(self):
        return self._exeStop
 

extraMetadata()

def python.trfExe.transformExecutor.extraMetadata (   self )

inherited

Definition at line 291 of file trfExe.py.

    def extraMetadata(self):
        return self._extraMetadata
    

first()

def python.trfExe.transformExecutor.first (   self )

inherited

Note

At the moment only athenaExecutor sets this property, but that might be changed...

Definition at line 324 of file trfExe.py.

    def first(self):
        if hasattr(self, '_first'):
            return self._first
        else:
            return None
 

hasExecuted()

def python.trfExe.transformExecutor.hasExecuted (   self )

inherited

Definition at line 295 of file trfExe.py.

    def hasExecuted(self):
        return self._hasExecuted
    

hasValidated()

def python.trfExe.transformExecutor.hasValidated (   self )

inherited

Definition at line 315 of file trfExe.py.

    def hasValidated(self):
        return self._hasValidated
    

inData() [1/2]

def python.trfExe.transformExecutor.inData (   self )

inherited

Definition at line 234 of file trfExe.py.

    def inData(self):
        
        if '_inData' in dir(self):
            return self._inData
        return None
    

inData() [2/2]

def python.trfExe.transformExecutor.inData (   self,   value  )

inherited

Definition at line 241 of file trfExe.py.

    def inData(self, value):
        self._inData = set(value)
        

inDataUpdate()

def python.trfExe.transformExecutor.inDataUpdate (   self,   value  )

inherited

Definition at line 244 of file trfExe.py.

    def inDataUpdate(self, value):
        
        if '_inData' in dir(self):
            self._inData.update(value)
        else:
            
            self.inData = value
 
 

input()

def python.trfExe.transformExecutor.input (   self )

inherited

Note

This returns the actual input data with which this executor ran (c.f. inData which returns all the possible data types this executor could run with)

Definition at line 275 of file trfExe.py.

    def input(self):
        
        if '_input' in dir(self):
            return self._input
        return None
    

isValidated()

def python.trfExe.transformExecutor.isValidated (   self )

inherited

Definition at line 319 of file trfExe.py.

    def isValidated(self):
        return self._isValidated
    

memAnalysis()

def python.trfExe.transformExecutor.memAnalysis (   self )

inherited

Definition at line 397 of file trfExe.py.

    def memAnalysis(self):
        return self._memLeakResult
 

memStats()

def python.trfExe.transformExecutor.memStats (   self )

inherited

Definition at line 393 of file trfExe.py.

    def memStats(self):
        return self._memStats
 

myMerger()

def python.trfExe.transformExecutor.myMerger (   self )

inherited

Now define properties for these data members.

Definition at line 206 of file trfExe.py.

    def myMerger(self):
        return self._myMerger
 

name() [1/2]

def python.trfExe.transformExecutor.name (   self )

inherited

Definition at line 210 of file trfExe.py.

    def name(self):
        return self._name
 

name() [2/2]

def python.trfExe.transformExecutor.name (   self,   value  )

inherited

Definition at line 214 of file trfExe.py.

    def name(self, value):
        self._name = value
 

outData() [1/2]

def python.trfExe.transformExecutor.outData (   self )

inherited

Definition at line 254 of file trfExe.py.

    def outData(self):
        
        if '_outData' in dir(self):
            return self._outData
        return None
 

outData() [2/2]

def python.trfExe.transformExecutor.outData (   self,   value  )

inherited

Definition at line 261 of file trfExe.py.

    def outData(self, value):
        self._outData = set(value)
    

outDataUpdate()

def python.trfExe.transformExecutor.outDataUpdate (   self,   value  )

inherited

Definition at line 264 of file trfExe.py.

    def outDataUpdate(self, value):
        
        if '_outData' in dir(self):
            self._outData.update(value)
        else:
            
            self.outData = value
    

output()

def python.trfExe.transformExecutor.output (   self )

inherited

Note

This returns the actual output data with which this executor ran (c.f. outData which returns all the possible data types this executor could run with)

Definition at line 284 of file trfExe.py.

    def output(self):
        
        if '_output' in dir(self):
            return self._output
        return None
    

postExeCpuTime()

def python.trfExe.transformExecutor.postExeCpuTime (   self )

inherited

Definition at line 401 of file trfExe.py.

    def postExeCpuTime(self):
        if self._exeStop and self._valStart:
            return calcCpuTime(self._exeStop, self._valStart)
        else:
            return None
 

postExecute()

def python.trfExe.scriptExecutor.postExecute (   self )

inherited

Reimplemented from python.trfExe.transformExecutor.

Reimplemented in python.trfExe.bsMergeExecutor, and python.trfExe.athenaExecutor.

Definition at line 787 of file trfExe.py.

    def postExecute(self):
        if hasattr(self._exeLogFile, 'close'):
            self._exeLogFile.close()
        if self._memMonitor:
            try:
                memFile = open(self._memSummaryFile)
                self._memStats = json.load(memFile)
            except Exception as e:
                msg.warning('Failed to load JSON memory summmary file {0}: {1}'.format(self._memSummaryFile, e))
                self._memMonitor = False
                self._memStats = {}
 
 

postExeWallTime()

def python.trfExe.transformExecutor.postExeWallTime (   self )

inherited

Definition at line 408 of file trfExe.py.

    def postExeWallTime(self):
        if self._exeStop and self._valStart:
            return calcWallTime(self._exeStop, self._valStart)
        else:
            return None
 

preExeCpuTime()

def python.trfExe.transformExecutor.preExeCpuTime (   self )

inherited

Definition at line 351 of file trfExe.py.

    def preExeCpuTime(self):
        if self._preExeStart and self._exeStart:
            return calcCpuTime(self._preExeStart, self._exeStart)
        else:
            return None
 

preExecute()

def python.trfExe.archiveExecutor.preExecute	(		self,
			input = set(),
			output = set()
	)

Reimplemented from python.trfExe.scriptExecutor.

Definition at line 2116 of file trfExe.py.

    def preExecute(self, input = set(), output = set()):
        self.setPreExeStart()
        self._memMonitor = False
 
        #archiving
        if self._exe == 'zip':
            if 'outputArchFile' not in self.conf.argdict:
                raise trfExceptions.TransformExecutionException(trfExit.nameToCode('TRF_ARG_MISSING'), 'Missing output file name')
 
            self._cmd = ['python']
            try:
                with open('zip_wrapper.py', 'w') as zip_wrapper:
                    print("import zipfile, os, shutil", file=zip_wrapper)
                    if os.path.exists(self.conf.argdict['outputArchFile'].value[0]):
                        #appending input file(s) to existing archive
                        print("zf = zipfile.ZipFile('{}', mode='a', allowZip64=True)".format(self.conf.argdict['outputArchFile'].value[0]), file=zip_wrapper)
                    else:
                        #creating new archive
                        print("zf = zipfile.ZipFile('{}', mode='w', allowZip64=True)".format(self.conf.argdict['outputArchFile'].value[0]), file=zip_wrapper)
                    print("for f in {}:".format(self.conf.argdict['inputDataFile'].value), file=zip_wrapper)
                    #This module gives false positives (as of python 3.7.0). Will also check the name for ".zip"
                    #print >> zip_wrapper, "    if zipfile.is_zipfile(f):"
                    print("    if zipfile.is_zipfile(f) and '.zip' in f:", file=zip_wrapper)
                    print("        archive = zipfile.ZipFile(f, mode='r')", file=zip_wrapper)
                    print("        print 'Extracting input zip file {0} to temporary directory {1}'.format(f,'tmp')", file=zip_wrapper)
                    print("        archive.extractall('tmp')", file=zip_wrapper)
                    print("        archive.close()", file=zip_wrapper)
                    # remove stuff as soon as it is saved to output in order to save disk space at worker node
                    print("        if os.access(f, os.F_OK):", file=zip_wrapper)
                    print("            print 'Removing input zip file {}'.format(f)", file=zip_wrapper)
                    print("            os.unlink(f)", file=zip_wrapper)
                    print("        if os.path.isdir('tmp'):", file=zip_wrapper)
                    print("            for root, dirs, files in os.walk('tmp'):", file=zip_wrapper)
                    print("                for name in files:", file=zip_wrapper)
                    print("                    print 'Zipping {}'.format(name)", file=zip_wrapper)
                    print("                    zf.write(os.path.join(root, name), name, compress_type=zipfile.ZIP_STORED)", file=zip_wrapper)
                    print("            shutil.rmtree('tmp')", file=zip_wrapper)
                    print("    else:", file=zip_wrapper)
                    print("        print 'Zipping {}'.format(os.path.basename(f))", file=zip_wrapper)
                    print("        zf.write(f, arcname=os.path.basename(f), compress_type=zipfile.ZIP_STORED)", file=zip_wrapper)
                    print("        if os.access(f, os.F_OK):", file=zip_wrapper)
                    print("            print 'Removing input file {}'.format(f)", file=zip_wrapper)
                    print("            os.unlink(f)", file=zip_wrapper)
                    print("zf.close()", file=zip_wrapper)
                os.chmod('zip_wrapper.py', 0o755)
            except OSError as e:
                errMsg = 'error writing zip wrapper {fileName}: {error}'.format(fileName = 'zip_wrapper.py',
                    error = e
                )
                msg.error(errMsg)
                raise trfExceptions.TransformExecutionException(trfExit.nameToCode('TRF_EXEC_SETUP_WRAPPER'),
                    errMsg
                )
            self._cmd.append('zip_wrapper.py')
 
        #unarchiving
        elif self._exe == 'unarchive':
            import zipfile
            for infile in self.conf.argdict['inputArchFile'].value:
                 if not zipfile.is_zipfile(infile):
                     raise trfExceptions.TransformExecutionException(trfExit.nameToCode('TRF_INPUT_FILE_ERROR'),
                                                                     'An input file is not a zip archive - aborting unpacking')
            self._cmd = ['python']
            try:
                with open('unarchive_wrapper.py', 'w') as unarchive_wrapper:
                    print("import zipfile", file=unarchive_wrapper)
                    print("for f in {}:".format(self.conf.argdict['inputArchFile'].value), file=unarchive_wrapper)
                    print("     archive = zipfile.ZipFile(f, mode='r')", file=unarchive_wrapper)
                    print("     path = '{}'".format(self.conf.argdict['path']), file=unarchive_wrapper)
                    print("     print 'Extracting archive {0} to {1}'.format(f,path)", file=unarchive_wrapper)
                    print("     archive.extractall(path)", file=unarchive_wrapper)
                    print("     archive.close()", file=unarchive_wrapper)
                os.chmod('unarchive_wrapper.py', 0o755)
            except OSError as e:
                errMsg = 'error writing unarchive wrapper {fileName}: {error}'.format(fileName = 'unarchive_wrapper.py',
                    error = e
                )
                msg.error(errMsg)
                raise trfExceptions.TransformExecutionException(trfExit.nameToCode('TRF_EXEC_SETUP_WRAPPER'),
                    errMsg
                )
            self._cmd.append('unarchive_wrapper.py')
        super(archiveExecutor, self).preExecute(input=input, output=output)
 

preExeStartTimes()

def python.trfExe.transformExecutor.preExeStartTimes (   self )

inherited

Definition at line 331 of file trfExe.py.

    def preExeStartTimes(self):
        return self._preExeStart
    

preExeWallTime()

def python.trfExe.transformExecutor.preExeWallTime (   self )

inherited

Definition at line 358 of file trfExe.py.

    def preExeWallTime(self):
        if self._preExeStart and self._exeStart:
            return calcWallTime(self._preExeStart, self._exeStart)
        else:
            return None
 

rc()

def python.trfExe.transformExecutor.rc (   self )

inherited

Definition at line 299 of file trfExe.py.

    def rc(self):
        return self._rc
    

reSimEvent()

def python.trfExe.transformExecutor.reSimEvent (   self )

inherited

Definition at line 447 of file trfExe.py.

    def reSimEvent(self):
        return self._resimevents
 

setPreExeStart()

def python.trfExe.transformExecutor.setPreExeStart (   self )

inherited

Definition at line 460 of file trfExe.py.

    def setPreExeStart(self):
        if self._preExeStart is None:
            self._preExeStart = os.times()
            msg.debug('preExeStart time is {0}'.format(self._preExeStart))
 

setValStart()

def python.trfExe.transformExecutor.setValStart (   self )

inherited

Definition at line 465 of file trfExe.py.

    def setValStart(self):
        if self._valStart is None:
            self._valStart = os.times()
            msg.debug('valStart time is {0}'.format(self._valStart))
 

substep()

def python.trfExe.transformExecutor.substep (   self )

inherited

Reimplemented in python.trfExe.athenaExecutor.

Definition at line 218 of file trfExe.py.

    def substep(self):
        if '_substep' in dir(self):
            return self._substep
        return None
    

sysTime()

def python.trfExe.transformExecutor.sysTime (   self )

inherited

Definition at line 379 of file trfExe.py.

    def sysTime(self):
        if self._exeStart and self._exeStop:
            return self._exeStop[3] - self._exeStart[3]
        else:
            return None
 

trf() [1/2]

def python.trfExe.transformExecutor.trf (   self )

inherited

Definition at line 224 of file trfExe.py.

    def trf(self):
        if '_trf' in dir(self):
            return self._trf
        return None
    

trf() [2/2]

def python.trfExe.transformExecutor.trf (   self,   value  )

inherited

Definition at line 230 of file trfExe.py.

    def trf(self, value):
        self._trf = value
        

usrTime()

def python.trfExe.transformExecutor.usrTime (   self )

inherited

Definition at line 372 of file trfExe.py.

    def usrTime(self):
        if self._exeStart and self._exeStop:
            return self._exeStop[2] - self._exeStart[2]
        else:
            return None
        

validate()

def python.trfExe.scriptExecutor.validate (   self )

inherited

Reimplemented from python.trfExe.transformExecutor.

Reimplemented in python.trfExe.DQMergeExecutor, python.trfExe.optionalAthenaExecutor, and python.trfExe.athenaExecutor.

Definition at line 800 of file trfExe.py.

    def validate(self):
        if self._valStart is None:
            self._valStart = os.times()
            msg.debug('valStart time is {0}'.format(self._valStart))
        self._hasValidated = True
        
        
        if self._rc == 0:
            msg.info('Executor {0} validated successfully (return code {1})'.format(self._name, self._rc))
            self._isValidated = True
            self._errMsg = ''
        else:
            # Want to learn as much as possible from the non-zero code
            # this is a bit hard in general, although one can do signals.
            # Probably need to be more specific per exe, i.e., athena non-zero codes
            self._isValidated = False
            if self._rc < 0:
                # Map return codes to what the shell gives (128 + SIGNUM)
                self._rc = 128 - self._rc
            if trfExit.codeToSignalname(self._rc) != "":
                self._errMsg = '{0} got a {1} signal (exit code {2})'.format(self._name, trfExit.codeToSignalname(self._rc), self._rc)
            else:
                self._errMsg = 'Non-zero return code from %s (%d)' % (self._name, self._rc)
            raise trfExceptions.TransformValidationException(trfExit.nameToCode('TRF_EXEC_FAIL'), self._errMsg)
 
        
        if 'checkEventCount' in self.conf.argdict and self.conf.argdict['checkEventCount'].returnMyValue(exe=self) is False:
            msg.info('Event counting for substep {0} is skipped'.format(self.name))
        else:
            checkcount=trfValidation.eventMatch(self)
            checkcount.decide()
            self._eventCount = checkcount.eventCount
            msg.info('Event counting for substep {0} passed'.format(self.name))
 
        self._valStop = os.times()
        msg.debug('valStop time is {0}'.format(self._valStop))
 
 
 

validation() [1/2]

def python.trfExe.transformExecutor.validation (   self )

inherited

Definition at line 307 of file trfExe.py.

    def validation(self):
        return self._validation
    

validation() [2/2]

def python.trfExe.transformExecutor.validation (   self,   value  )

inherited

Definition at line 311 of file trfExe.py.

    def validation(self, value):
        self._validation = value
        

validationCpuTime()

def python.trfExe.transformExecutor.validationCpuTime (   self )

inherited

Definition at line 415 of file trfExe.py.

    def validationCpuTime(self):
        if self._valStart and self._valStop:
            return calcCpuTime(self._valStart, self._valStop)
        else:
            return None
    

validationWallTime()

def python.trfExe.transformExecutor.validationWallTime (   self )

inherited

Definition at line 422 of file trfExe.py.

    def validationWallTime(self):
        if self._valStart and self._valStop:
            return calcWallTime(self._valStart, self._valStop)
        else:
            return None
 

valStartTimes()

def python.trfExe.transformExecutor.valStartTimes (   self )

inherited

Definition at line 343 of file trfExe.py.

    def valStartTimes(self):
        return self._valStart
 

valStopTimes()

def python.trfExe.transformExecutor.valStopTimes (   self )

inherited

Definition at line 347 of file trfExe.py.

    def valStopTimes(self):
        return self._valStop
    

wallTime()

def python.trfExe.transformExecutor.wallTime (   self )

inherited

Definition at line 386 of file trfExe.py.

    def wallTime(self):
        if self._exeStart and self._exeStop:
            return calcWallTime(self._exeStart, self._exeStop)
        else:
            return None
        

wallTimeTotal()

def python.trfExe.transformExecutor.wallTimeTotal (   self )

inherited

Definition at line 436 of file trfExe.py.

    def wallTimeTotal(self):
        if self._preExeStart and self._valStop:
            return calcWallTime(self._preExeStart, self._valStop)
        else:
            return None
        

Member Data Documentation

_alreadyInContainer

python.trfExe.transformExecutor._alreadyInContainer

privateinherited

Definition at line 197 of file trfExe.py.

_athenaConcurrentEvents

python.trfExe.transformExecutor._athenaConcurrentEvents

privateinherited

Definition at line 194 of file trfExe.py.

_athenaMP

python.trfExe.transformExecutor._athenaMP

privateinherited

Definition at line 192 of file trfExe.py.

_athenaMT

python.trfExe.transformExecutor._athenaMT

privateinherited

Definition at line 193 of file trfExe.py.

_cmd

python.trfExe.archiveExecutor._cmd

private

Definition at line 2125 of file trfExe.py.

_containerSetup

python.trfExe.transformExecutor._containerSetup

privateinherited

Definition at line 198 of file trfExe.py.

_dbMonitor

python.trfExe.transformExecutor._dbMonitor

privateinherited

Definition at line 195 of file trfExe.py.

_echologger

python.trfExe.scriptExecutor._echologger

privateinherited

Definition at line 691 of file trfExe.py.

_echoOutput

python.trfExe.scriptExecutor._echoOutput

privateinherited

Note

Query the environment for echo configuration Let the manual envars always win over auto-detected settings

Definition at line 631 of file trfExe.py.

_echostream

python.trfExe.scriptExecutor._echostream

privateinherited

Definition at line 701 of file trfExe.py.

_errMsg

python.trfExe.scriptExecutor._errMsg

privateinherited

Definition at line 810 of file trfExe.py.

_eventCount

python.trfExe.scriptExecutor._eventCount

privateinherited

Check event counts (always do this by default) Do this here so that all script executors have this by default (covers most use cases with events)

Definition at line 832 of file trfExe.py.

_exe

python.trfExe.archiveExecutor._exe

private

Definition at line 2121 of file trfExe.py.

_exeArgs

python.trfExe.scriptExecutor._exeArgs

privateinherited

Definition at line 624 of file trfExe.py.

_exeLogFile

python.trfExe.scriptExecutor._exeLogFile

privateinherited

Definition at line 696 of file trfExe.py.

_exeStart

python.trfExe.scriptExecutor._exeStart

privateinherited

Definition at line 726 of file trfExe.py.

_exeStop

python.trfExe.scriptExecutor._exeStop

privateinherited

Definition at line 769 of file trfExe.py.

_extraMetadata

python.trfExe.transformExecutor._extraMetadata

privateinherited

Definition at line 180 of file trfExe.py.

_hasExecuted

python.trfExe.scriptExecutor._hasExecuted

privateinherited

Definition at line 723 of file trfExe.py.

_hasValidated

python.trfExe.scriptExecutor._hasValidated

privateinherited

Definition at line 804 of file trfExe.py.

_inData

python.trfExe.transformExecutor._inData

privateinherited

Definition at line 143 of file trfExe.py.

_input

python.trfExe.scriptExecutor._input

privateinherited

Definition at line 661 of file trfExe.py.

_isValidated

python.trfExe.scriptExecutor._isValidated

privateinherited

Definition at line 809 of file trfExe.py.

_logFileName

python.trfExe.scriptExecutor._logFileName

privateinherited

Note

If an inherited class has set self._cmd leave it alone

Definition at line 670 of file trfExe.py.

_memFullFile

python.trfExe.scriptExecutor._memFullFile

privateinherited

Definition at line 749 of file trfExe.py.

_memLeakResult

python.trfExe.transformExecutor._memLeakResult

privateinherited

Definition at line 189 of file trfExe.py.

_memMonitor

python.trfExe.archiveExecutor._memMonitor

private

Definition at line 2118 of file trfExe.py.

_memStats

python.trfExe.scriptExecutor._memStats

privateinherited

Definition at line 793 of file trfExe.py.

_memSummaryFile

python.trfExe.scriptExecutor._memSummaryFile

privateinherited

Definition at line 748 of file trfExe.py.

_myMerger

python.trfExe.transformExecutor._myMerger

privateinherited

Definition at line 201 of file trfExe.py.

_name

python.trfExe.transformExecutor._name

privateinherited

Definition at line 140 of file trfExe.py.

_outData

python.trfExe.transformExecutor._outData

privateinherited

Definition at line 144 of file trfExe.py.

_output

python.trfExe.scriptExecutor._output

privateinherited

Definition at line 662 of file trfExe.py.

_preExeStart

python.trfExe.transformExecutor._preExeStart

privateinherited

Note

Place holders for resource consumption. CPU and walltime are available for all executors but currently only athena is instrumented to fill in memory stats (and then only if PerfMonSD is enabled).

Definition at line 185 of file trfExe.py.

_rc

python.trfExe.scriptExecutor._rc

privateinherited

Check rc.

Definition at line 767 of file trfExe.py.

_resimevents

python.trfExe.transformExecutor._resimevents

privateinherited

Definition at line 196 of file trfExe.py.

_trf

python.trfExe.transformExecutor._trf

privateinherited

Definition at line 231 of file trfExe.py.

_validation

python.trfExe.transformExecutor._validation

privateinherited

Definition at line 312 of file trfExe.py.

_valStart

python.trfExe.scriptExecutor._valStart

privateinherited

Definition at line 802 of file trfExe.py.

_valStop

python.trfExe.scriptExecutor._valStop

privateinherited

Definition at line 835 of file trfExe.py.

conf

python.trfExe.transformExecutor.conf

inherited

Executor configuration:

Note

that if conf and trf are None then we'll probably set the conf up later (this is allowed and expected to be done once the master transform has figured out what it's doing for this job)

Definition at line 162 of file trfExe.py.

inData

python.trfExe.transformExecutor.inData

inherited

Note

Protect against _inData not yet being defined

Use normal setter

Definition at line 250 of file trfExe.py.

outData

python.trfExe.transformExecutor.outData

inherited

Note

Protect against _outData not yet being defined

Use normal setter

Definition at line 270 of file trfExe.py.

---

The documentation for this class was generated from the following file:

• trfExe.py