python.trfExe.optionalAthenaExecutor Class Reference

Athena executor where failure is not consisered fatal. More...

Inheritance diagram for python.trfExe.optionalAthenaExecutor:

Collaboration diagram for python.trfExe.optionalAthenaExecutor:

Public Member Functions
def	validate (self)
def	name (self)
def	name (self, value)
def	inputDataTypeCountCheck (self)
def	inputDataTypeCountCheck (self, value)
def	substep (self)
def	disableMP (self)
def	disableMP (self, value)
def	disableMT (self)
def	disableMT (self, value)
def	onlyMP (self)
def	onlyMP (self, value)
def	onlyMT (self)
def	onlyMT (self, value)
def	preExecute (self, input=set(), output=set())
def	postExecute (self)
def	exe (self)
def	exe (self, value)
def	exeArgs (self)
def	exeArgs (self, value)
def	execute (self)
def	inData (self)
def	inData (self, value)
def	outData (self)
def	outData (self, value)
def	myMerger (self)
	Now define properties for these data members. More...
def	trf (self)
def	trf (self, value)
def	inDataUpdate (self, value)
def	outDataUpdate (self, value)
def	input (self)
def	output (self)
def	extraMetadata (self)
def	hasExecuted (self)
def	rc (self)
def	errMsg (self)
def	validation (self)
def	validation (self, value)
def	hasValidated (self)
def	isValidated (self)
def	first (self)
def	preExeStartTimes (self)
def	exeStartTimes (self)
def	exeStopTimes (self)
def	valStartTimes (self)
def	valStopTimes (self)
def	preExeCpuTime (self)
def	preExeWallTime (self)
def	cpuTime (self)
def	usrTime (self)
def	sysTime (self)
def	wallTime (self)
def	memStats (self)
def	memAnalysis (self)
def	postExeCpuTime (self)
def	postExeWallTime (self)
def	validationCpuTime (self)
def	validationWallTime (self)
def	cpuTimeTotal (self)
def	wallTimeTotal (self)
def	eventCount (self)
def	reSimEvent (self)
def	athenaMP (self)
def	dbMonitor (self)
def	setPreExeStart (self)
def	setValStart (self)
def	doAll (self, input=set(), output=set())
	Convenience function. More...

Public Attributes
	name
	conf
	Executor configuration: More...
	inData
	outData

Private Member Functions
def	_isCAEnabled (self)
	Check if running with CA. More...
def	_prepAthenaCommandLine (self)
	Prepare the correct command line to be used to invoke athena. More...
def	_writeAthenaWrapper (self, asetup=None, dbsetup=None, ossetup=None)
	Write a wrapper script which runs asetup and then Athena. More...
def	_smartMerge (self, fileArg)
	Manage smart merging of output files. More...
def	_targzipJiveXML (self)
def	_buildStandardCommand (self)

Private Attributes
	_isValidated
	_errMsg
	_rc
	_valStop
	_substep
	_inputEventTest
	_tryDropAndReload
	_extraRunargs
	_runtimeRunargs
	_literalRunargs
	_dataArgs
	_errorMaskFiles
	_inputDataTypeCountCheck
	_disableMT
	_disableMP
	_onlyMP
	Do we need to run asetup first? For a certain substep (e.g. More...
	_onlyMT
	_onlyMPWithRunargs
	_skeletonCA
	_perfMonFile
	_skeleton
	_jobOptionsTemplate
	_athenaConcurrentEvents
	_athenaMP
	_athenaMT
	_athenaMPWorkerTopDir
	_athenaMPFileReport
	_athenaMPEventOrdersFile
	_athenaMPReadEventOrders
	_athenaMPStrategy
	_topOptionsFiles
	Write the skeleton file and prep athena. More...
	_envUpdate
	Add input/output file information - this can't be done in init as we don't know what our inputs and outputs will be then. More...
	_logFileName
	_resimevents
	_hasValidated
	_memLeakResult
	Our parent will check the RC for us. More...
	_logScan
	_dbMonitor
	_exe
	Start building up the command line N.B. More...
	_cmd
	_originalCmd
	_asetup
	_dbsetup
	_containerSetup
	_workdir
	_alreadyInContainer
	_wrapperFile
	_setupFile
	_exeArgs
	_echoOutput
	_memMonitor
	_input
	_output
	_echologger
	_exeLogFile
	_echostream
	_hasExecuted
	_exeStart
	_memSummaryFile
	_memFullFile
	_exeStop
	_memStats
	_valStart
	_eventCount
	Check event counts (always do this by default) Do this here so that all script executors have this by default (covers most use cases with events) More...
	_name
	_inData
	_outData
	_extraMetadata
	_preExeStart
	_myMerger
	_trf
	_validation

Static Private Attributes
	_exitMessageLimit
	_defaultIgnorePatternFile

Detailed Description

Athena executor where failure is not consisered fatal.

Definition at line 1806 of file trfExe.py.

Member Function Documentation

_buildStandardCommand()

def python.trfExe.scriptExecutor._buildStandardCommand (   self )

privateinherited

Definition at line 705 of file trfExe.py.

    def _buildStandardCommand(self):
        if self._exe:
            self._cmd = [self.exe, ]
        else:
            raise trfExceptions.TransformExecutionException(trfExit.nameToCode('TRF_EXEC_SETUP_FAIL'), 
                                                            'No executor set in {0}'.format(self.__class__.__name__))
        for arg in self.exeArgs:
            if arg in self.conf.argdict:
                # If we have a list then add each element to our list, else just str() the argument value
                # Note if there are arguments which need more complex transformations then
                # consider introducing a special toExeArg() method.
                if isinstance(self.conf.argdict[arg].value, list):
                    self._cmd.extend([ str(v) for v in self.conf.argdict[arg].value])
                else:
                    self._cmd.append(str(self.conf.argdict[arg].value))
 
 

_isCAEnabled()

def python.trfExe.athenaExecutor._isCAEnabled (   self )

privateinherited

Check if running with CA.

Definition at line 1413 of file trfExe.py.

    def _isCAEnabled(self):
        # CA not present
        if 'CA' not in self.conf.argdict:
            # If there is no legacy skeleton, then we are running with CA
            if not self._skeleton:
                return True
            else:
                return False
 
        # CA present but None, all substeps running with CA
        if self.conf.argdict['CA'] is None:
            return True
 
        # CA enabled for a substep, running with CA
        if self.conf.argdict['CA'].returnMyValue(name=self.name, substep=self.substep) is True:
            return True
 
        return False
 

_prepAthenaCommandLine()

def python.trfExe.athenaExecutor._prepAthenaCommandLine (   self )

privateinherited

Prepare the correct command line to be used to invoke athena.

Definition at line 1433 of file trfExe.py.

    def _prepAthenaCommandLine(self):
        
        if 'athena' in self.conf.argdict:
            self._exe = self.conf.argdict['athena'].value
        self._cmd = [self._exe]
        
        # Find options for the current substep. Name is prioritised (e.g. RAWtoALL) over alias (e.g. r2a). Last look for 'all'
        currentSubstep = None
        if 'athenaopts' in self.conf.argdict:
            currentName = commonExecutorStepName(self.name)
            if currentName in self.conf.argdict['athenaopts'].value:
                currentSubstep = currentName
                if self.substep in self.conf.argdict['athenaopts'].value:
                    msg.info('Athenaopts found for {0} and {1}, joining options. '
                             'Consider changing your configuration to use just the name or the alias of the substep.'
                             .format(currentSubstep, self.substep))
                    self.conf.argdict['athenaopts'].value[currentSubstep].extend(self.conf.argdict['athenaopts'].value[self.substep])
                    del self.conf.argdict['athenaopts'].value[self.substep]
                    msg.debug('Athenaopts: {0}'.format(self.conf.argdict['athenaopts'].value))
            elif self.substep in self.conf.argdict['athenaopts'].value:
                currentSubstep = self.substep
            elif 'all' in self.conf.argdict['athenaopts'].value:
                currentSubstep = 'all'
 
        # See if there's a preloadlibs and a request to update LD_PRELOAD for athena
        preLoadUpdated = dict()
        if 'LD_PRELOAD' in self._envUpdate._envdict:
            preLoadUpdated[currentSubstep] = False
            if 'athenaopts' in self.conf.argdict:
                if currentSubstep is not None:
                    for athArg in self.conf.argdict['athenaopts'].value[currentSubstep]:
                        # This code is pretty ugly as the athenaopts argument contains
                        # strings which are really key/value pairs
                        if athArg.startswith('--preloadlib'):
                            try:
                                i = self.conf.argdict['athenaopts'].value[currentSubstep].index(athArg)
                                v = athArg.split('=', 1)[1]
                                msg.info('Updating athena --preloadlib option for substep {1} with: {0}'.format(self._envUpdate.value('LD_PRELOAD'), self.name))
                                newPreloads = ":".join(set(v.split(":")) | set(self._envUpdate.value('LD_PRELOAD').split(":")))
                                self.conf.argdict['athenaopts']._value[currentSubstep][i] = '--preloadlib={0}'.format(newPreloads)
                            except Exception as e:
                                msg.warning('Failed to interpret athena option: {0} ({1})'.format(athArg, e))
                            preLoadUpdated[currentSubstep] = True
                        break
            if not preLoadUpdated[currentSubstep]:
                msg.info('Setting athena preloadlibs for substep {1} to: {0}'.format(self._envUpdate.value('LD_PRELOAD'), self.name))
                if 'athenaopts' in self.conf.argdict:
                    if currentSubstep is not None:
                        self.conf.argdict['athenaopts'].value[currentSubstep].append("--preloadlib={0}".format(self._envUpdate.value('LD_PRELOAD')))
                    else:
                        self.conf.argdict['athenaopts'].value['all'] = ["--preloadlib={0}".format(self._envUpdate.value('LD_PRELOAD'))]
                else:
                    self.conf.argdict['athenaopts'] = trfArgClasses.argSubstepList(["--preloadlib={0}".format(self._envUpdate.value('LD_PRELOAD'))])
 
        # Now update command line with the options we have (including any changes to preload)
        if 'athenaopts' in self.conf.argdict:
            if currentSubstep is None and "all" in self.conf.argdict['athenaopts'].value:
                self._cmd.extend(self.conf.argdict['athenaopts'].value['all'])
            elif currentSubstep in self.conf.argdict['athenaopts'].value:
                self._cmd.extend(self.conf.argdict['athenaopts'].value[currentSubstep])
        
        if currentSubstep is None:
            currentSubstep = 'all'
        
        if self._tryDropAndReload:
            if self._isCAEnabled():
                msg.info('ignoring "--drop-and-reload" for CA-based transforms, config cleaned up anyway')
            elif 'valgrind' in self.conf._argdict and self.conf._argdict['valgrind'].value is True:
                msg.info('Disabling "--drop-and-reload" because the job is configured to use Valgrind')
            elif 'athenaopts' in self.conf.argdict:
                athenaConfigRelatedOpts = ['--config-only','--drop-and-reload']
                # Note for athena options we split on '=' so that we properly get the option and not the whole "--option=value" string
                if currentSubstep in self.conf.argdict['athenaopts'].value:
                    conflictOpts = set(athenaConfigRelatedOpts).intersection(set([opt.split('=')[0] for opt in self.conf.argdict['athenaopts'].value[currentSubstep]]))
                    if len(conflictOpts) > 0:
                        msg.info('Not appending "--drop-and-reload" to athena command line because these options conflict: {0}'.format(list(conflictOpts)))
                    else:
                        msg.info('Appending "--drop-and-reload" to athena options')
                        self._cmd.append('--drop-and-reload')
                else:
                    msg.info('No Athenaopts for substep {0}, appending "--drop-and-reload" to athena options'.format(self.name))
                    self._cmd.append('--drop-and-reload')
            else:
                # This is the 'standard' case - so drop and reload should be ok
                msg.info('Appending "--drop-and-reload" to athena options')
                self._cmd.append('--drop-and-reload')
        else:
            msg.info('Skipping test for "--drop-and-reload" in this executor')
            
        if not self._isCAEnabled():  #For CA-jobs, threads and nproc set in runargs file
            # For AthenaMT apply --threads=N if threads have been configured via ATHENA_CORE_NUMBER + multithreaded
            if self._athenaMT > 0 and not self._disableMT:
                if not ('athenaopts' in self.conf.argdict and
                        any('--threads' in opt for opt in self.conf.argdict['athenaopts'].value[currentSubstep])):
                        self._cmd.append('--threads=%s' % str(self._athenaMT))
 
            # For AthenaMP apply --nprocs=N if threads have been configured via ATHENA_CORE_NUMBER + multiprocess
            if self._athenaMP > 0 and not self._disableMP:
                if not ('athenaopts' in self.conf.argdict and
                        any('--nprocs' in opt for opt in self.conf.argdict['athenaopts'].value[currentSubstep])):
                        self._cmd.append('--nprocs=%s' % str(self._athenaMP))
 
        # Add topoptions
        if self._skeleton or self._skeletonCA:
            self._cmd += self._topOptionsFiles
            msg.info('Updated script arguments with topoptions: %s', self._cmd)
 
 

_smartMerge()

def python.trfExe.athenaExecutor._smartMerge (   self,   fileArg  )

privateinherited

Manage smart merging of output files.

Parameters

fileArg

File argument to merge

Definition at line 1719 of file trfExe.py.

    def _smartMerge(self, fileArg):
        
        if 'selfMerge' not in dir(fileArg):
            msg.info('Files in {0} cannot merged (no selfMerge() method is implemented)'.format(fileArg.name))
            return
        
        if fileArg.mergeTargetSize == 0:
            msg.info('Files in {0} will not be merged as target size is set to 0'.format(fileArg.name))
            return
 
        
        mergeCandidates = [list()]
        currentMergeSize = 0
        for fname in fileArg.value:
            size = fileArg.getSingleMetadata(fname, 'file_size')
            if not isinstance(size, int):
                msg.warning('File size metadata for {0} was not correct, found type {1}. Aborting merge attempts.'.format(fileArg, type(size)))
                return
            # if there is no file in the job, then we must add it
            if len(mergeCandidates[-1]) == 0:
                msg.debug('Adding file {0} to current empty merge list'.format(fname))
                mergeCandidates[-1].append(fname)
                currentMergeSize += size
                continue
            # see if adding this file gets us closer to the target size (but always add if target size is negative)
            if fileArg.mergeTargetSize < 0 or math.fabs(currentMergeSize + size - fileArg.mergeTargetSize) < math.fabs(currentMergeSize - fileArg.mergeTargetSize):
                msg.debug('Adding file {0} to merge list {1} as it gets closer to the target size'.format(fname, mergeCandidates[-1]))
                mergeCandidates[-1].append(fname)
                currentMergeSize += size
                continue
            # close this merge list and start a new one
            msg.debug('Starting a new merge list with file {0}'.format(fname))
            mergeCandidates.append([fname])
            currentMergeSize = size
            
        msg.debug('First pass splitting will merge files in this way: {0}'.format(mergeCandidates))
        
        if len(mergeCandidates) == 1:
            # Merging to a single file, so use the original filename that the transform
            # was started with
            mergeNames = [fileArg.originalName]
        else:
            # Multiple merge targets, so we need a set of unique names
            counter = 0
            mergeNames = []
            for mergeGroup in mergeCandidates:
                # Note that the individual worker files get numbered with 3 digit padding,
                # so these non-padded merges should be fine
                mergeName = fileArg.originalName + '_{0}'.format(counter)
                while path.exists(mergeName):
                    counter += 1
                    mergeName = fileArg.originalName + '_{0}'.format(counter)
                mergeNames.append(mergeName)
                counter += 1
        # Now actually do the merges
        for targetName, mergeGroup, counter in zip(mergeNames, mergeCandidates, list(range(len(mergeNames)))):
            msg.info('Want to merge files {0} to {1}'.format(mergeGroup, targetName))
            if len(mergeGroup) <= 1:
                msg.info('Skip merging for single file')
            else:
                
                self._myMerger.append(fileArg.selfMerge(output=targetName, inputs=mergeGroup, counter=counter, argdict=self.conf.argdict))
 
 

_targzipJiveXML()

def python.trfExe.athenaExecutor._targzipJiveXML (   self )

privateinherited

Definition at line 1783 of file trfExe.py.

    def _targzipJiveXML(self):
        #tgzipping JiveXML files
        targetTGZName = self.conf.dataDictionary['TXT_JIVEXMLTGZ'].value[0]
        if os.path.exists(targetTGZName):
            os.remove(targetTGZName)
 
        import tarfile
        fNameRE = re.compile(r"JiveXML\_\d+\_\d+.xml")
 
        # force gz compression
        tar = tarfile.open(targetTGZName, "w:gz")
        for fName in os.listdir('.'):
            matches = fNameRE.findall(fName)
            if len(matches) > 0:
                if fNameRE.findall(fName)[0] == fName:
                    msg.info('adding %s to %s', fName, targetTGZName)
                    tar.add(fName)
 
        tar.close()
        msg.info('JiveXML compression: %s has been written and closed.', targetTGZName)
 
 

_writeAthenaWrapper()

def python.trfExe.athenaExecutor._writeAthenaWrapper (   self,   asetup = None,   dbsetup = None,   ossetup = None  )

privateinherited

Write a wrapper script which runs asetup and then Athena.

Definition at line 1544 of file trfExe.py.

    def _writeAthenaWrapper(
        self,
        asetup = None,
        dbsetup = None,
        ossetup = None
        ):
        self._originalCmd        = self._cmd
        self._asetup             = asetup
        self._dbsetup            = dbsetup
        self._containerSetup     = ossetup
        self._workdir            = os.getcwd()
        self._alreadyInContainer = self._workdir.startswith("/srv")
        self._wrapperFile        = 'runwrapper.{name}.sh'.format(name = self._name)
        self._setupFile          = 'setup.{name}.sh'.format(name = self._name)
 
        # Create a setupATLAS script
        setupATLAS = 'my_setupATLAS.sh'
        with open(setupATLAS, 'w') as f:
            print("#!/bin/bash", file=f)
            print("""
if [ -z $ATLAS_LOCAL_ROOT_BASE ]; then
  export ATLAS_LOCAL_ROOT_BASE=/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase
fi
source ${ATLAS_LOCAL_ROOT_BASE}/user/atlasLocalSetup.sh"""
                  , file=f)
        os.chmod(setupATLAS, 0o755)
 
        msg.debug(
            'Preparing wrapper file {wrapperFileName} with '
            'asetup={asetupStatus} and dbsetup={dbsetupStatus}'.format(
                wrapperFileName = self._wrapperFile,
                asetupStatus    = self._asetup,
                dbsetupStatus   = self._dbsetup
            )
        )
 
        container_cmd = None
        try:
            with open(self._wrapperFile, 'w') as wrapper:
                print('#!/bin/sh', file=wrapper)
                if self._containerSetup is not None:
                    container_cmd = [ os.path.abspath(setupATLAS),
                                     "-c",
                                     self._containerSetup,
                                     "--pwd",
                                     self._workdir,
                                     "-s",
                                     os.path.join('.', self._setupFile),
                                     "-r"]
                    print('echo "This wrapper is executed within a container! For a local re-run, do:"', file=wrapper)
                    print('echo " '+ " ".join(['setupATLAS'] + container_cmd[1:] + [path.join('.', self._wrapperFile)]) + '"', file=wrapper)
                    print('echo "N.B.: if launching a nested container, navigate to /srv before running the above command"',
                          file = wrapper)
                    print('echo "      and use --pwd workdir, where workdir is the transform running directory within /srv"',
                          file=wrapper)
                    print('echo', file=wrapper)
 
                if asetup:
                    wfile = wrapper
                    asetupFile = None
                    # if the substep is executed within a container, setup athena with a separate script
                    # e.g. setupATLAS -c el9 -s setup.RDOtoRDOTrigger.sh -r runwrapper.RDOtoRDOTrigger.sh ...
                    if self._containerSetup is not None:
                        asetupFile = open(self._setupFile, 'w')
                        wfile = asetupFile
                    print(f'source ./{setupATLAS} -q', file=wfile)
                    print(f'asetup {asetup}', file=wfile)
                    print('if [ ${?} != "0" ]; then exit 255; fi', file=wfile)
                if dbsetup:
                    dbroot = path.dirname(dbsetup)
                    dbversion = path.basename(dbroot)
                    print("# DBRelease setup", file=wrapper)
                    print('echo Setting up DBRelease {dbroot} environment'.format(dbroot = dbroot), file=wrapper)
                    print('export DBRELEASE={dbversion}'.format(dbversion = dbversion), file=wrapper)
                    print('export CORAL_AUTH_PATH={directory}'.format(directory = path.join(dbroot, 'XMLConfig')), file=wrapper)
                    print('export CORAL_DBLOOKUP_PATH={directory}'.format(directory = path.join(dbroot, 'XMLConfig')), file=wrapper)
                    print('export TNS_ADMIN={directory}'.format(directory = path.join(dbroot, 'oracle-admin')), file=wrapper)
                    print('DATAPATH={dbroot}:$DATAPATH'.format(dbroot = dbroot), file=wrapper)
                if self._disableMT:
                    print("# AthenaMT explicitly disabled for this executor", file=wrapper)
                if self._disableMP:
                    print("# AthenaMP explicitly disabled for this executor", file=wrapper)
                if self._envUpdate.len > 0:
                    for envSetting in  self._envUpdate.values:
                        if not envSetting.startswith('LD_PRELOAD'):
                            print("export", envSetting, file=wrapper)
                # If Valgrind is engaged, a serialised Athena configuration file
                # is generated for use with a subsequent run of Athena with
                # Valgrind.
                if 'valgrind' in self.conf._argdict and self.conf._argdict['valgrind'].value is True:
                    msg.info('Valgrind engaged')
                    # Define the file name of the serialised Athena
                    # configuration.
                    AthenaSerialisedConfigurationFile = "{name}Conf.pkl".format(
                        name = self._name
                    )
                    # Run Athena for generation of its serialised configuration.
                    print(' '.join(self._cmd), "--config-only={0}".format(AthenaSerialisedConfigurationFile), file=wrapper)
                    print('if [ $? != "0" ]; then exit 255; fi', file=wrapper)
                    # Generate a Valgrind command, suppressing or ussing default
                    # options as requested and extra options as requested.
                    if 'valgrindDefaultOpts' in self.conf._argdict:
                        defaultOptions = self.conf._argdict['valgrindDefaultOpts'].value
                    else:
                        defaultOptions = True
                    if 'valgrindExtraOpts' in self.conf._argdict:
                        extraOptionsList = self.conf._argdict['valgrindExtraOpts'].value
                    else:
                        extraOptionsList = None
                    msg.debug("requested Valgrind command basic options: {options}".format(options = defaultOptions))
                    msg.debug("requested Valgrind command extra options: {options}".format(options = extraOptionsList))
                    command = ValgrindCommand(
                        defaultOptions = defaultOptions,
                        extraOptionsList = extraOptionsList,
                        AthenaSerialisedConfigurationFile = \
                            AthenaSerialisedConfigurationFile
                    )
                    msg.debug("Valgrind command: {command}".format(command = command))
                    print(command, file=wrapper)
                # If VTune is engaged, a serialised Athena configuration file
                # is generated for use with a subsequent run of Athena with
                # VTune.
                elif 'vtune' in self.conf._argdict and self.conf._argdict['vtune'].value is True:
                    msg.info('VTune engaged')
                    # Define the file name of the serialised Athena
                    # configuration.
                    AthenaSerialisedConfigurationFile = "{name}Conf.pkl".format(
                        name = self._name
                    )
                    # Run Athena for generation of its serialised configuration.
                    print(' '.join(self._cmd), "--config-only={0}".format(AthenaSerialisedConfigurationFile), file=wrapper)
                    print('if [ $? != "0" ]; then exit 255; fi', file=wrapper)
                    # Generate a VTune command, suppressing or ussing default
                    # options as requested and extra options as requested.
                    if 'vtuneDefaultOpts' in self.conf._argdict:
                        defaultOptions = self.conf._argdict['vtuneDefaultOpts'].value
                    else:
                        defaultOptions = True
                    if 'vtuneExtraOpts' in self.conf._argdict:
                        extraOptionsList = self.conf._argdict['vtuneExtraOpts'].value
                    else:
                        extraOptionsList = None
                    msg.debug("requested VTune command basic options: {options}".format(options = defaultOptions))
                    msg.debug("requested VTune command extra options: {options}".format(options = extraOptionsList))
                    command = VTuneCommand(
                        defaultOptions = defaultOptions,
                        extraOptionsList = extraOptionsList,
                        AthenaSerialisedConfigurationFile = \
                            AthenaSerialisedConfigurationFile
                    )
                    msg.debug("VTune command: {command}".format(command = command))
                    print(command, file=wrapper)
                else:
                    msg.info('Valgrind/VTune not engaged')
                    # run Athena command
                    print(' '.join(self._cmd), file=wrapper)
            os.chmod(self._wrapperFile, 0o755)
        except OSError as e:
            errMsg = 'error writing athena wrapper {fileName}: {error}'.format(
                fileName = self._wrapperFile,
                error = e
            )
            msg.error(errMsg)
            raise trfExceptions.TransformExecutionException(
                trfExit.nameToCode('TRF_EXEC_SETUP_WRAPPER'),
                errMsg
            )
        self._cmd = [ path.join('.', self._wrapperFile) ]
        if self._containerSetup is not None:
            asetupFile.close()
            self._cmd = container_cmd + self._cmd
 
 

athenaMP()

def python.trfExe.transformExecutor.athenaMP (   self )

inherited

Definition at line 451 of file trfExe.py.

    def athenaMP(self):
        return self._athenaMP
 

cpuTime()

def python.trfExe.transformExecutor.cpuTime (   self )

inherited

Definition at line 365 of file trfExe.py.

    def cpuTime(self):
        if self._exeStart and self._exeStop:
            return calcCpuTime(self._exeStart, self._exeStop)
        else:
            return None
 

cpuTimeTotal()

def python.trfExe.transformExecutor.cpuTimeTotal (   self )

inherited

Definition at line 429 of file trfExe.py.

    def cpuTimeTotal(self):
        if self._preExeStart and self._valStop:
            return calcCpuTime(self._preExeStart, self._valStop)
        else:
            return None
 

dbMonitor()

def python.trfExe.transformExecutor.dbMonitor (   self )

inherited

Definition at line 455 of file trfExe.py.

    def dbMonitor(self):
        return self._dbMonitor
 
 

disableMP() [1/2]

def python.trfExe.athenaExecutor.disableMP (   self )

inherited

Definition at line 938 of file trfExe.py.

    def disableMP(self):
        return self._disableMP
    

disableMP() [2/2]

def python.trfExe.athenaExecutor.disableMP (   self,   value  )

inherited

Definition at line 942 of file trfExe.py.

    def disableMP(self, value):
        self._disableMP = value
    

disableMT() [1/2]

def python.trfExe.athenaExecutor.disableMT (   self )

inherited

Definition at line 946 of file trfExe.py.

    def disableMT(self):
        return self._disableMT
    

disableMT() [2/2]

def python.trfExe.athenaExecutor.disableMT (   self,   value  )

inherited

Definition at line 950 of file trfExe.py.

    def disableMT(self, value):
        self._disableMT = value
 

doAll()

def python.trfExe.transformExecutor.doAll (   self,   input = set(),   output = set()  )

inherited

Convenience function.

Definition at line 498 of file trfExe.py.

    def doAll(self, input=set(), output=set()):
        self.preExecute(input, output)
        self.execute()
        self.postExecute()
        self.validate()
 

errMsg()

def python.trfExe.transformExecutor.errMsg (   self )

inherited

Definition at line 303 of file trfExe.py.

    def errMsg(self):
        return self._errMsg
    

eventCount()

def python.trfExe.transformExecutor.eventCount (   self )

inherited

Definition at line 443 of file trfExe.py.

    def eventCount(self):
        return self._eventCount
 

exe() [1/2]

def python.trfExe.scriptExecutor.exe (   self )

inherited

Definition at line 640 of file trfExe.py.

    def exe(self):
        return self._exe
    

exe() [2/2]

def python.trfExe.scriptExecutor.exe (   self,   value  )

inherited

Definition at line 644 of file trfExe.py.

    def exe(self, value):
        self._exe = value
        self._extraMetadata['script'] = value
 

exeArgs() [1/2]

def python.trfExe.scriptExecutor.exeArgs (   self )

inherited

Definition at line 649 of file trfExe.py.

    def exeArgs(self):
        return self._exeArgs
    

exeArgs() [2/2]

def python.trfExe.scriptExecutor.exeArgs (   self,   value  )

inherited

Definition at line 653 of file trfExe.py.

    def exeArgs(self, value):
        self._exeArgs = value
#        self._extraMetadata['scriptArgs'] = value
 

execute()

def python.trfExe.scriptExecutor.execute (   self )

inherited

Reimplemented from python.trfExe.transformExecutor.

Reimplemented in python.trfExe.bsMergeExecutor, and python.trfExe.POOLMergeExecutor.

Definition at line 722 of file trfExe.py.

    def execute(self):
        self._hasExecuted = True
        msg.info('Starting execution of {0} ({1})'.format(self._name, self._cmd))
        
        self._exeStart = os.times()
        msg.debug('exeStart time is {0}'.format(self._exeStart))
        if ('execOnly' in self.conf.argdict and self.conf.argdict['execOnly'] is True):
            msg.info('execOnly flag is set - execution will now switch, replacing the transform')
            os.execvp(self._cmd[0], self._cmd)
 
        encargs = {'encoding' : 'utf8'}
        try:
            # if we are already inside a container, then
            # must map /srv of the nested container to /srv of the parent container:
            # https://twiki.atlas-canada.ca/bin/view/AtlasCanada/Containers#Mount_point_summary
            if self._alreadyInContainer and self._containerSetup is not None:
                msg.info("chdir /srv to launch a nested container for the substep")
                os.chdir("/srv")
            p = subprocess.Popen(self._cmd, shell = False, stdout = subprocess.PIPE, stderr = subprocess.STDOUT, bufsize = 1, **encargs)
            # go back to the original working directory immediately after to store prmon in it
            if self._alreadyInContainer and self._containerSetup is not None:
                msg.info("chdir {} after launching the nested container".format(self._workdir))
                os.chdir(self._workdir)
 
            if self._memMonitor:
                try:
                    self._memSummaryFile = 'prmon.summary.' + self._name + '.json'
                    self._memFullFile = 'prmon.full.' + self._name
                    memMonitorCommand = ['prmon', '--pid', str(p.pid), '--filename', 'prmon.full.' + self._name, 
                                         '--json-summary', self._memSummaryFile, '--log-filename', 'prmon.' + self._name + '.log',
                                         '--interval', '30']
                    mem_proc = subprocess.Popen(memMonitorCommand, shell = False, close_fds=True, **encargs)
                    # TODO - link mem.full.current to mem.full.SUBSTEP
                except Exception as e:
                    msg.warning('Failed to spawn memory monitor for {0}: {1}'.format(self._name, e))
                    self._memMonitor = False
            
            while p.poll() is None:
                line = p.stdout.readline()
                if line:
                    self._echologger.info(line.rstrip())
            # Hoover up remaining buffered output lines
            for line in p.stdout:
                self._echologger.info(line.rstrip())
    
            self._rc = p.returncode
            msg.info('%s executor returns %d', self._name, self._rc)
            self._exeStop = os.times()
            msg.debug('exeStop time is {0}'.format(self._exeStop))
        except OSError as e:
            errMsg = 'Execution of {0} failed and raised OSError: {1}'.format(self._cmd[0], e)
            msg.error(errMsg)
            raise trfExceptions.TransformExecutionException(trfExit.nameToCode('TRF_EXEC'), errMsg)
        finally:
            if self._memMonitor:
                try:
                    mem_proc.send_signal(signal.SIGUSR1)
                    countWait = 0
                    while (not mem_proc.poll()) and countWait < 10:
                        time.sleep(0.1)
                        countWait += 1
                except OSError:
                    pass
        
        

exeStartTimes()

def python.trfExe.transformExecutor.exeStartTimes (   self )

inherited

Definition at line 335 of file trfExe.py.

    def exeStartTimes(self):
        return self._exeStart
 

exeStopTimes()

def python.trfExe.transformExecutor.exeStopTimes (   self )

inherited

Definition at line 339 of file trfExe.py.

    def exeStopTimes(self):
        return self._exeStop
 

extraMetadata()

def python.trfExe.transformExecutor.extraMetadata (   self )

inherited

Definition at line 291 of file trfExe.py.

    def extraMetadata(self):
        return self._extraMetadata
    

first()

def python.trfExe.transformExecutor.first (   self )

inherited

Note

At the moment only athenaExecutor sets this property, but that might be changed...

Definition at line 324 of file trfExe.py.

    def first(self):
        if hasattr(self, '_first'):
            return self._first
        else:
            return None
 

hasExecuted()

def python.trfExe.transformExecutor.hasExecuted (   self )

inherited

Definition at line 295 of file trfExe.py.

    def hasExecuted(self):
        return self._hasExecuted
    

hasValidated()

def python.trfExe.transformExecutor.hasValidated (   self )

inherited

Definition at line 315 of file trfExe.py.

    def hasValidated(self):
        return self._hasValidated
    

inData() [1/2]

def python.trfExe.transformExecutor.inData (   self )

inherited

Definition at line 234 of file trfExe.py.

    def inData(self):
        
        if '_inData' in dir(self):
            return self._inData
        return None
    

inData() [2/2]

def python.trfExe.transformExecutor.inData (   self,   value  )

inherited

Definition at line 241 of file trfExe.py.

    def inData(self, value):
        self._inData = set(value)
        

inDataUpdate()

def python.trfExe.transformExecutor.inDataUpdate (   self,   value  )

inherited

Definition at line 244 of file trfExe.py.

    def inDataUpdate(self, value):
        
        if '_inData' in dir(self):
            self._inData.update(value)
        else:
            
            self.inData = value
 
 

input()

def python.trfExe.transformExecutor.input (   self )

inherited

Note

This returns the actual input data with which this executor ran (c.f. inData which returns all the possible data types this executor could run with)

Definition at line 275 of file trfExe.py.

    def input(self):
        
        if '_input' in dir(self):
            return self._input
        return None
    

inputDataTypeCountCheck() [1/2]

def python.trfExe.athenaExecutor.inputDataTypeCountCheck (   self )

inherited

Definition at line 926 of file trfExe.py.

    def inputDataTypeCountCheck(self):
        return self._inputDataTypeCountCheck
    

inputDataTypeCountCheck() [2/2]

def python.trfExe.athenaExecutor.inputDataTypeCountCheck (   self,   value  )

inherited

Definition at line 930 of file trfExe.py.

    def inputDataTypeCountCheck(self, value):
        self._inputDataTypeCountCheck = value
        

isValidated()

def python.trfExe.transformExecutor.isValidated (   self )

inherited

Definition at line 319 of file trfExe.py.

    def isValidated(self):
        return self._isValidated
    

memAnalysis()

def python.trfExe.transformExecutor.memAnalysis (   self )

inherited

Definition at line 397 of file trfExe.py.

    def memAnalysis(self):
        return self._memLeakResult
 

memStats()

def python.trfExe.transformExecutor.memStats (   self )

inherited

Definition at line 393 of file trfExe.py.

    def memStats(self):
        return self._memStats
 

myMerger()

def python.trfExe.transformExecutor.myMerger (   self )

inherited

Now define properties for these data members.

Definition at line 206 of file trfExe.py.

    def myMerger(self):
        return self._myMerger
 

name() [1/2]

def python.trfExe.transformExecutor.name (   self )

inherited

Definition at line 210 of file trfExe.py.

    def name(self):
        return self._name
 

name() [2/2]

def python.trfExe.transformExecutor.name (   self,   value  )

inherited

Definition at line 214 of file trfExe.py.

    def name(self, value):
        self._name = value
 

onlyMP() [1/2]

def python.trfExe.athenaExecutor.onlyMP (   self )

inherited

Definition at line 954 of file trfExe.py.

    def onlyMP(self):
        return self._onlyMP
    

onlyMP() [2/2]

def python.trfExe.athenaExecutor.onlyMP (   self,   value  )

inherited

Definition at line 958 of file trfExe.py.

    def onlyMP(self, value):
        self._onlyMP = value
        

onlyMT() [1/2]

def python.trfExe.athenaExecutor.onlyMT (   self )

inherited

Definition at line 962 of file trfExe.py.

    def onlyMT(self):
        return self._onlyMT
 

onlyMT() [2/2]

def python.trfExe.athenaExecutor.onlyMT (   self,   value  )

inherited

Definition at line 966 of file trfExe.py.

    def onlyMT(self, value):
        self._onlyMT = value
 

outData() [1/2]

def python.trfExe.transformExecutor.outData (   self )

inherited

Definition at line 254 of file trfExe.py.

    def outData(self):
        
        if '_outData' in dir(self):
            return self._outData
        return None
 

outData() [2/2]

def python.trfExe.transformExecutor.outData (   self,   value  )

inherited

Definition at line 261 of file trfExe.py.

    def outData(self, value):
        self._outData = set(value)
    

outDataUpdate()

def python.trfExe.transformExecutor.outDataUpdate (   self,   value  )

inherited

Definition at line 264 of file trfExe.py.

    def outDataUpdate(self, value):
        
        if '_outData' in dir(self):
            self._outData.update(value)
        else:
            
            self.outData = value
    

output()

def python.trfExe.transformExecutor.output (   self )

inherited

Note

This returns the actual output data with which this executor ran (c.f. outData which returns all the possible data types this executor could run with)

Definition at line 284 of file trfExe.py.

    def output(self):
        
        if '_output' in dir(self):
            return self._output
        return None
    

postExeCpuTime()

def python.trfExe.transformExecutor.postExeCpuTime (   self )

inherited

Definition at line 401 of file trfExe.py.

    def postExeCpuTime(self):
        if self._exeStop and self._valStart:
            return calcCpuTime(self._exeStop, self._valStart)
        else:
            return None
 

postExecute()

def python.trfExe.athenaExecutor.postExecute (   self )

inherited

Reimplemented from python.trfExe.scriptExecutor.

Definition at line 1233 of file trfExe.py.

    def postExecute(self):
        super(athenaExecutor, self).postExecute()
 
        # Handle executor substeps
        if self.conf.totalExecutorSteps > 1:
            if self._athenaMP > 0:
                outputDataDictionary = dict([ (dataType, self.conf.dataDictionary[dataType]) for dataType in self._output ])
                athenaMPOutputHandler(self._athenaMPFileReport, self._athenaMPWorkerTopDir, outputDataDictionary, self._athenaMP, False, self.conf.argdict)
            if self.conf.executorStep == self.conf.totalExecutorSteps - 1:
                # first loop over datasets for the output
                for dataType in self._output:
                    newValue = []
                    if self._athenaMP > 0:
                        # assume the same number of workers all the time
                        for i in range(self.conf.totalExecutorSteps):
                            for v in self.conf.dataDictionary[dataType].value:
                                newValue.append(v.replace('_{0}{1}_'.format(executorStepSuffix, self.conf.executorStep),
                                                          '_{0}{1}_'.format(executorStepSuffix, i)))
                    else:
                        self.conf.dataDictionary[dataType].multipleOK = True
                        # just combine all executors
                        for i in range(self.conf.totalExecutorSteps):
                            newValue.append(self.conf.dataDictionary[dataType].originalName + '_{0}{1}'.format(executorStepSuffix, i))
                    self.conf.dataDictionary[dataType].value = newValue
 
                    # do the merging if needed
                    if self.conf.dataDictionary[dataType].io == "output" and len(self.conf.dataDictionary[dataType].value) > 1:
                        self._smartMerge(self.conf.dataDictionary[dataType])
 
        # If this was an athenaMP run then we need to update output files
        elif self._athenaMP > 0:
            outputDataDictionary = dict([ (dataType, self.conf.dataDictionary[dataType]) for dataType in self._output ])
            
            skipFileChecks=False
            if 'eventService' in self.conf.argdict and self.conf.argdict['eventService'].value:
                skipFileChecks=True
            athenaMPOutputHandler(self._athenaMPFileReport, self._athenaMPWorkerTopDir, outputDataDictionary, self._athenaMP, skipFileChecks, self.conf.argdict)
            for dataType in self._output:
                if self.conf.dataDictionary[dataType].io == "output" and len(self.conf.dataDictionary[dataType].value) > 1:
                    self._smartMerge(self.conf.dataDictionary[dataType])
        
        if 'TXT_JIVEXMLTGZ' in self.conf.dataDictionary:
            self._targzipJiveXML()
 
        # Summarise events passed the filter ISF_SimEventFilter from log.ReSim
        # This is a bit ugly to have such a specific feature here though
        # TODO
        # The best is to have a general approach so that user can extract useful info from log
        # Instead of hard coding a pattern, one idea could be that user provides a regExp pattern
        # in which the wanted variable is grouped by a name, then transforms could decode the pattern
        # and use it to extract required info and do the summation during log scan.
        if self._logFileName=='log.ReSim' and self.name=='ReSim':
            msg.info('scanning {0} for reporting events passed the filter ISF_SimEventFilter'.format(self._logFileName))
            self._resimevents = reportEventsPassedSimFilter(self._logFileName)
 
        # Remove intermediate input/output files of sub-steps
        # Delete only files with io="temporay" which are files with pattern "tmp*"
        # Some stubs like tmp.RDO_TRIG_000 created in AthenaMP mode or
        # tmp.HIST_ESD_INT, tmp.HIST_AOD_INT as input to DQHistogramMerge.py are not deleted
        # Enable if --deleteIntermediateOutputfiles is set
        if ('deleteIntermediateOutputfiles' in self.conf._argdict and self.conf._argdict['deleteIntermediateOutputfiles'].value):
          inputDataDictionary = dict([ (dataType, self.conf.dataDictionary[dataType]) for dataType in self._input ])
 
          for k, v in inputDataDictionary.items():
            if not v.io == 'temporary':
              continue
            for filename in v.value:
              if os.access(filename, os.R_OK) and not filename.startswith("/cvmfs"):
                msg.info("Removing intermediate {0} input file {1}".format(k, filename))
                # Check if symbolic link and delete also linked file
                if (os.path.realpath(filename) != filename):
                  targetpath = os.path.realpath(filename)
                os.unlink(filename)
                if (targetpath) and os.access(targetpath, os.R_OK):
                  os.unlink(targetpath)
 
 

postExeWallTime()

def python.trfExe.transformExecutor.postExeWallTime (   self )

inherited

Definition at line 408 of file trfExe.py.

    def postExeWallTime(self):
        if self._exeStop and self._valStart:
            return calcWallTime(self._exeStop, self._valStart)
        else:
            return None
 

preExeCpuTime()

def python.trfExe.transformExecutor.preExeCpuTime (   self )

inherited

Definition at line 351 of file trfExe.py.

    def preExeCpuTime(self):
        if self._preExeStart and self._exeStart:
            return calcCpuTime(self._preExeStart, self._exeStart)
        else:
            return None
 

preExecute()

def python.trfExe.athenaExecutor.preExecute (   self,   input = set(),   output = set()  )

inherited

Reimplemented from python.trfExe.scriptExecutor.

Reimplemented in python.trfExe.reductionFrameworkExecutor, and python.trfExe.POOLMergeExecutor.

Definition at line 969 of file trfExe.py.

    def preExecute(self, input = set(), output = set()):
        self.setPreExeStart()
        msg.debug('Preparing for execution of {0} with inputs {1} and outputs {2}'.format(self.name, input, output))
        
        # Check we actually have events to process!
        inputEvents = 0
        dt = ""
        if self._inputDataTypeCountCheck is None:
            self._inputDataTypeCountCheck = input
        for dataType in self._inputDataTypeCountCheck:
            if self.conf.dataDictionary[dataType].nentries == 'UNDEFINED':
                continue
 
            thisInputEvents = self.conf.dataDictionary[dataType].nentries
            if thisInputEvents > inputEvents:
                inputEvents = thisInputEvents
                dt = dataType
 
        # Now take into account skipEvents and maxEvents
        if ('skipEvents' in self.conf.argdict and 
            self.conf.argdict['skipEvents'].returnMyValue(name=self._name, substep=self._substep, first=self.conf.firstExecutor) is not None):
            mySkipEvents = self.conf.argdict['skipEvents'].returnMyValue(name=self._name, substep=self._substep, first=self.conf.firstExecutor)
        else:
            mySkipEvents = 0
 
        if ('maxEvents' in self.conf.argdict and 
            self.conf.argdict['maxEvents'].returnMyValue(name=self._name, substep=self._substep, first=self.conf.firstExecutor) is not None):
            myMaxEvents = self.conf.argdict['maxEvents'].returnMyValue(name=self._name, substep=self._substep, first=self.conf.firstExecutor)
        else:
            myMaxEvents = -1
        
        # Any events to process...?
        if (self._inputEventTest and mySkipEvents > 0 and mySkipEvents >= inputEvents):
            raise trfExceptions.TransformExecutionException(trfExit.nameToCode('TRF_NOEVENTS'),
                                                           'No events to process: {0} (skipEvents) >= {1} (inputEvents of {2}'.format(mySkipEvents, inputEvents, dt))
        
        try:
            # Expected events to process
            if (myMaxEvents != -1):
                if (self.inData and next(iter(self.inData)) == 'inNULL'):
                    expectedEvents = myMaxEvents
                else:
                    expectedEvents = min(inputEvents-mySkipEvents, myMaxEvents)
            else:
                expectedEvents = inputEvents-mySkipEvents
        except TypeError:
            # catching type error from UNDEFINED inputEvents count
            msg.info('input event count is UNDEFINED, setting expectedEvents to 0')
            expectedEvents = 0
        
        
        OSSetupString = None
 
        # Extract the asetup string
        asetupString = None
        legacyThreadingRelease = False
        if 'asetup' in self.conf.argdict:
            asetupString = self.conf.argdict['asetup'].returnMyValue(name=self._name, substep=self._substep, first=self.conf.firstExecutor)
            legacyThreadingRelease = asetupReleaseIsOlderThan(asetupString, 22)
        else:
            msg.info('Asetup report: {0}'.format(asetupReport()))
 
        if asetupString is not None:
            legacyOSRelease = asetupReleaseIsOlderThan(asetupString, 24)
            currentOS = os.environ['ALRB_USER_PLATFORM']
            if legacyOSRelease and "centos7" not in currentOS:
                OSSetupString = "centos7"
                msg.info('Legacy release required for the substep {}, will setup a container running {}'.format(self._substep, OSSetupString))
 
 
        # allow overriding the container OS using a flag
        if 'runInContainer' in self.conf.argdict:
            OSSetupString = self.conf.argdict['runInContainer'].returnMyValue(name=self._name, substep=self._substep, first=self.conf.firstExecutor)
            msg.info('The step {} will be performed in a container running {}, as explicitly requested'.format(self._substep, OSSetupString))
            if OSSetupString is not None and asetupString is None:
                raise trfExceptions.TransformExecutionException(trfExit.nameToCode('TRF_EXEC_SETUP_FAIL'),
                                                                '--asetup must be used for the substep which requires --runInContainer')
 
        # Conditional MP based on runtime arguments
        if self._onlyMPWithRunargs:
            for k in self._onlyMPWithRunargs:
                if k in self.conf._argdict:
                    self._onlyMP = True
 
        # Check the consistency of parallel configuration: CLI flags + evnironment.
        if ((('multithreaded' in self.conf._argdict and self.conf._argdict['multithreaded'].value) or ('multiprocess' in self.conf._argdict and self.conf._argdict['multiprocess'].value)) and
            ('ATHENA_CORE_NUMBER' not in os.environ)):
            # At least one of the parallel command-line flags has been provided but ATHENA_CORE_NUMBER environment has not been set
            msg.warning('either --multithreaded or --multiprocess argument used but ATHENA_CORE_NUMBER environment not set. Athena will continue in Serial mode')
        else:
            # Try to detect AthenaMT mode, number of threads and number of concurrent events
            if not self._disableMT:
                self._athenaMT, self._athenaConcurrentEvents = detectAthenaMTThreads(self.conf.argdict, self.name, legacyThreadingRelease)
 
            # Try to detect AthenaMP mode and number of workers
            if not self._disableMP:
                self._athenaMP = detectAthenaMPProcs(self.conf.argdict, self.name, legacyThreadingRelease)
 
            # Check that we actually support MT
            if self._onlyMP and self._athenaMT > 0:
                msg.info("This configuration does not support MT, falling back to MP")
                if self._athenaMP == 0:
                    self._athenaMP = self._athenaMT
                self._athenaMT = 0
                self._athenaConcurrentEvents = 0
 
            # Check that we actually support MP
            if self._onlyMT and self._athenaMP > 0:
                    msg.info("This configuration does not support MP, using MT")
                    if self._athenaMT == 0:
                        self._athenaMT = self._athenaMP
                        self._athenaConcurrentEvents = self._athenaMP
                    self._athenaMP = 0
 
        # Small hack to detect cases where there are so few events that it's not worthwhile running in MP mode
        # which also avoids issues with zero sized files
        if not self._disableMP and expectedEvents < self._athenaMP:
            msg.info("Disabling AthenaMP as number of input events to process is too low ({0} events for {1} workers)".format(expectedEvents, self._athenaMP))
            self._disableMP = True
            self._athenaMP = 0
 
        # Handle executor steps
        if self.conf.totalExecutorSteps > 1:
            for dataType in output:
                if self.conf._dataDictionary[dataType].originalName:
                    self.conf._dataDictionary[dataType].value[0] = self.conf._dataDictionary[dataType].originalName
                else:
                    self.conf._dataDictionary[dataType].originalName = self.conf._dataDictionary[dataType].value[0]
                self.conf._dataDictionary[dataType].value[0] += "_{0}{1}".format(executorStepSuffix, self.conf.executorStep)
                msg.info("Updated athena output filename for {0} to {1}".format(dataType, self.conf._dataDictionary[dataType].value[0]))
 
        # And if this is (still) athenaMP, then set some options for workers and output file report
        if self._athenaMP > 0:
            self._athenaMPWorkerTopDir = 'athenaMP-workers-{0}-{1}'.format(self._name, self._substep)
            self._athenaMPFileReport = 'athenaMP-outputs-{0}-{1}'.format(self._name, self._substep)
            self._athenaMPEventOrdersFile = 'athenamp_eventorders.txt.{0}'.format(self._name)
            if 'athenaMPUseEventOrders' in self.conf.argdict and self.conf._argdict['athenaMPUseEventOrders'].value is True:
                self._athenaMPReadEventOrders = True
            else:
                self._athenaMPReadEventOrders = False          
            # Decide on scheduling
            if ('athenaMPStrategy' in self.conf.argdict and 
                (self.conf.argdict['athenaMPStrategy'].returnMyValue(name=self._name, substep=self._substep, first=self.conf.firstExecutor) is not None)):
                self._athenaMPStrategy = self.conf.argdict['athenaMPStrategy'].returnMyValue(name=self._name, substep=self._substep, first=self.conf.firstExecutor)
            else:
                self._athenaMPStrategy = 'SharedQueue'
 
            # See if we have options for the target output file size
            if 'athenaMPMergeTargetSize' in self.conf.argdict:
                for dataType in output:
                    if dataType in self.conf.argdict['athenaMPMergeTargetSize'].value:
                        self.conf._dataDictionary[dataType].mergeTargetSize = self.conf.argdict['athenaMPMergeTargetSize'].value[dataType] * 1000000 # Convert from MB to B
                        msg.info('Set target merge size for {0} to {1}'.format(dataType, self.conf._dataDictionary[dataType].mergeTargetSize))
                    else:
                        # Use a globbing strategy
                        matchedViaGlob = False
                        for mtsType, mtsSize in self.conf.argdict['athenaMPMergeTargetSize'].value.items():
                            if fnmatch(dataType, mtsType):
                                self.conf._dataDictionary[dataType].mergeTargetSize = mtsSize * 1000000 # Convert from MB to B
                                msg.info('Set target merge size for {0} to {1} from "{2}" glob'.format(dataType, self.conf._dataDictionary[dataType].mergeTargetSize, mtsType))
                                matchedViaGlob = True
                                break
                        if not matchedViaGlob and "ALL" in self.conf.argdict['athenaMPMergeTargetSize'].value:
                            self.conf._dataDictionary[dataType].mergeTargetSize = self.conf.argdict['athenaMPMergeTargetSize'].value["ALL"] * 1000000 # Convert from MB to B
                            msg.info('Set target merge size for {0} to {1} from "ALL" value'.format(dataType, self.conf._dataDictionary[dataType].mergeTargetSize))
         
            # For AthenaMP jobs we ensure that the athena outputs get the suffix _000
            # so that the mother process output file (if it exists) can be used directly
            # as soft linking can lead to problems in the PoolFileCatalog (see ATLASJT-317)
            for dataType in output:
                if self.conf.totalExecutorSteps <= 1:
                    self.conf._dataDictionary[dataType].originalName = self.conf._dataDictionary[dataType].value[0]
                if 'eventService' not in self.conf.argdict or 'eventService' in self.conf.argdict and self.conf.argdict['eventService'].value is False:
                    if 'sharedWriter' in self.conf.argdict and self.conf.argdict['sharedWriter'].value:
                        msg.info("SharedWriter: not updating athena output filename for {0}".format(dataType))
                    else:
                        self.conf._dataDictionary[dataType].value[0] += "_000"
                        msg.info("Updated athena output filename for {0} to {1}".format(dataType, self.conf._dataDictionary[dataType].value[0]))
        else:
            self._athenaMPWorkerTopDir = self._athenaMPFileReport = None
 
 
        
        if self._skeleton or self._skeletonCA:
            inputFiles = dict()
            for dataType in input:
                inputFiles[dataType] = self.conf.dataDictionary[dataType]
            outputFiles = dict()
            for dataType in output:
                outputFiles[dataType] = self.conf.dataDictionary[dataType]
 
            # See if we have any 'extra' file arguments
            nameForFiles = commonExecutorStepName(self._name)
            for dataType, dataArg in self.conf.dataDictionary.items():
                if isinstance(dataArg, list) and dataArg:
                    if self.conf.totalExecutorSteps <= 1:
                        raise ValueError('Multiple input arguments provided but only running one substep')
                    if self.conf.totalExecutorSteps != len(dataArg):
                        raise ValueError(f'{len(dataArg)} input arguments provided but running {self.conf.totalExecutorSteps} substeps')
 
                    if dataArg[self.conf.executorStep].io == 'input' and nameForFiles in dataArg[self.conf.executorStep].executor:
                        inputFiles[dataArg[self.conf.executorStep].subtype] = dataArg
                else:
                    if dataArg.io == 'input' and nameForFiles in dataArg.executor:
                        inputFiles[dataArg.subtype] = dataArg
 
            msg.debug('Input Files: {0}; Output Files: {1}'.format(inputFiles, outputFiles))
            
            # Get the list of top options files that will be passed to athena (=runargs file + all skeletons)
            self._topOptionsFiles = self._jobOptionsTemplate.getTopOptions(input = inputFiles, 
                                                                           output = outputFiles)
 
        
        if len(input) > 0:
            self._extraMetadata['inputs'] = list(input)
        if len(output) > 0:
            self._extraMetadata['outputs'] = list(output)
 
        
        dbrelease = dbsetup = None
        if 'DBRelease' in self.conf.argdict:
            dbrelease = self.conf.argdict['DBRelease'].returnMyValue(name=self._name, substep=self._substep, first=self.conf.firstExecutor)
            if path.islink(dbrelease):
                dbrelease = path.realpath(dbrelease)
            if dbrelease:
                # Classic tarball - filename format is DBRelease-X.Y.Z.tar.gz
                dbdMatch = re.match(r'DBRelease-([\d\.]+)\.tar\.gz', path.basename(dbrelease))
                if dbdMatch:
                    msg.debug('DBRelease setting {0} matches classic tarball file'.format(dbrelease))
                    if not os.access(dbrelease, os.R_OK):
                        msg.warning('Transform was given tarball DBRelease file {0}, but this is not there'.format(dbrelease))
                        msg.warning('I will now try to find DBRelease {0} in cvmfs'.format(dbdMatch.group(1)))
                        dbrelease = dbdMatch.group(1)
                        dbsetup = cvmfsDBReleaseCheck(dbrelease)
                    else:
                        # Check if the DBRelease is setup
                        msg.debug('Setting up {0} from {1}'.format(dbdMatch.group(1), dbrelease))
                        unpacked, dbsetup = unpackDBRelease(tarball=dbrelease, dbversion=dbdMatch.group(1))
                        if unpacked:
                            # Now run the setup.py script to customise the paths to the current location...
                            setupDBRelease(dbsetup)
                # For cvmfs we want just the X.Y.Z release string (and also support 'current')
                else:
                    dbsetup = cvmfsDBReleaseCheck(dbrelease)
        
        
        self._envUpdate = trfEnv.environmentUpdate()
        self._envUpdate.setStandardEnvironment(self.conf.argdict, name=self.name, substep=self.substep)
        self._prepAthenaCommandLine() 
        
                
        super(athenaExecutor, self).preExecute(input, output)
        
        # Now we always write a wrapper, because it's very convenient for re-running individual substeps
        # This will have asetup and/or DB release setups in it
        # Do this last in this preExecute as the _cmd needs to be finalised
        msg.info('Now writing wrapper for substep executor {0}'.format(self._name))
        self._writeAthenaWrapper(asetup=asetupString, dbsetup=dbsetup, ossetup=OSSetupString)
        msg.info('Athena will be executed in a subshell via {0}'.format(self._cmd))
        
                

preExeStartTimes()

def python.trfExe.transformExecutor.preExeStartTimes (   self )

inherited

Definition at line 331 of file trfExe.py.

    def preExeStartTimes(self):
        return self._preExeStart
    

preExeWallTime()

def python.trfExe.transformExecutor.preExeWallTime (   self )

inherited

Definition at line 358 of file trfExe.py.

    def preExeWallTime(self):
        if self._preExeStart and self._exeStart:
            return calcWallTime(self._preExeStart, self._exeStart)
        else:
            return None
 

rc()

def python.trfExe.transformExecutor.rc (   self )

inherited

Definition at line 299 of file trfExe.py.

    def rc(self):
        return self._rc
    

reSimEvent()

def python.trfExe.transformExecutor.reSimEvent (   self )

inherited

Definition at line 447 of file trfExe.py.

    def reSimEvent(self):
        return self._resimevents
 

setPreExeStart()

def python.trfExe.transformExecutor.setPreExeStart (   self )

inherited

Definition at line 460 of file trfExe.py.

    def setPreExeStart(self):
        if self._preExeStart is None:
            self._preExeStart = os.times()
            msg.debug('preExeStart time is {0}'.format(self._preExeStart))
 

setValStart()

def python.trfExe.transformExecutor.setValStart (   self )

inherited

Definition at line 465 of file trfExe.py.

    def setValStart(self):
        if self._valStart is None:
            self._valStart = os.times()
            msg.debug('valStart time is {0}'.format(self._valStart))
 

substep()

def python.trfExe.athenaExecutor.substep (   self )

inherited

Reimplemented from python.trfExe.transformExecutor.

Definition at line 934 of file trfExe.py.

    def substep(self):
        return self._substep
 

sysTime()

def python.trfExe.transformExecutor.sysTime (   self )

inherited

Definition at line 379 of file trfExe.py.

    def sysTime(self):
        if self._exeStart and self._exeStop:
            return self._exeStop[3] - self._exeStart[3]
        else:
            return None
 

trf() [1/2]

def python.trfExe.transformExecutor.trf (   self )

inherited

Definition at line 224 of file trfExe.py.

    def trf(self):
        if '_trf' in dir(self):
            return self._trf
        return None
    

trf() [2/2]

def python.trfExe.transformExecutor.trf (   self,   value  )

inherited

Definition at line 230 of file trfExe.py.

    def trf(self, value):
        self._trf = value
        

usrTime()

def python.trfExe.transformExecutor.usrTime (   self )

inherited

Definition at line 372 of file trfExe.py.

    def usrTime(self):
        if self._exeStart and self._exeStop:
            return self._exeStop[2] - self._exeStart[2]
        else:
            return None
        

validate()

def python.trfExe.optionalAthenaExecutor.validate

(

self

)

Reimplemented from python.trfExe.athenaExecutor.

Definition at line 1809 of file trfExe.py.

    def validate(self):
        self.setValStart()
        try:
            super(optionalAthenaExecutor, self).validate()
        except trfExceptions.TransformValidationException as e:
            # In this case we hold this exception until the logfile has been scanned
            msg.warning('Validation failed for {0}: {1}'.format(self._name, e))
            self._isValidated = False
            self._errMsg = e.errMsg
            self._rc = e.errCode
        self._valStop = os.times()
        msg.debug('valStop time is {0}'.format(self._valStop))
 
 

validation() [1/2]

def python.trfExe.transformExecutor.validation (   self )

inherited

Definition at line 307 of file trfExe.py.

    def validation(self):
        return self._validation
    

validation() [2/2]

def python.trfExe.transformExecutor.validation (   self,   value  )

inherited

Definition at line 311 of file trfExe.py.

    def validation(self, value):
        self._validation = value
        

validationCpuTime()

def python.trfExe.transformExecutor.validationCpuTime (   self )

inherited

Definition at line 415 of file trfExe.py.

    def validationCpuTime(self):
        if self._valStart and self._valStop:
            return calcCpuTime(self._valStart, self._valStop)
        else:
            return None
    

validationWallTime()

def python.trfExe.transformExecutor.validationWallTime (   self )

inherited

Definition at line 422 of file trfExe.py.

    def validationWallTime(self):
        if self._valStart and self._valStop:
            return calcWallTime(self._valStart, self._valStop)
        else:
            return None
 

valStartTimes()

def python.trfExe.transformExecutor.valStartTimes (   self )

inherited

Definition at line 343 of file trfExe.py.

    def valStartTimes(self):
        return self._valStart
 

valStopTimes()

def python.trfExe.transformExecutor.valStopTimes (   self )

inherited

Definition at line 347 of file trfExe.py.

    def valStopTimes(self):
        return self._valStop
    

wallTime()

def python.trfExe.transformExecutor.wallTime (   self )

inherited

Definition at line 386 of file trfExe.py.

    def wallTime(self):
        if self._exeStart and self._exeStop:
            return calcWallTime(self._exeStart, self._exeStop)
        else:
            return None
        

wallTimeTotal()

def python.trfExe.transformExecutor.wallTimeTotal (   self )

inherited

Definition at line 436 of file trfExe.py.

    def wallTimeTotal(self):
        if self._preExeStart and self._valStop:
            return calcWallTime(self._preExeStart, self._valStop)
        else:
            return None
        

Member Data Documentation

_alreadyInContainer

python.trfExe.athenaExecutor._alreadyInContainer

privateinherited

Definition at line 1550 of file trfExe.py.

_asetup

python.trfExe.athenaExecutor._asetup

privateinherited

Definition at line 1546 of file trfExe.py.

_athenaConcurrentEvents

python.trfExe.athenaExecutor._athenaConcurrentEvents

privateinherited

Definition at line 1063 of file trfExe.py.

_athenaMP

python.trfExe.athenaExecutor._athenaMP

privateinherited

Definition at line 1067 of file trfExe.py.

_athenaMPEventOrdersFile

python.trfExe.athenaExecutor._athenaMPEventOrdersFile

privateinherited

Definition at line 1106 of file trfExe.py.

_athenaMPFileReport

python.trfExe.athenaExecutor._athenaMPFileReport

privateinherited

Definition at line 1105 of file trfExe.py.

_athenaMPReadEventOrders

python.trfExe.athenaExecutor._athenaMPReadEventOrders

privateinherited

Definition at line 1108 of file trfExe.py.

_athenaMPStrategy

python.trfExe.athenaExecutor._athenaMPStrategy

privateinherited

Definition at line 1114 of file trfExe.py.

_athenaMPWorkerTopDir

python.trfExe.athenaExecutor._athenaMPWorkerTopDir

privateinherited

Definition at line 1104 of file trfExe.py.

_athenaMT

python.trfExe.athenaExecutor._athenaMT

privateinherited

Definition at line 1074 of file trfExe.py.

_cmd

python.trfExe.athenaExecutor._cmd

privateinherited

Definition at line 1439 of file trfExe.py.

_containerSetup

python.trfExe.athenaExecutor._containerSetup

privateinherited

Definition at line 1548 of file trfExe.py.

_dataArgs

python.trfExe.athenaExecutor._dataArgs

privateinherited

Definition at line 889 of file trfExe.py.

_dbMonitor

python.trfExe.athenaExecutor._dbMonitor

privateinherited

Definition at line 1364 of file trfExe.py.

_dbsetup

python.trfExe.athenaExecutor._dbsetup

privateinherited

Definition at line 1547 of file trfExe.py.

_defaultIgnorePatternFile

python.trfExe.athenaExecutor._defaultIgnorePatternFile

staticprivateinherited

Definition at line 842 of file trfExe.py.

_disableMP

python.trfExe.athenaExecutor._disableMP

privateinherited

Definition at line 893 of file trfExe.py.

_disableMT

python.trfExe.athenaExecutor._disableMT

privateinherited

Definition at line 892 of file trfExe.py.

_echologger

python.trfExe.scriptExecutor._echologger

privateinherited

Definition at line 691 of file trfExe.py.

_echoOutput

python.trfExe.scriptExecutor._echoOutput

privateinherited

Note

Query the environment for echo configuration Let the manual envars always win over auto-detected settings

Definition at line 631 of file trfExe.py.

_echostream

python.trfExe.scriptExecutor._echostream

privateinherited

Definition at line 701 of file trfExe.py.

_envUpdate

python.trfExe.athenaExecutor._envUpdate

privateinherited

Add input/output file information - this can't be done in init as we don't know what our inputs and outputs will be then.

DBRelease configuration Look for environment updates and perpare the athena command line

Definition at line 1218 of file trfExe.py.

_errMsg

python.trfExe.optionalAthenaExecutor._errMsg

private

Definition at line 1817 of file trfExe.py.

_errorMaskFiles

python.trfExe.athenaExecutor._errorMaskFiles

privateinherited

Definition at line 890 of file trfExe.py.

_eventCount

python.trfExe.scriptExecutor._eventCount

privateinherited

Check event counts (always do this by default) Do this here so that all script executors have this by default (covers most use cases with events)

Definition at line 832 of file trfExe.py.

_exe

python.trfExe.athenaExecutor._exe

privateinherited

Start building up the command line N.B.

it's possible we might have cases where 'athena' and 'athenaopt' should be substep args but at the moment this hasn't been requested.

Definition at line 1438 of file trfExe.py.

_exeArgs

python.trfExe.scriptExecutor._exeArgs

privateinherited

Definition at line 624 of file trfExe.py.

_exeLogFile

python.trfExe.scriptExecutor._exeLogFile

privateinherited

Definition at line 696 of file trfExe.py.

_exeStart

python.trfExe.scriptExecutor._exeStart

privateinherited

Definition at line 726 of file trfExe.py.

_exeStop

python.trfExe.scriptExecutor._exeStop

privateinherited

Definition at line 769 of file trfExe.py.

_exitMessageLimit

python.trfExe.athenaExecutor._exitMessageLimit

staticprivateinherited

Definition at line 841 of file trfExe.py.

_extraMetadata

python.trfExe.transformExecutor._extraMetadata

privateinherited

Definition at line 180 of file trfExe.py.

_extraRunargs

python.trfExe.athenaExecutor._extraRunargs

privateinherited

Definition at line 886 of file trfExe.py.

_hasExecuted

python.trfExe.scriptExecutor._hasExecuted

privateinherited

Definition at line 723 of file trfExe.py.

_hasValidated

python.trfExe.athenaExecutor._hasValidated

privateinherited

Definition at line 1312 of file trfExe.py.

_inData

python.trfExe.transformExecutor._inData

privateinherited

Definition at line 143 of file trfExe.py.

_input

python.trfExe.scriptExecutor._input

privateinherited

Definition at line 661 of file trfExe.py.

_inputDataTypeCountCheck

python.trfExe.athenaExecutor._inputDataTypeCountCheck

privateinherited

Definition at line 891 of file trfExe.py.

_inputEventTest

python.trfExe.athenaExecutor._inputEventTest

privateinherited

Definition at line 884 of file trfExe.py.

_isValidated

python.trfExe.optionalAthenaExecutor._isValidated

private

Definition at line 1816 of file trfExe.py.

_jobOptionsTemplate

python.trfExe.athenaExecutor._jobOptionsTemplate

privateinherited

Definition at line 917 of file trfExe.py.

_literalRunargs

python.trfExe.athenaExecutor._literalRunargs

privateinherited

Definition at line 888 of file trfExe.py.

_logFileName

python.trfExe.athenaExecutor._logFileName

privateinherited

Note

Update argFile values to have the correct outputs from the MP workers

Definition at line 1284 of file trfExe.py.

_logScan

python.trfExe.athenaExecutor._logScan

privateinherited

Definition at line 1361 of file trfExe.py.

_memFullFile

python.trfExe.scriptExecutor._memFullFile

privateinherited

Definition at line 749 of file trfExe.py.

_memLeakResult

python.trfExe.athenaExecutor._memLeakResult

privateinherited

Our parent will check the RC for us.

Get results of memory monitor analysis (slope and chi2) the analysis is a linear fit to 'pss' va 'Time' (fit to at least 5 data points) to obtain a good fit, tails are excluded from data if the slope of 'pss' is higher than 'memLeakThreshold' and an error is already caught, a message will be added to the exit message the memory leak threshold is defined based on analysing several jobs with memory leak, however it is rather arbitrary and could be modified

Definition at line 1334 of file trfExe.py.

_memMonitor

python.trfExe.scriptExecutor._memMonitor

privateinherited

Definition at line 636 of file trfExe.py.

_memStats

python.trfExe.scriptExecutor._memStats

privateinherited

Definition at line 793 of file trfExe.py.

_memSummaryFile

python.trfExe.scriptExecutor._memSummaryFile

privateinherited

Definition at line 748 of file trfExe.py.

_myMerger

python.trfExe.transformExecutor._myMerger

privateinherited

Definition at line 201 of file trfExe.py.

_name

python.trfExe.transformExecutor._name

privateinherited

Definition at line 140 of file trfExe.py.

_onlyMP

python.trfExe.athenaExecutor._onlyMP

privateinherited

Do we need to run asetup first? For a certain substep (e.g.

trigger), might need to run in a container using a legacy release Container run will only work if asetup is executed

Definition at line 894 of file trfExe.py.

_onlyMPWithRunargs

python.trfExe.athenaExecutor._onlyMPWithRunargs

privateinherited

Definition at line 896 of file trfExe.py.

_onlyMT

python.trfExe.athenaExecutor._onlyMT

privateinherited

Definition at line 895 of file trfExe.py.

_originalCmd

python.trfExe.athenaExecutor._originalCmd

privateinherited

Definition at line 1545 of file trfExe.py.

_outData

python.trfExe.transformExecutor._outData

privateinherited

Definition at line 144 of file trfExe.py.

_output

python.trfExe.scriptExecutor._output

privateinherited

Definition at line 662 of file trfExe.py.

_perfMonFile

python.trfExe.athenaExecutor._perfMonFile

privateinherited

Definition at line 900 of file trfExe.py.

_preExeStart

python.trfExe.transformExecutor._preExeStart

privateinherited

Note

Place holders for resource consumption. CPU and walltime are available for all executors but currently only athena is instrumented to fill in memory stats (and then only if PerfMonSD is enabled).

Definition at line 185 of file trfExe.py.

_rc

python.trfExe.optionalAthenaExecutor._rc

private

Definition at line 1818 of file trfExe.py.

_resimevents

python.trfExe.athenaExecutor._resimevents

privateinherited

Definition at line 1286 of file trfExe.py.

_runtimeRunargs

python.trfExe.athenaExecutor._runtimeRunargs

privateinherited

Definition at line 887 of file trfExe.py.

_setupFile

python.trfExe.athenaExecutor._setupFile

privateinherited

Definition at line 1552 of file trfExe.py.

_skeleton

python.trfExe.athenaExecutor._skeleton

privateinherited

Definition at line 905 of file trfExe.py.

_skeletonCA

python.trfExe.athenaExecutor._skeletonCA

privateinherited

Definition at line 897 of file trfExe.py.

_substep

python.trfExe.athenaExecutor._substep

privateinherited

Definition at line 883 of file trfExe.py.

_topOptionsFiles

python.trfExe.athenaExecutor._topOptionsFiles

privateinherited

Write the skeleton file and prep athena.

Definition at line 1180 of file trfExe.py.

_trf

python.trfExe.transformExecutor._trf

privateinherited

Definition at line 231 of file trfExe.py.

_tryDropAndReload

python.trfExe.athenaExecutor._tryDropAndReload

privateinherited

Definition at line 885 of file trfExe.py.

_validation

python.trfExe.transformExecutor._validation

privateinherited

Definition at line 312 of file trfExe.py.

_valStart

python.trfExe.scriptExecutor._valStart

privateinherited

Definition at line 802 of file trfExe.py.

_valStop

python.trfExe.optionalAthenaExecutor._valStop

private

Definition at line 1819 of file trfExe.py.

_workdir

python.trfExe.athenaExecutor._workdir

privateinherited

Definition at line 1549 of file trfExe.py.

_wrapperFile

python.trfExe.athenaExecutor._wrapperFile

privateinherited

Definition at line 1551 of file trfExe.py.

conf

python.trfExe.transformExecutor.conf

inherited

Executor configuration:

Note

that if conf and trf are None then we'll probably set the conf up later (this is allowed and expected to be done once the master transform has figured out what it's doing for this job)

Definition at line 162 of file trfExe.py.

inData

python.trfExe.transformExecutor.inData

inherited

Note

Protect against _inData not yet being defined

Use normal setter

Definition at line 250 of file trfExe.py.

name

python.trfExe.athenaExecutor.name

inherited

Definition at line 1284 of file trfExe.py.

outData

python.trfExe.transformExecutor.outData

inherited

Note

Protect against _outData not yet being defined

Use normal setter

Definition at line 270 of file trfExe.py.

---

The documentation for this class was generated from the following file:

• trfExe.py