ATLAS Offline Software
Loading...
Searching...
No Matches
python.trfExe.reductionFrameworkExecutor Class Reference

Specialist executor to manage the handling of multiple implicit input and output files within the derivation framework. More...

Inheritance diagram for python.trfExe.reductionFrameworkExecutor:
Collaboration diagram for python.trfExe.reductionFrameworkExecutor:

Public Member Functions

 preExecute (self, input=set(), output=set())
 Take inputDAODFile and setup the actual outputs needed in this job.
 substep (self)
 inputDataTypeCountCheck (self)
 inputDataTypeCountCheck (self, value)
 disableMP (self)
 disableMP (self, value)
 disableMT (self)
 disableMT (self, value)
 onlyMP (self)
 onlyMP (self, value)
 onlyMT (self)
 onlyMT (self, value)
 skeletonCA (self)
 postExecute (self)
 validate (self)
 exeArgs (self)
 exeArgs (self, value)
 exe (self)
 exe (self, value)
 execute (self)
 inData (self)
 inData (self, value)
 outData (self)
 outData (self, value)
 myMerger (self)
 Now define properties for these data members.
 name (self)
 name (self, value)
 trf (self)
 trf (self, value)
 inDataUpdate (self, value)
 outDataUpdate (self, value)
 input (self)
 output (self)
 extraMetadata (self)
 hasExecuted (self)
 rc (self)
 errMsg (self)
 validation (self)
 validation (self, value)
 hasValidated (self)
 isValidated (self)
 first (self)
 preExeStartTimes (self)
 exeStartTimes (self)
 exeStopTimes (self)
 valStartTimes (self)
 valStopTimes (self)
 preExeCpuTime (self)
 preExeWallTime (self)
 cpuTime (self)
 usrTime (self)
 sysTime (self)
 wallTime (self)
 memStats (self)
 memAnalysis (self)
 postExeCpuTime (self)
 postExeWallTime (self)
 validationCpuTime (self)
 validationWallTime (self)
 cpuTimeTotal (self)
 wallTimeTotal (self)
 eventCount (self)
 reSimEvent (self)
 athenaMP (self)
 dbMonitor (self)
 setPreExeStart (self)
 setValStart (self)
 doAll (self, input=set(), output=set())
 Convenience function.

Public Attributes

 exeArgs
 conf = conf
 Executor configuration:
 inData = value
 outData = value

Protected Member Functions

 _isCAEnabled (self)
 Check if running with CA.
 _prepAthenaCommandLine (self)
 Prepare the correct command line to be used to invoke athena.
 _writeAthenaWrapper (self, asetup=None, dbsetup=None, ossetup=None)
 Write a wrapper script which runs asetup and then Athena.
 _smartMerge (self, fileArg)
 Manage smart merging of output files.
 _targzipJiveXML (self)
 _buildStandardCommand (self)

Protected Attributes

 _substep = forceToAlphaNum(substep)
 _inputEventTest = inputEventTest
 _tryDropAndReload = tryDropAndReload
 Add –drop-and-reload if possible (and allowed!).
 _extraRunargs = extraRunargs
 _runtimeRunargs = runtimeRunargs
 _literalRunargs = literalRunargs
 _dataArgs = dataArgs
 _errorMaskFiles = errorMaskFiles
 _inputDataTypeCountCheck = inputDataTypeCountCheck
 _disableMT = disableMT
bool _disableMP = disableMP
bool _onlyMP = onlyMP
 _onlyMT = onlyMT
 _onlyMPWithRunargs = onlyMPWithRunargs
 _skeletonCA = skeletonCA
 Handle MPI setup.
 _perfMonFile = None
list _skeleton = [skeletonFile]
 _jobOptionsTemplate = JobOptionsTemplate(exe = self, version = '$Id: trfExe.py 792052 2017-01-13 13:36:51Z mavogel $')
str _athenaMPWorkerTopDir = 'athenaMP-workers-{0}-{1}'.format(self._name, self._substep)
str _athenaMPFileReport = 'athenaMP-outputs-{0}-{1}'.format(self._name, self._substep)
str _athenaMPEventOrdersFile = 'athenamp_eventorders.txt.{0}'.format(self._name)
bool _athenaMPReadEventOrders = True
str _athenaMPStrategy = self.conf.argdict['athenaMPStrategy'].returnMyValue(name=self._name, substep=self._substep, first=self.conf.firstExecutor)
 _topOptionsFiles
 _envUpdate = trfEnv.environmentUpdate()
 Look for environment updates and perpare the athena command line.
 _logScan
list _originalCmd = self._cmd
 _asetup = asetup
 _dbsetup = dbsetup
str _wrapperFile = 'runwrapper.{name}.sh'.format(name = self._name)
str _setupFile = 'setup.{name}.sh'.format(name = self._name)
 _exe = exe
 _exeArgs = exeArgs
bool _echoOutput = False
list _cmd = None
bool _memMonitor = memMonitor
 _input = input
 _output = output
str _logFileName = "log.{0}".format(self._name)
 _echologger = logging.getLogger(self._name)
 _exeLogFile = logging.FileHandler(self._logFileName, mode='w', **encargs)
 _echostream = logging.StreamHandler(_encoding_stream(sys.stdout))
 _workdir
str _memSummaryFile = 'prmon.summary.' + self._name + '.json'
 _name = forceToAlphaNum(name)
 _inData = set(inData)
 _outData = set(outData)
bool _hasExecuted = False
int _rc = -1
str _errMsg = None
bool _hasValidated = False
bool _isValidated = False
dict _extraMetadata = {}
 _preExeStart = None
 _exeStart = None
 _valStart = None
dict _memStats = {}
dict _memLeakResult = {}
 _memFullFile = None
 _eventCount = None
int _athenaMP = 0
int _athenaMT = 0
int _athenaConcurrentEvents = 0
 _dbMonitor = None
 _resimevents = None
 _alreadyInContainer = None
 _containerSetup = None
list _myMerger = []
 _trf = value
 _validation = value
 _exeStop
 _valStop

Static Protected Attributes

int _exitMessageLimit = 200
list _defaultIgnorePatternFile = ['atlas_error_mask.db']

Detailed Description

Specialist executor to manage the handling of multiple implicit input and output files within the derivation framework.

Definition at line 1901 of file trfExe.py.

Member Function Documentation

◆ _buildStandardCommand()

python.trfExe.scriptExecutor._buildStandardCommand ( self)
protectedinherited

Definition at line 706 of file trfExe.py.

706 def _buildStandardCommand(self):
707 if self._exe:
708 self._cmd = [self.exe, ]
709 else:
710 raise trfExceptions.TransformExecutionException(trfExit.nameToCode('TRF_EXEC_SETUP_FAIL'),
711 'No executor set in {0}'.format(self.__class__.__name__))
712 for arg in self.exeArgs:
713 if arg in self.conf.argdict:
714 # If we have a list then add each element to our list, else just str() the argument value
715 # Note if there are arguments which need more complex transformations then
716 # consider introducing a special toExeArg() method.
717 if isinstance(self.conf.argdict[arg].value, list):
718 self._cmd.extend([ str(v) for v in self.conf.argdict[arg].value])
719 else:
720 self._cmd.append(str(self.conf.argdict[arg].value))
721
722

◆ _isCAEnabled()

python.trfExe.athenaExecutor._isCAEnabled ( self)
protectedinherited

Check if running with CA.

Definition at line 1440 of file trfExe.py.

1440 def _isCAEnabled(self):
1441 # CA is now enabled by default, except for Gen_tf.py
1442 if self._name != 'generate' and self._name != 'afterburn':
1443 if 'CA' in self.conf.argdict and (self.conf.argdict['CA'] is False or self.conf.argdict['CA'].returnMyValue(name=self.name, substep=self.substep) is False):
1444 return False
1445 else:
1446 return True
1447 # the old logic is kept for Gen_tf.py, to be dropped once migrated to CA
1448 else:
1449 # CA not present
1450 if 'CA' not in self.conf.argdict:
1451 # If there is no legacy skeleton, then we are running with CA
1452 if not self._skeleton:
1453 return True
1454 else:
1455 return False
1456
1457 # CA present but None, all substeps running with CA
1458 if self.conf.argdict['CA'] is None:
1459 return True
1460
1461 # CA enabled for a substep, running with CA
1462 if self.conf.argdict['CA'].returnMyValue(name=self.name, substep=self.substep) is True:
1463 return True
1464
1465 return False
1466

◆ _prepAthenaCommandLine()

python.trfExe.athenaExecutor._prepAthenaCommandLine ( self)
protectedinherited

Prepare the correct command line to be used to invoke athena.

Definition at line 1468 of file trfExe.py.

1468 def _prepAthenaCommandLine(self):
1469
1472 if 'athena' in self.conf.argdict:
1473 self._exe = self.conf.argdict['athena'].value
1474 self._cmd = [self._exe]
1475
1476 # Find options for the current substep. Name is prioritised (e.g. RAWtoALL) over alias (e.g. r2a). Last look for 'all'
1477 currentSubstep = None
1478 if 'athenaopts' in self.conf.argdict:
1479 currentName = commonExecutorStepName(self.name)
1480 if currentName in self.conf.argdict['athenaopts'].value:
1481 currentSubstep = currentName
1482 if self.substep in self.conf.argdict['athenaopts'].value:
1483 msg.info('Athenaopts found for {0} and {1}, joining options. '
1484 'Consider changing your configuration to use just the name or the alias of the substep.'
1485 .format(currentSubstep, self.substep))
1486 self.conf.argdict['athenaopts'].value[currentSubstep].extend(self.conf.argdict['athenaopts'].value[self.substep])
1487 del self.conf.argdict['athenaopts'].value[self.substep]
1488 msg.debug('Athenaopts: {0}'.format(self.conf.argdict['athenaopts'].value))
1489 elif self.substep in self.conf.argdict['athenaopts'].value:
1490 currentSubstep = self.substep
1491 elif 'all' in self.conf.argdict['athenaopts'].value:
1492 currentSubstep = 'all'
1493
1494 # See if there's a preloadlibs and a request to update LD_PRELOAD for athena
1495 preLoadUpdated = dict()
1496 if 'LD_PRELOAD' in self._envUpdate._envdict:
1497 preLoadUpdated[currentSubstep] = False
1498 if 'athenaopts' in self.conf.argdict:
1499 if currentSubstep is not None:
1500 for athArg in self.conf.argdict['athenaopts'].value[currentSubstep]:
1501 # This code is pretty ugly as the athenaopts argument contains
1502 # strings which are really key/value pairs
1503 if athArg.startswith('--preloadlib'):
1504 try:
1505 i = self.conf.argdict['athenaopts'].value[currentSubstep].index(athArg)
1506 v = athArg.split('=', 1)[1]
1507 msg.info('Updating athena --preloadlib option for substep {1} with: {0}'.format(self._envUpdate.value('LD_PRELOAD'), self.name))
1508 newPreloads = ":".join(set(v.split(":")) | set(self._envUpdate.value('LD_PRELOAD').split(":")))
1509 self.conf.argdict['athenaopts']._value[currentSubstep][i] = '--preloadlib={0}'.format(newPreloads)
1510 except Exception as e:
1511 msg.warning('Failed to interpret athena option: {0} ({1})'.format(athArg, e))
1512 preLoadUpdated[currentSubstep] = True
1513 break
1514 if not preLoadUpdated[currentSubstep]:
1515 msg.info('Setting athena preloadlibs for substep {1} to: {0}'.format(self._envUpdate.value('LD_PRELOAD'), self.name))
1516 if 'athenaopts' in self.conf.argdict:
1517 if currentSubstep is not None:
1518 self.conf.argdict['athenaopts'].value[currentSubstep].append("--preloadlib={0}".format(self._envUpdate.value('LD_PRELOAD')))
1519 else:
1520 self.conf.argdict['athenaopts'].value['all'] = ["--preloadlib={0}".format(self._envUpdate.value('LD_PRELOAD'))]
1521 else:
1522 self.conf.argdict['athenaopts'] = trfArgClasses.argSubstepList(["--preloadlib={0}".format(self._envUpdate.value('LD_PRELOAD'))])
1523
1524 # Now update command line with the options we have (including any changes to preload)
1525 if 'athenaopts' in self.conf.argdict:
1526 if currentSubstep is None and "all" in self.conf.argdict['athenaopts'].value:
1527 self._cmd.extend(self.conf.argdict['athenaopts'].value['all'])
1528 elif currentSubstep in self.conf.argdict['athenaopts'].value:
1529 self._cmd.extend(self.conf.argdict['athenaopts'].value[currentSubstep])
1530
1531 if currentSubstep is None:
1532 currentSubstep = 'all'
1533
1534 if self._tryDropAndReload:
1535 if self._isCAEnabled():
1536 msg.info('ignoring "--drop-and-reload" for CA-based transforms, config cleaned up anyway')
1537 elif 'valgrind' in self.conf._argdict and self.conf._argdict['valgrind'].value is True:
1538 msg.info('Disabling "--drop-and-reload" because the job is configured to use Valgrind')
1539 elif 'athenaopts' in self.conf.argdict:
1540 athenaConfigRelatedOpts = ['--config-only','--drop-and-reload']
1541 # Note for athena options we split on '=' so that we properly get the option and not the whole "--option=value" string
1542 if currentSubstep in self.conf.argdict['athenaopts'].value:
1543 conflictOpts = set(athenaConfigRelatedOpts).intersection(set([opt.split('=')[0] for opt in self.conf.argdict['athenaopts'].value[currentSubstep]]))
1544 if len(conflictOpts) > 0:
1545 msg.info('Not appending "--drop-and-reload" to athena command line because these options conflict: {0}'.format(list(conflictOpts)))
1546 else:
1547 msg.info('Appending "--drop-and-reload" to athena options')
1548 self._cmd.append('--drop-and-reload')
1549 else:
1550 msg.info('No Athenaopts for substep {0}, appending "--drop-and-reload" to athena options'.format(self.name))
1551 self._cmd.append('--drop-and-reload')
1552 else:
1553 # This is the 'standard' case - so drop and reload should be ok
1554 msg.info('Appending "--drop-and-reload" to athena options')
1555 self._cmd.append('--drop-and-reload')
1556 else:
1557 msg.info('Skipping test for "--drop-and-reload" in this executor')
1558
1559 if not self._isCAEnabled(): #For CA-jobs, threads and nproc set in runargs file
1560 # For AthenaMT apply --threads=N if threads have been configured via ATHENA_CORE_NUMBER + multithreaded
1561 if self._athenaMT > 0 and not self._disableMT:
1562 if not ('athenaopts' in self.conf.argdict and
1563 any('--threads' in opt for opt in self.conf.argdict['athenaopts'].value[currentSubstep])):
1564 self._cmd.append('--threads=%s' % str(self._athenaMT))
1565
1566 # For AthenaMP apply --nprocs=N if threads have been configured via ATHENA_CORE_NUMBER + multiprocess
1567 if self._athenaMP > 0 and not self._disableMP:
1568 if not ('athenaopts' in self.conf.argdict and
1569 any('--nprocs' in opt for opt in self.conf.argdict['athenaopts'].value[currentSubstep])):
1570 self._cmd.append('--nprocs=%s' % str(self._athenaMP))
1571
1572 # Add topoptions
1573 # Note that _writeAthenaWrapper removes this from the end of _cmd when preparing the options for VTuneCommand, so assumes it comes last.
1574 if self._skeleton or self._skeletonCA:
1575 self._cmd += self._topOptionsFiles
1576 msg.info('Updated script arguments with topoptions: %s', self._cmd)
1577
1578
STL class.
std::vector< std::string > intersection(std::vector< std::string > &v1, std::vector< std::string > &v2)
std::vector< std::string > split(const std::string &s, const std::string &t=":")
Definition hcg.cxx:179
Definition index.py:1

◆ _smartMerge()

python.trfExe.athenaExecutor._smartMerge ( self,
fileArg )
protectedinherited

Manage smart merging of output files.

Parameters
fileArgFile argument to merge

Definition at line 1762 of file trfExe.py.

1762 def _smartMerge(self, fileArg):
1763
1764 if 'selfMerge' not in dir(fileArg):
1765 msg.info('Files in {0} cannot merged (no selfMerge() method is implemented)'.format(fileArg.name))
1766 return
1767
1768 if fileArg.mergeTargetSize == 0:
1769 msg.info('Files in {0} will not be merged as target size is set to 0'.format(fileArg.name))
1770 return
1771
1772
1773 mergeCandidates = [list()]
1774 currentMergeSize = 0
1775 for fname in fileArg.value:
1776 size = fileArg.getSingleMetadata(fname, 'file_size')
1777 if not isinstance(size, int):
1778 msg.warning('File size metadata for {0} was not correct, found type {1}. Aborting merge attempts.'.format(fileArg, type(size)))
1779 return
1780 # if there is no file in the job, then we must add it
1781 if len(mergeCandidates[-1]) == 0:
1782 msg.debug('Adding file {0} to current empty merge list'.format(fname))
1783 mergeCandidates[-1].append(fname)
1784 currentMergeSize += size
1785 continue
1786 # see if adding this file gets us closer to the target size (but always add if target size is negative)
1787 if fileArg.mergeTargetSize < 0 or math.fabs(currentMergeSize + size - fileArg.mergeTargetSize) < math.fabs(currentMergeSize - fileArg.mergeTargetSize):
1788 msg.debug('Adding file {0} to merge list {1} as it gets closer to the target size'.format(fname, mergeCandidates[-1]))
1789 mergeCandidates[-1].append(fname)
1790 currentMergeSize += size
1791 continue
1792 # close this merge list and start a new one
1793 msg.debug('Starting a new merge list with file {0}'.format(fname))
1794 mergeCandidates.append([fname])
1795 currentMergeSize = size
1796
1797 msg.debug('First pass splitting will merge files in this way: {0}'.format(mergeCandidates))
1798
1799 if len(mergeCandidates) == 1:
1800 # Merging to a single file, so use the original filename that the transform
1801 # was started with
1802 mergeNames = [fileArg.originalName]
1803 else:
1804 # Multiple merge targets, so we need a set of unique names
1805 counter = 0
1806 mergeNames = []
1807 for mergeGroup in mergeCandidates:
1808 # Note that the individual worker files get numbered with 3 digit padding,
1809 # so these non-padded merges should be fine
1810 mergeName = fileArg.originalName + '_{0}'.format(counter)
1811 while path.exists(mergeName):
1812 counter += 1
1813 mergeName = fileArg.originalName + '_{0}'.format(counter)
1814 mergeNames.append(mergeName)
1815 counter += 1
1816 # Now actually do the merges
1817 for targetName, mergeGroup, counter in zip(mergeNames, mergeCandidates, list(range(len(mergeNames)))):
1818 msg.info('Want to merge files {0} to {1}'.format(mergeGroup, targetName))
1819 if len(mergeGroup) <= 1:
1820 msg.info('Skip merging for single file')
1821 else:
1822
1823 self._myMerger.append(fileArg.selfMerge(output=targetName, inputs=mergeGroup, counter=counter, argdict=self.conf.argdict))
1824
1825

◆ _targzipJiveXML()

python.trfExe.athenaExecutor._targzipJiveXML ( self)
protectedinherited

Definition at line 1826 of file trfExe.py.

1826 def _targzipJiveXML(self):
1827 #tgzipping JiveXML files
1828 targetTGZName = self.conf.dataDictionary['TXT_JIVEXMLTGZ'].value[0]
1829 if os.path.exists(targetTGZName):
1830 os.remove(targetTGZName)
1831
1832 import tarfile
1833 fNameRE = re.compile(r"JiveXML\_\d+\_\d+.xml")
1834
1835 # force gz compression
1836 tar = tarfile.open(targetTGZName, "w:gz")
1837 for fName in os.listdir('.'):
1838 matches = fNameRE.findall(fName)
1839 if len(matches) > 0:
1840 if fNameRE.findall(fName)[0] == fName:
1841 msg.info('adding %s to %s', fName, targetTGZName)
1842 tar.add(fName)
1843
1844 tar.close()
1845 msg.info('JiveXML compression: %s has been written and closed.', targetTGZName)
1846
1847

◆ _writeAthenaWrapper()

python.trfExe.athenaExecutor._writeAthenaWrapper ( self,
asetup = None,
dbsetup = None,
ossetup = None )
protectedinherited

Write a wrapper script which runs asetup and then Athena.

Definition at line 1580 of file trfExe.py.

1585 ):
1586 self._originalCmd = self._cmd
1587 self._asetup = asetup
1588 self._dbsetup = dbsetup
1589 self._containerSetup = ossetup
1590 self._workdir = os.getcwd()
1591 self._alreadyInContainer = self._workdir.startswith("/srv")
1592 self._wrapperFile = 'runwrapper.{name}.sh'.format(name = self._name)
1593 self._setupFile = 'setup.{name}.sh'.format(name = self._name)
1594
1595 # Create a setupATLAS script
1596 setupATLAS = 'my_setupATLAS.sh'
1597 with open(setupATLAS, 'w') as f:
1598 print("#!/bin/bash", file=f)
1599 print("""
1600if [ -z $ATLAS_LOCAL_ROOT_BASE ]; then
1601 export ATLAS_LOCAL_ROOT_BASE=/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase
1602fi
1603source ${ATLAS_LOCAL_ROOT_BASE}/user/atlasLocalSetup.sh"""
1604 , file=f)
1605 os.chmod(setupATLAS, 0o755)
1606
1607 msg.debug(
1608 'Preparing wrapper file {wrapperFileName} with '
1609 'asetup={asetupStatus} and dbsetup={dbsetupStatus}'.format(
1610 wrapperFileName = self._wrapperFile,
1611 asetupStatus = self._asetup,
1612 dbsetupStatus = self._dbsetup
1613 )
1614 )
1615
1616 container_cmd = None
1617 try:
1618 with open(self._wrapperFile, 'w') as wrapper:
1619 print('#!/bin/sh', file=wrapper)
1620 if self._containerSetup is not None:
1621 container_cmd = [ os.path.abspath(setupATLAS),
1622 "-c",
1623 self._containerSetup,
1624 "--pwd",
1625 self._workdir,
1626 "-s",
1627 os.path.join('.', self._setupFile),
1628 "-r"]
1629 print('echo "This wrapper is executed within a container! For a local re-run, do:"', file=wrapper)
1630 print('echo " '+ " ".join(['setupATLAS'] + container_cmd[1:] + [path.join('.', self._wrapperFile)]) + '"', file=wrapper)
1631 print('echo "N.B.: if launching a nested container, navigate to /srv before running the above command"',
1632 file = wrapper)
1633 print('echo " and use --pwd workdir, where workdir is the transform running directory within /srv"',
1634 file=wrapper)
1635 print('echo', file=wrapper)
1636
1637 if asetup:
1638 wfile = wrapper
1639 asetupFile = None
1640 # if the substep is executed within a container, setup athena with a separate script
1641 # e.g. setupATLAS -c el9 -s setup.RDOtoRDOTrigger.sh -r runwrapper.RDOtoRDOTrigger.sh ...
1642 if self._containerSetup is not None:
1643 asetupFile = open(self._setupFile, 'w')
1644 wfile = asetupFile
1645 print(f'source ./{setupATLAS} -q', file=wfile)
1646 print(f'asetup {asetup}', file=wfile)
1647 print('if [ ${?} != "0" ]; then exit 255; fi', file=wfile)
1648 if dbsetup:
1649 dbroot = path.dirname(dbsetup)
1650 dbversion = path.basename(dbroot)
1651 print("# DBRelease setup", file=wrapper)
1652 print('echo Setting up DBRelease {dbroot} environment'.format(dbroot = dbroot), file=wrapper)
1653 print('export DBRELEASE={dbversion}'.format(dbversion = dbversion), file=wrapper)
1654 print('export CORAL_AUTH_PATH={directory}'.format(directory = path.join(dbroot, 'XMLConfig')), file=wrapper)
1655 print('export CORAL_DBLOOKUP_PATH={directory}'.format(directory = path.join(dbroot, 'XMLConfig')), file=wrapper)
1656 print('export TNS_ADMIN={directory}'.format(directory = path.join(dbroot, 'oracle-admin')), file=wrapper)
1657 print('DATAPATH={dbroot}:$DATAPATH'.format(dbroot = dbroot), file=wrapper)
1658 if self._disableMT:
1659 print("# AthenaMT explicitly disabled for this executor", file=wrapper)
1660 if self._disableMP:
1661 print("# AthenaMP explicitly disabled for this executor", file=wrapper)
1662 if self._envUpdate.len > 0:
1663 for envSetting in self._envUpdate.values:
1664 if not envSetting.startswith('LD_PRELOAD'):
1665 print("export", envSetting, file=wrapper)
1666 # If Valgrind is engaged, a serialised Athena configuration file
1667 # is generated for use with a subsequent run of Athena with
1668 # Valgrind.
1669 if 'valgrind' in self.conf._argdict and self.conf._argdict['valgrind'].value is True:
1670 msg.info('Valgrind engaged')
1671 # Define the file name of the serialised Athena
1672 # configuration.
1673 AthenaSerialisedConfigurationFile = "{name}Conf.pkl".format(
1674 name = self._name
1675 )
1676 # Run Athena for generation of its serialised configuration.
1677 print(' '.join(self._cmd), "--config-only={0}".format(AthenaSerialisedConfigurationFile), file=wrapper)
1678 print('if [ $? != "0" ]; then exit 255; fi', file=wrapper)
1679 # Generate a Valgrind command, suppressing or ussing default
1680 # options as requested and extra options as requested.
1681 if 'valgrindDefaultOpts' in self.conf._argdict:
1682 defaultOptions = self.conf._argdict['valgrindDefaultOpts'].value
1683 else:
1684 defaultOptions = True
1685 if 'valgrindExtraOpts' in self.conf._argdict:
1686 extraOptionsList = self.conf._argdict['valgrindExtraOpts'].value
1687 else:
1688 extraOptionsList = None
1689 msg.debug("requested Valgrind command basic options: {options}".format(options = defaultOptions))
1690 msg.debug("requested Valgrind command extra options: {options}".format(options = extraOptionsList))
1691 command = ValgrindCommand(
1692 defaultOptions = defaultOptions,
1693 extraOptionsList = extraOptionsList,
1694 AthenaSerialisedConfigurationFile = \
1695 AthenaSerialisedConfigurationFile
1696 )
1697 msg.debug("Valgrind command: {command}".format(command = command))
1698 print(command, file=wrapper)
1699 # If VTune is engaged, a serialised Athena configuration file
1700 # is generated for use with a subsequent run of Athena with
1701 # VTune.
1702 elif 'vtune' in self.conf._argdict and self.conf._argdict['vtune'].value is True:
1703 msg.info('VTune engaged')
1704 # Define the file name of the serialised Athena
1705 # configuration.
1706 AthenaSerialisedConfigurationFile = "{name}Conf.pkl".format(
1707 name = self._name
1708 )
1709 # Run Athena for generation of its serialised configuration.
1710 print(' '.join(self._cmd), "--config-only={0}".format(AthenaSerialisedConfigurationFile), file=wrapper)
1711 print('if [ $? != "0" ]; then exit 255; fi', file=wrapper)
1712 # Generate a VTune command, suppressing or ussing default
1713 # options as requested and extra options as requested.
1714 if 'vtuneDefaultOpts' in self.conf._argdict:
1715 defaultOptions = self.conf._argdict['vtuneDefaultOpts'].value
1716 else:
1717 defaultOptions = True
1718 if 'vtuneExtraOpts' in self.conf._argdict:
1719 extraOptionsList = self.conf._argdict['vtuneExtraOpts'].value
1720 else:
1721 extraOptionsList = None
1722
1723 # replace the _topOptionsFiles from the Athena command with the AthenaSerialisedConfigurationFile.
1724 if (self._skeleton or self._skeletonCA) and len(self._topOptionsFiles) > 0:
1725 AthenaCommand = self._cmd[:-len(self._topOptionsFiles)]
1726 else:
1727 AthenaCommand = self._cmd
1728 AthenaCommand.append(AthenaSerialisedConfigurationFile)
1729
1730 msg.debug("requested VTune command basic options: {options}".format(options = defaultOptions))
1731 msg.debug("requested VTune command extra options: {options}".format(options = extraOptionsList))
1732 command = VTuneCommand(
1733 defaultOptions = defaultOptions,
1734 extraOptionsList = extraOptionsList,
1735 AthenaCommand = AthenaCommand
1736 )
1737 msg.debug("VTune command: {command}".format(command = command))
1738 print(command, file=wrapper)
1739 else:
1740 msg.info('Valgrind/VTune not engaged')
1741 # run Athena command
1742 print(' '.join(self._cmd), file=wrapper)
1743 os.chmod(self._wrapperFile, 0o755)
1744 except OSError as e:
1745 errMsg = 'error writing athena wrapper {fileName}: {error}'.format(
1746 fileName = self._wrapperFile,
1747 error = e
1748 )
1749 msg.error(errMsg)
1750 raise trfExceptions.TransformExecutionException(
1751 trfExit.nameToCode('TRF_EXEC_SETUP_WRAPPER'),
1752 errMsg
1753 )
1754 self._cmd = [ path.join('.', self._wrapperFile) ]
1755 if self._containerSetup is not None:
1756 asetupFile.close()
1757 self._cmd = container_cmd + self._cmd
1758
1759
void print(char *figname, TCanvas *c1)

◆ athenaMP()

python.trfExe.transformExecutor.athenaMP ( self)
inherited

Definition at line 452 of file trfExe.py.

452 def athenaMP(self):
453 return self._athenaMP
454

◆ cpuTime()

python.trfExe.transformExecutor.cpuTime ( self)
inherited

Definition at line 366 of file trfExe.py.

366 def cpuTime(self):
367 if self._exeStart and self._exeStop:
368 return calcCpuTime(self._exeStart, self._exeStop)
369 else:
370 return None
371

◆ cpuTimeTotal()

python.trfExe.transformExecutor.cpuTimeTotal ( self)
inherited

Definition at line 430 of file trfExe.py.

430 def cpuTimeTotal(self):
431 if self._preExeStart and self._valStop:
432 return calcCpuTime(self._preExeStart, self._valStop)
433 else:
434 return None
435

◆ dbMonitor()

python.trfExe.transformExecutor.dbMonitor ( self)
inherited

Definition at line 456 of file trfExe.py.

456 def dbMonitor(self):
457 return self._dbMonitor
458
459

◆ disableMP() [1/2]

python.trfExe.athenaExecutor.disableMP ( self)
inherited

Definition at line 945 of file trfExe.py.

945 def disableMP(self):
946 return self._disableMP
947

◆ disableMP() [2/2]

python.trfExe.athenaExecutor.disableMP ( self,
value )
inherited

Definition at line 949 of file trfExe.py.

949 def disableMP(self, value):
950 self._disableMP = value
951

◆ disableMT() [1/2]

python.trfExe.athenaExecutor.disableMT ( self)
inherited

Definition at line 953 of file trfExe.py.

953 def disableMT(self):
954 return self._disableMT
955

◆ disableMT() [2/2]

python.trfExe.athenaExecutor.disableMT ( self,
value )
inherited

Definition at line 957 of file trfExe.py.

957 def disableMT(self, value):
958 self._disableMT = value
959

◆ doAll()

python.trfExe.transformExecutor.doAll ( self,
input = set(),
output = set() )
inherited

Convenience function.

Definition at line 499 of file trfExe.py.

499 def doAll(self, input=set(), output=set()):
500 self.preExecute(input, output)
501 self.execute()
502 self.postExecute()
503 self.validate()
504

◆ errMsg()

python.trfExe.transformExecutor.errMsg ( self)
inherited

Definition at line 304 of file trfExe.py.

304 def errMsg(self):
305 return self._errMsg
306

◆ eventCount()

python.trfExe.transformExecutor.eventCount ( self)
inherited

Definition at line 444 of file trfExe.py.

444 def eventCount(self):
445 return self._eventCount
446

◆ exe() [1/2]

python.trfExe.scriptExecutor.exe ( self)
inherited

Definition at line 641 of file trfExe.py.

641 def exe(self):
642 return self._exe
643

◆ exe() [2/2]

python.trfExe.scriptExecutor.exe ( self,
value )
inherited

Definition at line 645 of file trfExe.py.

645 def exe(self, value):
646 self._exe = value
647 self._extraMetadata['script'] = value
648

◆ exeArgs() [1/2]

python.trfExe.scriptExecutor.exeArgs ( self)
inherited

Definition at line 650 of file trfExe.py.

650 def exeArgs(self):
651 return self._exeArgs
652

◆ exeArgs() [2/2]

python.trfExe.scriptExecutor.exeArgs ( self,
value )
inherited

Definition at line 654 of file trfExe.py.

654 def exeArgs(self, value):
655 self._exeArgs = value
656# self._extraMetadata['scriptArgs'] = value
657

◆ execute()

python.trfExe.scriptExecutor.execute ( self)
inherited

Reimplemented from python.trfExe.transformExecutor.

Reimplemented in python.trfExe.bsMergeExecutor, and python.trfExe.POOLMergeExecutor.

Definition at line 723 of file trfExe.py.

723 def execute(self):
724 self._hasExecuted = True
725 msg.info('Starting execution of {0} ({1})'.format(self._name, self._cmd))
726
727 self._exeStart = os.times()
728 msg.debug('exeStart time is {0}'.format(self._exeStart))
729 if ('execOnly' in self.conf.argdict and self.conf.argdict['execOnly'] is True):
730 msg.info('execOnly flag is set - execution will now switch, replacing the transform')
731 os.execvp(self._cmd[0], self._cmd)
732
733 encargs = {'encoding' : 'utf8'}
734 try:
735 # if we are already inside a container, then
736 # must map /srv of the nested container to /srv of the parent container:
737 # https://twiki.atlas-canada.ca/bin/view/AtlasCanada/Containers#Mount_point_summary
738 if self._alreadyInContainer and self._containerSetup is not None:
739 msg.info("chdir /srv to launch a nested container for the substep")
740 os.chdir("/srv")
741 p = subprocess.Popen(self._cmd, shell = False, stdout = subprocess.PIPE, stderr = subprocess.STDOUT, bufsize = 1, **encargs)
742 # go back to the original working directory immediately after to store prmon in it
743 if self._alreadyInContainer and self._containerSetup is not None:
744 msg.info("chdir {} after launching the nested container".format(self._workdir))
745 os.chdir(self._workdir)
746
747 if self._memMonitor:
748 try:
749 self._memSummaryFile = 'prmon.summary.' + self._name + '.json'
750 self._memFullFile = 'prmon.full.' + self._name
751 memMonitorCommand = ['prmon', '--pid', str(p.pid), '--filename', 'prmon.full.' + self._name,
752 '--json-summary', self._memSummaryFile, '--log-filename', 'prmon.' + self._name + '.log',
753 '--interval', '30']
754 mem_proc = subprocess.Popen(memMonitorCommand, shell = False, close_fds=True, **encargs)
755 # TODO - link mem.full.current to mem.full.SUBSTEP
756 except Exception as e:
757 msg.warning('Failed to spawn memory monitor for {0}: {1}'.format(self._name, e))
758 self._memMonitor = False
759
760 while p.poll() is None:
761 try:
762 line = p.stdout.readline()
763 if line:
764 self._echologger.info(line.rstrip())
765 except UnicodeDecodeError as e:
766 msg.warning('Exception raised processing athena log: {0}'.format(e))
767 # Hoover up remaining buffered output lines
768 for line in p.stdout:
769 self._echologger.info(line.rstrip())
770
771 self._rc = p.returncode
772 msg.info('%s executor returns %d', self._name, self._rc)
773 self._exeStop = os.times()
774 msg.debug('exeStop time is {0}'.format(self._exeStop))
775 except OSError as e:
776 errMsg = 'Execution of {0} failed and raised OSError: {1}'.format(self._cmd[0], e)
777 msg.error(errMsg)
778 raise trfExceptions.TransformExecutionException(trfExit.nameToCode('TRF_EXEC'), errMsg)
779 finally:
780 if self._memMonitor:
781 try:
782 mem_proc.send_signal(signal.SIGUSR1)
783 countWait = 0
784 while (not mem_proc.poll()) and countWait < 10:
785 time.sleep(0.1)
786 countWait += 1
787 except OSError:
788 pass
789
790

◆ exeStartTimes()

python.trfExe.transformExecutor.exeStartTimes ( self)
inherited

Definition at line 336 of file trfExe.py.

336 def exeStartTimes(self):
337 return self._exeStart
338

◆ exeStopTimes()

python.trfExe.transformExecutor.exeStopTimes ( self)
inherited

Definition at line 340 of file trfExe.py.

340 def exeStopTimes(self):
341 return self._exeStop
342

◆ extraMetadata()

python.trfExe.transformExecutor.extraMetadata ( self)
inherited

Definition at line 292 of file trfExe.py.

292 def extraMetadata(self):
293 return self._extraMetadata
294

◆ first()

python.trfExe.transformExecutor.first ( self)
inherited
Note
At the moment only athenaExecutor sets this property, but that might be changed...

Definition at line 325 of file trfExe.py.

325 def first(self):
326 if hasattr(self, '_first'):
327 return self._first
328 else:
329 return None
330

◆ hasExecuted()

python.trfExe.transformExecutor.hasExecuted ( self)
inherited

Definition at line 296 of file trfExe.py.

296 def hasExecuted(self):
297 return self._hasExecuted
298

◆ hasValidated()

python.trfExe.transformExecutor.hasValidated ( self)
inherited

Definition at line 316 of file trfExe.py.

316 def hasValidated(self):
317 return self._hasValidated
318

◆ inData() [1/2]

python.trfExe.transformExecutor.inData ( self)
inherited

Definition at line 235 of file trfExe.py.

235 def inData(self):
236
237 if '_inData' in dir(self):
238 return self._inData
239 return None
240

◆ inData() [2/2]

python.trfExe.transformExecutor.inData ( self,
value )
inherited

Definition at line 242 of file trfExe.py.

242 def inData(self, value):
243 self._inData = set(value)
244

◆ inDataUpdate()

python.trfExe.transformExecutor.inDataUpdate ( self,
value )
inherited

Definition at line 245 of file trfExe.py.

245 def inDataUpdate(self, value):
246
247 if '_inData' in dir(self):
248 self._inData.update(value)
249 else:
250
251 self.inData = value
252
253

◆ input()

python.trfExe.transformExecutor.input ( self)
inherited
Note
This returns the actual input data with which this executor ran (c.f. inData which returns all the possible data types this executor could run with)

Definition at line 276 of file trfExe.py.

276 def input(self):
277
278 if '_input' in dir(self):
279 return self._input
280 return None
281

◆ inputDataTypeCountCheck() [1/2]

python.trfExe.athenaExecutor.inputDataTypeCountCheck ( self)
inherited

Definition at line 933 of file trfExe.py.

933 def inputDataTypeCountCheck(self):
934 return self._inputDataTypeCountCheck
935

◆ inputDataTypeCountCheck() [2/2]

python.trfExe.athenaExecutor.inputDataTypeCountCheck ( self,
value )
inherited

Definition at line 937 of file trfExe.py.

937 def inputDataTypeCountCheck(self, value):
938 self._inputDataTypeCountCheck = value
939

◆ isValidated()

python.trfExe.transformExecutor.isValidated ( self)
inherited

Definition at line 320 of file trfExe.py.

320 def isValidated(self):
321 return self._isValidated
322

◆ memAnalysis()

python.trfExe.transformExecutor.memAnalysis ( self)
inherited

Definition at line 398 of file trfExe.py.

398 def memAnalysis(self):
399 return self._memLeakResult
400

◆ memStats()

python.trfExe.transformExecutor.memStats ( self)
inherited

Definition at line 394 of file trfExe.py.

394 def memStats(self):
395 return self._memStats
396

◆ myMerger()

python.trfExe.transformExecutor.myMerger ( self)
inherited

Now define properties for these data members.

Definition at line 207 of file trfExe.py.

207 def myMerger(self):
208 return self._myMerger
209

◆ name() [1/2]

python.trfExe.transformExecutor.name ( self)
inherited

Definition at line 211 of file trfExe.py.

211 def name(self):
212 return self._name
213

◆ name() [2/2]

python.trfExe.transformExecutor.name ( self,
value )
inherited

Definition at line 215 of file trfExe.py.

215 def name(self, value):
216 self._name = value
217

◆ onlyMP() [1/2]

python.trfExe.athenaExecutor.onlyMP ( self)
inherited

Definition at line 961 of file trfExe.py.

961 def onlyMP(self):
962 return self._onlyMP
963

◆ onlyMP() [2/2]

python.trfExe.athenaExecutor.onlyMP ( self,
value )
inherited

Definition at line 965 of file trfExe.py.

965 def onlyMP(self, value):
966 self._onlyMP = value
967

◆ onlyMT() [1/2]

python.trfExe.athenaExecutor.onlyMT ( self)
inherited

Definition at line 969 of file trfExe.py.

969 def onlyMT(self):
970 return self._onlyMT
971

◆ onlyMT() [2/2]

python.trfExe.athenaExecutor.onlyMT ( self,
value )
inherited

Definition at line 973 of file trfExe.py.

973 def onlyMT(self, value):
974 self._onlyMT = value
975

◆ outData() [1/2]

python.trfExe.transformExecutor.outData ( self)
inherited

Definition at line 255 of file trfExe.py.

255 def outData(self):
256
257 if '_outData' in dir(self):
258 return self._outData
259 return None
260

◆ outData() [2/2]

python.trfExe.transformExecutor.outData ( self,
value )
inherited

Definition at line 262 of file trfExe.py.

262 def outData(self, value):
263 self._outData = set(value)
264

◆ outDataUpdate()

python.trfExe.transformExecutor.outDataUpdate ( self,
value )
inherited

Definition at line 265 of file trfExe.py.

265 def outDataUpdate(self, value):
266
267 if '_outData' in dir(self):
268 self._outData.update(value)
269 else:
270
271 self.outData = value
272

◆ output()

python.trfExe.transformExecutor.output ( self)
inherited
Note
This returns the actual output data with which this executor ran (c.f. outData which returns all the possible data types this executor could run with)

Definition at line 285 of file trfExe.py.

285 def output(self):
286
287 if '_output' in dir(self):
288 return self._output
289 return None
290

◆ postExeCpuTime()

python.trfExe.transformExecutor.postExeCpuTime ( self)
inherited

Definition at line 402 of file trfExe.py.

402 def postExeCpuTime(self):
403 if self._exeStop and self._valStart:
404 return calcCpuTime(self._exeStop, self._valStart)
405 else:
406 return None
407

◆ postExecute()

python.trfExe.athenaExecutor.postExecute ( self)
inherited

Reimplemented from python.trfExe.scriptExecutor.

Definition at line 1247 of file trfExe.py.

1247 def postExecute(self):
1248 super(athenaExecutor, self).postExecute()
1249 # MPI merging
1250 if 'mpi' in self.conf.argdict and self.conf.argdict['mpi'].value:
1251 mpi.mergeOutputs()
1252
1253 # Handle executor substeps
1254 if self.conf.totalExecutorSteps > 1:
1255 if self._athenaMP > 0:
1256 outputDataDictionary = dict([ (dataType, self.conf.dataDictionary[dataType]) for dataType in self._output ])
1257 athenaMPOutputHandler(self._athenaMPFileReport, self._athenaMPWorkerTopDir, outputDataDictionary, self._athenaMP, False, self.conf.argdict)
1258 if self.conf.executorStep == self.conf.totalExecutorSteps - 1:
1259 # first loop over datasets for the output
1260 for dataType in self._output:
1261 newValue = []
1262 if self._athenaMP > 0:
1263 # assume the same number of workers all the time
1264 for i in range(self.conf.totalExecutorSteps):
1265 for v in self.conf.dataDictionary[dataType].value:
1266 newValue.append(v.replace('_{0}{1}_'.format(executorStepSuffix, self.conf.executorStep),
1267 '_{0}{1}_'.format(executorStepSuffix, i)))
1268 else:
1269 self.conf.dataDictionary[dataType].multipleOK = True
1270 # just combine all executors
1271 for i in range(self.conf.totalExecutorSteps):
1272 newValue.append(self.conf.dataDictionary[dataType].originalName + '_{0}{1}'.format(executorStepSuffix, i))
1273 self.conf.dataDictionary[dataType].value = newValue
1274
1275 # do the merging if needed
1276 if self.conf.dataDictionary[dataType].io == "output" and len(self.conf.dataDictionary[dataType].value) > 1:
1277 self._smartMerge(self.conf.dataDictionary[dataType])
1278
1279 # If this was an athenaMP run then we need to update output files
1280 elif self._athenaMP > 0:
1281 outputDataDictionary = dict([ (dataType, self.conf.dataDictionary[dataType]) for dataType in self._output ])
1282
1283 skipFileChecks=False
1284 if 'eventService' in self.conf.argdict and self.conf.argdict['eventService'].value:
1285 skipFileChecks=True
1286 athenaMPOutputHandler(self._athenaMPFileReport, self._athenaMPWorkerTopDir, outputDataDictionary, self._athenaMP, skipFileChecks, self.conf.argdict)
1287 for dataType in self._output:
1288 if self.conf.dataDictionary[dataType].io == "output" and len(self.conf.dataDictionary[dataType].value) > 1:
1289 self._smartMerge(self.conf.dataDictionary[dataType])
1290
1291 if 'TXT_JIVEXMLTGZ' in self.conf.dataDictionary:
1292 self._targzipJiveXML()
1293
1294 # Summarise events passed the filter ISF_SimEventFilter from log.ReSim
1295 # This is a bit ugly to have such a specific feature here though
1296 # TODO
1297 # The best is to have a general approach so that user can extract useful info from log
1298 # Instead of hard coding a pattern, one idea could be that user provides a regExp pattern
1299 # in which the wanted variable is grouped by a name, then transforms could decode the pattern
1300 # and use it to extract required info and do the summation during log scan.
1301 if self._logFileName=='log.ReSim' and self.name=='ReSim':
1302 msg.info('scanning {0} for reporting events passed the filter ISF_SimEventFilter'.format(self._logFileName))
1303 self._resimevents = reportEventsPassedSimFilter(self._logFileName)
1304
1305 # Remove intermediate input/output files of sub-steps
1306 # Delete only files with io="temporay" which are files with pattern "tmp*"
1307 # Some stubs like tmp.RDO_TRIG_000 created in AthenaMP mode or
1308 # tmp.HIST_ESD_INT, tmp.HIST_AOD_INT as input to DQHistogramMerge.py are not deleted
1309 # Enable if --deleteIntermediateOutputfiles is set
1310 if ('deleteIntermediateOutputfiles' in self.conf._argdict and self.conf._argdict['deleteIntermediateOutputfiles'].value):
1311 inputDataDictionary = dict([ (dataType, self.conf.dataDictionary[dataType]) for dataType in self._input ])
1312
1313 for k, v in inputDataDictionary.items():
1314 if not v.io == 'temporary':
1315 continue
1316 for filename in v.value:
1317 if os.access(filename, os.R_OK) and not filename.startswith("/cvmfs"):
1318 msg.info("Removing intermediate {0} input file {1}".format(k, filename))
1319 # Check if symbolic link and delete also linked file
1320 if (os.path.realpath(filename) != filename):
1321 targetpath = os.path.realpath(filename)
1322 os.unlink(filename)
1323 if (targetpath) and os.access(targetpath, os.R_OK):
1324 os.unlink(targetpath)
1325
1326

◆ postExeWallTime()

python.trfExe.transformExecutor.postExeWallTime ( self)
inherited

Definition at line 409 of file trfExe.py.

409 def postExeWallTime(self):
410 if self._exeStop and self._valStart:
411 return calcWallTime(self._exeStop, self._valStart)
412 else:
413 return None
414

◆ preExeCpuTime()

python.trfExe.transformExecutor.preExeCpuTime ( self)
inherited

Definition at line 352 of file trfExe.py.

352 def preExeCpuTime(self):
353 if self._preExeStart and self._exeStart:
354 return calcCpuTime(self._preExeStart, self._exeStart)
355 else:
356 return None
357

◆ preExecute()

python.trfExe.reductionFrameworkExecutor.preExecute ( self,
input = set(),
output = set() )

Take inputDAODFile and setup the actual outputs needed in this job.

Reimplemented from python.trfExe.athenaExecutor.

Definition at line 1904 of file trfExe.py.

1904 def preExecute(self, input=set(), output=set()):
1905 self.setPreExeStart()
1906 msg.debug('Preparing for execution of {0} with inputs {1} and outputs {2}'.format(self.name, input, output))
1907 if 'NTUP_PILEUP' not in output:
1908 # New derivation framework transform uses "formats"
1909 if 'formats' not in self.conf.argdict:
1910 raise trfExceptions.TransformExecutionException(trfExit.nameToCode('TRF_REDUCTION_CONFIG_ERROR'),
1911 'No derivation configuration specified')
1912
1913 if ('DAOD' not in output) and ('D2AOD' not in output):
1914 raise trfExceptions.TransformExecutionException(trfExit.nameToCode('TRF_REDUCTION_CONFIG_ERROR'),
1915 'No base name for DAOD output')
1916
1917 formatList = []
1918 if 'formats' in self.conf.argdict: formatList = self.conf.argdict['formats'].value
1919 for reduction in formatList:
1920 if ('DAOD' in output):
1921 dataType = 'DAOD_' + reduction
1922 if 'augmentations' not in self.conf.argdict:
1923 outputName = 'DAOD_' + reduction + '.' + self.conf.argdict['outputDAODFile'].value[0]
1924 else:
1925 for val in self.conf.argdict['augmentations'].value:
1926 if reduction in val.split(':')[0]:
1927 outputName = 'DAOD_' + val.split(':')[1] + '.' + self.conf.argdict['outputDAODFile'].value[0]
1928 break
1929 else:
1930 outputName = 'DAOD_' + reduction + '.' + self.conf.argdict['outputDAODFile'].value[0]
1931
1932 if ('D2AOD' in output):
1933 dataType = 'D2AOD_' + reduction
1934 outputName = 'D2AOD_' + reduction + '.' + self.conf.argdict['outputD2AODFile'].value[0]
1935
1936 msg.info('Adding reduction output type {0}'.format(dataType))
1937 output.add(dataType)
1938 newReduction = trfArgClasses.argPOOLFile(outputName, io='output', runarg=True, type='AOD',
1939 name=reduction)
1940 # References to _trf - can this be removed?
1941 self.conf.dataDictionary[dataType] = newReduction
1942
1943 # Clean up the stub file from the executor input and the transform's data dictionary
1944 # (we don't remove the actual argFile instance)
1945 if ('DAOD' in output):
1946 output.remove('DAOD')
1947 del self.conf.dataDictionary['DAOD']
1948 del self.conf.argdict['outputDAODFile']
1949 if ('D2AOD' in output):
1950 output.remove('D2AOD')
1951 del self.conf.dataDictionary['D2AOD']
1952 del self.conf.argdict['outputD2AODFile']
1953
1954 msg.info('Data dictionary is now: {0}'.format(self.conf.dataDictionary))
1955 msg.info('Input/Output: {0}/{1}'.format(input, output))
1956
1957 msg.info('Data dictionary is now: {0}'.format(self.conf.dataDictionary))
1958 msg.info('Input/Output: {0}/{1}'.format(input, output))
1959 super(reductionFrameworkExecutor, self).preExecute(input, output)
1960
1961

◆ preExeStartTimes()

python.trfExe.transformExecutor.preExeStartTimes ( self)
inherited

Definition at line 332 of file trfExe.py.

332 def preExeStartTimes(self):
333 return self._preExeStart
334

◆ preExeWallTime()

python.trfExe.transformExecutor.preExeWallTime ( self)
inherited

Definition at line 359 of file trfExe.py.

359 def preExeWallTime(self):
360 if self._preExeStart and self._exeStart:
361 return calcWallTime(self._preExeStart, self._exeStart)
362 else:
363 return None
364

◆ rc()

python.trfExe.transformExecutor.rc ( self)
inherited

Definition at line 300 of file trfExe.py.

300 def rc(self):
301 return self._rc
302
static Double_t rc

◆ reSimEvent()

python.trfExe.transformExecutor.reSimEvent ( self)
inherited

Definition at line 448 of file trfExe.py.

448 def reSimEvent(self):
449 return self._resimevents
450

◆ setPreExeStart()

python.trfExe.transformExecutor.setPreExeStart ( self)
inherited

Definition at line 461 of file trfExe.py.

461 def setPreExeStart(self):
462 if self._preExeStart is None:
463 self._preExeStart = os.times()
464 msg.debug('preExeStart time is {0}'.format(self._preExeStart))
465

◆ setValStart()

python.trfExe.transformExecutor.setValStart ( self)
inherited

Definition at line 466 of file trfExe.py.

466 def setValStart(self):
467 if self._valStart is None:
468 self._valStart = os.times()
469 msg.debug('valStart time is {0}'.format(self._valStart))
470

◆ skeletonCA()

python.trfExe.athenaExecutor.skeletonCA ( self)
inherited

Definition at line 976 of file trfExe.py.

976 def skeletonCA(self):
977 return self._skeletonCA
978

◆ substep()

python.trfExe.athenaExecutor.substep ( self)
inherited

Reimplemented from python.trfExe.transformExecutor.

Definition at line 941 of file trfExe.py.

941 def substep(self):
942 return self._substep
943

◆ sysTime()

python.trfExe.transformExecutor.sysTime ( self)
inherited

Definition at line 380 of file trfExe.py.

380 def sysTime(self):
381 if self._exeStart and self._exeStop:
382 return self._exeStop[3] - self._exeStart[3]
383 else:
384 return None
385

◆ trf() [1/2]

python.trfExe.transformExecutor.trf ( self)
inherited

Definition at line 225 of file trfExe.py.

225 def trf(self):
226 if '_trf' in dir(self):
227 return self._trf
228 return None
229

◆ trf() [2/2]

python.trfExe.transformExecutor.trf ( self,
value )
inherited

Definition at line 231 of file trfExe.py.

231 def trf(self, value):
232 self._trf = value
233

◆ usrTime()

python.trfExe.transformExecutor.usrTime ( self)
inherited

Definition at line 373 of file trfExe.py.

373 def usrTime(self):
374 if self._exeStart and self._exeStop:
375 return self._exeStop[2] - self._exeStart[2]
376 else:
377 return None
378

◆ validate()

python.trfExe.athenaExecutor.validate ( self)
inherited

Reimplemented from python.trfExe.scriptExecutor.

Reimplemented in python.trfExe.optionalAthenaExecutor.

Definition at line 1327 of file trfExe.py.

1327 def validate(self):
1328 self.setValStart()
1329 self._hasValidated = True
1330 deferredException = None
1331 memLeakThreshold = 5000
1332 _hasMemLeak = False
1333
1334
1335 try:
1336 super(athenaExecutor, self).validate()
1337 except trfExceptions.TransformValidationException as e:
1338 # In this case we hold this exception until the logfile has been scanned
1339 msg.error('Validation of return code failed: {0!s}'.format(e))
1340 deferredException = e
1341
1342
1349 if self._memFullFile:
1350 msg.info('Analysing memory monitor output file {0} for possible memory leak'.format(self._memFullFile))
1351 self._memLeakResult = analytic().getFittedData(self._memFullFile)
1352 if self._memLeakResult:
1353 if self._memLeakResult['slope'] > memLeakThreshold:
1354 _hasMemLeak = True
1355 msg.warning('Possible memory leak; abnormally high values in memory monitor parameters (ignore this message if the job has finished successfully)')
1356 else:
1357 msg.warning('Failed to analyse the memory monitor file {0}'.format(self._memFullFile))
1358 else:
1359 msg.info('No memory monitor file to be analysed')
1360
1361 # Logfile scan setup
1362 # Always use ignorePatterns from the command line
1363 # For patterns in files, pefer the command line first, then any special settings for
1364 # this executor, then fallback to the standard default (atlas_error_mask.db)
1365 if 'ignorePatterns' in self.conf.argdict:
1366 igPat = self.conf.argdict['ignorePatterns'].value
1367 else:
1368 igPat = []
1369 if 'ignoreFiles' in self.conf.argdict:
1370 ignorePatterns = trfValidation.ignorePatterns(files = self.conf.argdict['ignoreFiles'].value, extraSearch=igPat)
1371 elif self._errorMaskFiles is not None:
1372 ignorePatterns = trfValidation.ignorePatterns(files = self._errorMaskFiles, extraSearch=igPat)
1373 else:
1374 ignorePatterns = trfValidation.ignorePatterns(files = athenaExecutor._defaultIgnorePatternFile, extraSearch=igPat)
1375
1376 # Now actually scan my logfile
1377 msg.info('Scanning logfile {0} for errors in substep {1}'.format(self._logFileName, self._substep))
1378 self._logScan = trfValidation.athenaLogFileReport(logfile=self._logFileName, substepName=self._substep,
1379 ignoreList=ignorePatterns)
1380 worstError = self._logScan.worstError()
1381 eventLoopWarnings = self._logScan.eventLoopWarnings()
1382 self._dbMonitor = self._logScan.dbMonitor()
1383
1384
1385 # In general we add the error message to the exit message, but if it's too long then don't do
1386 # that and just say look in the jobReport
1387 if worstError['firstError']:
1388 if len(worstError['firstError']['message']) > athenaExecutor._exitMessageLimit:
1389 if 'CoreDumpSvc' in worstError['firstError']['message']:
1390 exitErrorMessage = "Core dump at line {0} (see jobReport for further details)".format(worstError['firstError']['firstLine'])
1391 elif 'G4Exception' in worstError['firstError']['message']:
1392 exitErrorMessage = "G4 exception at line {0} (see jobReport for further details)".format(worstError['firstError']['firstLine'])
1393 else:
1394 exitErrorMessage = "Long {0} message at line {1} (see jobReport for further details)".format(worstError['level'], worstError['firstError']['firstLine'])
1395 else:
1396 exitErrorMessage = "Logfile error in {0}: \"{1}\"".format(self._logFileName, worstError['firstError']['message'])
1397 else:
1398 exitErrorMessage = "Error level {0} found (see athena logfile for details)".format(worstError['level'])
1399
1400 # If we failed on the rc, then abort now
1401 if deferredException is not None:
1402 # Add any logfile information we have
1403 if worstError['nLevel'] >= stdLogLevels['ERROR']:
1404 deferredException.errMsg = deferredException.errMsg + "; {0}".format(exitErrorMessage)
1405 # Add the result of memory analysis
1406 if _hasMemLeak:
1407 deferredException.errMsg = deferredException.errMsg + "; Possible memory leak: 'pss' slope: {0} KB/s".format(self._memLeakResult['slope'])
1408 raise deferredException
1409
1410
1411 # Very simple: if we get ERROR or worse, we're dead, except if ignoreErrors=True
1412 if worstError['nLevel'] == stdLogLevels['ERROR'] and ('ignoreErrors' in self.conf.argdict and self.conf.argdict['ignoreErrors'].value is True):
1413 msg.warning('Found ERRORs in the logfile, but ignoring this as ignoreErrors=True (see jobReport for details)')
1414 # Act as if ignoreErrors=True if running in MPI, because we want to be tolerant to the occasional event failure
1415 elif worstError['nLevel'] >= stdLogLevels['ERROR'] and (not mpi.mpiShouldValidate()):
1416 msg.warning(f'Found {worstError["level"]} in the logfile in MPI rank {mpi.getMPIRank()} but moving on to be failure-tolerant')
1417 elif worstError['nLevel'] >= stdLogLevels['ERROR']:
1418 self._isValidated = False
1419 msg.error('Fatal error in athena logfile (level {0})'.format(worstError['level']))
1420 # Add the result of memory analysis
1421 if _hasMemLeak:
1422 exitErrorMessage = exitErrorMessage + "; Possible memory leak: 'pss' slope: {0} KB/s".format(self._memLeakResult['slope'])
1423 raise trfExceptions.TransformLogfileErrorException(trfExit.nameToCode('TRF_EXEC_LOGERROR'),
1424 'Fatal error in athena logfile: "{0}"'.format(exitErrorMessage))
1425
1426 # Print event loop warnings
1427 if (len(eventLoopWarnings) > 0):
1428 msg.warning('Found WARNINGS in the event loop, as follows:')
1429 for element in eventLoopWarnings:
1430 msg.warning('{0} {1} ({2} instances)'.format(element['item']['service'],element['item']['message'],element['count']))
1431
1432 # Must be ok if we got here!
1433 msg.info('Executor {0} has validated successfully'.format(self.name))
1434 self._isValidated = True
1435
1436 self._valStop = os.times()
1437 msg.debug('valStop time is {0}'.format(self._valStop))
1438

◆ validation() [1/2]

python.trfExe.transformExecutor.validation ( self)
inherited

Definition at line 308 of file trfExe.py.

308 def validation(self):
309 return self._validation
310

◆ validation() [2/2]

python.trfExe.transformExecutor.validation ( self,
value )
inherited

Definition at line 312 of file trfExe.py.

312 def validation(self, value):
313 self._validation = value
314

◆ validationCpuTime()

python.trfExe.transformExecutor.validationCpuTime ( self)
inherited

Definition at line 416 of file trfExe.py.

416 def validationCpuTime(self):
417 if self._valStart and self._valStop:
418 return calcCpuTime(self._valStart, self._valStop)
419 else:
420 return None
421

◆ validationWallTime()

python.trfExe.transformExecutor.validationWallTime ( self)
inherited

Definition at line 423 of file trfExe.py.

423 def validationWallTime(self):
424 if self._valStart and self._valStop:
425 return calcWallTime(self._valStart, self._valStop)
426 else:
427 return None
428

◆ valStartTimes()

python.trfExe.transformExecutor.valStartTimes ( self)
inherited

Definition at line 344 of file trfExe.py.

344 def valStartTimes(self):
345 return self._valStart
346

◆ valStopTimes()

python.trfExe.transformExecutor.valStopTimes ( self)
inherited

Definition at line 348 of file trfExe.py.

348 def valStopTimes(self):
349 return self._valStop
350

◆ wallTime()

python.trfExe.transformExecutor.wallTime ( self)
inherited

Definition at line 387 of file trfExe.py.

387 def wallTime(self):
388 if self._exeStart and self._exeStop:
389 return calcWallTime(self._exeStart, self._exeStop)
390 else:
391 return None
392

◆ wallTimeTotal()

python.trfExe.transformExecutor.wallTimeTotal ( self)
inherited

Definition at line 437 of file trfExe.py.

437 def wallTimeTotal(self):
438 if self._preExeStart and self._valStop:
439 return calcWallTime(self._preExeStart, self._valStop)
440 else:
441 return None
442

Member Data Documentation

◆ _alreadyInContainer

python.trfExe.transformExecutor._alreadyInContainer = None
protectedinherited

Definition at line 198 of file trfExe.py.

◆ _asetup

python.trfExe.athenaExecutor._asetup = asetup
protectedinherited

Definition at line 1587 of file trfExe.py.

◆ _athenaConcurrentEvents

int python.trfExe.transformExecutor._athenaConcurrentEvents = 0
protectedinherited

Definition at line 195 of file trfExe.py.

◆ _athenaMP

int python.trfExe.transformExecutor._athenaMP = 0
protectedinherited

Definition at line 193 of file trfExe.py.

◆ _athenaMPEventOrdersFile

str python.trfExe.athenaExecutor._athenaMPEventOrdersFile = 'athenamp_eventorders.txt.{0}'.format(self._name)
protectedinherited

Definition at line 1116 of file trfExe.py.

◆ _athenaMPFileReport

python.trfExe.athenaExecutor._athenaMPFileReport = 'athenaMP-outputs-{0}-{1}'.format(self._name, self._substep)
protectedinherited

Definition at line 1115 of file trfExe.py.

◆ _athenaMPReadEventOrders

bool python.trfExe.athenaExecutor._athenaMPReadEventOrders = True
protectedinherited

Definition at line 1118 of file trfExe.py.

◆ _athenaMPStrategy

str python.trfExe.athenaExecutor._athenaMPStrategy = self.conf.argdict['athenaMPStrategy'].returnMyValue(name=self._name, substep=self._substep, first=self.conf.firstExecutor)
protectedinherited

Definition at line 1124 of file trfExe.py.

◆ _athenaMPWorkerTopDir

python.trfExe.athenaExecutor._athenaMPWorkerTopDir = 'athenaMP-workers-{0}-{1}'.format(self._name, self._substep)
protectedinherited

Definition at line 1114 of file trfExe.py.

◆ _athenaMT

int python.trfExe.transformExecutor._athenaMT = 0
protectedinherited

Definition at line 194 of file trfExe.py.

◆ _cmd

python.trfExe.scriptExecutor._cmd = None
protectedinherited
Note
If an inherited class has set self._cmd leave it alone

Definition at line 636 of file trfExe.py.

◆ _containerSetup

python.trfExe.transformExecutor._containerSetup = None
protectedinherited

Definition at line 199 of file trfExe.py.

◆ _dataArgs

python.trfExe.athenaExecutor._dataArgs = dataArgs
protectedinherited

Definition at line 900 of file trfExe.py.

◆ _dbMonitor

python.trfExe.transformExecutor._dbMonitor = None
protectedinherited

Definition at line 196 of file trfExe.py.

◆ _dbsetup

python.trfExe.athenaExecutor._dbsetup = dbsetup
protectedinherited

Definition at line 1588 of file trfExe.py.

◆ _defaultIgnorePatternFile

list python.trfExe.athenaExecutor._defaultIgnorePatternFile = ['atlas_error_mask.db']
staticprotectedinherited

Definition at line 849 of file trfExe.py.

◆ _disableMP

bool python.trfExe.athenaExecutor._disableMP = disableMP
protectedinherited

Definition at line 904 of file trfExe.py.

◆ _disableMT

python.trfExe.athenaExecutor._disableMT = disableMT
protectedinherited

Definition at line 903 of file trfExe.py.

◆ _echologger

python.trfExe.scriptExecutor._echologger = logging.getLogger(self._name)
protectedinherited
Note
Query the environment for echo configuration Let the manual envars always win over auto-detected settings

Definition at line 692 of file trfExe.py.

◆ _echoOutput

bool python.trfExe.scriptExecutor._echoOutput = False
protectedinherited
Note
Query the environment for echo configuration Let the manual envars always win over auto-detected settings

Definition at line 633 of file trfExe.py.

◆ _echostream

python.trfExe.scriptExecutor._echostream = logging.StreamHandler(_encoding_stream(sys.stdout))
protectedinherited

Definition at line 702 of file trfExe.py.

◆ _envUpdate

python.trfExe.athenaExecutor._envUpdate = trfEnv.environmentUpdate()
protectedinherited

Look for environment updates and perpare the athena command line.

Definition at line 1232 of file trfExe.py.

◆ _errMsg

str python.trfExe.transformExecutor._errMsg = None
protectedinherited

Definition at line 172 of file trfExe.py.

◆ _errorMaskFiles

python.trfExe.athenaExecutor._errorMaskFiles = errorMaskFiles
protectedinherited

Definition at line 901 of file trfExe.py.

◆ _eventCount

python.trfExe.transformExecutor._eventCount = None
protectedinherited

Definition at line 192 of file trfExe.py.

◆ _exe

python.trfExe.scriptExecutor._exe = exe
protectedinherited

Definition at line 623 of file trfExe.py.

◆ _exeArgs

python.trfExe.scriptExecutor._exeArgs = exeArgs
protectedinherited

Definition at line 626 of file trfExe.py.

◆ _exeLogFile

python.trfExe.scriptExecutor._exeLogFile = logging.FileHandler(self._logFileName, mode='w', **encargs)
protectedinherited

Definition at line 697 of file trfExe.py.

◆ _exeStart

python.trfExe.transformExecutor._exeStart = None
protectedinherited

Definition at line 187 of file trfExe.py.

◆ _exeStop

python.trfExe.transformExecutor._exeStop
protectedinherited

Definition at line 367 of file trfExe.py.

◆ _exitMessageLimit

int python.trfExe.athenaExecutor._exitMessageLimit = 200
staticprotectedinherited

Definition at line 848 of file trfExe.py.

◆ _extraMetadata

dict python.trfExe.transformExecutor._extraMetadata = {}
protectedinherited

Definition at line 181 of file trfExe.py.

◆ _extraRunargs

python.trfExe.athenaExecutor._extraRunargs = extraRunargs
protectedinherited

Definition at line 897 of file trfExe.py.

◆ _hasExecuted

bool python.trfExe.transformExecutor._hasExecuted = False
protectedinherited

Definition at line 170 of file trfExe.py.

◆ _hasValidated

bool python.trfExe.transformExecutor._hasValidated = False
protectedinherited

Definition at line 175 of file trfExe.py.

◆ _inData

python.trfExe.transformExecutor._inData = set(inData)
protectedinherited

Definition at line 144 of file trfExe.py.

◆ _input

python.trfExe.scriptExecutor._input = input
protectedinherited

Definition at line 662 of file trfExe.py.

◆ _inputDataTypeCountCheck

python.trfExe.athenaExecutor._inputDataTypeCountCheck = inputDataTypeCountCheck
protectedinherited

Definition at line 902 of file trfExe.py.

◆ _inputEventTest

python.trfExe.athenaExecutor._inputEventTest = inputEventTest
protectedinherited

Definition at line 895 of file trfExe.py.

◆ _isValidated

bool python.trfExe.transformExecutor._isValidated = False
protectedinherited

Definition at line 176 of file trfExe.py.

◆ _jobOptionsTemplate

python.trfExe.athenaExecutor._jobOptionsTemplate = JobOptionsTemplate(exe = self, version = '$Id: trfExe.py 792052 2017-01-13 13:36:51Z mavogel $')
protectedinherited

Definition at line 928 of file trfExe.py.

◆ _literalRunargs

python.trfExe.athenaExecutor._literalRunargs = literalRunargs
protectedinherited

Definition at line 899 of file trfExe.py.

◆ _logFileName

python.trfExe.scriptExecutor._logFileName = "log.{0}".format(self._name)
protectedinherited
Note
If an inherited class has set self._cmd leave it alone
Query the environment for echo configuration Let the manual envars always win over auto-detected settings

Definition at line 671 of file trfExe.py.

◆ _logScan

python.trfExe.athenaExecutor._logScan
protectedinherited
Initial value:
= trfValidation.athenaLogFileReport(logfile=self._logFileName, substepName=self._substep,
ignoreList=ignorePatterns)

Definition at line 1378 of file trfExe.py.

◆ _memFullFile

python.trfExe.transformExecutor._memFullFile = None
protectedinherited

Definition at line 191 of file trfExe.py.

◆ _memLeakResult

dict python.trfExe.transformExecutor._memLeakResult = {}
protectedinherited

Definition at line 190 of file trfExe.py.

◆ _memMonitor

bool python.trfExe.scriptExecutor._memMonitor = memMonitor
protectedinherited

Definition at line 638 of file trfExe.py.

◆ _memStats

dict python.trfExe.transformExecutor._memStats = {}
protectedinherited

Definition at line 189 of file trfExe.py.

◆ _memSummaryFile

python.trfExe.scriptExecutor._memSummaryFile = 'prmon.summary.' + self._name + '.json'
protectedinherited

Definition at line 749 of file trfExe.py.

◆ _myMerger

list python.trfExe.transformExecutor._myMerger = []
protectedinherited

Definition at line 202 of file trfExe.py.

◆ _name

python.trfExe.transformExecutor._name = forceToAlphaNum(name)
protectedinherited

Definition at line 141 of file trfExe.py.

◆ _onlyMP

bool python.trfExe.athenaExecutor._onlyMP = onlyMP
protectedinherited

Definition at line 905 of file trfExe.py.

◆ _onlyMPWithRunargs

python.trfExe.athenaExecutor._onlyMPWithRunargs = onlyMPWithRunargs
protectedinherited

Definition at line 907 of file trfExe.py.

◆ _onlyMT

python.trfExe.athenaExecutor._onlyMT = onlyMT
protectedinherited

Definition at line 906 of file trfExe.py.

◆ _originalCmd

list python.trfExe.athenaExecutor._originalCmd = self._cmd
protectedinherited

Definition at line 1586 of file trfExe.py.

◆ _outData

python.trfExe.transformExecutor._outData = set(outData)
protectedinherited

Definition at line 145 of file trfExe.py.

◆ _output

python.trfExe.scriptExecutor._output = output
protectedinherited

Definition at line 663 of file trfExe.py.

◆ _perfMonFile

python.trfExe.athenaExecutor._perfMonFile = None
protectedinherited

Definition at line 911 of file trfExe.py.

◆ _preExeStart

python.trfExe.transformExecutor._preExeStart = None
protectedinherited
Note
Place holders for resource consumption. CPU and walltime are available for all executors but currently only athena is instrumented to fill in memory stats (and then only if PerfMonSD is enabled).

Definition at line 186 of file trfExe.py.

◆ _rc

python.trfExe.transformExecutor._rc = -1
protectedinherited

Definition at line 171 of file trfExe.py.

◆ _resimevents

python.trfExe.transformExecutor._resimevents = None
protectedinherited

Definition at line 197 of file trfExe.py.

◆ _runtimeRunargs

python.trfExe.athenaExecutor._runtimeRunargs = runtimeRunargs
protectedinherited

Definition at line 898 of file trfExe.py.

◆ _setupFile

str python.trfExe.athenaExecutor._setupFile = 'setup.{name}.sh'.format(name = self._name)
protectedinherited

Definition at line 1593 of file trfExe.py.

◆ _skeleton

list python.trfExe.athenaExecutor._skeleton = [skeletonFile]
protectedinherited

Definition at line 916 of file trfExe.py.

◆ _skeletonCA

python.trfExe.athenaExecutor._skeletonCA = skeletonCA
protectedinherited

Handle MPI setup.

Write the skeleton file and prep athena

Definition at line 908 of file trfExe.py.

◆ _substep

python.trfExe.athenaExecutor._substep = forceToAlphaNum(substep)
protectedinherited

Definition at line 894 of file trfExe.py.

◆ _topOptionsFiles

python.trfExe.athenaExecutor._topOptionsFiles
protectedinherited
Initial value:
= self._jobOptionsTemplate.getTopOptions(input = inputFiles,
output = outputFiles)

Definition at line 1194 of file trfExe.py.

◆ _trf

python.trfExe.transformExecutor._trf = value
protectedinherited

Definition at line 232 of file trfExe.py.

◆ _tryDropAndReload

python.trfExe.athenaExecutor._tryDropAndReload = tryDropAndReload
protectedinherited

Add –drop-and-reload if possible (and allowed!).

Definition at line 896 of file trfExe.py.

◆ _validation

python.trfExe.transformExecutor._validation = value
protectedinherited

Definition at line 313 of file trfExe.py.

◆ _valStart

python.trfExe.transformExecutor._valStart = None
protectedinherited

Definition at line 188 of file trfExe.py.

◆ _valStop

python.trfExe.transformExecutor._valStop
protectedinherited

Definition at line 417 of file trfExe.py.

◆ _workdir

python.trfExe.scriptExecutor._workdir
protectedinherited

Definition at line 744 of file trfExe.py.

◆ _wrapperFile

python.trfExe.athenaExecutor._wrapperFile = 'runwrapper.{name}.sh'.format(name = self._name)
protectedinherited

Definition at line 1592 of file trfExe.py.

◆ conf

python.trfExe.transformExecutor.conf = conf
inherited

Executor configuration:

Note
that if conf and trf are None then we'll probably set the conf up later (this is allowed and expected to be done once the master transform has figured out what it's doing for this job)

Definition at line 163 of file trfExe.py.

◆ exeArgs

python.trfExe.scriptExecutor.exeArgs
inherited

Definition at line 712 of file trfExe.py.

◆ inData

python.trfExe.transformExecutor.inData = value
inherited
Note
Protect against _inData not yet being defined
Use normal setter

Definition at line 251 of file trfExe.py.

◆ outData

python.trfExe.transformExecutor.outData = value
inherited
Note
Protect against _outData not yet being defined
Use normal setter

Definition at line 271 of file trfExe.py.


The documentation for this class was generated from the following file: