sct_calib_tf.SCTCalibExecutor Class Reference

Inheritance diagram for sct_calib_tf.SCTCalibExecutor:

Collaboration diagram for sct_calib_tf.SCTCalibExecutor:

Public Member Functions
def	__init__ (self, skeleton)
def	preExecute (self, input=set(), output=set())
def	execute (self)
def	postExecute (self)
def	validate (self)

Private Attributes
	_errMsg
	Try to avoid validation of output files self.skipOutputFileValidation=True. More...
	_isValidated
	_hasValidated
	_logScan

Detailed Description

Definition at line 313 of file sct_calib_tf.py.

Constructor & Destructor Documentation

__init__()

def sct_calib_tf.SCTCalibExecutor.__init__	(		self,
			skeleton
	)

Definition at line 314 of file sct_calib_tf.py.

    def __init__(self, skeleton):
        athenaExecutor.__init__(self,
                                name = 'sctcalib',
                                skeletonCA='SCT_CalibAlgs.SCTCalib_Skeleton')
 

Member Function Documentation

execute()

def sct_calib_tf.SCTCalibExecutor.execute

(

self

)

Definition at line 446 of file sct_calib_tf.py.

    def execute(self):
 
        runArgs=self.conf._argdict
        # Check the run for criteria in runSelector
        if runArgs['doRunSelector']._value:
            import SCT_CalibAlgs.runSelector as runSelector
            part=runArgs['part']._value
            if runArgs['splitHitMap']._value == 1 :
              skipQueue = 1
            else:
              skipQueue = 0
            checkRun=runSelector.main(RunNumber,part,skipQueue,Stream)
            if not checkRun:
 
                print ("Run ", RunNumber, " didn't pass run selection criteria. It will not be processed and no output will be generated. Finish execution and exit gracefully")
                emptyDic = {}
                self._trf._dataDictionary = emptyDic
                    
                self._isValidated = True
                self._trf._exitMsg = 'Did not pass run selection criteria. Finish execution and exit gracefully.'
                self._trf._exitCode = 0
                resetTrfSignalHandlers()
                self._trf.generateReport(fast=True)
                sys.exit(0)
 
        rootHitmapFiles = []
        rootLbFiles = []
        rootBSerrFiles = []
        for inputFileName in runArgs['input'] :
            if inputFileName.find("SCTHitMaps") != -1:
                rootHitmapFiles.append(inputFileName)
            if inputFileName.find("SCTLB") != -1:
                rootLbFiles.append(inputFileName)
            if inputFileName.find("SCTBSErrors") != -1:
                rootBSerrFiles.append(inputFileName)
 
        if runArgs['splitHitMap']._value ==2 :
            
            if len(rootHitmapFiles) > 0 :
                
                fileutil.remove('SCTHitMaps.root')
 
                cmd = "cp -v $ROOTSYS/bin/hadd . \n"
                cmd += "hadd -n 10 SCTHitMaps.root " 
                for inputFileName in rootHitmapFiles :
                    cmd += "%s " %(inputFileName)
                cmd += "\n"
            
                print (cmd)
                self._echologger.info('Merging Hitmap files!')
                retcode=1
                try:
                    retcode = os.system(cmd)
                except OSError:
                    retcode = 1
                if retcode == 0:
                    self._echologger.info('Root merge successful')
                else:
                    self._echologger.error("FAILED to merge root files")
            
            if ( len(rootLbFiles) > 0 and (len(rootLbFiles) == len(rootHitmapFiles)) ):
                
                fileutil.remove('SCTLB.root')
 
                cmd = "cp -v $ROOTSYS/bin/hadd . \n"
                cmd += "hadd -n 10 SCTLB.root " 
                for inputFileName in rootLbFiles :
                    cmd += "%s " %(inputFileName)
                cmd += "\n"
            
                print (cmd)
                self._echologger.info('Merging LBHitmap files!')
                retcode=1
                try:
                    retcode = os.system(cmd)
                except OSError:
                    retcode = 1
                if retcode == 0:
                    self._echologger.info('Root merge successful')
                else:
                    self._echologger.error("FAILED to merge root files")
 
            if ( len(rootBSerrFiles) > 0 and (len(rootBSerrFiles) == len(rootHitmapFiles)) ):
 
                fileutil.remove('SCTBSErrors.root')
 
                cmd = "cp -v $ROOTSYS/bin/hadd . \n"
                cmd += "hadd -n 10 SCTBSErrors.root " 
                for inputFileName in rootBSerrFiles :
                    cmd += "%s " %(inputFileName)
                cmd += "\n"
            
                print (cmd)
                self._echologger.info('Merging BSerr files!')
                retcode=1
                try:
                    retcode = os.system(cmd)
                except OSError:
                    retcode = 1
                if retcode == 0:
                    self._echologger.info('Root merge successful')
                else:
                    self._echologger.error("FAILED to merge root files")
        
        super(SCTCalibExecutor, self).execute()
 
        if self._rc != 0:
            try:
                if 'less than the required minimum number of events' in open('log.sctcalib').read():
                    self._errMsg = 'Successful but warrants further investigation'
                    raise trfExceptions.TransformValidationException(trfExit.nameToCode('TRF_UNKOWN'), self._errMsg)
            except trfExceptions.TransformValidationException:
                pass
 

postExecute()

def sct_calib_tf.SCTCalibExecutor.postExecute

(

self

)

Definition at line 560 of file sct_calib_tf.py.

    def postExecute(self):
 
        runArgs=self.conf._argdict
        prefix=runArgs['prefix']._value
 
        # After processing Hitmaps, change Metadata of SCTHitMaps and SCTLB (and SCTBSErrors) files so
        # they contain the number of events. This value can be used when processing
        # noisy strips to avoid running over empty files
        
        listOfKeys = self._trf.dataDictionary
 
        if 'doNoisyStrip' in runArgs['part']._value and runArgs['splitHitMap']._value == 1:
            outInstance0 = self.conf.dataDictionary[list(self._output)[0]]
            outTFile0 = TFile(outInstance0._value[0])
            print (outTFile0.GetName())
            outNentries0 = int(outTFile0.Get('GENERAL/events').GetEntries())
            outInstance0._setMetadata(outInstance0._value,{'nentries': outNentries0})
            
            outInstance1 = self.conf.dataDictionary[list(self._output)[1]]
            outTFile1 = TFile(outInstance1._value[0])
            print (outTFile1.GetName())
            outNentries1 = int(outTFile1.Get('GENERAL/events').GetEntries())
            outInstance1._setMetadata(outInstance1._value,{'nentries': outNentries1})
 
        if ('doDeadStrip' in runArgs['part']._value or 'doDeadChip' in runArgs['part']._value or 'doQuietStrip' in runArgs['part']._value or 'doQuietChip' in runArgs['part']._value ) and runArgs['splitHitMap']._value == 1:
            outInstance0 = self.conf.dataDictionary[list(self._output)[0]]
            outTFile0 = TFile(outInstance0._value[0])
            print (outTFile0.GetName())
            outNentries0 = int(outTFile0.Get('GENERAL/events').GetEntries())
            outInstance0._setMetadata(outInstance0._value,{'nentries': outNentries0})
            
            outInstance1 = self.conf.dataDictionary[list(self._output)[1]]
            outTFile1 = TFile(outInstance1._value[0])
            print (outTFile1.GetName())
            outNentries1 = int(outTFile1.Get('GENERAL/events').GetEntries())
            outInstance1._setMetadata(outInstance1._value,{'nentries': outNentries1})
 
        if 'doDeadStrip' in runArgs['part']._value and runArgs['splitHitMap']._value != 1:
            pwd=os.getcwd()
            deadFile=pwd+'/'+prefix+'.DeadStripsFile.xml'
            deadSummary=pwd+'/'+prefix+'.DeadSummaryFile.xml'
                            
            numLinesFile = 0
            numLinesSummary = 0
            if os.path.exists(deadFile):
                numLinesFile = sum(1 for line in open(deadFile))
            if os.path.exists(deadSummary):
                numLinesSummary = sum(1 for line in open(deadSummary))
                    
            # if the files exist, but there were no dead strips there won't be COOL file, making the job fail                         
            # remove the COOL file of the list of output files. Clunky, but temporal fix                                                   
                
            if ( numLinesFile == 2  and  numLinesSummary == 20 ):
                dataDic =  self._trf.dataDictionary
                listOfKeys = []
 
                for key in dataDic:
                    if key != 'COOL':
                        listOfKeys.append(key)
 
                redDict = {key:dataDic[key] for key in listOfKeys}
                self._trf._dataDictionary = redDict
 
        if 'doDeadChip' in runArgs['part']._value and runArgs['splitHitMap']._value != 1:
            pwd=os.getcwd()
            deadFile=pwd+'/'+prefix+'.DeadChipsFile.xml'
            deadSummary=pwd+'/'+prefix+'.DeadSummaryFile.xml'
 
            numLinesFile = 0
            numLinesSummary = 0
            if os.path.exists(deadFile):
                numLinesFile = sum(1 for line in open(deadFile))
            if os.path.exists(deadSummary):
                numLinesSummary = sum(1 for line in open(deadSummary))
 
            # if the files exist, but there were no dead strips there won't be COOL file, making the job fail              
            # remove the COOL file of the list of output files. Clunky, but temporal fix                                                      
 
            if ( numLinesFile == 2 and numLinesSummary == 20 ):
                dataDic =  self._trf.dataDictionary
                listOfKeys = []
 
                for key in dataDic:
                    if key != 'COOL':
                        listOfKeys.append(key)
 
                redDict = {key:dataDic[key] for key in listOfKeys}
                self._trf._dataDictionary = redDict
 
        if 'doQuietStrip' in runArgs['part']._value and runArgs['splitHitMap']._value != 1:
            pwd=os.getcwd()
            deadFile=pwd+'/'+prefix+'.QuietStripsFile.xml'
            deadSummary=pwd+'/'+prefix+'.QuietSummaryFile.xml'
                            
            numLinesFile = 0
            numLinesSummary = 0
            if os.path.exists(deadFile):
                numLinesFile = sum(1 for line in open(deadFile))
            if os.path.exists(deadSummary):
                numLinesSummary = sum(1 for line in open(deadSummary))
                    
            # if the files exist, but there were no dead strips there won't be COOL file, making the job fail                         
            # remove the COOL file of the list of output files. Clunky, but temporal fix                                                   
                
            if ( numLinesFile == 2  and  numLinesSummary == 20 ):
                dataDic =  self._trf.dataDictionary
                listOfKeys = []
 
                for key in dataDic:
                    if key != 'COOL':
                        listOfKeys.append(key)
 
                redDict = {key:dataDic[key] for key in listOfKeys}
                self._trf._dataDictionary = redDict
 
        if 'doQuietChip' in runArgs['part']._value and runArgs['splitHitMap']._value != 1:
            pwd=os.getcwd()
            deadFile=pwd+'/'+prefix+'.QuietChipsFile.xml'
            deadSummary=pwd+'/'+prefix+'.QuietSummaryFile.xml'
 
            numLinesFile = 0
            numLinesSummary = 0
            if os.path.exists(deadFile):
                numLinesFile = sum(1 for line in open(deadFile))
            if os.path.exists(deadSummary):
                numLinesSummary = sum(1 for line in open(deadSummary))
 
            # if the files exist, but there were no dead strips there won't be COOL file, making the job fail              
            # remove the COOL file of the list of output files. Clunky, but temporal fix                                                      
 
            if ( numLinesFile == 2 and numLinesSummary == 20 ):
                dataDic =  self._trf.dataDictionary
                listOfKeys = []
 
                for key in dataDic:
                    if key != 'COOL':
                        listOfKeys.append(key)
 
                redDict = {key:dataDic[key] for key in listOfKeys}
                self._trf._dataDictionary = redDict
 
        if prefix != '':
            try:
                if runArgs['splitHitMap']._value !=1 and 'COOL' in listOfKeys:
                    os.rename('mycool.db',prefix+'.mycool.db')                
                if runArgs['splitHitMap']._value == 2:
                    os.rename('SCTHitMaps.root',prefix+'.SCTHitMaps.root')
                    if 'doNoisyStrip' in runArgs['part']._value:
                        os.rename('SCTLB.root',prefix+'.SCTLB.root')
                    if ('doDeadStrip' in runArgs['part']._value or 'doDeadChip' in runArgs['part']._value or 'doQuietStrip' in runArgs['part']._value or 'doQuietChip' in runArgs['part']._value ):
                        os.rename('SCTBSErrors.root',prefix+'.SCTBSErrors.root')
            except OSError:
                self._echologger.warning('failed to rename DB, ROOT or LOG file.' )
            
        super(SCTCalibExecutor, self).postExecute()
    

preExecute()

def sct_calib_tf.SCTCalibExecutor.preExecute	(		self,
			input = set(),
			output = set()
	)

Execute runInfo, set environment and check inputtype

Definition at line 319 of file sct_calib_tf.py.

    def preExecute(self, input=set(), output=set()):
 
        """ Execute runInfo, set environment and check inputtype"""
        # Execute runInfo.py
        runArgs=self.conf._argdict
 
        checkFileList(runArgs['input'])
        namelist=[]
        for i in range(0,len(dsDict['input'])):
            namelist.append(dsDict['input'][i]['file'])
            
        self.conf.addToArgdict('inputNames', trfArgClasses.argList(namelist))
 
        nName=namelist[0].count('/')
        fileName=namelist[0].split('/')[nName]
        projectName=str(fileName.split('.')[0])
 
        
 
        if 'doRunInfo' not in runArgs:
            self.conf.addToArgdict('doRunInfo', trfArgClasses.argBool(False))
        else:
            if runArgs['doRunInfo']._value:
                import SCT_CalibAlgs.runInfo as runInfo
 
                print ("RunNumber for the runInfo = ", str(RunNumber), " ", Stream)
                runInfo.main(RunNumber, projectName)
 
        if 'splitHitMap' not in runArgs:
            self.conf.addToArgdict('splitHitMap', trfArgClasses.argInt(0))
        if 'doRunSelector' not in runArgs:
            self.conf.addToArgdict('doRunSelector', trfArgClasses.argBool(False))
 
        
        if 'EventNumber' not in runArgs:
            self.conf.addToArgdict('EventNumber', trfArgClasses.argInt(0))
 
        # Set STAGE_SVCCLASS
        if SvcClass != '' and SvcClass is not None:
            os.environ['STAGE_SVCCLASS']=SvcClass
 
        # Check input type
        inputtype=dsDict['input'][0]['dataset'].split('.')[4]
        print ("Input type = ", inputtype)
        self.conf.addToArgdict('InputType', trfArgClasses.argString(inputtype))
 
        # check which parts to be run
        if 'part' not in runArgs:
            self.conf.addToArgdict('part', trfArgClasses.argString('doNoisyStrip'))
 
        part=runArgs['part']._value
 
        for ipart in part:
            if ipart not in ['doNoisyStrip','doNoiseOccupancy','doDeadChip','doDeadStrip','doQuietChip','doQuietStrip','doHV','doBSErrorDB','doRawOccupancy','doEfficiency','doLorentzAngle','doNoisyLB']:
                self._errMsg = 'Argument part=%s does not match any of the possible candidates' % ipart
                raise trfExceptions.TransformValidationException(trfExit.nameToCode('TRF_ARG_ERRO'), self._errMsg)
 
        # get prefix
        if 'prefix' not in runArgs:
            self.conf.addToArgdict('prefix', trfArgClasses.argString(''))
 
        prefix=runArgs['prefix']._value
 
        # set job number
        jobnb=''
        # find seperator for jobnumber
        if prefix != '' : 
            sep=prefix.find('._')
            if ( sep != -1 ) :
                jobnb=prefix[sep+1:]
            elif ( prefix.rfind('#_') != -1 ):
                sep=prefix.rfind('#_')
                jobnb=prefix[sep+1:]
            
            # find seperator for prefix
            sep=prefix.find('#')
            if (sep != -1) :
                prefix=prefix[:sep]
            elif (prefix.find('._') != -1):
                sep=prefix.rfind('._')
                prefix=prefix[:sep]
 
            # set prefix and jobnumber
            prefix+='.'+jobnb
            runArgs['prefix']._value = prefix
 
        # When ATLAS is NOT in standby the SCT is, the hitmap root files have 0 events,
        # even though the calibration_SCTNoise streams has 10k+ events.
        # If the noisy strips task is generated, the jobs will fail. A.N has implemented
        # a condition a t0 level so they won't be defined. However,
        # when runSelector uses AtlRunQuery to look for the runs that have 10k+ events
        # in the calibration_SCTNoise stream, those runs that failed or were skipped
        # will appear as waiting to be uploaded, making the rest keep on hold.
 
        # We include a protection against those cases: if the summed number of events
        # of hitmap files is <10k, we don't execute the noisy strips. Rather, we exit
        # with 'success' status, so the job won't fail at t0, and update the value
        # of the last run uploaded as if this run had been uploaded, to avoid the
        # next run being indefinitely on hold
        # print 'Number of events: ', NumberOfEvents
 
        if ('doNoisyStrip' in part or 'doDeadStrip' in part or 'doDeadChip' in part or 'doQuietStrip' in part or 'doQuietChip' in part) and runArgs['splitHitMap']._value==2 and NumberOfEvents<1:
            self._isValidated = True
            self._trf._exitCode = 0
            self._trf._exitMsg = 'Noisy/dead/quiet strips/chips trying to read root files with 0 events. Gracefully exit and update lastRun counter to %s' %(RunNumber)
 
            updateLastRun(RunNumber)
            emptyDic = {}
            self._trf._dataDictionary = emptyDic
 
            resetTrfSignalHandlers()
            self._trf.generateReport(fast=True)
            sys.exit(0)
 
        if jobnb != '':
            self.conf.addToArgdict('JobNumber', trfArgClasses.argString(jobnb))
 
        # get RunNumber from datasetName
        if not RunNumber == -1:
            self.conf.addToArgdict('RunNumber', trfArgClasses.argInt(RunNumber))
        if not Stream == '':
            self.conf.addToArgdict('Stream', trfArgClasses.argString(Stream))
 
        # Do other prerun actions
        super(SCTCalibExecutor, self).preExecute(input,output)
        

validate()

def sct_calib_tf.SCTCalibExecutor.validate

(

self

)

Definition at line 716 of file sct_calib_tf.py.

    def validate(self):
        self._hasValidated = True
        deferredException = None
 
        if 'ignorePatterns' in self.conf._argdict:
            igPat = self.conf.argdict['ignorePatterns'].value
        else:
            igPat = []
        if 'ignoreFiles' in self.conf._argdict:
            ignorePatterns = trfValidation.ignorePatterns(files = self.conf._argdict['ignoreFiles'].value, extraSearch=igPat)
        elif self._errorMaskFiles is not None:
            ignorePatterns = trfValidation.ignorePatterns(files = self._errorMaskFiles, extraSearch=igPat)
        else:
            ignorePatterns = trfValidation.ignorePatterns(files = athenaExecutor._defaultIgnorePatternFile, extraSearch=igPat)
       
        # Now actually scan my logfile
        msg.info('Scanning logfile {0} for errors'.format(self._logFileName))
        self._logScan = trfValidation.athenaLogFileReport(logfile = self._logFileName, ignoreList = ignorePatterns)
        worstError = self._logScan.worstError()
 
        # In general we add the error message to the exit message, but if it's too long then don't do
        # that and just say look in the jobReport
        if worstError['firstError']:
            if len(worstError['firstError']['message']) > athenaExecutor._exitMessageLimit:
                if 'CoreDumpSvc' in worstError['firstError']['message']:
                    exitErrorMessage = "Core dump at line {0} (see jobReport for further details)".format(worstError['firstError']['firstLine'])
                elif 'G4Exception' in worstError['firstError']['message']:
                    exitErrorMessage = "G4 exception at line {0} (see jobReport for further details)".format(worstError['firstError']['firstLine'])
                else:
                    exitErrorMessage = "Long {0} message at line {1} (see jobReport for further details)".format(worstError['level'], worstError['firstError']['firstLine'])
            else:
                exitErrorMessage = "Logfile error in {0}: \"{1}\"".format(self._logFileName, worstError['firstError']['message'])
        else:
            exitErrorMessage = "Error level {0} found (see athena logfile for details)".format(worstError['level'])
 
        # If we failed on the rc, then abort now
        if deferredException is not None:
            # Add any logfile information we have
            if worstError['nLevel'] >= stdLogLevels['ERROR']:
                deferredException.errMsg = deferredException.errMsg + "; {0}".format(exitErrorMessage)
            raise deferredException
       
        # ignore instances of "unknown offline id..."
        # less than ~10/event are admisible
        # if > 10/event, event is skipped in SCT_CalibEventInfo
 
        if worstError['firstError'] is not None:
            if 'ERROR Unknown offlineId for OnlineId' in worstError['firstError']['message']:
                worstError['nLevel'] = 30
                worstError['level'] = 'WARNING'
                
        # Very simple: if we get ERROR or worse, we're dead, except if ignoreErrors=True
        if worstError['nLevel'] == stdLogLevels['ERROR'] and ('ignoreErrors' in self.conf._argdict and self.conf._argdict['ignoreErrors'].value is True):
            msg.warning('Found ERRORs in the logfile, but ignoring this as ignoreErrors=True (see jobReport for details)')
        elif worstError['nLevel'] >= stdLogLevels['ERROR']:
            self._isValidated = False
            msg.error('Fatal error in athena logfile (level {0})'.format(worstError['level']))
            raise trfExceptions.TransformLogfileErrorException(trfExit.nameToCode('TRF_EXEC_LOGERROR'),
                                                               ' Fatal error in athena logfile: "{0}"'.format(exitErrorMessage))
 
        # Must be ok if we got here!
        msg.info('Executor {0} has validated successfully'.format(self.name))
        self._isValidated = True
 

Member Data Documentation

_errMsg

sct_calib_tf.SCTCalibExecutor._errMsg

private

Try to avoid validation of output files self.skipOutputFileValidation=True.

This is a try to set event number manually if run over HIST files

Definition at line 374 of file sct_calib_tf.py.

_hasValidated

sct_calib_tf.SCTCalibExecutor._hasValidated

private

Definition at line 717 of file sct_calib_tf.py.

_isValidated

sct_calib_tf.SCTCalibExecutor._isValidated

private

Definition at line 422 of file sct_calib_tf.py.

_logScan

sct_calib_tf.SCTCalibExecutor._logScan

private

Definition at line 733 of file sct_calib_tf.py.

---

The documentation for this class was generated from the following file:

• sct_calib_tf.py