python.trfValidation.athenaLogFileReport Class Reference

Logfile suitable for scanning logfiles with an athena flavour, i.e., lines of the form "SERVICE LOGLEVEL MESSAGE". More...

Inheritance diagram for python.trfValidation.athenaLogFileReport:

Collaboration diagram for python.trfValidation.athenaLogFileReport:

Public Member Functions
def	__init__ (self, logfile, substepName=None, msgLimit=10, msgDetailLevel=stdLogLevels['ERROR'], ignoreList=None)
	Class constructor. More...
def	python (self)
	Produce a python dictionary summary of the log file report for inclusion in the executor report. More...
def	resetReport (self)
def	knowledgeFileHandler (self, knowledgefile)
	Generally, a knowledge file consists of non-standard logging error/abnormal lines which are left out during log scan and could help diagnose job failures. More...
def	scanLogFile (self, resetReport=False)
def	dbMonitor (self)
	Return data volume and time spend to retrieve information from the database. More...
def	worstError (self)
	Return the worst error found in the logfile (first error of the most serious type) More...
def	firstError (self, floor='ERROR')
	Return the first error found in the logfile above a certain loglevel. More...
def	eventLoopWarnings (self)
def	moreDetails (self, log, firstline, firstLineCount, knowledgeFile, offset=0)
def	coreDumpSvcParser (self, log, lineGenerator, firstline, firstLineCount)
	Attempt to suck a core dump report from the current logfile This function scans logs in two different directions: 1) downwards, to exctract information after CoreDumpSvc; and 2) upwards, to find abnormal lines. More...
def	g494ExceptionParser (self, lineGenerator, firstline, firstLineCount)
def	g4ExceptionParser (self, lineGenerator, firstline, firstLineCount, g4ExceptionLineDepth)
def	pythonExceptionParser (self, log, lineGenerator, firstline, firstLineCount)
def	badAllocExceptionParser (self, lineGenerator, firstline, firstLineCount)
def	rootSysErrorParser (self, lineGenerator, firstline, firstLineCount)
def	__str__ (self)
def	scanLogFile (self)
def	firstError (self)

Private Attributes
	_ignoreList
	_regExp
	_metaPat
	_metaData
	_eventLoopWarnings
	_substepName
	_msgLimit
	_levelCounter
	_errorDetails
	_dbbytes
	_dbtime
	_logfile
	_msgDetails
	_re

Detailed Description

Logfile suitable for scanning logfiles with an athena flavour, i.e., lines of the form "SERVICE LOGLEVEL MESSAGE".

Definition at line 211 of file trfValidation.py.

Constructor & Destructor Documentation

__init__()

def python.trfValidation.athenaLogFileReport.__init__	(		self,
			logfile,
			substepName = None,
			msgLimit = 10,
			msgDetailLevel = stdLogLevels['ERROR'],
			ignoreList = None
	)

Class constructor.

Parameters

logfile	Logfile (or list of logfiles) to scan
substepName	Name of the substep executor, that has requested this log scan
msgLimit	The number of messages in each category on which a

Definition at line 216 of file trfValidation.py.

    def __init__(self, logfile, substepName=None, msgLimit=10, msgDetailLevel=stdLogLevels['ERROR'], ignoreList=None):
        if ignoreList:
            self._ignoreList = ignoreList
        else:
            self._ignoreList = ignorePatterns()
 
        
        self._regExp = re.compile(r'(?P<service>[^\s]+\w)(.*)\s+(?P<level>' + '|'.join(stdLogLevels) + r')\s+(?P<message>.*)')
 
        self._metaPat = re.compile(r"MetaData:\s+(.*?)\s*=\s*(.*)$")
        self._metaData = {}
        self._eventLoopWarnings = []
        self._substepName = substepName
        self._msgLimit = msgLimit
 
        self.resetReport()
 
        super(athenaLogFileReport, self).__init__(logfile, msgLimit, msgDetailLevel)
 

Member Function Documentation

__str__()

def python.trfValidation.athenaLogFileReport.__str__

(

self

)

Reimplemented from python.trfValidation.logFileReport.

Definition at line 705 of file trfValidation.py.

    def __str__(self):
        return str(self._levelCounter) + str(self._errorDetails)
 
 

badAllocExceptionParser()

def python.trfValidation.athenaLogFileReport.badAllocExceptionParser	(		self,
			lineGenerator,
			firstline,
			firstLineCount
	)

Definition at line 693 of file trfValidation.py.

    def badAllocExceptionParser(self, lineGenerator, firstline, firstLineCount):
        badAllocExceptionReport = 'terminate after \'std::bad_alloc\'.'
 
        msg.debug('Identified bad_alloc - adding to error detail report')
        self._levelCounter['CATASTROPHE'] += 1
        self._errorDetails['CATASTROPHE'].append({'message': badAllocExceptionReport, 'firstLine': firstLineCount, 'count': 1})
 

coreDumpSvcParser()

def python.trfValidation.athenaLogFileReport.coreDumpSvcParser	(		self,
			log,
			lineGenerator,
			firstline,
			firstLineCount
	)

Attempt to suck a core dump report from the current logfile This function scans logs in two different directions: 1) downwards, to exctract information after CoreDumpSvc; and 2) upwards, to find abnormal lines.

Note

: Current downwards scan just eats lines until a 'normal' line is seen. There is a slight problem here in that the end of core dump trigger line will not get parsed TODO: fix this (OTOH core dump is usually the very last thing and fatal!)

Definition at line 539 of file trfValidation.py.

    def coreDumpSvcParser(self, log, lineGenerator, firstline, firstLineCount):
        _eventCounter = _run = _event = _currentAlgorithm = _functionLine = _currentFunction = None
        coreDumpReport = 'Core dump from CoreDumpSvc'
        # Number of lines to ignore above 'core dump' when looking for abnormal lines
        offset = 1
        coreDumpDetailsReport = {}
 
        for line, linecounter in lineGenerator:
            m = self._regExp.match(line)
            if m is None:
                if 'Caught signal 11(Segmentation fault)' in line:
                    coreDumpReport = 'Segmentation fault'
                if 'Event counter' in line:
                    _eventCounter = line
 
                #Lookup: 'EventID: [Run,Evt,Lumi,Time,BunchCross,DetMask] = [267599,7146597,1,1434123751:0,0,0x0,0x0,0x0]'
                if 'EventID' in line:
                    match = re.findall(r'\[.*?\]', line)
                    if match and match.__len__() >= 2:      # Assuming the line contains at-least one key-value pair.
                        brackets = "[]"
                        commaDelimer = ','
                        keys = (match[0].strip(brackets)).split(commaDelimer)
                        values = (match[1].strip(brackets)).split(commaDelimer)
 
                        if 'Run' in keys:
                            _run = 'Run: ' + values[keys.index('Run')]
 
                        if 'Evt' in keys:
                            _event = 'Evt: ' + values[keys.index('Evt')]
 
                if 'Current algorithm' in line:
                    _currentAlgorithm = line
                if '<signal handler called>' in line:
                    _functionLine = linecounter+1
                if _functionLine and linecounter is _functionLine:
                    if ' in ' in line:
                        _currentFunction = 'Current Function: ' + line.split(' in ')[1].split()[0]
                    else:
                        _currentFunction = 'Current Function: ' + line.split()[1]
            else:
                # Can this be done - we want to push the line back into the generator to be
                # reparsed in the normal way (might need to make the generator a class with the
                # __exec__ method supported (to get the line), so that we can then add a
                # pushback onto an internal FIFO stack
                # lineGenerator.pushback(line)
                break
        _eventCounter = 'Event counter: unknown' if not _eventCounter else _eventCounter
        _run = 'Run: unknown' if not _run else _run
        _event = 'Evt: unknown' if not _event else _event
        _currentAlgorithm = 'Current algorithm: unknown' if not _currentAlgorithm else _currentAlgorithm
        _currentFunction = 'Current Function: unknown' if not _currentFunction else _currentFunction
        coreDumpReport = '{0}: {1}; {2}; {3}; {4}; {5}'.format(coreDumpReport, _eventCounter, _run, _event, _currentAlgorithm, _currentFunction)
 
        coreDumpDetailsReport = self.moreDetails(log, firstline, firstLineCount, 'knowledgeFile.db', offset)
        abnormalLines = coreDumpDetailsReport['abnormalLines']
 
        # concatenate an extract of first seen abnormal line to the core dump message
        if 'message0' in abnormalLines.keys():
            coreDumpReport += '; Abnormal line seen just before core dump: ' + abnormalLines['message0'][0:30] + '...[truncated] ' + '(see the jobReport)'
 
        # Core dumps are always fatal...
        msg.debug('Identified core dump - adding to error detail report')
        self._levelCounter['FATAL'] += 1
        self._errorDetails['FATAL'].append({'moreDetails': coreDumpDetailsReport, 'message': coreDumpReport, 'firstLine': firstLineCount, 'count': 1})
 
 

dbMonitor()

def python.trfValidation.athenaLogFileReport.dbMonitor

(

self

)

Return data volume and time spend to retrieve information from the database.

Definition at line 445 of file trfValidation.py.

    def dbMonitor(self):
        return {'bytes' : self._dbbytes, 'time' : self._dbtime} if self._dbbytes > 0 or self._dbtime > 0 else None
 

eventLoopWarnings()

def python.trfValidation.athenaLogFileReport.eventLoopWarnings

(

self

)

Definition at line 478 of file trfValidation.py.

    def eventLoopWarnings(self):
        eventLoopWarnings = []
        for item in self._eventLoopWarnings:
            if item in [element['item'] for element in eventLoopWarnings]:
                continue
            count = self._eventLoopWarnings.count(item)
            eventLoopWarnings.append({'item':item, 'count': count})
        return eventLoopWarnings          
 

firstError() [1/2]

def python.trfValidation.logFileReport.firstError (   self )

inherited

Definition at line 201 of file trfValidation.py.

    def firstError(self):
        pass
 

firstError() [2/2]

def python.trfValidation.athenaLogFileReport.firstError	(		self,
			floor = 'ERROR'
	)

Return the first error found in the logfile above a certain loglevel.

Definition at line 464 of file trfValidation.py.

    def firstError(self, floor='ERROR'):
        firstLine = firstError = None
        firstLevel = stdLogLevels[floor]
        firstName = floor
        for lvl, count in self._levelCounter.items():
            if (count > 0 and stdLogLevels.get(lvl, 0) >= stdLogLevels[floor] and
                (firstError is None or self._errorDetails[lvl][0]['firstLine'] < firstLine)):
                firstLine = self._errorDetails[lvl][0]['firstLine']
                firstLevel = stdLogLevels[lvl]
                firstName = lvl
                firstError = self._errorDetails[lvl][0]
 
        return {'level': firstName, 'nLevel': firstLevel, 'firstError': firstError}
 

g494ExceptionParser()

def python.trfValidation.athenaLogFileReport.g494ExceptionParser	(		self,
			lineGenerator,
			firstline,
			firstLineCount
	)

Definition at line 605 of file trfValidation.py.

    def g494ExceptionParser(self, lineGenerator, firstline, firstLineCount):
        g4Report = firstline
        g4lines = 1
        if 'Aborting execution' not in g4Report:
            for line, linecounter in lineGenerator:
                g4Report += os.linesep + line
                g4lines += 1
                # Test for the closing string
                if '*** ' in line:
                    break
                if g4lines >= 25:
                    msg.warning('G4 exception closing string not found within {0} log lines of line {1}'.format(g4lines, firstLineCount))
                    break
 
        # G4 exceptions can be fatal or they can be warnings...
        msg.debug('Identified G4 exception - adding to error detail report')
        if "just a warning" in g4Report:
            if self._levelCounter['WARNING'] <= self._msgLimit:
                self._levelCounter['WARNING'] += 1
                self._errorDetails['WARNING'].append({'message': g4Report, 'firstLine': firstLineCount, 'count': 1})
            elif self._levelCounter['WARNING'] == self._msgLimit + 1:
                msg.warning("Found message number {0} at level WARNING - this and further messages will be supressed from the report".format(self._levelCounter['WARNING']))
        else:
            self._levelCounter['FATAL'] += 1
            self._errorDetails['FATAL'].append({'message': g4Report, 'firstLine': firstLineCount, 'count': 1})
 

g4ExceptionParser()

def python.trfValidation.athenaLogFileReport.g4ExceptionParser	(		self,
			lineGenerator,
			firstline,
			firstLineCount,
			g4ExceptionLineDepth
	)

Definition at line 631 of file trfValidation.py.

    def g4ExceptionParser(self, lineGenerator, firstline, firstLineCount, g4ExceptionLineDepth):
        g4Report = firstline
        g4lines = 1
        for line, linecounter in lineGenerator:
            g4Report += os.linesep + line
            g4lines += 1
            # Test for the closing string
            if 'G4Exception-END' in line:
                break
            if g4lines >= g4ExceptionLineDepth:
                msg.warning('G4 exception closing string not found within {0} log lines of line {1}'.format(g4lines, firstLineCount))
                break
 
        # G4 exceptions can be fatal or they can be warnings...
        msg.debug('Identified G4 exception - adding to error detail report')
        if "-------- WWWW -------" in g4Report:
            if self._levelCounter['WARNING'] <= self._msgLimit:
                self._levelCounter['WARNING'] += 1
                self._errorDetails['WARNING'].append({'message': g4Report, 'firstLine': firstLineCount, 'count': 1})
            elif self._levelCounter['WARNING'] == self._msgLimit + 1:
                msg.warning("Found message number {0} at level WARNING - this and further messages will be supressed from the report".format(self._levelCounter['WARNING'])) 
        else:
            self._levelCounter['FATAL'] += 1
            self._errorDetails['FATAL'].append({'message': g4Report, 'firstLine': firstLineCount, 'count': 1})
 
 

knowledgeFileHandler()

def python.trfValidation.athenaLogFileReport.knowledgeFileHandler	(		self,
			knowledgefile
	)

Generally, a knowledge file consists of non-standard logging error/abnormal lines which are left out during log scan and could help diagnose job failures.

Definition at line 268 of file trfValidation.py.

    def knowledgeFileHandler(self, knowledgefile):
        # load abnormal/error line(s) from the knowledge file(s)
        linesList = []
        fullName = trfUtils.findFile(os.environ['DATAPATH'], knowledgefile)
        if not fullName:
            msg.warning('Knowledge file {0} could not be found in DATAPATH'.format(knowledgefile))
        else:
            try:
                with open(fullName) as knowledgeFileHandle:
                    msg.debug('Opened knowledge file {0} from here: {1}'.format(knowledgefile, fullName))
 
                    for line in knowledgeFileHandle:
                        if line.startswith('#') or line == '' or line =='\n':
                            continue
                        line = line.rstrip('\n')
                        linesList.append(line)
            except OSError as e:
                msg.warning('Failed to open knowledge file {0}: {1}'.format(fullName, e))
        return linesList
 

moreDetails()

def python.trfValidation.athenaLogFileReport.moreDetails	(		self,
			log,
			firstline,
			firstLineCount,
			knowledgeFile,
			offset = 0
	)

Definition at line 487 of file trfValidation.py.

    def moreDetails(self, log, firstline, firstLineCount, knowledgeFile, offset=0):
        # Look for "abnormal" and "last normal" line(s)
        # Make a list of last e.g. 50 lines before core dump
        abnormalLinesList = self.knowledgeFileHandler(knowledgeFile)
        linesToBeScanned = 50
        seenAbnormalLines = []
        abnormalLinesReport = {}
        lastNormalLineReport = {}
 
        linesList = [] 
        myGen = trfUtils.lineByLine(log)
        for line, linecounter in myGen:
            if linecounter in range(firstLineCount - linesToBeScanned, firstLineCount-offset):
                linesList.append([linecounter, line])
            elif linecounter == firstLineCount:
                break
 
        for linecounter, line in reversed(linesList):
            if re.findall(r'|'.join(abnormalLinesList), line):
                seenLine = False
                for dic in seenAbnormalLines:
                    # count repetitions or similar (e.g. first 15 char) abnormal lines
                    if dic['message'] == line or dic['message'][0:15] == line[0:15]:
                        dic['count'] += 1
                        seenLine = True
                        break
                if seenLine is False:
                    seenAbnormalLines.append({'message': line, 'firstLine': linecounter, 'count': 1})
            else:
                if line != '':
                    lastNormalLineReport = {'message': line, 'firstLine': linecounter, 'count': 1}
                    break
                else:
                    continue
 
        # Write the list of abnormal lines into the abnormalLinesReport dictionary
        # The keys of each abnormal line have a number suffix starting with 0
        # e.g., first abnormal line's keys are :{'mesage0', 'firstLine0', 'count0'}        
 
        for a in range(len(seenAbnormalLines)):
            abnormalLinesReport.update({'message{0}'.format(a): seenAbnormalLines[a]['message'], 'firstLine{0}'.format(a): seenAbnormalLines[a]['firstLine'],
                                        'count{0}'.format(a): seenAbnormalLines[a]['count']})
 
        return {'abnormalLines': abnormalLinesReport, 'lastNormalLine': lastNormalLineReport}
 
 

python()

def python.trfValidation.athenaLogFileReport.python

(

self

)

Produce a python dictionary summary of the log file report for inclusion in the executor report.

Definition at line 242 of file trfValidation.py.

    def python(self):
        errorDict = {'countSummary': {}, 'details': {}}
        for level, count in self._levelCounter.items():
            errorDict['countSummary'][level] = count
            if self._levelCounter[level] > 0 and len(self._errorDetails[level]) > 0:
                errorDict['details'][level] = []
                for error in self._errorDetails[level]:
                    errorDict['details'][level].append(error)
        return errorDict
 

pythonExceptionParser()

def python.trfValidation.athenaLogFileReport.pythonExceptionParser	(		self,
			log,
			lineGenerator,
			firstline,
			firstLineCount
	)

Definition at line 657 of file trfValidation.py.

    def pythonExceptionParser(self, log, lineGenerator, firstline, firstLineCount):
        pythonExceptionReport = ""
        lastLine = firstline
        lastLine2 = firstline
        pythonErrorLine = firstLineCount
        pyLines = 1
        for line, linecounter in lineGenerator:
            if 'Py:Athena' in line and 'INFO leaving with code' in line:
                if len(lastLine)> 0:
                    pythonExceptionReport = lastLine
                    pythonErrorLine = linecounter-1
                else: # Sometimes there is a blank line after the exception
                    pythonExceptionReport = lastLine2
                    pythonErrorLine = linecounter-2
                break
            if pyLines >= 25:
                msg.warning('Could not identify python exception correctly scanning {0} log lines after line {1}'.format(pyLines, firstLineCount))
                pythonExceptionReport = "Unable to identify specific exception"
                pythonErrorLine = firstLineCount
                break
            lastLine2 = lastLine
            lastLine = line
            pyLines += 1
 
        pythonExceptionDetailsReport = self.moreDetails(log, firstline, firstLineCount, 'knowledgeFile.db')
        abnormalLines = pythonExceptionDetailsReport['abnormalLines']
 
        # concatenate an extract of first seen abnormal line to pythonExceptionReport
        if 'message0' in abnormalLines.keys():
            pythonExceptionReport += '; Abnormal line seen just before python exception: ' + abnormalLines['message0'][0:30] + '...[truncated] ' + '(see the jobReport)'
 
        msg.debug('Identified python exception - adding to error detail report')
        self._levelCounter['FATAL'] += 1
        self._errorDetails['FATAL'].append({'moreDetails': pythonExceptionDetailsReport, 'message': pythonExceptionReport, 'firstLine': pythonErrorLine, 'count': 1})
 
 

resetReport()

def python.trfValidation.athenaLogFileReport.resetReport

(

self

)

Reimplemented from python.trfValidation.logFileReport.

Definition at line 252 of file trfValidation.py.

    def resetReport(self):
        self._levelCounter = {}
        for level in list(stdLogLevels) + ['UNKNOWN', 'IGNORED']:
            self._levelCounter[level] = 0
 
        self._errorDetails = {}
        self._eventLoopWarnings = []
        for level in self._levelCounter:
            self._errorDetails[level] = []
            # Format:
            # List of dicts {'message': errMsg, 'firstLine': lineNo, 'count': N}
        self._dbbytes = 0
        self._dbtime  = 0.0
 

rootSysErrorParser()

def python.trfValidation.athenaLogFileReport.rootSysErrorParser	(		self,
			lineGenerator,
			firstline,
			firstLineCount
	)

Definition at line 700 of file trfValidation.py.

    def rootSysErrorParser(self, lineGenerator, firstline, firstLineCount):
        msg.debug('Identified ROOT IO problem - adding to error detail report')
        self._levelCounter['FATAL'] += 1
        self._errorDetails['FATAL'].append({'message': firstline, 'firstLine': firstLineCount, 'count': 1})
 

scanLogFile() [1/2]

def python.trfValidation.logFileReport.scanLogFile (   self )

inherited

Definition at line 195 of file trfValidation.py.

    def scanLogFile(self):
        pass
 

scanLogFile() [2/2]

def python.trfValidation.athenaLogFileReport.scanLogFile	(		self,
			resetReport = False
	)

Definition at line 288 of file trfValidation.py.

    def scanLogFile(self, resetReport=False):
        
        nonStandardErrorsList = self.knowledgeFileHandler('nonStandardErrors.db')
 
        if resetReport:
            self.resetReport()
 
        for log in self._logfile:
            msg.debug('Now scanning logfile {0}'.format(log))
            seenNonStandardError = ''
            customLogParser = None
            if log == 'log.generate':
                from EvgenProdTools.EvgenParserTool import evgenParserTool
                customLogParser = evgenParserTool()
            # N.B. Use the generator so that lines can be grabbed by subroutines, e.g., core dump svc reporter
            try:
                myGen = trfUtils.lineByLine(log, substepName=self._substepName)
            except IOError as e:
                msg.error('Failed to open transform logfile {0}: {1:s}'.format(log, e))
                # Return this as a small report
                self._levelCounter['ERROR'] = 1
                self._errorDetails['ERROR'] = {'message': str(e), 'firstLine': 0, 'count': 1}
                return
            # Detect whether we are in the event loop part of the log file
            inEventLoop = False
            for line, lineCounter in myGen:  
                if '===>>>  start processing event' in line: inEventLoop = True
                if 'Application Manager Stopped successfully' in line: inEventLoop = False
 
                # In case we have enabled a custom log parser, run the line through it first
                if customLogParser is not None:
                    customLogParser.processLine(line)
                # Search for metadata strings
                m = self._metaPat.search(line)
                if m is not None:
                    key, value = m.groups()
                    self._metaData[key] = value
 
                m = self._regExp.match(line)
                if m is None:
                    # We didn't manage to get a recognised standard line from the file
                    # But we can check for certain other interesting things, like core dumps
                    if 'Core dump from CoreDumpSvc' in line:
                        msg.warning('Detected CoreDumpSvc report - activating core dump svc grabber')
                        self.coreDumpSvcParser(log, myGen, line, lineCounter)
                        continue
                    # Add the G4 exceptipon parsers
                    if 'G4Exception-START' in line:
                        msg.warning('Detected G4 exception report - activating G4 exception grabber')
                        self.g4ExceptionParser(myGen, line, lineCounter, 40)
                        continue
                    if '*** G4Exception' in line:
                        msg.warning('Detected G4 9.4 exception report - activating G4 exception grabber')
                        self.g494ExceptionParser(myGen, line, lineCounter)
                        continue
                    # Add the python exception parser
                    if 'Shortened traceback (most recent user call last)' in line:
                        msg.warning('Detected python exception - activating python exception grabber')
                        self.pythonExceptionParser(log, myGen, line, lineCounter)
                        continue
                    # Add parser for missed bad_alloc
                    if 'terminate called after throwing an instance of \'std::bad_alloc\'' in line:
                        msg.warning('Detected bad_alloc!')
                        self.badAllocExceptionParser(myGen, line, lineCounter)
                        continue
                    # Parser for ROOT reporting a stale file handle (see ATLASG-448)
                    # Amendment: Generalize the search (see ATLASRECTS-7121)
                    if 'Error in <TFile::ReadBuffer>' in line:
                        self.rootSysErrorParser(myGen, line, lineCounter)
                        continue
 
                    if 'Error in <TFile::WriteBuffer>' in line:
                        self.rootSysErrorParser(myGen, line, lineCounter)
                        continue
                    # Check if the line is among the non-standard logging errors from the knowledge file
                    if any(line in l for l in nonStandardErrorsList):
                        seenNonStandardError = line
                        continue
 
                    msg.debug('Non-standard line in %s: %s', log, line)
                    self._levelCounter['UNKNOWN'] += 1
                    continue
 
                # Line was matched successfully
                fields = {}
                for matchKey in ('service', 'level', 'message'):
                    fields[matchKey] = m.group(matchKey)
                msg.debug('Line parsed as: {0}'.format(fields))
 
                # If this is a WARNING and we passed the start of the event loop, 
                # add it to special list 
                if (fields['level'] == 'WARNING') and inEventLoop:
                    self._eventLoopWarnings.append(fields)
 
                # Check this is not in our ignore list
                ignoreFlag = False
                for ignorePat in self._ignoreList.structuredPatterns:
                    serviceMatch = ignorePat['service'].match(fields['service'])
                    levelMatch = (ignorePat['level'] == "" or ignorePat['level'] == fields['level'])
                    messageMatch = ignorePat['message'].match(fields['message'])
                    if serviceMatch and levelMatch and messageMatch:
                        msg.info('Error message "{0}" was ignored at line {1} (structured match)'.format(line, lineCounter))
                        ignoreFlag = True
                        break
                if ignoreFlag is False:
                    for searchPat in self._ignoreList.searchPatterns:
                        if searchPat.search(line):
                            msg.info('Error message "{0}" was ignored at line {1} (search match)'.format(line, lineCounter))
                            ignoreFlag = True
                            break
                if ignoreFlag:
                    # Got an ignore - message this to a special IGNORED error
                    fields['level'] = 'IGNORED'
                else:
                    # Some special handling for specific errors (maybe generalise this if
                    # there end up being too many special cases)
                    # Upgrade bad_alloc to CATASTROPHE to allow for better automated handling of
                    # jobs that run out of memory
                    if 'std::bad_alloc' in fields['message']:
                        fields['level'] = 'CATASTROPHE'
 
                # concatenate the seen non-standard logging error to the FATAL
                if fields['level'] == 'FATAL':
                    if seenNonStandardError:
                        line += '; ' + seenNonStandardError
 
                # Count this error
                self._levelCounter[fields['level']] += 1
 
                # Record some error details
                # N.B. We record 'IGNORED' errors as these really should be flagged for fixing
                if fields['level'] == 'IGNORED' or stdLogLevels[fields['level']] >= self._msgDetails:
                    if self._levelCounter[fields['level']] <= self._msgLimit: 
                        detailsHandled = False
                        for seenError in self._errorDetails[fields['level']]:
                            if seenError['message'] == line:
                                seenError['count'] += 1
                                detailsHandled = True
                                break
                        if detailsHandled is False:
                            self._errorDetails[fields['level']].append({'message': line, 'firstLine': lineCounter, 'count': 1})
                    elif self._levelCounter[fields['level']] == self._msgLimit + 1:
                        msg.warning("Found message number {0} at level {1} - this and further messages will be supressed from the report".format(self._levelCounter[fields['level']], fields['level']))
                    else:
                        # Overcounted
                        pass
                if 'Total payload read from IOVDb' in fields['message']:
                    msg.debug("Found COOL payload information at line {0}".format(line))
                    a = re.match(r'(\D+)(?P<bytes>\d+)(\D+)(?P<time>\d+[.]?\d*)(\D+)', fields['message'])
                    self._dbbytes += int(a.group('bytes'))
                    self._dbtime  += float(a.group('time'))
            # Finally, if we have a custom log parser, use it to update the metadata dictionary
            if customLogParser is not None:
                customLogParser.report()
                self._metaData = customLogParser.updateMetadata( self._metaData )
 

worstError()

def python.trfValidation.athenaLogFileReport.worstError

(

self

)

Return the worst error found in the logfile (first error of the most serious type)

Reimplemented from python.trfValidation.logFileReport.

Definition at line 449 of file trfValidation.py.

    def worstError(self):
        worst = stdLogLevels['DEBUG']
        worstName = 'DEBUG'
        for lvl, count in self._levelCounter.items():
            if count > 0 and stdLogLevels.get(lvl, 0) > worst:
                worstName = lvl
                worst = stdLogLevels[lvl]
        if len(self._errorDetails[worstName]) > 0:
            firstError = self._errorDetails[worstName][0]
        else:
            firstError = None
 
        return {'level': worstName, 'nLevel': worst, 'firstError': firstError}
 

Member Data Documentation

_dbbytes

python.trfValidation.athenaLogFileReport._dbbytes

private

Definition at line 263 of file trfValidation.py.

_dbtime

python.trfValidation.athenaLogFileReport._dbtime

private

Definition at line 264 of file trfValidation.py.

_errorDetails

python.trfValidation.athenaLogFileReport._errorDetails

private

Definition at line 257 of file trfValidation.py.

_eventLoopWarnings

python.trfValidation.athenaLogFileReport._eventLoopWarnings

private

Definition at line 231 of file trfValidation.py.

_ignoreList

python.trfValidation.athenaLogFileReport._ignoreList

private

Definition at line 218 of file trfValidation.py.

_levelCounter

python.trfValidation.athenaLogFileReport._levelCounter

private

Definition at line 253 of file trfValidation.py.

_logfile

python.trfValidation.logFileReport._logfile

privateinherited

Definition at line 181 of file trfValidation.py.

_metaData

python.trfValidation.athenaLogFileReport._metaData

private

Definition at line 230 of file trfValidation.py.

_metaPat

python.trfValidation.athenaLogFileReport._metaPat

private

Definition at line 229 of file trfValidation.py.

_msgDetails

python.trfValidation.logFileReport._msgDetails

privateinherited

Definition at line 186 of file trfValidation.py.

_msgLimit

python.trfValidation.athenaLogFileReport._msgLimit

private

Definition at line 233 of file trfValidation.py.

_re

python.trfValidation.logFileReport._re

privateinherited

Definition at line 187 of file trfValidation.py.

_regExp

python.trfValidation.athenaLogFileReport._regExp

private

Note

This is the regular expression match for athena logfile lines Match first strips off any HH:MM:SS prefix the transform has added, then takes the next group of non-whitespace characters as the service, then then matches from the list of known levels, then finally, ignores any last pieces of whitespace prefix and takes the rest of the line as the message

Definition at line 227 of file trfValidation.py.

_substepName

python.trfValidation.athenaLogFileReport._substepName

private

Definition at line 232 of file trfValidation.py.

---

The documentation for this class was generated from the following file:

• trfValidation.py