python.trfValidation.athenaLogFileReport Class Reference

Logfile suitable for scanning logfiles with an athena flavour, i.e., lines of the form "SERVICE LOGLEVEL MESSAGE". More...

Inheritance diagram for python.trfValidation.athenaLogFileReport:

Collaboration diagram for python.trfValidation.athenaLogFileReport:

Public Types
typedef HLT::TypeInformation::for_each_type_c< typenameEDMLIST::map, my_functor, my_result<>, my_arg< HLT::TypeInformation::get_cont, CONTAINER > >::type	result

Public Member Functions
	__init__ (self, logfile, substepName=None, msgLimit=10, msgDetailLevel=stdLogLevels['ERROR'], ignoreList=None)
	Class constructor.
	python (self)
	Produce a python dictionary summary of the log file report for inclusion in the executor report.
	resetReport (self)
	knowledgeFileHandler (self, knowledgefile)
	Generally, a knowledge file consists of non-standard logging error/abnormal lines which are left out during log scan and could help diagnose job failures.
	scanLogFile (self, resetReport=False)
	dbMonitor (self)
	Return data volume and time spend to retrieve information from the database.
	worstError (self)
	Return the worst error found in the logfile (first error of the most serious type)
	firstError (self, floor='ERROR')
	Return the first error found in the logfile above a certain loglevel.
	eventLoopWarnings (self)
	moreDetails (self, log, firstline, firstLineCount, knowledgeFile, offset=0)
	coreDumpSvcParser (self, log, lineGenerator, firstline, firstLineCount)
	Attempt to suck a core dump report from the current logfile This function scans logs in two different directions: 1) downwards, to exctract information after CoreDumpSvc; and 2) upwards, to find abnormal lines.
	g494ExceptionParser (self, lineGenerator, firstline, firstLineCount)
	g4ExceptionParser (self, lineGenerator, firstline, firstLineCount, g4ExceptionLineDepth)
	pythonExceptionParser (self, log, lineGenerator, firstline, firstLineCount)
	badAllocExceptionParser (self, lineGenerator, firstline, firstLineCount)
	rootSysErrorParser (self, lineGenerator, firstline, firstLineCount)
	__str__ (self)

Protected Attributes
	_ignoreList = ignoreList
	_regExp = re.compile(r'(?P<service>[^\s]+\w)(.*)\s+(?P<level>' + '|'.join(stdLogLevels) + r')\s+(?P<message>.*)')
	_metaPat = re.compile(r"MetaData:\s+(.*?)\s*=\s*(.*)$")
dict	_metaData = {}
list	_eventLoopWarnings = []
	_substepName = substepName
dict	_levelCounter = {}
dict	_errorDetails = {}
int	_dbbytes = 0
float	_dbtime = 0.0
list	_logfile = [logfile, ]
	_msgLimit = msgLimit
	_msgDetails = msgDetailLevel
	_re = None

Detailed Description

Logfile suitable for scanning logfiles with an athena flavour, i.e., lines of the form "SERVICE LOGLEVEL MESSAGE".

Definition at line 211 of file trfValidation.py.

Member Typedef Documentation

result

typedef HLT::TypeInformation::for_each_type_c<typenameEDMLIST::map,my_functor,my_result<>,my_arg<HLT::TypeInformation::get_cont,CONTAINER>>::type master_search< typenameEDMLIST::map, HLT::TypeInformation::get_cont, CONTAINER >::result

inherited

Definition at line 90 of file EDM_MasterSearch.h.

Constructor & Destructor Documentation

__init__()

python.trfValidation.athenaLogFileReport.__init__	(		self,
			logfile,
			substepName = None,
			msgLimit = 10,
			msgDetailLevel = stdLogLevels['ERROR'],
			ignoreList = None )

Class constructor.

Parameters

logfile	Logfile (or list of logfiles) to scan
substepName	Name of the substep executor, that has requested this log scan
msgLimit	The number of messages in each category on which a

Definition at line 216 of file trfValidation.py.

    def __init__(self, logfile, substepName=None, msgLimit=10, msgDetailLevel=stdLogLevels['ERROR'], ignoreList=None):
        if ignoreList:
            self._ignoreList = ignoreList
        else:
            self._ignoreList = ignorePatterns()
 
        
        self._regExp = re.compile(r'(?P<service>[^\s]+\w)(.*)\s+(?P<level>' + '|'.join(stdLogLevels) + r')\s+(?P<message>.*)')
 
        self._metaPat = re.compile(r"MetaData:\s+(.*?)\s*=\s*(.*)$")
        self._metaData = {}
        self._eventLoopWarnings = []
        self._substepName = substepName
        self._msgLimit = msgLimit
 
        self.resetReport()
 
        super(athenaLogFileReport, self).__init__(logfile, msgLimit, msgDetailLevel)
 

Member Function Documentation

__str__()

python.trfValidation.athenaLogFileReport.__str__

(

self

)

Definition at line 705 of file trfValidation.py.

    def __str__(self):
        return str(self._levelCounter) + str(self._errorDetails)
 
 

badAllocExceptionParser()

python.trfValidation.athenaLogFileReport.badAllocExceptionParser	(		self,
			lineGenerator,
			firstline,
			firstLineCount )

Definition at line 693 of file trfValidation.py.

    def badAllocExceptionParser(self, lineGenerator, firstline, firstLineCount):
        badAllocExceptionReport = 'terminate after \'std::bad_alloc\'.'
 
        msg.debug('Identified bad_alloc - adding to error detail report')
        self._levelCounter['CATASTROPHE'] += 1
        self._errorDetails['CATASTROPHE'].append({'message': badAllocExceptionReport, 'firstLine': firstLineCount, 'count': 1})
 

coreDumpSvcParser()

python.trfValidation.athenaLogFileReport.coreDumpSvcParser	(		self,
			log,
			lineGenerator,
			firstline,
			firstLineCount )

Attempt to suck a core dump report from the current logfile This function scans logs in two different directions: 1) downwards, to exctract information after CoreDumpSvc; and 2) upwards, to find abnormal lines.

Note

: Current downwards scan just eats lines until a 'normal' line is seen. There is a slight problem here in that the end of core dump trigger line will not get parsed TODO: fix this (OTOH core dump is usually the very last thing and fatal!)

Definition at line 539 of file trfValidation.py.

    def coreDumpSvcParser(self, log, lineGenerator, firstline, firstLineCount):
        _eventCounter = _run = _event = _currentAlgorithm = _functionLine = _currentFunction = None
        coreDumpReport = 'Core dump from CoreDumpSvc'
        # Number of lines to ignore above 'core dump' when looking for abnormal lines
        offset = 1
        coreDumpDetailsReport = {}
 
        for line, linecounter in lineGenerator:
            m = self._regExp.match(line)
            if m is None:
                if 'Caught signal 11(Segmentation fault)' in line:
                    coreDumpReport = 'Segmentation fault'
                if 'Event counter' in line:
                    _eventCounter = line
 
                #Lookup: 'EventID: [Run,Evt,Lumi,Time,BunchCross,DetMask] = [267599,7146597,1,1434123751:0,0,0x0,0x0,0x0]'
                if 'EventID' in line:
                    match = re.findall(r'\[.*?\]', line)
                    if match and match.__len__() >= 2:      # Assuming the line contains at-least one key-value pair.
                        brackets = "[]"
                        commaDelimer = ','
                        keys = (match[0].strip(brackets)).split(commaDelimer)
                        values = (match[1].strip(brackets)).split(commaDelimer)
 
                        if 'Run' in keys:
                            _run = 'Run: ' + values[keys.index('Run')]
 
                        if 'Evt' in keys:
                            _event = 'Evt: ' + values[keys.index('Evt')]
 
                if 'Current algorithm' in line:
                    _currentAlgorithm = line
                if '<signal handler called>' in line:
                    _functionLine = linecounter+1
                if _functionLine and linecounter is _functionLine:
                    if ' in ' in line:
                        _currentFunction = 'Current Function: ' + line.split(' in ')[1].split()[0]
                    else:
                        _currentFunction = 'Current Function: ' + line.split()[1]
            else:
                # Can this be done - we want to push the line back into the generator to be
                # reparsed in the normal way (might need to make the generator a class with the
                # __exec__ method supported (to get the line), so that we can then add a
                # pushback onto an internal FIFO stack
                # lineGenerator.pushback(line)
                break
        _eventCounter = 'Event counter: unknown' if not _eventCounter else _eventCounter
        _run = 'Run: unknown' if not _run else _run
        _event = 'Evt: unknown' if not _event else _event
        _currentAlgorithm = 'Current algorithm: unknown' if not _currentAlgorithm else _currentAlgorithm
        _currentFunction = 'Current Function: unknown' if not _currentFunction else _currentFunction
        coreDumpReport = '{0}: {1}; {2}; {3}; {4}; {5}'.format(coreDumpReport, _eventCounter, _run, _event, _currentAlgorithm, _currentFunction)
 
        coreDumpDetailsReport = self.moreDetails(log, firstline, firstLineCount, 'knowledgeFile.db', offset)
        abnormalLines = coreDumpDetailsReport['abnormalLines']
 
        # concatenate an extract of first seen abnormal line to the core dump message
        if 'message0' in abnormalLines.keys():
            coreDumpReport += '; Abnormal line seen just before core dump: ' + abnormalLines['message0'][0:30] + '...[truncated] ' + '(see the jobReport)'
 
        # Core dumps are always fatal...
        msg.debug('Identified core dump - adding to error detail report')
        self._levelCounter['FATAL'] += 1
        self._errorDetails['FATAL'].append({'moreDetails': coreDumpDetailsReport, 'message': coreDumpReport, 'firstLine': firstLineCount, 'count': 1})
 
 

dbMonitor()

python.trfValidation.athenaLogFileReport.dbMonitor

(

self

)

Return data volume and time spend to retrieve information from the database.

Definition at line 445 of file trfValidation.py.

    def dbMonitor(self):
        return {'bytes' : self._dbbytes, 'time' : self._dbtime} if self._dbbytes > 0 or self._dbtime > 0 else None
 

eventLoopWarnings()

python.trfValidation.athenaLogFileReport.eventLoopWarnings

(

self

)

Definition at line 478 of file trfValidation.py.

    def eventLoopWarnings(self):
        eventLoopWarnings = []
        for item in self._eventLoopWarnings:
            if item in [element['item'] for element in eventLoopWarnings]:
                continue
            count = self._eventLoopWarnings.count(item)
            eventLoopWarnings.append({'item':item, 'count': count})
        return eventLoopWarnings          
 

firstError()

python.trfValidation.athenaLogFileReport.firstError	(		self,
			floor = 'ERROR' )

Return the first error found in the logfile above a certain loglevel.

Reimplemented from python.trfValidation.logFileReport.

Definition at line 464 of file trfValidation.py.

    def firstError(self, floor='ERROR'):
        firstLine = firstError = None
        firstLevel = stdLogLevels[floor]
        firstName = floor
        for lvl, count in self._levelCounter.items():
            if (count > 0 and stdLogLevels.get(lvl, 0) >= stdLogLevels[floor] and
                (firstError is None or self._errorDetails[lvl][0]['firstLine'] < firstLine)):
                firstLine = self._errorDetails[lvl][0]['firstLine']
                firstLevel = stdLogLevels[lvl]
                firstName = lvl
                firstError = self._errorDetails[lvl][0]
 
        return {'level': firstName, 'nLevel': firstLevel, 'firstError': firstError}
 

g494ExceptionParser()

python.trfValidation.athenaLogFileReport.g494ExceptionParser	(		self,
			lineGenerator,
			firstline,
			firstLineCount )

Definition at line 605 of file trfValidation.py.

    def g494ExceptionParser(self, lineGenerator, firstline, firstLineCount):
        g4Report = firstline
        g4lines = 1
        if 'Aborting execution' not in g4Report:
            for line, linecounter in lineGenerator:
                g4Report += os.linesep + line
                g4lines += 1
                # Test for the closing string
                if '*** ' in line:
                    break
                if g4lines >= 25:
                    msg.warning('G4 exception closing string not found within {0} log lines of line {1}'.format(g4lines, firstLineCount))
                    break
 
        # G4 exceptions can be fatal or they can be warnings...
        msg.debug('Identified G4 exception - adding to error detail report')
        if "just a warning" in g4Report:
            if self._levelCounter['WARNING'] <= self._msgLimit:
                self._levelCounter['WARNING'] += 1
                self._errorDetails['WARNING'].append({'message': g4Report, 'firstLine': firstLineCount, 'count': 1})
            elif self._levelCounter['WARNING'] == self._msgLimit + 1:
                msg.warning("Found message number {0} at level WARNING - this and further messages will be supressed from the report".format(self._levelCounter['WARNING']))
        else:
            self._levelCounter['FATAL'] += 1
            self._errorDetails['FATAL'].append({'message': g4Report, 'firstLine': firstLineCount, 'count': 1})
 

g4ExceptionParser()

python.trfValidation.athenaLogFileReport.g4ExceptionParser	(		self,
			lineGenerator,
			firstline,
			firstLineCount,
			g4ExceptionLineDepth )

Definition at line 631 of file trfValidation.py.

    def g4ExceptionParser(self, lineGenerator, firstline, firstLineCount, g4ExceptionLineDepth):
        g4Report = firstline
        g4lines = 1
        for line, linecounter in lineGenerator:
            g4Report += os.linesep + line
            g4lines += 1
            # Test for the closing string
            if 'G4Exception-END' in line:
                break
            if g4lines >= g4ExceptionLineDepth:
                msg.warning('G4 exception closing string not found within {0} log lines of line {1}'.format(g4lines, firstLineCount))
                break
 
        # G4 exceptions can be fatal or they can be warnings...
        msg.debug('Identified G4 exception - adding to error detail report')
        if "-------- WWWW -------" in g4Report:
            if self._levelCounter['WARNING'] <= self._msgLimit:
                self._levelCounter['WARNING'] += 1
                self._errorDetails['WARNING'].append({'message': g4Report, 'firstLine': firstLineCount, 'count': 1})
            elif self._levelCounter['WARNING'] == self._msgLimit + 1:
                msg.warning("Found message number {0} at level WARNING - this and further messages will be supressed from the report".format(self._levelCounter['WARNING'])) 
        else:
            self._levelCounter['FATAL'] += 1
            self._errorDetails['FATAL'].append({'message': g4Report, 'firstLine': firstLineCount, 'count': 1})
 
 

knowledgeFileHandler()

python.trfValidation.athenaLogFileReport.knowledgeFileHandler	(		self,
			knowledgefile )

Generally, a knowledge file consists of non-standard logging error/abnormal lines which are left out during log scan and could help diagnose job failures.

Definition at line 268 of file trfValidation.py.

    def knowledgeFileHandler(self, knowledgefile):
        # load abnormal/error line(s) from the knowledge file(s)
        linesList = []
        fullName = trfUtils.findFile(os.environ['DATAPATH'], knowledgefile)
        if not fullName:
            msg.warning('Knowledge file {0} could not be found in DATAPATH'.format(knowledgefile))
        else:
            try:
                with open(fullName) as knowledgeFileHandle:
                    msg.debug('Opened knowledge file {0} from here: {1}'.format(knowledgefile, fullName))
 
                    for line in knowledgeFileHandle:
                        if line.startswith('#') or line == '' or line =='\n':
                            continue
                        line = line.rstrip('\n')
                        linesList.append(line)
            except OSError as e:
                msg.warning('Failed to open knowledge file {0}: {1}'.format(fullName, e))
        return linesList
 

moreDetails()

python.trfValidation.athenaLogFileReport.moreDetails	(		self,
			log,
			firstline,
			firstLineCount,
			knowledgeFile,
			offset = 0 )

Definition at line 487 of file trfValidation.py.

    def moreDetails(self, log, firstline, firstLineCount, knowledgeFile, offset=0):
        # Look for "abnormal" and "last normal" line(s)
        # Make a list of last e.g. 50 lines before core dump
        abnormalLinesList = self.knowledgeFileHandler(knowledgeFile)
        linesToBeScanned = 50
        seenAbnormalLines = []
        abnormalLinesReport = {}
        lastNormalLineReport = {}
 
        linesList = [] 
        myGen = trfUtils.lineByLine(log)
        for line, linecounter in myGen:
            if linecounter in range(firstLineCount - linesToBeScanned, firstLineCount-offset):
                linesList.append([linecounter, line])
            elif linecounter == firstLineCount:
                break
 
        for linecounter, line in reversed(linesList):
            if re.findall(r'|'.join(abnormalLinesList), line):
                seenLine = False
                for dic in seenAbnormalLines:
                    # count repetitions or similar (e.g. first 15 char) abnormal lines
                    if dic['message'] == line or dic['message'][0:15] == line[0:15]:
                        dic['count'] += 1
                        seenLine = True
                        break
                if seenLine is False:
                    seenAbnormalLines.append({'message': line, 'firstLine': linecounter, 'count': 1})
            else:
                if line != '':
                    lastNormalLineReport = {'message': line, 'firstLine': linecounter, 'count': 1}
                    break
                else:
                    continue
 
        # Write the list of abnormal lines into the abnormalLinesReport dictionary
        # The keys of each abnormal line have a number suffix starting with 0
        # e.g., first abnormal line's keys are :{'mesage0', 'firstLine0', 'count0'}        
 
        for a in range(len(seenAbnormalLines)):
            abnormalLinesReport.update({'message{0}'.format(a): seenAbnormalLines[a]['message'], 'firstLine{0}'.format(a): seenAbnormalLines[a]['firstLine'],
                                        'count{0}'.format(a): seenAbnormalLines[a]['count']})
 
        return {'abnormalLines': abnormalLinesReport, 'lastNormalLine': lastNormalLineReport}
 
 

python()

python.trfValidation.athenaLogFileReport.python

(

self

)

Produce a python dictionary summary of the log file report for inclusion in the executor report.

Definition at line 242 of file trfValidation.py.

    def python(self):
        errorDict = {'countSummary': {}, 'details': {}}
        for level, count in self._levelCounter.items():
            errorDict['countSummary'][level] = count
            if self._levelCounter[level] > 0 and len(self._errorDetails[level]) > 0:
                errorDict['details'][level] = []
                for error in self._errorDetails[level]:
                    errorDict['details'][level].append(error)
        return errorDict
 

pythonExceptionParser()

python.trfValidation.athenaLogFileReport.pythonExceptionParser	(		self,
			log,
			lineGenerator,
			firstline,
			firstLineCount )

Definition at line 657 of file trfValidation.py.

    def pythonExceptionParser(self, log, lineGenerator, firstline, firstLineCount):
        pythonExceptionReport = ""
        lastLine = firstline
        lastLine2 = firstline
        pythonErrorLine = firstLineCount
        pyLines = 1
        for line, linecounter in lineGenerator:
            if 'Py:Athena' in line and 'INFO leaving with code' in line:
                if len(lastLine)> 0:
                    pythonExceptionReport = lastLine
                    pythonErrorLine = linecounter-1
                else: # Sometimes there is a blank line after the exception
                    pythonExceptionReport = lastLine2
                    pythonErrorLine = linecounter-2
                break
            if pyLines >= 25:
                msg.warning('Could not identify python exception correctly scanning {0} log lines after line {1}'.format(pyLines, firstLineCount))
                pythonExceptionReport = "Unable to identify specific exception"
                pythonErrorLine = firstLineCount
                break
            lastLine2 = lastLine
            lastLine = line
            pyLines += 1
 
        pythonExceptionDetailsReport = self.moreDetails(log, firstline, firstLineCount, 'knowledgeFile.db')
        abnormalLines = pythonExceptionDetailsReport['abnormalLines']
 
        # concatenate an extract of first seen abnormal line to pythonExceptionReport
        if 'message0' in abnormalLines.keys():
            pythonExceptionReport += '; Abnormal line seen just before python exception: ' + abnormalLines['message0'][0:30] + '...[truncated] ' + '(see the jobReport)'
 
        msg.debug('Identified python exception - adding to error detail report')
        self._levelCounter['FATAL'] += 1
        self._errorDetails['FATAL'].append({'moreDetails': pythonExceptionDetailsReport, 'message': pythonExceptionReport, 'firstLine': pythonErrorLine, 'count': 1})
 
 

resetReport()

python.trfValidation.athenaLogFileReport.resetReport

(

self

)

Reimplemented from python.trfValidation.logFileReport.

Definition at line 252 of file trfValidation.py.

    def resetReport(self):
        self._levelCounter = {}
        for level in list(stdLogLevels) + ['UNKNOWN', 'IGNORED']:
            self._levelCounter[level] = 0
 
        self._errorDetails = {}
        self._eventLoopWarnings = []
        for level in self._levelCounter:
            self._errorDetails[level] = []
            # Format:
            # List of dicts {'message': errMsg, 'firstLine': lineNo, 'count': N}
        self._dbbytes = 0
        self._dbtime  = 0.0
 

rootSysErrorParser()

python.trfValidation.athenaLogFileReport.rootSysErrorParser	(		self,
			lineGenerator,
			firstline,
			firstLineCount )

Definition at line 700 of file trfValidation.py.

    def rootSysErrorParser(self, lineGenerator, firstline, firstLineCount):
        msg.debug('Identified ROOT IO problem - adding to error detail report')
        self._levelCounter['FATAL'] += 1
        self._errorDetails['FATAL'].append({'message': firstline, 'firstLine': firstLineCount, 'count': 1})
 

scanLogFile()

python.trfValidation.athenaLogFileReport.scanLogFile	(		self,
			resetReport = False )

Reimplemented from python.trfValidation.logFileReport.

Definition at line 288 of file trfValidation.py.

    def scanLogFile(self, resetReport=False):
        
        nonStandardErrorsList = self.knowledgeFileHandler('nonStandardErrors.db')
 
        if resetReport:
            self.resetReport()
 
        for log in self._logfile:
            msg.debug('Now scanning logfile {0}'.format(log))
            seenNonStandardError = ''
            customLogParser = None
            if log == 'log.generate':
                from EvgenProdTools.EvgenParserTool import evgenParserTool
                customLogParser = evgenParserTool()
            # N.B. Use the generator so that lines can be grabbed by subroutines, e.g., core dump svc reporter
            try:
                myGen = trfUtils.lineByLine(log, substepName=self._substepName)
            except IOError as e:
                msg.error('Failed to open transform logfile {0}: {1:s}'.format(log, e))
                # Return this as a small report
                self._levelCounter['ERROR'] = 1
                self._errorDetails['ERROR'] = {'message': str(e), 'firstLine': 0, 'count': 1}
                return
            # Detect whether we are in the event loop part of the log file
            inEventLoop = False
            for line, lineCounter in myGen:  
                if '===>>>  start processing event' in line: inEventLoop = True
                if 'Application Manager Stopped successfully' in line: inEventLoop = False
 
                # In case we have enabled a custom log parser, run the line through it first
                if customLogParser is not None:
                    customLogParser.processLine(line)
                # Search for metadata strings
                m = self._metaPat.search(line)
                if m is not None:
                    key, value = m.groups()
                    self._metaData[key] = value
 
                m = self._regExp.match(line)
                if m is None:
                    # We didn't manage to get a recognised standard line from the file
                    # But we can check for certain other interesting things, like core dumps
                    if 'Core dump from CoreDumpSvc' in line:
                        msg.warning('Detected CoreDumpSvc report - activating core dump svc grabber')
                        self.coreDumpSvcParser(log, myGen, line, lineCounter)
                        continue
                    # Add the G4 exceptipon parsers
                    if 'G4Exception-START' in line:
                        msg.warning('Detected G4 exception report - activating G4 exception grabber')
                        self.g4ExceptionParser(myGen, line, lineCounter, 40)
                        continue
                    if '*** G4Exception' in line:
                        msg.warning('Detected G4 9.4 exception report - activating G4 exception grabber')
                        self.g494ExceptionParser(myGen, line, lineCounter)
                        continue
                    # Add the python exception parser
                    if 'Shortened traceback (most recent user call last)' in line:
                        msg.warning('Detected python exception - activating python exception grabber')
                        self.pythonExceptionParser(log, myGen, line, lineCounter)
                        continue
                    # Add parser for missed bad_alloc
                    if 'terminate called after throwing an instance of \'std::bad_alloc\'' in line:
                        msg.warning('Detected bad_alloc!')
                        self.badAllocExceptionParser(myGen, line, lineCounter)
                        continue
                    # Parser for ROOT reporting a stale file handle (see ATLASG-448)
                    # Amendment: Generalize the search (see ATLASRECTS-7121)
                    if 'Error in <TFile::ReadBuffer>' in line:
                        self.rootSysErrorParser(myGen, line, lineCounter)
                        continue
 
                    if 'Error in <TFile::WriteBuffer>' in line:
                        self.rootSysErrorParser(myGen, line, lineCounter)
                        continue
                    # Check if the line is among the non-standard logging errors from the knowledge file
                    if any(line in l for l in nonStandardErrorsList):
                        seenNonStandardError = line
                        continue
 
                    msg.debug('Non-standard line in %s: %s', log, line)
                    self._levelCounter['UNKNOWN'] += 1
                    continue
 
                # Line was matched successfully
                fields = {}
                for matchKey in ('service', 'level', 'message'):
                    fields[matchKey] = m.group(matchKey)
                msg.debug('Line parsed as: {0}'.format(fields))
 
                # If this is a WARNING and we passed the start of the event loop, 
                # add it to special list 
                if (fields['level'] == 'WARNING') and inEventLoop:
                    self._eventLoopWarnings.append(fields)
 
                # Check this is not in our ignore list
                ignoreFlag = False
                for ignorePat in self._ignoreList.structuredPatterns:
                    serviceMatch = ignorePat['service'].match(fields['service'])
                    levelMatch = (ignorePat['level'] == "" or ignorePat['level'] == fields['level'])
                    messageMatch = ignorePat['message'].match(fields['message'])
                    if serviceMatch and levelMatch and messageMatch:
                        msg.info('Error message "{0}" was ignored at line {1} (structured match)'.format(line, lineCounter))
                        ignoreFlag = True
                        break
                if ignoreFlag is False:
                    for searchPat in self._ignoreList.searchPatterns:
                        if searchPat.search(line):
                            msg.info('Error message "{0}" was ignored at line {1} (search match)'.format(line, lineCounter))
                            ignoreFlag = True
                            break
                if ignoreFlag:
                    # Got an ignore - message this to a special IGNORED error
                    fields['level'] = 'IGNORED'
                else:
                    # Some special handling for specific errors (maybe generalise this if
                    # there end up being too many special cases)
                    # Upgrade bad_alloc to CATASTROPHE to allow for better automated handling of
                    # jobs that run out of memory
                    if 'std::bad_alloc' in fields['message']:
                        fields['level'] = 'CATASTROPHE'
 
                # concatenate the seen non-standard logging error to the FATAL
                if fields['level'] == 'FATAL':
                    if seenNonStandardError:
                        line += '; ' + seenNonStandardError
 
                # Count this error
                self._levelCounter[fields['level']] += 1
 
                # Record some error details
                # N.B. We record 'IGNORED' errors as these really should be flagged for fixing
                if fields['level'] == 'IGNORED' or stdLogLevels[fields['level']] >= self._msgDetails:
                    if self._levelCounter[fields['level']] <= self._msgLimit: 
                        detailsHandled = False
                        for seenError in self._errorDetails[fields['level']]:
                            if seenError['message'] == line:
                                seenError['count'] += 1
                                detailsHandled = True
                                break
                        if detailsHandled is False:
                            self._errorDetails[fields['level']].append({'message': line, 'firstLine': lineCounter, 'count': 1})
                    elif self._levelCounter[fields['level']] == self._msgLimit + 1:
                        msg.warning("Found message number {0} at level {1} - this and further messages will be supressed from the report".format(self._levelCounter[fields['level']], fields['level']))
                    else:
                        # Overcounted
                        pass
                if 'Total payload read from IOVDb' in fields['message']:
                    msg.debug("Found COOL payload information at line {0}".format(line))
                    a = re.match(r'(\D+)(?P<bytes>\d+)(\D+)(?P<time>\d+[.]?\d*)(\D+)', fields['message'])
                    self._dbbytes += int(a.group('bytes'))
                    self._dbtime  += float(a.group('time'))
            # Finally, if we have a custom log parser, use it to update the metadata dictionary
            if customLogParser is not None:
                customLogParser.report()
                self._metaData = customLogParser.updateMetadata( self._metaData )
 

worstError()

python.trfValidation.athenaLogFileReport.worstError

(

self

)

Return the worst error found in the logfile (first error of the most serious type)

Reimplemented from python.trfValidation.logFileReport.

Definition at line 449 of file trfValidation.py.

    def worstError(self):
        worst = stdLogLevels['DEBUG']
        worstName = 'DEBUG'
        for lvl, count in self._levelCounter.items():
            if count > 0 and stdLogLevels.get(lvl, 0) > worst:
                worstName = lvl
                worst = stdLogLevels[lvl]
        if len(self._errorDetails[worstName]) > 0:
            firstError = self._errorDetails[worstName][0]
        else:
            firstError = None
 
        return {'level': worstName, 'nLevel': worst, 'firstError': firstError}
 

Member Data Documentation

_dbbytes

python.trfValidation.athenaLogFileReport._dbbytes = 0

protected

Definition at line 263 of file trfValidation.py.

_dbtime

float python.trfValidation.athenaLogFileReport._dbtime = 0.0

protected

Definition at line 264 of file trfValidation.py.

_errorDetails

python.trfValidation.athenaLogFileReport._errorDetails = {}

protected

Definition at line 257 of file trfValidation.py.

_eventLoopWarnings

list python.trfValidation.athenaLogFileReport._eventLoopWarnings = []

protected

Definition at line 231 of file trfValidation.py.

_ignoreList

python.trfValidation.athenaLogFileReport._ignoreList = ignoreList

protected

Definition at line 218 of file trfValidation.py.

_levelCounter

python.trfValidation.athenaLogFileReport._levelCounter = {}

protected

Definition at line 253 of file trfValidation.py.

_logfile

list python.trfValidation.logFileReport._logfile = [logfile, ]

protectedinherited

Definition at line 181 of file trfValidation.py.

_metaData

dict python.trfValidation.athenaLogFileReport._metaData = {}

protected

Definition at line 230 of file trfValidation.py.

_metaPat

python.trfValidation.athenaLogFileReport._metaPat = re.compile(r"MetaData:\s+(.*?)\s*=\s*(.*)$")

protected

Definition at line 229 of file trfValidation.py.

_msgDetails

python.trfValidation.logFileReport._msgDetails = msgDetailLevel

protectedinherited

Definition at line 186 of file trfValidation.py.

_msgLimit

python.trfValidation.logFileReport._msgLimit = msgLimit

protectedinherited

Definition at line 185 of file trfValidation.py.

_re

python.trfValidation.logFileReport._re = None

protectedinherited

Definition at line 187 of file trfValidation.py.

_regExp

python.trfValidation.athenaLogFileReport._regExp = re.compile(r'(?P<service>[^\s]+\w)(.*)\s+(?P<level>' + '|'.join(stdLogLevels) + r')\s+(?P<message>.*)')

protected

Note

This is the regular expression match for athena logfile lines Match first strips off any HH:MM:SS prefix the transform has added, then takes the next group of non-whitespace characters as the service, then then matches from the list of known levels, then finally, ignores any last pieces of whitespace prefix and takes the rest of the line as the message

Definition at line 227 of file trfValidation.py.

_substepName

python.trfValidation.athenaLogFileReport._substepName = substepName

protected

Definition at line 232 of file trfValidation.py.

---

The documentation for this class was generated from the following file:

• trfValidation.py