ATLAS Offline Software
Public Member Functions | Public Attributes | List of all members
EvgenParserTool.evgenParserTool Class Reference
Collaboration diagram for EvgenParserTool.evgenParserTool:

Public Member Functions

def __init__ (self, name='evgenLogParser', **kw)
 
def processLine (self, line)
 
def report (self)
 
def updateMetadata (self, metadata)
 

Public Attributes

 msg
 
 FixHepMCDict
 
 TestHepMCDict
 
 FilterSeqDict
 
 MetadataDict
 
 isMP
 

Detailed Description

Definition at line 12 of file EvgenParserTool.py.

Constructor & Destructor Documentation

◆ __init__()

def EvgenParserTool.evgenParserTool.__init__ (   self,
  name = 'evgenLogParser',
**  kw 
)

Definition at line 14 of file EvgenParserTool.py.

14  def __init__ ( self, name = 'evgenLogParser', **kw ):
15  self.msg = Logging.logging.getLogger(name)
16  # For FixHepMC, keep a list of reasons for rejection, counts, and lines for logging
17  self.FixHepMCDict = {'reasons':{},'denominator':0,'lines':{}}
18  # For TestHepMC, we need summary statistics as well as all the individual numbers
19  # Keep number passing and failing, p/f line (for printing), all the rates with the lines, which ones are(n't) included in the efficiency
20  # the line for the rates, the line for the final efficiency, and the number of events in this log passing/failing (for converting % to count)
21  self.TestHepMCDict = {'p':0,'f':0,'pfline':None,'rate':{},'notinc':[],'rline':None,'effline':None,'lastpf':0}
22  # For the filter sequence, keep the weighted and unweighted numbers and the line
23  self.FilterSeqDict = {'num':0,'den':0,'wnum':0,'wden':0,'line':None,'wline':None}
24  # Process the Metadata as well
25  self.MetadataDict = {'sumOfPosWeights':0.,'sumOfNegWeights':0.,'sumOfSqrWeights':0.,
26  'sumOfPosWeightsNoFilter':0.,'sumOfNegWeightsNoFilter':0.,'sumOfSqrWeightsNoFilter':0.,
27  'xsec_holder':0.,'xsec_weight':0.,'xsec_sum':0.}
28  self.isMP = -1
29 

Member Function Documentation

◆ processLine()

def EvgenParserTool.evgenParserTool.processLine (   self,
  line 
)
Function to process a log line and keep what's needed for final reporting

Definition at line 30 of file EvgenParserTool.py.

30  def processLine( self, line ):
31  ''' Function to process a log line and keep what's needed for final reporting'''
32 
33  # First check for lines from FixHepMC
34  if 'FixHepMC' in line and 'INFO Removed' in line:
35  # Use the loops line to count the denominator
36  if 'because of loops' in line:
37  self.FixHepMCDict['denominator'] += int( line.split(' of ')[1].split()[0].strip() )
38  # Grab the reason for failure from the line; they all follow the same formula
39  reason = line.split('particles')[1]
40  # Add to the count for that reason
41  if reason not in self.FixHepMCDict['reasons']:
42  self.FixHepMCDict['reasons'][reason] = 0
43  self.FixHepMCDict['reasons'][reason] += int( line.split('Removed')[1].split()[0].strip() )
44  # Make sure that we also have a log line for printing later if we need it
45  if reason not in self.FixHepMCDict['lines']:
46  self.FixHepMCDict['lines'][reason] = [ line.split('Removed')[0] , line.split('particles')[1] ]
47  # Second up: Filter statistics
48  elif 'Py:EvgenFilterSeq' in line:
49  # First check weighted, then unweighted (one is a substring of the other...)
50  if 'Weighted Filter Efficiency' in line:
51  # Grab all the numbers from the log. The line prints the numerator and denominator explicitly at the end
52  numbers = re.findall(r'[\d.]+',line)
53  self.FilterSeqDict['wnum'] += float( numbers[-2] )
54  self.FilterSeqDict['wden'] += float( numbers[-1] )
55  # Grab the log line to print
56  self.FilterSeqDict['wline'] = line.split('=')[0]
57  elif 'Filter Efficiency' in line:
58  # Grab all the numbers from the log. The line prints the numerator and denominator explicitly at the end
59  numbers = re.findall(r'[\d.]+',line)
60  self.FilterSeqDict['num'] += int( numbers[-2] )
61  self.FilterSeqDict['den'] += int( numbers[-1] )
62  # Grab the log line to print
63  if self.FilterSeqDict['line'] is None:
64  self.FilterSeqDict['line'] = line.split('=')[0]
65  # Third and finally: TestHepMC
66  elif 'TestHepMC' in line and 'Event' in line:
67  if 'Events passed' in line:
68  # Simplest line, just shows the numbers of events passing and failing
69  numbers = re.findall(r'[\d.]+',line)
70  self.TestHepMCDict['p'] += int(numbers[-2])
71  self.TestHepMCDict['f'] += int(numbers[-1])
72  # Save the log line so that we can reproduce it later
73  self.TestHepMCDict['pfline'] = line.split('=')[0]
74  # Keep track of the denominators to ensure we get the percentages right
75  self.TestHepMCDict['lastpf'] = int(numbers[-2]) + int(numbers[-1])
76  else:
77  # Otherwise this is an event rate line. First figure out what the reason in the line is, and index count on that
78  reason = line.split('Event rate')[1].split('=')[0]
79  if reason not in self.TestHepMCDict['rate']:
80  self.TestHepMCDict['rate'][reason] = 0
81  # Some rates are not included in the test efficiency; we should be ready to point that out as well
82  if 'not included in test efficiency' in line:
83  self.TestHepMCDict['notinc'] += [reason]
84  # We have to convert the logged percentage back to counts, and then back to percentage in the report
85  my_perc = float( line.split('=')[1].split('%')[0] )
86  self.TestHepMCDict['rate'][reason] += my_perc * self.TestHepMCDict['lastpf']/100.
87  # And last we just have to get the log format for an event rate line right
88  self.TestHepMCDict['rline'] = line.split('Event rate')[0]
89  # Extra catch for Efficiency line for TestHepMC
90  elif 'TestHepMC' in line and 'Efficiency' in line:
91  # If this is just the efficiency line, all we need is the log line format
92  self.TestHepMCDict['effline'] = line.split('=')[0]
93  # Because this is a unique line, we will also use it to check if we are running MP
94  self.isMP += 1
95  elif 'MetaData:' in line:
96  # If it's a metadata line, we just sum them up, with one exception
97  field = line.split('MetaData:')[1].split('=')[0].strip()
98  if field in self.MetadataDict:
99  self.MetadataDict[field] += float( line.split('=')[1] )
100  # Cross section requires special attention
101  # The cross section field itself comes first, so we have to just stash it
102  if field == 'cross-section (nb)':
103  self.MetadataDict['xsec_holder'] = float( line.split('=')[1] )
104  # The weights fields are also there; we need the 'no filter' versions
105  # Use = here to reset each round
106  elif field == 'sumOfPosWeightsNoFilter':
107  self.MetadataDict['xsec_weight'] = float( line.split('=')[1] )
108  # Sum of negative weights is last, and now we have all the info we need
109  elif field == 'sumOfNegWeightsNoFilter':
110  my_negw = float( line.split('=')[1] )
111  self.MetadataDict['xsec_sum'] += self.MetadataDict['xsec_holder']*(self.MetadataDict['xsec_weight']-my_negw)
112  # We don't need to keep the sum of weights here, because we have it elsewhere
113 

◆ report()

def EvgenParserTool.evgenParserTool.report (   self)
Function to print final statistics grabbed from the logs

Definition at line 114 of file EvgenParserTool.py.

114  def report(self):
115  ''' Function to print final statistics grabbed from the logs'''
116  # If we aren't running MP, then forget it
117  if self.isMP<1:
118  self.msg.debug('Not running MP, nothing to do')
119  return
120  # Now we are going to print updated statistics for all the handlers
121  # Because in an MP job we get a primary worker report as well, the number of workers is just self.isMP
122  self.msg.info(f'Printing final summary statistics from {self.isMP} MP workers')
123  # First print all the FixHepMC stuff - just rates for each reason we remove particles
124  for reason in self.FixHepMCDict['reasons']:
125  print(f"{self.FixHepMCDict['lines'][reason][0]}Removed {self.FixHepMCDict['reasons'][reason]} of {self.FixHepMCDict['denominator']} particles {self.FixHepMCDict['lines'][reason][1]}")
126  # Next print all the information from TestHepMC, starting with the pass/fail summary
127  print(f"{self.TestHepMCDict['pfline']}= {self.TestHepMCDict['p']}, Events Failed = {self.TestHepMCDict['f']}")
128  # We will have the same denominator for all the ratios that we print next
129  denom = self.TestHepMCDict['p'] + self.TestHepMCDict['f']
130  # Now go through all the TestHepMC checks, and for each one recreate the log line
131  for rate in self.TestHepMCDict['rate']:
132  # Again, have to handle zero denominators correctly
133  pct = 0.
134  if denom>0:
135  pct = (self.TestHepMCDict['rate'][rate] / denom)*100.
136  print(f"{self.TestHepMCDict['rline']}Event rate {rate} = {pct}%{' (not included in test efficiency)' if rate in self.TestHepMCDict['notinc'] else ''}")
137  # Now print the final efficiency; make sure we handle zeroes correctly
138  final_eff = 1.
139  if denom:
140  final_eff = self.TestHepMCDict['p']/denom
141  print(f"{self.TestHepMCDict['effline']}= {final_eff*100.}%")
142  # Now print our filter sequence summary - again, make sure we handle zeroes correctly
143  eff = (self.FilterSeqDict['num']/self.FilterSeqDict['den']) if self.FilterSeqDict['den'] > 0 else 1.
144  print(f"{self.FilterSeqDict['line']} = {eff} [{self.FilterSeqDict['num']} / {self.FilterSeqDict['den']}]")
145  eff = (self.FilterSeqDict['wnum']/self.FilterSeqDict['wden']) if self.FilterSeqDict['wden'] >0 else 1.
146  print(f"{self.FilterSeqDict['wline']} = {eff} [{self.FilterSeqDict['wnum']} / {self.FilterSeqDict['wden']}]")
147  # Done!
148 

◆ updateMetadata()

def EvgenParserTool.evgenParserTool.updateMetadata (   self,
  metadata 
)
Update the metadata based on the logfile information

Definition at line 149 of file EvgenParserTool.py.

149  def updateMetadata(self, metadata):
150  ''' Update the metadata based on the logfile information'''
151  # If we aren't running MP, then forget it
152  if self.isMP<1:
153  self.msg.debug('Not running MP, nothing to do')
154  return metadata
155  # Print the updated metadata as we go, as well as updating the dictionary
156  # First by convention is the cross-section, which we have to calculate
157  denom = self.MetadataDict['sumOfPosWeightsNoFilter']-self.MetadataDict['sumOfNegWeightsNoFilter']
158  my_xsec = 0.
159  if denom > 0.:
160  my_xsec = self.MetadataDict['xsec_sum'] / denom
161  self.msg.info(f'cross-section (nb)= {my_xsec:e}')
162  metadata['cross-section (nb)'] = f'{my_xsec:e}'
163  # Now come all the fields that we had saved
164  self.msg.info('Updated metadata:')
165  for field in self.MetadataDict:
166  # Need to skip the cross section related fields
167  if field in metadata:
168  self.msg.info(f'{field} = {self.MetadataDict[field]:e}')
169  metadata[field] = f'{self.MetadataDict[field]:e}'
170  # Generator filter efficiency needs some special handling
171  geneff = 1.
172  if denom>0:
173  geneff = (self.MetadataDict['sumOfPosWeights']-self.MetadataDict['sumOfNegWeights'])/denom
174  self.msg.info(f'GenFiltEff = {geneff:e}')
175  if 'GenFiltEff' in metadata:
176  metadata['GenFiltEff'] = f'{geneff:e}'
177  return metadata
178 

Member Data Documentation

◆ FilterSeqDict

EvgenParserTool.evgenParserTool.FilterSeqDict

Definition at line 23 of file EvgenParserTool.py.

◆ FixHepMCDict

EvgenParserTool.evgenParserTool.FixHepMCDict

Definition at line 17 of file EvgenParserTool.py.

◆ isMP

EvgenParserTool.evgenParserTool.isMP

Definition at line 28 of file EvgenParserTool.py.

◆ MetadataDict

EvgenParserTool.evgenParserTool.MetadataDict

Definition at line 25 of file EvgenParserTool.py.

◆ msg

EvgenParserTool.evgenParserTool.msg

Definition at line 15 of file EvgenParserTool.py.

◆ TestHepMCDict

EvgenParserTool.evgenParserTool.TestHepMCDict

Definition at line 21 of file EvgenParserTool.py.


The documentation for this class was generated from the following file:
checkTP.report
report
Definition: checkTP.py:125
print
void print(char *figname, TCanvas *c1)
Definition: TRTCalib_StrawStatusPlots.cxx:25
debug
const bool debug
Definition: MakeUncertaintyPlots.cxx:53
python.processes.powheg.ZZ.ZZ.__init__
def __init__(self, base_directory, **kwargs)
Constructor: all process options are set here.
Definition: ZZ.py:18
python.CaloAddPedShiftConfig.int
int
Definition: CaloAddPedShiftConfig.py:45
python.ParticleTypeUtil.info
def info
Definition: ParticleTypeUtil.py:87
Trk::split
@ split
Definition: LayerMaterialProperties.h:38
python.LArMinBiasAlgConfig.float
float
Definition: LArMinBiasAlgConfig.py:65