ATLAS Offline Software
Histogram Merging and Post-Processing

Histogram Merging

On Tier0 the raw data is processed one input file per job and the histogram outputs from each job are merged in a later step. Merge methods can be defined with the histograms so that the merging is done correctly. This is not an issue online as a single job processes all the events in a run.

The available custom merge methods are defined in MonitoringFile_MergeAlgs.cxx and the name strings to use in the code in MonitoringFile::mergeObjs() both from the package DataQuality/DataQualityUtils. If no merge method is specified the default is ROOT h1->Add(h2).

When using custom merge methods or unusual filling schemes the merge should be tested to show that it gives the same result as a single job with no merges and the same result regardless of the order of merging. This can be done with the rootDiff command from Sami Kama. A slightly modified version of this is available in the test directory of the TrigT1CaloMonitoringTools package. This can be used in the following way:

Set up a suitable Tier0 release then compile the test executable:

g++ -o rootDiff rootDiff.cxx

Run three Reco_trf.py jobs on the same raw data file making sure that the histograms of interest have some entries (or the test is useless!):

Reco_trf.py maxEvents=500 ......
Reco_trf.py maxEvents=240 ......
Reco_trf.py maxEvents=260 skipEvents=240 ......

If the histogram output files from these jobs are Hist1.root, Hist2.root, Hist3.root say, then create a file hist_merge_list1.txt say, containing the names:

Hist3.root
Hist2.root

Note the reverse order. Then do:

DQHistogramMerge hist_merge_list1.txt Hist4.root False >& DQHistogramMerge1.log
./rootDiff Hist1.root Hist4.root >& rootDiff.log

and check rootDiff.log for any L1Calo or LVL1_Interfaces errors.


Custom Merges Used in TrigT1CaloMonitoring

Merge Method Description from DataQualityUtils
eventSample Merge histograms containing, for example, event numbers of events with particular types of errors.
lowerLB Merge "status" histograms, i.e filled at start of run/LB.
mergeRebinned This method provides a correct summation for histograms that have different binning.
perBinEffPerCent This code assumes that the histogram content is the efficiency of a given cut or selection in each bin.
weightedAverage Merge histograms of weighted averages.

Histogram Post-Processing

Sometimes it's not possible to get the result you want with merging. In this case it is necessary to use the DQ histogram post-processing algorithms which are run after all the merging is complete. The L1calo post-processing code can be found in MonitoringFile_L1CaloPostProcess.cxx from DataQualityUtils.

Note that this code is not run in the standard Reco_trf.py job so to test it it is necessary to rerun DQHistogramMerge like this:

DQHistogramMerge hist_merge_list.txt myHIST2.root True >& DQHistogramMerge2.log

in the output directory of the Reco_trf.py job.

True
#define True
Definition: ALFA_SvdCalc.h:37
jetMakeRefSamples.skipEvents
int skipEvents
Definition: jetMakeRefSamples.py:56
python.SCT_ByteStreamErrorsTestAlgConfig.maxEvents
maxEvents
Definition: SCT_ByteStreamErrorsTestAlgConfig.py:43
python.CaloCondTools.g
g
Definition: CaloCondTools.py:15
False
#define False
Definition: ALFA_SvdCalc.h:38
DQHistogramMerge
Definition: DQHistogramMerge.py:1