LVL1CaloMonitoringConfig.L1CaloMonitorCfgHelper Class Reference

Inheritance diagram for LVL1CaloMonitoringConfig.L1CaloMonitorCfgHelper:

Collaboration diagram for LVL1CaloMonitoringConfig.L1CaloMonitorCfgHelper:

Public Member Functions
def	__init__ (self, flags, algClassOrObj=None, name=None, *args, **kwargs)
def	defineDQAlgorithm (self, name, hanConfig, thresholdConfig=None)
def	defineHistogram (self, *args, fillGroup=None, hanConfig={}, paths=[], **kwargs)
def	defineTree (self, *args, fillGroup=None, **kwargs)
def	result (self)

Static Public Member Functions
def	createXmls ()
def	printHanConfig (filename="collisions_run.config")

Public Attributes
	helper
	alg
	fillGroups
	dqEnv

Static Public Attributes
	hanConfigs
	hanAlgConfigs
	hanThresholdConfigs
	xmlConfigs
	SIGNATURES
	HELPURL

Detailed Description

This class is designed to handle registering all of L1Calo's monitoring histograms and trees in a
coherent way. It will also generate the han config file for all the histograms.

Definition at line 5 of file LVL1CaloMonitoringConfig.py.

Constructor & Destructor Documentation

__init__()

def LVL1CaloMonitoringConfig.L1CaloMonitorCfgHelper.__init__	(		self,
			flags,
			algClassOrObj = None,
			name = None,
		*	args,
		**	kwargs
	)

Create the configuration helper.

Arguments:
flags -- the configuration flag object
algClassOrObj -- the name you want to assign the family of algorithms

Definition at line 205 of file LVL1CaloMonitoringConfig.py.

    def __init__(self, flags, algClassOrObj = None, name = None, *args, **kwargs):
        '''
        Create the configuration helper.
 
        Arguments:
        flags -- the configuration flag object
        algClassOrObj -- the name you want to assign the family of algorithms
        '''
        from AthenaMonitoring import AthMonitorCfgHelper
        self.helper = AthMonitorCfgHelper(flags,name)
        self.alg = self.helper.addAlgorithm(algClassOrObj,name,*args, **kwargs) if algClassOrObj is not None else None
        self.fillGroups = {}
        self.dqEnv = flags.DQ.Environment # used to decide if should defineTree or not ...
 

Member Function Documentation

createXmls()

def LVL1CaloMonitoringConfig.L1CaloMonitorCfgHelper.createXmls ( )

static

Definition at line 24 of file LVL1CaloMonitoringConfig.py.

    def createXmls():
        # First create the DQAlgorithm config file
        import config
        db = config.Configuration()
        db.create_db("l1calo_gen_phase1.data.xml",["daq/schema/dqm.schema.xml","daq/sw/dqm-algorithms.data.xml","daq/sw/repository.data.xml","daq/sw/dqm-default-streams.data.xml"])
 
        # first ensure all the algorithms exist
        for confName,conf in L1CaloMonitorCfgHelper.hanAlgConfigs.items():
            # only need algorithms for Shifter histograms
            algUsed = False
            for histPath,value in L1CaloMonitorCfgHelper.xmlConfigs.items():
                if value.get("algorithm","") == confName:
                    algUsed = True
                    break
            if not algUsed: continue
            updated = False
            try:
                dqAlgo = db.get_dal("DQAlgorithm",conf["name"])
            except RuntimeError:
                # algorithm doesn't exist, create it ...
                updated=True
                db.create_obj("DQAlgorithm",conf["name"])
                dqAlgo = db.get_dal("DQAlgorithm",conf["name"])
                dqAlgo.LibraryName = conf["libname"]
            # ensure all parameters appear in ParametersNames, and thresholds in ThresholdsNames
            for par,val in conf.items():
                if par=="thresholds" and val in L1CaloMonitorCfgHelper.hanThresholdConfigs:
                    for thresh,vals in L1CaloMonitorCfgHelper.hanThresholdConfigs[val].items():
                        if thresh not in dqAlgo.ThresholdsNames:
                            updated=True
                            dqAlgo.ThresholdsNames += [thresh]
                elif par not in ["name","libname","thresholds"]+dqAlgo.ParametersNames:
                    updated=True
                    dqAlgo.ParametersNames += [par]
            if updated:
                db.update_dal(dqAlgo)
 
 
        # now create DQParameter object for each histogram with a DQ algorithm
        # need to have them all in a DQRegion that belongs to a DQLayout
        db.create_obj("DQShape","L1CaloShapeDeveloper")
        db.create_obj("DQLayout","L1CaloLayoutDeveloper")
        db.create_obj("DQRegion","L1CaloRegionDeveloper")
        dqLayout = db.get_dal("DQLayout","L1CaloLayoutDeveloper")
        dqLayout.DefaultChildrenShape = db.get_dal("DQShape","L1CaloShapeDeveloper")
        db.update_dal(dqLayout)
        dqRegion = db.get_dal("DQRegion","L1CaloRegionDeveloper")
        dqRegion.Description = "L1Calo Monitoring Development Folder"
        dqRegion.Label = "Developer"
        dqRegion.DQLayout = dqLayout
        dqRegion.DQSummaryMaker = db.get_dal("DQSummaryMaker","WorstCaseSummary")
 
        for histPath,value in L1CaloMonitorCfgHelper.xmlConfigs.items():
            if "algorithm" not in value.keys(): continue # only add histograms that have an algorithm
 
            name = "_".join(histPath.split("/"))
            db.create_obj("DQParameter",name)
            dqPar = db.get_dal("DQParameter",name)
            dqPar.InputDataSource = ["Histogramming.l1calo-athenaHLT-Phase1./"+histPath]
            dqPar.Description = value["description"]
 
            if value["algorithm"] in L1CaloMonitorCfgHelper.hanAlgConfigs:
                algConf = L1CaloMonitorCfgHelper.hanAlgConfigs[value["algorithm"]]
                dqPar.DQAlgorithm = db.get_dal("DQAlgorithm",algConf["name"])
                for par,val in algConf.items():
                    if par in dqPar.DQAlgorithm.ParametersNames:
                        dqPar.Parameters += [f"{par}={val}"]
                if "thresholds" in algConf:
                    for thresh,vals in L1CaloMonitorCfgHelper.hanThresholdConfigs[algConf["thresholds"]].items():
                        dqPar.GreenThresholds += [f"{thresh}={vals['warning']}"]
                        dqPar.RedThresholds += [f"{thresh}={vals['error']}"]
            else:
                dqPar.DQAlgorithm = db.get_dal("DQAlgorithm",value["algorithm"]) # catches cases of e.g. predefined parameterless alg like Histogram_Empty
 
            dqRegion.DQParameters +=  [dqPar]
            db.update_dal(dqPar)
 
        db.update_dal(dqRegion)
        db.commit("Autogenerated")
 
 

defineDQAlgorithm()

def LVL1CaloMonitoringConfig.L1CaloMonitorCfgHelper.defineDQAlgorithm	(		self,
			name,
			hanConfig,
			thresholdConfig = None
	)

:param name: name of algorithm
:param hanConfig: dict of algo properties
:param thresholdConfig: dict of thresholds, keys in form of ParName.level
:return:

Definition at line 219 of file LVL1CaloMonitoringConfig.py.

    def defineDQAlgorithm(self,name,hanConfig,thresholdConfig=None):
 
        """
 
        :param name: name of algorithm
        :param hanConfig: dict of algo properties
        :param thresholdConfig: dict of thresholds, keys in form of ParName.level
        :return:
        """
 
        # note: this method will replace any existing alg definition
 
        #thresNum = len(self.hanThresholdConfigs)
        if thresholdConfig is not None:
            hanConfig["thresholds"] = f"{name}Thresholds"
            threshDict = {}
            for parName,limVals in thresholdConfig.items():
                if len(limVals) != 2:
                    raise Exception("must specify two limits: warning and error")
                if parName not in threshDict: threshDict[parName] = {}
                threshDict[parName]["warning"] = limVals[0]
                threshDict[parName]["error"] = limVals[1]
            # see if any existing thresholds are identical, if so we can reuse
            for threshName,thresh in self.hanThresholdConfigs.items():
                if str(thresh)==str(threshDict):
                    threshDict = None
                    hanConfig["thresholds"] = threshName
                    break
            if threshDict is not None: self.hanThresholdConfigs[hanConfig["thresholds"]] = threshDict
        self.hanAlgConfigs[name] = hanConfig
 
        return
 
 

defineHistogram()

def LVL1CaloMonitoringConfig.L1CaloMonitorCfgHelper.defineHistogram	(		self,
		*	args,
			fillGroup = None,
			hanConfig = {},
			paths = [],
		**	kwargs
	)

:param path:
:param fillGroup:
:param args:
:param kwargs:
:return:

Definition at line 253 of file LVL1CaloMonitoringConfig.py.

    def defineHistogram(self,*args,fillGroup=None,hanConfig={},paths=[],**kwargs):
        '''
 
        :param path:
        :param fillGroup:
        :param args:
        :param kwargs:
        :return:
        '''
 
        hanConfig = dict(hanConfig) # create a copy since will modify below (otherwise we end up modifying the default empty dict)
 
        if paths != []:
            for path in paths:
                # create a copy of the histogram in each of the extra locations
                self.defineHistogram(*args,fillGroup=fillGroup,hanConfig=hanConfig,paths=[],path=path,**kwargs)
            return None
 
        argsCopy = list(args) # need to convert tuple to list to convert it
        if ";" not in args[0]:
            argsCopy[0] += ";h_" + argsCopy[0].replace(":","_")
 
        if kwargs.get("path",None) is None:
            # put in the Developer path, under the name of the algorithm
            kwargs["path"] = "Developer/" + self.alg.name
        elif kwargs["path"][-1] == '/':
            kwargs["path"] = kwargs["path"][:-1] # strip trailing slash
        # verify path obeys convention
        splitPath = kwargs["path"].split("/")
        if splitPath[0] not in ["Shifter","Expert","Developer"]:
            raise Exception("Path of histogram invalid, does not start with one of the allowed audiences (Shifter,Expert,Developer)")
 
        # don't bother registering Shifter plots in tier0 ... only needed in p1 and user environments
        if splitPath[0]=="Shifter" and self.dqEnv == 'tier0':
            return None
 
        # require a hanConfig not in Developer or a detail dir
        if splitPath[0] != "Developer" and splitPath[-1] != "detail" and ("algorithm" not in hanConfig):
            # will default to using GatherData as long as there is a description
            if "description" not in hanConfig:
                raise Exception("Must specify a hanConfig for a Shifter or Expert (non-detail) histogram")
            else:
                hanConfig["algorithm"] = "GatherData" # must have an algo, otherwise wont be valid han config
        elif "algorithm" in hanConfig and hanConfig["algorithm"] not in self.hanAlgConfigs and hanConfig["algorithm"] not in ["All_Bins_Filled","Histogram_Effective_Empty","Histogram_Empty","Histogram_Not_Empty","No_OverFlows","No_UnderFlows"]:
            # if algorithm specified, must have been defined
            raise Exception(f'DQ Algorithm {hanConfig["algorithm"]} for histogram {args[0]} not defined. Please use defineDQAlgorithm method to define it')
 
 
        if fillGroup is None: fillGroup = self.alg.name + "_fillGroup"
        if fillGroup not in self.fillGroups:
            self.fillGroups[fillGroup] = self.helper.addGroup(self.alg,fillGroup,topPath="L1Calo")
 
 
        if "merge" not in kwargs and kwargs.get("type","") !="TEfficiency":
            kwargs["merge"] = "merge" # ensures we don't get a warning about not specifying merge method
        out = self.fillGroups[fillGroup].defineHistogram(*argsCopy,**kwargs)
        histName = argsCopy[0].split(";")[-1]
 
        # add help link for all expert plots
        if splitPath[0] == "Expert":
            linkUrl = self.HELPURL + "#" + "".join(splitPath[1:]+[histName])
            linkUrl = f"<a href=\"{linkUrl}\">Help</a>"
            if "description" not in hanConfig: hanConfig["description"] = linkUrl
            else: hanConfig["description"] += " - " + linkUrl
 
        # only add to hanConfig if in Expert or Developer folder
        if splitPath[0] == "Expert" or splitPath[0] == "Developer":
            if "description" in hanConfig and hanConfig["description"]=="":
                del hanConfig["description"]
            splitPathWithHist = splitPath + [histName]
            x = L1CaloMonitorCfgHelper.hanConfigs
            for i,p in enumerate(splitPathWithHist):
                key = ("dir " if i!=len(splitPathWithHist)-1 else "hist ") + p
                if key not in x:
                    x[key] = {}
                x = x[key]
            hanConfig["output"] = "/".join(["L1Calo"]+splitPath)
            x.update(hanConfig)
        elif splitPath[0] == "Shifter": # record shifter histograms in another map, for generating xmls
            L1CaloMonitorCfgHelper.xmlConfigs["/".join(["L1Calo"]+splitPath+[histName])] = {}
            L1CaloMonitorCfgHelper.xmlConfigs["/".join(["L1Calo"]+splitPath+[histName])].update(hanConfig) # copy in the config
 
        return out
 

defineTree()

def LVL1CaloMonitoringConfig.L1CaloMonitorCfgHelper.defineTree	(		self,
		*	args,
			fillGroup = None,
		**	kwargs
	)

Definition at line 337 of file LVL1CaloMonitoringConfig.py.

    def defineTree(self,*args,fillGroup=None,**kwargs):
 
        if ";" not in args[0]:
            raise Exception("Must specify a tree name using ';name' suffix")
        treeName = args[0].split(";")[-1]
 
        if "," in args[1]: # catch a subtle typo that can screw up monitoring
            raise Exception("Should not have comma in list of branch names and types")
 
        if kwargs.get("path",None) is None:
            # put in the Developer path, under the name of the algorithm
            kwargs["path"] = "Developer/" + self.alg.name
 
        # verify path obeys convention
        splitPath = kwargs["path"].split("/")
        if splitPath[0] not in ["Developer"]:
            raise Exception("Path of tree invalid, must be in the audience=Developer directory")
 
 
        if fillGroup is None: fillGroup = self.alg.name + "_fillGroup"
        if fillGroup not in self.fillGroups:
            self.fillGroups[fillGroup] = self.helper.addGroup(self.alg,fillGroup,topPath="L1Calo")
 
        # always define a histogram to go with the tree, which will indicate how many entries there are
        # this also ensures we satisfy the requirement that every fillGroup has an object defined
        # (in the cases of running online/tier0 the defineTrees are blocked, but we still need a hist then)
        histName = "h_" + treeName + "_entries"
        argsCopy = list(args)
        argsCopy[0] = argsCopy[0].replace(";"+treeName,";"+histName)
        kwargsCopy = dict(kwargs)
        kwargsCopy["title"] = f"Number of Entries in {treeName} TTree" + ";" + ";".join(kwargsCopy.get("title","").split(";")[1:])
        kwargsCopy["opt"] = ['kCanRebin','kAddBinsDynamically','kAlwaysCreate']
        kwargsCopy["merge"] = "merge"
        is2d = (kwargsCopy["title"].count(";")>1)
        self.defineHistogram(argsCopy[0],type="TH2I" if is2d else "TH1I",xbins=1,xmin=0,xmax=1,ybins=1 if is2d else None,ymin=0,ymax=1,fillGroup=fillGroup,**kwargsCopy)
        if not any([x in self.dqEnv for x in ['tier0','online']]):
            out = self.fillGroups[fillGroup].defineTree(*args,**kwargs)
        else:
            out = None
        return out
 

printHanConfig()

def LVL1CaloMonitoringConfig.L1CaloMonitorCfgHelper.printHanConfig (   filename = "collisions_run.config" )

static

Definition at line 106 of file LVL1CaloMonitoringConfig.py.

    def printHanConfig(filename="collisions_run.config"):
        L1CaloMonitorCfgHelper.createXmls()
        from contextlib import redirect_stdout
        from PathResolver import PathResolver
        header = PathResolver.FindCalibFile("TrigT1CaloMonitoring/hanConfig_header.txt")
        with open(filename,'w') as f:
            # copy the header if we have one
            if header != "":
                with open(header,'r') as fh:
                    for line in fh: f.write(line)
            with redirect_stdout(f):
                # Note: common configs (see common dir in DataQualityConfiguration) provides following algorithms by default:
                #     algorithm All_Bins_Filled
                #     algorithm Histogram_Effective_Empty
                #     algorithm Histogram_Empty
                #     algorithm Histogram_Not_Empty
                #     algorithm No_OverFlows
                #     algorithm No_UnderFlows
 
                outputs = set()
                def printConf(d,prefix=""):
                    for key,value in d.items():
                        if type(value)==dict:
                            print(prefix,key,"{")
                            if(key=="dir detail" or key=="dir Developer"):
                                print(prefix+"  algorithm = GatherData") # define GatherData as default algo for all of these hists
                            if key.startswith("dir ") and any([x.startswith("hist ") for x in value.keys()]):
                                # this is a dir with hists, so specify output as the path .. take from the first hist child
                                for childKey,childVal in value.items():
                                    if childKey.startswith("hist ") and "output" in childVal:
                                        print(prefix+"  output = "+childVal["output"])
                                        break
                            printConf(value,prefix + "  ")
                            print(prefix,"}")
                        else:
                            # save all output paths to add to output block (don't need to print here b.c. specified at dir level
                            if key == "output": outputs.add(value)
                            else: print(prefix,key,"=",value)
 
 
                print("#inputs")
                print("dir L1Calo {")
                printConf(L1CaloMonitorCfgHelper.hanConfigs," ")
                print("}")
                print("#outputs")
                print("output top_level {")
                def printOutputs(d,prefix=""):
                    for key,value in d.items():
                        if(key.startswith("hist ")):
                            pass # do nothing
                        elif type(value)==dict:
                            print(prefix,key.replace("dir ","output "),"{")
                            if key=="dir Developer": print(prefix,"  algorithm = L1Calo_AlwaysUndefinedSummary")
                            printOutputs(value,prefix + "  ")
                            print(prefix,"}")
                print("  output L1Calo {")
                printOutputs(L1CaloMonitorCfgHelper.hanConfigs,"   ")
                print("  }")
                print("}")
                # include example of adding algorithms and thresholds
                print("""
#algorithms
algorithm AnyNonZeroBinIsError {
  # Use this algo if want error on any non-zero bin content
  libname = libdqm_summaries.so
  name = Bins_NotEqual_Threshold
  BinThreshold = 0.
  thresholds = th_AnyBinIsError
}
algorithm L1Calo_AlwaysUndefinedSummary {
    libname = libdqm_summaries.so
    name = AlwaysUndefinedSummary
}
""")
                for algName,algProps in L1CaloMonitorCfgHelper.hanAlgConfigs.items():
                    print(f"algorithm {algName} {{")
                    for propName,propVal in algProps.items():
                        print(f"  {propName} = {propVal}")
                    print("  }")
                print("""
#thresholds
thresholds th_AnyBinIsError {
    limits NBins {
      warning = 0
      error = 1
    }
}
""")
                for threshName,threshProps in L1CaloMonitorCfgHelper.hanThresholdConfigs.items():
                    print(f"thresholds {threshName} {{")
                    for parName,parLims in threshProps.items():
                        print(f"  limits {parName} {{")
                        for limName,limVal in parLims.items():
                            print(f"    {limName} = {limVal}")
                        print("  }")
                    print("}")
 
 
 

result()

def LVL1CaloMonitoringConfig.L1CaloMonitorCfgHelper.result

(

self

)

Definition at line 378 of file LVL1CaloMonitoringConfig.py.

    def result(self):
        return self.helper.result()
 
 

Member Data Documentation

alg

LVL1CaloMonitoringConfig.L1CaloMonitorCfgHelper.alg

Definition at line 215 of file LVL1CaloMonitoringConfig.py.

dqEnv

LVL1CaloMonitoringConfig.L1CaloMonitorCfgHelper.dqEnv

Definition at line 217 of file LVL1CaloMonitoringConfig.py.

fillGroups

LVL1CaloMonitoringConfig.L1CaloMonitorCfgHelper.fillGroups

Definition at line 216 of file LVL1CaloMonitoringConfig.py.

hanAlgConfigs

LVL1CaloMonitoringConfig.L1CaloMonitorCfgHelper.hanAlgConfigs

static

Definition at line 16 of file LVL1CaloMonitoringConfig.py.

hanConfigs

LVL1CaloMonitoringConfig.L1CaloMonitorCfgHelper.hanConfigs

static

Definition at line 15 of file LVL1CaloMonitoringConfig.py.

hanThresholdConfigs

LVL1CaloMonitoringConfig.L1CaloMonitorCfgHelper.hanThresholdConfigs

static

Definition at line 17 of file LVL1CaloMonitoringConfig.py.

helper

LVL1CaloMonitoringConfig.L1CaloMonitorCfgHelper.helper

Definition at line 214 of file LVL1CaloMonitoringConfig.py.

HELPURL

LVL1CaloMonitoringConfig.L1CaloMonitorCfgHelper.HELPURL

static

Definition at line 21 of file LVL1CaloMonitoringConfig.py.

SIGNATURES

LVL1CaloMonitoringConfig.L1CaloMonitorCfgHelper.SIGNATURES

static

Definition at line 20 of file LVL1CaloMonitoringConfig.py.

xmlConfigs

LVL1CaloMonitoringConfig.L1CaloMonitorCfgHelper.xmlConfigs

static

Definition at line 18 of file LVL1CaloMonitoringConfig.py.

---

The documentation for this class was generated from the following file:

• LVL1CaloMonitoringConfig.py