ATLAS Offline Software
Public Member Functions | Static Public Member Functions | Public Attributes | List of all members
python.TriggerConfigAccessBase.ConfigDBLoader Class Reference
Inheritance diagram for python.TriggerConfigAccessBase.ConfigDBLoader:
Collaboration diagram for python.TriggerConfigAccessBase.ConfigDBLoader:

Public Member Functions

def __init__ (self, configType, dbalias, dbkey)
 
def setQuery (self, query)
 
def getQueryDefinition (self, schemaVersion)
 
def load (self)
 
def getWriteFilename (self)
 
def confirmConfigType (self, config)
 

Static Public Member Functions

def getResolvedFileName (filename, pathenv="")
 
def getConnectionParameters (dbalias)
 
def getSchema (connStr)
 
def readSchemaVersion (qdict, session)
 
def getCoralQuery (session, queryStr, qdict=None)
 

Public Attributes

 dbalias
 
 dbkey
 
 query
 
 schema
 
 configType
 

Detailed Description

Definition at line 87 of file TriggerConfigAccessBase.py.

Constructor & Destructor Documentation

◆ __init__()

def python.TriggerConfigAccessBase.ConfigDBLoader.__init__ (   self,
  configType,
  dbalias,
  dbkey 
)

Definition at line 88 of file TriggerConfigAccessBase.py.

88  def __init__(self, configType, dbalias, dbkey):
89  super().__init__(configType)
90  self.dbalias = dbalias
91  self.dbkey = dbkey
92  self.query = None
93  self.schema = None
94 

Member Function Documentation

◆ confirmConfigType()

def python.TriggerConfigAccessBase.ConfigLoader.confirmConfigType (   self,
  config 
)
inherited
checks that the in-file specification of the configuration type matches the expected type

Definition at line 49 of file TriggerConfigAccessBase.py.

49  def confirmConfigType(self,config):
50  """
51  checks that the in-file specification of the configuration type matches the expected type
52  """
53  if config['filetype'] != self.configType:
54  raise RuntimeError("Can not load file with filetype '%s' when expecting '%s'" % (config['filetype'], self.configType.filetype))
55 

◆ getConnectionParameters()

def python.TriggerConfigAccessBase.ConfigDBLoader.getConnectionParameters (   dbalias)
static

Definition at line 115 of file TriggerConfigAccessBase.py.

115  def getConnectionParameters(dbalias):
116  dblookupFile = ConfigDBLoader.getResolvedFileName("dblookup.xml", "CORAL_DBLOOKUP_PATH")
117  dbp = ET.parse(dblookupFile)
118  listOfServices = None
119  for logSvc in dbp.iter("logicalservice"):
120  if logSvc.attrib["name"] != dbalias:
121  continue
122  listOfServices = [ serv.attrib["name"] for serv in logSvc.iter("service") ]
123  if len(listOfServices) == 0:
124  raise RuntimeError("DB %s has no services listed in %s" % (dbalias, dblookupFile))
125  break
126  if listOfServices is None:
127  raise RuntimeError("DB %s not available in %s" % (dbalias, dblookupFile))
128 
129  if "FRONTIER_SERVER" not in os.environ:
130  # remove all frontier connnections in the list if the environment FRONTIER_SERVER variable does not exist
131  # this speeds up the resolution of the connection specification (dbalias)
132  listOfServices = [svc for svc in listOfServices if not svc.startswith("frontier:")]
133 
134  # now get the account and pw for oracle connections
135  credentials = dict.fromkeys(listOfServices)
136 
137  for svc in filter(lambda s : s.startswith("frontier:"), listOfServices):
138  credentials[svc] = dict()
139  credentials[svc]["user"] = svc
140  credentials[svc]["password"] = ""
141 
142  try:
143  authFile = ConfigDBLoader.getResolvedFileName("authentication.xml", "CORAL_AUTH_PATH")
144  except Exception as e:
145  log.warning("File authentication.xml is not available! Oracle connection cannot be established. Exception message is: %s",e)
146  else:
147  for svc in filter(lambda s : s.startswith("oracle:"), listOfServices):
148  ap = ET.parse(authFile)
149  count = 0
150  for con in filter( lambda c: c.attrib["name"]==svc, ap.iter("connection")):
151  credentials[svc] = dict([(par.attrib["name"],par.attrib["value"]) for par in con])
152  count += 1
153  if count==0:
154  raise RuntimeError("No credentials found for connection %s from service %s for db %s" % (con,svc,dbalias))
155  if count>1:
156  raise RuntimeError("More than 1 connection found in %s for service %s" % (authFile, svc))
157 
158  return credentials
159 

◆ getCoralQuery()

def python.TriggerConfigAccessBase.ConfigDBLoader.getCoralQuery (   session,
  queryStr,
  qdict = None 
)
static
Parse output, tables and condition from the query string into coral query object

Definition at line 206 of file TriggerConfigAccessBase.py.

206  def getCoralQuery(session, queryStr, qdict = None):
207  ''' Parse output, tables and condition from the query string into coral query object'''
208  query = session.nominalSchema().newQuery()
209 
210  if qdict is not None:
211  queryStr = queryStr.format(**qdict)
212 
213  # bind vars
214  bindVars = coral.AttributeList()
215  bindVarsInQuery = re.findall(r":(\w*)", queryStr)
216  if len(bindVarsInQuery) > 0 and qdict is None:
217  log.error("Query has bound-variable syntax but no value dictionary is provided. Query: %s", queryStr)
218  for k in bindVarsInQuery:
219  bindVars.extend(k, "int")
220  bindVars[k].setData(qdict[k])
221 
222  output = queryStr.split("SELECT")[1].split("FROM")[0]
223  for field in output.split(','):
224  query.addToOutputList(field)
225 
226  log.debug("Conversion for Coral of query: %s", queryStr)
227 
228  for table in queryStr.split("FROM")[1].split("WHERE")[0].split(","):
229  tableSplit = list(filter(None, table.split(" ")))
230  # Schema name is stripped from TableList in Coral query
231  query.addToTableList(tableSplit[0].split(".")[1], tableSplit[1])
232 
233  if "WHERE" in queryStr:
234  cond = queryStr.split("WHERE")[1]
235  m = re.match("(.*)(?i: ORDER *BY )(.*)", cond) # check for "order by" clause
236  if m:
237  where, order = m.groups()
238  query.setCondition(where, bindVars)
239  query.addToOrderList(order)
240  else:
241  query.setCondition(cond, bindVars)
242 
243  return query
244 

◆ getQueryDefinition()

def python.TriggerConfigAccessBase.ConfigDBLoader.getQueryDefinition (   self,
  schemaVersion 
)
Choose query based on schema version, based on TrigConf::TrigDBLoader::getQueryDefinition 

Definition at line 245 of file TriggerConfigAccessBase.py.

245  def getQueryDefinition(self, schemaVersion):
246  '''Choose query based on schema version, based on TrigConf::TrigDBLoader::getQueryDefinition '''
247  maxDefVersion = 0
248  for vkey in self.query.keys():
249  if vkey>maxDefVersion and vkey<=schemaVersion:
250  maxDefVersion = vkey
251 
252  if maxDefVersion == 0:
253  raise RuntimeError("No query available for schema version {0}".format(schemaVersion))
254 
255  return self.query[maxDefVersion]
256 

◆ getResolvedFileName()

def python.TriggerConfigAccessBase.ConfigDBLoader.getResolvedFileName (   filename,
  pathenv = "" 
)
static
looks for file, first absolute, then by resolving envvar pathenv

Definition at line 103 of file TriggerConfigAccessBase.py.

103  def getResolvedFileName(filename, pathenv=""):
104  """ looks for file, first absolute, then by resolving envvar pathenv"""
105  if os.access(filename,os.R_OK):
106  return filename
107  pathlist = os.getenv(pathenv,'').split(os.pathsep)
108  for path in pathlist:
109  f = os.path.join( path, filename )
110  if os.access( f, os.R_OK ):
111  return f
112  raise RuntimeError("Can't read file %s, neither locally nor in %s" % (filename, pathenv) )
113 

◆ getSchema()

def python.TriggerConfigAccessBase.ConfigDBLoader.getSchema (   connStr)
static
Read schema from connection string 

Definition at line 161 of file TriggerConfigAccessBase.py.

161  def getSchema(connStr):
162  ''' Read schema from connection string '''
163  if connStr.startswith("oracle:"):
164  [_, schema] = connStr.split("/")[-2:]
165  return schema
166 
167  if connStr.startswith("frontier:"):
168  import re
169  pattern = r"frontier://ATLF/\‍(\‍)/(.*)"
170  m = re.match(pattern, connStr)
171  if not m:
172  raise RuntimeError("connection string '%s' doesn't match the pattern '%s'?" % (connStr, pattern))
173  (schema, ) = m.groups()
174  return schema
175 
176  if connStr.startswith("sqlite_file:"):
177  raise NotImplementedError("Python-loading of trigger configuration from sqlite has not yet been implemented")
178 

◆ getWriteFilename()

def python.TriggerConfigAccessBase.ConfigDBLoader.getWriteFilename (   self)

Definition at line 337 of file TriggerConfigAccessBase.py.

337  def getWriteFilename(self):
338  return "{basename}_{schema}_{dbkey}.json".format(basename = self.configType.basename, schema = self.schema, dbkey = self.dbkey)
339 

◆ load()

def python.TriggerConfigAccessBase.ConfigDBLoader.load (   self)

Definition at line 257 of file TriggerConfigAccessBase.py.

257  def load(self):
258  credentials: dict[str,Any] = ConfigDBLoader.getConnectionParameters(self.dbalias)
259 
260  if not credentials:
261  log.error("No TriggerDB connections found for %s", self.dbalias)
262  raise RuntimeError(f"No TriggerDB connections found for {self.dbalias}")
263 
264  svc = coral.ConnectionService()
265  svcconfig = svc.configuration()
266  svcconfig.disablePoolAutomaticCleanUp()
267  svcconfig.setConnectionTimeOut(0)
268 
269  failureMode = 0
270  for credential in credentials:
271  log.debug("Trying credentials %s",credential)
272 
273  try:
274  session = svc.connect(credential, coral.access_ReadOnly)
275  except Exception as e:
276  log.warning("Failed to establish connection: %s",e)
277  failureMode = max(1, failureMode)
278  continue
279 
280  # Check that the FRONTIER_SERVER is set properly, if not reduce the retrial period and time out values
281  if 'FRONTIER_SERVER' in os.environ and os.environ['FRONTIER_SERVER']:
282  svcconfig.setConnectionRetrialPeriod(300)
283  svcconfig.setConnectionRetrialTimeOut(3600)
284  else:
285  svcconfig.setConnectionRetrialPeriod(1)
286  svcconfig.setConnectionRetrialTimeOut(1)
287 
288  try:
289  session.transaction().start(True) # readOnly
290  self.schema = ConfigDBLoader.getSchema(credential)
291  qdict = { "schema" : self.schema, "dbkey" : self.dbkey }
292 
293  # Choose query based on schema
294  schemaVersion = ConfigDBLoader.readSchemaVersion(qdict, session)
295  qstr = self.getQueryDefinition(schemaVersion)
296  # execute coral query
297  query = ConfigDBLoader.getCoralQuery(session, qstr, qdict)
298  cursor = query.execute()
299 
300  except Exception as e:
301  log.warning(f"DB query on {credential} failed to execute.")
302  log.warning("Exception message: %r", e)
303  failureMode = max(2, failureMode)
304  continue # to next source
305 
306  # Read query result
307  if not cursor.next():
308  # empty result
309  log.warning(f"DB query on {credential} returned empty result, likely due to non-existing key {self.dbkey}")
310  failureMode = 3
311  continue # to next source
312 
313  configblob = cursor.currentRow()[0].data()
314  if type(configblob) is not str:
315  configblob = configblob.readline()
316  config = json.loads(configblob)
317  session.transaction().commit()
318 
319  self.confirmConfigType(config)
320  return config
321 
322  if failureMode == 1:
323  log.error("TriggerDB query: could not connect to any source for %s", self.configType.basename)
324  log.error("Considered sources: %s", ", ".join(credentials))
325  raise RuntimeError("TriggerDB query: could not connect to any source", self.configType.basename)
326  if failureMode == 2:
327  log.error("Query failed due to wrong definition for %s", self.configType.basename)
328  log.error("DB query was: %s", qstr.format(**qdict))
329  raise RuntimeError("Query failed due to wrong definition", self.configType.basename)
330  elif failureMode == 3:
331  log.error("DB key %s does not exist for %s", self.dbkey, self.configType.basename)
332  raise KeyError("DB key does not exist", self.dbkey, self.configType.basename)
333  else:
334  raise RuntimeError("Query failed for unknown reason")
335 

◆ readSchemaVersion()

def python.TriggerConfigAccessBase.ConfigDBLoader.readSchemaVersion (   qdict,
  session 
)
static
Read schema version form database, based on TrigConf::TrigDBLoader::schemaVersion 

Definition at line 180 of file TriggerConfigAccessBase.py.

180  def readSchemaVersion(qdict, session):
181  ''' Read schema version form database, based on TrigConf::TrigDBLoader::schemaVersion '''
182  try:
183  q = "SELECT TS_TAG FROM {schema}.TRIGGER_SCHEMA TS"
184  query = ConfigDBLoader.getCoralQuery(session, q.format(**qdict))
185  cursor = query.execute()
186  cursor.next()
187 
188  versionTag = cursor.currentRow()[0].data()
189 
190  versionTagPrefix = "Trigger-Run3-Schema-v"
191  if not versionTag.startswith(versionTagPrefix):
192  raise RuntimeError( "Tag format error: Trigger schema version tag %s does not start with %s", versionTag, versionTagPrefix)
193 
194  vstr = versionTag[len(versionTagPrefix)]
195 
196  if not vstr.isdigit():
197  raise RuntimeError( "Invalid argument when interpreting the version part %s of schema tag %s is %s", vstr, versionTag, type(vstr))
198 
199  log.debug("Found schema version %s", vstr)
200  return int(vstr)
201 
202  except Exception as e:
203  log.warning("Failed to read schema version: %r", e)
204 

◆ setQuery()

def python.TriggerConfigAccessBase.ConfigDBLoader.setQuery (   self,
  query 
)
query template is a dictionary of queries, identified by schema version, 
similar to TrigConf::TrigDBMenuLoader::m_hltQueries and TrigConf::TrigDBMenuLoader::m_l1Queries

Definition at line 95 of file TriggerConfigAccessBase.py.

95  def setQuery(self, query):
96  """
97  query template is a dictionary of queries, identified by schema version,
98  similar to TrigConf::TrigDBMenuLoader::m_hltQueries and TrigConf::TrigDBMenuLoader::m_l1Queries
99  """
100  self.query = query
101 

Member Data Documentation

◆ configType

python.TriggerConfigAccessBase.ConfigLoader.configType
inherited

Definition at line 48 of file TriggerConfigAccessBase.py.

◆ dbalias

python.TriggerConfigAccessBase.ConfigDBLoader.dbalias

Definition at line 90 of file TriggerConfigAccessBase.py.

◆ dbkey

python.TriggerConfigAccessBase.ConfigDBLoader.dbkey

Definition at line 91 of file TriggerConfigAccessBase.py.

◆ query

python.TriggerConfigAccessBase.ConfigDBLoader.query

Definition at line 92 of file TriggerConfigAccessBase.py.

◆ schema

python.TriggerConfigAccessBase.ConfigDBLoader.schema

Definition at line 93 of file TriggerConfigAccessBase.py.


The documentation for this class was generated from the following file:
data
char data[hepevt_bytes_allocation_ATLAS]
Definition: HepEvt.cxx:11
vtune_athena.format
format
Definition: vtune_athena.py:14
max
constexpr double max()
Definition: ap_fixedTest.cxx:33
mergePhysValFiles.start
start
Definition: DataQuality/DataQualityUtils/scripts/mergePhysValFiles.py:13
python.CaloAddPedShiftConfig.type
type
Definition: CaloAddPedShiftConfig.py:42
covarianceTool.filter
filter
Definition: covarianceTool.py:514
histSizes.list
def list(name, path='/')
Definition: histSizes.py:38
TCS::join
std::string join(const std::vector< std::string > &v, const char c=',')
Definition: Trigger/TrigT1/L1Topo/L1TopoCommon/Root/StringUtils.cxx:10
python.processes.powheg.ZZ.ZZ.__init__
def __init__(self, base_directory, **kwargs)
Constructor: all process options are set here.
Definition: ZZ.py:18
python.CaloAddPedShiftConfig.int
int
Definition: CaloAddPedShiftConfig.py:45
python.Bindings.keys
keys
Definition: Control/AthenaPython/python/Bindings.py:801
calibdata.commit
bool commit
Definition: calibdata.py:831
python.root_pickle.load
def load(f, use_proxy=1, key=None)
Definition: root_pickle.py:476
Trk::split
@ split
Definition: LayerMaterialProperties.h:38