ATLAS Offline Software
Loading...
Searching...
No Matches
Classes | Functions | Variables
python.db Namespace Reference

Classes

class  Databases
class  DefectsDB

Functions

 make_safe_fields (fields)
 get_query_range (since, until, runs)
 fetch_iovs (folder_name, since=None, until=None, channels=None, tag="", what="all", max_records=-1, with_channel=True, loud=False, database=None, convert_time=False, named_channels=False, selection=None, runs=None, with_time=False, unicode_strings=False)
 write_iovs (folder_name, iovs, record, multiversion=True, tag="", create=False, storage_buffer=False)

Variables

 log
str DEFAULT_DBNAME = "CONDBR2"

Detailed Description

Authors: Peter Waller <peter.waller@cern.ch> and "Peter Onyisi" <peter.onyisi@cern.ch>

Please see https://twiki.cern.ch/twiki/bin/viewauth/Atlas/DQDefects

This file defines DefectsDB and some of its public interface.

To separate out the code into digestable units, this class is split into mixins
which can be found defined in other files in this directory.

The bulk of the core functionality is found in this file.

Function Documentation

◆ fetch_iovs()

python.db.fetch_iovs ( folder_name,
since = None,
until = None,
channels = None,
tag = "",
what = "all",
max_records = -1,
with_channel = True,
loud = False,
database = None,
convert_time = False,
named_channels = False,
selection = None,
runs = None,
with_time = False,
unicode_strings = False ) 
Helper to fetch objects in a pythonic manner
`folder_name` may be an abbreviated name (DQMFONL) or a fully-qualified name
    (e.g. /GLOBAL/DETSTATUS/DQMFONL)
`since`, `until` can be (run, lumi) tuples, or standard iov keys
`channels` can be a cool ChannelSelection object or a list of ids/names
`tag` COOL folder tag
`what` is a list of strings specifying which records should be fetched
    if it is the string "all" (not a list), then all records are fetched,
    and naming is turned on.
`max_records` specifies the maximum number of records to fetch. -1 means all
`with_channel` specifies whether the channel number should be in the result
    list of each tuple
`loud` specifies whether quick_retrieve (C++ function) should print its
    status every 1000 objects
`database` can be used to specify an abbreviated database, or a connection 
    string
`convert_time` performs a conversion of `since` and `until` from runlumi
    to nanoseconds since the epoch.
`named_channels` causes the iovs returned to contain strings in the channel 
    identifier
`selection` [NOT IMPLEMENTED YET] create a cool selection object
`runs` if it is an integer, it is a run number. If it is a tuple, it is a
    run range.
`with_time` retrieve insertiontime for iovs
`unicode_strings` return unicode string objects, assuming database content 
                  is UTF-8

Definition at line 65 of file DQUtils/python/db.py.

68 selection=None, runs=None, with_time=False, unicode_strings=False):
69 """
70 Helper to fetch objects in a pythonic manner
71 `folder_name` may be an abbreviated name (DQMFONL) or a fully-qualified name
72 (e.g. /GLOBAL/DETSTATUS/DQMFONL)
73 `since`, `until` can be (run, lumi) tuples, or standard iov keys
74 `channels` can be a cool ChannelSelection object or a list of ids/names
75 `tag` COOL folder tag
76 `what` is a list of strings specifying which records should be fetched
77 if it is the string "all" (not a list), then all records are fetched,
78 and naming is turned on.
79 `max_records` specifies the maximum number of records to fetch. -1 means all
80 `with_channel` specifies whether the channel number should be in the result
81 list of each tuple
82 `loud` specifies whether quick_retrieve (C++ function) should print its
83 status every 1000 objects
84 `database` can be used to specify an abbreviated database, or a connection
85 string
86 `convert_time` performs a conversion of `since` and `until` from runlumi
87 to nanoseconds since the epoch.
88 `named_channels` causes the iovs returned to contain strings in the channel
89 identifier
90 `selection` [NOT IMPLEMENTED YET] create a cool selection object
91 `runs` if it is an integer, it is a run number. If it is a tuple, it is a
92 run range.
93 `with_time` retrieve insertiontime for iovs
94 `unicode_strings` return unicode string objects, assuming database content
95 is UTF-8
96 """
97 from .quick_retrieve import quick_retrieve, browse_coracool, get_coracool_payload_spec
98
99 if channels == []: return IOVSet()
100
101 since, until = get_query_range(since, until, runs)
102
103 channel_mapping = None
104 if isinstance(folder_name, str):
105 folder = Databases.get_folder(folder_name, database)
106 else:
107 try:
108 folder = folder_name
109 folder_name = folder.fullPath()
110 except Exception:
111 log.error("Exception when interpreting folder: {0}".format(folder_name))
112 raise
113
114 log.info("Querying %s", folder_name)
115 log.debug("Query range: [%s, %s]", since, until)
116
117 short_folder = folder.fullPath().split("/")[-1]
118
119 time_based_folder = "<timeStamp>time</timeStamp>" in folder.description()
120 coracool_folder = "<coracool>" in folder.description()
121 iov_key_type = TimestampType if time_based_folder else RunLumiType
122
123 if time_based_folder and (convert_time or runs):
124 # Perform a conversion of the run IoV to a time-based one.
125 # Note: probably inadvisable to do this for long ranges since
126 # it has to retrieve all of the luminosity blocks that took place
127 # in the query range.
128
129 until = min(until, RunLumi(100000000, 0))
130 runrange = fetch_iovs("LBLB", since, until)
131 if runrange:
132 # If the runrange is empty for some reason, fall back.
133 since, until = runrange.first.StartTime, runrange.last.EndTime
134 return fetch_iovs(folder_name, since, until, channels, tag, what,
135 max_records, with_channel, loud,
136 database, convert_time=False,
137 named_channels=named_channels, selection=selection,
138 with_time=with_time,
139 unicode_strings=unicode_strings)
140 else:
141 return IOVSet()
142
143 detstatus_names = "DQMFOFL", "DCSOFL", "DQMFONL", "SHIFTOFL", "SHIFTONL", "LBSUMM"
144 if any(short_folder.endswith(x) for x in detstatus_names):
145 channel_mapping = None # get channel mapping from channel_mapping.py
146 else:
147 _, _, channelmap = get_channel_ids_names(folder)
148 cm_reversed = {value: key for key, value in channelmap.items()}
149 channelmap.update(cm_reversed)
150 channel_mapping = channelmap
151
152 channels = make_channelselection(channels, channel_mapping)
153
154 field_name = "%s_VAL" % short_folder
155
156 if not coracool_folder:
157 if what == "all":
158 what = folder.folderSpecification().payloadSpecification().keys()
159
160 channelmap = None
161 if named_channels:
162 _, _, channelmap = get_channel_ids_names(folder)
163
164 folder.setPrefetchAll(False)
165
166 if selection:
167 sel = make_browse_objects_selection(folder, selection)
168 iterator = folder.browseObjects(since, until, channels, tag, sel)
169 else:
170 iterator = folder.browseObjects(since, until, channels, tag)
171
172 fields = []
173 if with_time:
174 fields.append("insertion_time")
175 if with_channel:
176 fields.append("channel")
177 fields.extend(what)
178
179 record = make_iov_type(field_name, make_safe_fields(fields))
180
181 result = quick_retrieve(iterator, record, what, max_records, with_channel,
182 loud, iov_key_type, channelmap, with_time,
183 unicode_strings)
184
185 else:
186 args = folder_name, database
187 database, folder_path = Databases.resolve_folder_string(*args)
188
189 # Coracool
190 assert database, "Coracool folder - you must specify a database"
191
192 db = Databases.get_instance(database)
193 spec = get_coracool_payload_spec(db, folder_path)
194 if what == "all":
195 what = spec.keys()
196
197 assert isinstance(what, list), ("Coracool folder - you must specify "
198 "`what` to fetch (it cannot be inferred, as with non-coracool.)")
199
200 record = make_iov_type(field_name, ["channel", "elements"])
201
202 element = namedtuple("element", " ".join(make_safe_fields(what)))
203
204 result = browse_coracool(db, folder_path, since, until, channels, "",
205 what, record, element, iov_key_type)
206
207 result = IOVSet(result, iov_type=record, origin=short_folder)
208
209 return result
210
min
#define min(a, b)
Definition cfImp.cxx:40
get_coracool_payload_spec
const cool::RecordSpecification get_coracool_payload_spec(IDatabasePtr cooldb, const string &folder)
Definition pythonic_coracool.cxx:175
quick_retrieve
PyObject * quick_retrieve(const IObjectIteratorPtr &objects, PyObject *object_converter, PyObject *to_fetch=NULL, const long max_records=-1, const bool with_channel=true, const bool loud=false, PyObject *iovkey_wrapper=NULL, PyObject *channel_name_mapping=NULL, const bool with_time=false, const bool as_unicode=false)
Definition quick_retrieve.cxx:219
browse_coracool
PyObject * browse_coracool(IDatabasePtr cooldb, const string &folder, ValidityKey since, ValidityKey until, const ChannelSelection &cs=ChannelSelection::all(), const char *tag="", PyObject *to_fetch=NULL, PyObject *object_converter=NULL, PyObject *inner_object_converter=NULL, PyObject *iovkey_wrapper=NULL)
Definition pythonic_coracool.cxx:189
split
std::vector< std::string > split(const std::string &s, const std::string &t=":")
Definition hcg.cxx:177

◆ get_query_range()

python.db.get_query_range ( since,
until,
runs ) 
Take `since`, `until` and `runs` and turn them into parameters.

If nothing is specified, an infinite range is used.
If `runs` is an integer, just that run is used
If `runs` is a two-tuple, then it is used as a (from_run, to_run)

Definition at line 27 of file DQUtils/python/db.py.

27def get_query_range(since, until, runs):
28 """
29 Take `since`, `until` and `runs` and turn them into parameters.
30
31 If nothing is specified, an infinite range is used.
32 If `runs` is an integer, just that run is used
33 If `runs` is a two-tuple, then it is used as a (from_run, to_run)
34 """
35 if runs and (since is None and until is None):
36 from builtins import int
37 if isinstance(runs, tuple):
38 since, until = (runs[0], 0), (runs[1], 0)
39
40 elif isinstance(runs, int):
41 since, until = (runs, 0), (runs+1, 0)
42 else:
43 raise RuntimeError("Invalid type for `runs`, should be int or tuple")
44
45 elif runs:
46 raise RuntimeError("Specify (since and/or until), OR runs, not both")
47
48 else:
49 if since is None: since = 0
50 if until is None: until = 2**63-1
51
52 if isinstance(since, tuple): since = RunLumi(*since)
53 if isinstance(until, tuple): until = RunLumi(*until)
54
55 if isinstance(since, str): since = TimestampType.from_string(since)
56 if isinstance(until, str): until = TimestampType.from_string(until)
57
58 if isinstance(since, datetime): since = TimestampType.from_date(since)
59 if isinstance(until, datetime): until = TimestampType.from_date(until)
60
61 assert since <= until, "Bad query range (since > until?)"
62
63 return since, until
64

◆ make_safe_fields()

python.db.make_safe_fields ( fields)

Definition at line 24 of file DQUtils/python/db.py.

24def make_safe_fields(fields):
25 return [(field + "_") if iskeyword(field) else field for field in fields]
26

◆ write_iovs()

python.db.write_iovs ( folder_name,
iovs,
record,
multiversion = True,
tag = "",
create = False,
storage_buffer = False )

Definition at line 211 of file DQUtils/python/db.py.

212 create=False, storage_buffer=False):
213 args = folder_name, multiversion, record, create
214 db, folder, payload = Databases.fetch_for_writing(*args)
215
216 if storage_buffer:
217 folder.setupStorageBuffer()
218
219 total_iovs = len(iovs)
220 for i, iov in enumerate(iovs):
221 for field_name, field_value in zip(iov._fields[3:], iov[3:]):
222 payload[field_name] = field_value
223
224 folder.storeObject(iov.since, iov.until, payload, iov.channel, tag)
225 if not i % 1000:
226 log.debug("Wrote %5i / %5i records", i, total_iovs)
227
228 if storage_buffer:
229 log.debug("Flushing records to database...")
230 folder.flushStorageBuffer()
231 log.debug("... done")
232

Variable Documentation

◆ DEFAULT_DBNAME

str python.db.DEFAULT_DBNAME = "CONDBR2"

Definition at line 22 of file DQUtils/python/db.py.

◆ log

python.db.log

Definition at line 17 of file DQDefects/python/db.py.

Generated on for ATLAS Offline Software by doxygen 1.14.0