ATLAS Offline Software
Loading...
Searching...
No Matches
Public Member Functions | Public Attributes | Protected Member Functions | Protected Attributes | List of all members
python.trfExe.archiveExecutor Class Reference

Archive transform. More...

Inheritance diagram for python.trfExe.archiveExecutor:
Collaboration diagram for python.trfExe.archiveExecutor:

Public Member Functions

 preExecute (self, input=set(), output=set())
 exeArgs (self)
 exeArgs (self, value)
 exe (self)
 exe (self, value)
 execute (self)
 postExecute (self)
 validate (self)
 inData (self)
 inData (self, value)
 outData (self)
 outData (self, value)
 myMerger (self)
 Now define properties for these data members.
 name (self)
 name (self, value)
 substep (self)
 trf (self)
 trf (self, value)
 inDataUpdate (self, value)
 outDataUpdate (self, value)
 input (self)
 output (self)
 extraMetadata (self)
 hasExecuted (self)
 rc (self)
 errMsg (self)
 validation (self)
 validation (self, value)
 hasValidated (self)
 isValidated (self)
 first (self)
 preExeStartTimes (self)
 exeStartTimes (self)
 exeStopTimes (self)
 valStartTimes (self)
 valStopTimes (self)
 preExeCpuTime (self)
 preExeWallTime (self)
 cpuTime (self)
 usrTime (self)
 sysTime (self)
 wallTime (self)
 memStats (self)
 memAnalysis (self)
 postExeCpuTime (self)
 postExeWallTime (self)
 validationCpuTime (self)
 validationWallTime (self)
 cpuTimeTotal (self)
 wallTimeTotal (self)
 eventCount (self)
 reSimEvent (self)
 athenaMP (self)
 dbMonitor (self)
 setPreExeStart (self)
 setValStart (self)
 doAll (self, input=set(), output=set())
 Convenience function.

Public Attributes

 exeArgs
 conf = conf
 Executor configuration:
 inData = value
 outData = value

Protected Member Functions

 _buildStandardCommand (self)

Protected Attributes

 _exe = exe
 _exeArgs = exeArgs
bool _echoOutput = False
list _cmd = None
bool _memMonitor = memMonitor
 _input = input
 _output = output
str _logFileName = "log.{0}".format(self._name)
 _echologger = logging.getLogger(self._name)
 _exeLogFile = logging.FileHandler(self._logFileName, mode='w', **encargs)
 _echostream = logging.StreamHandler(_encoding_stream(sys.stdout))
 _workdir
str _memSummaryFile = 'prmon.summary.' + self._name + '.json'
 _name = forceToAlphaNum(name)
 _inData = set(inData)
 _outData = set(outData)
bool _hasExecuted = False
int _rc = -1
str _errMsg = None
bool _hasValidated = False
bool _isValidated = False
dict _extraMetadata = {}
 _preExeStart = None
 _exeStart = None
 _valStart = None
dict _memStats = {}
dict _memLeakResult = {}
 _memFullFile = None
 _eventCount = None
int _athenaMP = 0
int _athenaMT = 0
int _athenaConcurrentEvents = 0
 _dbMonitor = None
 _resimevents = None
 _alreadyInContainer = None
 _containerSetup = None
list _myMerger = []
 _trf = value
 _validation = value
 _exeStop
 _valStop

Detailed Description

Archive transform.

Definition at line 2273 of file trfExe.py.

Member Function Documentation

◆ _buildStandardCommand()

python.trfExe.scriptExecutor._buildStandardCommand ( self)
protectedinherited

Definition at line 706 of file trfExe.py.

706 def _buildStandardCommand(self):
707 if self._exe:
708 self._cmd = [self.exe, ]
709 else:
710 raise trfExceptions.TransformExecutionException(trfExit.nameToCode('TRF_EXEC_SETUP_FAIL'),
711 'No executor set in {0}'.format(self.__class__.__name__))
712 for arg in self.exeArgs:
713 if arg in self.conf.argdict:
714 # If we have a list then add each element to our list, else just str() the argument value
715 # Note if there are arguments which need more complex transformations then
716 # consider introducing a special toExeArg() method.
717 if isinstance(self.conf.argdict[arg].value, list):
718 self._cmd.extend([ str(v) for v in self.conf.argdict[arg].value])
719 else:
720 self._cmd.append(str(self.conf.argdict[arg].value))
721
722

◆ athenaMP()

python.trfExe.transformExecutor.athenaMP ( self)
inherited

Definition at line 452 of file trfExe.py.

452 def athenaMP(self):
453 return self._athenaMP
454

◆ cpuTime()

python.trfExe.transformExecutor.cpuTime ( self)
inherited

Definition at line 366 of file trfExe.py.

366 def cpuTime(self):
367 if self._exeStart and self._exeStop:
368 return calcCpuTime(self._exeStart, self._exeStop)
369 else:
370 return None
371

◆ cpuTimeTotal()

python.trfExe.transformExecutor.cpuTimeTotal ( self)
inherited

Definition at line 430 of file trfExe.py.

430 def cpuTimeTotal(self):
431 if self._preExeStart and self._valStop:
432 return calcCpuTime(self._preExeStart, self._valStop)
433 else:
434 return None
435

◆ dbMonitor()

python.trfExe.transformExecutor.dbMonitor ( self)
inherited

Definition at line 456 of file trfExe.py.

456 def dbMonitor(self):
457 return self._dbMonitor
458
459

◆ doAll()

python.trfExe.transformExecutor.doAll ( self,
input = set(),
output = set() )
inherited

Convenience function.

Definition at line 499 of file trfExe.py.

499 def doAll(self, input=set(), output=set()):
500 self.preExecute(input, output)
501 self.execute()
502 self.postExecute()
503 self.validate()
504
set
STL class.

◆ errMsg()

python.trfExe.transformExecutor.errMsg ( self)
inherited

Definition at line 304 of file trfExe.py.

304 def errMsg(self):
305 return self._errMsg
306

◆ eventCount()

python.trfExe.transformExecutor.eventCount ( self)
inherited

Definition at line 444 of file trfExe.py.

444 def eventCount(self):
445 return self._eventCount
446

◆ exe() [1/2]

python.trfExe.scriptExecutor.exe ( self)
inherited

Definition at line 641 of file trfExe.py.

641 def exe(self):
642 return self._exe
643

◆ exe() [2/2]

python.trfExe.scriptExecutor.exe ( self,
value )
inherited

Definition at line 645 of file trfExe.py.

645 def exe(self, value):
646 self._exe = value
647 self._extraMetadata['script'] = value
648

◆ exeArgs() [1/2]

python.trfExe.scriptExecutor.exeArgs ( self)
inherited

Definition at line 650 of file trfExe.py.

650 def exeArgs(self):
651 return self._exeArgs
652

◆ exeArgs() [2/2]

python.trfExe.scriptExecutor.exeArgs ( self,
value )
inherited

Definition at line 654 of file trfExe.py.

654 def exeArgs(self, value):
655 self._exeArgs = value
656# self._extraMetadata['scriptArgs'] = value
657

◆ execute()

python.trfExe.scriptExecutor.execute ( self)
inherited

Reimplemented from python.trfExe.transformExecutor.

Reimplemented in python.trfExe.bsMergeExecutor, and python.trfExe.POOLMergeExecutor.

Definition at line 723 of file trfExe.py.

723 def execute(self):
724 self._hasExecuted = True
725 msg.info('Starting execution of {0} ({1})'.format(self._name, self._cmd))
726
727 self._exeStart = os.times()
728 msg.debug('exeStart time is {0}'.format(self._exeStart))
729 if ('execOnly' in self.conf.argdict and self.conf.argdict['execOnly'] is True):
730 msg.info('execOnly flag is set - execution will now switch, replacing the transform')
731 os.execvp(self._cmd[0], self._cmd)
732
733 encargs = {'encoding' : 'utf8'}
734 try:
735 # if we are already inside a container, then
736 # must map /srv of the nested container to /srv of the parent container:
737 # https://twiki.atlas-canada.ca/bin/view/AtlasCanada/Containers#Mount_point_summary
738 if self._alreadyInContainer and self._containerSetup is not None:
739 msg.info("chdir /srv to launch a nested container for the substep")
740 os.chdir("/srv")
741 p = subprocess.Popen(self._cmd, shell = False, stdout = subprocess.PIPE, stderr = subprocess.STDOUT, bufsize = 1, **encargs)
742 # go back to the original working directory immediately after to store prmon in it
743 if self._alreadyInContainer and self._containerSetup is not None:
744 msg.info("chdir {} after launching the nested container".format(self._workdir))
745 os.chdir(self._workdir)
746
747 if self._memMonitor:
748 try:
749 self._memSummaryFile = 'prmon.summary.' + self._name + '.json'
750 self._memFullFile = 'prmon.full.' + self._name
751 memMonitorCommand = ['prmon', '--pid', str(p.pid), '--filename', 'prmon.full.' + self._name,
752 '--json-summary', self._memSummaryFile, '--log-filename', 'prmon.' + self._name + '.log',
753 '--interval', '30']
754 mem_proc = subprocess.Popen(memMonitorCommand, shell = False, close_fds=True, **encargs)
755 # TODO - link mem.full.current to mem.full.SUBSTEP
756 except Exception as e:
757 msg.warning('Failed to spawn memory monitor for {0}: {1}'.format(self._name, e))
758 self._memMonitor = False
759
760 while p.poll() is None:
761 line = p.stdout.readline()
762 if line:
763 self._echologger.info(line.rstrip())
764 # Hoover up remaining buffered output lines
765 for line in p.stdout:
766 self._echologger.info(line.rstrip())
767
768 self._rc = p.returncode
769 msg.info('%s executor returns %d', self._name, self._rc)
770 self._exeStop = os.times()
771 msg.debug('exeStop time is {0}'.format(self._exeStop))
772 except OSError as e:
773 errMsg = 'Execution of {0} failed and raised OSError: {1}'.format(self._cmd[0], e)
774 msg.error(errMsg)
775 raise trfExceptions.TransformExecutionException(trfExit.nameToCode('TRF_EXEC'), errMsg)
776 finally:
777 if self._memMonitor:
778 try:
779 mem_proc.send_signal(signal.SIGUSR1)
780 countWait = 0
781 while (not mem_proc.poll()) and countWait < 10:
782 time.sleep(0.1)
783 countWait += 1
784 except OSError:
785 pass
786
787

◆ exeStartTimes()

python.trfExe.transformExecutor.exeStartTimes ( self)
inherited

Definition at line 336 of file trfExe.py.

336 def exeStartTimes(self):
337 return self._exeStart
338

◆ exeStopTimes()

python.trfExe.transformExecutor.exeStopTimes ( self)
inherited

Definition at line 340 of file trfExe.py.

340 def exeStopTimes(self):
341 return self._exeStop
342

◆ extraMetadata()

python.trfExe.transformExecutor.extraMetadata ( self)
inherited

Definition at line 292 of file trfExe.py.

292 def extraMetadata(self):
293 return self._extraMetadata
294

◆ first()

python.trfExe.transformExecutor.first ( self)
inherited
Note
At the moment only athenaExecutor sets this property, but that might be changed...

Definition at line 325 of file trfExe.py.

325 def first(self):
326 if hasattr(self, '_first'):
327 return self._first
328 else:
329 return None
330

◆ hasExecuted()

python.trfExe.transformExecutor.hasExecuted ( self)
inherited

Definition at line 296 of file trfExe.py.

296 def hasExecuted(self):
297 return self._hasExecuted
298

◆ hasValidated()

python.trfExe.transformExecutor.hasValidated ( self)
inherited

Definition at line 316 of file trfExe.py.

316 def hasValidated(self):
317 return self._hasValidated
318

◆ inData() [1/2]

python.trfExe.transformExecutor.inData ( self)
inherited

Definition at line 235 of file trfExe.py.

235 def inData(self):
236
237 if '_inData' in dir(self):
238 return self._inData
239 return None
240

◆ inData() [2/2]

python.trfExe.transformExecutor.inData ( self,
value )
inherited

Definition at line 242 of file trfExe.py.

242 def inData(self, value):
243 self._inData = set(value)
244

◆ inDataUpdate()

python.trfExe.transformExecutor.inDataUpdate ( self,
value )
inherited

Definition at line 245 of file trfExe.py.

245 def inDataUpdate(self, value):
246
247 if '_inData' in dir(self):
248 self._inData.update(value)
249 else:
250
251 self.inData = value
252
253

◆ input()

python.trfExe.transformExecutor.input ( self)
inherited
Note
This returns the actual input data with which this executor ran (c.f. inData which returns all the possible data types this executor could run with)

Definition at line 276 of file trfExe.py.

276 def input(self):
277
278 if '_input' in dir(self):
279 return self._input
280 return None
281

◆ isValidated()

python.trfExe.transformExecutor.isValidated ( self)
inherited

Definition at line 320 of file trfExe.py.

320 def isValidated(self):
321 return self._isValidated
322

◆ memAnalysis()

python.trfExe.transformExecutor.memAnalysis ( self)
inherited

Definition at line 398 of file trfExe.py.

398 def memAnalysis(self):
399 return self._memLeakResult
400

◆ memStats()

python.trfExe.transformExecutor.memStats ( self)
inherited

Definition at line 394 of file trfExe.py.

394 def memStats(self):
395 return self._memStats
396

◆ myMerger()

python.trfExe.transformExecutor.myMerger ( self)
inherited

Now define properties for these data members.

Definition at line 207 of file trfExe.py.

207 def myMerger(self):
208 return self._myMerger
209

◆ name() [1/2]

python.trfExe.transformExecutor.name ( self)
inherited

Definition at line 211 of file trfExe.py.

211 def name(self):
212 return self._name
213

◆ name() [2/2]

python.trfExe.transformExecutor.name ( self,
value )
inherited

Definition at line 215 of file trfExe.py.

215 def name(self, value):
216 self._name = value
217

◆ outData() [1/2]

python.trfExe.transformExecutor.outData ( self)
inherited

Definition at line 255 of file trfExe.py.

255 def outData(self):
256
257 if '_outData' in dir(self):
258 return self._outData
259 return None
260

◆ outData() [2/2]

python.trfExe.transformExecutor.outData ( self,
value )
inherited

Definition at line 262 of file trfExe.py.

262 def outData(self, value):
263 self._outData = set(value)
264

◆ outDataUpdate()

python.trfExe.transformExecutor.outDataUpdate ( self,
value )
inherited

Definition at line 265 of file trfExe.py.

265 def outDataUpdate(self, value):
266
267 if '_outData' in dir(self):
268 self._outData.update(value)
269 else:
270
271 self.outData = value
272

◆ output()

python.trfExe.transformExecutor.output ( self)
inherited
Note
This returns the actual output data with which this executor ran (c.f. outData which returns all the possible data types this executor could run with)

Definition at line 285 of file trfExe.py.

285 def output(self):
286
287 if '_output' in dir(self):
288 return self._output
289 return None
290

◆ postExeCpuTime()

python.trfExe.transformExecutor.postExeCpuTime ( self)
inherited

Definition at line 402 of file trfExe.py.

402 def postExeCpuTime(self):
403 if self._exeStop and self._valStart:
404 return calcCpuTime(self._exeStop, self._valStart)
405 else:
406 return None
407

◆ postExecute()

python.trfExe.scriptExecutor.postExecute ( self)
inherited

Reimplemented from python.trfExe.transformExecutor.

Reimplemented in python.trfExe.athenaExecutor, and python.trfExe.bsMergeExecutor.

Definition at line 788 of file trfExe.py.

788 def postExecute(self):
789 if hasattr(self._exeLogFile, 'close'):
790 self._exeLogFile.close()
791 if self._memMonitor:
792 try:
793 memFile = open(self._memSummaryFile)
794 self._memStats = json.load(memFile)
795 except Exception as e:
796 msg.warning('Failed to load JSON memory summmary file {0}: {1}'.format(self._memSummaryFile, e))
797 self._memMonitor = False
798 self._memStats = {}
799
800

◆ postExeWallTime()

python.trfExe.transformExecutor.postExeWallTime ( self)
inherited

Definition at line 409 of file trfExe.py.

409 def postExeWallTime(self):
410 if self._exeStop and self._valStart:
411 return calcWallTime(self._exeStop, self._valStart)
412 else:
413 return None
414

◆ preExeCpuTime()

python.trfExe.transformExecutor.preExeCpuTime ( self)
inherited

Definition at line 352 of file trfExe.py.

352 def preExeCpuTime(self):
353 if self._preExeStart and self._exeStart:
354 return calcCpuTime(self._preExeStart, self._exeStart)
355 else:
356 return None
357

◆ preExecute()

python.trfExe.archiveExecutor.preExecute ( self,
input = set(),
output = set() )

Reimplemented from python.trfExe.scriptExecutor.

Definition at line 2275 of file trfExe.py.

2275 def preExecute(self, input = set(), output = set()):
2276 self.setPreExeStart()
2277 self._memMonitor = False
2278
2279 #archiving
2280 if self._exe == 'zip':
2281 if 'outputArchFile' not in self.conf.argdict:
2282 raise trfExceptions.TransformExecutionException(trfExit.nameToCode('TRF_ARG_MISSING'), 'Missing output file name')
2283
2284 self._cmd = ['python']
2285 try:
2286 with open('zip_wrapper.py', 'w') as zip_wrapper:
2287 print("import zipfile, os, shutil", file=zip_wrapper)
2288 if os.path.exists(self.conf.argdict['outputArchFile'].value[0]):
2289 #appending input file(s) to existing archive
2290 print("zf = zipfile.ZipFile('{}', mode='a', allowZip64=True)".format(self.conf.argdict['outputArchFile'].value[0]), file=zip_wrapper)
2291 else:
2292 #creating new archive
2293 print("zf = zipfile.ZipFile('{}', mode='w', allowZip64=True)".format(self.conf.argdict['outputArchFile'].value[0]), file=zip_wrapper)
2294 print("for f in {}:".format(self.conf.argdict['inputDataFile'].value), file=zip_wrapper)
2295 #This module gives false positives (as of python 3.7.0). Will also check the name for ".zip"
2296 #print >> zip_wrapper, " if zipfile.is_zipfile(f):"
2297 print(" if zipfile.is_zipfile(f) and '.zip' in f:", file=zip_wrapper)
2298 print(" archive = zipfile.ZipFile(f, mode='r')", file=zip_wrapper)
2299 print(" print 'Extracting input zip file {0} to temporary directory {1}'.format(f,'tmp')", file=zip_wrapper)
2300 print(" archive.extractall('tmp')", file=zip_wrapper)
2301 print(" archive.close()", file=zip_wrapper)
2302 # remove stuff as soon as it is saved to output in order to save disk space at worker node
2303 print(" if os.access(f, os.F_OK):", file=zip_wrapper)
2304 print(" print 'Removing input zip file {}'.format(f)", file=zip_wrapper)
2305 print(" os.unlink(f)", file=zip_wrapper)
2306 print(" if os.path.isdir('tmp'):", file=zip_wrapper)
2307 print(" for root, dirs, files in os.walk('tmp'):", file=zip_wrapper)
2308 print(" for name in files:", file=zip_wrapper)
2309 print(" print 'Zipping {}'.format(name)", file=zip_wrapper)
2310 print(" zf.write(os.path.join(root, name), name, compress_type=zipfile.ZIP_STORED)", file=zip_wrapper)
2311 print(" shutil.rmtree('tmp')", file=zip_wrapper)
2312 print(" else:", file=zip_wrapper)
2313 print(" print 'Zipping {}'.format(os.path.basename(f))", file=zip_wrapper)
2314 print(" zf.write(f, arcname=os.path.basename(f), compress_type=zipfile.ZIP_STORED)", file=zip_wrapper)
2315 print(" if os.access(f, os.F_OK):", file=zip_wrapper)
2316 print(" print 'Removing input file {}'.format(f)", file=zip_wrapper)
2317 print(" os.unlink(f)", file=zip_wrapper)
2318 print("zf.close()", file=zip_wrapper)
2319 os.chmod('zip_wrapper.py', 0o755)
2320 except OSError as e:
2321 errMsg = 'error writing zip wrapper {fileName}: {error}'.format(fileName = 'zip_wrapper.py',
2322 error = e
2323 )
2324 msg.error(errMsg)
2325 raise trfExceptions.TransformExecutionException(trfExit.nameToCode('TRF_EXEC_SETUP_WRAPPER'),
2326 errMsg
2327 )
2328 self._cmd.append('zip_wrapper.py')
2329
2330 #unarchiving
2331 elif self._exe == 'unarchive':
2332 import zipfile
2333 for infile in self.conf.argdict['inputArchFile'].value:
2334 if not zipfile.is_zipfile(infile):
2335 raise trfExceptions.TransformExecutionException(trfExit.nameToCode('TRF_INPUT_FILE_ERROR'),
2336 'An input file is not a zip archive - aborting unpacking')
2337 self._cmd = ['python']
2338 try:
2339 with open('unarchive_wrapper.py', 'w') as unarchive_wrapper:
2340 print("import zipfile", file=unarchive_wrapper)
2341 print("for f in {}:".format(self.conf.argdict['inputArchFile'].value), file=unarchive_wrapper)
2342 print(" archive = zipfile.ZipFile(f, mode='r')", file=unarchive_wrapper)
2343 print(" path = '{}'".format(self.conf.argdict['path']), file=unarchive_wrapper)
2344 print(" print 'Extracting archive {0} to {1}'.format(f,path)", file=unarchive_wrapper)
2345 print(" archive.extractall(path)", file=unarchive_wrapper)
2346 print(" archive.close()", file=unarchive_wrapper)
2347 os.chmod('unarchive_wrapper.py', 0o755)
2348 except OSError as e:
2349 errMsg = 'error writing unarchive wrapper {fileName}: {error}'.format(fileName = 'unarchive_wrapper.py',
2350 error = e
2351 )
2352 msg.error(errMsg)
2353 raise trfExceptions.TransformExecutionException(trfExit.nameToCode('TRF_EXEC_SETUP_WRAPPER'),
2354 errMsg
2355 )
2356 self._cmd.append('unarchive_wrapper.py')
2357 super(archiveExecutor, self).preExecute(input=input, output=output)
2358
print
void print(char *figname, TCanvas *c1)
Definition TRTCalib_StrawStatusPlots.cxx:26

◆ preExeStartTimes()

python.trfExe.transformExecutor.preExeStartTimes ( self)
inherited

Definition at line 332 of file trfExe.py.

332 def preExeStartTimes(self):
333 return self._preExeStart
334

◆ preExeWallTime()

python.trfExe.transformExecutor.preExeWallTime ( self)
inherited

Definition at line 359 of file trfExe.py.

359 def preExeWallTime(self):
360 if self._preExeStart and self._exeStart:
361 return calcWallTime(self._preExeStart, self._exeStart)
362 else:
363 return None
364

◆ rc()

python.trfExe.transformExecutor.rc ( self)
inherited

Definition at line 300 of file trfExe.py.

300 def rc(self):
301 return self._rc
302
rc
static Double_t rc
Definition LArPhysWaveHECTool.cxx:37

◆ reSimEvent()

python.trfExe.transformExecutor.reSimEvent ( self)
inherited

Definition at line 448 of file trfExe.py.

448 def reSimEvent(self):
449 return self._resimevents
450

◆ setPreExeStart()

python.trfExe.transformExecutor.setPreExeStart ( self)
inherited

Definition at line 461 of file trfExe.py.

461 def setPreExeStart(self):
462 if self._preExeStart is None:
463 self._preExeStart = os.times()
464 msg.debug('preExeStart time is {0}'.format(self._preExeStart))
465

◆ setValStart()

python.trfExe.transformExecutor.setValStart ( self)
inherited

Definition at line 466 of file trfExe.py.

466 def setValStart(self):
467 if self._valStart is None:
468 self._valStart = os.times()
469 msg.debug('valStart time is {0}'.format(self._valStart))
470

◆ substep()

python.trfExe.transformExecutor.substep ( self)
inherited

Reimplemented in python.trfExe.athenaExecutor.

Definition at line 219 of file trfExe.py.

219 def substep(self):
220 if '_substep' in dir(self):
221 return self._substep
222 return None
223

◆ sysTime()

python.trfExe.transformExecutor.sysTime ( self)
inherited

Definition at line 380 of file trfExe.py.

380 def sysTime(self):
381 if self._exeStart and self._exeStop:
382 return self._exeStop[3] - self._exeStart[3]
383 else:
384 return None
385

◆ trf() [1/2]

python.trfExe.transformExecutor.trf ( self)
inherited

Definition at line 225 of file trfExe.py.

225 def trf(self):
226 if '_trf' in dir(self):
227 return self._trf
228 return None
229

◆ trf() [2/2]

python.trfExe.transformExecutor.trf ( self,
value )
inherited

Definition at line 231 of file trfExe.py.

231 def trf(self, value):
232 self._trf = value
233

◆ usrTime()

python.trfExe.transformExecutor.usrTime ( self)
inherited

Definition at line 373 of file trfExe.py.

373 def usrTime(self):
374 if self._exeStart and self._exeStop:
375 return self._exeStop[2] - self._exeStart[2]
376 else:
377 return None
378

◆ validate()

python.trfExe.scriptExecutor.validate ( self)
inherited

Reimplemented from python.trfExe.transformExecutor.

Reimplemented in python.trfExe.athenaExecutor, python.trfExe.DQMergeExecutor, python.trfExe.DQMPostProcessExecutor, and python.trfExe.optionalAthenaExecutor.

Definition at line 801 of file trfExe.py.

801 def validate(self):
802 if self._valStart is None:
803 self._valStart = os.times()
804 msg.debug('valStart time is {0}'.format(self._valStart))
805 self._hasValidated = True
806
807
808 if self._rc == 0:
809 msg.info('Executor {0} validated successfully (return code {1})'.format(self._name, self._rc))
810 self._isValidated = True
811 self._errMsg = ''
812 else:
813 # Want to learn as much as possible from the non-zero code
814 # this is a bit hard in general, although one can do signals.
815 # Probably need to be more specific per exe, i.e., athena non-zero codes
816 self._isValidated = False
817 if self._rc < 0:
818 # Map return codes to what the shell gives (128 + SIGNUM)
819 self._rc = 128 - self._rc
820 if trfExit.codeToSignalname(self._rc) != "":
821 self._errMsg = '{0} got a {1} signal (exit code {2})'.format(self._name, trfExit.codeToSignalname(self._rc), self._rc)
822 else:
823 self._errMsg = 'Non-zero return code from %s (%d)' % (self._name, self._rc)
824 raise trfExceptions.TransformValidationException(trfExit.nameToCode('TRF_EXEC_FAIL'), self._errMsg)
825
826
828 if 'checkEventCount' in self.conf.argdict and self.conf.argdict['checkEventCount'].returnMyValue(exe=self) is False:
829 msg.info('Event counting for substep {0} is skipped'.format(self.name))
830 else:
831 if 'mpi' in self.conf.argdict and not mpi.mpiShouldValidate():
832 msg.info('MPI mode -- skipping output event count check')
833 else:
834 checkcount=trfValidation.eventMatch(self)
835 checkcount.decide()
836 self._eventCount = checkcount.eventCount
837 msg.info('Event counting for substep {0} passed'.format(self.name))
838
839 self._valStop = os.times()
840 msg.debug('valStop time is {0}'.format(self._valStop))
841
842
843

◆ validation() [1/2]

python.trfExe.transformExecutor.validation ( self)
inherited

Definition at line 308 of file trfExe.py.

308 def validation(self):
309 return self._validation
310

◆ validation() [2/2]

python.trfExe.transformExecutor.validation ( self,
value )
inherited

Definition at line 312 of file trfExe.py.

312 def validation(self, value):
313 self._validation = value
314

◆ validationCpuTime()

python.trfExe.transformExecutor.validationCpuTime ( self)
inherited

Definition at line 416 of file trfExe.py.

416 def validationCpuTime(self):
417 if self._valStart and self._valStop:
418 return calcCpuTime(self._valStart, self._valStop)
419 else:
420 return None
421

◆ validationWallTime()

python.trfExe.transformExecutor.validationWallTime ( self)
inherited

Definition at line 423 of file trfExe.py.

423 def validationWallTime(self):
424 if self._valStart and self._valStop:
425 return calcWallTime(self._valStart, self._valStop)
426 else:
427 return None
428

◆ valStartTimes()

python.trfExe.transformExecutor.valStartTimes ( self)
inherited

Definition at line 344 of file trfExe.py.

344 def valStartTimes(self):
345 return self._valStart
346

◆ valStopTimes()

python.trfExe.transformExecutor.valStopTimes ( self)
inherited

Definition at line 348 of file trfExe.py.

348 def valStopTimes(self):
349 return self._valStop
350

◆ wallTime()

python.trfExe.transformExecutor.wallTime ( self)
inherited

Definition at line 387 of file trfExe.py.

387 def wallTime(self):
388 if self._exeStart and self._exeStop:
389 return calcWallTime(self._exeStart, self._exeStop)
390 else:
391 return None
392

◆ wallTimeTotal()

python.trfExe.transformExecutor.wallTimeTotal ( self)
inherited

Definition at line 437 of file trfExe.py.

437 def wallTimeTotal(self):
438 if self._preExeStart and self._valStop:
439 return calcWallTime(self._preExeStart, self._valStop)
440 else:
441 return None
442

Member Data Documentation

◆ _alreadyInContainer

python.trfExe.transformExecutor._alreadyInContainer = None
protectedinherited

Definition at line 198 of file trfExe.py.

◆ _athenaConcurrentEvents

int python.trfExe.transformExecutor._athenaConcurrentEvents = 0
protectedinherited

Definition at line 195 of file trfExe.py.

◆ _athenaMP

int python.trfExe.transformExecutor._athenaMP = 0
protectedinherited

Definition at line 193 of file trfExe.py.

◆ _athenaMT

int python.trfExe.transformExecutor._athenaMT = 0
protectedinherited

Definition at line 194 of file trfExe.py.

◆ _cmd

python.trfExe.scriptExecutor._cmd = None
protectedinherited
Note
If an inherited class has set self._cmd leave it alone

Definition at line 636 of file trfExe.py.

◆ _containerSetup

python.trfExe.transformExecutor._containerSetup = None
protectedinherited

Definition at line 199 of file trfExe.py.

◆ _dbMonitor

python.trfExe.transformExecutor._dbMonitor = None
protectedinherited

Definition at line 196 of file trfExe.py.

◆ _echologger

python.trfExe.scriptExecutor._echologger = logging.getLogger(self._name)
protectedinherited
Note
Query the environment for echo configuration Let the manual envars always win over auto-detected settings

Definition at line 692 of file trfExe.py.

◆ _echoOutput

bool python.trfExe.scriptExecutor._echoOutput = False
protectedinherited
Note
Query the environment for echo configuration Let the manual envars always win over auto-detected settings

Definition at line 633 of file trfExe.py.

◆ _echostream

python.trfExe.scriptExecutor._echostream = logging.StreamHandler(_encoding_stream(sys.stdout))
protectedinherited

Definition at line 702 of file trfExe.py.

◆ _errMsg

str python.trfExe.transformExecutor._errMsg = None
protectedinherited

Definition at line 172 of file trfExe.py.

◆ _eventCount

python.trfExe.transformExecutor._eventCount = None
protectedinherited

Definition at line 192 of file trfExe.py.

◆ _exe

python.trfExe.scriptExecutor._exe = exe
protectedinherited

Definition at line 623 of file trfExe.py.

◆ _exeArgs

python.trfExe.scriptExecutor._exeArgs = exeArgs
protectedinherited

Definition at line 626 of file trfExe.py.

◆ _exeLogFile

python.trfExe.scriptExecutor._exeLogFile = logging.FileHandler(self._logFileName, mode='w', **encargs)
protectedinherited

Definition at line 697 of file trfExe.py.

◆ _exeStart

python.trfExe.transformExecutor._exeStart = None
protectedinherited

Definition at line 187 of file trfExe.py.

◆ _exeStop

python.trfExe.transformExecutor._exeStop
protectedinherited

Definition at line 367 of file trfExe.py.

◆ _extraMetadata

dict python.trfExe.transformExecutor._extraMetadata = {}
protectedinherited

Definition at line 181 of file trfExe.py.

◆ _hasExecuted

bool python.trfExe.transformExecutor._hasExecuted = False
protectedinherited

Definition at line 170 of file trfExe.py.

◆ _hasValidated

bool python.trfExe.transformExecutor._hasValidated = False
protectedinherited

Definition at line 175 of file trfExe.py.

◆ _inData

python.trfExe.transformExecutor._inData = set(inData)
protectedinherited

Definition at line 144 of file trfExe.py.

◆ _input

python.trfExe.scriptExecutor._input = input
protectedinherited

Definition at line 662 of file trfExe.py.

◆ _isValidated

bool python.trfExe.transformExecutor._isValidated = False
protectedinherited

Definition at line 176 of file trfExe.py.

◆ _logFileName

python.trfExe.scriptExecutor._logFileName = "log.{0}".format(self._name)
protectedinherited
Note
If an inherited class has set self._cmd leave it alone
Query the environment for echo configuration Let the manual envars always win over auto-detected settings

Definition at line 671 of file trfExe.py.

◆ _memFullFile

python.trfExe.transformExecutor._memFullFile = None
protectedinherited

Definition at line 191 of file trfExe.py.

◆ _memLeakResult

dict python.trfExe.transformExecutor._memLeakResult = {}
protectedinherited

Definition at line 190 of file trfExe.py.

◆ _memMonitor

bool python.trfExe.scriptExecutor._memMonitor = memMonitor
protectedinherited

Definition at line 638 of file trfExe.py.

◆ _memStats

dict python.trfExe.transformExecutor._memStats = {}
protectedinherited

Definition at line 189 of file trfExe.py.

◆ _memSummaryFile

python.trfExe.scriptExecutor._memSummaryFile = 'prmon.summary.' + self._name + '.json'
protectedinherited

Definition at line 749 of file trfExe.py.

◆ _myMerger

list python.trfExe.transformExecutor._myMerger = []
protectedinherited

Definition at line 202 of file trfExe.py.

◆ _name

python.trfExe.transformExecutor._name = forceToAlphaNum(name)
protectedinherited

Definition at line 141 of file trfExe.py.

◆ _outData

python.trfExe.transformExecutor._outData = set(outData)
protectedinherited

Definition at line 145 of file trfExe.py.

◆ _output

python.trfExe.scriptExecutor._output = output
protectedinherited

Definition at line 663 of file trfExe.py.

◆ _preExeStart

python.trfExe.transformExecutor._preExeStart = None
protectedinherited
Note
Place holders for resource consumption. CPU and walltime are available for all executors but currently only athena is instrumented to fill in memory stats (and then only if PerfMonSD is enabled).

Definition at line 186 of file trfExe.py.

◆ _rc

python.trfExe.transformExecutor._rc = -1
protectedinherited

Definition at line 171 of file trfExe.py.

◆ _resimevents

python.trfExe.transformExecutor._resimevents = None
protectedinherited

Definition at line 197 of file trfExe.py.

◆ _trf

python.trfExe.transformExecutor._trf = value
protectedinherited

Definition at line 232 of file trfExe.py.

◆ _validation

python.trfExe.transformExecutor._validation = value
protectedinherited

Definition at line 313 of file trfExe.py.

◆ _valStart

python.trfExe.transformExecutor._valStart = None
protectedinherited

Definition at line 188 of file trfExe.py.

◆ _valStop

python.trfExe.transformExecutor._valStop
protectedinherited

Definition at line 417 of file trfExe.py.

◆ _workdir

python.trfExe.scriptExecutor._workdir
protectedinherited

Definition at line 744 of file trfExe.py.

◆ conf

python.trfExe.transformExecutor.conf = conf
inherited

Executor configuration:

Note
that if conf and trf are None then we'll probably set the conf up later (this is allowed and expected to be done once the master transform has figured out what it's doing for this job)

Definition at line 163 of file trfExe.py.

◆ exeArgs

python.trfExe.scriptExecutor.exeArgs
inherited

Definition at line 712 of file trfExe.py.

◆ inData

python.trfExe.transformExecutor.inData = value
inherited
Note
Protect against _inData not yet being defined
Use normal setter

Definition at line 251 of file trfExe.py.

◆ outData

python.trfExe.transformExecutor.outData = value
inherited
Note
Protect against _outData not yet being defined
Use normal setter

Definition at line 271 of file trfExe.py.

The documentation for this class was generated from the following file:
Generated on for ATLAS Offline Software by doxygen 1.14.0