python.TaskManager Namespace Reference

Classes
class	DbParam
class	JobAnalyzer
class	TaskAnalyzer
class	TaskManager
class	TaskManagerCheckError
class	TaskManagerDatabaseError

Functions
	dictFactory (cursor, row)
	getKey (d, v)
	getStatusClass (status)
	appendUnique (s, v)
	getFullTaskNames (taskman, dsname, taskname, requireSingleTask=False, confirmWithUser=False, addWildCards=True)
	getJobConfig (jobDir, dsName, taskName, jobName=' *')

Variables
str	__author__ = 'Juerg Beringer'
str	__version__ = 'TaskManager.py atlas/athena'

Detailed Description

TaskManager is a tool for keeping track of JobRunner jobs using a
database. TaskManager uses the notion of a task as the primary unit for
bookkeeping. Each task includes one or more jobs run using the same
JobRunner template and typically resulting from a single invocation of
JobRunner through one of the makeDPD.py or runDPD.py
scripts. TaskManager keeps track of the status of tasks and associated
jobs, runs postprocessing scripts, and accumulates results for later
analysis accross tasks.

Written by Juerg Beringer (LBNL) in 2009.

Function Documentation

appendUnique()

python.TaskManager.appendUnique	(		s,
			v )

Definition at line 64 of file TaskManager.py.

def appendUnique(s,v):
    if not s:
        s = ''   # make sure it is not None
    if v not in s.split():
        s = ' '.join([s,v])
    return s
 
 

dictFactory()

python.TaskManager.dictFactory	(		cursor,
			row )

Convert a tuple from a database query into a dictonary.

Definition at line 33 of file TaskManager.py.

def dictFactory(cursor, row):
    """Convert a tuple from a database query into a dictonary."""
    d = {}
    for idx, col in enumerate(cursor.description):
        d[col[0]] = row[idx]
    return d
 
 

getFullTaskNames()

python.TaskManager.getFullTaskNames	(		taskman,
			dsname,
			taskname,
			requireSingleTask = False,
			confirmWithUser = False,
			addWildCards = True )

Retrieve the full dataset and task names given a pair of (dsname,task) that may
   contain wildcards or be just a parital name such as the run number. Depending
   on the requireSingleTask and confirmWithUser settings a TaskManagerCheckError
   is raised if there are multiple tasks or if the user doesn't confirm.

Definition at line 72 of file TaskManager.py.

def getFullTaskNames(taskman,dsname,taskname,requireSingleTask=False,confirmWithUser=False,addWildCards=True):
    """Retrieve the full dataset and task names given a pair of (dsname,task) that may
       contain wildcards or be just a parital name such as the run number. Depending
       on the requireSingleTask and confirmWithUser settings a TaskManagerCheckError
       is raised if there are multiple tasks or if the user doesn't confirm."""
    taskList = taskman.getTaskNames(dsname,taskname,addWildCards)
    if len(taskList)==0:
        raise  TaskManagerCheckError ('ERROR: No tasks found for dataset = %s, task = %s' % (dsname,taskname))
    if requireSingleTask and len(taskList)!=1:
        m = "ERROR: Multiple data set names found for dataset = %s, task = %s\n       Please use full dataset or task name from list below, using option -n if necessary:\n\n" % (dsname,taskname)
        m += "    %-50s  %s\n" % ('DATASET NAME','TASK NAME')
        m += "    %s\n" % (75*'-')
        for t in taskList:
            m += "    %-50s  %s\n" % (t[0],t[1])
        m += '\n'
        raise TaskManagerCheckError (m)
    if confirmWithUser:
        print ('Please confirm that you want to execute this command for the following tasks:\n')
        print ("    %-50s  %s" % ('DATASET NAME','TASK NAME'))
        print ("    %s" % (75*'-'))
        for t in taskList:
            print ("    %-50s  %s" % (t[0],t[1]))
        a = input('\nARE YOU SURE [n] ? ')
        if a!='y':
            raise TaskManagerCheckError ('ERROR: Aborted by user')
        print()
    return taskList
 
 

getJobConfig()

python.TaskManager.getJobConfig	(		jobDir,
			dsName,
			taskName,
			jobName = '*' )

Read config dict from job files.

Definition at line 101 of file TaskManager.py.

def getJobConfig(jobDir,dsName,taskName,jobName='*'):
    """Read config dict from job files."""
    config = {}
    configFile = glob.glob('%s/%s/%s/%s/%s' % (jobDir,dsName,taskName,jobName,'*.config.py.final.py'))
    if not configFile:
        configFile = glob.glob('%s/%s/%s/%s/%s' % (jobDir,dsName,taskName,jobName,'*.config.py'))
    if configFile:
        exec(open(configFile[0]).read(),config)    # Eval config file and put defs into config dict
    return config['jobConfig']
 
 

getKey()

python.TaskManager.getKey	(		d,
			v )

Get the key for which dictonary d has an entry with value v.
Returns 'Undefined' if there's no such key, if several values are found.

Definition at line 41 of file TaskManager.py.

def getKey(d, v):
    """Get the key for which dictonary d has an entry with value v.
    Returns 'Undefined' if there's no such key, if several values are found."""
    l = [k for k in d.keys() if d[k]==v]
    if len(l)==1:
        return l[0]
    else:
        return 'Undefined'
 
 

getStatusClass()

python.TaskManager.getStatusClass

(

status

)

Returns 'ok', 'warn' or 'bad' depending on the value of status.

Definition at line 51 of file TaskManager.py.

def getStatusClass(status):
    """Returns 'ok', 'warn' or 'bad' depending on the value of status."""
    statusMap = {
        7: 'bad',
        10: 'ok',
        11: 'bad'
        }
    if status in statusMap:
        return statusMap[status]
    else:
        return 'warn'
 
 

Variable Documentation

__author__

str python.TaskManager.__author__ = 'Juerg Beringer'

private

Definition at line 17 of file TaskManager.py.

__version__

str python.TaskManager.__version__ = 'TaskManager.py atlas/athena'

private

Definition at line 18 of file TaskManager.py.