python.trfArgClasses.argFile Class Reference

File argument class. More...

Inheritance diagram for python.trfArgClasses.argFile:

Collaboration diagram for python.trfArgClasses.argFile:

Public Member Functions
def	__init__ (self, value=list(), type=None, subtype=None, io='output', splitter=',', runarg=True, guid=None, multipleOK=None, name=None, executor=list(), mergeTargetSize=-1, auxiliaryFile=False)
	Initialise an argFile. More...
def	value (self)
	Argument value getter. More...
def	value (self, value)
	Argument value setter. More...
def	multipleOK (self)
	multipleOK getter More...
def	multipleOK (self, value)
	multipleOK value setter More...
def	mergeTargetSize (self)
	mergeTargeSize value getter More...
def	mergeTargetSize (self, value)
	mergeTargeSize value setter More...
def	prodsysDescription (self)
def	executor (self)
	Executor status getter. More...
def	valueSetter (self, value)
	Set the argFile value, but allow parameters here. More...
def	io (self)
def	io (self, value)
def	dataset (self)
def	dataset (self, value)
def	orignalName (self)
def	originalName (self, value)
def	type (self)
def	type (self, value)
def	subtype (self)
def	subtype (self, value)
def	name (self)
	Name getter. More...
def	name (self, value)
	Name setter. More...
def	auxiliaryFile (self)
def	metadata (self)
	Returns the whole kit and kaboodle... More...
def	nentries (self)
	Return total number of events in all constituent files. More...
def	getnentries (self, fast=False)
	Explicit getter, offering fast switch. More...
def	getMetadata (self, files=None, metadataKeys=None, maskMetadataKeys=None, populate=True, flush=False)
	Return specific keys for specific files. More...
def	getSingleMetadata (self, fname, metadataKey, populate=True, flush=False)
	Convenience function to extract a single metadata key for a single file. More...
def	isCached (self, files=None, metadataKeys=None)
	Test if certain metadata elements are already cached. More...
def	__str__ (self)
	String representation of a file argument. More...
def	append (self, addme)
	Append a value to the list. More...
def	__repr__ (self)
	Repr conversion. More...
def	isRunarg (self)
	Return runarg status. More...
def	__eq__ (self, other)
	Comparison is based on value attribute. More...
def	__nq__ (self, other)
def	__lt__ (self, other)
def	__gt__ (self, other)

Public Attributes
	io
	dataset
	value

Private Member Functions
def	_resetMetadata (self, files=[])
	Resets all metadata files in this instance. More...
def	_readMetadata (self, files, metadataKeys)
	Check metadata is in the cache or generate it if it's missing. More...
def	_setMetadata (self, files=None, metadataKeys={})
	Set metadata values into the cache. More...
def	_getDatasetFromFilename (self, reset=False)
	Look for dataset name in dataset::filename Tier0 convention. More...
def	_getSize (self, files)
	Determines the size of files. More...
def	_getIntegrity (self, files)
	File integrity checker. More...
def	_generateGUID (self, files)
	Generate a GUID on demand - no intrinsic for this file type More...
def	_exists (self, files)
	Try to determine if a file actually exists... More...
def	_mergeArgs (self, argdict, copyArgs=None)
	Utility to strip arguments which should not be passed to the selfMerge methods of our child classes. More...

Private Attributes
	_dataset
	_urlType
	_type
	_subtype
	_guid
	_mergeTargetSize
	_auxiliaryFile
	_originalName
	_exe
	_metadataKeys
	_fileMetadata
	_io
	Input file globbing and expansion. More...
	_multipleOK
	_value
	_name
	_splitter
	_supressEmptyStrings
	_runarg

Detailed Description

File argument class.

Inherits from argList

Definition at line 522 of file trfArgClasses.py.

Constructor & Destructor Documentation

__init__()

def python.trfArgClasses.argFile.__init__	(		self,
			value = list(),
			type = None,
			subtype = None,
			io = 'output',
			splitter = ',',
			runarg = True,
			guid = None,
			multipleOK = None,
			name = None,
			executor = list(),
			mergeTargetSize = -1,
			auxiliaryFile = False
	)

Initialise an argFile.

Parameters

io	input, output or temporary file, default output.
splitter	changes character a string list is split on, default is a comma (see argList).
type	Datatype in this instance - this should be the major datatype (ESD, AOD, etc).
subtype	The data subtype, which should match the DATATYPE portion of the corresponding argument name, e.g., outputDESD_SGLMUFile -> DESD_SGLMU
guid	This is a non-standard option and allows the GUID for files without an intrinsic GUID to be set explicitly at initialisation. The parameter should be a dictionary, keyed by filename, which contains the GUID string, e.g., {'file1' : '930de3de-de8d-4819-9129-beef3bb4fadb', 'file2' : ... }
multipleOK	Explicit declaration of whether multiple arguments are allowed; default is True for input, False for output and temporary
name	The corresponding key for this argument in the argdict of the transform (e.g., inputESDFile)
executor	List of execution substeps where this file type should be added explicitly (e.g., minbias HITS for digitisation)
mergeTargetSize	Target merge size if this instance supports a selfMerge method. Value is in bytes, with the special values: -1 Always merge to a single file, 0 never merge these files
auxiliaryFile	Is set to True then all validation for this file is disabled - used for non-primary input files, e.g., pileup inputs

Note

When used in argument parser, set nargs='+' to get auto-concatenation of multiple arguments (should be used when multipleOK is True)

Definition at line 544 of file trfArgClasses.py.

    def __init__(self, value=list(), type=None, subtype=None, io = 'output', splitter=',', runarg=True, guid=None, 
                 multipleOK = None, name=None, executor=list(), mergeTargetSize=-1, auxiliaryFile=False):
        # Set these values before invoking super().__init__ to make sure they can be
        # accessed in our setter 
        self._dataset = None
        self._urlType = None
        self._type = type
        self._subtype = subtype
        self._guid = guid
        self._mergeTargetSize = mergeTargetSize
        self._auxiliaryFile = auxiliaryFile
        self._originalName = None
 
        # User setter to get valid value check
        self.io = io
 
        self._exe = executor
 
        
 
        self._metadataKeys = {'file_size': self._getSize,
                              'integrity': self._getIntegrity,
                              'file_guid': self._generateGUID,
                              '_exists': self._exists,
                              }
        self._fileMetadata = {}
        if multipleOK is None:
            if self._io == 'input':
                self._multipleOK = True
            else:
                self._multipleOK = False
        else:
            self._multipleOK = multipleOK
            
        
        super(argFile, self).__init__(value=value, splitter=splitter, runarg=runarg, name=name)
    
 

Member Function Documentation

__eq__()

def python.trfArgClasses.argument.__eq__ (   self,   other  )

inherited

Comparison is based on value attribute.

Definition at line 161 of file trfArgClasses.py.

    def __eq__(self,other):
        return self.value == other.value
 

__gt__()

def python.trfArgClasses.argument.__gt__ (   self,   other  )

inherited

Definition at line 170 of file trfArgClasses.py.

    def __gt__(self, other):
        return self.value > other.value
 

__lt__()

def python.trfArgClasses.argument.__lt__ (   self,   other  )

inherited

Definition at line 167 of file trfArgClasses.py.

    def __lt__(self, other):
        return self.value < other.value
 

__nq__()

def python.trfArgClasses.argument.__nq__ (   self,   other  )

inherited

Definition at line 164 of file trfArgClasses.py.

    def __nq__(self, other):
        return self.value != other.value
 

__repr__()

def python.trfArgClasses.argList.__repr__ (   self )

inherited

Repr conversion.

Return a python parsable string

Reimplemented from python.trfArgClasses.argument.

Definition at line 409 of file trfArgClasses.py.

    def __repr__(self):
        return '[' + ','.join([ repr(s) for s in self._value ]) + ']'
 
 

__str__()

def python.trfArgClasses.argFile.__str__

(

self

)

String representation of a file argument.

Reimplemented from python.trfArgClasses.argList.

Definition at line 1201 of file trfArgClasses.py.

    def __str__(self):
        return "{0}={1} (Type {2}, Dataset {3}, IO {4})".format(self.name, self.value, self.type, self.dataset, self.io)
    
    

_exists()

def python.trfArgClasses.argFile._exists (   self,   files  )

private

Try to determine if a file actually exists...

For a posix file, just call stat; for anything else call TFile.Open A small optimisation is to retieve the file_size metadatum at the same time.

Parameters

Definition at line 1176 of file trfArgClasses.py.

    def _exists(self, files):
        msg.debug('Testing existance for {0}'.format(files))
        for fname in files:
            if self._urlType == 'posix':
                try:
                    size = os.stat(fname).st_size
                    self._fileMetadata[fname]['file_size'] = size
                    self._fileMetadata[fname]['_exists'] = True
                    msg.debug('POSIX file {0} exists'.format(fname))
                except OSError as e:
                    msg.error('Got exception {0!s} raised while stating file {1}  - probably it does not exist'.format(e, fname))
                    self._fileMetadata[fname]['_exists'] = False
            else:
                # OK, let's see if ROOT can do it...
                msg.debug('Calling ROOT TFile.GetSize({0})'.format(fname))
                size = ROOTGetSize(fname)
                if size is None:
                    self._fileMetadata[fname]['_exists'] = False
                    msg.error('Non-POSIX file {0} could not be opened - probably it does not exist'.format(fname))
                else:
                    msg.debug('Non-POSIX file {0} exists'.format(fname))
                    self._fileMetadata[fname]['file_size'] = size
                    self._fileMetadata[fname]['_exists'] = True
 

_generateGUID()

def python.trfArgClasses.argFile._generateGUID (   self,   files  )

private

Generate a GUID on demand - no intrinsic for this file type

Use uuid.uuid4() call to generate a GUID

Note

This generation method will be superceeded in any file type which actually has an intrinsic GUID (e.g. BS or POOL files)

Definition at line 1165 of file trfArgClasses.py.

    def _generateGUID(self, files):
        for fname in files:
            msg.debug('Generating a GUID for file {0}'.format(fname))
            self._fileMetadata[fname]['file_guid'] = str(uuid.uuid4()).upper()
            
            

_getDatasetFromFilename()

def python.trfArgClasses.argFile._getDatasetFromFilename (   self,   reset = False  )

private

Look for dataset name in dataset::filename Tier0 convention.

At the moment all files must be in the same dataset if it's specified. (To change this dataset will need to become a per-file metadatum.)

Note

dsn::lfn notation must be used for all input values and all dsn values must be the same

Parameters

Definition at line 1089 of file trfArgClasses.py.

    def _getDatasetFromFilename(self, reset = False):
        if reset:
            self._dataset = None
        newValue = []
        for filename in self._value:
            if filename.find('#') > -1:
                (dataset, fname) = filename.split('#', 1)
                newValue.append(fname)
                msg.debug('Current dataset: {0}; New dataset {1}'.format(self._dataset, dataset))
                if self._dataset and (self._dataset != dataset):
                    raise trfExceptions.TransformArgException(trfExit.nameToCode('TRF_ARG_DATASET'), 
                                                              'Found inconsistent dataset assignment in argFile setup: %s != %s' % (self._dataset, dataset))
                self._dataset = dataset
        if len(newValue) == 0:
            return
        elif len(newValue) != len (self._value):
            raise trfExceptions.TransformArgException(trfExit.nameToCode('TRF_ARG_DATASET'), 
                                                      'Found partial dataset assignment in argFile setup from {0} (dsn#lfn notation must be uniform for all inputs)'.format(self._value))
        self._value = newValue
    

_getIntegrity()

def python.trfArgClasses.argFile._getIntegrity (   self,   files  )

private

File integrity checker.

For a 'plain' file, integrity just checks that we can read it

Parameters

Reimplemented in python.trfArgClasses.argBZ2File, python.trfArgClasses.argNTUPFile, python.trfArgClasses.argHISTFile, python.trfArgClasses.argPOOLFile, and python.trfArgClasses.argBSFile.

Definition at line 1131 of file trfArgClasses.py.

    def _getIntegrity(self, files):
        for fname in files:
            is_binary = False
            with open(fname) as f: 
                try:
                    while True:
                        chunk = len(f.read(1024*1024))
                        msg.debug('Read {0} bytes from {1}'.format(chunk, fname)) 
                        if chunk == 0:
                            break
                    self._fileMetadata[fname]['integrity'] = True
                except OSError as e:
                    msg.error('Got exception {0!s} raised while checking integrity of file {1}'.format(e, fname))
                    self._fileMetadata[fname]['integrity'] = False
                except UnicodeDecodeError:
                    msg.debug('Problem reading file as unicode, attempting with binary')
                    is_binary = True
            if is_binary:
                with open(fname,'rb') as f:
                    try:
                        while True:
                            chunk = len(f.read(1024*1024))
                            msg.debug('Read {0} bytes from {1}'.format(chunk, fname))
                            if chunk == 0:
                                break
                        self._fileMetadata[fname]['integrity'] = True
                    except OSError as e:
                        msg.error('Got exception {0!s} raised while checking integrity of file {1}'.format(e, fname))
                        self._fileMetadata[fname]['integrity'] = False
 

_getSize()

def python.trfArgClasses.argFile._getSize (   self,   files  )

private

Determines the size of files.

Currently only for statable files (posix fs). Caches the

Parameters

files

List of paths to the files for which the size is determined.

Returns

None (internal self._fileMetadata cache is updated)

Definition at line 1113 of file trfArgClasses.py.

    def _getSize(self, files):
        for fname in files:
            if self._urlType == 'posix':
                try:
                    self._fileMetadata[fname]['size'] = os.stat(fname).st_size
                except OSError as e:
                    msg.error('Got exception {0!s} raised while stating file {1}'.format(e, fname))
                    self._fileMetadata[fname]['size'] = None
            else:
                # OK, let's see if ROOT can do it...
                msg.debug('Calling ROOT TFile.GetSize({0})'.format(fname))
                self._fileMetadata[fname]['size'] = ROOTGetSize(fname)
 
 

_mergeArgs()

def python.trfArgClasses.argFile._mergeArgs (   self,   argdict,   copyArgs = None  )

private

Utility to strip arguments which should not be passed to the selfMerge methods of our child classes.

Parameters

copyArgs

If None copy all arguments by default, otherwise only copy the listed keys

Definition at line 1209 of file trfArgClasses.py.

    def _mergeArgs(self, argdict, copyArgs=None):
        if copyArgs:
            myargdict = {}
            for arg in copyArgs:
                if arg in argdict:
                    myargdict[arg] = copy.copy(argdict[arg])
            
        else:
            myargdict = copy.copy(argdict)
        # Never do event count checks for self merging
        myargdict['checkEventCount'] = argSubstepBool('False', runarg=False) 
        newopts = []
        if 'athenaopts' in myargdict:
            # Need to ensure that "nprocs" is not passed to merger
            # and prevent multiple '--threads' options when there are multiple sub-steps in 'athenopts'
            for subStep in myargdict['athenaopts'].value:
                hasNprocs = False
                hasNthreads = False
                for opt in myargdict['athenaopts'].value[subStep]:
                    if opt.startswith('--nprocs'):
                        hasNprocs = True
                        continue
                    # Keep at least one '--threads'
                    elif opt.startswith('--threads'):
                        hasNthreads = True
                        if opt in newopts:
                            continue
                    newopts.append(opt)
                # If we have hybrid MP+MT job make sure --threads is not passed to merger
                if hasNprocs and hasNthreads:
                    tmpopts = []
                    for opt in newopts:
                        if opt.startswith('--threads'):
                            continue
                        tmpopts.append(opt)
                    newopts = tmpopts
            myargdict['athenaopts'] = argSubstepList(newopts, runarg=False)
        return myargdict
 
 

_readMetadata()

def python.trfArgClasses.argFile._readMetadata (   self,   files,   metadataKeys  )

private

Check metadata is in the cache or generate it if it's missing.

Returns

: dictionary of files with metadata, for any unknown keys 'UNDEFINED' is returned

Definition at line 996 of file trfArgClasses.py.

    def _readMetadata(self, files, metadataKeys):
        msg.debug('Retrieving metadata keys {1!s} for files {0!s}'.format(files, metadataKeys))
        for fname in files:
            if fname not in self._fileMetadata:
                self._fileMetadata[fname] = {}
        for fname in files:
            # Always try for a simple existence test first before producing misleading error messages
            # from metadata populator functions
            if '_exists' not in self._fileMetadata[fname]:
                self._metadataKeys['_exists'](files)
            if self._fileMetadata[fname]['_exists'] is False:
                # N.B. A log ERROR message has printed by the existence test, so do not repeat that news here
                for key in metadataKeys:
                    if key != '_exists':
                        self._fileMetadata[fname][key] = None
            else:
                # OK, file seems to exist at least... 
                for key in metadataKeys:
                    if key not in self._metadataKeys:
                        msg.debug('Metadata key {0} is unknown for {1}'.format(key, self.__class__.__name__))
                        self._fileMetadata[fname][key] = 'UNDEFINED'
                    else:
                        if key in self._fileMetadata[fname]:
                            msg.debug('Found cached value for {0}:{1} = {2!s}'.format(fname, key, self._fileMetadata[fname][key]))
                        else:
                            msg.debug('No cached value for {0}:{1}. Calling generator function {2} ({3})'.format(fname, key, self._metadataKeys[key].__name__, self._metadataKeys[key]))
                            try:
                                # For efficiency call this routine with all files we have
                                msg.info("Metadata generator called to obtain {0} for {1}".format(key, files))
                                self._metadataKeys[key](files)
                            except trfExceptions.TransformMetadataException as e:
                                msg.error('Calling {0!s} raised an exception: {1!s}'.format(self._metadataKeys[key].__name__, e))
                            if key not in self._fileMetadata[fname]:
                                msg.warning('Call to function {0} for {1} file {2} failed to populate metadata key {3}'.format(self._metadataKeys[key].__name__, self.__class__.__name__, fname, key))
                                self._fileMetadata[fname][key] = None
                            msg.debug('Now have {0}:{1} = {2!s}'.format(fname, key, self._fileMetadata[fname][key]))
    
    

_resetMetadata()

def python.trfArgClasses.argFile._resetMetadata (   self,   files = []  )

private

Resets all metadata files in this instance.

Metadata dictionary entry is reset for any files given (default all files) and any files that are no longer in this instance have any metadata removed (useful for self merging).

Note

Metadata is set to {}, except for the case when an explicit GUID option was given

Definition at line 913 of file trfArgClasses.py.

    def _resetMetadata(self, files=[]):
        if files == [] or '_fileMetadata' not in dir(self):
            self._fileMetadata = {}
            for fname in self.value:
                self._fileMetadata[fname] = {}
        else:
            for fname in files:
                if fname in self.value:
                    self._fileMetadata[fname] = {}
                elif fname in self._fileMetadata:
                    del self._fileMetadata[fname]
        msg.debug('Metadata dictionary now {0}'.format(self._fileMetadata))
        
        # If we have the special guid option, then manually try to set GUIDs we find
        if self._guid is not None:
            msg.debug('Now trying to set file GUID metadata using {0}'.format(self._guid))
            for fname, guid in self._guid.items():
                if fname in self._value:
                    self._fileMetadata[fname]['file_guid'] = guid
                else:
                    msg.warning('Explicit GUID {0} was passed for file {1}, but this file is not a member of this instance'.format(guid, fname))
    

_setMetadata()

def python.trfArgClasses.argFile._setMetadata (   self,   files = None,   metadataKeys = {}  )

private

Set metadata values into the cache.

Manually sets the metadata cache values to the values given in the metadata key dictionary here. This is useful for setting values to make checks on file metadata handling.

Note

To really suppress any external function calls that gather metadata be careful to also set the _exists metadatum to True.

Warning

No checks are done on the values or keys given here, so you'd better know what you are doing.

Parameters

files	Files to set metadata for (None means "all")
metadataKeys	Dictionary with metadata keys and values

Definition at line 1044 of file trfArgClasses.py.

    def _setMetadata(self, files=None, metadataKeys={}):
        if files is None:
            files = self._value
        for fname in files:
            if fname not in self._fileMetadata:
                self._fileMetadata[fname] = {}
            for k, v in metadataKeys.items():
                msg.debug('Manualy setting {0} for file {1} to {2}'.format(k, fname, v))
                self._fileMetadata[fname][k] = v
    
    

append()

def python.trfArgClasses.argList.append (   self,   addme  )

inherited

Append a value to the list.

Parameters

addme

Item to add

Definition at line 398 of file trfArgClasses.py.

    def append(self, addme):
        self._value.append(addme)
    

auxiliaryFile()

def python.trfArgClasses.argFile.auxiliaryFile

(

self

)

Definition at line 874 of file trfArgClasses.py.

    def auxiliaryFile(self):
        return self._auxiliaryFile
    

dataset() [1/2]

def python.trfArgClasses.argFile.dataset

(

self

)

Definition at line 814 of file trfArgClasses.py.

    def dataset(self):
        return self._dataset
    

dataset() [2/2]

def python.trfArgClasses.argFile.dataset	(		self,
			value
	)

Definition at line 818 of file trfArgClasses.py.

    def dataset(self, value):
        self._dataset = value
    

executor()

def python.trfArgClasses.argFile.executor

(

self

)

Executor status getter.

Definition at line 638 of file trfArgClasses.py.

    def executor(self):
        return self._exe
        

getMetadata()

def python.trfArgClasses.argFile.getMetadata	(		self,
			files = None,
			metadataKeys = None,
			maskMetadataKeys = None,
			populate = True,
			flush = False
	)

Return specific keys for specific files.

Parameters

files	List of files to return metadata for (default - all files in this instance)
metadataKeys	Keys to return (default - all keys valid for this class of files)
maskMetadataKeys	Keys to NOT return (useful when metadataKeys is left as default)
populate	If missing keys should be generated by calling the population subroutines
flush	If cached data should be flushed and the generators rerun

Definition at line 941 of file trfArgClasses.py.

    def getMetadata(self, files = None, metadataKeys = None, maskMetadataKeys = None, populate = True, flush = False):
        # Normalise the files and keys parameter
        if files is None:
            files = self._value
        elif isinstance(files, str):
            files = (files,)
        msg.debug('getMetadata will examine these files: {0!s}'.format(files))
 
        if metadataKeys is None:
            metadataKeys = list(self._metadataKeys)
        elif isinstance(metadataKeys, str):
            metadataKeys = [metadataKeys,]
        if maskMetadataKeys is not None:
            metadataKeys = [k for k in metadataKeys if k not in maskMetadataKeys]
        msg.debug('getMetadata will retrieve these keys: {0!s}'.format(metadataKeys))
        
        if flush is True:
            msg.debug('Flushing cached metadata values')
            self._resetMetadata()
            
        if populate is True:
            msg.debug('Checking metadata values')
            self._readMetadata(files, metadataKeys)
 
        metadata = {}
        for fname in files:
            metadata[fname] = {}
            for mdkey in metadataKeys:
                try:
                    metadata[fname][mdkey] = self._fileMetadata[fname][mdkey]
                except KeyError:
                    # This should not happen, unless we skipped populating
                    if populate:
                        msg.error('Did not find metadata key {0!s} for file {1!s} - setting to None'.format(mdkey, fname))
                    metadata[fname][mdkey] = None
        return metadata
    

getnentries()

def python.trfArgClasses.argFile.getnentries	(		self,
			fast = False
	)

Explicit getter, offering fast switch.

Definition at line 890 of file trfArgClasses.py.

    def getnentries(self, fast=False):
        totalEvents = 0
        for fname in self._value:
            events = self.getSingleMetadata(fname=fname, metadataKey='nentries', populate = not fast)
            if events is None:
                msg.debug('Got events=None for file {0} - returning None for this instance'.format(fname))
                return None
            if events == 'UNDEFINED':
                msg.debug('Got events=UNDEFINED for file {0} - returning UNDEFINED for this instance'.format(fname))
                return 'UNDEFINED'
            if not isinstance(events, int):
                msg.warning('Got unexpected events metadata for file {0}: {1!s} - returning None for this instance'.format(fname, events))
                return None
            totalEvents += events
        
        return totalEvents
                
    

getSingleMetadata()

def python.trfArgClasses.argFile.getSingleMetadata	(		self,
			fname,
			metadataKey,
			populate = True,
			flush = False
	)

Convenience function to extract a single metadata key for a single file.

Retrieves a single metadata item for a single file, returning it directly

Returns

Single metadata value

Parameters

fname	File to return metadata for
metadataKey	Key to return
populate	If missing key should be generated by calling the population subroutines
flush	If cached data should be flushed and the generator rerun

Definition at line 985 of file trfArgClasses.py.

    def getSingleMetadata(self, fname, metadataKey, populate = True, flush = False):
        if not (isinstance(fname, str) and isinstance(metadataKey, str)):
            raise trfExceptions.TransformInternalException(trfExit.nameToCode('TRF_INTERNAL'), 
                                                           'Illegal call to getSingleMetadata function: {0!s} {1!s}'.format(fname, metadataKey))
        md = self.getMetadata(files = fname, metadataKeys = metadataKey, populate = populate, flush = flush)
        return md[fname][metadataKey]
    
    

io() [1/2]

def python.trfArgClasses.argFile.io

(

self

)

Definition at line 803 of file trfArgClasses.py.

    def io(self):
        return (self._io)
 

io() [2/2]

def python.trfArgClasses.argFile.io	(		self,
			value
	)

Definition at line 807 of file trfArgClasses.py.

    def io(self, value):
        if value not in ('input', 'output', 'temporary'):
            raise trfExceptions.TransformArgException(trfExit.nameToCode('TRF_RUNTIME_ERROR'), 
                                                      'File arguments must be specified as input, output or temporary - got {0}'.format(value))
        self._io = value
    

isCached()

def python.trfArgClasses.argFile.isCached	(		self,
			files = None,
			metadataKeys = None
	)

Test if certain metadata elements are already cached.

Will test for a cached value for all files and all keys given, aborting as soon as it finds a single uncached value.

Parameters

files	Files to check (defaults to all files)
metadataKeys	Keys to check (defaults to all keys)

Returns

Boolean if all keys are cached for all files

Definition at line 1061 of file trfArgClasses.py.

    def isCached(self, files = None, metadataKeys = None):
        msg.debug('Testing for cached values for files {0} and keys {1}'.format(files, metadataKeys))
        if files is None:
            files = self._value
        elif isinstance(files, str):
            files = (files,)
        if metadataKeys is None:
            metadataKeys = list(self._metadataKeys)
        elif isinstance(metadataKeys, str):
            metadataKeys = (metadataKeys,)
            
        isCachedFlag = True
        for fname in files:
            for key in metadataKeys:
                if key not in self._fileMetadata[fname]:
                    isCachedFlag = False
                    break
            if isCachedFlag is False:
                break
            
        return isCachedFlag
    

isRunarg()

def python.trfArgClasses.argument.isRunarg (   self )

inherited

Return runarg status.

Definition at line 134 of file trfArgClasses.py.

    def isRunarg(self):
        return self._runarg
    

mergeTargetSize() [1/2]

def python.trfArgClasses.argFile.mergeTargetSize

(

self

)

mergeTargeSize value getter

Definition at line 613 of file trfArgClasses.py.

    def mergeTargetSize(self):
        return self._mergeTargetSize
 

mergeTargetSize() [2/2]

def python.trfArgClasses.argFile.mergeTargetSize	(		self,
			value
	)

mergeTargeSize value setter

Definition at line 618 of file trfArgClasses.py.

    def mergeTargetSize(self, value):
        if value is None:
            self._mergeTargetSize = 0
        else:
            self._mergeTargetSize = value
 

metadata()

def python.trfArgClasses.argFile.metadata

(

self

)

Returns the whole kit and kaboodle...

Note

Populates the whole metadata dictionary for this instance

Definition at line 880 of file trfArgClasses.py.

    def metadata(self):
        self.getMetadata()
        return self._fileMetadata
    

multipleOK() [1/2]

def python.trfArgClasses.argFile.multipleOK

(

self

)

multipleOK getter

Returns

Current value

Definition at line 603 of file trfArgClasses.py.

    def multipleOK(self):
        return self._multipleOK
    

multipleOK() [2/2]

def python.trfArgClasses.argFile.multipleOK	(		self,
			value
	)

multipleOK value setter

Definition at line 608 of file trfArgClasses.py.

    def multipleOK(self, value):
        self._multipleOK = value
        

name() [1/2]

def python.trfArgClasses.argFile.name

(

self

)

Name getter.

Reimplemented from python.trfArgClasses.argument.

Definition at line 847 of file trfArgClasses.py.

    def name(self):
        return self._name
    

name() [2/2]

def python.trfArgClasses.argFile.name	(		self,
			value
	)

Name setter.

Note

This property setter will also set the type and subtype of the argFile if they are not yet set. This means that for most arguments the type and subtype are automatically set correctly.

Reimplemented from python.trfArgClasses.argument.

Definition at line 855 of file trfArgClasses.py.

    def name(self, value):
        self._name = value
        m = re.match(r'(input|output|tmp.)([A-Za-z0-9_]+?)(File)?$', value)
        if m:
            msg.debug("ArgFile name setter matched this: {0}".format(m.groups()))
            if self._type is None:
                dtype = m.group(2).split('_', 1)[0]
                # But DRAW/DESD/DAOD are really just RAW, ESD, AOD in  format
                if re.match(r'D(RAW|ESD|AOD)', dtype):
                    dtype = dtype[1:]
                msg.debug("Autoset data type to {0}".format(dtype))
                self._type = dtype
            if self._subtype is None:
                msg.debug("Autoset data subtype to {0}".format(m.group(2)))
                self._subtype = m.group(2)
        else:
            msg.debug("ArgFile name setter did not match against '{0}'".format(value))
 

nentries()

def python.trfArgClasses.argFile.nentries

(

self

)

Return total number of events in all constituent files.

Definition at line 886 of file trfArgClasses.py.

    def nentries(self):
        return self.getnentries()
        

originalName()

def python.trfArgClasses.argFile.originalName	(		self,
			value
	)

Definition at line 826 of file trfArgClasses.py.

    def originalName(self, value):
        self._originalName = value
    

orignalName()

def python.trfArgClasses.argFile.orignalName

(

self

)

Definition at line 822 of file trfArgClasses.py.

    def orignalName(self):
        return self._originalName
    

prodsysDescription()

def python.trfArgClasses.argFile.prodsysDescription

(

self

)

Reimplemented from python.trfArgClasses.argList.

Reimplemented in python.trfArgClasses.argFTKIPFile, python.trfArgClasses.argBZ2File, python.trfArgClasses.argNTUPFile, python.trfArgClasses.argHISTFile, python.trfArgClasses.argPOOLFile, python.trfArgClasses.argBSFile, and python.trfArgClasses.argAthenaFile.

Definition at line 625 of file trfArgClasses.py.

    def prodsysDescription(self):
        if isinstance(self._type, dict):
            if self._type=={}:
                desc = {'type' : 'file', 'subtype' : "NONE" }
            else:
                desc = {'type' : 'file', 'subtype' : dict((str(k).upper(), str(v).upper()) for (k,v) in self._type.items())}
        else:
            desc = {'type' : 'file', 'subtype' : str(self._type).upper()}
            desc['multiple'] = self._multipleOK
        return desc 
    

subtype() [1/2]

def python.trfArgClasses.argFile.subtype

(

self

)

Definition at line 838 of file trfArgClasses.py.

    def subtype(self):
        return self._subtype
    

subtype() [2/2]

def python.trfArgClasses.argFile.subtype	(		self,
			value
	)

Definition at line 842 of file trfArgClasses.py.

    def subtype(self, value):
        self._subtype = value
 

type() [1/2]

def python.trfArgClasses.argFile.type

(

self

)

Definition at line 830 of file trfArgClasses.py.

    def type(self):
        return self._type
    

type() [2/2]

def python.trfArgClasses.argFile.type	(		self,
			value
	)

Definition at line 834 of file trfArgClasses.py.

    def type(self, value):
        self._type = value
 

value() [1/2]

def python.trfArgClasses.argFile.value

(

self

)

Argument value getter.

Returns

Current value

Reimplemented from python.trfArgClasses.argList.

Definition at line 591 of file trfArgClasses.py.

    def value(self):
        return self._value
 

value() [2/2]

def python.trfArgClasses.argFile.value	(		self,
			value
	)

Argument value setter.

Calls the valueSetter function with the standard options

Reimplemented from python.trfArgClasses.argList.

Definition at line 597 of file trfArgClasses.py.

    def value(self, value):
        self.valueSetter(value)
        

valueSetter()

def python.trfArgClasses.argFile.valueSetter	(		self,
			value
	)

Set the argFile value, but allow parameters here.

Note

Normally athena only takes a single value for an output file, but when AthenaMP runs it can produce multiple output files - this is allowed by setting allowMultiOutputs = True

The setter protects against the same file being added multiple times

Definition at line 645 of file trfArgClasses.py.

    def valueSetter(self, value):
        
        if isinstance(value, (list, tuple)):
            if len(value) > 0 and isinstance(value[0], dict): # Tier-0 style expanded argument with metadata
                self._value=[]
                for myfile in value:
                    try:
                        self._value.append(myfile['lfn'])
                        self._resetMetadata(files = [myfile['lfn']])
                    except KeyError:
                        raise trfExceptions.TransformArgException(trfExit.nameToCode('TRF_ARG_CONV_FAIL'),
                                                                  'Filename (key "lfn") not found in Tier-0 file dictionary: {0}'.format(myfile))
                    for k, v in myfile.items():
                        if k == 'guid':
                            self._setMetadata([myfile['lfn']], {'file_guid': v})
                        elif k == 'events':
                            self._setMetadata([myfile['lfn']], {'nentries': v})
                        elif k == 'checksum':
                            self._setMetadata([myfile['lfn']], {'checksum': v})
                        elif k == 'dsn':
                            if not self._dataset:
                                self.dataset = v
                            elif self.dataset != v:
                                raise trfExceptions.TransformArgException(trfExit.nameToCode('TRF_ARG_DATASET'),
                                                                          'Inconsistent dataset names in Tier-0 dictionary: {0} != {1}'.format(self.dataset, v))
            else:
                self._value = list(value)
                self._getDatasetFromFilename(reset = False)
                self._resetMetadata()
        elif value is None:
            self._value = []
            return
        else:
            try:
                if value.lower().startswith('lfn'):
                    # Resolve physical filename using pool file catalog.
                    from PyUtils.PoolFile import file_name
                    protocol, pfn = file_name(value)
                    self._value = [pfn]
                    self._getDatasetFromFilename(reset = False)
                    self._resetMetadata()
                else:
                    self._value = value.split(self._splitter)
                    self._getDatasetFromFilename(reset = False)
                    self._resetMetadata()
            except (AttributeError, TypeError):
                raise trfExceptions.TransformArgException(trfExit.nameToCode('TRF_ARG_CONV_FAIL'),
                                                          'Failed to convert %s to a list' % str(value))
 
        
        deDuplicatedValue = []
        for fname in self._value:
            if fname not in deDuplicatedValue:
                deDuplicatedValue.append(fname)
            else:
                msg.warning("Removing duplicated file {0} from file list".format(fname))
        if len(self._value) != len(deDuplicatedValue):
            self._value = deDuplicatedValue
            msg.warning('File list after duplicate removal: {0}'.format(self._value))
        
        # Find our URL type (if we actually have files!)
        # At the moment this is assumed to be the same for all files in this instance
        # although in principle one could mix different access methods in the one input file type
        if len(self._value) > 0:
            self._urlType = urlType(self._value[0])
        else:
            self._urlType = None
 
        
        if self._io == 'input':
            
            if self._urlType == 'posix':
                msg.debug('Found POSIX filesystem input - activating globbing')
                newValue = []
                for filename in self._value:
                    # Simple case
                    globbedFiles = glob.glob(filename)
                    if len(globbedFiles) == 0:          # No files globbed for this 'filename' argument.
                        raise trfExceptions.TransformArgException(trfExit.nameToCode('TRF_INPUT_FILE_ERROR'),
                        'Input file argument {0} globbed to NO input files - probably the file(s) are missing'.format(filename))
 
                    globbedFiles.sort()
                    newValue.extend(globbedFiles)
 
                self._value = newValue
                msg.debug ('File input is globbed to %s' % self._value)
 
            elif self._urlType == 'root':
                msg.debug('Found root filesystem input - activating globbing')
                newValue = []
                for filename in self._value:
                    if str(filename).startswith("root"):
                        msg.debug('Found input file name starting with "root," setting XRD_RUNFORKHANDLER=1, which enables fork handlers for xrootd in direct I/O')
                        os.environ["XRD_RUNFORKHANDLER"] = "1"
                    if str(filename).startswith("https") or str(filename).startswith("davs") or not(str(filename).endswith('/')) and '*' not in filename and '?' not in filename:
                        msg.debug('Seems that only one file was given: {0}'.format(filename))
                        newValue.extend(([filename]))
                    else:
                        # Hopefully this recognised wildcards...
                        path = filename
                        fileMask = ''
                        if '*' in filename or '?' in filename:
                            msg.debug('Split input into path for listdir() and a filemask to select available files.')
                            path = filename[0:filename.rfind('/')+1]
                            msg.debug('path: {0}'.format(path))
                            fileMask = filename[filename.rfind('/')+1:len(filename)]
                            msg.debug('Will select according to: {0}'.format(fileMask))
 
                        cmd = ['/afs/cern.ch/project/eos/installation/atlas/bin/eos.select' ]
                        if not os.access ('/afs/cern.ch/project/eos/installation/atlas/bin/eos.select', os.X_OK ):
                            raise trfExceptions.TransformArgException(trfExit.nameToCode('TRF_INPUT_FILE_ERROR'),
                            'No execute access to "eos.select" - could not glob EOS input files.')
 
                        cmd.extend(['ls'])
                        cmd.extend([path])
 
                        myFiles = []
                        try:
                            proc = subprocess.Popen(args = cmd,bufsize = 1, shell = False, stdout = subprocess.PIPE, stderr = subprocess.STDOUT)
                            rc = proc.wait()
                            output = proc.stdout.readlines()
                            if rc!=0:
                                raise trfExceptions.TransformArgException(trfExit.nameToCode('TRF_INPUT_FILE_ERROR'),
                                                            'EOS list command ("{0!s}") failed: rc {1}, output {2}'.format(cmd, rc, output))
                            msg.debug("eos returned: {0}".format(output))
                            for line in output:
                                if "root" in line:
                                    myFiles += [str(path)+str(line.rstrip('\n'))]
 
                            patt = re.compile(fileMask.replace('*','.*').replace('?','.'))
                            for srmFile in myFiles:
                                if fileMask != '':
                                    if(patt.search(srmFile)) is not None:
                                    #if fnmatch.fnmatch(srmFile, fileMask):
                                        msg.debug('match: ',srmFile)
                                        newValue.extend(([srmFile]))
                                else:
                                    newValue.extend(([srmFile]))
                                
                            msg.debug('Selected files: ', newValue)
                        except (AttributeError, TypeError, OSError):
                            raise trfExceptions.TransformArgException(trfExit.nameToCode('TRF_RUNTIME_ERROR'),
                                                                      'Failed to convert %s to a list' % str(value))
                if len(self._value) > 0 and len(newValue) == 0:
                    # Woops - no files!
                    raise trfExceptions.TransformArgException(trfExit.nameToCode('TRF_INPUT_FILE_ERROR'),
                                                              'Input file argument(s) {0!s} globbed to NO input files - ls command failed')
                self._value = newValue
                msg.debug ('File input is globbed to %s' % self._value)
        # Check if multiple outputs are ok for this object
        elif self._multipleOK is False and len(self._value) > 1:
            raise trfExceptions.TransformArgException(trfExit.nameToCode('TRF_OUTPUT_FILE_ERROR'), 
                                                      'Multiple file arguments are not supported for {0} (was given: {1}'.format(self, self._value))
            

Member Data Documentation

_auxiliaryFile

python.trfArgClasses.argFile._auxiliaryFile

private

Definition at line 553 of file trfArgClasses.py.

_dataset

python.trfArgClasses.argFile._dataset

private

Definition at line 547 of file trfArgClasses.py.

_exe

python.trfArgClasses.argFile._exe

private

Definition at line 559 of file trfArgClasses.py.

_fileMetadata

python.trfArgClasses.argFile._fileMetadata

private

Definition at line 574 of file trfArgClasses.py.

_guid

python.trfArgClasses.argFile._guid

private

Definition at line 551 of file trfArgClasses.py.

_io

python.trfArgClasses.argFile._io

private

Input file globbing and expansion.

Definition at line 576 of file trfArgClasses.py.

_mergeTargetSize

python.trfArgClasses.argFile._mergeTargetSize

private

Definition at line 552 of file trfArgClasses.py.

_metadataKeys

python.trfArgClasses.argFile._metadataKeys

private

Note

Variable listing set of file metadata which corresponds to this class, Key is the metadata variable name, the value is the function to call to populate/refresh this metadata value. Function must take a single parameter, which is the list of files to get metadata for. It must return a metadata dictionary: {file1 : {key1: value1, key2: value2}, file2: ...} Keys which start with _ are for transform internal use and should not appear in jobReports

Definition at line 569 of file trfArgClasses.py.

_multipleOK

python.trfArgClasses.argFile._multipleOK

private

Definition at line 577 of file trfArgClasses.py.

_name

python.trfArgClasses.argFile._name

private

Definition at line 856 of file trfArgClasses.py.

_originalName

python.trfArgClasses.argFile._originalName

private

Definition at line 554 of file trfArgClasses.py.

_runarg

python.trfArgClasses.argument._runarg

privateinherited

Definition at line 110 of file trfArgClasses.py.

_splitter

python.trfArgClasses.argList._splitter

privateinherited

Definition at line 357 of file trfArgClasses.py.

_subtype

python.trfArgClasses.argFile._subtype

private

Definition at line 550 of file trfArgClasses.py.

_supressEmptyStrings

python.trfArgClasses.argList._supressEmptyStrings

privateinherited

Definition at line 358 of file trfArgClasses.py.

_type

python.trfArgClasses.argFile._type

private

Definition at line 549 of file trfArgClasses.py.

_urlType

python.trfArgClasses.argFile._urlType

private

Note

TODO: Non-posix URLs Problem is not so much the [] expansion, but the invisible .N attempt number One can only deal with this with a listdir() functionality N.B. Current transforms only do globbing on posix fs too (see trfutil.expandStringToList())

Definition at line 548 of file trfArgClasses.py.

_value

python.trfArgClasses.argFile._value

private

Note

First do parsing of string vs. lists to get list of files

Check for duplicates (N.B. preserve the order, just remove the duplicates)

Definition at line 649 of file trfArgClasses.py.

dataset

python.trfArgClasses.argFile.dataset

Definition at line 666 of file trfArgClasses.py.

io

python.trfArgClasses.argFile.io

Definition at line 557 of file trfArgClasses.py.

value

python.trfArgClasses.argument.value

inherited

Note

We have a default of None here, but all derived classes should definitely have their own value setter and translate this value to something sensible for their underlying value type. N.B. As most argument classes use this default constructor it must call the @value .setter function!

Definition at line 118 of file trfArgClasses.py.

---

The documentation for this class was generated from the following file:

• trfArgClasses.py