TarBZ filetype. More...

Inheritance diagram for python.trfArgClasses.argBZ2File:

Collaboration diagram for python.trfArgClasses.argBZ2File:

Public Types
typedef HLT::TypeInformation::for_each_type_c< typenameEDMLIST::map, my_functor, my_result<>, my_arg< HLT::TypeInformation::get_cont, CONTAINER > >::type	result

Public Member Functions
	prodsysDescription (self)
	io (self)
	io (self, value)
	dataset (self)
	dataset (self, value)
	value (self)
	Argument value getter.
	value (self, value)
	Argument value setter.
	name (self)
	Name getter.
	name (self, value)
	Name setter.
	type (self)
	type (self, value)
	multipleOK (self)
	multipleOK getter
	multipleOK (self, value)
	multipleOK value setter
	mergeTargetSize (self)
	mergeTargeSize value getter
	mergeTargetSize (self, value)
	mergeTargeSize value setter
	executor (self)
	Executor status getter.
	valueSetter (self, value)
	Set the argFile value, but allow parameters here.
	orignalName (self)
	originalName (self, value)
	subtype (self)
	subtype (self, value)
	auxiliaryFile (self)
	metadata (self)
	Returns the whole kit and kaboodle...
	nentries (self)
	Return total number of events in all constituent files.
	getnentries (self, fast=False)
	Explicit getter, offering fast switch.
	getMetadata (self, files=None, metadataKeys=None, maskMetadataKeys=None, populate=True, flush=False)
	Return specific keys for specific files.
	getSingleMetadata (self, fname, metadataKey, populate=True, flush=False)
	Convenience function to extract a single metadata key for a single file.
	isCached (self, files=None, metadataKeys=None)
	Test if certain metadata elements are already cached.
	__str__ (self)
	String representation of a file argument.
	append (self, addme)
	Append a value to the list.
	__repr__ (self)
	Repr conversion.
	isRunarg (self)
	Return runarg status.
	__eq__ (self, other)
	Comparison is based on value attribute.
	__nq__ (self, other)
	__lt__ (self, other)
	__gt__ (self, other)

Public Attributes
	io = io
	dataset = v
	type

Protected Member Functions
	_getIntegrity (self, files)
	File integrity checker.
	_resetMetadata (self, files=[])
	Resets all metadata files in this instance.
	_readMetadata (self, files, metadataKeys)
	Check metadata is in the cache or generate it if it's missing.
	_setMetadata (self, files=None, metadataKeys={})
	Set metadata values into the cache.
	_getDatasetFromFilename (self, reset=False)
	Look for dataset name in dataset#filename Tier0 convention.
	_getSize (self, files)
	Determines the size of files.
	_generateGUID (self, files)
	Generate a GUID on demand - no intrinsic for this file type.
	_exists (self, files)
	Try to determine if a file actually exists...
	_mergeArgs (self, argdict, copyArgs=None)
	Utility to strip arguments which should not be passed to the selfMerge methods of our child classes.

Protected Attributes
	_dataset = None
str	_urlType = None
	Input file globbing and expansion.
dict	_type = type
	_subtype = subtype
	_guid = guid
int	_mergeTargetSize = mergeTargetSize
	_auxiliaryFile = auxiliaryFile
	_originalName = None
	_exe = executor
dict	_metadataKeys
dict	_fileMetadata = {}
str	_io = 'input':
	Input file globbing and expansion.
bool	_multipleOK = True
	_splitter = splitter
	_supressEmptyStrings = supressEmptyStrings
	_runarg = runarg
	_name = name
	_value = value

Detailed Description

TarBZ filetype.

Definition at line 1798 of file trfArgClasses.py.

Member Typedef Documentation

◆ result

typedef HLT::TypeInformation::for_each_type_c<typenameEDMLIST::map,my_functor,my_result<>,my_arg<HLT::TypeInformation::get_cont,CONTAINER>>::type master_search< typenameEDMLIST::map, HLT::TypeInformation::get_cont, CONTAINER >::result

inherited

Definition at line 90 of file EDM_MasterSearch.h.

Member Function Documentation

◆ eq()

python.trfArgClasses.argument.__eq__	(		self,
			other )

inherited

Comparison is based on value attribute.

Definition at line 161 of file trfArgClasses.py.

    def __eq__(self,other):
        return self.value == other.value
 

◆ gt()

python.trfArgClasses.argument.__gt__	(		self,
			other )

inherited

Definition at line 170 of file trfArgClasses.py.

    def __gt__(self, other):
        return self.value > other.value
 

◆ lt()

python.trfArgClasses.argument.__lt__	(		self,
			other )

inherited

Definition at line 167 of file trfArgClasses.py.

    def __lt__(self, other):
        return self.value < other.value
 

◆ nq()

python.trfArgClasses.argument.__nq__	(		self,
			other )

inherited

Definition at line 164 of file trfArgClasses.py.

    def __nq__(self, other):
        return self.value != other.value
 

◆ repr()

python.trfArgClasses.argList.__repr__ ( self )

inherited

Repr conversion.

Return a python parsable string

Definition at line 409 of file trfArgClasses.py.

    def __repr__(self):
        return '[' + ','.join([ repr(s) for s in self._value ]) + ']'

◆ str()

python.trfArgClasses.argFile.__str__ ( self )

inherited

String representation of a file argument.

Definition at line 1216 of file trfArgClasses.py.

    def __str__(self):
        return "{0}={1} (Type {2}, Dataset {3}, IO {4})".format(self.name, self.value, self.type, self.dataset, self.io)

◆ _exists()

python.trfArgClasses.argFile._exists	(		self,
			files )

protectedinherited

Try to determine if a file actually exists...

For a posix file, just call stat; for anything else call TFile.Open A small optimisation is to retieve the file_size metadatum at the same time.

Parameters

c files List of paths to test for existance

Returns: None (internal self._fileMetadata cache is updated)

Definition at line 1180 of file trfArgClasses.py.

    def _exists(self, files):
        import re
        msg.debug('Testing existance for {0}'.format(files))
        def split_filelist(fn):
            if self.io != 'output':
                return [fn]
            file_split_regex = re.compile(r"(.+)\[(.+)](.+)")
            if ('[' in fn) and (']' in fn):
                match = file_split_regex.match(fn)
                return [f"{match.group(1)}{it}{match.group(3)}" for it in match.group(2).split(',')]
            else:
                return [fn]
        for fname in files:
            file_list = split_filelist(fname)
            if self._urlType == 'posix':
                try:
                    size = map(lambda fn: os.stat(fn).st_size, file_list)
                    self._fileMetadata[fname]['file_size'] = sum(size)
                    self._fileMetadata[fname]['_exists'] = True
                    msg.debug('POSIX file {0} exists (or all elements of list)'.format(fname))
                except OSError as e:
                    msg.error('Got exception {0!s} raised while stating file {1} (or some element of list)  - probably it does not exist'.format(e, fname))
                    self._fileMetadata[fname]['_exists'] = False
            else:
                # OK, let's see if ROOT can do it...
                msg.debug('Calling ROOT TFile.GetSize on {0} (or elements of list)'.format(fname))
                size = map(ROOTGetSize, file_list)
                if None in size:
                    self._fileMetadata[fname]['_exists'] = False
                    msg.error('Non-POSIX file {0} (or element of list) could not be opened - probably it does not exist'.format(fname))
                else:
                    msg.debug('Non-POSIX file {0} (or all elements of list) exists'.format(fname))
                    self._fileMetadata[fname]['file_size'] = sum(size)
                    self._fileMetadata[fname]['_exists'] = True
 

◆ _generateGUID()

python.trfArgClasses.argFile._generateGUID	(		self,
			files )

protectedinherited

Generate a GUID on demand - no intrinsic for this file type.

Use uuid.uuid4() call to generate a GUID

Note: This generation method will be superceeded in any file type which actually has an intrinsic GUID (e.g. BS or POOL files)

Definition at line 1169 of file trfArgClasses.py.

    def _generateGUID(self, files):
        for fname in files:
            msg.debug('Generating a GUID for file {0}'.format(fname))
            self._fileMetadata[fname]['file_guid'] = str(uuid.uuid4()).upper()
            
            

◆ _getDatasetFromFilename()

python.trfArgClasses.argFile._getDatasetFromFilename	(		self,
			reset = False )

protectedinherited

Look for dataset name in dataset#filename Tier0 convention.

At the moment all files must be in the same dataset if it's specified. (To change this dataset will need to become a per-file metadatum.)

Note: dsn#lfn notation must be used for all input values and all dsn values must be the same

Parameters

c reset If True then forget previous dataset setting. Default is False.

Returns: None. Side effect is to set self._metadata.

Definition at line 1093 of file trfArgClasses.py.

    def _getDatasetFromFilename(self, reset = False):
        if reset:
            self._dataset = None
        newValue = []
        for filename in self._value:
            if filename.find('#') > -1:
                (dataset, fname) = filename.split('#', 1)
                newValue.append(fname)
                msg.debug('Current dataset: {0}; New dataset {1}'.format(self._dataset, dataset))
                if self._dataset and (self._dataset != dataset):
                    raise trfExceptions.TransformArgException(trfExit.nameToCode('TRF_ARG_DATASET'), 
                                                              'Found inconsistent dataset assignment in argFile setup: %s != %s' % (self._dataset, dataset))
                self._dataset = dataset
        if len(newValue) == 0:
            return
        elif len(newValue) != len (self._value):
            raise trfExceptions.TransformArgException(trfExit.nameToCode('TRF_ARG_DATASET'), 
                                                      'Found partial dataset assignment in argFile setup from {0} (dsn#lfn notation must be uniform for all inputs)'.format(self._value))
        self._value = newValue
    

◆ _getIntegrity()

python.trfArgClasses.argBZ2File._getIntegrity	(		self,
			files )

protected

File integrity checker.

For a 'plain' file, integrity just checks that we can read it

Parameters

c files List of paths to the files for which the integrity is determined

Returns: None (internal self._fileMetadata cache is updated)

Reimplemented from python.trfArgClasses.argFile.

Definition at line 1799 of file trfArgClasses.py.

    def _getIntegrity(self, files):
        for fname in files:
            # bz2 only supports 'with' from python 2.7
            try:
                f = bz2.BZ2File(fname, 'r')
                while True:
                    chunk = len(f.read(1024*1024))
                    msg.debug('Read {0} bytes from {1}'.format(chunk, fname)) 
                    if chunk == 0:
                        break
                self._fileMetadata[fname]['integrity'] = True
                f.close()
            except OSError as e:
                msg.error('Got exception {0!s} raised while checking integrity of file {1}'.format(e, fname))
                self._fileMetadata[fname]['integrity'] = False
                
 

◆ _getSize()

python.trfArgClasses.argFile._getSize	(		self,
			files )

protectedinherited

Determines the size of files.

Currently only for statable files (posix fs). Caches the

Parameters

files List of paths to the files for which the size is determined.

Returns: None (internal self._fileMetadata cache is updated)

Definition at line 1117 of file trfArgClasses.py.

    def _getSize(self, files):
        for fname in files:
            if self._urlType == 'posix':
                try:
                    self._fileMetadata[fname]['size'] = os.stat(fname).st_size
                except OSError as e:
                    msg.error('Got exception {0!s} raised while stating file {1}'.format(e, fname))
                    self._fileMetadata[fname]['size'] = None
            else:
                # OK, let's see if ROOT can do it...
                msg.debug('Calling ROOT TFile.GetSize({0})'.format(fname))
                self._fileMetadata[fname]['size'] = ROOTGetSize(fname)
 
 

◆ _mergeArgs()

python.trfArgClasses.argFile._mergeArgs	(	self,
		argdict,
		copyArgs = None )

protectedinherited

Utility to strip arguments which should not be passed to the selfMerge methods of our child classes.

Parameters

copyArgs If None copy all arguments by default, otherwise only copy the listed keys

Definition at line 1224 of file trfArgClasses.py.

    def _mergeArgs(self, argdict, copyArgs=None):
        if copyArgs:
            myargdict = {}
            for arg in copyArgs:
                if arg in argdict:
                    myargdict[arg] = copy.copy(argdict[arg])
            
        else:
            myargdict = copy.copy(argdict)
        # Never do event count checks for self merging
        myargdict['checkEventCount'] = argSubstepBool('False', runarg=False) 
        newopts = []
        if 'athenaopts' in myargdict:
            # Need to ensure that "nprocs" is not passed to merger
            # and prevent multiple '--threads' options when there are multiple sub-steps in 'athenopts'
            for subStep in myargdict['athenaopts'].value:
                hasNprocs = False
                hasNthreads = False
                for opt in myargdict['athenaopts'].value[subStep]:
                    if opt.startswith('--nprocs'):
                        hasNprocs = True
                        continue
                    # Keep at least one '--threads'
                    elif opt.startswith('--threads'):
                        hasNthreads = True
                        if opt in newopts:
                            continue
                    newopts.append(opt)
                # If we have hybrid MP+MT job make sure --threads is not passed to merger
                if hasNprocs and hasNthreads:
                    tmpopts = []
                    for opt in newopts:
                        if opt.startswith('--threads'):
                            continue
                        tmpopts.append(opt)
                    newopts = tmpopts
            myargdict['athenaopts'] = argSubstepList(newopts, runarg=False)
        return myargdict
 
 

◆ _readMetadata()

python.trfArgClasses.argFile._readMetadata	(	self,
		files,
		metadataKeys )

protectedinherited

Check metadata is in the cache or generate it if it's missing.

Returns: : dictionary of files with metadata, for any unknown keys 'UNDEFINED' is returned

Definition at line 1000 of file trfArgClasses.py.

    def _readMetadata(self, files, metadataKeys):
        msg.debug('Retrieving metadata keys {1!s} for files {0!s}'.format(files, metadataKeys))
        for fname in files:
            if fname not in self._fileMetadata:
                self._fileMetadata[fname] = {}
        for fname in files:
            # Always try for a simple existence test first before producing misleading error messages
            # from metadata populator functions
            if '_exists' not in self._fileMetadata[fname]:
                self._metadataKeys['_exists'](files)
            if self._fileMetadata[fname]['_exists'] is False:
                # N.B. A log ERROR message has printed by the existence test, so do not repeat that news here
                for key in metadataKeys:
                    if key != '_exists':
                        self._fileMetadata[fname][key] = None
            else:
                # OK, file seems to exist at least... 
                for key in metadataKeys:
                    if key not in self._metadataKeys:
                        msg.debug('Metadata key {0} is unknown for {1}'.format(key, self.__class__.__name__))
                        self._fileMetadata[fname][key] = 'UNDEFINED'
                    else:
                        if key in self._fileMetadata[fname]:
                            msg.debug('Found cached value for {0}:{1} = {2!s}'.format(fname, key, self._fileMetadata[fname][key]))
                        else:
                            msg.debug('No cached value for {0}:{1}. Calling generator function {2} ({3})'.format(fname, key, self._metadataKeys[key].__name__, self._metadataKeys[key]))
                            try:
                                # For efficiency call this routine with all files we have
                                msg.info("Metadata generator called to obtain {0} for {1}".format(key, files))
                                self._metadataKeys[key](files)
                            except trfExceptions.TransformMetadataException as e:
                                msg.error('Calling {0!s} raised an exception: {1!s}'.format(self._metadataKeys[key].__name__, e))
                            if key not in self._fileMetadata[fname]:
                                msg.warning('Call to function {0} for {1} file {2} failed to populate metadata key {3}'.format(self._metadataKeys[key].__name__, self.__class__.__name__, fname, key))
                                self._fileMetadata[fname][key] = None
                            msg.debug('Now have {0}:{1} = {2!s}'.format(fname, key, self._fileMetadata[fname][key]))
    
    

◆ _resetMetadata()

python.trfArgClasses.argFile._resetMetadata	(		self,
			files = [] )

protectedinherited

Resets all metadata files in this instance.

Metadata dictionary entry is reset for any files given (default all files) and any files that are no longer in this instance have any metadata removed (useful for self merging).

Note: Metadata is set to {}, except for the case when an explicit GUID option was given

Definition at line 917 of file trfArgClasses.py.

    def _resetMetadata(self, files=[]):
        if files == [] or '_fileMetadata' not in dir(self):
            self._fileMetadata = {}
            for fname in self.value:
                self._fileMetadata[fname] = {}
        else:
            for fname in files:
                if fname in self.value:
                    self._fileMetadata[fname] = {}
                elif fname in self._fileMetadata:
                    del self._fileMetadata[fname]
        msg.debug('Metadata dictionary now {0}'.format(self._fileMetadata))
        
        # If we have the special guid option, then manually try to set GUIDs we find
        if self._guid is not None:
            msg.debug('Now trying to set file GUID metadata using {0}'.format(self._guid))
            for fname, guid in self._guid.items():
                if fname in self._value:
                    self._fileMetadata[fname]['file_guid'] = guid
                else:
                    msg.warning('Explicit GUID {0} was passed for file {1}, but this file is not a member of this instance'.format(guid, fname))
    

◆ _setMetadata()

python.trfArgClasses.argFile._setMetadata	(	self,
		files = None,
		metadataKeys = {} )

protectedinherited

Set metadata values into the cache.

Manually sets the metadata cache values to the values given in the metadata key dictionary here. This is useful for setting values to make checks on file metadata handling.

Note: To really suppress any external function calls that gather metadata be careful to also set the _exists metadatum to True.

Warning: No checks are done on the values or keys given here, so you'd better know what you are doing.

Parameters

files	Files to set metadata for (`None` means "all")
metadataKeys	Dictionary with metadata keys and values

Definition at line 1048 of file trfArgClasses.py.

    def _setMetadata(self, files=None, metadataKeys={}):
        if files is None:
            files = self._value
        for fname in files:
            if fname not in self._fileMetadata:
                self._fileMetadata[fname] = {}
            for k, v in metadataKeys.items():
                msg.debug('Manualy setting {0} for file {1} to {2}'.format(k, fname, v))
                self._fileMetadata[fname][k] = v
    
    

◆ append()

python.trfArgClasses.argList.append	(		self,
			addme )

inherited

Append a value to the list.

Parameters

addme Item to add

Definition at line 398 of file trfArgClasses.py.

    def append(self, addme):
        self._value.append(addme)
    

◆ auxiliaryFile()

python.trfArgClasses.argFile.auxiliaryFile ( self )

inherited

Definition at line 878 of file trfArgClasses.py.

    def auxiliaryFile(self):
        return self._auxiliaryFile
    

◆ dataset() [1/2]

python.trfArgClasses.argFile.dataset ( self )

inherited

Definition at line 818 of file trfArgClasses.py.

    def dataset(self):
        return self._dataset
    

◆ dataset() [2/2]

python.trfArgClasses.argFile.dataset	(		self,
			value )

inherited

Definition at line 822 of file trfArgClasses.py.

    def dataset(self, value):
        self._dataset = value
    

◆ executor()

python.trfArgClasses.argFile.executor ( self )

inherited

Executor status getter.

Definition at line 638 of file trfArgClasses.py.

    def executor(self):
        return self._exe
        

◆ getMetadata()

python.trfArgClasses.argFile.getMetadata	(	self,
		files = None,
		metadataKeys = None,
		maskMetadataKeys = None,
		populate = True,
		flush = False )

inherited

Return specific keys for specific files.

Parameters

files	List of files to return metadata for (default - all files in this instance)
metadataKeys	Keys to return (default - all keys valid for this class of files)
maskMetadataKeys	Keys to NOT return (useful when metadataKeys is left as default)
populate	If missing keys should be generated by calling the population subroutines
flush	If cached data should be flushed and the generators rerun

Definition at line 945 of file trfArgClasses.py.

    def getMetadata(self, files = None, metadataKeys = None, maskMetadataKeys = None, populate = True, flush = False):
        # Normalise the files and keys parameter
        if files is None:
            files = self._value
        elif isinstance(files, str):
            files = (files,)
        msg.debug('getMetadata will examine these files: {0!s}'.format(files))
 
        if metadataKeys is None:
            metadataKeys = list(self._metadataKeys)
        elif isinstance(metadataKeys, str):
            metadataKeys = [metadataKeys,]
        if maskMetadataKeys is not None:
            metadataKeys = [k for k in metadataKeys if k not in maskMetadataKeys]
        msg.debug('getMetadata will retrieve these keys: {0!s}'.format(metadataKeys))
        
        if flush is True:
            msg.debug('Flushing cached metadata values')
            self._resetMetadata()
            
        if populate is True:
            msg.debug('Checking metadata values')
            self._readMetadata(files, metadataKeys)
 
        metadata = {}
        for fname in files:
            metadata[fname] = {}
            for mdkey in metadataKeys:
                try:
                    metadata[fname][mdkey] = self._fileMetadata[fname][mdkey]
                except KeyError:
                    # This should not happen, unless we skipped populating
                    if populate:
                        msg.error('Did not find metadata key {0!s} for file {1!s} - setting to None'.format(mdkey, fname))
                    metadata[fname][mdkey] = None
        return metadata
    

◆ getnentries()

python.trfArgClasses.argFile.getnentries	(		self,
			fast = False )

inherited

Explicit getter, offering fast switch.

Definition at line 894 of file trfArgClasses.py.

    def getnentries(self, fast=False):
        totalEvents = 0
        for fname in self._value:
            events = self.getSingleMetadata(fname=fname, metadataKey='nentries', populate = not fast)
            if events is None:
                msg.debug('Got events=None for file {0} - returning None for this instance'.format(fname))
                return None
            if events == 'UNDEFINED':
                msg.debug('Got events=UNDEFINED for file {0} - returning UNDEFINED for this instance'.format(fname))
                return 'UNDEFINED'
            if not isinstance(events, int):
                msg.warning('Got unexpected events metadata for file {0}: {1!s} - returning None for this instance'.format(fname, events))
                return None
            totalEvents += events
        
        return totalEvents
                
    

◆ getSingleMetadata()

python.trfArgClasses.argFile.getSingleMetadata	(	self,
		fname,
		metadataKey,
		populate = True,
		flush = False )

inherited

Convenience function to extract a single metadata key for a single file.

Retrieves a single metadata item for a single file, returning it directly

Returns: Single metadata value

Parameters

fname	File to return metadata for
metadataKey	Key to return
populate	If missing key should be generated by calling the population subroutines
flush	If cached data should be flushed and the generator rerun

Definition at line 989 of file trfArgClasses.py.

    def getSingleMetadata(self, fname, metadataKey, populate = True, flush = False):
        if not (isinstance(fname, str) and isinstance(metadataKey, str)):
            raise trfExceptions.TransformInternalException(trfExit.nameToCode('TRF_INTERNAL'), 
                                                           'Illegal call to getSingleMetadata function: {0!s} {1!s}'.format(fname, metadataKey))
        md = self.getMetadata(files = fname, metadataKeys = metadataKey, populate = populate, flush = flush)
        return md[fname][metadataKey]
    
    

◆ io() [1/2]

python.trfArgClasses.argFile.io ( self )

inherited

Definition at line 807 of file trfArgClasses.py.

    def io(self):
        return (self._io)
 

◆ io() [2/2]

python.trfArgClasses.argFile.io	(		self,
			value )

inherited

Definition at line 811 of file trfArgClasses.py.

    def io(self, value):
        if value not in ('input', 'output', 'temporary'):
            raise trfExceptions.TransformArgException(trfExit.nameToCode('TRF_RUNTIME_ERROR'), 
                                                      'File arguments must be specified as input, output or temporary - got {0}'.format(value))
        self._io = value
    

◆ isCached()

python.trfArgClasses.argFile.isCached	(	self,
		files = None,
		metadataKeys = None )

inherited

Test if certain metadata elements are already cached.

Will test for a cached value for all files and all keys given, aborting as soon as it finds a single uncached value.

Parameters

files	Files to check (defaults to all files)
metadataKeys	Keys to check (defaults to all keys)

Returns: Boolean if all keys are cached for all files

Definition at line 1065 of file trfArgClasses.py.

    def isCached(self, files = None, metadataKeys = None):
        msg.debug('Testing for cached values for files {0} and keys {1}'.format(files, metadataKeys))
        if files is None:
            files = self._value
        elif isinstance(files, str):
            files = (files,)
        if metadataKeys is None:
            metadataKeys = list(self._metadataKeys)
        elif isinstance(metadataKeys, str):
            metadataKeys = (metadataKeys,)
            
        isCachedFlag = True
        for fname in files:
            for key in metadataKeys:
                if key not in self._fileMetadata[fname]:
                    isCachedFlag = False
                    break
            if isCachedFlag is False:
                break
            
        return isCachedFlag
    

◆ isRunarg()

python.trfArgClasses.argument.isRunarg ( self )

inherited

Return runarg status.

Definition at line 134 of file trfArgClasses.py.

    def isRunarg(self):
        return self._runarg
    

◆ mergeTargetSize() [1/2]

python.trfArgClasses.argFile.mergeTargetSize ( self )

inherited

mergeTargeSize value getter

Definition at line 613 of file trfArgClasses.py.

    def mergeTargetSize(self):
        return self._mergeTargetSize
 

◆ mergeTargetSize() [2/2]

python.trfArgClasses.argFile.mergeTargetSize	(		self,
			value )

inherited

mergeTargeSize value setter

Definition at line 618 of file trfArgClasses.py.

    def mergeTargetSize(self, value):
        if value is None:
            self._mergeTargetSize = 0
        else:
            self._mergeTargetSize = value
 

◆ metadata()

python.trfArgClasses.argFile.metadata ( self )

inherited

Returns the whole kit and kaboodle...

Note: Populates the whole metadata dictionary for this instance

Definition at line 884 of file trfArgClasses.py.

    def metadata(self):
        self.getMetadata()
        return self._fileMetadata
    

◆ multipleOK() [1/2]

python.trfArgClasses.argFile.multipleOK ( self )

inherited

multipleOK getter

Returns: Current value

Definition at line 603 of file trfArgClasses.py.

    def multipleOK(self):
        return self._multipleOK
    

◆ multipleOK() [2/2]

python.trfArgClasses.argFile.multipleOK	(		self,
			value )

inherited

multipleOK value setter

Definition at line 608 of file trfArgClasses.py.

    def multipleOK(self, value):
        self._multipleOK = value
        

◆ name() [1/2]

python.trfArgClasses.argFile.name ( self )

inherited

Name getter.

Reimplemented from python.trfArgClasses.argument.

Definition at line 851 of file trfArgClasses.py.

    def name(self):
        return self._name
    

◆ name() [2/2]

python.trfArgClasses.argFile.name	(		self,
			value )

inherited

Name setter.

Note: This property setter will also set the type and subtype of the argFile if they are not yet set. This means that for most arguments the type and subtype are automatically set correctly.

Reimplemented from python.trfArgClasses.argument.

Definition at line 859 of file trfArgClasses.py.

    def name(self, value):
        self._name = value
        m = re.match(r'(input|output|tmp.)([A-Za-z0-9_]+?)(File)?$', value)
        if m:
            msg.debug("ArgFile name setter matched this: {0}".format(m.groups()))
            if self._type is None:
                dtype = m.group(2).split('_', 1)[0]
                # But DRAW/DESD/DAOD are really just RAW, ESD, AOD in  format
                if re.match(r'D(RAW|ESD|AOD)', dtype):
                    dtype = dtype[1:]
                msg.debug("Autoset data type to {0}".format(dtype))
                self._type = dtype
            if self._subtype is None:
                msg.debug("Autoset data subtype to {0}".format(m.group(2)))
                self._subtype = m.group(2)
        else:
            msg.debug("ArgFile name setter did not match against '{0}'".format(value))
 

◆ nentries()

python.trfArgClasses.argFile.nentries ( self )

inherited

Return total number of events in all constituent files.

Definition at line 890 of file trfArgClasses.py.

    def nentries(self):
        return self.getnentries()
        

◆ originalName()

python.trfArgClasses.argFile.originalName	(		self,
			value )

inherited

Definition at line 830 of file trfArgClasses.py.

    def originalName(self, value):
        self._originalName = value
    

◆ orignalName()

python.trfArgClasses.argFile.orignalName ( self )

inherited

Definition at line 826 of file trfArgClasses.py.

    def orignalName(self):
        return self._originalName
    

◆ prodsysDescription()

python.trfArgClasses.argBZ2File.prodsysDescription ( self )

Reimplemented from python.trfArgClasses.argFile.

Reimplemented in python.trfArgClasses.argFTKIPFile.

Definition at line 1817 of file trfArgClasses.py.

    def prodsysDescription(self):
        desc=super(argBZ2File, self).prodsysDescription
        return desc  
 
 

◆ subtype() [1/2]

python.trfArgClasses.argFile.subtype ( self )

inherited

Definition at line 842 of file trfArgClasses.py.

    def subtype(self):
        return self._subtype
    

◆ subtype() [2/2]

python.trfArgClasses.argFile.subtype	(		self,
			value )

inherited

Definition at line 846 of file trfArgClasses.py.

    def subtype(self, value):
        self._subtype = value
 

◆ type() [1/2]

python.trfArgClasses.argFile.type ( self )

inherited

Definition at line 834 of file trfArgClasses.py.

    def type(self):
        return self._type
    

◆ type() [2/2]

python.trfArgClasses.argFile.type	(		self,
			value )

inherited

Definition at line 838 of file trfArgClasses.py.

    def type(self, value):
        self._type = value
 

◆ value() [1/2]

python.trfArgClasses.argFile.value ( self )

inherited

Argument value getter.

Returns: Current value

Reimplemented from python.trfArgClasses.argList.

Definition at line 591 of file trfArgClasses.py.

    def value(self):
        return self._value
 

◆ value() [2/2]

python.trfArgClasses.argFile.value	(		self,
			value )

inherited

Argument value setter.

Calls the valueSetter function with the standard options

Reimplemented from python.trfArgClasses.argList.

Definition at line 597 of file trfArgClasses.py.

    def value(self, value):
        self.valueSetter(value)
        

◆ valueSetter()

python.trfArgClasses.argFile.valueSetter	(		self,
			value )

inherited

Set the argFile value, but allow parameters here.

Note: Normally athena only takes a single value for an output file, but when AthenaMP runs it can produce multiple output files - this is allowed by setting allowMultiOutputs = True; The setter protects against the same file being added multiple times

Definition at line 645 of file trfArgClasses.py.

    def valueSetter(self, value):
        
        if isinstance(value, (list, tuple)):
            if len(value) > 0 and isinstance(value[0], dict): # Tier-0 style expanded argument with metadata
                self._value=[]
                for myfile in value:
                    try:
                        self._value.append(myfile['lfn'])
                        self._resetMetadata(files = [myfile['lfn']])
                    except KeyError:
                        raise trfExceptions.TransformArgException(trfExit.nameToCode('TRF_ARG_CONV_FAIL'),
                                                                  'Filename (key "lfn") not found in Tier-0 file dictionary: {0}'.format(myfile))
                    for k, v in myfile.items():
                        if k == 'guid':
                            self._setMetadata([myfile['lfn']], {'file_guid': v})
                        elif k == 'events':
                            self._setMetadata([myfile['lfn']], {'nentries': v})
                        elif k == 'checksum':
                            self._setMetadata([myfile['lfn']], {'checksum': v})
                        elif k == 'dsn':
                            if not self._dataset:
                                self.dataset = v
                            elif self.dataset != v:
                                raise trfExceptions.TransformArgException(trfExit.nameToCode('TRF_ARG_DATASET'),
                                                                          'Inconsistent dataset names in Tier-0 dictionary: {0} != {1}'.format(self.dataset, v))
            else:
                self._value = list(value)
                self._getDatasetFromFilename(reset = False)
                self._resetMetadata()
        elif value is None:
            self._value = []
            return
        else:
            try:
                if value.lower().startswith('lfn'):
                    # Resolve physical filename using pool file catalog.
                    from PyUtils.PoolFile import file_name
                    protocol, pfn = file_name(value)
                    self._value = [pfn]
                    self._getDatasetFromFilename(reset = False)
                    self._resetMetadata()
                else:
                    # Don't split output filename if it contains a list in square brackets
                    if self._io == 'output' and ('[' in value) and (']' in value):
                        self._value = [value]
                    else:
                        self._value = value.split(self._splitter)
                    self._getDatasetFromFilename(reset = False)
                    self._resetMetadata()
            except (AttributeError, TypeError):
                raise trfExceptions.TransformArgException(trfExit.nameToCode('TRF_ARG_CONV_FAIL'),
                                                          'Failed to convert %s to a list' % str(value))
 
        
        deDuplicatedValue = []
        for fname in self._value:
            if fname not in deDuplicatedValue:
                deDuplicatedValue.append(fname)
            else:
                msg.warning("Removing duplicated file {0} from file list".format(fname))
        if len(self._value) != len(deDuplicatedValue):
            self._value = deDuplicatedValue
            msg.warning('File list after duplicate removal: {0}'.format(self._value))
        
        # Find our URL type (if we actually have files!)
        # At the moment this is assumed to be the same for all files in this instance
        # although in principle one could mix different access methods in the one input file type
        if len(self._value) > 0:
            self._urlType = urlType(self._value[0])
        else:
            self._urlType = None
 
        
        if self._io == 'input':
            
            if self._urlType == 'posix':
                msg.debug('Found POSIX filesystem input - activating globbing')
                newValue = []
                for filename in self._value:
                    # Simple case
                    globbedFiles = glob.glob(filename)
                    if len(globbedFiles) == 0:          # No files globbed for this 'filename' argument.
                        raise trfExceptions.TransformArgException(trfExit.nameToCode('TRF_INPUT_FILE_ERROR'),
                        'Input file argument {0} globbed to NO input files - probably the file(s) are missing'.format(filename))
 
                    globbedFiles.sort()
                    newValue.extend(globbedFiles)
 
                self._value = newValue
                msg.debug ('File input is globbed to %s' % self._value)
 
            elif self._urlType == 'root':
                msg.debug('Found root filesystem input - activating globbing')
                newValue = []
                for filename in self._value:
                    if str(filename).startswith("root"):
                        msg.debug('Found input file name starting with "root," setting XRD_RUNFORKHANDLER=1, which enables fork handlers for xrootd in direct I/O')
                        os.environ["XRD_RUNFORKHANDLER"] = "1"
                    if str(filename).startswith("https") or str(filename).startswith("davs") or not(str(filename).endswith('/')) and '*' not in filename and '?' not in filename:
                        msg.debug('Seems that only one file was given: {0}'.format(filename))
                        newValue.extend(([filename]))
                    else:
                        # Hopefully this recognised wildcards...
                        path = filename
                        fileMask = ''
                        if '*' in filename or '?' in filename:
                            msg.debug('Split input into path for listdir() and a filemask to select available files.')
                            path = filename[0:filename.rfind('/')+1]
                            msg.debug('path: {0}'.format(path))
                            fileMask = filename[filename.rfind('/')+1:len(filename)]
                            msg.debug('Will select according to: {0}'.format(fileMask))
 
                        cmd = ['/afs/cern.ch/project/eos/installation/atlas/bin/eos.select' ]
                        if not os.access ('/afs/cern.ch/project/eos/installation/atlas/bin/eos.select', os.X_OK ):
                            raise trfExceptions.TransformArgException(trfExit.nameToCode('TRF_INPUT_FILE_ERROR'),
                            'No execute access to "eos.select" - could not glob EOS input files.')
 
                        cmd.extend(['ls'])
                        cmd.extend([path])
 
                        myFiles = []
                        try:
                            proc = subprocess.Popen(args = cmd,bufsize = 1, shell = False, stdout = subprocess.PIPE, stderr = subprocess.STDOUT)
                            rc = proc.wait()
                            output = proc.stdout.readlines()
                            if rc!=0:
                                raise trfExceptions.TransformArgException(trfExit.nameToCode('TRF_INPUT_FILE_ERROR'),
                                                            'EOS list command ("{0!s}") failed: rc {1}, output {2}'.format(cmd, rc, output))
                            msg.debug("eos returned: {0}".format(output))
                            for line in output:
                                if "root" in line:
                                    myFiles += [str(path)+str(line.rstrip('\n'))]
 
                            patt = re.compile(fileMask.replace('*','.*').replace('?','.'))
                            for srmFile in myFiles:
                                if fileMask != '':
                                    if(patt.search(srmFile)) is not None:
                                    #if fnmatch.fnmatch(srmFile, fileMask):
                                        msg.debug('match: %s',srmFile)
                                        newValue.extend(([srmFile]))
                                else:
                                    newValue.extend(([srmFile]))
                                
                            msg.debug('Selected files: %s', newValue)
                        except (AttributeError, TypeError, OSError):
                            raise trfExceptions.TransformArgException(trfExit.nameToCode('TRF_RUNTIME_ERROR'),
                                                                      'Failed to convert %s to a list' % str(value))
                if len(self._value) > 0 and len(newValue) == 0:
                    # Woops - no files!
                    raise trfExceptions.TransformArgException(trfExit.nameToCode('TRF_INPUT_FILE_ERROR'),
                                                              'Input file argument(s) {0!s} globbed to NO input files - ls command failed')
                self._value = newValue
                msg.debug ('File input is globbed to %s' % self._value)
        # Check if multiple outputs are ok for this object
        elif self._multipleOK is False and len(self._value) > 1:
            raise trfExceptions.TransformArgException(trfExit.nameToCode('TRF_OUTPUT_FILE_ERROR'), 
                                                      'Multiple file arguments are not supported for {0} (was given: {1}'.format(self, self._value))
            

Member Data Documentation

◆ _auxiliaryFile

python.trfArgClasses.argFile._auxiliaryFile = auxiliaryFile

protectedinherited

Definition at line 554 of file trfArgClasses.py.

◆ _dataset

python.trfArgClasses.argFile._dataset = None

protectedinherited

Definition at line 548 of file trfArgClasses.py.

◆ _exe

python.trfArgClasses.argFile._exe = executor

protectedinherited

Definition at line 560 of file trfArgClasses.py.

◆ _fileMetadata

python.trfArgClasses.argFile._fileMetadata = {}

protectedinherited

Definition at line 575 of file trfArgClasses.py.

◆ _guid

python.trfArgClasses.argFile._guid = guid

protectedinherited

Definition at line 552 of file trfArgClasses.py.

◆ _io

python.trfArgClasses.argFile._io = 'input':

protectedinherited

Input file globbing and expansion.

Definition at line 577 of file trfArgClasses.py.

◆ _mergeTargetSize

int python.trfArgClasses.argFile._mergeTargetSize = mergeTargetSize

protectedinherited

Definition at line 553 of file trfArgClasses.py.

◆ _metadataKeys

dict python.trfArgClasses.argFile._metadataKeys

protectedinherited

Initial value:

=  {'file_size': self._getSize,
                              'integrity': self._getIntegrity,
                              'file_guid': self._generateGUID,
                              '_exists': self._exists,
                              }

Note: Variable listing set of file metadata which corresponds to this class, Key is the metadata variable name, the value is the function to call to populate/refresh this metadata value. Function must take a single parameter, which is the list of files to get metadata for. It must return a metadata dictionary: {file1 : {key1: value1, key2: value2}, file2: ...} Keys which start with _ are for transform internal use and should not appear in jobReports

Definition at line 570 of file trfArgClasses.py.

◆ _multipleOK

bool python.trfArgClasses.argFile._multipleOK = True

protectedinherited

Definition at line 578 of file trfArgClasses.py.

◆ _name

python.trfArgClasses.argument._name = name

protectedinherited

Definition at line 111 of file trfArgClasses.py.

◆ _originalName

python.trfArgClasses.argFile._originalName = None

protectedinherited

Definition at line 555 of file trfArgClasses.py.

◆ _runarg

python.trfArgClasses.argument._runarg = runarg

protectedinherited

Definition at line 110 of file trfArgClasses.py.

◆ _splitter

python.trfArgClasses.argList._splitter = splitter

protectedinherited

Definition at line 357 of file trfArgClasses.py.

◆ _subtype

python.trfArgClasses.argFile._subtype = subtype

protectedinherited

Definition at line 551 of file trfArgClasses.py.

◆ _supressEmptyStrings

python.trfArgClasses.argList._supressEmptyStrings = supressEmptyStrings

protectedinherited

Definition at line 358 of file trfArgClasses.py.

◆ _type

python.trfArgClasses.argFile._type = type

protectedinherited

Definition at line 550 of file trfArgClasses.py.

◆ _urlType

str python.trfArgClasses.argFile._urlType = None

protectedinherited

Input file globbing and expansion.

Note: TODO: Non-posix URLs Problem is not so much the [] expansion, but the invisible .N attempt number One can only deal with this with a listdir() functionality N.B. Current transforms only do globbing on posix fs too (see trfutil.expandStringToList())

Definition at line 549 of file trfArgClasses.py.

◆ _value

python.trfArgClasses.argument._value = value

protectedinherited

Definition at line 130 of file trfArgClasses.py.

◆ dataset

python.trfArgClasses.argFile.dataset = v

inherited

Definition at line 666 of file trfArgClasses.py.

◆ io

python.trfArgClasses.argFile.io = io

inherited

Definition at line 558 of file trfArgClasses.py.

◆ type

python.trfArgClasses.argFile.type

inherited

Definition at line 1217 of file trfArgClasses.py.

The documentation for this class was generated from the following file:

trfArgClasses.py

Public Types

Public Member Functions

Public Attributes

Protected Member Functions

Protected Attributes

Detailed Description

Member Typedef Documentation

◆ result

Member Function Documentation

◆ __eq__()

◆ __gt__()

◆ __lt__()

◆ __nq__()

◆ __repr__()

◆ __str__()

◆ _exists()

◆ _generateGUID()

◆ _getDatasetFromFilename()

◆ _getIntegrity()

◆ _getSize()

◆ _mergeArgs()

◆ _readMetadata()

◆ _resetMetadata()

◆ _setMetadata()

◆ append()

◆ auxiliaryFile()

◆ dataset() [1/2]

◆ dataset() [2/2]

◆ executor()

◆ getMetadata()

◆ getnentries()

◆ getSingleMetadata()

◆ io() [1/2]

◆ io() [2/2]

◆ isCached()

◆ isRunarg()

◆ mergeTargetSize() [1/2]

◆ mergeTargetSize() [2/2]

◆ metadata()

◆ multipleOK() [1/2]

◆ multipleOK() [2/2]

◆ name() [1/2]

◆ name() [2/2]

◆ nentries()

◆ originalName()

◆ orignalName()

◆ prodsysDescription()

◆ subtype() [1/2]

◆ subtype() [2/2]

◆ type() [1/2]

◆ type() [2/2]

◆ value() [1/2]

◆ value() [2/2]

◆ valueSetter()

Member Data Documentation

◆ _auxiliaryFile

◆ _dataset

◆ _exe

◆ _fileMetadata

◆ _guid

◆ _io

◆ _mergeTargetSize

◆ _metadataKeys

◆ _multipleOK

◆ _name

◆ _originalName

◆ _runarg

◆ _splitter

◆ _subtype

◆ _supressEmptyStrings

◆ _type

◆ _urlType

◆ _value

◆ dataset

◆ io

◆ type

◆ eq()

◆ gt()

◆ lt()

◆ nq()

◆ repr()

◆ str()