python.trfReports.trfFileReport Class Reference

Class to contain metadata for file types. More...

Inheritance diagram for python.trfReports.trfFileReport:

Collaboration diagram for python.trfReports.trfFileReport:

Public Member Functions
def	__init__ (self, fileArg)
	Constructor. More...
def	python (self, fast=False, type='full')
	Get a python representation of file report @detail Returns the python representation of this file report which is a list of dictionaries (a la Tier 0) More...
def	singleFilePython (self, filename, fast=False, type='full', basename=True)
	Return unique metadata for a single file in an argFile class. More...
def	classicEltreeList (self, fast=False)
	Return an element tree representation of the classic file report @detail Returns a list of eltree entities giving the classic metadata of a files for this argument. More...
def	classicSingleEltree (self, filename, fast=False)
	Return file metadata in 'classic' POOLFILECATALOG format. More...
def	classicPython (self, fast=False)
	Return file metadata in classic Tier 0 python style. More...
def	classicSinglePython (self, filename, fast=False)
	Return single file metadata in classic Tier 0 python style. More...

Private Attributes
	_fileArg

Static Private Attributes
	_internalToClassicMap
	_internalToGpickleMap
	_internalToGpickleMoreMap

Detailed Description

Class to contain metadata for file types.

Definition at line 393 of file trfReports.py.

Constructor & Destructor Documentation

__init__()

def python.trfReports.trfFileReport.__init__	(		self,
			fileArg
	)

Constructor.

Parameters

Definition at line 416 of file trfReports.py.

    def __init__(self, fileArg):
        self._fileArg = fileArg
 

Member Function Documentation

classicEltreeList()

def python.trfReports.trfFileReport.classicEltreeList	(		self,
			fast = False
	)

Return an element tree representation of the classic file report @detail Returns a list of eltree entities giving the classic metadata of a files for this argument.

This is a list so that it can be attached as a series of nodes into the final PFC description of the job

Returns

List of ElementTree.Element objects

Definition at line 501 of file trfReports.py.

    def classicEltreeList(self, fast = False):
        treeList = []
        for fname in self._fileArg.value:
            treeList.append(self.classicSingleEltree(fname, fast = fast))
 
        return treeList
 
 

classicPython()

def python.trfReports.trfFileReport.classicPython	(		self,
			fast = False
	)

Return file metadata in classic Tier 0 python style.

Parameters

fast	Prevent external metadata calls if true

Returns

List of file metadata dictionaries

Definition at line 543 of file trfReports.py.

    def classicPython(self, fast = False):
        fileList = []
        for fname in self._fileArg.value:
            fileList.append(self.classicSinglePython(fname, fast = fast))
        return fileList
 

classicSingleEltree()

def python.trfReports.trfFileReport.classicSingleEltree	(		self,
			filename,
			fast = False
	)

Return file metadata in 'classic' POOLFILECATALOG format.

Parameters

filename	Filename to generate Element object for
fast	Prevent external metadata calls if true

Definition at line 512 of file trfReports.py.

    def classicSingleEltree(self, filename, fast = False):
        if filename not in self._fileArg.value:
            raise trfExceptions.TransformReportException(trfExit.nameToCode('TRF_INTERNAL_REPORT_ERROR'),
                                                         'Unknown file ({0}) in the file report for {1}'.format(filename, self._fileArg))
        tree = ElementTree.Element('File', ID = str(self._fileArg.getSingleMetadata(fname = filename, metadataKey = 'file_guid', populate = not fast)))
        for myKey, classicKey in self._internalToClassicMap.items():
            # beam_type is tricky - we return only the first list value,
            # (but remember, protect against funny stuff!)
            if myKey == 'beam_type':
                beamType = self._fileArg.getSingleMetadata(fname = filename, metadataKey = myKey, populate = not fast)
                if isinstance(beamType, list):
                    if len(beamType) == 0:
                        ElementTree.SubElement(tree, 'metadata', att_name = classicKey, att_value = '')
                    else:
                        ElementTree.SubElement(tree, 'metadata', att_name = classicKey, att_value = str(beamType[0]))
                else:
                    # This is really not normal, but best we can do is str conversion
                    ElementTree.SubElement(tree, 'metadata', att_name = classicKey, att_value = str(beamType))
            else:
                ElementTree.SubElement(tree, 'metadata', att_name = classicKey,
                                       att_value = str(self._fileArg.getSingleMetadata(fname = filename, metadataKey = myKey, populate = not fast)))
        # Now add the metadata which is stored at the whole argument level
        ElementTree.SubElement(tree, 'metadata', att_name = 'fileType', att_value = str(self._fileArg.type))
        if self._fileArg.dataset is not None:
            ElementTree.SubElement(tree, 'metadata', att_name = 'dataset', att_value = self._fileArg.dataset)
 
        return tree
 

classicSinglePython()

def python.trfReports.trfFileReport.classicSinglePython	(		self,
			filename,
			fast = False
	)

Return single file metadata in classic Tier 0 python style.

Parameters

fast	Prevent external metadata calls if true

Definition at line 552 of file trfReports.py.

    def classicSinglePython(self, filename, fast = False):
        if filename not in self._fileArg.value:
            raise trfExceptions.TransformReportException(trfExit.nameToCode('TRF_INTERNAL_REPORT_ERROR'),
                                                         'Unknown file ({0}) in the file report for {1}'.format(filename, self._fileArg))
        # Direct population of some keys
        fileDict = {'lfn' : filename,
                    'dataset' : self._fileArg.dataset,
                    }
        # Fill in the mapped 'primary' keys
        for myKey, classicKey in self._internalToGpickleMap.items():
            fileDict[classicKey] = self._fileArg.getSingleMetadata(fname = filename, metadataKey = myKey, populate = not fast)
            if classicKey == 'checkSum' and fileDict[classicKey] == 'UNDEFINED':
                # Old style is that we give back None when we don't know
                fileDict[classicKey] = None
            elif fileDict[classicKey] == 'UNDEFINED':
                # Suppress things we don't generally expect to know
                del fileDict[classicKey]
        # Base 'more' stuff which is known by the argFile itself
        fileDict['more'] = {'metadata' : {'fileType' : self._fileArg.type}}
        for myKey, classicKey in self._internalToGpickleMoreMap.items():
            value = self._fileArg.getSingleMetadata(fname = filename, metadataKey = myKey, populate = not fast)
            if value != 'UNDEFINED':
                fileDict['more']['metadata'][classicKey] = value
 
        return fileDict
 
 

python()

def python.trfReports.trfFileReport.python	(		self,
			fast = False,
			type = 'full'
	)

Get a python representation of file report @detail Returns the python representation of this file report which is a list of dictionaries (a la Tier 0)

Parameters

type	Report type (e.g., None = normal or fast)
base	How extensive to make the report: name or full

Definition at line 424 of file trfReports.py.

    def python(self, fast = False, type = 'full'):
        # First entity contains shared properties - same for all files in this argFile
        if type == 'name':
            fileArgProps = {'dataset': self._fileArg.dataset,
                            'nentries': self._fileArg.getnentries(fast),
                            'subFiles' : []}
        elif type == 'full':
            fileArgProps = {'dataset' : self._fileArg.dataset,
                            'type' : self._fileArg.type,
                            'subFiles' : [],
                            'argName' : self._fileArg.name,
                            }
        else:
            raise trfExceptions.TransformReportException(trfExit.nameToCode('TRF_INTERNAL_REPORT_ERROR'),
                                                         'Unknown file report type ({0}) in the file report for {1}'.format(type, self._fileArg))
 
        
        uniqueBasenames = set([ os.path.basename(fname) for fname in self._fileArg.value ])
        uniqueDirectories = set([ os.path.dirname(os.path.relpath(os.path.normpath(fname))) for fname in self._fileArg.value ])
        if len(uniqueBasenames) != len(self._fileArg.value):
            msg.info('Detected two files with the same basename in a file argument - report for file {0} will be produced with the path as a key'.format(self._fileArg))
            basenameReport = False
        elif len(uniqueDirectories) > 1:
            msg.warning('Detected output files in different directories - report for file {0} will be produced with the path as a key'.format(self._fileArg))
            basenameReport = False
        else:
            basenameReport = True
        suppressed = []
        for fname in self._fileArg.value:
            subFile = None
            if basenameReport:
                subFile = self.singleFilePython(fname, fast = fast, type = type)
            else:
                subFile = self.singleFilePython(fname, fast = fast, type = type, basename = False)
            if subFile is not None:
                # if nentries == 0 for DRAW, suppress subfile from report
                if 'nentries' in subFile and subFile['nentries'] == 0 and isinstance(self._fileArg, trfArgClasses.argBSFile):
                    msg.info('Suppressing file {0}, nentries is 0'.format(subFile['name']))
                    suppressed.append(subFile['name'])
                else:
                    fileArgProps['subFiles'].append(subFile)
 
        return fileArgProps
 

singleFilePython()

def python.trfReports.trfFileReport.singleFilePython	(		self,
			filename,
			fast = False,
			type = 'full',
			basename = True
	)

Return unique metadata for a single file in an argFile class.

Parameters

filename	File for which report should be generated
type	Type of report to produce
fast	Generate a fast report, i.e., no external matadata calls

Returns

Python report dictionary

Definition at line 475 of file trfReports.py.

    def singleFilePython(self, filename, fast = False, type = 'full', basename = True):
        if filename not in self._fileArg.value:
            raise trfExceptions.TransformReportException(trfExit.nameToCode('TRF_INTERNAL_REPORT_ERROR'),
                                                         'Unknown file ({0}) in the file report for {1}'.format(filename, self._fileArg))
        if basename:
            entry = {'name': os.path.basename(filename)}
        else:
            entry = {'name': os.path.relpath(os.path.normpath(filename))}
        if type == 'name':
            # For 'name' we return only the GUID
            entry.update(self._fileArg.getMetadata(files = filename, populate = not fast, metadataKeys = ['file_guid'])[filename])
        elif type == 'full':
            # Suppress io because it's the key at a higher level and _exists because it's internal
            entry.update(self._fileArg.getMetadata(files = filename, populate = not fast, maskMetadataKeys = ['io', '_exists', 'integrity', 'file_type'])[filename])
        else:
            raise trfExceptions.TransformReportException(trfExit.nameToCode('TRF_INTERNAL_REPORT_ERROR'),
                                                         'Unknown file report type ({0}) in the file report for {1}'.format(type, self._fileArg))
 
        return entry
 
 

Member Data Documentation

_fileArg

python.trfReports.trfFileReport._fileArg

private

Definition at line 417 of file trfReports.py.

_internalToClassicMap

python.trfReports.trfFileReport._internalToClassicMap

staticprivate

Definition at line 396 of file trfReports.py.

_internalToGpickleMap

python.trfReports.trfFileReport._internalToGpickleMap

staticprivate

Definition at line 403 of file trfReports.py.

_internalToGpickleMoreMap

python.trfReports.trfFileReport._internalToGpickleMoreMap

staticprivate

Definition at line 409 of file trfReports.py.

---

The documentation for this class was generated from the following file:

• trfReports.py