xAOD::RDataSource Class Reference

Data source for xAOD input files. More...

#include <RDataSource.h>

Inheritance diagram for xAOD::RDataSource:

Collaboration diagram for xAOD::RDataSource:

Public Types
typedef std::vector< std::pair< ULong64_t, ULong64_t > >	EntryRanges_t
	Type describing the entry ranges of the input file(s) More...

Public Member Functions
	RDataSource (std::string_view fileNameGlob, std::string_view treeName="CollectionTree")
	Constructor with the file name pattern. More...
	RDataSource (const std::vector< std::string > &fileNames, std::string_view treeName="CollectionTree")
	Constructor with list of file names. More...
	~RDataSource ()
	Destructor. More...
void	setVerboseOutput (Bool_t value=kTRUE)
	Set whether verbose output should be printed (for debugging) More...
Bool_t	isVerboseOutput () const
	Check whether verbose output is set up to be printed. More...
Functions implemented from @c ROOT::RDF::RDataSource
virtual void	SetNSlots (unsigned int slots) override final
	Set the number of threads/slots that the data source should use. More...
virtual void	Initialize () override final
	Initialize the data source, before the start of the event loop. More...
virtual void	InitSlot (unsigned int slot, ULong64_t firstEntry) override final
	Initialize one of the slots/threads. More...
virtual void	FinalizeSlot (unsigned int slot) override final
	Close the input file reading in one of the slots/threads. More...
virtual void	Finalize () override final
	Finalize the data source, after the event loop. More...
virtual const std::vector< std::string > &	GetColumnNames () const override final
	Get the column/object names for the input file(s) More...
virtual bool	HasColumn (std::string_view name) const override final
	Check if the dataset has a certain column/object. More...
virtual std::string	GetTypeName (std::string_view column) const override final
	Get the type name of a given column/object. More...
virtual EntryRanges_t	GetEntryRanges () override final
	Get the entry ranges in the input file(s) More...
virtual bool	SetEntry (unsigned int slot, ULong64_t entry) override final
	Set which entry a give slot/thread should be processing. More...

Private Member Functions
virtual Record_t	GetColumnReadersImpl (std::string_view column, const std::type_info &typeInfo) override final
	Return the type-erased vector of pointers to pointers to column values. More...
void	readInputMetadata ()
	Fill the metadata variables. More...

Private Attributes
Arguments provided by the user
std::vector< std::string >	m_fileNames
	Files to read. More...
std::string	m_treeName
	Name of the event tree in the input files. More...
Bool_t	m_verboseOutput
	Whether verbose output should be printed or not. More...
Variables valid right after construction
std::vector< std::string >	m_columnNames
	Names of the columns/objects on the input. More...
std::unordered_map< std::string, std::string >	m_classNameMap
	The object name -> class name map. More...
Variables that become valid after calling @c Initialize()
EntryRanges_t	m_entryRanges
	Optimal entry ranges to split the processing into. More...
Variables that become valid after calling @c SetNSlots(...)
std::vector< std::unique_ptr< TChain > >	m_chains
	Chains used in the file I/O. More...
std::vector< std::unique_ptr< RDataSourceEvent > >	m_events
	Event objects performing the file I/O. More...
std::vector< std::unique_ptr< TStore > >	m_stores
	In-memory whiteboards used during the event loop. More...

Detailed Description

Data source for xAOD input files.

This data source can be used to allow ROOT::RDataFrame to process (D)xAOD inputs.

Author

Umesh Worlikar Umesh.nosp@m..Dha.nosp@m.rmaji.nosp@m..Wor.nosp@m.likar.nosp@m.@cer.nosp@m.n.ch

Attila Krasznahorkay Attil.nosp@m.a.Kr.nosp@m.aszna.nosp@m.hork.nosp@m.ay@ce.nosp@m.rn.c.nosp@m.h

Definition at line 35 of file RDataSource.h.

Member Typedef Documentation

EntryRanges_t

typedef std::vector< std::pair< ULong64_t, ULong64_t > > xAOD::RDataSource::EntryRanges_t

Type describing the entry ranges of the input file(s)

Definition at line 48 of file RDataSource.h.

Constructor & Destructor Documentation

RDataSource() [1/2]

xAOD::RDataSource::RDataSource	(	std::string_view	fileNameGlob,
		std::string_view	treeName = "CollectionTree"
	)

Constructor with the file name pattern.

Definition at line 105 of file RDataSource.cxx.

      : RDataSource( std::vector< std::string >( { fileNameGlob.data() } ),
                     treeName ) {
 
   }

RDataSource() [2/2]

xAOD::RDataSource::RDataSource	(	const std::vector< std::string > &	fileNames,
		std::string_view	treeName = "CollectionTree"
	)

Constructor with list of file names.

Definition at line 112 of file RDataSource.cxx.

      : m_fileNames( fileNames ), m_treeName( treeName ),
        m_verboseOutput( kFALSE ) {
 
      readInputMetadata();
   }

~RDataSource()

xAOD::RDataSource::~RDataSource

(

)

Destructor.

Definition at line 120 of file RDataSource.cxx.

                             {
 
      // I don't understand why, but ROOT really doesn't like it if the
      // chains are not the first to be deleted from memory. :-/
      m_chains.clear();
   }

Member Function Documentation

Finalize()

void xAOD::RDataSource::Finalize ( )

finaloverridevirtual

Finalize the data source, after the event loop.

Definition at line 308 of file RDataSource.cxx.

                              {
 
      // Simply print what's happening.
      PRINT_VERBOSE( "Finalize: Function called" );
 
      // Return gracefully.
      return;
   }

FinalizeSlot()

void xAOD::RDataSource::FinalizeSlot ( unsigned int  slot )

finaloverridevirtual

Close the input file reading in one of the slots/threads.

Definition at line 299 of file RDataSource.cxx.

                                                     {
 
      // Simply print what's happening.
      PRINT_VERBOSE( "FinalizeSlot: Called for slot " << slot );
 
      // Return gracefully.
      return;
   }

GetColumnNames()

const std::vector< std::string > & xAOD::RDataSource::GetColumnNames ( ) const

finaloverridevirtual

Get the column/object names for the input file(s)

Definition at line 318 of file RDataSource.cxx.

                                                                   {
 
      return m_columnNames;
   }

GetColumnReadersImpl()

RDataSource::Record_t xAOD::RDataSource::GetColumnReadersImpl ( std::string_view  column, const std::type_info &  typeInfo  )

finaloverrideprivatevirtual

Return the type-erased vector of pointers to pointers to column values.

Definition at line 397 of file RDataSource.cxx.

                                                                     {
 
      // Make sure that the column/object is known.
      if( ! HasColumn( column ) ) {
         ::Error( "xAOD::RDataSource::GetColumnReadersImpl",
                  XAOD_MESSAGE( "Column/object \"%s\" not available" ),
                  column.data() );
         throw std::runtime_error( "Column/object \"" + std::string( column ) +
                                   "\" not available" );
      }
      PRINT_VERBOSE( "GetColumnReadersImpl: Creating column readers for \""
                     << column << "/" << SG::normalizedTypeinfoName( typeInfo )
                     << "\"" );
 
      // Create the comlumn reader pointers.
      Record_t result( m_events.size() );
      for( size_t i = 0; i < m_events.size(); ++i ) {
         result[ i ] = m_events[ i ]->columnReader( column, typeInfo );
      }
      return result;
   }

GetEntryRanges()

RDataSource::EntryRanges_t xAOD::RDataSource::GetEntryRanges ( )

finaloverridevirtual

Get the entry ranges in the input file(s)

Definition at line 352 of file RDataSource.cxx.

                                                        {
 
      // When ROOT asks for the entry ranges, we have to tell it which ones
      // have not been processed yet. Since we process all entries right away
      // (SetEntry(...) does not have logic for not processing a requested
      // entry), the logic here is to empty out the m_entryRanges variable on
      // this call. So that on the next call an empty range would be returned.
      const EntryRanges_t dummy( std::move( m_entryRanges ) );
      return dummy;
   }

GetTypeName()

std::string xAOD::RDataSource::GetTypeName ( std::string_view  column ) const

finaloverridevirtual

Get the type name of a given column/object.

Definition at line 329 of file RDataSource.cxx.

                                                                   {
 
      // Make sure that the column/object is known.
      if( ! HasColumn( column ) ) {
         ::Error( "xAOD::RDataSource::GetTypeName",
                  XAOD_MESSAGE( "Column/object \"%s\" not available" ),
                  column.data() );
         throw std::runtime_error( "Column/object \"" + std::string( column ) +
                                   "\" not available" );
      }
 
      // Get the type.
      auto itr = m_classNameMap.find( column.data() );
      if( itr == m_classNameMap.end() ) {
         // Note that the fatal message will abort the entire job in all cases.
         ::Fatal( "xAOD::RDataSource::GetTypeName",
                  XAOD_MESSAGE( "Internal logic error found" ) );
      }
      PRINT_VERBOSE( "GetTypeName: Type name for column \"" << column
                     << "\" is: " << itr->second );
      return itr->second;
   }

HasColumn()

bool xAOD::RDataSource::HasColumn ( std::string_view  name ) const

finaloverridevirtual

Check if the dataset has a certain column/object.

Definition at line 323 of file RDataSource.cxx.

                                                          {
 
      return std::find( m_columnNames.begin(), m_columnNames.end(),
                        name ) != m_columnNames.end();
   }

Initialize()

void xAOD::RDataSource::Initialize ( )

finaloverridevirtual

Initialize the data source, before the start of the event loop.

Definition at line 192 of file RDataSource.cxx.

                                {
#endif
 
      // A sanity check.
      if( m_entryRanges.size() != 0 ) {
         ::Fatal( "xAOD::RDataSource::Initialize",
                  XAOD_MESSAGE( "Function called on an initialized object" ) );
      }
      PRINT_VERBOSE( "Initialize: Initializing the data source" );
 
      // Create a chain that will help determine the optimal entry ranges
      // to process.
      auto chain = ::makeChain( m_fileNames, m_treeName );
      TObjArray* filesInChain = chain->GetListOfFiles();
 
      // Loop over the input files of the chain.
      Long64_t fileOffset = 0;
      for( Int_t ifile = 0; ifile < filesInChain->GetEntries(); ++ifile ) {
 
         // Open the file directly.
         const char* fileName = filesInChain->At( ifile )->GetTitle();
         auto file = std::unique_ptr< TFile >( TFile::Open( fileName,
                                                            "READ" ) );
         if( ( ! file ) || file->IsZombie() ) {
            ::Error( "xAOD::RDataSource::Initialize",
                     XAOD_MESSAGE( "Failed to open file: %s" ), fileName );
            throw std::runtime_error( "Failed to open file: " +
                                      std::string( fileName ) );
         }
 
         // Access the event tree inside of it.
         TTree* tree =
            dynamic_cast< TTree* >( file->Get( m_treeName.c_str() ) );
         if( ! tree ) {
            // A file with no event tree is not necessarily a problem. It could
            // just be a file that has no events left in it after all
            // selections.
            continue;
         }
 
         // Extract the ideal entry ranges from the file.
         const Long64_t entries = tree->GetEntries();
         TTree::TClusterIterator clusterIter( tree->GetClusterIterator( 0 ) );
         Long64_t clusterStart = 0;
         while( ( clusterStart = clusterIter() ) < entries ) {
            m_entryRanges.emplace_back( fileOffset + clusterStart,
                                        fileOffset +
                                        clusterIter.GetNextEntry() );
         }
 
         // Increment the file offset value.
         fileOffset += entries;
      }
      PRINT_VERBOSE( "Initialize: Created entry ranges: " << m_entryRanges );
 
      // Return gracefully.
      return;
   }

InitSlot()

void xAOD::RDataSource::InitSlot ( unsigned int  slot, ULong64_t  firstEntry  )

finaloverridevirtual

Initialize one of the slots/threads.

Definition at line 251 of file RDataSource.cxx.

                                                                       {
 
      // A sanity check.
      if( m_events.size() <= slot ) {
         ::Error( "xAOD::RDataSource::InitSlot",
                  XAOD_MESSAGE( "Invalid slot (%u) received" ), slot );
         throw std::runtime_error( "Invalid slot received" );
      }
 
      // Load the first entry for it.
      if( m_events[ slot ]->getEntry( firstEntry ) < 0 ) {
         ::Error( "xAOD::RDataSource::InitSlot",
                  XAOD_MESSAGE( "Failed to load entry %lld for slot %u" ),
                  firstEntry, slot );
         throw std::runtime_error( "Failed to load entry for slot" );
      }
      PRINT_VERBOSE( "InitSlot: Retrieved entry " << firstEntry << " for slot "
                     << slot );
 
      // Activate and clear the store.
      m_stores[ slot ]->setActive();
      m_stores[ slot ]->clear();
      PRINT_VERBOSE( "InitSlot: Activated and cleared transient store for slot "
                     << slot );
 
      // Return gracefully.
      return;
   }

isVerboseOutput()

Bool_t xAOD::RDataSource::isVerboseOutput

(

)

const

Check whether verbose output is set up to be printed.

Definition at line 391 of file RDataSource.cxx.

                                             {
 
      return m_verboseOutput;
   }

readInputMetadata()

void xAOD::RDataSource::readInputMetadata ( )

private

Fill the metadata variables.

Definition at line 420 of file RDataSource.cxx.

                                       {
 
      // Create a temporary event object.
      auto chain = ::makeChain( m_fileNames, m_treeName );
      RDataSourceEvent event;
      if( ! event.readFrom( chain.get() ).isSuccess() ) {
         ::Error( "xAOD::RDataSource::readInputMetadata",
                  XAOD_MESSAGE( "Failed to connect to the input chain" ) );
         throw std::runtime_error( "Failed to connect to the input chain" );
      }
 
      // Load the first event of the input, if one is available.
      if( event.getEntries() > 0 ) {
         if( event.getEntry( 0 ) < 0 ) {
            ::Error( "xAOD::RDataSource::readInputMetadata",
                     "Couldn't load the first event of the input" );
            throw std::runtime_error( "Couldn't load the first event of the "
                                      "input" );
         }
      }
 
      // Fill the column and type name variables.
      m_columnNames.clear(); m_classNameMap.clear();
      auto names = event.columnAndTypeNames();
      m_columnNames.reserve( names.size() );
      m_classNameMap.reserve( names.size() );
      for( const auto& pair : names ) {
         m_columnNames.push_back( pair.first );
         m_classNameMap[ pair.first ] = pair.second;
      }
      PRINT_VERBOSE( "readInputMetadata: m_columnNames = " << m_columnNames );
      PRINT_VERBOSE( "readInputMetadata: m_classNameMap = " << m_classNameMap );
 
      // ROOT memory management is weird... We must delete the chain first,
      // before the TEvent object on top of it would be deleted...
      chain.reset();
 
      // Return gracefully.
      return;
   }

SetEntry()

bool xAOD::RDataSource::SetEntry ( unsigned int  slot, ULong64_t  entry  )

finaloverridevirtual

Set which entry a give slot/thread should be processing.

Definition at line 363 of file RDataSource.cxx.

                                                                  {
 
      // A sanity check.
      if( m_events.size() <= slot ) {
         ::Error( "xAOD::RDataSource::SetEntry",
                  XAOD_MESSAGE( "Invalid slot (%u) received" ), slot );
         throw std::runtime_error( "Invalid slot received" );
      }
      PRINT_VERBOSE( "SetEntry: Called for slot " << slot << " and entry "
                     << entry );
 
      // Switch to the requested entry.
      m_events[ slot ]->updateObjectsForEntry( entry );
 
      // Activate and clear the store.
      m_stores[ slot ]->setActive();
      m_stores[ slot ]->clear();
 
      // The entry is always processed.
      return true;
   }

SetNSlots()

void xAOD::RDataSource::SetNSlots ( unsigned int  slots )

finaloverridevirtual

Set the number of threads/slots that the data source should use.

Definition at line 127 of file RDataSource.cxx.

                                                   {
 
      // Some sanity checks.
      if( slots == 0 ) {
         ::Error( "xAOD::RDataSource::SetNSlots",
                  XAOD_MESSAGE( "Zero slots requested" ) );
         throw std::invalid_argument( "Zero slots requested" );
      }
      if( m_events.size() != 0 ) {
         ::Error( "xAOD::RDataSource::SetNSlots",
                  XAOD_MESSAGE( "Function called multiple times" ) );
         throw std::runtime_error( "Function called multiple times" );
      }
 
      // Reserve the correct number of elements.
      m_chains.reserve( slots );
      m_events.reserve( slots );
      m_stores.reserve( slots );
      PRINT_VERBOSE( "SetNSlots: Reserved " << slots
                     << " slots for the chains, events and stores" );
 
      // Create the event objects already at this point.
      for( unsigned int i = 0; i < slots; ++i ) {
 
         // Set up the chain, event and store.
         m_chains.push_back( ::makeChain( m_fileNames, m_treeName ) );
         m_events.push_back( std::make_unique< RDataSourceEvent >() );
         m_stores.push_back( std::make_unique< TStore >() );
         TChain* chain = m_chains.back().get();
         RDataSourceEvent* event = m_events.back().get();
 
         // Initialize the event object.
         if( ! event->readFrom( chain ).isSuccess() ) {
            ::Error( "xAOD::RDataSource::SetNSlots",
                     XAOD_MESSAGE( "Failed to set up xAOD::RDataSourceEvent "
                                   "for slot %u" ), i );
            throw std::runtime_error( "Failed to set up "
                                      "xAOD::RDataSourceEvent" );
         }
 
         // Load entry 0 for it. Notice that this is a waste of CPU and I/O
         // on the surface. But it's not... This triggers the initialization of
         // the files/trees used by these chains. Which happens much more
         // quickly in a serial way in a single thread than in multiple threads
         // at the same time. To be followed up with the ROOT developers...
         if( event->getEntry( 0 ) < 0 ) {
            ::Error( "xAOD::RDataSource::SetNSlots",
                     XAOD_MESSAGE( "Failed to load entry 0 for slot %u" ), i );
            throw std::runtime_error( "Failed to load entry for slot" );
         }
         PRINT_VERBOSE( "SetNSlots: Initialized objects for slot " << i );
      }
 
#if ROOT_VERSION_CODE >= ROOT_VERSION(6,35,99)
      // Pass on to the base class.
      ROOT::RDF::RDataSource::SetNSlots( slots );
#endif
 
      // Return gracefully.
      return;
   }

setVerboseOutput()

void xAOD::RDataSource::setVerboseOutput

(

Bool_t

value = kTRUE

)

Set whether verbose output should be printed (for debugging)

Definition at line 385 of file RDataSource.cxx.

                                                    {
 
      m_verboseOutput = value;
      return;
   }

Member Data Documentation

m_chains

std::vector< std::unique_ptr< TChain > > xAOD::RDataSource::m_chains

private

Chains used in the file I/O.

Definition at line 141 of file RDataSource.h.

m_classNameMap

std::unordered_map< std::string, std::string > xAOD::RDataSource::m_classNameMap

private

The object name -> class name map.

Definition at line 125 of file RDataSource.h.

m_columnNames

std::vector< std::string > xAOD::RDataSource::m_columnNames

private

Names of the columns/objects on the input.

Definition at line 123 of file RDataSource.h.

m_entryRanges

EntryRanges_t xAOD::RDataSource::m_entryRanges

private

Optimal entry ranges to split the processing into.

Definition at line 133 of file RDataSource.h.

m_events

std::vector< std::unique_ptr< RDataSourceEvent > > xAOD::RDataSource::m_events

private

Event objects performing the file I/O.

Definition at line 143 of file RDataSource.h.

m_fileNames

std::vector< std::string > xAOD::RDataSource::m_fileNames

private

Files to read.

Definition at line 111 of file RDataSource.h.

m_stores

std::vector< std::unique_ptr< TStore > > xAOD::RDataSource::m_stores

private

In-memory whiteboards used during the event loop.

Definition at line 145 of file RDataSource.h.

m_treeName

std::string xAOD::RDataSource::m_treeName

private

Name of the event tree in the input files.

Definition at line 113 of file RDataSource.h.

m_verboseOutput

Bool_t xAOD::RDataSource::m_verboseOutput

private

Whether verbose output should be printed or not.

Definition at line 115 of file RDataSource.h.

---

The documentation for this class was generated from the following files:

• RDataSource.h
• RDataSource.cxx