xAOD::TFileAccessTracer Class Reference

Helper class keeping track of the files that got accessed. More...

#include <TFileAccessTracer.h>

Collaboration diagram for xAOD::TFileAccessTracer:

Classes
class	AccessedFile
	Helper class storing information about the accessed files. More...

Public Member Functions
	TFileAccessTracer ()
	Default constructor. More...
	~TFileAccessTracer ()
	Destructor. More...
void	add (const ::TFile &file)
	Add information about a new file that got accessed. More...
const std::string &	serverAddress () const
	The address of the server that information is sent to. More...
void	setServerAddress (const std::string &addr)
	Set the address of the server that information is sent to. More...
::Double_t	monitoredFraction () const
	Fraction of jobs that should send monitoring information. More...
void	setMonitoredFraction (::Double_t value)
	Set the fraction of jobs that should send monitoring information. More...

Static Public Member Functions
static void	enableDataSubmission (::Bool_t value)
	Function for turning data submission on/off. More...

Private Attributes
std::set< AccessedFile >	m_accessedFiles
	List of all the input files that were accessed in the job. More...
std::string	m_serverAddress
	Address of the server to send monitoring information to. More...
::TInetAddress	m_serverInetAddress
	Technical address of the server. More...
::Double_t	m_monitoredFraction
	Fraction of jobs that should send monitoring information. More...
ReadStats *	m_readStats
	Object describing the job's xAOD access statistics. More...
std::mutex	m_mutex
	Mutex for modifying the object. More...

Detailed Description

Helper class keeping track of the files that got accessed.

This class helps in keeping track of which files get accessed during a job. To be able to report about them to the DDM (Rucio) system in a way that doesn't rely on the grid middleware. (So backdoor access to files stored at data centres becomes visible to the DDM system.)

The class can also send information about which branches/variables got accessed in the job. This it simply gets from the xAODCore code.

xAOD::TEvent uses this class to send information about xAOD access to the central Rucio monitoring infrastructure. This default behaviour can be influenced in two ways:

• Use the TFileAccessTrancer::enableDataSubmission function to disable data submission from a particular application;
• Set the environment variable "XAOD_ACCESSTRACER_FRACTION" to a floating point value, which is the fraction of jobs that should be monitored within that session. This is mostly for grid purposes...

Author

Attila Krasznahorkay Attil.nosp@m.a.Kr.nosp@m.aszna.nosp@m.hork.nosp@m.ay@ce.nosp@m.rn.c.nosp@m.h

Definition at line 46 of file TFileAccessTracer.h.

Constructor & Destructor Documentation

TFileAccessTracer()

xAOD::TFileAccessTracer::TFileAccessTracer

(

)

Default constructor.

Definition at line 30 of file TFileAccessTracer.cxx.

   : m_accessedFiles(),
     m_serverAddress( "http://rucio-lb-prod.cern.ch:18762/traces/" ),
     m_serverInetAddress(),
     m_monitoredFraction( 1.0 ),
     m_readStats( &( IOStats::instance().stats() ) ) {
 
      // Construct the "technical address" of the host:
      const ::TUrl url( m_serverAddress.c_str() );
      m_serverInetAddress = gSystem->GetHostByName( url.GetHost() );
   }

~TFileAccessTracer()

xAOD::TFileAccessTracer::~TFileAccessTracer

(

)

Destructor.

The destructor of the class is the one doing most of the heavy lifting.

If the $SEND_XAOD_FILE_ACCESS_STAT environment variable is set when the object gets deleted, it posts all the information it collected, to the address defined by SERVER_ADDRESS. By constructing an HTTP message from scratch.

Definition at line 48 of file TFileAccessTracer.cxx.

                                         {
 
      // If the user turned off the data submission, then stop already here...
      if( ! s_enableDataSumbission ) {
         return;
      }
 
      // Decide what monitoring fraction to take. To make it possible for Panda
      // to override it with larger/smaller values if needed.
      Double_t monitoredFraction = m_monitoredFraction;
      const char* fractionString =
         gSystem->Getenv( "XAOD_ACCESSTRACER_FRACTION" );
      if( fractionString ) {
         char* endptr = 0;
         const Double_t fraction = strtod( fractionString, &endptr );
         if( endptr != fractionString ) {
            monitoredFraction = fraction;
         }
      }
 
      // Decide randomly whether to send the monitoring data or not. While
      // TRandom is not good enough for statistical purposes, it's fast, and is
      // perfectly good to make this decision...
      ::TRandom rng( ::TTimeStamp().GetNanoSec() );
      if( rng.Rndm() > monitoredFraction ) {
         return;
      }
 
      // Open a socket to the server:
      const ::TUrl url( m_serverAddress.c_str() );
      TSocket socket;
      if( ! socket.connect( m_serverInetAddress, url.GetPort() ).isSuccess() ) {
         // Just exit silently. If we can't send the info, we can't send the
         // info. It's not a problem.
         return;
      }
 
      // Let the user know what's happening:
      ::Info( "xAOD::TFileAccessTracer",
              "Sending file access statistics to %s", m_serverAddress.c_str() );
 
      // Start constructing the header of the message to send to the server:
      ::TString hdr = "POST /";
      hdr += url.GetFile();
      hdr += " HTTP/1.0";
      hdr += "\r\n";
      hdr += "From: ";
      hdr += gSystem->HostName();
      hdr += "\r\n";
      hdr += "User-Agent: xAODRootAccess\r\n";
      hdr += "Content-Type: application/json\r\n";
      hdr += "Content-Length: ";
 
      //
      // Now construct the message payload:
      //
      ::TString pld = "{";
 
      //
      // Collect the names of all the accessed files:
      //
      pld += "\"accessedFiles\": [";
      bool first = true;
      for( auto& info : m_accessedFiles ) {
         if( ! first ) {
            pld += ", ";
         }
         pld += "\"";
         pld += info.fullFilePath();
         pld += "\"";
         first = false;
      }
      pld += "], ";
      //
      // Collect the names of all the containers that were accessed:
      //
      pld += "\"accessedContainers\": [";
      first = true;
      for( const auto& bs : m_readStats->containers() ) {
         if( ! bs.second.readEntries() ) {
            continue;
         }
         if( ! first ) {
            pld += ", ";
         }
         pld += "{\"";
         pld += bs.second.GetName();
         pld += "\": ";
         pld += bs.second.readEntries();
         pld += "}";
         first = false;
      }
      pld += "], ";
      //
      // Collect the names of all the branches that were accessed:
      //
      pld += "\"accessedBranches\": [";
      first = true;
      for( const auto& branch : m_readStats->branches() ) {
         for( const xAOD::BranchStats* bs : branch.second ) {
            if( ( ! bs ) || ( ! bs->readEntries() ) ) {
               continue;
            }
            if( ! first ) {
               pld += ", ";
            }
            pld += "{\"";
            pld += bs->GetName();
            pld += "\": ";
            pld += bs->readEntries();
            pld += "}";
            first = false;
         }
      }
      pld += "]";
      //
      // Collect some possible Panda information in case the job is running
      // on the grid:
      //
      const char* pandaID = gSystem->Getenv( "PandaID" );
      if( pandaID ) {
         pld += ", \"PandaID\": ";
         pld += pandaID;
         pld += "";
      }
      const char* taskID = gSystem->Getenv( "PanDA_TaskID" );
      if( taskID ) {
         pld += ", \"PanDA_TaskID\": ";
         pld += taskID;
         pld += "";
      }
      //
      // Add some simple information about the host:
      //
      pld += ", \"ROOT_RELEASE\": \"";
      pld += ROOT_RELEASE;
      pld += "\"";
      pld += ", \"ReportRate\": ";
      pld += monitoredFraction;
      //
      // Add some information about the file access pattern:
      //
      pld += ", \"ReadCalls\": ";
      pld += m_readStats->fileReads();
      pld += ", \"ReadSize\": ";
      pld += ( m_readStats->fileReads() != 0 ?
               m_readStats->bytesRead() / m_readStats->fileReads() : 0 );
      pld += ", \"CacheSize\": ";
      pld += m_readStats->cacheSize();
      pld += "}";
 
      // Now finish constructing the header, and merge the two into a single
      // message:
      hdr += TString::Format( "%i", pld.Length() );
      hdr += "\r\n\r\n";
      const ::TString msg = hdr + pld;
 
      // A debug message for the time being:
      /*
      Info( "xAOD::TFileAccessTracer::~TFileAccessTracer",
            "Sending message:\n\n%s", msg.Data() );
       */
 
      // Finally, send the message:
      if( ! socket.send( msg ).isSuccess() ) {
         // Don't be vocal about the issue...
         return;
      }
   }

Member Function Documentation

add()

void xAOD::TFileAccessTracer::add

(

const ::TFile &

file

)

Add information about a new file that got accessed.

This function is called by TEvent to record which files were read from during the job.

Parameters

file	The file object that is being read from

Definition at line 223 of file TFileAccessTracer.cxx.

                                                  {
 
      // Protect this call:
      std::lock_guard< std::mutex > lock( m_mutex );
 
      // Remember this file:
      m_accessedFiles.insert(
         AccessedFile{ gSystem->DirName( file.GetName() ),
                       gSystem->BaseName( file.GetName() ) } );
 
      // Return gracefully:
      return;
   }

enableDataSubmission()

void xAOD::TFileAccessTracer::enableDataSubmission ( ::Bool_t  value )

static

Function for turning data submission on/off.

This function can be used by concerned users to turn off data collection and submission for their jobs completely.

Parameters

value

Setting whether data submission should be enabled or not

Definition at line 281 of file TFileAccessTracer.cxx.

                                                              {
 
      s_enableDataSumbission = value;
      return;
   }

monitoredFraction()

Double_t xAOD::TFileAccessTracer::monitoredFraction

(

)

const

Fraction of jobs that should send monitoring information.

Definition at line 259 of file TFileAccessTracer.cxx.

                                                       {
 
      // Protect this call:
      std::lock_guard< std::mutex > lock( m_mutex );
 
      return m_monitoredFraction;
   }

serverAddress()

const std::string & xAOD::TFileAccessTracer::serverAddress

(

)

const

The address of the server that information is sent to.

Definition at line 237 of file TFileAccessTracer.cxx.

                                                           {
 
      // Protect this call:
      std::lock_guard< std::mutex > lock( m_mutex );
 
      return m_serverAddress;
   }

setMonitoredFraction()

void xAOD::TFileAccessTracer::setMonitoredFraction

(

::Double_t

value

)

Set the fraction of jobs that should send monitoring information.

Definition at line 267 of file TFileAccessTracer.cxx.

                                                                {
 
      // Protect this call:
      std::lock_guard< std::mutex > lock( m_mutex );
 
      m_monitoredFraction = value;
      return;
   }

setServerAddress()

void xAOD::TFileAccessTracer::setServerAddress

(

const std::string &

addr

)

Set the address of the server that information is sent to.

Definition at line 245 of file TFileAccessTracer.cxx.

                                                                   {
 
      // Protect this call:
      std::lock_guard< std::mutex > lock( m_mutex );
 
      // Set the address itself:
      m_serverAddress = addr;
 
      // Construct the "technical address" of the host:
      ::TUrl url( m_serverAddress.c_str() );
      m_serverInetAddress = gSystem->GetHostByName( url.GetHost() );
      return;
   }

Member Data Documentation

m_accessedFiles

std::set< AccessedFile > xAOD::TFileAccessTracer::m_accessedFiles

private

List of all the input files that were accessed in the job.

Definition at line 85 of file TFileAccessTracer.h.

m_monitoredFraction

::Double_t xAOD::TFileAccessTracer::m_monitoredFraction

private

Fraction of jobs that should send monitoring information.

Definition at line 93 of file TFileAccessTracer.h.

m_mutex

std::mutex xAOD::TFileAccessTracer::m_mutex

mutableprivate

Mutex for modifying the object.

Definition at line 99 of file TFileAccessTracer.h.

m_readStats

ReadStats* xAOD::TFileAccessTracer::m_readStats

private

Object describing the job's xAOD access statistics.

Definition at line 96 of file TFileAccessTracer.h.

m_serverAddress

std::string xAOD::TFileAccessTracer::m_serverAddress

private

Address of the server to send monitoring information to.

Definition at line 88 of file TFileAccessTracer.h.

m_serverInetAddress

::TInetAddress xAOD::TFileAccessTracer::m_serverInetAddress

private

Technical address of the server.

Definition at line 90 of file TFileAccessTracer.h.

---

The documentation for this class was generated from the following files:

• TFileAccessTracer.h
• TFileAccessTracer.cxx