xAOD::TFileAccessTracer Class Reference

Helper class keeping track of the files that got accessed. More...

#include <TFileAccessTracer.h>

Collaboration diagram for xAOD::TFileAccessTracer:

Classes
struct	Impl

Public Member Functions
	~TFileAccessTracer ()
	Destructor.
void	add (std::string_view fileName)
	Add information about a new file that got accessed.
const std::string &	serverAddress () const
	The address of the server that information is sent to.
void	setServerAddress (const std::string &addr)
	Set the address of the server that information is sent to.
double	monitoredFraction () const
	Fraction of jobs that should send monitoring information.
void	setMonitoredFraction (double value)
	Set the fraction of jobs that should send monitoring information.
void	enableDataSubmission (bool value)
	Function for turning data submission on/off.

Static Public Member Functions
static TFileAccessTracer &	instance ()
	Access the singleton instance of this class.

Private Member Functions
	TFileAccessTracer ()
	Default constructor.

Private Attributes
std::unique_ptr< Impl >	m_impl
	Pointer to the implementation of the class.

Detailed Description

Helper class keeping track of the files that got accessed.

This class helps in keeping track of which files get accessed during a job. To be able to report about them to the DDM (Rucio) system in a way that doesn't rely on the grid middleware. (So backdoor access to files stored at data centres becomes visible to the DDM system.)

The class can also send information about which branches/variables got accessed in the job. This it simply gets from the xAODCore code.

xAOD::TEvent and xAOD::REvent use this class to send information about xAOD access to the central Rucio monitoring infrastructure. This default behaviour can be influenced in two ways:

• Using the the public functions on the singleton instance of this class;
• Setting the following environment variables before running a job:
  • "XAOD_ACCESSTRACER_FRACTION": This is a floating point value, which is the fraction of jobs that should be monitored within that session.
  • "XAOD_ACCESSTRACER_SERVER": This is the address of the server to which the monitoring information should be sent.

Definition at line 37 of file TFileAccessTracer.h.

Constructor & Destructor Documentation

~TFileAccessTracer()

xAOD::TFileAccessTracer::~TFileAccessTracer

(

)

Destructor.

The destructor of the class is the one doing most of the heavy lifting.

If the $SEND_XAOD_FILE_ACCESS_STAT environment variable is set when the object gets deleted, it posts all the information it collected, to the address defined by SERVER_ADDRESS. By constructing an HTTP message from scratch.

Definition at line 81 of file TFileAccessTracer.cxx.

                                      {
 
  // Make sure that nobody is trying to update the object at this point.
  std::lock_guard<std::mutex> lock(m_impl->m_mutex);
 
  // If the user turned off the data submission, then stop already here...
  if (m_impl->m_enableDataSumbission == false) {
    return;
  }
 
  // Decide what monitoring fraction to take. To make it possible for Panda
  // to override it with larger/smaller values if needed.
  double monitoredFraction = m_impl->m_monitoredFraction;
  const char* fractionString = gSystem->Getenv("XAOD_ACCESSTRACER_FRACTION");
  if (fractionString) {
    char* endptr = 0;
    const double fraction = strtod(fractionString, &endptr);
    if (endptr != fractionString) {
      monitoredFraction = fraction;
    }
  }
 
  // Decide randomly whether to send the monitoring data or not. While
  // TRandom is not good enough for statistical purposes, it's fast, and is
  // perfectly good to make this decision...
  ::TRandom rng(::TTimeStamp().GetNanoSec());
  if (rng.Rndm() > monitoredFraction) {
    return;
  }
 
  // Decide what server to send the information to. To make it possible for
  // Panda to override it if needed.
  std::string serverAddress = m_impl->m_serverAddress;
  const char* serverAddressString = gSystem->Getenv("XAOD_ACCESSTRACER_SERVER");
  if (serverAddressString) {
    serverAddress = serverAddressString;
  }
 
  // Construct the "technical address" of the host:
  const ::TUrl url(serverAddress.c_str());
  const ::TInetAddress serverInetAddress =
      gSystem->GetHostByName(url.GetHost());
 
  // Open a socket to the server:
  TSocket socket;
  if (socket.connect(serverInetAddress, url.GetPort()).isSuccess() == false) {
    // Just exit silently. If we can't send the info, we can't send the
    // info. It's not a problem.
    return;
  }
 
  // Let the user know what's happening:
  ::Info("xAOD::TFileAccessTracer", "Sending file access statistics to %s",
         serverAddress.c_str());
 
  // Start constructing the header of the message to send to the server:
  ::TString hdr = "POST /";
  hdr += url.GetFile();
  hdr += " HTTP/1.1\r\n";
  hdr += "From: ";
  hdr += gSystem->HostName();
  hdr += "\r\n";
  hdr += "Host: ";
  hdr += url.GetHost();
  hdr += "\r\n";
  hdr += "User-Agent: xAODRootAccess\r\n";
  hdr += "Content-Type: application/json\r\n";
  hdr += "Connection: close\r\n";
  hdr += "Content-Length: ";
 
  //
  // Now construct the message payload:
  //
  ::TString pld = "{";
 
  //
  // Collect the names of all the accessed files:
  //
  pld += "\"accessedFiles\": [";
  bool first = true;
  for (const Impl::AccessedFile& info : m_impl->m_accessedFiles) {
    if (!first) {
      pld += ", ";
    }
    pld += "\"";
    pld += info.fullFilePath();
    pld += "\"";
    first = false;
  }
  pld += "], ";
  //
  // Collect the names of all the containers that were accessed:
  //
  pld += "\"accessedContainers\": [";
  first = true;
  for (const auto& bs : m_impl->m_readStats.containers()) {
    if (!bs.second.readEntries()) {
      continue;
    }
    if (!first) {
      pld += ", ";
    }
    pld += "{\"";
    pld += bs.second.GetName();
    pld += "\": ";
    pld += bs.second.readEntries();
    pld += "}";
    first = false;
  }
  pld += "], ";
  //
  // Collect the names of all the branches that were accessed:
  //
  pld += "\"accessedBranches\": [";
  first = true;
  for (const auto& branch : m_impl->m_readStats.branches()) {
    for (const xAOD::BranchStats* bs : branch.second) {
      if ((!bs) || (!bs->readEntries())) {
        continue;
      }
      if (!first) {
        pld += ", ";
      }
      pld += "{\"";
      pld += bs->GetName();
      pld += "\": ";
      pld += bs->readEntries();
      pld += "}";
      first = false;
    }
  }
  pld += "]";
  //
  // Collect some possible Panda information in case the job is running
  // on the grid:
  //
  const char* pandaID = gSystem->Getenv("PandaID");
  if (pandaID) {
    pld += ", \"PandaID\": ";
    pld += pandaID;
    pld += "";
  }
  const char* taskID = gSystem->Getenv("PanDA_TaskID");
  if (taskID) {
    pld += ", \"PanDA_TaskID\": ";
    pld += taskID;
    pld += "";
  }
  //
  // Add some simple information about the host:
  //
  pld += ", \"ROOT_RELEASE\": \"";
  pld += ROOT_RELEASE;
  pld += "\"";
  pld += ", \"ReportRate\": ";
  pld += monitoredFraction;
  //
  // Add some information about the file access pattern:
  //
  pld += ", \"ReadCalls\": ";
  pld += m_impl->m_readStats.fileReads();
  pld += ", \"ReadSize\": ";
  pld +=
      (m_impl->m_readStats.fileReads() != 0
           ? m_impl->m_readStats.bytesRead() / m_impl->m_readStats.fileReads()
           : 0);
  pld += ", \"CacheSize\": ";
  pld += m_impl->m_readStats.cacheSize();
  pld += "}";
 
  // Now finish constructing the header, and merge the two into a single
  // message:
  hdr += TString::Format("%i", pld.Length());
  hdr += "\r\n\r\n";
  const ::TString msg = hdr + pld;
 
  // Send the message, and try to receive an answer.
  socket.send(msg).ignore();
  TString response;
  socket.receive(4096, response).ignore();
}

TFileAccessTracer()

xAOD::TFileAccessTracer::TFileAccessTracer ( )

private

Default constructor.

Definition at line 327 of file TFileAccessTracer.cxx.

: m_impl(std::make_unique<Impl>()) {}

Member Function Documentation

add()

void xAOD::TFileAccessTracer::add

(

std::string_view

fileName

)

Add information about a new file that got accessed.

This function is called by TEvent to record which files were read from during the job.

Parameters

fileName

The name of the file that is being read from

Definition at line 274 of file TFileAccessTracer.cxx.

                                                   {
 
  // Protect this call:
  std::lock_guard<std::mutex> lock(m_impl->m_mutex);
 
  // Remember this file:
  m_impl->m_accessedFiles.insert(
      {gSystem->DirName(fileName.data()), gSystem->BaseName(fileName.data())});
}

enableDataSubmission()

void xAOD::TFileAccessTracer::enableDataSubmission

(

bool

value

)

Function for turning data submission on/off.

This function can be used by concerned users to turn off data collection and submission for their jobs completely.

Parameters

value

Setting whether data submission should be enabled or not

Definition at line 322 of file TFileAccessTracer.cxx.

                                                         {
 
  m_impl->m_enableDataSumbission = value;
}

instance()

TFileAccessTracer & xAOD::TFileAccessTracer::instance ( )

static

Access the singleton instance of this class.

Definition at line 263 of file TFileAccessTracer.cxx.

                                               {
 
  static TFileAccessTracer instance;
  return instance;
}

monitoredFraction()

double xAOD::TFileAccessTracer::monitoredFraction

(

)

const

Fraction of jobs that should send monitoring information.

Definition at line 301 of file TFileAccessTracer.cxx.

                                                  {
 
  // Protect this call:
  std::lock_guard<std::mutex> lock(m_impl->m_mutex);
 
  return m_impl->m_monitoredFraction;
}

serverAddress()

const std::string & xAOD::TFileAccessTracer::serverAddress

(

)

const

The address of the server that information is sent to.

Definition at line 284 of file TFileAccessTracer.cxx.

                                                        {
 
  // Protect this call:
  std::lock_guard<std::mutex> lock(m_impl->m_mutex);
 
  return m_impl->m_serverAddress;
}

setMonitoredFraction()

void xAOD::TFileAccessTracer::setMonitoredFraction

(

double

value

)

Set the fraction of jobs that should send monitoring information.

Definition at line 309 of file TFileAccessTracer.cxx.

                                                         {
 
  // Protect this call:
  std::lock_guard<std::mutex> lock(m_impl->m_mutex);
 
  m_impl->m_monitoredFraction = value;
}

setServerAddress()

void xAOD::TFileAccessTracer::setServerAddress

(

const std::string &

addr

)

Set the address of the server that information is sent to.

Definition at line 292 of file TFileAccessTracer.cxx.

                                                              {
 
  // Protect this call:
  std::lock_guard<std::mutex> lock(m_impl->m_mutex);
 
  // Set the address itself:
  m_impl->m_serverAddress = addr;
}

Member Data Documentation

m_impl

std::unique_ptr<Impl> xAOD::TFileAccessTracer::m_impl

private

Pointer to the implementation of the class.

Definition at line 69 of file TFileAccessTracer.h.

---

The documentation for this class was generated from the following files:

• TFileAccessTracer.h
• TFileAccessTracer.cxx