EL::AlgorithmMemoryWrapper Class Reference

an IAlgorithmWrapper that adds a memory monitor to an algorithm More...

#include <AlgorithmMemoryWrapper.h>

Inheritance diagram for EL::AlgorithmMemoryWrapper:

Collaboration diagram for EL::AlgorithmMemoryWrapper:

Public Types
using	clock_type = std::chrono::high_resolution_clock
	the clock we use for our Memory More...

Public Member Functions
void	testInvariant () const
	test the invariant of this object More...
	AlgorithmMemoryWrapper ()
	standard default constructor for serialization More...
	AlgorithmMemoryWrapper (std::unique_ptr< IAlgorithmWrapper > &&val_algorithm)
	standard constructor More...
virtual std::string_view	getName () const override
virtual bool	hasName (const std::string &name) const override
	whether this algorithm has the given name More...
virtual std::unique_ptr< IAlgorithmWrapper >	makeClone () const override
	make a clone of this algorithm More...
virtual Algorithm *	getLegacyAlg () override
	get the legacy algorithm, if we wrap one More...
virtual StatusCode	initialize (const AlgorithmWorkerData &workerData) override
	call initialize on the algorithm More...
virtual StatusCode	execute () override
	call execute on the algorithm More...
virtual StatusCode	postExecute () override
	call postExecute on the algorithm More...
virtual StatusCode	finalize () override
	call finalize on the algorithm More...
virtual StatusCode	fileExecute () override
	call fileExecute on the algorithm More...
virtual StatusCode	beginInputFile () override
	call beginInputFile on the algorithm More...
virtual StatusCode	endInputFile () override
	call endInputFile on the algorithm More...

Private Member Functions
StatusCode	recordPreMemory ()
StatusCode	recordPostMemory ()

Private Attributes
std::unique_ptr< IAlgorithmWrapper >	m_algorithm
	the actual algorithm More...
std::vector< Long_t >	m_mem_resident
	the Memory Consumption for different calls More...
std::vector< Long_t >	m_mem_virtual
Long_t	m_preMem_resident = 0
Long_t	m_preMem_virtual = 0

Detailed Description

an IAlgorithmWrapper that adds a memory monitor to an algorithm

This is a little weird in how its done: We can't just count the total memory allocated as all algorithms allocate memory which gets released at the end of the event. We also can't just count what gets allocated in initialize (which would seem natural), as some tools and algorithms do not allocate all their memory until we hit execute. And it may not even be the first execute either, it depends on what's in the event. So I'm summing up the memory increase from the top ten calls to functions on the algorithm. This is a bit of a hack, but the whole mechanism is meant to be quick and dirty. If you want a proper estimate, you should use valgrind.

Note

This module is specifically intended to debug the issue with analysis jobs running out of memory (Feb 24). Once that issue is resolved for good, or if there are fundamental issues that break this module it can be removed.

There is a dedicated test in AnalysisAlgorithms config that runs a test job with this module enabled to ensure it runs and doesn't break the output.

Definition at line 43 of file AlgorithmMemoryWrapper.h.

Member Typedef Documentation

clock_type

using EL::AlgorithmMemoryWrapper::clock_type = std::chrono::high_resolution_clock

the clock we use for our Memory

Public Members

Definition at line 51 of file AlgorithmMemoryWrapper.h.

Constructor & Destructor Documentation

AlgorithmMemoryWrapper() [1/2]

EL::AlgorithmMemoryWrapper::AlgorithmMemoryWrapper ( )

inline

standard default constructor for serialization

Definition at line 57 of file AlgorithmMemoryWrapper.h.

{};

AlgorithmMemoryWrapper() [2/2]

EL::AlgorithmMemoryWrapper::AlgorithmMemoryWrapper

(

std::unique_ptr< IAlgorithmWrapper > &&

val_algorithm

)

standard constructor

Definition at line 35 of file AlgorithmMemoryWrapper.cxx.

    : m_algorithm (std::move (val_algorithm))
  {
    RCU_NEW_INVARIANT (this);
  }

Member Function Documentation

beginInputFile()

StatusCode EL::AlgorithmMemoryWrapper::beginInputFile ( )

overridevirtual

call beginInputFile on the algorithm

Implements EL::IAlgorithmWrapper.

Definition at line 160 of file AlgorithmMemoryWrapper.cxx.

  {
    using namespace msgEventLoop;
    RCU_CHANGE_INVARIANT (this);
 
    ANA_CHECK (recordPreMemory());
    auto result = m_algorithm->beginInputFile ();
    ANA_CHECK (recordPostMemory());
    return result;
  }

endInputFile()

StatusCode EL::AlgorithmMemoryWrapper::endInputFile ( )

overridevirtual

call endInputFile on the algorithm

Implements EL::IAlgorithmWrapper.

Definition at line 174 of file AlgorithmMemoryWrapper.cxx.

  {
    using namespace msgEventLoop;
    RCU_CHANGE_INVARIANT (this);
 
    ANA_CHECK (recordPreMemory());
    auto result = m_algorithm->endInputFile ();
    ANA_CHECK (recordPostMemory());
    return result;
  }

execute()

StatusCode EL::AlgorithmMemoryWrapper::execute ( )

overridevirtual

call execute on the algorithm

Implements EL::IAlgorithmWrapper.

Definition at line 96 of file AlgorithmMemoryWrapper.cxx.

  {
    using namespace msgEventLoop;
    RCU_CHANGE_INVARIANT (this);
 
    ANA_CHECK (recordPreMemory());
    auto result = m_algorithm->execute ();
    ANA_CHECK (recordPostMemory());
    return result;
  }

fileExecute()

StatusCode EL::AlgorithmMemoryWrapper::fileExecute ( )

overridevirtual

call fileExecute on the algorithm

Implements EL::IAlgorithmWrapper.

Definition at line 146 of file AlgorithmMemoryWrapper.cxx.

  {
    using namespace msgEventLoop;
    RCU_CHANGE_INVARIANT (this);
 
    ANA_CHECK (recordPreMemory());
    auto result = m_algorithm->fileExecute ();
    ANA_CHECK (recordPostMemory());
    return result;
  }

finalize()

StatusCode EL::AlgorithmMemoryWrapper::finalize ( )

overridevirtual

call finalize on the algorithm

Implements EL::IAlgorithmWrapper.

Definition at line 124 of file AlgorithmMemoryWrapper.cxx.

  {
    using namespace msgEventLoop;
    RCU_CHANGE_INVARIANT (this);
 
    ANA_CHECK (recordPreMemory());
    auto result = m_algorithm->finalize ();
    ANA_CHECK (recordPostMemory());
    std::ostringstream str;
    str << "algorithm memory: " << m_algorithm->getName() << " rss=";
    for (auto& rss : m_mem_resident)
      str << rss << ",";
    str << " vss=";
    for (auto& vss : m_mem_virtual)
      str << vss << ",";
    ANA_MSG_INFO (str.str());
    return result;
  }

getLegacyAlg()

Algorithm * EL::AlgorithmMemoryWrapper::getLegacyAlg ( )

overridevirtual

get the legacy algorithm, if we wrap one

Reimplemented from EL::IAlgorithmWrapper.

Definition at line 73 of file AlgorithmMemoryWrapper.cxx.

  {
    RCU_READ_INVARIANT (this);
    return m_algorithm->getLegacyAlg();
  }

getName()

std::string_view EL::AlgorithmMemoryWrapper::getName ( ) const

overridevirtual

Inherited Members

Implements EL::IAlgorithmWrapper.

Definition at line 44 of file AlgorithmMemoryWrapper.cxx.

  {
    RCU_READ_INVARIANT (this);
    return m_algorithm->getName();
  }

hasName()

bool EL::AlgorithmMemoryWrapper::hasName ( const std::string &  name ) const

overridevirtual

whether this algorithm has the given name

Implements EL::IAlgorithmWrapper.

Definition at line 53 of file AlgorithmMemoryWrapper.cxx.

  {
    RCU_READ_INVARIANT (this);
    return m_algorithm->hasName (name);
  }

initialize()

StatusCode EL::AlgorithmMemoryWrapper::initialize ( const AlgorithmWorkerData &  workerData )

overridevirtual

call initialize on the algorithm

Implements EL::IAlgorithmWrapper.

Definition at line 82 of file AlgorithmMemoryWrapper.cxx.

  {
    using namespace msgEventLoop;
    RCU_CHANGE_INVARIANT (this);
 
    ANA_CHECK (recordPreMemory());
    auto result = m_algorithm->initialize (workerData);
    ANA_CHECK (recordPostMemory());
    return result;
  }

makeClone()

std::unique_ptr< IAlgorithmWrapper > EL::AlgorithmMemoryWrapper::makeClone ( ) const

overridevirtual

make a clone of this algorithm

Implements EL::IAlgorithmWrapper.

Definition at line 62 of file AlgorithmMemoryWrapper.cxx.

  {
    using namespace msgEventLoop;
    RCU_READ_INVARIANT (this);
 
    return std::make_unique<AlgorithmMemoryWrapper> (m_algorithm->makeClone());
  }

postExecute()

StatusCode EL::AlgorithmMemoryWrapper::postExecute ( )

overridevirtual

call postExecute on the algorithm

Reimplemented from EL::IAlgorithmWrapper.

Definition at line 110 of file AlgorithmMemoryWrapper.cxx.

  {
    using namespace msgEventLoop;
    RCU_CHANGE_INVARIANT (this);
 
    ANA_CHECK (recordPreMemory());
    auto result = m_algorithm->postExecute ();
    ANA_CHECK (recordPostMemory());
    return result;
  }

recordPostMemory()

StatusCode EL::AlgorithmMemoryWrapper::recordPostMemory ( )

private

Definition at line 205 of file AlgorithmMemoryWrapper.cxx.

  {
    constexpr std::size_t max_entries = 3;
 
    using namespace msgEventLoop;
    RCU_CHANGE_INVARIANT (this);
    ::ProcInfo_t pinfo;
    if (gSystem->GetProcInfo (&pinfo) != 0) {
      ANA_MSG_ERROR ("Could not get memory usage information");
      return StatusCode::FAILURE;
    }
    m_mem_resident.push_back (pinfo.fMemResident - m_preMem_resident);
    std::sort (m_mem_resident.begin()+1, m_mem_resident.end(), std::greater<Long_t>());
    if (m_mem_resident.size() > max_entries)
      m_mem_resident.resize (max_entries);
    m_mem_virtual.push_back (pinfo.fMemVirtual - m_preMem_virtual);
    std::sort (m_mem_virtual.begin()+1, m_mem_virtual.end(), std::greater<Long_t>());
    if (m_mem_virtual.size() > max_entries)
      m_mem_virtual.resize (max_entries);
    return StatusCode::SUCCESS;    
  }

recordPreMemory()

StatusCode EL::AlgorithmMemoryWrapper::recordPreMemory ( )

private

Definition at line 188 of file AlgorithmMemoryWrapper.cxx.

  {
    using namespace msgEventLoop;
    RCU_CHANGE_INVARIANT (this);
    ::ProcInfo_t pinfo;
    if (gSystem->GetProcInfo (&pinfo) != 0) {
      ANA_MSG_ERROR ("Could not get memory usage information");
      return StatusCode::FAILURE;
    }
    m_preMem_resident = pinfo.fMemResident;
    m_preMem_virtual = pinfo.fMemVirtual;
    return StatusCode::SUCCESS;    
  }

testInvariant()

void EL::AlgorithmMemoryWrapper::testInvariant

(

)

const

test the invariant of this object

Definition at line 28 of file AlgorithmMemoryWrapper.cxx.

  {
  }

Member Data Documentation

m_algorithm

std::unique_ptr<IAlgorithmWrapper> EL::AlgorithmMemoryWrapper::m_algorithm

private

the actual algorithm

Private Members

Definition at line 99 of file AlgorithmMemoryWrapper.h.

m_mem_resident

std::vector<Long_t> EL::AlgorithmMemoryWrapper::m_mem_resident

private

the Memory Consumption for different calls

Definition at line 102 of file AlgorithmMemoryWrapper.h.

m_mem_virtual

std::vector<Long_t> EL::AlgorithmMemoryWrapper::m_mem_virtual

private

Definition at line 103 of file AlgorithmMemoryWrapper.h.

m_preMem_resident

Long_t EL::AlgorithmMemoryWrapper::m_preMem_resident = 0

private

Definition at line 105 of file AlgorithmMemoryWrapper.h.

m_preMem_virtual

Long_t EL::AlgorithmMemoryWrapper::m_preMem_virtual = 0

private

Definition at line 106 of file AlgorithmMemoryWrapper.h.

---

The documentation for this class was generated from the following files:

• AlgorithmMemoryWrapper.h
• AlgorithmMemoryWrapper.cxx