a Module monitoring memory usage at various points during the job More...

#include <MemoryMonitorModule.h>

Inheritance diagram for EL::Detail::MemoryMonitorModule:

Collaboration diagram for EL::Detail::MemoryMonitorModule:

Public Member Functions
virtual StatusCode	firstInitialize (ModuleData &data) override
	action at the the very beginning of the worker job More...

virtual StatusCode	onInitialize (ModuleData &data) override
	action just before algorithms are initialized More...

virtual StatusCode	onExecute (ModuleData &data) override
	actions just before `execute` is called on algorithms More...

virtual StatusCode	postFirstEvent (ModuleData &data) override
	action after processing first event More...

virtual StatusCode	onFinalize (ModuleData &data) override
	actions just before algorithms are finalized More...

virtual StatusCode	onWorkerEnd (ModuleData &data) override
	action at the end of the worker job More...

virtual StatusCode	preFileInitialize (ModuleData &data)
	action before opening the first file in the worker job More...

virtual StatusCode	processInputs (ModuleData &data, IInputModuleActions &actions)
	process all input files More...

virtual StatusCode	onNewInputFile (ModuleData &data)
	actions after opening a new input file More...

virtual StatusCode	onCloseInputFile (ModuleData &data)
	actions before closing an input file More...

virtual StatusCode	postCloseInputFile (ModuleData &data)
	actions after `CloseInputFile` is called on the algorithms More...

virtual StatusCode	onFileExecute (ModuleData &data)
	actions just before `fileExecute` is called on algorithms More...

virtual StatusCode	postFinalize (ModuleData &data)
	actions after algorithms have been finalized More...

virtual StatusCode	postFileClose (ModuleData &data)
	action at end of the worker job More...

virtual void	reportInputFailure (ModuleData &data)
	report that we failed to open our input file More...

Public Attributes
std::optional< std::uint64_t >	lastRSS
	the last RSS value read More...

Private Member Functions
StatusCode	printMemoryUsage (const std::string &location)

Private Attributes
std::uint64_t	m_numExecute = 0

std::uint64_t	m_executeNext = 1

std::uint64_t	m_executeStep = 1

std::uint64_t	m_executeTarget = 30

Detailed Description

a Module monitoring memory usage at various points during the job

Note: This module is specifically intended to debug the issue with analysis jobs running out of memory (Feb 24). Once that issue is resolved for good, or if there are fundamental issues that break this module it can be removed.; There is a dedicated test in AnalysisAlgorithms config that runs a test job with this module enabled to ensure it runs and doesn't break the output.

Definition at line 35 of file MemoryMonitorModule.h.

Member Function Documentation

◆ firstInitialize()

StatusCode EL::Detail::MemoryMonitorModule::firstInitialize ( ModuleData & data )

overridevirtual

action at the the very beginning of the worker job

This gets called as early as possible in the worker initialization. Essentially all that should happen before this is to load all the modules. The main purpose is to start any benchmarks that are meant to capture the initialization process as well.

Reimplemented from EL::Detail::Module.

Definition at line 26 of file MemoryMonitorModule.cxx.

     {
       ANA_CHECK (printMemoryUsage ("firstInitialize"));
       return StatusCode::SUCCESS;
     }

◆ onCloseInputFile()

StatusCode EL::Detail::Module::onCloseInputFile ( ModuleData & data )

virtualinherited

actions before closing an input file

Reimplemented in EL::Detail::AlgorithmStateModule, and EL::Detail::TreeCacheModule.

Definition at line 43 of file Module.cxx.

     {
       return ::StatusCode::SUCCESS;
     }

◆ onExecute()

StatusCode EL::Detail::MemoryMonitorModule::onExecute ( ModuleData & data )

overridevirtual

actions just before execute is called on algorithms

For now that is mostly used to point input modules to the right event

Reimplemented from EL::Detail::Module.

Definition at line 44 of file MemoryMonitorModule.cxx.

     {
       if (++ m_numExecute == m_executeNext)
       {
         m_executeNext += m_executeStep;
         if (m_executeNext == m_executeTarget)
           m_executeStep *= 10, m_executeTarget *= 10;
  
         ANA_CHECK (printMemoryUsage ("onExecute(" + std::to_string (m_numExecute) + ")"));
       }
       return StatusCode::SUCCESS;
     }

◆ onFileExecute()

StatusCode EL::Detail::Module::onFileExecute ( ModuleData & data )

virtualinherited

actions just before fileExecute is called on algorithms

Reimplemented in EL::Detail::AlgorithmStateModule, EL::Detail::FileExecutedModule, and EL::Detail::StopwatchModule.

Definition at line 65 of file Module.cxx.

     {
       return ::StatusCode::SUCCESS;
     }

◆ onFinalize()

StatusCode EL::Detail::MemoryMonitorModule::onFinalize ( ModuleData & data )

overridevirtual

actions just before algorithms are finalized

Reimplemented from EL::Detail::Module.

Definition at line 69 of file MemoryMonitorModule.cxx.

     {
       ANA_CHECK (printMemoryUsage ("onFinalize"));
       return StatusCode::SUCCESS;
     }

◆ onInitialize()

StatusCode EL::Detail::MemoryMonitorModule::onInitialize ( ModuleData & data )

overridevirtual

action just before algorithms are initialized

This is typically used for any setup that this module needs to do.

Reimplemented from EL::Detail::Module.

Definition at line 35 of file MemoryMonitorModule.cxx.

     {
       ANA_CHECK (printMemoryUsage ("onInitialize"));
       return StatusCode::SUCCESS;
     }

◆ onNewInputFile()

StatusCode EL::Detail::Module::onNewInputFile ( ModuleData & data )

virtualinherited

actions after opening a new input file

Reimplemented in EL::Detail::TEventModule, EL::Detail::AlgorithmStateModule, EL::Detail::GridReportingModule, and EL::Detail::TreeCacheModule.

Definition at line 37 of file Module.cxx.

     {
       return ::StatusCode::SUCCESS;
     }

◆ onWorkerEnd()

StatusCode EL::Detail::MemoryMonitorModule::onWorkerEnd ( ModuleData & data )

overridevirtual

action at the end of the worker job

This is mostly meant/used to print job summary statements at the very end of worker job, and actually have them show up at or at least near the end of any log file.

Reimplemented from EL::Detail::Module.

Definition at line 78 of file MemoryMonitorModule.cxx.

     {
       ANA_CHECK (printMemoryUsage ("onWorkerEnd"));
       return StatusCode::SUCCESS;
     }

◆ postCloseInputFile()

StatusCode EL::Detail::Module::postCloseInputFile ( ModuleData & data )

virtualinherited

actions after CloseInputFile is called on the algorithms

Right now that is only used to disconnect the TEvent object from the input file.

Reimplemented in EL::Detail::TEventModule.

Definition at line 49 of file Module.cxx.

     {
       return ::StatusCode::SUCCESS;
     }

◆ postFileClose()

StatusCode EL::Detail::Module::postFileClose ( ModuleData & data )

virtualinherited

action at end of the worker job

Executed just right before worker exit successfully at this stage all outputs have been created

Reimplemented in EL::Detail::GridReportingModule, and EL::Detail::PostClosedOutputsModule.

Definition at line 107 of file Module.cxx.

     {
       return ::StatusCode::SUCCESS;
     }

◆ postFinalize()

StatusCode EL::Detail::Module::postFinalize ( ModuleData & data )

virtualinherited

actions after algorithms have been finalized

This is usually used to calculate some job summary information that is to be stored it in the histogram output file.

Reimplemented in EL::Detail::TEventModule, EL::Detail::FileExecutedModule, EL::Detail::LeakCheckModule, EL::Detail::StopwatchModule, and EL::Detail::EventCountModule.

Definition at line 95 of file Module.cxx.

     {
       return ::StatusCode::SUCCESS;
     }

◆ postFirstEvent()

StatusCode EL::Detail::MemoryMonitorModule::postFirstEvent ( ModuleData & data )

overridevirtual

action after processing first event

This is mostly meant to set up benchmarks that record per-event performance. While a lot of initialization happens during initialize() there is a fair amount of initialization that happens on the first event, so when recording per-event performance the first event is sort of "special" and may need to be omitted.

Reimplemented from EL::Detail::Module.

Definition at line 60 of file MemoryMonitorModule.cxx.

     {
       ANA_CHECK (printMemoryUsage ("postFirstEvent"));
       return StatusCode::SUCCESS;
     }

◆ preFileInitialize()

StatusCode EL::Detail::Module::preFileInitialize ( ModuleData & data )

virtualinherited

action before opening the first file in the worker job

This is mostly meant to allow loading the dictionaries before any files and associated information is loaded.

Definition at line 31 of file Module.cxx.

     {
       return ::StatusCode::SUCCESS;
     }

◆ printMemoryUsage()

StatusCode EL::Detail::MemoryMonitorModule::printMemoryUsage ( const std::string & location )

private

Definition at line 87 of file MemoryMonitorModule.cxx.

     {
       ::ProcInfo_t pinfo;
       if (gSystem->GetProcInfo (&pinfo) != 0) {
         ANA_MSG_ERROR ("Could not get memory usage information");
         return StatusCode::FAILURE;
       }
       ANA_MSG_INFO ("Memory usage at " << location << ": " << pinfo.fMemResident << " kB resident, " << pinfo.fMemVirtual << " kB virtual");
       lastRSS = pinfo.fMemResident;
       return StatusCode::SUCCESS;
     }

◆ processInputs()

StatusCode EL::Detail::Module::processInputs	(	ModuleData &	data,
		IInputModuleActions &	actions
	)

virtualinherited

process all input files

This deviates slightly from the usual pattern for module functions in that I pass in the possible actions as an argument. See IInputModuleActions for details.

Reimplemented in EL::Detail::BatchInputModule, and EL::Detail::DirectInputModule.

Definition at line 83 of file Module.cxx.

     {
       return ::StatusCode::SUCCESS;
     }

◆ reportInputFailure()

void EL::Detail::Module::reportInputFailure ( ModuleData & data )

virtualinherited

report that we failed to open our input file

Reimplemented in EL::Detail::GridReportingModule.

Definition at line 61 of file Module.cxx.

63 {}

Member Data Documentation

◆ lastRSS

std::optional<std::uint64_t> EL::Detail::MemoryMonitorModule::lastRSS

the last RSS value read

This is mostly meant for convenient testing. It is not used by the module itself.

Definition at line 54 of file MemoryMonitorModule.h.

◆ m_executeNext

std::uint64_t EL::Detail::MemoryMonitorModule::m_executeNext = 1

private

Definition at line 63 of file MemoryMonitorModule.h.

◆ m_executeStep

std::uint64_t EL::Detail::MemoryMonitorModule::m_executeStep = 1

private

Definition at line 64 of file MemoryMonitorModule.h.

◆ m_executeTarget

std::uint64_t EL::Detail::MemoryMonitorModule::m_executeTarget = 30

private

Definition at line 65 of file MemoryMonitorModule.h.

◆ m_numExecute

std::uint64_t EL::Detail::MemoryMonitorModule::m_numExecute = 0

private

Private Members

Definition at line 62 of file MemoryMonitorModule.h.

The documentation for this class was generated from the following files:

Public Member Functions

Public Attributes

Private Member Functions

Private Attributes

Detailed Description

Member Function Documentation

◆ firstInitialize()

◆ onCloseInputFile()

◆ onExecute()

◆ onFileExecute()

◆ onFinalize()

◆ onInitialize()

◆ onNewInputFile()

◆ onWorkerEnd()

◆ postCloseInputFile()

◆ postFileClose()

◆ postFinalize()

◆ postFirstEvent()

◆ preFileInitialize()

◆ printMemoryUsage()

◆ processInputs()

◆ reportInputFailure()

Member Data Documentation

◆ lastRSS

◆ m_executeNext

◆ m_executeStep

◆ m_executeTarget

◆ m_numExecute

Private Members