ATLAS Offline Software
Public Member Functions | Private Types | Private Member Functions | Private Attributes | List of all members
PerfMonMTSvc Class Reference

#include <PerfMonMTSvc.h>

Inheritance diagram for PerfMonMTSvc:
Collaboration diagram for PerfMonMTSvc:

Public Member Functions

 PerfMonMTSvc (const std::string &name, ISvcLocator *pSvcLocator)
 Standard Gaudi Service constructor. More...
 
virtual ~PerfMonMTSvc ()=default
 
virtual void handle (const Incident &incident) override
 Incident service handle for post-finalize. More...
 
virtual StatusCode initialize () override
 Standard Gaudi Service initialization. More...
 
virtual StatusCode finalize () override
 Standard Gaudi Service finalization. More...
 
virtual void startAud (const std::string &stepName, const std::string &compName, const EventContext &ctx) override
 Start Auditing. More...
 
virtual void stopAud (const std::string &stepName, const std::string &compName, const EventContext &ctx) override
 Stop Auditing. More...
 
void startSnapshotAud (const std::string &stepName, const std::string &compName)
 Snapshot Auditing: Take snapshots at the beginning and at the end of each step. More...
 
void stopSnapshotAud (const std::string &stepName, const std::string &compName)
 
void startCompAud (const std::string &stepName, const std::string &compName, const EventContext &ctx)
 Component Level Auditing: Take measurements at the beginning and at the end of each component call. More...
 
void stopCompAud (const std::string &stepName, const std::string &compName, const EventContext &ctx)
 
void report ()
 Report the results. More...
 
void report2Log ()
 Report to log. More...
 
void report2Log_Description () const
 
void report2Log_ComponentLevel ()
 
void report2Log_EventLevel_instant () const
 
void report2Log_EventLevel ()
 
void report2Log_Summary ()
 
void report2Log_CpuInfo () const
 
void report2Log_EnvInfo () const
 
void report2JsonFile ()
 Report to the JSON File. More...
 
void report2JsonFile_Summary (nlohmann::json &j) const
 
void report2JsonFile_ComponentLevel (nlohmann::json &j) const
 
void report2JsonFile_EventLevel (nlohmann::json &j) const
 
void aggregateSlotData ()
 A few helper functions. More...
 
void divideData2Steps ()
 
std::string scaleTime (double timeMeas) const
 
std::string scaleMem (int64_t memMeas) const
 
std::string get_info_from_file (const std::string &fileName, const std::string &fieldName) const
 A few helper methods to get system information These should be carried to PerfMonMTUtils at some point. More...
 
std::string get_cpu_model_info () const
 
int get_cpu_core_info () const
 
uint64_t get_memory_info () const
 
PMonMT::StepComp generate_state (const std::string &stepName, const std::string &compName) const
 

Private Types

enum  Snapshots {
  CONFIGURE, INITIALIZE, FIRSTEVENT, EXECUTE,
  FINALIZE, NSNAPSHOTS
}
 
typedef std::map< PMonMT::StepComp, PMonMT::ComponentData * > data_map_t
 
typedef std::map< PMonMT::StepComp, std::unique_ptr< PMonMT::ComponentData > > data_map_unique_t
 

Private Member Functions

int getCpuEfficiency () const
 

Private Attributes

PMonMT::SnapshotMeasurement m_measurementSnapshots
 Measurement to capture snapshots. More...
 
PMonMT::SnapshotMeasurement m_measurementEvents
 Measurement to capture events. More...
 
Gaudi::Property< bool > m_doEventLoopMonitoring
 Do event loop monitoring. More...
 
Gaudi::Property< bool > m_doComponentLevelMonitoring
 Do component level monitoring. More...
 
Gaudi::Property< bool > m_reportResultsToJSON {this, "reportResultsToJSON", true, "Report results into the json file."}
 Report results to JSON. More...
 
Gaudi::Property< std::string > m_jsonFileName
 Name of the JSON file. More...
 
Gaudi::Property< bool > m_printDetailedTables
 Print detailed tables. More...
 
Gaudi::Property< uint64_t > m_memFitLowerLimit
 Lower limit (in number of events) for the memory fit. More...
 
Gaudi::Property< uint64_t > m_checkPointThreshold
 Frequency of event level monitoring. More...
 
Gaudi::Property< double > m_wallTimeOffset {this, "wallTimeOffset", 0, "Job start wall time in miliseconds."}
 Offset for the wall-time, comes from configuration. More...
 
Gaudi::Property< int > m_printNComps
 Print the top N components. More...
 
Gaudi::Property< int > m_numberOfThreads {this, "numberOfThreads", 1, "Number of threads in the job."}
 Get the number of threads. More...
 
Gaudi::Property< int > m_numberOfSlots {this, "numberOfSlots", 1, "Number of slots in the job."}
 Get the number of slots. More...
 
Gaudi::Property< uint64_t > m_eventLoopMsgLimit {this, "eventLoopMsgLimit", 10, "Maximum number of event-level messages."}
 Set the number of messages for the event-level report. More...
 
const std::set< std::string > m_exclusionSet
 Exclude some common components from monitoring In the future this might be converted to a inclusion set which would allow user to monitor only a set of algorithms... More...
 
int m_motherPID
 Snapshots data. More...
 
std::vector< PMonMT::SnapshotDatam_snapshotData
 
std::vector< std::string > m_snapshotStepNames = {"Configure", "Initialize", "FirstEvent", "Execute", "Finalize"}
 
PMonMT::EventLevelData m_eventLevelData {}
 
std::mutex m_mutex_capture
 
std::atomic< bool > m_isFirstEvent
 
std::atomic< uint64_t > m_eventCounter
 
std::atomic< uint64_t > m_eventLoopMsgCounter
 
std::atomic< double > m_checkPointTime
 
std::atomic< bool > m_isEvtLoopStopped
 
data_map_t m_compLevelDataMap
 
std::vector< data_map_unique_tm_compLevelDataMapVec
 
data_map_t m_compLevelDataMap_ini
 
data_map_t m_compLevelDataMap_1stevt
 
data_map_t m_compLevelDataMap_evt
 
data_map_t m_compLevelDataMap_fin
 
data_map_t m_compLevelDataMap_plp
 
data_map_t m_compLevelDataMap_cbk
 
std::vector< data_map_tm_stdoutVec_serial
 
PerfMon::LinFitSglPass m_fit_vmem
 
PerfMon::LinFitSglPass m_fit_pss
 

Detailed Description

Definition at line 41 of file PerfMonMTSvc.h.

Member Typedef Documentation

◆ data_map_t

Definition at line 189 of file PerfMonMTSvc.h.

◆ data_map_unique_t

typedef std::map<PMonMT::StepComp, std::unique_ptr<PMonMT::ComponentData> > PerfMonMTSvc::data_map_unique_t
private

Definition at line 190 of file PerfMonMTSvc.h.

Member Enumeration Documentation

◆ Snapshots

Enumerator
CONFIGURE 
INITIALIZE 
FIRSTEVENT 
EXECUTE 
FINALIZE 
NSNAPSHOTS 

Definition at line 163 of file PerfMonMTSvc.h.

Constructor & Destructor Documentation

◆ PerfMonMTSvc()

PerfMonMTSvc::PerfMonMTSvc ( const std::string &  name,
ISvcLocator *  pSvcLocator 
)

Standard Gaudi Service constructor.

Definition at line 32 of file PerfMonMTSvc.cxx.

33  : base_class(name, pSvcLocator), m_isFirstEvent{false}, m_eventCounter{0}, m_eventLoopMsgCounter{0}, m_checkPointTime{0}, m_isEvtLoopStopped{false} {
34  // Five main snapshots : Configure, Initialize, FirstEvent, Execute, and Finalize
35  m_motherPID = getpid();
36  m_snapshotData.resize(NSNAPSHOTS); // Default construct
37 
38  // Initial capture upon construction
42 }

◆ ~PerfMonMTSvc()

virtual PerfMonMTSvc::~PerfMonMTSvc ( )
virtualdefault

Member Function Documentation

◆ aggregateSlotData()

void PerfMonMTSvc::aggregateSlotData ( )

A few helper functions.

Definition at line 793 of file PerfMonMTSvc.cxx.

793  {
794  // Loop over data from all slots
795  for (const auto& slotData : m_compLevelDataMapVec) {
796  for (const auto& it : slotData) {
797  // Copy the first slot data and sum the rest
798  if(m_compLevelDataMap.find(it.first) == m_compLevelDataMap.end()) {
799  m_compLevelDataMap.insert({it.first, it.second.get()});
800  } else {
801  m_compLevelDataMap[it.first]->add2CallCount(it.second->getCallCount());
802  m_compLevelDataMap[it.first]->add2DeltaCPU(it.second->getDeltaCPU());
803  m_compLevelDataMap[it.first]->add2DeltaWall(it.second->getDeltaWall());
804  m_compLevelDataMap[it.first]->add2DeltaVmem(it.second->getDeltaVmem());
805  m_compLevelDataMap[it.first]->add2DeltaMalloc(it.second->getDeltaMalloc());
806  }
807  // Do a quick consistency check here and print any suspicious measurements.
808  // Timing measurements should always be positive definite
809  if(it.second->getDeltaCPU() < 0) {
810  ATH_MSG_WARNING("Negative CPU-time measurement of " << it.second->getDeltaCPU() <<
811  " ms for component " << it.first.compName <<
812  " in step " << it.first.stepName);
813  }
814  if(it.second->getDeltaWall() < 0) {
815  ATH_MSG_WARNING("Negative Wall-time measurement of " << it.second->getDeltaWall() <<
816  " ms for component " << it.first.compName <<
817  " in step " << it.first.stepName);
818  }
819  }
820  }
821 }

◆ divideData2Steps()

void PerfMonMTSvc::divideData2Steps ( )

Definition at line 826 of file PerfMonMTSvc.cxx.

826  {
827  for (const auto &it : m_compLevelDataMap) {
828  if (it.first.stepName == "Initialize")
829  m_compLevelDataMap_ini[it.first] = it.second;
830  else if (it.first.stepName == "FirstEvent")
831  m_compLevelDataMap_1stevt[it.first] = it.second;
832  else if (it.first.stepName == "Execute")
833  m_compLevelDataMap_evt[it.first] = it.second;
834  else if (it.first.stepName == "Finalize")
835  m_compLevelDataMap_fin[it.first] = it.second;
836  else if (it.first.stepName == "preLoadProxy")
837  m_compLevelDataMap_plp[it.first] = it.second;
838  else if (it.first.stepName == "Callback")
839  m_compLevelDataMap_cbk[it.first] = it.second;
840  }
847 }

◆ finalize()

StatusCode PerfMonMTSvc::finalize ( )
overridevirtual

Standard Gaudi Service finalization.

Definition at line 93 of file PerfMonMTSvc.cxx.

93  {
94  // Print where we are
95  ATH_MSG_INFO("Finalizing " << name());
96 
97  return StatusCode::SUCCESS;
98 }

◆ generate_state()

PMonMT::StepComp PerfMonMTSvc::generate_state ( const std::string &  stepName,
const std::string &  compName 
) const

Definition at line 783 of file PerfMonMTSvc.cxx.

783  {
784  PMonMT::StepComp currentState;
785  currentState.stepName = (m_isFirstEvent && stepName == "Execute") ? "FirstEvent" : stepName;
786  currentState.compName = compName;
787  return currentState;
788 }

◆ get_cpu_core_info()

int PerfMonMTSvc::get_cpu_core_info ( ) const

Definition at line 936 of file PerfMonMTSvc.cxx.

936  {
937  std::string val = get_info_from_file("/proc/cpuinfo","processor");
938  if (val.empty()) return 0;
939  return std::stoi(val) + 1;
940 }

◆ get_cpu_model_info()

std::string PerfMonMTSvc::get_cpu_model_info ( ) const

Definition at line 931 of file PerfMonMTSvc.cxx.

931  {
932  return get_info_from_file("/proc/cpuinfo","model name") +
933  get_info_from_file("/proc/cpuinfo","cache size");
934 }

◆ get_info_from_file()

std::string PerfMonMTSvc::get_info_from_file ( const std::string &  fileName,
const std::string &  fieldName 
) const

A few helper methods to get system information These should be carried to PerfMonMTUtils at some point.

Definition at line 903 of file PerfMonMTSvc.cxx.

904  {
905  // Helper function to read files of type Key : Value
906  // Returns the last instance if there are multiple matches
907  // This is because we use this method to get the processor count
908  std::string result{""};
909 
910  std::ifstream file{fileName};
911  std::string line{""};
912 
913  while (std::getline(file, line)) {
914  if (line.empty()) continue;
915  size_t splitIdx = line.find(':');
916  if (splitIdx != std::string::npos) {
917  std::string val = line.substr(splitIdx + 1);
918  if (val.empty()) continue;
919  if (line.size() >= fieldName.size() &&
920  line.compare(0, fieldName.size(), fieldName) == 0) {
921  result = std::move(val);
922  }
923  }
924  }
925 
926  file.close();
927 
928  return result;
929 }

◆ get_memory_info()

uint64_t PerfMonMTSvc::get_memory_info ( ) const

Definition at line 942 of file PerfMonMTSvc.cxx.

942  {
943  std::string val = get_info_from_file("/proc/meminfo","MemTotal");
944  if (val.empty()) return 0;
945  val.resize(val.size() - 3); // strip the trailing kB
946  return std::stoull(val);
947 }

◆ getCpuEfficiency()

int PerfMonMTSvc::getCpuEfficiency ( ) const
private

Definition at line 363 of file PerfMonMTSvc.cxx.

363  {
364 
365  // In AthenaMT only the event-loop is executed concurrently
366  // In this metric, we scale the event-loop wall-time by
367  // the number of slots to take the concurrency into account
368  // Then we divide the total cpu-time by this number
369  // It's A metric not THE metric...
370 
371  const double totalCpuTime =
372  m_snapshotData[CONFIGURE].getDeltaCPU() +
373  m_snapshotData[INITIALIZE].getDeltaCPU() +
374  m_snapshotData[FIRSTEVENT].getDeltaCPU() +
375  m_snapshotData[EXECUTE].getDeltaCPU() +
376  m_snapshotData[FINALIZE].getDeltaCPU();
377 
378  const double scaledWallTime =
379  m_snapshotData[CONFIGURE].getDeltaWall() * 1. +
380  m_snapshotData[INITIALIZE].getDeltaWall() * 1. +
381  m_snapshotData[FIRSTEVENT].getDeltaWall() * 1. +
382  m_snapshotData[EXECUTE].getDeltaWall() * m_numberOfSlots +
383  m_snapshotData[FINALIZE].getDeltaWall() * 1.;
384 
385  return ( scaledWallTime > 0 ? totalCpuTime / scaledWallTime * 100. : 0 );
386 
387 }

◆ handle()

void PerfMonMTSvc::handle ( const Incident &  incident)
overridevirtual

Incident service handle for post-finalize.

Definition at line 103 of file PerfMonMTSvc.cxx.

103  {
104  // Begin event processing
105  if (inc.type() == IncidentType::BeginEvent) {
106  // Lock for data integrity
107  std::lock_guard<std::mutex> lock(m_mutex_capture);
108 
109  // Increment the internal counter
110  m_eventCounter++;
111 
112  // Get current time in seconds
113  double currentTime = PMonMT::get_wall_time()*0.001;
114 
115  // Monitor
117  // Overwrite the last measurement time
118  m_checkPointTime = currentTime;
119 
120  // Capture
123  // Report instantly - no more than m_eventLoopMsgLimit times
127  }
128  }
129  }
130  // End event processing (as signaled by SG clean-up)
131  // By convention the first event is executed serially
132  // Therefore, we treat it a little bit differently
133  else if (m_eventCounter == 1 && inc.type() == "EndAlgorithms") {
134  // In AthenaMP w/ fork-after-initialize, the loop starts
135  // in the mother process but the first event is actually
136  // executed in the worker. Here, we try to work around this
137  // by resetting the first event measurement if we think
138  // we're in AthenaMP. This is not an ideal approach but
139  // gets the job done for the fork-after-initialize case.
140  if (m_motherPID != getpid()) {
141  m_snapshotData[FIRSTEVENT].m_tmp_cpu = 0;
142  m_snapshotData[FIRSTEVENT].m_memMonTmpMap["vmem"] = 0;
143  m_snapshotData[FIRSTEVENT].m_memMonTmpMap["pss"] = 0;
144  m_snapshotData[FIRSTEVENT].m_memMonTmpMap["rss"] = 0;
145  m_snapshotData[FIRSTEVENT].m_memMonTmpMap["swap"] = 0;
146  }
150  // Normally this flag is set in stopCompAud but we don't
151  // go in there unless m_doComponentLevelMonitoring is true.
152  // If it's false, we toggle it here but
153  // this is mostly for completeness since in that mode
154  // this flag is not really used at the moment.
156  m_isFirstEvent = false;
157  }
158  }
159  // This incident is fired by only some loop managers to signal the end of event processing
160  else if (inc.type() == "EndEvtLoop") {
161  m_isEvtLoopStopped = true;
164  }
165  // Finalize ourself and print the metrics in SvcPostFinalize
166  else if (inc.type() == IncidentType::SvcPostFinalize) {
167  // Final capture upon post-finalization
170 
171  // Report everything
172  report();
173  }
174  return;
175 }

◆ initialize()

StatusCode PerfMonMTSvc::initialize ( )
overridevirtual

Standard Gaudi Service initialization.

Configure the auditor

Definition at line 47 of file PerfMonMTSvc.cxx.

47  {
48  // Print where we are
49  ATH_MSG_INFO("Initializing " << name());
50 
51  // Set to be listener to SvcPostFinalize
52  ServiceHandle<IIncidentSvc> incSvc("IncidentSvc/IncidentSvc", name());
53  ATH_CHECK(incSvc.retrieve());
54  const long highestPriority = static_cast<long>(-1);
55  const long lowestPriority = 0;
56  incSvc->addListener(this, IncidentType::BeginEvent, highestPriority);
57  incSvc->addListener(this, "EndAlgorithms", lowestPriority);
58  incSvc->addListener(this, "EndEvtLoop", highestPriority);
59  incSvc->addListener(this, IncidentType::SvcPostFinalize);
60 
61  // Check if /proc exists, if not memory statistics are not available
62  const bool procExists = PMonMT::doesDirectoryExist("/proc");
63  if(!procExists) {
64  ATH_MSG_INFO("The system doesn't support /proc. Therefore, memory measurements are not available");
65  }
66 
67  // Print some information minimal information about our configuration
68  ATH_MSG_INFO("Service is configured for [" << m_numberOfThreads.toString() << "] threads " <<
69  "analyzing [" << m_numberOfSlots.toString() << "] events concurrently");
70  ATH_MSG_INFO("Component-level measurements are [" << (m_doComponentLevelMonitoring ? "Enabled" : "Disabled") << "]");
72  ATH_MSG_INFO(" >> Component-level memory monitoring in the event-loop is disabled in jobs with more than 1 thread");
73  }
74 
75  // Thread specific component-level data map
76  m_compLevelDataMapVec.resize(m_numberOfThreads+1); // Default construct
77 
78  // Set wall time offset
80  if (m_wallTimeOffset > 0) {
81  m_snapshotData[CONFIGURE].add2DeltaWall(-m_wallTimeOffset);
82  }
83 
85  ATH_CHECK(auditorSvc()->addAuditor("PerfMonMTAuditor"));
86 
87  return StatusCode::SUCCESS;
88 }

◆ report()

void PerfMonMTSvc::report ( )

Report the results.

Definition at line 392 of file PerfMonMTSvc.cxx.

392  {
393  // Write into log file
394  report2Log();
395 
396  // Write into JSON
397  if (m_reportResultsToJSON) {
398  report2JsonFile();
399  }
400 }

◆ report2JsonFile()

void PerfMonMTSvc::report2JsonFile ( )

Report to the JSON File.

Definition at line 624 of file PerfMonMTSvc.cxx.

624  {
625  nlohmann::json j;
626 
627  // CPU and Wall-time
628  report2JsonFile_Summary(j); // Snapshots
629 
630  // Memory
632  report2JsonFile_ComponentLevel(j); // Component-level
633  }
635  report2JsonFile_EventLevel(j); // Event-level
636  }
637 
638  // Write and close the JSON file
639  std::ofstream o(m_jsonFileName);
640  o << std::setw(4) << j << std::endl;
641  o.close();
642 
643  // Compress the JSON file into tar.gz
644  auto cmd = "tar -czf " + m_jsonFileName + ".tar.gz " + m_jsonFileName + ";";
645  int rc = std::system(cmd.c_str());
646  if(rc!=0) {
647  ATH_MSG_WARNING("Couldn't compress the JSON file...");
648  return;
649  }
650 
651  // Remove the uncompressed JSON file to save disk-space
652  rc = std::remove(m_jsonFileName.toString().c_str());
653  if(rc!=0) {
654  ATH_MSG_WARNING("Couldn't remove the uncompressed JSON file...");
655  return;
656  }
657 }

◆ report2JsonFile_ComponentLevel()

void PerfMonMTSvc::report2JsonFile_ComponentLevel ( nlohmann::json j) const

Definition at line 732 of file PerfMonMTSvc.cxx.

732  {
733 
734  for (const auto& dataMapPerStep : m_stdoutVec_serial) {
735 
736  for(const auto& meas : dataMapPerStep){
737 
738  const std::string step = meas.first.stepName;
739  const std::string component = meas.first.compName;
740  const uint64_t count = meas.second->getCallCount();
741  const double cpuTime = meas.second->getDeltaCPU();
742  const double wallTime = meas.second->getDeltaWall();
743  const int64_t vmem = meas.second->getDeltaVmem();
744  const int64_t mall = meas.second->getDeltaMalloc();
745 
746  j["componentLevel"][step][component] = {{"count", count},
747  {"cpuTime", cpuTime},
748  {"wallTime", wallTime},
749  {"vmem", vmem},
750  {"malloc", mall}};
751  }
752 
753  }
754 
755 }

◆ report2JsonFile_EventLevel()

void PerfMonMTSvc::report2JsonFile_EventLevel ( nlohmann::json j) const

Definition at line 757 of file PerfMonMTSvc.cxx.

757  {
758 
759  for (const auto& it : m_eventLevelData.getEventLevelData()) {
760 
761  const uint64_t event = it.first;
762  const double cpuTime = it.second.cpu_time;
763  const double wallTime = it.second.wall_time;
764  const int64_t vmem = it.second.mem_stats.at("vmem");
765  const int64_t rss = it.second.mem_stats.at("rss");
766  const int64_t pss = it.second.mem_stats.at("pss");
767  const int64_t swap = it.second.mem_stats.at("swap");
768 
769  j["eventLevel"][std::to_string(event)] = {{"cpuTime", cpuTime},
770  {"wallTime", wallTime},
771  {"vmem", vmem},
772  {"rss", rss},
773  {"pss", pss},
774  {"swap", swap}};
775 
776 
777  }
778 }

◆ report2JsonFile_Summary()

void PerfMonMTSvc::report2JsonFile_Summary ( nlohmann::json j) const

Definition at line 662 of file PerfMonMTSvc.cxx.

662  {
663 
664  // Report snapshot level results
665  for(int i=0; i < NSNAPSHOTS; i++){
666 
667  const std::string step = m_snapshotStepNames[i];
668  const double dCPU = m_snapshotData[i].getDeltaCPU();
669  const double dWall = m_snapshotData[i].getDeltaWall();
670  const double cpuUtil = dCPU / dWall;
671  const int64_t dVmem = m_snapshotData[i].getMemMonDeltaMap("vmem");
672  const int64_t dRss = m_snapshotData[i].getMemMonDeltaMap("rss");
673  const int64_t dPss = m_snapshotData[i].getMemMonDeltaMap("pss");
674  const int64_t dSwap = m_snapshotData[i].getMemMonDeltaMap("swap");
675 
676  j["summary"]["snapshotLevel"][step] = {{"dCPU", dCPU},
677  {"dWall", dWall},
678  {"cpuUtil", cpuUtil},
679  {"dVmem", dVmem},
680  {"dRss", dRss},
681  {"dPss", dPss},
682  {"dSwap", dSwap}};
683 
684  }
685 
686  // Report the total number of events
687  const int64_t nEvents = m_eventCounter;
688  j["summary"]["nEvents"] = nEvents;
689 
690  // Report Peaks
691  const int64_t vmemPeak = m_eventLevelData.getEventLevelMemoryMax("vmem");
692  const int64_t rssPeak = m_eventLevelData.getEventLevelMemoryMax("rss");
693  const int64_t pssPeak = m_eventLevelData.getEventLevelMemoryMax("pss");
694  const int64_t swapPeak = m_eventLevelData.getEventLevelMemoryMax("swap");
695 
696  j["summary"]["peaks"] = {{"vmemPeak", vmemPeak},
697  {"rssPeak", rssPeak},
698  {"pssPeak", pssPeak},
699  {"swapPeak", swapPeak}};
700 
701  // Report leak estimates
702  const int64_t vmemLeak = m_fit_vmem.slope();
703  const int64_t pssLeak = m_fit_pss.slope();
704  const int64_t nPoints = m_fit_vmem.nPoints();
705 
706  j["summary"]["leakEstimates"] = {{"vmemLeak", vmemLeak},
707  {"pssLeak", pssLeak},
708  {"nPoints", nPoints}};
709 
710  // Report Sys info
711  const std::string cpuModel = get_cpu_model_info();
712  const int coreNum = get_cpu_core_info();
713  const int64_t totMem = get_memory_info();
714 
715  j["summary"]["sysInfo"] = {{"cpuModel", cpuModel},
716  {"coreNum", coreNum},
717  {"totMem", totMem}};
718 
719  // Report Enviroment info
720  const std::string mallocLib = std::filesystem::path(PMonMT::symb2lib("malloc")).filename().string();
721  const std::string mathLib = std::filesystem::path(PMonMT::symb2lib("atan2")).filename().string();
722 
723  j["summary"]["envInfo"] = {{"mallocLib", mallocLib},
724  {"mathLib", mathLib}};
725 
726  // Report CPU utilization efficiency;
727  const int cpuUtilEff = getCpuEfficiency();
728  j["summary"]["misc"] = {{"cpuUtilEff", cpuUtilEff}};
729 
730 }

◆ report2Log()

void PerfMonMTSvc::report2Log ( )

Report to log.

Definition at line 405 of file PerfMonMTSvc.cxx.

405  {
406  // Header
408 
409  // Component-level
412  }
413 
414  // Event-level
417  }
418 
419  // Summary and system information
423 }

◆ report2Log_ComponentLevel()

void PerfMonMTSvc::report2Log_ComponentLevel ( )

Definition at line 445 of file PerfMonMTSvc.cxx.

445  {
446 
447  ATH_MSG_INFO("=======================================================================================");
448  ATH_MSG_INFO(" Component Level Monitoring ");
449  ATH_MSG_INFO("=======================================================================================");
450 
451  ATH_MSG_INFO(std::format("{:<10} {:<15} {:<25} {:<40} {:<55} {:<75}","Step", "Count", "CPU Time [ms]",
452  "Vmem [kB]", "Malloc [kB]", "Component"));
453 
454  ATH_MSG_INFO("---------------------------------------------------------------------------------------");
455 
456  aggregateSlotData(); // aggregate data from slots
457  divideData2Steps(); // divive data into steps for ordered printing
458 
459  for (auto vec_itr : m_stdoutVec_serial) {
460  // Sort the results by CPU time for the time being
461  std::vector<std::pair<PMonMT::StepComp, PMonMT::ComponentData*>> pairs;
462  for (auto itr = vec_itr.begin(); itr != vec_itr.end(); ++itr) pairs.push_back(*itr);
463 
464  sort(pairs.begin(), pairs.end(),
465  [=](std::pair<PMonMT::StepComp, PMonMT::ComponentData*>& a,
466  std::pair<PMonMT::StepComp, PMonMT::ComponentData*>& b) {
467  return a.second->getDeltaCPU() > b.second->getDeltaCPU();
468  });
469 
470  int counter = 0;
471  for (auto it : pairs) {
472  // Only write out a certian number of components
473  if (counter >= m_printNComps) {
474  break;
475  }
476  counter++;
477 
478  ATH_MSG_INFO(std::format("{:<10} {:<15} {:<25.2f} {:<40.0f} {:<55.0f} {:<75}",it.first.stepName,
479  it.second->getCallCount(),it.second->getDeltaCPU(),it.second->getDeltaVmem(),
480  it.second->getDeltaMalloc(),it.first.compName));
481  }
482  if(counter>0) {
483  ATH_MSG_INFO("=======================================================================================");
484  }
485  }
486 }

◆ report2Log_CpuInfo()

void PerfMonMTSvc::report2Log_CpuInfo ( ) const

Definition at line 595 of file PerfMonMTSvc.cxx.

595  {
596 
597  ATH_MSG_INFO(" System Information ");
598  ATH_MSG_INFO("=======================================================================================");
599 
600  ATH_MSG_INFO(std::format("{:<34} {}", "CPU Model:", get_cpu_model_info()));
601  ATH_MSG_INFO(std::format("{:<35} {}", "Number of Available Cores:", get_cpu_core_info()));
602  ATH_MSG_INFO(std::format("{:<35} {}", "Total Memory:", scaleMem(get_memory_info())));
603  ATH_MSG_INFO("=======================================================================================");
604 }

◆ report2Log_Description()

void PerfMonMTSvc::report2Log_Description ( ) const

Definition at line 428 of file PerfMonMTSvc.cxx.

428  {
429  ATH_MSG_INFO("=======================================================================================");
430  ATH_MSG_INFO(" PerfMonMTSvc Report ");
431  ATH_MSG_INFO("=======================================================================================");
432  if (m_reportResultsToJSON) {
433  ATH_MSG_INFO("*** Full set of information can also be found in: " << m_jsonFileName.toString());
434  ATH_MSG_INFO("*** In order to make plots using the results run the following commands:");
435  ATH_MSG_INFO("*** $ perfmonmt-plotter.py -i " << m_jsonFileName.toString());
436  ATH_MSG_INFO("*** In order to print tables using the results run the following commands:");
437  ATH_MSG_INFO("*** $ perfmonmt-printer.py -i " << m_jsonFileName.toString());
438  ATH_MSG_INFO("=======================================================================================");
439  }
440 }

◆ report2Log_EnvInfo()

void PerfMonMTSvc::report2Log_EnvInfo ( ) const

Definition at line 609 of file PerfMonMTSvc.cxx.

609  {
610 
611  ATH_MSG_INFO(" Environment Information ");
612  ATH_MSG_INFO("=======================================================================================");
613 
614  ATH_MSG_INFO(std::format("{:<35} {}","Malloc Library:", std::filesystem::path(PMonMT::symb2lib("malloc")).filename().string()));
615  ATH_MSG_INFO(std::format("{:<35} {}","Math Library:", std::filesystem::path(PMonMT::symb2lib("atan2")).filename().string()));
616 
617  ATH_MSG_INFO("=======================================================================================");
618 
619 }

◆ report2Log_EventLevel()

void PerfMonMTSvc::report2Log_EventLevel ( )

Definition at line 508 of file PerfMonMTSvc.cxx.

508  {
509 
510  ATH_MSG_INFO(" Event Level Monitoring ");
511  ATH_MSG_INFO(" (Only the first " << m_eventLoopMsgLimit.toString() <<
512  " and the last measurements are explicitly printed)");
513  ATH_MSG_INFO("=======================================================================================");
514 
515  ATH_MSG_INFO(std::format("{:<16} {:<12} {:<12} {:<12} {:<12} {:<12} {:<12}","Event", "CPU [s]",
516  "Wall [s]", "Vmem [kB]", "Rss [kB]", "Pss [kB]", "Swap [kB]"));
517 
518  ATH_MSG_INFO("---------------------------------------------------------------------------------------");
519 
520  m_eventLoopMsgCounter = 0; // reset counter
521  uint64_t nMeasurements = m_eventLevelData.getNMeasurements();
522 
523  for (const auto& it : m_eventLevelData.getEventLevelData()) {
524  // Print
525  if(m_eventLoopMsgCounter < m_eventLoopMsgLimit || m_eventLoopMsgCounter == nMeasurements - 1) {
527  ATH_MSG_INFO(std::format("{:=<87}", "..."));
528  }
529  ATH_MSG_INFO(std::format("{:<16} {:>12.2f} {:>12.2f} {:>12} {:>12} {:>12} {:>12}", it.first,
530  it.second.cpu_time * 0.001,it.second.wall_time * 0.001,it.second.mem_stats.at("vmem"),
531  it.second.mem_stats.at("rss"),it.second.mem_stats.at("pss"),it.second.mem_stats.at("swap")));
532  }
534  // Add to leak estimate
535  if (it.first >= m_memFitLowerLimit) {
536  m_fit_vmem.addPoint(it.first, it.second.mem_stats.at("vmem"));
537  m_fit_pss.addPoint(it.first, it.second.mem_stats.at("pss"));
538  }
539  }
540  ATH_MSG_INFO("=======================================================================================");
541 }

◆ report2Log_EventLevel_instant()

void PerfMonMTSvc::report2Log_EventLevel_instant ( ) const

Definition at line 491 of file PerfMonMTSvc.cxx.

491  {
494 
495  int64_t vmem = m_eventLevelData.getEventLevelMemory(m_eventCounter, "vmem");
499 
500  ATH_MSG_INFO("Event [" << std::setw(5) << m_eventCounter << "] CPU Time: " << scaleTime(cpu_time) <<
501  ", Wall Time: " << scaleTime(wall_time) << ", Vmem: " << scaleMem(vmem) <<
502  ", Rss: " << scaleMem(rss) << ", Pss: " << scaleMem(pss) << ", Swap: " << scaleMem(swap));
503 }

◆ report2Log_Summary()

void PerfMonMTSvc::report2Log_Summary ( )

Definition at line 546 of file PerfMonMTSvc.cxx.

546  {
547 
548  ATH_MSG_INFO(" Snapshots Summary ");
549  ATH_MSG_INFO("=======================================================================================");
550 
551  ATH_MSG_INFO(std::format("{:<13} {:<12} {:<12} {:<7} {:<11} {:<11} {:<11} {:<11}","Step",
552  "dCPU [s]","dWall [s]","<CPU>","dVmem [kB]","dRss [kB]","dPss [kB]","dSwap [kB]"));
553 
554  ATH_MSG_INFO("---------------------------------------------------------------------------------------");
555 
556  for (unsigned int idx = 0; idx < NSNAPSHOTS; idx++) {
557  ATH_MSG_INFO(std::format("{:<13} {:<12.2f} {:<12.2f} {:<7.2f} {:<11} {:<11} {:<11} {:<11}",
558  m_snapshotStepNames[idx], m_snapshotData[idx].getDeltaCPU() * 0.001,
559  m_snapshotData[idx].getDeltaWall() * 0.001,
560  m_snapshotData[idx].getDeltaCPU() / m_snapshotData[idx].getDeltaWall(),
561  m_snapshotData[idx].getMemMonDeltaMap("vmem"),m_snapshotData[idx].getMemMonDeltaMap("rss"),
562  m_snapshotData[idx].getMemMonDeltaMap("pss"),m_snapshotData[idx].getMemMonDeltaMap("swap")));
563  }
564 
565  ATH_MSG_INFO("***************************************************************************************");
566  const double cpu_exec_total = m_snapshotData[FIRSTEVENT].getDeltaCPU() + m_snapshotData[EXECUTE].getDeltaCPU();
567  const double wall_exec_total = m_snapshotData[FIRSTEVENT].getDeltaWall() + m_snapshotData[EXECUTE].getDeltaWall();
568 
569  ATH_MSG_INFO(std::format("{:<35} {}", "Number of events processed:", static_cast<int>(m_eventCounter)));
570  ATH_MSG_INFO(std::format("{:<35} {:.0f}", "CPU usage per event [ms]:",
571  (m_eventCounter > 0 ? cpu_exec_total / m_eventCounter : 0)));
572  ATH_MSG_INFO(std::format("{:<35} {:.3f}", "Events per second:",
573  (wall_exec_total > 0 ? m_eventCounter / wall_exec_total * 1000. : 0)));
574  ATH_MSG_INFO(std::format("{:<35} {}", "CPU utilization efficiency [%]:", getCpuEfficiency()));
576  ATH_MSG_INFO("***************************************************************************************");
577  ATH_MSG_INFO(std::format("{:<35} {}", "Max Vmem:", scaleMem(m_eventLevelData.getEventLevelMemoryMax("vmem"))));
578  ATH_MSG_INFO(std::format("{:<35} {}", "Max Rss:", scaleMem(m_eventLevelData.getEventLevelMemoryMax("rss"))));
579  ATH_MSG_INFO(std::format("{:<35} {}", "Max Pss:", scaleMem(m_eventLevelData.getEventLevelMemoryMax("pss"))));
580  ATH_MSG_INFO(std::format("{:<35} {}", "Max Swap:", scaleMem(m_eventLevelData.getEventLevelMemoryMax("swap"))));
581  ATH_MSG_INFO("***************************************************************************************");
582  ATH_MSG_INFO(std::format("{:<35} {}", "Leak estimate per event Vmem:", scaleMem(m_fit_vmem.slope())));
583  ATH_MSG_INFO(std::format("{:<35} {}", "Leak estimate per event Pss:", scaleMem(m_fit_pss.slope())));
584  ATH_MSG_INFO(" >> Estimated using the last " << m_fit_vmem.nPoints()
585  << " measurements from the Event Level Monitoring");
586  ATH_MSG_INFO(" >> Events prior to the first " << m_memFitLowerLimit.toString() << " are omitted...");
587  }
588 
589  ATH_MSG_INFO("=======================================================================================");
590 }

◆ scaleMem()

std::string PerfMonMTSvc::scaleMem ( int64_t  memMeas) const

Definition at line 873 of file PerfMonMTSvc.cxx.

873  {
874 
875  // Check if there is anything to be done
876  if (memMeas == 0) {
877  return "0.00 KB" ;
878  }
879 
880  // Prepare for the result
881  std::ostringstream ss;
882  ss << std::fixed;
883  ss << std::setprecision(2);
884 
885  // The input is in KB
886  std::vector<std::string> significance = {"KB", "MB", "GB", "TB"};
887 
888  // Get the absolute value
889  int64_t absMemMeas = std::abs(memMeas);
890  // Find the order, note that this is an int operation
891  int64_t order = std::log(absMemMeas)/std::log(1024);
892  // Compute the final value preserving the sign
893  double value = memMeas/std::pow(1024, order);
894  // Convert the result to a string
895  ss << value;
896 
897  return ss.str() + " " + significance[order];
898 }

◆ scaleTime()

std::string PerfMonMTSvc::scaleTime ( double  timeMeas) const

Definition at line 849 of file PerfMonMTSvc.cxx.

849  {
850  // Not a huge fan of this, we should eventually unify the types
851  // Just want to be explicit about what's happening
852  auto ms = static_cast<int64_t>(timeMeas);
853 
854  // Compute hrs and offset
855  auto hrs = ms / 3600000;
856  ms -= hrs * 3600000;
857  // Compute mins and offset
858  auto mins = ms / 60000;
859  ms -= mins * 60000;
860  // Compute secs and offset
861  auto secs = ms / 1000;
862  ms -= secs * 1000;
863 
864  // Primarily care about H:M:S
865  std::stringstream ss;
866  ss.fill('0');
867  ss << std::setw(2) << hrs << "h" <<
868  std::setw(2) << mins << "m" <<
869  std::setw(2) << secs << "s";
870  return ss.str();
871 }

◆ startAud()

void PerfMonMTSvc::startAud ( const std::string &  stepName,
const std::string &  compName,
const EventContext &  ctx 
)
overridevirtual

Start Auditing.

Definition at line 188 of file PerfMonMTSvc.cxx.

188  {
189  // Snapshots, i.e. Initialize, Event Loop, etc.
190  startSnapshotAud(stepName, compName);
191 
192  /*
193  * Perform component monitoring only if the user asked for it.
194  * By default we don't monitor a set of common components.
195  */
197  // Start component auditing (for tools/services we have to resort to thread-local context)
198  startCompAud(stepName, compName, ctx.valid() ? ctx : getCurrentContext());
199  }
200 }

◆ startCompAud()

void PerfMonMTSvc::startCompAud ( const std::string &  stepName,
const std::string &  compName,
const EventContext &  ctx 
)

Component Level Auditing: Take measurements at the beginning and at the end of each component call.

Definition at line 256 of file PerfMonMTSvc.cxx.

256  {
257  // Get the thread index
258  const unsigned int ithread = (ctx.valid() && tbb::this_task_arena::current_thread_index() > -1) ? tbb::this_task_arena::current_thread_index() : 0;
259 
260  // Memory measurement is only done outside the loop except when there is only a single thread
261  const bool doMem = !ctx.valid() || (m_numberOfThreads == 1);
262 
263  // Generate State
264  PMonMT::StepComp currentState = generate_state(stepName, compName);
265 
266  // Check if this is the first time calling if so create the mesurement data if not use the existing one.
267  // Metrics are collected per thread then aggregated before reporting
268  data_map_unique_t& compLevelDataMap = m_compLevelDataMapVec[ithread];
269  if(compLevelDataMap.find(currentState) == compLevelDataMap.end()) {
270  compLevelDataMap.insert({currentState, std::make_unique<PMonMT::ComponentData>()});
271  }
272 
273  // Capture and store
275  meas.capture(); // No memory in the event-loop
276  if (doMem) {
277  // we made sure this is only run outside event loop or single-threaded
278  [[maybe_unused]] bool dummy ATLAS_THREAD_SAFE = meas.capture_memory();
279  }
280 
281  compLevelDataMap[currentState]->addPointStart(meas, doMem);
282 
283  // Debug
284  ATH_MSG_DEBUG("Start Audit - Component " << compName << " , "
285  "Step " << stepName << " , "
286  "Event " << ctx.evt() << " , "
287  "Slot " << ctx.slot() << " , "
288  "Context " << ctx.valid() << " , "
289  "Thread " << ithread << " , "
290  "Cpu " << meas.cpu_time << " ms, "
291  "Wall " << meas.wall_time << " ms, "
292  "Vmem " << meas.vmem << " kb, "
293  "Malloc " << meas.malloc << " kb");
294 }

◆ startSnapshotAud()

void PerfMonMTSvc::startSnapshotAud ( const std::string &  stepName,
const std::string &  compName 
)

Snapshot Auditing: Take snapshots at the beginning and at the end of each step.

Definition at line 219 of file PerfMonMTSvc.cxx.

219  {
220  // Last thing to be called before the event loop begins
221  if (compName == "AthOutSeq" && stepName == "Start") {
224  m_isFirstEvent = true;
225  }
226 
227  // Last thing to be called before finalize step begins
228  if (compName == "AthMasterSeq" && stepName == "Finalize") {
231  }
232 }

◆ stopAud()

void PerfMonMTSvc::stopAud ( const std::string &  stepName,
const std::string &  compName,
const EventContext &  ctx 
)
overridevirtual

Stop Auditing.

Definition at line 205 of file PerfMonMTSvc.cxx.

205  {
206  // Snapshots, i.e. Initialize, Event Loop, etc.
207  stopSnapshotAud(stepName, compName);
208 
209  // Check if we should monitor this component
211  // Stop component auditing (for tools/services we have to resort to thread-local context)
212  stopCompAud(stepName, compName, ctx.valid() ? ctx : getCurrentContext());
213  }
214 }

◆ stopCompAud()

void PerfMonMTSvc::stopCompAud ( const std::string &  stepName,
const std::string &  compName,
const EventContext &  ctx 
)

Definition at line 299 of file PerfMonMTSvc.cxx.

299  {
300  // Get the thread index
301  const unsigned int ithread = (ctx.valid() && tbb::this_task_arena::current_thread_index() > -1) ? tbb::this_task_arena::current_thread_index() : 0;
302 
303  // Memory measurement is only done outside the loop except when there is only a single thread
304  const bool doMem = !ctx.valid() || (m_numberOfThreads == 1);
305 
306  // Capture
308  meas.capture(); // No memory in the event-loop
309  if (doMem) {
310  // we made sure this is only run outside event loop or single-threaded
311  [[maybe_unused]] bool dummy ATLAS_THREAD_SAFE = meas.capture_memory();
312  }
313 
314  // Generate State
315  PMonMT::StepComp currentState = generate_state(stepName, compName);
316 
317  // Store
318  data_map_unique_t& compLevelDataMap = m_compLevelDataMapVec[ithread];
319  auto itr = compLevelDataMap.find(currentState);
320 
321  // This can happen if we never got the startCompAud call.
322  // Usually because Gaudi's AuditorSvc was not fully initialized yet.
323  if (itr==compLevelDataMap.end()) return;
324 
325  itr->second->addPointStop(meas, doMem);
326 
327  // Once the first time IncidentProcAlg3 is excuted, toggle m_isFirstEvent to false.
328  // Doing it this way, instead of at EndAlgorithms incident, makes sure there is no
329  // mismatch in start-stop calls to IncidentProcAlg3.
330  // It's a little ad-hoc but I don't think this workflow will change much anytime soon.
331  if ( m_isFirstEvent && compName == "IncidentProcAlg3" && stepName == "Execute") {
332  m_isFirstEvent = false;
333  }
334 
335  // Debug
336  ATH_MSG_DEBUG("Stop Audit - Component " << compName << " , "
337  "Step " << stepName << " , "
338  "Event " << ctx.evt() << " , "
339  "Slot " << ctx.slot() << " , "
340  "Context " << ctx.valid() << " , "
341  "Thread " << ithread << " , "
342  "Cpu (" << compLevelDataMap[currentState]->m_tmp_cpu << ":"
343  << meas.cpu_time << ":"
344  << (meas.cpu_time - compLevelDataMap[currentState]->m_tmp_cpu) << ":"
345  << compLevelDataMap[currentState]->m_delta_cpu << ") ms, "
346  "Wall (" << compLevelDataMap[currentState]->m_tmp_wall << ":"
347  << meas.wall_time << ":"
348  << (meas.wall_time - compLevelDataMap[currentState]->m_tmp_wall) << ":"
349  << compLevelDataMap[currentState]->m_delta_wall << ") ms, "
350  "Vmem (" << compLevelDataMap[currentState]->m_tmp_vmem << ":"
351  << meas.vmem << ":"
352  << (meas.vmem - compLevelDataMap[currentState]->m_tmp_vmem) << ":"
353  << compLevelDataMap[currentState]->m_delta_vmem << ") kb, "
354  "Malloc (" << compLevelDataMap[currentState]->m_tmp_malloc << ":"
355  << meas.malloc << ":"
356  << (meas.malloc - compLevelDataMap[currentState]->m_tmp_malloc) << ":"
357  << compLevelDataMap[currentState]->m_delta_malloc << ") kb");
358 }

◆ stopSnapshotAud()

void PerfMonMTSvc::stopSnapshotAud ( const std::string &  stepName,
const std::string &  compName 
)

Definition at line 237 of file PerfMonMTSvc.cxx.

237  {
238  // First thing to be called after the initialize step ends
239  if (compName == "AthMasterSeq" && stepName == "Initialize") {
242  }
243 
244  // First thing to be called after the event loop ends
245  // Some loop managers fire a dedicated incident to signal the end of the event loop
246  // That preceeds the AthMasterSeq Stop and if it is already handled we don't do anything here
247  if (compName == "AthMasterSeq" && stepName == "Stop" && m_eventCounter > 0 && !m_isEvtLoopStopped) {
250  }
251 }

Member Data Documentation

◆ m_checkPointThreshold

Gaudi::Property<uint64_t> PerfMonMTSvc::m_checkPointThreshold
private
Initial value:
{
this, "checkPointThreshold", 30,
"Least amount of time (in seconds) between event-level checks."}

Frequency of event level monitoring.

Definition at line 137 of file PerfMonMTSvc.h.

◆ m_checkPointTime

std::atomic<double> PerfMonMTSvc::m_checkPointTime
private

Definition at line 181 of file PerfMonMTSvc.h.

◆ m_compLevelDataMap

data_map_t PerfMonMTSvc::m_compLevelDataMap
private

Definition at line 194 of file PerfMonMTSvc.h.

◆ m_compLevelDataMap_1stevt

data_map_t PerfMonMTSvc::m_compLevelDataMap_1stevt
private

Definition at line 200 of file PerfMonMTSvc.h.

◆ m_compLevelDataMap_cbk

data_map_t PerfMonMTSvc::m_compLevelDataMap_cbk
private

Definition at line 204 of file PerfMonMTSvc.h.

◆ m_compLevelDataMap_evt

data_map_t PerfMonMTSvc::m_compLevelDataMap_evt
private

Definition at line 201 of file PerfMonMTSvc.h.

◆ m_compLevelDataMap_fin

data_map_t PerfMonMTSvc::m_compLevelDataMap_fin
private

Definition at line 202 of file PerfMonMTSvc.h.

◆ m_compLevelDataMap_ini

data_map_t PerfMonMTSvc::m_compLevelDataMap_ini
private

Definition at line 199 of file PerfMonMTSvc.h.

◆ m_compLevelDataMap_plp

data_map_t PerfMonMTSvc::m_compLevelDataMap_plp
private

Definition at line 203 of file PerfMonMTSvc.h.

◆ m_compLevelDataMapVec

std::vector<data_map_unique_t> PerfMonMTSvc::m_compLevelDataMapVec
private

Definition at line 198 of file PerfMonMTSvc.h.

◆ m_doComponentLevelMonitoring

Gaudi::Property<bool> PerfMonMTSvc::m_doComponentLevelMonitoring
private
Initial value:
{
this, "doComponentLevelMonitoring", false,
"True if component level monitoring is enabled, false o/w. Component monitoring may cause a decrease in the "
"performance due to the usage of locks."}

Do component level monitoring.

Definition at line 120 of file PerfMonMTSvc.h.

◆ m_doEventLoopMonitoring

Gaudi::Property<bool> PerfMonMTSvc::m_doEventLoopMonitoring
private
Initial value:
{
this, "doEventLoopMonitoring", true,
"True if event loop monitoring is enabled, false o/w. Event loop monitoring may cause a decrease in the "
"performance due to the usage of locks."}

Do event loop monitoring.

Definition at line 115 of file PerfMonMTSvc.h.

◆ m_eventCounter

std::atomic<uint64_t> PerfMonMTSvc::m_eventCounter
private

Definition at line 175 of file PerfMonMTSvc.h.

◆ m_eventLevelData

PMonMT::EventLevelData PerfMonMTSvc::m_eventLevelData {}
private

Definition at line 166 of file PerfMonMTSvc.h.

◆ m_eventLoopMsgCounter

std::atomic<uint64_t> PerfMonMTSvc::m_eventLoopMsgCounter
private

Definition at line 178 of file PerfMonMTSvc.h.

◆ m_eventLoopMsgLimit

Gaudi::Property<uint64_t> PerfMonMTSvc::m_eventLoopMsgLimit {this, "eventLoopMsgLimit", 10, "Maximum number of event-level messages."}
private

Set the number of messages for the event-level report.

Definition at line 150 of file PerfMonMTSvc.h.

◆ m_exclusionSet

const std::set<std::string> PerfMonMTSvc::m_exclusionSet
private
Initial value:
= {"AthMasterSeq", "AthAlgEvtSeq", "AthAllAlgSeq", "AthAlgSeq", "AthOutSeq",
"AthCondSeq", "AthBeginSeq", "AthEndSeq", "AthenaEventLoopMgr", "AthenaHiveEventLoopMgr", "AthMpEvtLoopMgr", "AthenaMtesEventLoopMgr",
"PerfMonMTSvc"}

Exclude some common components from monitoring In the future this might be converted to a inclusion set which would allow user to monitor only a set of algorithms...

Definition at line 155 of file PerfMonMTSvc.h.

◆ m_fit_pss

PerfMon::LinFitSglPass PerfMonMTSvc::m_fit_pss
private

Definition at line 210 of file PerfMonMTSvc.h.

◆ m_fit_vmem

PerfMon::LinFitSglPass PerfMonMTSvc::m_fit_vmem
private

Definition at line 209 of file PerfMonMTSvc.h.

◆ m_isEvtLoopStopped

std::atomic<bool> PerfMonMTSvc::m_isEvtLoopStopped
private

Definition at line 184 of file PerfMonMTSvc.h.

◆ m_isFirstEvent

std::atomic<bool> PerfMonMTSvc::m_isFirstEvent
private

Definition at line 172 of file PerfMonMTSvc.h.

◆ m_jsonFileName

Gaudi::Property<std::string> PerfMonMTSvc::m_jsonFileName
private
Initial value:
{this, "jsonFileName", "PerfMonMTSvc_result.json",
"Name of the JSON file that contains the results."}

Name of the JSON file.

Definition at line 127 of file PerfMonMTSvc.h.

◆ m_measurementEvents

PMonMT::SnapshotMeasurement PerfMonMTSvc::m_measurementEvents
private

Measurement to capture events.

Definition at line 112 of file PerfMonMTSvc.h.

◆ m_measurementSnapshots

PMonMT::SnapshotMeasurement PerfMonMTSvc::m_measurementSnapshots
private

Measurement to capture snapshots.

Definition at line 109 of file PerfMonMTSvc.h.

◆ m_memFitLowerLimit

Gaudi::Property<uint64_t> PerfMonMTSvc::m_memFitLowerLimit
private
Initial value:
{
this, "memFitLowerLimit", 25,
"Lower limit (in number of events) for the memory fit."}

Lower limit (in number of events) for the memory fit.

Definition at line 133 of file PerfMonMTSvc.h.

◆ m_motherPID

int PerfMonMTSvc::m_motherPID
private

Snapshots data.

Definition at line 160 of file PerfMonMTSvc.h.

◆ m_mutex_capture

std::mutex PerfMonMTSvc::m_mutex_capture
private

Definition at line 169 of file PerfMonMTSvc.h.

◆ m_numberOfSlots

Gaudi::Property<int> PerfMonMTSvc::m_numberOfSlots {this, "numberOfSlots", 1, "Number of slots in the job."}
private

Get the number of slots.

Definition at line 148 of file PerfMonMTSvc.h.

◆ m_numberOfThreads

Gaudi::Property<int> PerfMonMTSvc::m_numberOfThreads {this, "numberOfThreads", 1, "Number of threads in the job."}
private

Get the number of threads.

Definition at line 146 of file PerfMonMTSvc.h.

◆ m_printDetailedTables

Gaudi::Property<bool> PerfMonMTSvc::m_printDetailedTables
private
Initial value:
{this, "printDetailedTables", true,
"Print detailed component-level metrics."}

Print detailed tables.

Definition at line 130 of file PerfMonMTSvc.h.

◆ m_printNComps

Gaudi::Property<int> PerfMonMTSvc::m_printNComps
private
Initial value:
{
this, "printNComps", 50, "Maximum number of components to be printed."}

Print the top N components.

Definition at line 143 of file PerfMonMTSvc.h.

◆ m_reportResultsToJSON

Gaudi::Property<bool> PerfMonMTSvc::m_reportResultsToJSON {this, "reportResultsToJSON", true, "Report results into the json file."}
private

Report results to JSON.

Definition at line 125 of file PerfMonMTSvc.h.

◆ m_snapshotData

std::vector<PMonMT::SnapshotData> PerfMonMTSvc::m_snapshotData
private

Definition at line 161 of file PerfMonMTSvc.h.

◆ m_snapshotStepNames

std::vector<std::string> PerfMonMTSvc::m_snapshotStepNames = {"Configure", "Initialize", "FirstEvent", "Execute", "Finalize"}
private

Definition at line 162 of file PerfMonMTSvc.h.

◆ m_stdoutVec_serial

std::vector<data_map_t> PerfMonMTSvc::m_stdoutVec_serial
private

Definition at line 206 of file PerfMonMTSvc.h.

◆ m_wallTimeOffset

Gaudi::Property<double> PerfMonMTSvc::m_wallTimeOffset {this, "wallTimeOffset", 0, "Job start wall time in miliseconds."}
private

Offset for the wall-time, comes from configuration.

Definition at line 141 of file PerfMonMTSvc.h.


The documentation for this class was generated from the following files:
nEvents
const int nEvents
Definition: fbtTestBasics.cxx:78
PerfMonMTSvc::m_doComponentLevelMonitoring
Gaudi::Property< bool > m_doComponentLevelMonitoring
Do component level monitoring.
Definition: PerfMonMTSvc.h:120
PerfMonMTSvc::report2JsonFile
void report2JsonFile()
Report to the JSON File.
Definition: PerfMonMTSvc.cxx:624
PerfMonMTSvc::m_isFirstEvent
std::atomic< bool > m_isFirstEvent
Definition: PerfMonMTSvc.h:172
PerfMonMTSvc::stopSnapshotAud
void stopSnapshotAud(const std::string &stepName, const std::string &compName)
Definition: PerfMonMTSvc.cxx:237
PerfMonMTSvc::report2Log_Description
void report2Log_Description() const
Definition: PerfMonMTSvc.cxx:428
PerfMonMTSvc::divideData2Steps
void divideData2Steps()
Definition: PerfMonMTSvc.cxx:826
PerfMonMTSvc::getCpuEfficiency
int getCpuEfficiency() const
Definition: PerfMonMTSvc.cxx:363
AddEmptyComponent.compName
compName
Definition: AddEmptyComponent.py:32
PerfMonMTSvc::report2Log_ComponentLevel
void report2Log_ComponentLevel()
Definition: PerfMonMTSvc.cxx:445
get_generator_info.result
result
Definition: get_generator_info.py:21
PMonMT::symb2lib
const char * symb2lib(const char *symbol, const char *failstr)
Definition: PerfMonMTUtils.h:435
PerfMonMTSvc::m_compLevelDataMap_fin
data_map_t m_compLevelDataMap_fin
Definition: PerfMonMTSvc.h:202
athena.path
path
python interpreter configuration --------------------------------------—
Definition: athena.py:128
PerfMonMTSvc::m_numberOfThreads
Gaudi::Property< int > m_numberOfThreads
Get the number of threads.
Definition: PerfMonMTSvc.h:146
PowhegControl_ttHplus_NLO.ss
ss
Definition: PowhegControl_ttHplus_NLO.py:83
PerfMonMTSvc::m_measurementSnapshots
PMonMT::SnapshotMeasurement m_measurementSnapshots
Measurement to capture snapshots.
Definition: PerfMonMTSvc.h:109
PerfMonMTSvc::m_compLevelDataMap
data_map_t m_compLevelDataMap
Definition: PerfMonMTSvc.h:194
vtune_athena.format
format
Definition: vtune_athena.py:14
ATH_MSG_INFO
#define ATH_MSG_INFO(x)
Definition: AthMsgStreamMacros.h:31
PerfMon::LinFitSglPass::addPoint
void addPoint(const double &, const double &)
Definition: LinFitSglPass.h:56
PerfMonMTSvc::m_motherPID
int m_motherPID
Snapshots data.
Definition: PerfMonMTSvc.h:160
PerfMonMTSvc::report2Log_CpuInfo
void report2Log_CpuInfo() const
Definition: PerfMonMTSvc.cxx:595
PerfMonMTSvc::report2Log_EnvInfo
void report2Log_EnvInfo() const
Definition: PerfMonMTSvc.cxx:609
json
nlohmann::json json
Definition: HistogramDef.cxx:9
PerfMonMTSvc::m_eventCounter
std::atomic< uint64_t > m_eventCounter
Definition: PerfMonMTSvc.h:175
PerfMonMTSvc::get_cpu_core_info
int get_cpu_core_info() const
Definition: PerfMonMTSvc.cxx:936
rerun_display.cmd
string cmd
Definition: rerun_display.py:67
PerfMonMTSvc::m_checkPointTime
std::atomic< double > m_checkPointTime
Definition: PerfMonMTSvc.h:181
PerfMonMTSvc::m_wallTimeOffset
Gaudi::Property< double > m_wallTimeOffset
Offset for the wall-time, comes from configuration.
Definition: PerfMonMTSvc.h:141
PerfMonMTSvc::report2JsonFile_ComponentLevel
void report2JsonFile_ComponentLevel(nlohmann::json &j) const
Definition: PerfMonMTSvc.cxx:732
skel.it
it
Definition: skel.GENtoEVGEN.py:407
PerfMonMTSvc::m_fit_vmem
PerfMon::LinFitSglPass m_fit_vmem
Definition: PerfMonMTSvc.h:209
PMonMT::StepComp
Definition: PerfMonMTUtils.h:63
PerfMonMTSvc::get_cpu_model_info
std::string get_cpu_model_info() const
Definition: PerfMonMTSvc.cxx:931
PerfMonMTSvc::stopCompAud
void stopCompAud(const std::string &stepName, const std::string &compName, const EventContext &ctx)
Definition: PerfMonMTSvc.cxx:299
PerfMonMTSvc::CONFIGURE
@ CONFIGURE
Definition: PerfMonMTSvc.h:163
PerfMonMTSvc::m_measurementEvents
PMonMT::SnapshotMeasurement m_measurementEvents
Measurement to capture events.
Definition: PerfMonMTSvc.h:112
athena.value
value
Definition: athena.py:124
python.RatesEmulationExample.lock
lock
Definition: RatesEmulationExample.py:148
PerfMonMTSvc::m_eventLoopMsgLimit
Gaudi::Property< uint64_t > m_eventLoopMsgLimit
Set the number of messages for the event-level report.
Definition: PerfMonMTSvc.h:150
PerfMonMTSvc::FINALIZE
@ FINALIZE
Definition: PerfMonMTSvc.h:163
PerfMonMTSvc::report2Log_EventLevel
void report2Log_EventLevel()
Definition: PerfMonMTSvc.cxx:508
PerfMonMTSvc::startSnapshotAud
void startSnapshotAud(const std::string &stepName, const std::string &compName)
Snapshot Auditing: Take snapshots at the beginning and at the end of each step.
Definition: PerfMonMTSvc.cxx:219
PerfMonMTSvc::m_isEvtLoopStopped
std::atomic< bool > m_isEvtLoopStopped
Definition: PerfMonMTSvc.h:184
PerfMonMTSvc::report2Log_Summary
void report2Log_Summary()
Definition: PerfMonMTSvc.cxx:546
dq_defect_bulk_create_defects.line
line
Definition: dq_defect_bulk_create_defects.py:27
std::sort
void sort(typename DataModel_detail::iterator< DVL > beg, typename DataModel_detail::iterator< DVL > end)
Specialization of sort for DataVector/List.
Definition: DVL_algorithms.h:554
PerfMonMTSvc::m_compLevelDataMap_ini
data_map_t m_compLevelDataMap_ini
Definition: PerfMonMTSvc.h:199
XMLtoHeader.count
count
Definition: XMLtoHeader.py:84
PMonMT::ComponentMeasurement::cpu_time
double cpu_time
Definition: PerfMonMTUtils.h:77
PerfMonMTSvc::m_compLevelDataMap_evt
data_map_t m_compLevelDataMap_evt
Definition: PerfMonMTSvc.h:201
python.CreateTierZeroArgdict.pairs
pairs
Definition: CreateTierZeroArgdict.py:201
PerfMonMTSvc::m_snapshotData
std::vector< PMonMT::SnapshotData > m_snapshotData
Definition: PerfMonMTSvc.h:161
PMonMT::ComponentMeasurement
Definition: PerfMonMTUtils.h:74
PerfMonMTSvc::INITIALIZE
@ INITIALIZE
Definition: PerfMonMTSvc.h:163
PerfMonMTSvc::report2Log_EventLevel_instant
void report2Log_EventLevel_instant() const
Definition: PerfMonMTSvc.cxx:491
PixelModuleFeMask_create_db.remove
string remove
Definition: PixelModuleFeMask_create_db.py:83
PMonMT::get_wall_time
double get_wall_time()
Definition: PerfMonMTUtils.h:330
PMonMT::EventLevelData::getEventLevelData
const EventMeasMap_t & getEventLevelData() const
Definition: PerfMonMTUtils.h:220
PerfMonMTSvc::m_fit_pss
PerfMon::LinFitSglPass m_fit_pss
Definition: PerfMonMTSvc.h:210
PerfMonMTSvc::report2Log
void report2Log()
Report to log.
Definition: PerfMonMTSvc.cxx:405
event
POOL::TEvent event(POOL::TEvent::kClassAccess)
PMonMT::ComponentMeasurement::wall_time
double wall_time
Definition: PerfMonMTUtils.h:77
lumiFormat.i
int i
Definition: lumiFormat.py:85
PMonMT::EventLevelData::getEventLevelMemoryMax
int64_t getEventLevelMemoryMax(const std::string &stat) const
Definition: PerfMonMTUtils.h:241
PerfMonMTSvc::m_reportResultsToJSON
Gaudi::Property< bool > m_reportResultsToJSON
Report results to JSON.
Definition: PerfMonMTSvc.h:125
ATH_MSG_DEBUG
#define ATH_MSG_DEBUG(x)
Definition: AthMsgStreamMacros.h:29
mc.order
order
Configure Herwig7.
Definition: mc.Herwig7_Dijet.py:12
PerfMonMTSvc::m_jsonFileName
Gaudi::Property< std::string > m_jsonFileName
Name of the JSON file.
Definition: PerfMonMTSvc.h:127
taskman.fieldName
fieldName
Definition: taskman.py:489
CalibDbCompareRT.dummy
dummy
Definition: CalibDbCompareRT.py:59
PerfMonMTSvc::report2JsonFile_EventLevel
void report2JsonFile_EventLevel(nlohmann::json &j) const
Definition: PerfMonMTSvc.cxx:757
file
TFile * file
Definition: tile_monitor.h:29
PerfMonMTSvc::m_eventLoopMsgCounter
std::atomic< uint64_t > m_eventLoopMsgCounter
Definition: PerfMonMTSvc.h:178
xAOD::uint64_t
uint64_t
Definition: EventInfo_v1.cxx:123
ATH_CHECK
#define ATH_CHECK
Definition: AthCheckMacros.h:40
PMonMT::SnapshotMeasurement::capture
void capture()
Definition: PerfMonMTUtils.h:177
PerfMonMTSvc::scaleMem
std::string scaleMem(int64_t memMeas) const
Definition: PerfMonMTSvc.cxx:873
PerfMonMTSvc::scaleTime
std::string scaleTime(double timeMeas) const
Definition: PerfMonMTSvc.cxx:849
PerfMonMTSvc::get_memory_info
uint64_t get_memory_info() const
Definition: PerfMonMTSvc.cxx:942
WriteCalibToCool.swap
swap
Definition: WriteCalibToCool.py:94
PerfMonMTSvc::NSNAPSHOTS
@ NSNAPSHOTS
Definition: PerfMonMTSvc.h:163
PerfMon::LinFitSglPass::nPoints
unsigned nPoints() const
Definition: LinFitSglPass.h:31
PerfMonMTSvc::report2JsonFile_Summary
void report2JsonFile_Summary(nlohmann::json &j) const
Definition: PerfMonMTSvc.cxx:662
PMonMT::StepComp::stepName
std::string stepName
Definition: PerfMonMTUtils.h:64
PerfMonMTSvc::m_snapshotStepNames
std::vector< std::string > m_snapshotStepNames
Definition: PerfMonMTSvc.h:162
PerfMonMTSvc::m_exclusionSet
const std::set< std::string > m_exclusionSet
Exclude some common components from monitoring In the future this might be converted to a inclusion s...
Definition: PerfMonMTSvc.h:155
name
std::string name
Definition: Control/AthContainers/Root/debug.cxx:240
ActsTrk::to_string
std::string to_string(const DetectorType &type)
Definition: GeometryDefs.h:34
plotBeamSpotMon.b
b
Definition: plotBeamSpotMon.py:76
PMonMT::EventLevelData::getEventLevelMemory
int64_t getEventLevelMemory(const uint64_t event_count, const std::string &stat) const
Definition: PerfMonMTUtils.h:236
PerfMonMTSvc::m_mutex_capture
std::mutex m_mutex_capture
Definition: PerfMonMTSvc.h:169
PMonMT::ComponentMeasurement::vmem
double vmem
Definition: PerfMonMTUtils.h:78
PerfMonMTSvc::m_numberOfSlots
Gaudi::Property< int > m_numberOfSlots
Get the number of slots.
Definition: PerfMonMTSvc.h:148
PerfMonMTSvc::m_printNComps
Gaudi::Property< int > m_printNComps
Print the top N components.
Definition: PerfMonMTSvc.h:143
PerfMonMTSvc::m_printDetailedTables
Gaudi::Property< bool > m_printDetailedTables
Print detailed tables.
Definition: PerfMonMTSvc.h:130
PerfMonMTSvc::aggregateSlotData
void aggregateSlotData()
A few helper functions.
Definition: PerfMonMTSvc.cxx:793
PMonMT::EventLevelData::set_wall_time_offset
void set_wall_time_offset(const double wall_time_offset)
Definition: PerfMonMTUtils.h:218
PerfMonMTSvc::m_stdoutVec_serial
std::vector< data_map_t > m_stdoutVec_serial
Definition: PerfMonMTSvc.h:206
PerfMonMTSvc::m_eventLevelData
PMonMT::EventLevelData m_eventLevelData
Definition: PerfMonMTSvc.h:166
a
TList * a
Definition: liststreamerinfos.cxx:10
PMonMT::EventLevelData::getNMeasurements
uint64_t getNMeasurements() const
Definition: PerfMonMTUtils.h:224
PerfMonMTSvc::m_doEventLoopMonitoring
Gaudi::Property< bool > m_doEventLoopMonitoring
Do event loop monitoring.
Definition: PerfMonMTSvc.h:115
PerfMonMTSvc::EXECUTE
@ EXECUTE
Definition: PerfMonMTSvc.h:163
PerfMonMTSvc::m_checkPointThreshold
Gaudi::Property< uint64_t > m_checkPointThreshold
Frequency of event level monitoring.
Definition: PerfMonMTSvc.h:137
ATH_MSG_WARNING
#define ATH_MSG_WARNING(x)
Definition: AthMsgStreamMacros.h:32
Pythia8_RapidityOrderMPI.val
val
Definition: Pythia8_RapidityOrderMPI.py:14
PMonMT::ComponentMeasurement::malloc
double malloc
Definition: PerfMonMTUtils.h:78
PerfMon::LinFitSglPass::slope
double slope() const
Definition: LinFitSglPass.h:75
python.CaloCondTools.log
log
Definition: CaloCondTools.py:20
LArNewCalib_DelayDump_OFC_Cali.idx
idx
Definition: LArNewCalib_DelayDump_OFC_Cali.py:69
CaloCellTimeCorrFiller.filename
filename
Definition: CaloCellTimeCorrFiller.py:23
PerfMonMTSvc::m_compLevelDataMap_1stevt
data_map_t m_compLevelDataMap_1stevt
Definition: PerfMonMTSvc.h:200
PerfMonMTSvc::m_memFitLowerLimit
Gaudi::Property< uint64_t > m_memFitLowerLimit
Lower limit (in number of events) for the memory fit.
Definition: PerfMonMTSvc.h:133
LArCellBinning.step
step
Definition: LArCellBinning.py:158
PMonMT::StepComp::compName
std::string compName
Definition: PerfMonMTUtils.h:65
PerfMonMTSvc::FIRSTEVENT
@ FIRSTEVENT
Definition: PerfMonMTSvc.h:163
ATLAS_THREAD_SAFE
#define ATLAS_THREAD_SAFE
Definition: checker_macros.h:211
jobOptions.fileName
fileName
Definition: jobOptions.SuperChic_ALP2.py:39
test_pyathena.counter
counter
Definition: test_pyathena.py:15
PerfMonMTSvc::get_info_from_file
std::string get_info_from_file(const std::string &fileName, const std::string &fieldName) const
A few helper methods to get system information These should be carried to PerfMonMTUtils at some poin...
Definition: PerfMonMTSvc.cxx:903
pow
constexpr int pow(int base, int exp) noexcept
Definition: ap_fixedTest.cxx:15
PMonMT::doesDirectoryExist
bool doesDirectoryExist(const std::string &dir)
Definition: PerfMonMTUtils.h:415
PerfMonMTSvc::m_compLevelDataMapVec
std::vector< data_map_unique_t > m_compLevelDataMapVec
Definition: PerfMonMTSvc.h:198
PerfMonMTSvc::m_compLevelDataMap_cbk
data_map_t m_compLevelDataMap_cbk
Definition: PerfMonMTSvc.h:204
PerfMonMTSvc::data_map_unique_t
std::map< PMonMT::StepComp, std::unique_ptr< PMonMT::ComponentData > > data_map_unique_t
Definition: PerfMonMTSvc.h:190
PerfMonMTSvc::generate_state
PMonMT::StepComp generate_state(const std::string &stepName, const std::string &compName) const
Definition: PerfMonMTSvc.cxx:783
PMonMT::ComponentMeasurement::capture
void capture()
Definition: PerfMonMTUtils.h:81
python.trfValidateRootFile.rc
rc
Definition: trfValidateRootFile.py:375
PerfMonMTSvc::report
void report()
Report the results.
Definition: PerfMonMTSvc.cxx:392
PerfMonMTSvc::startCompAud
void startCompAud(const std::string &stepName, const std::string &compName, const EventContext &ctx)
Component Level Auditing: Take measurements at the beginning and at the end of each component call.
Definition: PerfMonMTSvc.cxx:256
PMonMT::EventLevelData::getEventLevelWallTime
double getEventLevelWallTime(const uint64_t event_count) const
Definition: PerfMonMTUtils.h:232
PerfMonMTSvc::m_compLevelDataMap_plp
data_map_t m_compLevelDataMap_plp
Definition: PerfMonMTSvc.h:203
PMonMT::EventLevelData::recordEvent
void recordEvent(const SnapshotMeasurement &meas, const int eventCount)
Definition: PerfMonMTUtils.h:203
PMonMT::EventLevelData::getEventLevelCpuTime
double getEventLevelCpuTime(const uint64_t event_count) const
Definition: PerfMonMTUtils.h:228
ServiceHandle< IIncidentSvc >
python.SystemOfUnits.ms
float ms
Definition: SystemOfUnits.py:148