Service to print additional information before a crash. More...

#include <CoreDumpSvc.h>

Inheritance diagram for CoreDumpSvc:

Collaboration diagram for CoreDumpSvc:

Classes
struct	sysDumpRec

Public Member Functions
	CoreDumpSvc (const std::string &name, ISvcLocator *pSvcLocator) ATLAS_CTORDTOR_NOT_THREAD_SAFE
	Constructor with parameters. More...

virtual	~CoreDumpSvc () ATLAS_CTORDTOR_NOT_THREAD_SAFE
	Destructor. More...

ICoreDumpSvc implementation
virtual void	setCoreDumpInfo (const std::string &name, const std::string &value) override
	Set a name/value pair in the core dump record. More...

virtual void	setCoreDumpInfo (const EventContext &ctx, const std::string &name, const std::string &value) override
	Set a name/value pair in the core dump record for given EventContext. More...

virtual std::string	dump () const override
	Print all core dump records. More...

Protected Member Functions
	CoreDumpSvc ()
	Default constructor (do not use) More...

Private Member Functions
void propertyHandler	ATLAS_NOT_THREAD_SAFE (Gaudi::Details::PropertyBase &p)
	Property handler. More...

void print	ATLAS_NOT_THREAD_SAFE ()
	Print core dump records to configured stream. More...

void	setSigInfo (siginfo_t *info)
	Set pointer to siginfo_t struct. More...

StatusCode installSignalHandler	ATLAS_NOT_THREAD_SAFE ()
	Install signal handlers. More...

StatusCode uninstallSignalHandler	ATLAS_NOT_THREAD_SAFE ()
	Uninstall signal handlers. More...

void	setAltStack ()
	Set up an alternate stack for the current thread. More...

Friends
void	CoreDumpSvcHandler::action (int sig, siginfo_t info, void extra)

Gaudi implementation
typedef tbb::concurrent_unordered_map< std::string, std::string >	UserCore_t

std::vector< UserCore_t >	m_usrCoreDumps
	User defined core dump info. More...

std::vector< sysDumpRec >	m_sysCoreDumps
	Core dump info collected by this service More...

siginfo_t *	m_siginfo {nullptr}
	Pointer to siginfo_t struct (set by signal handler) More...

std::atomic< EventID::event_number_t >	m_eventCounter {0}
	Event counter. More...

Gaudi::Property< std::vector< int > >	m_signals
	Alternate stack for signal handler. More...

Gaudi::Property< bool >	m_callOldHandler

Gaudi::Property< bool >	m_dumpCoreFile

Gaudi::Property< bool >	m_stackTrace

Gaudi::Property< bool >	m_fastStackTrace

Gaudi::Property< std::string >	m_coreDumpStream

Gaudi::Property< int >	m_fatalHandlerFlags

Gaudi::Property< double >	m_timeout

Gaudi::Property< bool >	m_killOnSigInt {this, "KillOnSigInt",true, "Terminate job on SIGINT (aka Ctrl-C)"}

static thread_local std::vector< uint8_t >	s_stack

virtual StatusCode initialize	ATLAS_NOT_THREAD_SAFE () override

virtual StatusCode	start () override

virtual StatusCode finalize	ATLAS_NOT_THREAD_SAFE () override

virtual void	handle (const Incident &incident) override
	Incident listener. More...

Detailed Description

Service to print additional information before a crash.

Author: Frank Winklmeier; Sami Kama

This service will catch fatal signals and print its internal core dump record. The service collects some information during event processing. Additional information can be added via setCoreDumpInfo().

For a list of job option properties see CoreDumpSvc::CoreDumpSvc().

Definition at line 44 of file CoreDumpSvc.h.

Member Typedef Documentation

◆ UserCore_t

typedef tbb::concurrent_unordered_map<std::string,std::string > CoreDumpSvc::UserCore_t

private

Definition at line 91 of file CoreDumpSvc.h.

Constructor & Destructor Documentation

◆ CoreDumpSvc() [1/2]

CoreDumpSvc::CoreDumpSvc ( )

protected

Default constructor (do not use)

◆ CoreDumpSvc() [2/2]

CoreDumpSvc::CoreDumpSvc	(	const std::string &	name,
		ISvcLocator *	pSvcLocator
	)

Constructor with parameters.

Definition at line 236 of file CoreDumpSvc.cxx.

                                                                           :
   base_class( name, pSvcLocator )
 {
   // Set us as the current instance
   CoreDumpSvcHandler::coreDumpSvc = this;
   
   m_callOldHandler.declareUpdateHandler(&CoreDumpSvc::propertyHandler, this);
   m_dumpCoreFile.declareUpdateHandler(&CoreDumpSvc::propertyHandler, this);
   m_stackTrace.declareUpdateHandler(&CoreDumpSvc::propertyHandler, this);
   m_fastStackTrace.declareUpdateHandler(&CoreDumpSvc::propertyHandler, this);
   m_coreDumpStream.declareUpdateHandler(&CoreDumpSvc::propertyHandler, this);
   m_fatalHandlerFlags.declareUpdateHandler(&CoreDumpSvc::propertyHandler, this);
   m_killOnSigInt.declareUpdateHandler(&CoreDumpSvc::propertyHandler, this);
   // Allocate for 2 slots just for now.
   m_usrCoreDumps.resize(2);
   m_sysCoreDumps.resize(2); 
 }

◆ ~CoreDumpSvc()

CoreDumpSvc::~CoreDumpSvc ( )

virtual

Destructor.

Definition at line 254 of file CoreDumpSvc.cxx.

 {
   CoreDumpSvcHandler::coreDumpSvc = nullptr;
 }

Member Function Documentation

◆ ATLAS_NOT_THREAD_SAFE() [1/6]

void print CoreDumpSvc::ATLAS_NOT_THREAD_SAFE ( )

private

Print core dump records to configured stream.

◆ ATLAS_NOT_THREAD_SAFE() [2/6]

StatusCode installSignalHandler CoreDumpSvc::ATLAS_NOT_THREAD_SAFE ( )

private

Install signal handlers.

◆ ATLAS_NOT_THREAD_SAFE() [3/6]

StatusCode uninstallSignalHandler CoreDumpSvc::ATLAS_NOT_THREAD_SAFE ( )

private

Uninstall signal handlers.

◆ ATLAS_NOT_THREAD_SAFE() [4/6]

virtual StatusCode initialize CoreDumpSvc::ATLAS_NOT_THREAD_SAFE ( )

overridevirtual

◆ ATLAS_NOT_THREAD_SAFE() [5/6]

virtual StatusCode finalize CoreDumpSvc::ATLAS_NOT_THREAD_SAFE ( )

overridevirtual

◆ ATLAS_NOT_THREAD_SAFE() [6/6]

void propertyHandler CoreDumpSvc::ATLAS_NOT_THREAD_SAFE ( Gaudi::Details::PropertyBase & p )

private

Property handler.

◆ dump()

std::string CoreDumpSvc::dump ( ) const

overridevirtual

Print all core dump records.

Definition at line 392 of file CoreDumpSvc.cxx.

 {
   std::ostringstream os;
   char buf[26];
   const time_t now = time(nullptr);
   
   os << "-------------------------------------------------------------------------------------" << "\n";
   os << "Core dump from " << name() << " on " << System::hostName()
      << " at " << ctime_r(&now, buf) /*<< "\n"*/; // ctime adds "\n"
   os << "\n";
  
   // Print additional information if available
   if (m_siginfo) {
     int signo = m_siginfo->si_signo;  // shorthand
     
     os << "Caught signal " << signo
        << "(" << strsignal(signo) << "). Details: "
        << "\n";   
  
     os << "  errno = " << m_siginfo->si_errno
        << ", code = " << m_siginfo->si_code
        << " (" << Athena::Signal::describe(signo, m_siginfo->si_code) << ")"
        << "\n";
     
     os << "  pid   = " << m_siginfo->si_pid
        << ", uid = " << m_siginfo->si_uid
        << "\n";
     
 #ifndef __APPLE__
     // These are set if the POSIX signal sender passed them.
     os << "  value = (" << m_siginfo->si_int << ", "
        << std::hex << m_siginfo->si_ptr << ")" << std::dec << "\n";
 #endif
  
     // memory usage informations
     athena_statm s = read_athena_statm();
     
     const long pagesz = sysconf(_SC_PAGESIZE);
     os << "  vmem = " << s.vm_pages*pagesz/1024./1024.  << " MB\n"
        << "  rss  = " << s.rss_pages*pagesz/1024./1024. << " MB\n";
  
 #ifndef __APPLE__
     // more memory usage informations (system wide stuff)
     // see sysinfo(2)
  
     {
       struct sysinfo sys;
       if ( 0 == sysinfo(&sys) ) {
         // all sizes are reported in sys.mem_unit bytes
         const float mem_units = sys.mem_unit/(1024.*1024.);
         os << "  total-ram = " << sys.totalram * mem_units << " MB\n"
            << "  free-ram  = " << sys.freeram  * mem_units << " MB\n"
            << "  buffer-ram= " << sys.bufferram* mem_units << " MB\n"
            << "  total-swap= " << sys.totalswap* mem_units << " MB\n"
            << "  free-swap = " << sys.freeswap * mem_units << " MB\n";
       }
     }
 #endif
  
     // This is the interesting address for memory faults.
     if (signo == SIGILL || signo == SIGFPE || signo == SIGSEGV || signo == SIGBUS)
       os << "  addr  = " << std::hex << m_siginfo->si_addr << std::dec << "\n";
     
     os << "\n";
   }
   
   os << "Event counter: " << m_eventCounter << "\n";  
  
  
   IAlgExecStateSvc* algExecStateSvc(nullptr);
   IAlgContextSvc* algContextSvc(nullptr);
  
   // Use AlgExecStateSvc in MT, otherwise AlgContextSvc
   if (Gaudi::Concurrency::ConcurrencyFlags::numConcurrentEvents() > 0) {
     service("AlgExecStateSvc", algExecStateSvc, /*createIf=*/ false).ignore();
   }
   else {
     service("AlgContextSvc", algContextSvc, /*createIf=*/ false).ignore();
   }
  
   // Loop over all slots
   for (size_t t=0; t < m_sysCoreDumps.size(); ++t){
  
     // Currently executing algorithm(s)
     std::string currentAlg;
     if (algExecStateSvc) {
       ATH_MSG_DEBUG("Using AlgExecStateSvc to determine current algorithm(s)");
       try {
         // We copy on purpose to avoid modification while we examine it
         auto states = algExecStateSvc->algExecStates(EventContext(0,t));
         for (const auto& kv : states) {
           if (kv.second.state()==AlgExecState::State::Executing)
             currentAlg += (kv.first + " ");
         }
       }
       catch (const GaudiException&) {  // can happen if we get called before any algo execution
         ATH_MSG_INFO("No information from AlgExecStateSvc because no algorithm was executed yet.");
       }
     }
     else if (algContextSvc) {
       ATH_MSG_DEBUG("Using AlgContextSvc to determine current algorithm");
       IAlgorithm* alg = algContextSvc->currentAlg();
       if (alg) currentAlg = alg->name();
     }
     else {
       ATH_MSG_WARNING("AlgExecStateSvc or AlgContextSvc not available. Cannot determine current algorithm.");
     }
  
     if (currentAlg.empty()) currentAlg = "<NONE>";
     os << "Slot " << std::setw(3) << t << " : Current algorithm = " << currentAlg << std::endl;
         
     // System core dump
     auto &sys = m_sysCoreDumps.at(t);
     if (!sys.LastInc.empty()) {
       os << "         : Last Incident = " << sys.LastInc << std::endl
          << "         : Event ID      = " << sys.EvId << std::endl;
     }
     
     // User core dump
     auto &usr = m_usrCoreDumps.at(t);
     if (!usr.empty()) {
       for (auto &s : usr) {
         os << "         : (usr) " << s.first << " = " << s.second << std::endl;
       }
     }
   }
  
   if (algContextSvc) {
     os << "Algorithm stack: ";
     if ( algContextSvc->algorithms().empty() ) os << "<EMPTY>" << "\n";
     else {
       os << "\n";
       for (auto alg : algContextSvc->algorithms()) {
         if (alg) os << "   " << alg->name() << "\n";
       }
     }
   }
  
   os << horizLine;
   os << "| AtlasBaseDir : " << std::setw(66) << getenv("AtlasBaseDir")  << " |\n";
   os << "| AtlasVersion : " << std::setw(66) << getenv("AtlasVersion")  << " |\n";
   os << "| BINARY_TAG   : " << std::setw(66) << getenv("BINARY_TAG")    << " |\n";
   os << horizLine;
   os << " Note: to see line numbers in below stacktrace you might consider running following :\n";
   os << "  atlasAddress2Line --file <logfile>\n";
  
   IAthenaSummarySvc *iass(nullptr);
   if (service("AthenaSummarySvc",iass,false).isSuccess() && iass) {
     iass->addSummary("CoreDumpSvc",os.str());
     iass->setStatus(1);
     iass->createSummary().ignore();
   }
   
   return os.str();
 }

◆ handle()

void CoreDumpSvc::handle ( const Incident & incident )

overridevirtual

Incident listener.

Definition at line 552 of file CoreDumpSvc.cxx.

 {
   //handle is single threaded in context;
   auto slot = incident.context().valid() ? incident.context().slot() : 0;  
   auto &currRec = m_sysCoreDumps.at(slot);
  
   currRec.LastInc = incident.source() + ":" + incident.type();
  
   std::ostringstream oss;
   oss << incident.context().eventID();
   currRec.EvId = oss.str();
  
   if (incident.type()==IncidentType::BeginEvent) {
     // Set up an alternate stack for this thread, if not already done.
     setAltStack();
     ++m_eventCounter;
   } else if (incident.type() == "StoreCleared") {
     // Try to force reallocation.
     auto newstr = currRec.EvId;
     // Intentional:
     // cppcheck-suppress selfAssignment
     newstr[0] = newstr[0];
     currRec.EvId = newstr;
   }
  
 }

◆ setAltStack()

void CoreDumpSvc::setAltStack ( )

private

Set up an alternate stack for the current thread.

Definition at line 647 of file CoreDumpSvc.cxx.

 {
   std::vector<uint8_t>& stack = s_stack;
   if (stack.empty()) {
     stack.resize (std::max (SIGSTKSZ, MINSIGSTKSZ) + 2*1024*1024);
     stack_t ss;
     ss.ss_sp = stack.data();
     ss.ss_flags = 0;
     ss.ss_size = stack.size();
     sigaltstack (&ss, nullptr);
   }
 }

◆ setCoreDumpInfo() [1/2]

void CoreDumpSvc::setCoreDumpInfo	(	const EventContext &	ctx,
		const std::string &	name,
		const std::string &	value
	)

overridevirtual

Set a name/value pair in the core dump record for given EventContext.

Definition at line 369 of file CoreDumpSvc.cxx.

 {
   auto slot = ctx.valid() ? ctx.slot() : 0;  
   m_usrCoreDumps.at(slot)[name] = value;
 }

◆ setCoreDumpInfo() [2/2]

void CoreDumpSvc::setCoreDumpInfo	(	const std::string &	name,
		const std::string &	value
	)

overridevirtual

Set a name/value pair in the core dump record.

Definition at line 364 of file CoreDumpSvc.cxx.

 {
   setCoreDumpInfo(Gaudi::Hive::currentContext(), name, value);
 }

◆ setSigInfo()

void CoreDumpSvc::setSigInfo ( siginfo_t * info )

inlineprivate

Set pointer to siginfo_t struct.

Definition at line 139 of file CoreDumpSvc.h.

139 { m_siginfo = info; }

◆ start()

StatusCode CoreDumpSvc::start ( )

overridevirtual

Definition at line 337 of file CoreDumpSvc.cxx.

 {
   auto numSlots = std::max<size_t>(1, Gaudi::Concurrency::ConcurrencyFlags::numConcurrentEvents());
   m_usrCoreDumps.resize(numSlots);
   m_sysCoreDumps.resize(numSlots);
   return StatusCode::SUCCESS;
 }

Friends And Related Function Documentation

◆ CoreDumpSvcHandler::action

void CoreDumpSvcHandler::action	(	int	sig,
		siginfo_t *	info,
		void *	extra
	)

friend

Member Data Documentation

◆ m_callOldHandler

Gaudi::Property<bool> CoreDumpSvc::m_callOldHandler

private

Initial value:

{this, "CallOldHandler", true,

"Call previous signal handler"}

Definition at line 104 of file CoreDumpSvc.h.

◆ m_coreDumpStream

Gaudi::Property<std::string> CoreDumpSvc::m_coreDumpStream

private

Initial value:

{this, "CoreDumpStream", "stdout",

"Stream to use for core dump [stdout,stderr]"}

Definition at line 116 of file CoreDumpSvc.h.

◆ m_dumpCoreFile

Gaudi::Property<bool> CoreDumpSvc::m_dumpCoreFile

private

Initial value:

{this, "DumpCoreFile", false,

"Produce a core dump file if resource limits (ulimit -c) allow"}

Definition at line 107 of file CoreDumpSvc.h.

◆ m_eventCounter

std::atomic<EventID::event_number_t> CoreDumpSvc::m_eventCounter {0}

private

Event counter.

Definition at line 95 of file CoreDumpSvc.h.

◆ m_fastStackTrace

Gaudi::Property<bool> CoreDumpSvc::m_fastStackTrace

private

Initial value:

{this, "FastStackTrace", false,

"Produce fast stack trace of current thread"}

Definition at line 113 of file CoreDumpSvc.h.

◆ m_fatalHandlerFlags

Gaudi::Property<int> CoreDumpSvc::m_fatalHandlerFlags

private

Initial value:

{this, "FatalHandler", 0, 
      "Flags given to the fatal handler this service installs\n"
      "if the flag is zero, no additional fatal handler is installed."}

Definition at line 119 of file CoreDumpSvc.h.

◆ m_killOnSigInt

Gaudi::Property<bool> CoreDumpSvc::m_killOnSigInt {this, "KillOnSigInt",true, "Terminate job on SIGINT (aka Ctrl-C)"}

private

Definition at line 127 of file CoreDumpSvc.h.

◆ m_siginfo

siginfo_t* CoreDumpSvc::m_siginfo {nullptr}

private

Pointer to siginfo_t struct (set by signal handler)

Definition at line 94 of file CoreDumpSvc.h.

◆ m_signals

Gaudi::Property<std::vector<int> > CoreDumpSvc::m_signals

private

Initial value:

{this, "Signals", {SIGSEGV,SIGBUS,SIGILL,SIGFPE,SIGALRM},

"List of signals to catch"}

Alternate stack for signal handler.

Properties

Definition at line 101 of file CoreDumpSvc.h.

◆ m_stackTrace

Gaudi::Property<bool> CoreDumpSvc::m_stackTrace

private

Initial value:

{this, "StackTrace", false,

"Produce (gdb) stack trace on crash. Useful if no other signal handler is used"}

Definition at line 110 of file CoreDumpSvc.h.

◆ m_sysCoreDumps

std::vector<sysDumpRec> CoreDumpSvc::m_sysCoreDumps

private

Core dump info collected by this service

Definition at line 93 of file CoreDumpSvc.h.

◆ m_timeout

Gaudi::Property<double> CoreDumpSvc::m_timeout

private

Initial value:

{this, "TimeOut", 30.0*60*1e9,
      "Terminate job after it this reaches the time out in Wallclock time, "
      "usually due to hanging during stack unwinding. Timeout given in nanoseconds despite seconds precision"}

Definition at line 123 of file CoreDumpSvc.h.

◆ m_usrCoreDumps

std::vector<UserCore_t> CoreDumpSvc::m_usrCoreDumps

private

User defined core dump info.

Definition at line 92 of file CoreDumpSvc.h.

◆ s_stack

thread_local std::vector< uint8_t > CoreDumpSvc::s_stack

staticprivate

Definition at line 97 of file CoreDumpSvc.h.

The documentation for this class was generated from the following files:

Classes

Public Member Functions

Protected Member Functions

Private Member Functions

Friends

Gaudi implementation

Detailed Description

Member Typedef Documentation

◆ UserCore_t

Constructor & Destructor Documentation

◆ CoreDumpSvc() [1/2]

◆ CoreDumpSvc() [2/2]

◆ ~CoreDumpSvc()

Member Function Documentation

◆ ATLAS_NOT_THREAD_SAFE() [1/6]

◆ ATLAS_NOT_THREAD_SAFE() [2/6]

◆ ATLAS_NOT_THREAD_SAFE() [3/6]

◆ ATLAS_NOT_THREAD_SAFE() [4/6]

◆ ATLAS_NOT_THREAD_SAFE() [5/6]

◆ ATLAS_NOT_THREAD_SAFE() [6/6]

◆ dump()

◆ handle()

◆ setAltStack()

◆ setCoreDumpInfo() [1/2]

◆ setCoreDumpInfo() [2/2]

◆ setSigInfo()

◆ start()

Friends And Related Function Documentation

◆ CoreDumpSvcHandler::action

Member Data Documentation

◆ m_callOldHandler

◆ m_coreDumpStream

◆ m_dumpCoreFile

◆ m_eventCounter

◆ m_fastStackTrace

◆ m_fatalHandlerFlags

◆ m_killOnSigInt

◆ m_siginfo

◆ m_signals

◆ m_stackTrace

◆ m_sysCoreDumps

◆ m_timeout

◆ m_usrCoreDumps

◆ s_stack