CoreDumpSvc Class Reference

Service to print additional information before a crash. More...

#include <CoreDumpSvc.h>

Inheritance diagram for CoreDumpSvc:

Collaboration diagram for CoreDumpSvc:

Classes
struct	sysDumpRec

Public Member Functions
	CoreDumpSvc (const std::string &name, ISvcLocator *pSvcLocator) ATLAS_CTORDTOR_NOT_THREAD_SAFE
	Constructor with parameters. More...
virtual	~CoreDumpSvc () ATLAS_CTORDTOR_NOT_THREAD_SAFE
	Destructor. More...
ICoreDumpSvc implementation
virtual void	setCoreDumpInfo (const std::string &name, const std::string &value) override
	Set a name/value pair in the core dump record. More...
virtual void	setCoreDumpInfo (const EventContext &ctx, const std::string &name, const std::string &value) override
	Set a name/value pair in the core dump record for given EventContext. More...
virtual std::string	dump () const override
	Print all core dump records. More...

Protected Member Functions
	CoreDumpSvc ()
	Default constructor (do not use) More...

Private Member Functions
void propertyHandler	ATLAS_NOT_THREAD_SAFE (Gaudi::Details::PropertyBase &p)
	Property handler. More...
void print	ATLAS_NOT_THREAD_SAFE ()
	Print core dump records to configured stream. More...
void	setSigInfo (siginfo_t *info)
	Set pointer to siginfo_t struct. More...
StatusCode installSignalHandler	ATLAS_NOT_THREAD_SAFE ()
	Install signal handlers. More...
StatusCode uninstallSignalHandler	ATLAS_NOT_THREAD_SAFE ()
	Uninstall signal handlers. More...
void	setAltStack ()
	Set up an alternate stack for the current thread. More...

Friends
void	CoreDumpSvcHandler::action (int sig, siginfo_t *info, void *extra)

Gaudi implementation
typedef tbb::concurrent_unordered_map< std::string, std::string >	UserCore_t
std::vector< UserCore_t >	m_usrCoreDumps
	User defined core dump info. More...
std::vector< sysDumpRec >	m_sysCoreDumps
	Core dump info collected by this service More...
siginfo_t *	m_siginfo {nullptr}
	Pointer to siginfo_t struct (set by signal handler) More...
std::atomic< EventID::event_number_t >	m_eventCounter {0}
	Event counter. More...
Gaudi::Property< std::vector< int > >	m_signals
	Alternate stack for signal handler. More...
Gaudi::Property< bool >	m_callOldHandler
Gaudi::Property< bool >	m_dumpCoreFile
Gaudi::Property< bool >	m_stackTrace
Gaudi::Property< bool >	m_fastStackTrace
Gaudi::Property< std::string >	m_coreDumpStream
Gaudi::Property< int >	m_fatalHandlerFlags
Gaudi::Property< double >	m_timeout
Gaudi::Property< bool >	m_killOnSigInt {this, "KillOnSigInt",true, "Terminate job on SIGINT (aka Ctrl-C)"}
static thread_local std::vector< uint8_t >	s_stack
virtual StatusCode initialize	ATLAS_NOT_THREAD_SAFE () override
virtual StatusCode	start () override
virtual StatusCode finalize	ATLAS_NOT_THREAD_SAFE () override
virtual void	handle (const Incident &incident) override
	Incident listener. More...

Detailed Description

Service to print additional information before a crash.

Author

Frank Winklmeier

Sami Kama

This service will catch fatal signals and print its internal core dump record. The service collects some information during event processing. Additional information can be added via setCoreDumpInfo().

For a list of job option properties see CoreDumpSvc::CoreDumpSvc().

Definition at line 41 of file CoreDumpSvc.h.

Member Typedef Documentation

UserCore_t

typedef tbb::concurrent_unordered_map<std::string,std::string > CoreDumpSvc::UserCore_t

private

Definition at line 88 of file CoreDumpSvc.h.

Constructor & Destructor Documentation

CoreDumpSvc() [1/2]

CoreDumpSvc::CoreDumpSvc ( )

protected

Default constructor (do not use)

CoreDumpSvc() [2/2]

CoreDumpSvc::CoreDumpSvc	(	const std::string &	name,
		ISvcLocator *	pSvcLocator
	)

Constructor with parameters.

Definition at line 237 of file CoreDumpSvc.cxx.

                                                                          :
  base_class( name, pSvcLocator )
{
  // Set us as the current instance
  CoreDumpSvcHandler::coreDumpSvc = this;
  
  m_callOldHandler.declareUpdateHandler(&CoreDumpSvc::propertyHandler, this);
  m_dumpCoreFile.declareUpdateHandler(&CoreDumpSvc::propertyHandler, this);
  m_stackTrace.declareUpdateHandler(&CoreDumpSvc::propertyHandler, this);
  m_fastStackTrace.declareUpdateHandler(&CoreDumpSvc::propertyHandler, this);
  m_coreDumpStream.declareUpdateHandler(&CoreDumpSvc::propertyHandler, this);
  m_fatalHandlerFlags.declareUpdateHandler(&CoreDumpSvc::propertyHandler, this);
  m_killOnSigInt.declareUpdateHandler(&CoreDumpSvc::propertyHandler, this);
  // Allocate for 2 slots just for now.
  m_usrCoreDumps.resize(2);
  m_sysCoreDumps.resize(2); 
}

~CoreDumpSvc()

CoreDumpSvc::~CoreDumpSvc ( )

virtual

Destructor.

Definition at line 255 of file CoreDumpSvc.cxx.

{
  CoreDumpSvcHandler::coreDumpSvc = nullptr;
}

Member Function Documentation

ATLAS_NOT_THREAD_SAFE() [1/6]

void print CoreDumpSvc::ATLAS_NOT_THREAD_SAFE ( )

private

Print core dump records to configured stream.

ATLAS_NOT_THREAD_SAFE() [2/6]

StatusCode installSignalHandler CoreDumpSvc::ATLAS_NOT_THREAD_SAFE ( )

private

Install signal handlers.

ATLAS_NOT_THREAD_SAFE() [3/6]

StatusCode uninstallSignalHandler CoreDumpSvc::ATLAS_NOT_THREAD_SAFE ( )

private

Uninstall signal handlers.

ATLAS_NOT_THREAD_SAFE() [4/6]

virtual StatusCode initialize CoreDumpSvc::ATLAS_NOT_THREAD_SAFE ( )

overridevirtual

ATLAS_NOT_THREAD_SAFE() [5/6]

virtual StatusCode finalize CoreDumpSvc::ATLAS_NOT_THREAD_SAFE ( )

overridevirtual

ATLAS_NOT_THREAD_SAFE() [6/6]

void propertyHandler CoreDumpSvc::ATLAS_NOT_THREAD_SAFE ( Gaudi::Details::PropertyBase &  p )

private

Property handler.

dump()

std::string CoreDumpSvc::dump ( ) const

overridevirtual

Print all core dump records.

Definition at line 393 of file CoreDumpSvc.cxx.

{
  std::ostringstream os;
  char buf[26];
  const time_t now = time(nullptr);
  
  os << "-------------------------------------------------------------------------------------" << "\n";
  os << "Core dump from " << name() << " on " << System::hostName()
     << " at " << ctime_r(&now, buf) /*<< "\n"*/; // ctime adds "\n"
  os << "\n";
 
  // Print additional information if available
  if (m_siginfo) {
    int signo = m_siginfo->si_signo;  // shorthand
    
    os << "Caught signal " << signo
       << "(" << strsignal(signo) << "). Details: "
       << "\n";   
 
    os << "  errno = " << m_siginfo->si_errno
       << ", code = " << m_siginfo->si_code
       << " (" << Athena::Signal::describe(signo, m_siginfo->si_code) << ")"
       << "\n";
    
    os << "  pid   = " << m_siginfo->si_pid
       << ", uid = " << m_siginfo->si_uid
       << "\n";
    
#ifndef __APPLE__
    // These are set if the POSIX signal sender passed them.
    os << "  value = (" << m_siginfo->si_int << ", "
       << std::hex << m_siginfo->si_ptr << ")" << std::dec << "\n";
#endif
 
    // memory usage informations
    athena_statm s = read_athena_statm();
    
    const long pagesz = sysconf(_SC_PAGESIZE);
    os << "  vmem = " << s.vm_pages*pagesz/1024./1024.  << " MB\n"
       << "  rss  = " << s.rss_pages*pagesz/1024./1024. << " MB\n";
 
#ifndef __APPLE__
    // more memory usage informations (system wide stuff)
    // see sysinfo(2)
 
    {
      struct sysinfo sys;
      if ( 0 == sysinfo(&sys) ) {
        // all sizes are reported in sys.mem_unit bytes
        const float mem_units = sys.mem_unit/(1024.*1024.);
        os << "  total-ram = " << sys.totalram * mem_units << " MB\n"
           << "  free-ram  = " << sys.freeram  * mem_units << " MB\n"
           << "  buffer-ram= " << sys.bufferram* mem_units << " MB\n"
           << "  total-swap= " << sys.totalswap* mem_units << " MB\n"
           << "  free-swap = " << sys.freeswap * mem_units << " MB\n";
      }
    }
#endif
 
    // This is the interesting address for memory faults.
    if (signo == SIGILL || signo == SIGFPE || signo == SIGSEGV || signo == SIGBUS)
      os << "  addr  = " << std::hex << m_siginfo->si_addr << std::dec << "\n";
    
    os << "\n";
  }
  
  os << "Event counter: " << m_eventCounter << "\n";  
 
  SmartIF<IAlgManager> algMgr{serviceLocator()->as<IAlgManager>()};
  SmartIF<IAlgContextSvc> algContextSvc;
 
 
  // For serial, retrieve AlgContextSvc
  if (Gaudi::Concurrency::ConcurrencyFlags::numConcurrentEvents() == 0) {
    algContextSvc = service("AlgContextSvc", /*createIf=*/ false);
  }
 
  // Loop over all slots
  for (size_t t=0; t < m_sysCoreDumps.size(); ++t){
 
    // Currently executing algorithm(s)
    std::string currentAlg;
 
    // Use AlgExecStateSvc in MT, otherwise AlgContextSvc
    if (Gaudi::Concurrency::ConcurrencyFlags::numConcurrentEvents() > 0) {
      const EventContext ctx(0,t);
      ATH_MSG_DEBUG("Using AlgExecStateSvc to determine current algorithm(s)");
      try {
        for (const IAlgorithm* alg : algMgr->getAlgorithms()) {
          auto aes = alg->execState(ctx);
          if (aes.state()==AlgExecState::State::Executing)
            currentAlg += (alg->name() + " ");
        }
      }
      catch (const GaudiException&) {  // can happen if we get called before any algo execution
        ATH_MSG_INFO("No information from AlgExecStateSvc because no algorithm was executed yet.");
      }
    }
    else if (algContextSvc) {
      ATH_MSG_DEBUG("Using AlgContextSvc to determine current algorithm");
      IAlgorithm* alg = algContextSvc->currentAlg();
      if (alg) currentAlg = alg->name();
    }
    else {
      ATH_MSG_WARNING("AlgExecStateSvc or AlgContextSvc not available. Cannot determine current algorithm.");
    }
 
    if (currentAlg.empty()) currentAlg = "<NONE>";
    os << "Slot " << std::setw(3) << t << " : Current algorithm = " << currentAlg << std::endl;
        
    // System core dump
    auto &sys = m_sysCoreDumps.at(t);
    if (!sys.LastInc.empty()) {
      os << "         : Last Incident = " << sys.LastInc << std::endl
         << "         : Event ID      = " << sys.EvId << std::endl;
    }
    
    // User core dump
    auto &usr = m_usrCoreDumps.at(t);
    if (!usr.empty()) {
      for (auto &s : usr) {
        os << "         : (usr) " << s.first << " = " << s.second << std::endl;
      }
    }
  }
 
  if (algContextSvc) {
    os << "Algorithm stack: ";
    if ( algContextSvc->algorithms().empty() ) os << "<EMPTY>" << "\n";
    else {
      os << "\n";
      for (auto alg : algContextSvc->algorithms()) {
        if (alg) os << "   " << alg->name() << "\n";
      }
    }
  }
 
  os << horizLine;
  os << "| AtlasBaseDir : " << std::setw(66) << getenv("AtlasBaseDir")  << " |\n";
  os << "| AtlasVersion : " << std::setw(66) << getenv("AtlasVersion")  << " |\n";
  os << "| BINARY_TAG   : " << std::setw(66) << getenv("BINARY_TAG")    << " |\n";
  os << horizLine;
  os << " Note: to see line numbers in below stacktrace you might consider running following :\n";
  os << "  atlasAddress2Line --file <logfile>\n";
 
  SmartIF<IAthenaSummarySvc> iass{service("AthenaSummarySvc", /*createIf*/false)};
  if (iass) {
    iass->addSummary("CoreDumpSvc",os.str());
    iass->setStatus(1);
    iass->createSummary().ignore();
  }
  
  return os.str();
}

handle()

void CoreDumpSvc::handle ( const Incident &  incident )

overridevirtual

Incident listener.

Definition at line 552 of file CoreDumpSvc.cxx.

{
  //handle is single threaded in context;
  auto slot = incident.context().valid() ? incident.context().slot() : 0;  
  auto &currRec = m_sysCoreDumps.at(slot);
 
  currRec.LastInc = incident.source() + ":" + incident.type();
 
  std::ostringstream oss;
  oss << incident.context().eventID();
  currRec.EvId = oss.str();
 
  if (incident.type()==IncidentType::BeginEvent) {
    // Set up an alternate stack for this thread, if not already done.
    setAltStack();
    ++m_eventCounter;
  } else if (incident.type() == "StoreCleared") {
    // Try to force reallocation.
    auto newstr = currRec.EvId;
    // Intentional:
    // cppcheck-suppress selfAssignment
    newstr[0] = newstr[0];
    currRec.EvId = std::move(newstr);
  }
 
}

setAltStack()

void CoreDumpSvc::setAltStack ( )

private

Set up an alternate stack for the current thread.

Definition at line 647 of file CoreDumpSvc.cxx.

{
  std::vector<uint8_t>& stack = s_stack;
  if (stack.empty()) {
    stack.resize (std::max (SIGSTKSZ, MINSIGSTKSZ) + 2*1024*1024);
    stack_t ss;
    ss.ss_sp = stack.data();
    ss.ss_flags = 0;
    ss.ss_size = stack.size();
    int ret = sigaltstack (&ss, nullptr);
    if ( ret!=0 ) {
      ATH_MSG_WARNING("Error on setting alternative stack! ");
    }
  }
}

setCoreDumpInfo() [1/2]

void CoreDumpSvc::setCoreDumpInfo ( const EventContext &  ctx, const std::string &  name, const std::string &  value  )

overridevirtual

Set a name/value pair in the core dump record for given EventContext.

Definition at line 370 of file CoreDumpSvc.cxx.

{
  auto slot = ctx.valid() ? ctx.slot() : 0;  
  m_usrCoreDumps.at(slot)[name] = value;
}

setCoreDumpInfo() [2/2]

void CoreDumpSvc::setCoreDumpInfo ( const std::string &  name, const std::string &  value  )

overridevirtual

Set a name/value pair in the core dump record.

Definition at line 365 of file CoreDumpSvc.cxx.

{
  setCoreDumpInfo(Gaudi::Hive::currentContext(), name, value);
}

setSigInfo()

void CoreDumpSvc::setSigInfo ( siginfo_t *  info )

inlineprivate

Set pointer to siginfo_t struct.

Definition at line 136 of file CoreDumpSvc.h.

{ m_siginfo = info; }  

start()

StatusCode CoreDumpSvc::start ( )

overridevirtual

Definition at line 338 of file CoreDumpSvc.cxx.

{
  auto numSlots = std::max<size_t>(1, Gaudi::Concurrency::ConcurrencyFlags::numConcurrentEvents());
  m_usrCoreDumps.resize(numSlots);
  m_sysCoreDumps.resize(numSlots);
  return StatusCode::SUCCESS;
}

Friends And Related Function Documentation

CoreDumpSvcHandler::action

void CoreDumpSvcHandler::action ( int  sig, siginfo_t *  info, void *  extra  )

friend

Member Data Documentation

m_callOldHandler

Gaudi::Property<bool> CoreDumpSvc::m_callOldHandler

private

Initial value:

{this, "CallOldHandler", true,
      "Call previous signal handler"}

Definition at line 101 of file CoreDumpSvc.h.

m_coreDumpStream

Gaudi::Property<std::string> CoreDumpSvc::m_coreDumpStream

private

Initial value:

{this, "CoreDumpStream", "stdout",
      "Stream to use for core dump [stdout,stderr]"}

Definition at line 113 of file CoreDumpSvc.h.

m_dumpCoreFile

Gaudi::Property<bool> CoreDumpSvc::m_dumpCoreFile

private

Initial value:

{this, "DumpCoreFile", false,
      "Produce a core dump file if resource limits (ulimit -c) allow"}

Definition at line 104 of file CoreDumpSvc.h.

m_eventCounter

std::atomic<EventID::event_number_t> CoreDumpSvc::m_eventCounter {0}

private

Event counter.

Definition at line 92 of file CoreDumpSvc.h.

m_fastStackTrace

Gaudi::Property<bool> CoreDumpSvc::m_fastStackTrace

private

Initial value:

{this, "FastStackTrace", false,
      "Produce fast stack trace of current thread"}

Definition at line 110 of file CoreDumpSvc.h.

m_fatalHandlerFlags

Gaudi::Property<int> CoreDumpSvc::m_fatalHandlerFlags

private

Initial value:

{this, "FatalHandler", 0, 
      "Flags given to the fatal handler this service installs\n"
      "if the flag is zero, no additional fatal handler is installed."}

Definition at line 116 of file CoreDumpSvc.h.

m_killOnSigInt

Gaudi::Property<bool> CoreDumpSvc::m_killOnSigInt {this, "KillOnSigInt",true, "Terminate job on SIGINT (aka Ctrl-C)"}

private

Definition at line 124 of file CoreDumpSvc.h.

m_siginfo

siginfo_t* CoreDumpSvc::m_siginfo {nullptr}

private

Pointer to siginfo_t struct (set by signal handler)

Definition at line 91 of file CoreDumpSvc.h.

m_signals

Gaudi::Property<std::vector<int> > CoreDumpSvc::m_signals

private

Initial value:

{this, "Signals", {SIGSEGV,SIGBUS,SIGILL,SIGABRT,SIGFPE,SIGALRM},
      "List of signals to catch"}

Alternate stack for signal handler.

Properties

Definition at line 98 of file CoreDumpSvc.h.

m_stackTrace

Gaudi::Property<bool> CoreDumpSvc::m_stackTrace

private

Initial value:

{this, "StackTrace", false,
      "Produce (gdb) stack trace on crash. Useful if no other signal handler is used"}

Definition at line 107 of file CoreDumpSvc.h.

m_sysCoreDumps

std::vector<sysDumpRec> CoreDumpSvc::m_sysCoreDumps

private

Core dump info collected by this service

Definition at line 90 of file CoreDumpSvc.h.

m_timeout

Gaudi::Property<double> CoreDumpSvc::m_timeout

private

Initial value:

{this, "TimeOut", 30.0*60*1e9,
      "Terminate job after it this reaches the time out in Wallclock time, "
      "usually due to hanging during stack unwinding. Timeout given in nanoseconds despite seconds precision"}

Definition at line 120 of file CoreDumpSvc.h.

m_usrCoreDumps

std::vector<UserCore_t> CoreDumpSvc::m_usrCoreDumps

private

User defined core dump info.

Definition at line 89 of file CoreDumpSvc.h.

s_stack

thread_local std::vector< uint8_t > CoreDumpSvc::s_stack

staticprivate

Definition at line 94 of file CoreDumpSvc.h.

---

The documentation for this class was generated from the following files:

• CoreDumpSvc.h
• CoreDumpSvc.cxx