EL::Job Class Reference

#include <Job.h>

Collaboration diagram for EL::Job:

Public Types
typedef OutputStream *	outputMIter
	description: the list of output datasets used guarantee: no-fail / strong failures(3): out of memory II requires(3,soft): !outputHas (val_output.label()) note: while it is not a requirement it is highly recommended that you add the outputs from Algorithm::doSetupJob.
typedef const OutputStream *	outputIter

Public Member Functions
void	testInvariant () const
	effects: test the invariant of this object guarantee: no-fail
	Job ()
	effects: standard default constructor guarantee: strong failures: low level errors I
	Job (const Job &that)
	effects: standard copy constructor guarantee: strong failures: out of memory II
	~Job ()
	effects: standard destructor guarantee: no-fail
Job &	operator= (const Job &that)
	effects: standard assignment operator returns: *this guarantee: strong failures: out of memory II
const SH::SampleHandler &	sampleHandler () const
	description: the sample handler used guarantee: no-fail / strong failures: out of memory II
void	sampleHandler (const SH::SampleHandler &val_sampleHandler)
void	algsAdd (std::unique_ptr< IAlgorithmWrapper > val_algorithm)
	description: the list of algorithms used guarantee: no-fail / strong failures: out of memory II invariant: alg != 0
void	algsAdd (std::unique_ptr< Algorithm > val_algorithm)
void	algsAdd (Algorithm *alg_swallow)
void	algsAdd (const AnaAlgorithmConfig &config)
void	algsAdd (const AnaReentrantAlgorithmConfig &config)
void	algsAdd (const asg::AsgServiceConfig &config)
void	algsAdd (const EL::PythonConfigBase &config)
void	algsAddClone (const Algorithm &alg)
	add a clone of the given algorithm
bool	algsHas (const std::string &name) const
	returns: whether we have the algorithm with the given name guarantee: no-fail
void	startNewAlgSequence ()
	start a new sub-sequence of algorithms
outputMIter	outputBegin ()
outputIter	outputBegin () const
outputMIter	outputEnd ()
outputIter	outputEnd () const
void	outputAdd (const OutputStream &val_output)
bool	outputHas (const std::string &name) const
	returns: whether we have an output with the given name guarantee: no-fail
void	useXAOD ()
	effects: register this job to use XAODs guarantee: strong failures: out of memory II failures: EventSvc not available
SH::MetaObject *	options ()
	description: the list of options to the job guarantee: no-fail postcondition: result != 0
const SH::MetaObject *	options () const
const JobConfig &	jobConfig () const noexcept
	the JobConfig object we are wrapping

Static Public Attributes
static const std::string	optRemoveSubmitDir = "nc_EventLoop_RemoveSubmitDir"
	description: the name of the option for overwriting the submission directory.
static const std::string	optSubmitDirMode = "nc_EventLoop_SubmitDirMode"
	the submit-dir mode (allowed values: "no-clobber", "overwrite", "unique", "unique-link")
static const std::string	optUniqueDateFormat = "nc_EventLoop_UniqueDateFormat"
	the date-format to use when generating unique submission directory names
static const std::string	optAlgorithmTimer = "nc_EventLoop_AlgorithmTimer"
	a boolean flag for whether to add a timer for the algorithms
static const std::string	optAlgorithmMemoryMonitor = "nc_EventLoop_AlgorithmMemoryMonitor"
	a boolean flag for whether to add a memory monitor for the algorithms
static const std::string	optFactoryPreload = "nc_EventLoop_FactoryPreload"
	a boolean flag for whether to perform a component factory preload
static const std::string	optStreamAliases = "nc_EventLoop_StreamAliases"
	an option for stream aliases
static const std::string	optMaxEvents = "nc_EventLoop_MaxEvents"
	description: the name of the option used for setting the maximum number of events to process per sample rationale: this is used for test runs where you don't want to process all events, but just want a quick look
static const std::string	optSkipEvents = "nc_EventLoop_SkipEvents"
	description: the name of the option used for skipping a certain number of events in the beginning rationale: this is used for test runs where you are only interested in a particular set of events
static const std::string	optWorkerConfigFile = "nc_EventLoop_WorkerConfigFile"
	a python configuration file that will be executed on the worker
static const std::string	optFilesPerWorker = "nc_EventLoop_FilesPerWorker"
	description: the name of the option for selecting the number of files per batch job.
static const std::string	optEventsPerWorker = "nc_EventLoop_EventsPerWorker"
	description: the name of the option for selecting the number of events per batch job.
static const std::string	optWorkerPostClosedOutputsExecutable = "nc_EventLoop_WorkerPostClosedOutputsExecutable"
	Optionnal executable to be provided by the user that will be called, by the worker, for every output stream.
static const std::string	optSubmitFlags = "nc_EventLoop_SubmitFlags"
	description: the name of the option for supplying extra submit parameters to batch systems rationale: the primary purpose of this is to allow selecting the right queue on your batch system, but it can be used for all kind of submission options that EventLoop doesn't support directly.
static const std::string	optCondorConf = "nc_EventLoop_CondorConf"
	description: the name of the option for supplying extra parameters for condor systems
static const std::string	optCacheSize = "nc_cache_size"
	description: this option allows to configure the TTreeCache size for this job.
static const std::string	optCacheLearnEntries = "nc_EventLoop_CacheLearnEntries"
	description: this option allows to configure the number of tree entries used for learning cache behavior.
static const std::string	optD3PDPerfStats = "nc_EventLoop_D3PDPerfStats"
	description: the name of the option for turning on D3PDPerfStats.
static const std::string	optD3PDReadStats = "nc_EventLoop_D3PDReadStats"
	description: the name of the D3PDPerfStats object produced as I gather it, as well as the name of the option passed into the job.
static const std::string	optXAODPerfStats = "nc_EventLoop_XAODPerfStats"
	description: the name of the option for turning on XAODPerfStats.
static const std::string	optXAODReadStats = "nc_EventLoop_XAODReadStats"
	description: the name of the XAODPerfStats object produced as I gather it, as well as the name of the option passed into the job.
static const std::string	optD3PDCacheMinEvent = "nc_EventLoop_D3PDCacheMinEvent"
	description: these options configure the D3PDReader TTreeCache settings.
static const std::string	optD3PDCacheMinEventFraction = "nc_EventLoop_D3PDCacheMinEventFraction"
static const std::string	optD3PDCacheMinByte = "nc_EventLoop_D3PDCacheMinByte"
static const std::string	optD3PDCacheMinByteFraction = "nc_EventLoop_D3PDCacheMinByteFraction"
static const std::string	optPerfTree = "nc_EventLoop_PerfTree"
	description: the option to turn on the performance tree in PROOF.
static const std::string	optXAODInput = "nc_EventLoop_XAODInput"
	the option to select whether our input is xAODs
static const std::string	optXAODSummaryReport = "nc_xaod_summary_report"
	the option to turn on/off the xAOD summary reporting at the end of the job
static const std::string	optPrintPerFileStats = "nc_print_per_file_stats"
	description: the option to turn on printing of i/o statistics at the end of each file rationale: while this is not as complete as whole sample statistics, it can be helpful in some circumstances when debugging the performance warning: this is not supported for all drivers
static const std::string	optDisableMetrics = "nc_disable_metrics"
	description: the option to turn off collection of performance data
static const std::string	optResetShell = "nc_reset_shell"
	description: the option to reset the shell on the worker nodes rationale: this is currently only used by the LSFDriver where it is enabled by default to reset it on lxbatch.
static const std::string	optLocalNoUnsetup = "nc_local_no_unsetup"
	the option not to unsetup the environment in LocalDriver
static const std::string	optNumParallelProcs = "nc_num_parallel_processes"
	the option to specify the number of parallel jobs in LocalDriver (0 = number of hardware cores) (default=1)
static const std::string	optBackgroundProcess = "nc_background_process"
	the option to do processing in a background process in PROOF
static const std::string	optOutputSampleName = "nc_outputSampleName"
	the output sample name
static const std::string	optGridDestSE = "nc_destSE"
	description: grid-specific options rationale: these are named so as to correspond to prun equivalents, bare the optGrid prefix.
static const std::string	optGridSite = "nc_site"
static const std::string	optGridExcludedSite = "nc_excludedSite"
static const std::string	optGridNGBPerJob = "nc_nGBPerJob"
static const std::string	optGridMemory = "nc_memory"
static const std::string	optGridMaxCpuCount = "nc_maxCpuCount"
static const std::string	optGridNFiles = "nc_nFiles"
static const std::string	optGridNFilesPerJob = "nc_nFilesPerJob"
static const std::string	optGridNJobs = "nc_nJobs"
static const std::string	optGridMaxFileSize = "nc_maxFileSize"
static const std::string	optGridMaxNFilesPerJob = "nc_maxNFilesPerJob"
static const std::string	optGridNEventsPerJob = "nc_nEventsPerJob"
static const std::string	optGridExpress = "nc_express"
static const std::string	optGridNoSubmit = "nc_noSubmit"
static const std::string	optGridMergeOutput = "nc_mergeOutput"
static const std::string	optGridAddNthFieldOfInDSToLFN = "nc_addNthFieldOfInDSToLFN"
static const std::string	optGridWorkingGroup = "nc_workingGroup"
static const std::string	optGridShowCmd = "nc_showCmd"
static const std::string	optGridCpuTimePerEvent = "nc_cpuTimePerEvent"
static const std::string	optGridMaxWalltime = "nc_maxWalltime"
static const std::string	optGridAvoidVP = "nc_avoidVP"
static const std::string	optGridPrunShipAdditionalFilesOrDirs = "nc_prunShipAdditionalFilesOrDirs"
	Enables to ship additional files to the tarbal sent to the grid Should be a list of comma separated path to files and/or directories Beware for root or heavy files you should also add their name (not their full path) to EL::Job::optUserFiles Otherwise prun ignore those files.
static const std::string	optGridPrunNRetrySubmitToGrid = "nc_prunNRetrySubmitToGrid"
	Will retry N-times submission of job to the grid in case of failure to submit job Sometimes submitting jobs to the grid can fail due to transient server issue This flag allows to retry submitting after waiting few seconds.
static const std::string	optTmpDir = "nc_tmpDir"
static const std::string	optRootVer = "nc_rootVer"
static const std::string	optCmtConfig = "nc_cmtConfig"
static const std::string	optGridDisableAutoRetry = "nc_disableAutoRetry"
static const std::string	optOfficial = "nc_official"
static const std::string	optVoms = "nc_voms"
static const std::string	optGridReporting = "nc_gridReporting"
	whether to use grid reporting even when not running on the grid
static const std::string	optUserFiles = "nc_EventLoop_UserFiles"
	a list of files that need to be available within the worker job
static const std::string	optBatchSharedFileSystem = "nc_sharedFileSystem"
	description: batch-specific options rationale: these options are for configuring batch drivers
static const std::string	optBatchSlurmExtraConfigLines = "nc_SlurmExtraConfig"
	The content of this string will be executed in the job script on the worker node before the main executable is run.
static const std::string	optBatchSlurmWrapperExec = "nc_SlurmWrapperExec"
	Append a command before the main executable is called This is useful is you want to execute the command e.g.
static const std::string	optBatchSetupCommand = "nc_BatchSetupCommand"
	This overrides the asetup command if you need to use a custom one.
static const std::string	optDockerImage = "nc_DockerImage"
	this is the name of the docker image, when using docker with a supported batch driver
static const std::string	optDockerOptions = "nc_DockerOptions"
	any extra options we may want to pass to docker
static const std::string	optBatchConfigFile = "nc_BatchConfigFile"
	the job submission configuration file (used by some drivers that need more complex configuration)
static const std::string	optBatchSetupFile = "nc_BatchSetupFile"
	the job submission setup file.
static const std::string	histogramStreamName = "HISTOGRAM"
	the name of the histogram output stream
static const std::string	optRetries = SH::MetaNames::openRetries()
	these options are defined in SH::MetaNames
static const std::string	optRetriesWait = SH::MetaNames::openRetriesWait()
Options controlling the memory monitoring behaviour of the job
static const std::string	optMemResidentPerEventIncreaseLimit
	The minimal per-event resident memory increase for triggering an error.
static const std::string	optMemVirtualPerEventIncreaseLimit
	The minimal per-event virtual memory increase for triggering an error.
static const std::string	optMemResidentIncreaseLimit = "nc_resMemAbsIncrease"
	The minimal resident memory increase necessary to trigger an error.
static const std::string	optMemVirtualIncreaseLimit = "nc_virtMemAbsIncrease"
	The minimal virtual memory increase necessary to trigger an error.
static const std::string	optMemFailOnLeak = "nc_failOnMemLeak"
	Failure behaviour of the code when a "significant memory leak" is found.

Private Attributes
SH::SampleHandler	m_sampleHandler
	description: members directly corresponding to accessors
std::vector< EL::OutputStream >	m_output
SH::MetaObject	m_options
EL::JobConfig	m_jobConfig
	the JobConfig object we use

Friends
void	swap (Job &a, Job &b)
	effects: standard swap guarantee: no-fail

Detailed Description

Definition at line 50 of file Job.h.

Member Typedef Documentation

outputIter

typedef const OutputStream* EL::Job::outputIter

Definition at line 148 of file Job.h.

outputMIter

typedef OutputStream* EL::Job::outputMIter

description: the list of output datasets used guarantee: no-fail / strong failures(3): out of memory II requires(3,soft): !outputHas (val_output.label()) note: while it is not a requirement it is highly recommended that you add the outputs from Algorithm::doSetupJob.

That way they get automatically enabled/disabled when you enable/disable the algorithms. Plus your code will be slightly less spread out.

Definition at line 147 of file Job.h.

Constructor & Destructor Documentation

Job() [1/2]

EL::Job::Job

(

)

effects: standard default constructor guarantee: strong failures: low level errors I

Definition at line 147 of file Job.cxx.

  {
    RCU::check_root_version ();
 
    RCU_NEW_INVARIANT (this);
  }

Job() [2/2]

EL::Job::Job

(

const Job &

that

)

effects: standard copy constructor guarantee: strong failures: out of memory II

Definition at line 157 of file Job.cxx.

    : m_sampleHandler ((RCU_READ_INVARIANT (&that),
            that.m_sampleHandler)),
      m_output (that.m_output),
      m_options (that.m_options),
      m_jobConfig (that.m_jobConfig)
  {
    RCU_NEW_INVARIANT (this);
  }

~Job()

EL::Job::~Job

(

)

effects: standard destructor guarantee: no-fail

Definition at line 170 of file Job.cxx.

  {
    RCU_DESTROY_INVARIANT (this);
  }

Member Function Documentation

algsAdd() [1/7]

void EL::Job::algsAdd

(

Algorithm *

alg_swallow

)

Definition at line 262 of file Job.cxx.

  {
    algsAdd (std::unique_ptr<Algorithm> (alg_swallow));
  }

algsAdd() [2/7]

void EL::Job::algsAdd

(

const AnaAlgorithmConfig &

config

)

Definition at line 270 of file Job.cxx.

  {
    // no invariant used
    if (config.useXAODs())
      useXAOD ();
    algsAdd (std::make_unique<AnaAlgorithmWrapper> (config));
  }

algsAdd() [3/7]

void EL::Job::algsAdd

(

const AnaReentrantAlgorithmConfig &

config

)

Definition at line 281 of file Job.cxx.

  {
    // no invariant used
    algsAdd (std::make_unique<AnaReentrantAlgorithmWrapper> (config));
  }

algsAdd() [4/7]

void EL::Job::algsAdd

(

const asg::AsgServiceConfig &

config

)

Definition at line 290 of file Job.cxx.

  {
    // no invariant used
    algsAdd (std::make_unique<AsgServiceWrapper> (config));
  }

algsAdd() [5/7]

void EL::Job::algsAdd

(

const EL::PythonConfigBase &

config

)

Definition at line 299 of file Job.cxx.

  {
    // no invariant used
    useXAOD ();
    if (config.componentType() == "AnaAlgorithm")
      algsAdd (std::make_unique<AnaAlgorithmWrapper> (AnaAlgorithmConfig (config)));
    else if (config.componentType() == "AnaReentrantAlgorithm")
      algsAdd (std::make_unique<AnaReentrantAlgorithmWrapper> (AnaReentrantAlgorithmConfig (config)));
    else if (config.componentType() == "AsgTool")
      algsAdd (std::make_unique<AsgToolWrapper> (asg::AsgToolConfig (config)));
    else if (config.componentType() == "AsgService")
      algsAdd (std::make_unique<AsgServiceWrapper> (asg::AsgServiceConfig (config)));
    else
      RCU_THROW_MSG ("unknown component type: \"" + config.componentType() + "\"");
  }

algsAdd() [6/7]

void EL::Job::algsAdd

(

std::unique_ptr< Algorithm >

val_algorithm

)

Definition at line 220 of file Job.cxx.

  {
    using namespace msgEventLoop;
 
    RCU_CHANGE_INVARIANT (this);
    RCU_REQUIRE_SOFT (val_algorithm != nullptr);
 
 
    std::string myname = val_algorithm->GetName();
    if (myname.empty() || algsHas (myname))
    {
      if (myname.empty())
        myname = "UnnamedAlgorithm";
      bool unique = false;
      for (unsigned iter = 1; !unique; ++ iter)
      {
        std::ostringstream str;
        str << myname << iter;
        if (!algsHas (str.str()))
        {
          myname = str.str();
          unique = true;
        }
      }
      if (strlen (val_algorithm->GetName()) > 0)
        ANA_MSG_WARNING ("renaming algorithm " << val_algorithm->GetName() << " to " << myname << " to make the name unique");
      val_algorithm->SetName (myname.c_str());
      if (val_algorithm->GetName() != myname)
      {
        std::ostringstream message;
        message << "failed to rename algorithm " << val_algorithm->GetName() << " to " << myname;
        RCU_THROW_MSG (message.str());
      }
    }
 
    val_algorithm->sysSetupJob (*this);
    algsAdd (std::make_unique<AlgorithmWrapper> (std::move (val_algorithm)));
  }

algsAdd() [7/7]

void EL::Job::algsAdd

(

std::unique_ptr< IAlgorithmWrapper >

val_algorithm

)

description: the list of algorithms used guarantee: no-fail / strong failures: out of memory II invariant: alg != 0

Definition at line 207 of file Job.cxx.

  {
    using namespace msgEventLoop;
 
    RCU_CHANGE_INVARIANT (this);
    RCU_REQUIRE_SOFT (val_algorithm != nullptr);
 
    ANA_CHECK_THROW (m_jobConfig.addAlgorithm (std::move (val_algorithm)));
  }

algsAddClone()

void EL::Job::algsAddClone

(

const Algorithm &

alg

)

add a clone of the given algorithm

This is to be used when the user wants to reuse an algorithm for multiple Job objects or for some reason needs to delete it himself.

Guarantee

strong

Failures

algorithm cloning failures
out of memory I

Definition at line 318 of file Job.cxx.

  {
    // no invariant used
    algsAdd (dynamic_cast<Algorithm*>(alg.Clone()));
  }

algsHas()

bool EL::Job::algsHas

(

const std::string &

name

)

const

returns: whether we have the algorithm with the given name guarantee: no-fail

Definition at line 327 of file Job.cxx.

  {
    RCU_READ_INVARIANT (this);
    return m_jobConfig.getAlgorithm (name) != nullptr;
  }

jobConfig()

const JobConfig & EL::Job::jobConfig ( ) const

noexcept

the JobConfig object we are wrapping

Guarantee

no-fail

Definition at line 434 of file Job.cxx.

  {
    RCU_READ_INVARIANT (this);
    return m_jobConfig;
  }

operator=()

Job & EL::Job::operator=

(

const Job &

that

)

effects: standard assignment operator returns: *this guarantee: strong failures: out of memory II

Definition at line 178 of file Job.cxx.

  {
    // no invariant used
    Job tmp (that);
    swap (*this, tmp);
    return *this;
  }

options() [1/2]

SH::MetaObject * EL::Job::options

(

)

description: the list of options to the job guarantee: no-fail postcondition: result != 0

Definition at line 416 of file Job.cxx.

  {
    RCU_READ_INVARIANT (this);
    return &m_options;
  }

options() [2/2]

const SH::MetaObject * EL::Job::options

(

)

const

Definition at line 425 of file Job.cxx.

  {
    RCU_READ_INVARIANT (this);
    return &m_options;
  }

outputAdd()

void EL::Job::outputAdd

(

const OutputStream &

val_output

)

Definition at line 381 of file Job.cxx.

  {
    RCU_CHANGE_INVARIANT (this);
    RCU_REQUIRE2_SOFT (!outputHas (val_output.label()), ("trying to create two output streams with the label " + val_output.label() + "\nplease make sure that every output stream has a unique label").c_str());
    m_output.push_back (val_output);
  }

outputBegin() [1/2]

Job::outputMIter EL::Job::outputBegin

(

)

Definition at line 345 of file Job.cxx.

  {
    RCU_READ_INVARIANT (this);
    return ( m_output.size() ? &m_output[0] : nullptr );
  }

outputBegin() [2/2]

Job::outputIter EL::Job::outputBegin

(

)

const

Definition at line 354 of file Job.cxx.

  {
    RCU_READ_INVARIANT (this);
    return ( m_output.size() ? &m_output[0] : nullptr );
  }

outputEnd() [1/2]

Job::outputMIter EL::Job::outputEnd

(

)

Definition at line 363 of file Job.cxx.

  {
    RCU_READ_INVARIANT (this);
    return ( m_output.size() ? &m_output[m_output.size()] : nullptr );
  }

outputEnd() [2/2]

Job::outputIter EL::Job::outputEnd

(

)

const

Definition at line 372 of file Job.cxx.

  {
    RCU_READ_INVARIANT (this);
    return ( m_output.size() ? &m_output[m_output.size()] : nullptr );
  }

outputHas()

bool EL::Job::outputHas

(

const std::string &

name

)

const

returns: whether we have an output with the given name guarantee: no-fail

Definition at line 391 of file Job.cxx.

  {
    RCU_CHANGE_INVARIANT (this);
    for (outputIter iter = outputBegin(),
       end = outputEnd(); iter != end; ++ iter)
    {
      if (iter->label() == name)
    return true;
    }
    return false;
  }

sampleHandler() [1/2]

const SH::SampleHandler & EL::Job::sampleHandler

(

)

const

description: the sample handler used guarantee: no-fail / strong failures: out of memory II

Definition at line 189 of file Job.cxx.

  {
    RCU_READ_INVARIANT (this);
    return m_sampleHandler;
  }

sampleHandler() [2/2]

void EL::Job::sampleHandler

(

const SH::SampleHandler &

val_sampleHandler

)

Definition at line 198 of file Job.cxx.

  {
    RCU_CHANGE_INVARIANT (this);
    m_sampleHandler = val_sampleHandler;
  }

startNewAlgSequence()

void EL::Job::startNewAlgSequence

(

)

start a new sub-sequence of algorithms

Definition at line 336 of file Job.cxx.

  {
    RCU_CHANGE_INVARIANT (this);
    m_jobConfig.startNewAlgSequence ();
  }

testInvariant()

void EL::Job::testInvariant

(

)

const

effects: test the invariant of this object guarantee: no-fail

Definition at line 139 of file Job.cxx.

  {
    RCU_INVARIANT (this);
  }

useXAOD()

void EL::Job::useXAOD

(

)

effects: register this job to use XAODs guarantee: strong failures: out of memory II failures: EventSvc not available

Definition at line 406 of file Job.cxx.

  {
    RCU_CHANGE_INVARIANT (this);
 
    options()->setBool (Job::optXAODInput, true);
  }

Friends And Related Symbol Documentation

swap

void swap ( Job & a, Job & b )

friend

effects: standard swap guarantee: no-fail

Definition at line 130 of file Job.cxx.

  {
    swap (a.m_sampleHandler, b.m_sampleHandler);
    a.m_jobConfig.swap (b.m_jobConfig);
    swap (a.m_output, b.m_output);
  }

Member Data Documentation

histogramStreamName

const std::string EL::Job::histogramStreamName = "HISTOGRAM"

static

the name of the histogram output stream

Normally users don't need to worry about the histogram output stream, and it just gets created for you automatically, but in some situations you will need to reconfigure the histogram output stream.

Definition at line 606 of file Job.h.

m_jobConfig

EL::JobConfig EL::Job::m_jobConfig

private

the JobConfig object we use

Definition at line 626 of file Job.h.

m_options

SH::MetaObject EL::Job::m_options

private

Definition at line 622 of file Job.h.

m_output

std::vector<EL::OutputStream> EL::Job::m_output

private

Definition at line 620 of file Job.h.

m_sampleHandler

SH::SampleHandler EL::Job::m_sampleHandler

private

description: members directly corresponding to accessors

Definition at line 618 of file Job.h.

optAlgorithmMemoryMonitor

const std::string EL::Job::optAlgorithmMemoryMonitor = "nc_EventLoop_AlgorithmMemoryMonitor"

static

a boolean flag for whether to add a memory monitor for the algorithms

Definition at line 215 of file Job.h.

optAlgorithmTimer

const std::string EL::Job::optAlgorithmTimer = "nc_EventLoop_AlgorithmTimer"

static

a boolean flag for whether to add a timer for the algorithms

Definition at line 210 of file Job.h.

optBackgroundProcess

const std::string EL::Job::optBackgroundProcess = "nc_background_process"

static

the option to do processing in a background process in PROOF

Definition at line 445 of file Job.h.

optBatchConfigFile

const std::string EL::Job::optBatchConfigFile = "nc_BatchConfigFile"

static

the job submission configuration file (used by some drivers that need more complex configuration)

Definition at line 536 of file Job.h.

optBatchSetupCommand

const std::string EL::Job::optBatchSetupCommand = "nc_BatchSetupCommand"

static

This overrides the asetup command if you need to use a custom one.

Definition at line 525 of file Job.h.

optBatchSetupFile

const std::string EL::Job::optBatchSetupFile = "nc_BatchSetupFile"

static

the job submission setup file.

unlike optBatchConfigFile this only gets used once per submission instead of once per job.

Definition at line 541 of file Job.h.

optBatchSharedFileSystem

const std::string EL::Job::optBatchSharedFileSystem = "nc_sharedFileSystem"

static

description: batch-specific options rationale: these options are for configuring batch drivers

Definition at line 517 of file Job.h.

optBatchSlurmExtraConfigLines

const std::string EL::Job::optBatchSlurmExtraConfigLines = "nc_SlurmExtraConfig"

static

The content of this string will be executed in the job script on the worker node before the main executable is run.

Definition at line 520 of file Job.h.

optBatchSlurmWrapperExec

const std::string EL::Job::optBatchSlurmWrapperExec = "nc_SlurmWrapperExec"

static

Append a command before the main executable is called This is useful is you want to execute the command e.g.

within shifter.

Definition at line 523 of file Job.h.

optCacheLearnEntries

const std::string EL::Job::optCacheLearnEntries = "nc_EventLoop_CacheLearnEntries"

static

description: this option allows to configure the number of tree entries used for learning cache behavior.

rationale: there is a trade-off here, if you set this too low you will fail to cache more rarely used variables. if you set it too high it will take too long until the cache kicks in. warning: right now this is performed on a per-file basis, which at some point will be raised to a per-job basis, which is the limit of how far caching can go in this approach. if you need multi-job support, either use D3PDReader or contact me on how to do this.

Definition at line 332 of file Job.h.

optCacheSize

const std::string EL::Job::optCacheSize = "nc_cache_size"

static

description: this option allows to configure the TTreeCache size for this job.

warning: this has to be synchronized with SampleHandler::MetaFields.

if it is smaller or equal to 0, the cache is turned off. rationale: if you read data from across the network using the cache mechanism will greatly increase your performance. for local disks you may gain or lose performance.

I can't just copy it here, because the order of initialization is undefined

Definition at line 317 of file Job.h.

optCmtConfig

const std::string EL::Job::optCmtConfig = "nc_cmtConfig"

static

Definition at line 492 of file Job.h.

optCondorConf

const std::string EL::Job::optCondorConf = "nc_EventLoop_CondorConf"

static

description: the name of the option for supplying extra parameters for condor systems

Definition at line 307 of file Job.h.

optD3PDCacheMinByte

const std::string EL::Job::optD3PDCacheMinByte = "nc_EventLoop_D3PDCacheMinByte"

static

Definition at line 383 of file Job.h.

optD3PDCacheMinByteFraction

const std::string EL::Job::optD3PDCacheMinByteFraction = "nc_EventLoop_D3PDCacheMinByteFraction"

static

Definition at line 384 of file Job.h.

optD3PDCacheMinEvent

const std::string EL::Job::optD3PDCacheMinEvent = "nc_EventLoop_D3PDCacheMinEvent"

static

description: these options configure the D3PDReader TTreeCache settings.

if you use more than one option, the result is undefined. warning: this only works if you use D3PDReader rationale: the idea is that in your first job you create the D3PDReadStats object, which you then pass to subsequent jobs.

Definition at line 381 of file Job.h.

optD3PDCacheMinEventFraction

const std::string EL::Job::optD3PDCacheMinEventFraction = "nc_EventLoop_D3PDCacheMinEventFraction"

static

Definition at line 382 of file Job.h.

optD3PDPerfStats

const std::string EL::Job::optD3PDPerfStats = "nc_EventLoop_D3PDPerfStats"

static

description: the name of the option for turning on D3PDPerfStats.

To use D3PDPerfStats set this to a non-zero value. warning: this only works if you read data through D3PDReader rationale: this can be used for fine-tuning TTreeCache or generally optimizing i/o performance

Definition at line 342 of file Job.h.

optD3PDReadStats

const std::string EL::Job::optD3PDReadStats = "nc_EventLoop_D3PDReadStats"

static

description: the name of the D3PDPerfStats object produced as I gather it, as well as the name of the option passed into the job.

rationale: I use the same name in two places to allow reading the stats object from one output and passing it to the next

Definition at line 351 of file Job.h.

optDisableMetrics

const std::string EL::Job::optDisableMetrics = "nc_disable_metrics"

static

description: the option to turn off collection of performance data

Definition at line 418 of file Job.h.

optDockerImage

const std::string EL::Job::optDockerImage = "nc_DockerImage"

static

this is the name of the docker image, when using docker with a supported batch driver

Definition at line 529 of file Job.h.

optDockerOptions

const std::string EL::Job::optDockerOptions = "nc_DockerOptions"

static

any extra options we may want to pass to docker

Definition at line 532 of file Job.h.

optEventsPerWorker

const std::string EL::Job::optEventsPerWorker = "nc_EventLoop_EventsPerWorker"

static

description: the name of the option for selecting the number of events per batch job.

(only BatchDriver and derived drivers). warning: this option will be ignored unless you have called SH::scanNEvents first. rationale: this allows to make your jobs all approximately equal size and gives you much finer control of the lengths of your jobs. if you did run SH::scanNEvents, but didn't specify a number of events per worker this will produce the same number of jobs as you would have otherwise, but spread the workload more evenly.

Definition at line 279 of file Job.h.

optFactoryPreload

const std::string EL::Job::optFactoryPreload = "nc_EventLoop_FactoryPreload"

static

a boolean flag for whether to perform a component factory preload

Definition at line 219 of file Job.h.

optFilesPerWorker

const std::string EL::Job::optFilesPerWorker = "nc_EventLoop_FilesPerWorker"

static

description: the name of the option for selecting the number of files per batch job.

(only BatchDriver and derived drivers). rationale: this is typically used if you are either processing fairly small input files or are producing a skim with a very high reduction factor. it will reduce the inefficiency associated with starting a lot of jobs or reading a lot of output files

Definition at line 264 of file Job.h.

optGridAddNthFieldOfInDSToLFN

const std::string EL::Job::optGridAddNthFieldOfInDSToLFN = "nc_addNthFieldOfInDSToLFN"

static

Definition at line 472 of file Job.h.

optGridAvoidVP

const std::string EL::Job::optGridAvoidVP = "nc_avoidVP"

static

Definition at line 477 of file Job.h.

optGridCpuTimePerEvent

const std::string EL::Job::optGridCpuTimePerEvent = "nc_cpuTimePerEvent"

static

Definition at line 475 of file Job.h.

optGridDestSE

const std::string EL::Job::optGridDestSE = "nc_destSE"

static

description: grid-specific options rationale: these are named so as to correspond to prun equivalents, bare the optGrid prefix.

Definition at line 457 of file Job.h.

optGridDisableAutoRetry

const std::string EL::Job::optGridDisableAutoRetry = "nc_disableAutoRetry"

static

Definition at line 493 of file Job.h.

optGridExcludedSite

const std::string EL::Job::optGridExcludedSite = "nc_excludedSite"

static

Definition at line 459 of file Job.h.

optGridExpress

const std::string EL::Job::optGridExpress = "nc_express"

static

Definition at line 469 of file Job.h.

optGridMaxCpuCount

const std::string EL::Job::optGridMaxCpuCount = "nc_maxCpuCount"

static

Definition at line 462 of file Job.h.

optGridMaxFileSize

const std::string EL::Job::optGridMaxFileSize = "nc_maxFileSize"

static

Definition at line 466 of file Job.h.

optGridMaxNFilesPerJob

const std::string EL::Job::optGridMaxNFilesPerJob = "nc_maxNFilesPerJob"

static

Definition at line 467 of file Job.h.

optGridMaxWalltime

const std::string EL::Job::optGridMaxWalltime = "nc_maxWalltime"

static

Definition at line 476 of file Job.h.

optGridMemory

const std::string EL::Job::optGridMemory = "nc_memory"

static

Definition at line 461 of file Job.h.

optGridMergeOutput

const std::string EL::Job::optGridMergeOutput = "nc_mergeOutput"

static

Definition at line 471 of file Job.h.

optGridNEventsPerJob

const std::string EL::Job::optGridNEventsPerJob = "nc_nEventsPerJob"

static

Definition at line 468 of file Job.h.

optGridNFiles

const std::string EL::Job::optGridNFiles = "nc_nFiles"

static

Definition at line 463 of file Job.h.

optGridNFilesPerJob

const std::string EL::Job::optGridNFilesPerJob = "nc_nFilesPerJob"

static

Definition at line 464 of file Job.h.

optGridNGBPerJob

const std::string EL::Job::optGridNGBPerJob = "nc_nGBPerJob"

static

Definition at line 460 of file Job.h.

optGridNJobs

const std::string EL::Job::optGridNJobs = "nc_nJobs"

static

Definition at line 465 of file Job.h.

optGridNoSubmit

const std::string EL::Job::optGridNoSubmit = "nc_noSubmit"

static

Definition at line 470 of file Job.h.

optGridPrunNRetrySubmitToGrid

const std::string EL::Job::optGridPrunNRetrySubmitToGrid = "nc_prunNRetrySubmitToGrid"

static

Will retry N-times submission of job to the grid in case of failure to submit job Sometimes submitting jobs to the grid can fail due to transient server issue This flag allows to retry submitting after waiting few seconds.

Definition at line 488 of file Job.h.

optGridPrunShipAdditionalFilesOrDirs

const std::string EL::Job::optGridPrunShipAdditionalFilesOrDirs = "nc_prunShipAdditionalFilesOrDirs"

static

Enables to ship additional files to the tarbal sent to the grid Should be a list of comma separated path to files and/or directories Beware for root or heavy files you should also add their name (not their full path) to EL::Job::optUserFiles Otherwise prun ignore those files.

Definition at line 482 of file Job.h.

optGridReporting

const std::string EL::Job::optGridReporting = "nc_gridReporting"

static

whether to use grid reporting even when not running on the grid

Definition at line 498 of file Job.h.

optGridShowCmd

const std::string EL::Job::optGridShowCmd = "nc_showCmd"

static

Definition at line 474 of file Job.h.

optGridSite

const std::string EL::Job::optGridSite = "nc_site"

static

Definition at line 458 of file Job.h.

optGridWorkingGroup

const std::string EL::Job::optGridWorkingGroup = "nc_workingGroup"

static

Definition at line 473 of file Job.h.

optLocalNoUnsetup

const std::string EL::Job::optLocalNoUnsetup = "nc_local_no_unsetup"

static

the option not to unsetup the environment in LocalDriver

Definition at line 430 of file Job.h.

optMaxEvents

const std::string EL::Job::optMaxEvents = "nc_EventLoop_MaxEvents"

static

description: the name of the option used for setting the maximum number of events to process per sample rationale: this is used for test runs where you don't want to process all events, but just want a quick look

Definition at line 230 of file Job.h.

optMemFailOnLeak

const std::string EL::Job::optMemFailOnLeak = "nc_failOnMemLeak"

static

Failure behaviour of the code when a "significant memory leak" is found.

This flag allows the user to select what should happen when the code finds a memory leak in the job that is larger than the values set by optMemResidentPerEventIncreaseLimit, optMemVirtualPerEventIncreaseLimit, optMemResidentIncreaseLimit and optMemVirtualIncreaseLimit.

It's a boolean property. When set to true, the job fails if a significant memory leak is detected. If set to false, only a warning is printed.

Definition at line 595 of file Job.h.

optMemResidentIncreaseLimit

const std::string EL::Job::optMemResidentIncreaseLimit = "nc_resMemAbsIncrease"

static

The minimal resident memory increase necessary to trigger an error.

It is an integer property, setting the limit in kilobytes. Jobs have to increase their resident memory usage by this amount to trigger a failure.

Definition at line 573 of file Job.h.

optMemResidentPerEventIncreaseLimit

const std::string EL::Job::optMemResidentPerEventIncreaseLimit

static

Initial value:

                                                        =
"nc_resMemPerEventIncrease"

The minimal per-event resident memory increase for triggering an error.

This is the main setting for triggering a failure in memory-leaking analysis jobs. It sets the limit on the per-event resident memory increase of the job for it to be still successful. It is an integer property, setting the limit in kilobytes.

Definition at line 555 of file Job.h.

optMemVirtualIncreaseLimit

const std::string EL::Job::optMemVirtualIncreaseLimit = "nc_virtMemAbsIncrease"

static

The minimal virtual memory increase necessary to trigger an error.

Implemented very similarly to optMemResidentIncreaseLimit. Since normally the virtual memory usage is not considered in producing a failure, it is set to zero by default.

Definition at line 581 of file Job.h.

optMemVirtualPerEventIncreaseLimit

const std::string EL::Job::optMemVirtualPerEventIncreaseLimit

static

Initial value:

                                                       =
"nc_virtMemPerEventIncrease"

The minimal per-event virtual memory increase for triggering an error.

Implemented very similarly to optMemResidentPerEventIncreaseLimit. But since normally we don't care about the virtual memory usage of the jobs that much, it is set to zero by default. Making optMemResidentPerEventIncreaseLimit control the behaviour of the job.

Definition at line 565 of file Job.h.

optNumParallelProcs

const std::string EL::Job::optNumParallelProcs = "nc_num_parallel_processes"

static

the option to specify the number of parallel jobs in LocalDriver (0 = number of hardware cores) (default=1)

As an intermediate between running a single job locally and running in batch, this allows to run multiple processes in parallel locally. It is still recommended to just run in batch instead, but sometimes this is more practical.

Definition at line 440 of file Job.h.

optOfficial

const std::string EL::Job::optOfficial = "nc_official"

static

Definition at line 494 of file Job.h.

optOutputSampleName

const std::string EL::Job::optOutputSampleName = "nc_outputSampleName"

static

the output sample name

Definition at line 450 of file Job.h.

optPerfTree

const std::string EL::Job::optPerfTree = "nc_EventLoop_PerfTree"

static

description: the option to turn on the performance tree in PROOF.

if this is set to 1, it will write out the tree rationale: this can help in tuning your PROOF cluster or tuning what you do in PROOF, but it may have some overhead, which is why we don't do it by default

Definition at line 393 of file Job.h.

optPrintPerFileStats

const std::string EL::Job::optPrintPerFileStats = "nc_print_per_file_stats"

static

description: the option to turn on printing of i/o statistics at the end of each file rationale: while this is not as complete as whole sample statistics, it can be helpful in some circumstances when debugging the performance warning: this is not supported for all drivers

Definition at line 413 of file Job.h.

optRemoveSubmitDir

const std::string EL::Job::optRemoveSubmitDir = "nc_EventLoop_RemoveSubmitDir"

static

description: the name of the option for overwriting the submission directory.

if you set this to a non-zero value it will remove any existing submit-directory before trying to create a new one. rationale: normally you don't want to silently remove an existing submission directory, since it may contain valuable data, but for some cases like debugging you may be annoyed to delete it manually.

Definition at line 194 of file Job.h.

optResetShell

const std::string EL::Job::optResetShell = "nc_reset_shell"

static

description: the option to reset the shell on the worker nodes rationale: this is currently only used by the LSFDriver where it is enabled by default to reset it on lxbatch.

Definition at line 425 of file Job.h.

optRetries

const std::string EL::Job::optRetries = SH::MetaNames::openRetries()

static

these options are defined in SH::MetaNames

Definition at line 503 of file Job.h.

optRetriesWait

const std::string EL::Job::optRetriesWait = SH::MetaNames::openRetriesWait()

static

Definition at line 504 of file Job.h.

optRootVer

const std::string EL::Job::optRootVer = "nc_rootVer"

static

Definition at line 491 of file Job.h.

optSkipEvents

const std::string EL::Job::optSkipEvents = "nc_EventLoop_SkipEvents"

static

description: the name of the option used for skipping a certain number of events in the beginning rationale: this is used for test runs where you are only interested in a particular set of events

Definition at line 238 of file Job.h.

optStreamAliases

const std::string EL::Job::optStreamAliases = "nc_EventLoop_StreamAliases"

static

an option for stream aliases

Definition at line 222 of file Job.h.

optSubmitDirMode

const std::string EL::Job::optSubmitDirMode = "nc_EventLoop_SubmitDirMode"

static

the submit-dir mode (allowed values: "no-clobber", "overwrite", "unique", "unique-link")

Definition at line 200 of file Job.h.

optSubmitFlags

const std::string EL::Job::optSubmitFlags = "nc_EventLoop_SubmitFlags"

static

description: the name of the option for supplying extra submit parameters to batch systems rationale: the primary purpose of this is to allow selecting the right queue on your batch system, but it can be used for all kind of submission options that EventLoop doesn't support directly.

Definition at line 301 of file Job.h.

optTmpDir

const std::string EL::Job::optTmpDir = "nc_tmpDir"

static

Definition at line 490 of file Job.h.

optUniqueDateFormat

const std::string EL::Job::optUniqueDateFormat = "nc_EventLoop_UniqueDateFormat"

static

the date-format to use when generating unique submission directory names

Definition at line 205 of file Job.h.

optUserFiles

const std::string EL::Job::optUserFiles = "nc_EventLoop_UserFiles"

static

a list of files that need to be available within the worker job

Definition at line 510 of file Job.h.

optVoms

const std::string EL::Job::optVoms = "nc_voms"

static

Definition at line 495 of file Job.h.

optWorkerConfigFile

const std::string EL::Job::optWorkerConfigFile = "nc_EventLoop_WorkerConfigFile"

static

a python configuration file that will be executed on the worker

This allows to inspect the meta-data of the first input file and configure the job accordingly.

EXPERIMENTAL: This feature is currently (23 Feb 23) new and experimental and details of its implementation and usage may still change.

Definition at line 251 of file Job.h.

optWorkerPostClosedOutputsExecutable

const std::string EL::Job::optWorkerPostClosedOutputsExecutable = "nc_EventLoop_WorkerPostClosedOutputsExecutable"

static

Optionnal executable to be provided by the user that will be called, by the worker, for every output stream.

It enables to perform action on output files once they are all closed. For every output stream the executable will be provided 2 arguments:

• the output stream name
• the output file path You can for instance implement an executable to check that the output files produced by your job are not corrupted. Or perform other type of actions if you want.

Definition at line 291 of file Job.h.

optXAODInput

const std::string EL::Job::optXAODInput = "nc_EventLoop_XAODInput"

static

the option to select whether our input is xAODs

Definition at line 398 of file Job.h.

optXAODPerfStats

const std::string EL::Job::optXAODPerfStats = "nc_EventLoop_XAODPerfStats"

static

description: the name of the option for turning on XAODPerfStats.

To use XAODPerfStats set this to a non-zero value. warning: this only works if you read data through XAODReader rationale: this can be used for fine-tuning TTreeCache or generally optimizing i/o performance

Definition at line 361 of file Job.h.

optXAODReadStats

const std::string EL::Job::optXAODReadStats = "nc_EventLoop_XAODReadStats"

static

description: the name of the XAODPerfStats object produced as I gather it, as well as the name of the option passed into the job.

rationale: I use the same name in two places to allow reading the stats object from one output and passing it to the next

Definition at line 370 of file Job.h.

optXAODSummaryReport

const std::string EL::Job::optXAODSummaryReport = "nc_xaod_summary_report"

static

the option to turn on/off the xAOD summary reporting at the end of the job

Definition at line 403 of file Job.h.

---

The documentation for this class was generated from the following files:

• Job.h
• Job.cxx