CP::MakeSystematicsVector Class Reference

This class handles turning the list of systematics into the actual list of nuisance parameter points to evaluate. More...

#include <MakeSystematicsVector.h>

Collaboration diagram for CP::MakeSystematicsVector:

Classes
struct	GroupConfig
	the configuration for the given group More...

Public Member Functions
void	testInvariant () const
	test the invariant of this object More...
	MakeSystematicsVector ()
	standard default constructor More...
const std::vector< SystematicSet > &	result (const std::string &label) const
	the list of nuisance parameter points generated with the given label More...
void	calc (const SystematicSet &sysList)
	fill in result More...
void	addGroup (const std::string &val_label)
	finish configuration for this group and add a new one More...
void	setPattern (const std::string &val_pattern)
	set the pattern for the current group More...
void	setSigma (float val_sigma)
	set the number of sigmas to vary this group by More...
void	setToys (unsigned val_toys)
	set the number of toys to run for this group More...
void	useForNominal ()
	set this group as the default, i.e. More...

Private Member Functions
std::vector< std::map< std::string, std::vector< SystematicVariation > > >	calcBaseSys (const SystematicSet &sysList)
	make the list of base systematics for calc More...

Private Attributes
std::map< std::string, std::vector< SystematicSet > >	m_result
	the value of result More...
std::vector< GroupConfig >	m_config
	the configuration on a per-group basis More...
std::string	m_useForNominal
	the group for which useForNominal was set More...

Detailed Description

This class handles turning the list of systematics into the actual list of nuisance parameter points to evaluate.

This is meant as a placeholder for a generic tool to be developed by the statistics forum (or as a prototype for it).

For now I decided to keep it as a single class, but there are other options, e.g. the member MakeSystematicsVector::Config could be made a class that the user configures directly and then passes in. However, for now this approach seems better, as it hides some of the mechanics from the user and gives me more freedom on the backend side.

Definition at line 33 of file MakeSystematicsVector.h.

Constructor & Destructor Documentation

MakeSystematicsVector()

CP::MakeSystematicsVector::MakeSystematicsVector

(

)

standard default constructor

Guarantee

no-fail

Definition at line 71 of file MakeSystematicsVector.cxx.

    : m_config (1)
  {
    RCU_NEW_INVARIANT (this);
  }

Member Function Documentation

addGroup()

void CP::MakeSystematicsVector::addGroup

(

const std::string &

val_label

)

finish configuration for this group and add a new one

Parameters

val_label

the label for the new group

Guarantee

strong

Failures

out of memory II

Definition at line 182 of file MakeSystematicsVector.cxx.

  {
    RCU_CHANGE_INVARIANT (this);
    GroupConfig config;
    config.label = val_label;
    m_config.push_back (config);
  }

calc()

void CP::MakeSystematicsVector::calc

(

const SystematicSet &

sysList

)

fill in result

Parameters

sysList

the list of systematics to use, usually the list of recommended systematics from your CP tools, framework or the systematics registry

Guarantee

strong

Failures

out of memory II
configuration errors

Definition at line 93 of file MakeSystematicsVector.cxx.

  {
    RCU_CHANGE_INVARIANT (this);
 
    auto baseSys = calcBaseSys (sysList);
 
    std::map<std::string,std::vector<SystematicSet>> myresult;
    myresult[m_useForNominal].push_back (SystematicSet ());
    for (std::size_t group = 0; group != m_config.size(); ++ group)
    {
      const auto& config = m_config[group];
 
      // note: this is not just a short-cut, but also makes sure that
      // we have an entry for each label, even if there are no
      // systematics for the label
      auto& subresult = myresult[config.label];
 
      // this skips groups that don't match any requested systematics,
      // which is mainly important for toy systematics as you wouldn't
      // want to generate a bunch of empty systematics
      if (baseSys[group].empty())
    continue;
 
      if (config.toys == 0)
      {
    for (auto sys : baseSys[group])
    {
      RCU_ASSERT (!sys.second.empty());
      RCU_ASSERT (!sys.second.front().isToyEnsemble());
      if (sys.second.front().isContinuousEnsemble())
      {
        // for continuous systematics
        subresult.push_back(CP::SystematicSet());
        subresult.back().insert (CP::SystematicVariation (sys.first, config.sigma));
        subresult.push_back(CP::SystematicSet());
        subresult.back().insert (CP::SystematicVariation (sys.first, -config.sigma));
      } else if (sys.second.front().isEnsemble())
      {
        // we must have added a new kind of ensemble after I wrote
        // this code
        RCU_THROW_MSG ("unsupported ensemble systematic: " + sys.first);
      } else
      {
        // otherwise just add all of them flat
        for (auto mysys : sys.second)
        {
          subresult.push_back(CP::SystematicSet());
          subresult.back().insert(mysys);
        }
      }
    }
      } else
      {
    std::vector<CP::SystematicSet> toys (config.toys);
 
    for (auto sys : baseSys[group])
    {
      RCU_ASSERT (!sys.second.empty());
      RCU_ASSERT (sys.second.front().isEnsemble());
 
      if (sys.second.front().isContinuousEnsemble())
      {
        std::unique_ptr<TRandom3> random (new TRandom3);
        random->SetSeed (hash_string (sys.first));
 
        for (auto& toy : toys)
          toy.insert (CP::SystematicVariation (sys.first, random->Gaus (0, config.sigma)));
      } else if (sys.second.front().isToyEnsemble())
      {
        for (unsigned toy = 0; toy != config.toys; ++ toy)
          toys[toy].insert (CP::SystematicVariation::makeToyVariation (sys.first, toy + 1, config.sigma));
      } else
      {
        // we must have added a new kind of ensemble after I
        // wrote this code
        RCU_THROW_MSG ("unsupported ensemble systematic for toys: " + sys.first);
      }
    }
    for (auto& toy : toys)
      subresult.push_back (std::move (toy));
      }
    }
 
    m_result = myresult;
  }

calcBaseSys()

std::vector< std::map< std::string, std::vector< SystematicVariation > > > CP::MakeSystematicsVector::calcBaseSys ( const SystematicSet &  sysList )

private

make the list of base systematics for calc

Guarantee

strong

Failures

out of memory II

Definition at line 232 of file MakeSystematicsVector.cxx.

  {
    std::map<std::string,std::vector<SystematicVariation> > basesys;
    for (auto sys : sysList)
    {
      basesys[sys.basename()].push_back (sys);
    }
    std::vector<std::map<std::string,std::vector<SystematicVariation> >>
      basesysList (m_config.size());
    for (auto sys : basesys)
    {
      // extract the ensemble if we have one
      SystematicVariation ensemble;
      for (auto mysys : sys.second)
      {
    if (mysys.isEnsemble())
    {
      if (!ensemble.empty())
        RCU_THROW_MSG ("inconsistent ensembles requested: " + ensemble.name() + " " + mysys.name());
      ensemble = mysys;
    }
      }
 
      // setting this beyond the valid groups in case none matches
      std::size_t group = m_config.size();
      for (std::size_t iter = 0; iter != m_config.size(); ++ iter)
      {
    if (m_config[iter].pattern.empty())
    {
      // only use empty patterns if no previous pattern already took this
      if (group == m_config.size())
      {
        if (m_config[iter].toys > 0)
        {
          if (ensemble.isToyEnsemble())
        group = iter;
        } else
        {
          if (!ensemble.isToyEnsemble())
        group = iter;
        }
      }
    } else if (RCU::match_expr (boost::regex (m_config[iter].pattern.c_str()), sys.first))
    {
      if (m_config[iter].toys > 0 && ensemble.empty())
        RCU_THROW_MSG ("toys only supported for ensemble systematics");
      group = iter;
    }
      }
      if (group == m_config.size())
    RCU_THROW_MSG ("no systematics group for systematic: " + sys.first);
 
      if (!ensemble.empty())
      {
    basesysList[group][sys.first].push_back (ensemble);
      } else
      {
    basesysList[group][sys.first] = std::move (sys.second);
      }
    }
    return basesysList;
  }

result()

const std::vector< SystematicSet > & CP::MakeSystematicsVector::result

(

const std::string &

label

)

const

the list of nuisance parameter points generated with the given label

Guarantee

strong

Failures

unknown label

Precondition

calculate() has been called

Definition at line 80 of file MakeSystematicsVector.cxx.

  {
    RCU_READ_INVARIANT (this);
    RCU_REQUIRE2 (!m_result.empty(), "calculate() has been called");
    auto iter = m_result.find (label);
    if (iter == m_result.end())
      RCU_THROW_MSG ("unknown systematics group: " + label);
    return iter->second;
  }

setPattern()

void CP::MakeSystematicsVector::setPattern

(

const std::string &

val_pattern

)

set the pattern for the current group

Guarantee

strong

Failures

out of memory II

Definition at line 193 of file MakeSystematicsVector.cxx.

  {
    RCU_CHANGE_INVARIANT (this);
    m_config.back().pattern = val_pattern;
  }

setSigma()

void CP::MakeSystematicsVector::setSigma

(

float

val_sigma

)

set the number of sigmas to vary this group by

Normally we are using just +/-1 sigma variations, but if the systematics are very small that can get lost in the statistical jitter. For those cases it is better to do a multi-sigma variation and then scale it back to +/-1 sigma thereby reducing the statistical jitter introduced into the systematic. A traditional scaling factor for these cases is to scale by five sigma.

Please note that if you do this, you normally only want to do this for small systematics for which statistical jitter is an issue. For large systematics there is a legitimate concern that a 5 sigma variation won't be 5 times the size of a 1 sigma variation, introducing a different kind of bias. To that end, if you use this, you should normally put the small systematics into a separate group from your regular systematics (which then also makes it easier for you to know which one to scale down).

Guarantee

no-fail

Precondition

val_sigma > 0

Definition at line 202 of file MakeSystematicsVector.cxx.

  {
    RCU_CHANGE_INVARIANT (this);
    RCU_REQUIRE (val_sigma > 0);
    m_config.back().sigma = val_sigma;
  }

setToys()

void CP::MakeSystematicsVector::setToys

(

unsigned

val_toys

)

set the number of toys to run for this group

This is a specialized mechanism pioneered for the e/gamma and muon scale factors. Instead of evaluating a large number of systematics separately, it allows to vary all of them at the same time repeatedly, so that instead of hundreds of systematic variations you only have to perform maybe 10 or 20 "toy" variations. You then just take the spread of the variations as the overall systematic uncertainty from the toys. There are reports that compared to the "old" method this can yield a factor five reduction in systematic uncertainty without increasing the number of systematic variations to evaluate.

Please note that this approach requires a more expert handling than "regular" systematics. The main point here is that the "toy" variations need a different post-processing than regular variations, i.e. you need to look at the spread between the output histograms for the different "toys" and use that to construct the combined systematic. The e/gamma group currently (Oct 15) provides a tool for that. For Bayesian marginalization you may alternatively consider to integrate the "toy" variations directly into the integral over nuisance parameter space to extract more information than you could with a single systematic.

A general concern for all methods that reduce the number of nuisance parameters (including the "toy" approach) is if you use profiling and are able to constrain the "toy" systematic, in which case we generally assume that such an approach is invalid. Similar precautions should be taken when used with Bayesian marginalization. This is typically of little practical concern as long as the systematic in question is small, but it is something that you should check for and that you should include in your supporting documentation.

The "toy" approach to systematics evaluation introduces an additional statistical jitter into your systematics, due to the sampling fluctuations of your "toy" variations. Naturally this uncertainty decreases if you use a larger number of "toy" variations. We currently (Oct 15) provide no recommendations for evaluating the size of that uncertainty or for determining whether your chosen number of "toys" is sufficient. Anecdotal evidence from the e/gamma group suggests that for evaluating their scale factor systematic as little as 10 or 20 "toys" may be sufficient. However, this will vary depending on your analysis and the systematics you use the "toys" for.

If you decide to go for a large number of "toy" variations it may be better to go for a "traditional" evaluation of your systematics instead, as that can cut down on the aforementioned statistical jitter by interpolating between the variations. The number of variations needed for the traditional approach to be better (or even feasible) will depend on the interpolation algorithm used and is likely to improve as we implement better interpolation algorithms.

Guarantee

no-fail

Definition at line 212 of file MakeSystematicsVector.cxx.

  {
    RCU_CHANGE_INVARIANT (this);
    RCU_REQUIRE (val_toys > 0);
    m_config.back().toys = val_toys;
  }

testInvariant()

void CP::MakeSystematicsVector::testInvariant

(

)

const

test the invariant of this object

Guarantee

no-fail

Definition at line 62 of file MakeSystematicsVector.cxx.

  {
    //RCU_INVARIANT (this != nullptr);
    RCU_INVARIANT (!m_config.empty());
  }

useForNominal()

void CP::MakeSystematicsVector::useForNominal

(

)

set this group as the default, i.e.

the group containing the nominal variation

Guarantee

no-fail

Definition at line 222 of file MakeSystematicsVector.cxx.

  {
    RCU_CHANGE_INVARIANT (this);
    m_useForNominal = m_config.back().label;
  }

Member Data Documentation

m_config

std::vector<GroupConfig> CP::MakeSystematicsVector::m_config

private

the configuration on a per-group basis

Definition at line 229 of file MakeSystematicsVector.h.

m_result

std::map<std::string,std::vector<SystematicSet> > CP::MakeSystematicsVector::m_result

private

the value of result

Definition at line 202 of file MakeSystematicsVector.h.

m_useForNominal

std::string CP::MakeSystematicsVector::m_useForNominal

private

the group for which useForNominal was set

Definition at line 234 of file MakeSystematicsVector.h.

---

The documentation for this class was generated from the following files:

• MakeSystematicsVector.h
• MakeSystematicsVector.cxx