SH::ScanDir Struct Reference

the class used for scanning local directories and file servers for samples More...

#include <ScanDir.h>

Collaboration diagram for SH::ScanDir:

Public Member Functions
	ScanDir ()
	standard constructor
const ScanDir &	scan (SampleHandler &sh, const std::string &dir) const
	scan the given directory and put the created samples into the sample handler
const ScanDir &	scanEOS (SampleHandler &sh, const std::string &eosDir) const
	scan the given directory in EOS and put the created samples into the sample handler
const ScanDir &	scan (SampleHandler &sh, DiskList &list) const
	scan the given directory and put the created samples into the sample handler
ScanDir &	sampleDepth (int val_sampleDepth)
	the index of the file hierarchy at which we gather the sample name.
ScanDir &	absSampleDepth (int val_absSampleDepth)
	the index of the file hierarchy at which we gather the sample name.
ScanDir &	sampleName (const std::string &val_sampleName)
	a single sample name into which all found files should be placed.
ScanDir &	minDepth (std::size_t val_minDepth)
	the minimum depth for files to make it into the sample
ScanDir &	maxDepth (std::size_t val_maxDepth)
	the maximum depth for files to make it into the sample
ScanDir &	filePattern (const std::string &val_filePattern)
	the pattern for files to be accepted
ScanDir &	fileRegex (const std::string &val_fileRegex)
	the regular expression for files to be accepted
ScanDir &	directoryPattern (const std::string &val_directoryPattern)
	the pattern for directories to be visited
ScanDir &	directoryRegex (const std::string &val_directoryRegex)
	the regular expression for directories to be visited
ScanDir &	samplePattern (const std::string &val_samplePattern)
	the pattern for samples to be accepted
ScanDir &	samplePostfix (const std::string &val_samplePostfix)
	the pattern for the postfix to be stripped from the sampleName
ScanDir &	sampleRename (const std::string &pattern, const std::string &name)
	rename any sample matching pattern to name
ScanDir &	extraNameComponent (int val_relSampleDepth)
	attach an extra name component to the sample based on a second component of the path

Private Types
using	SampleRenameIter = std::vector<std::pair<std::regex,std::string> >::const_iterator
	the list of entries from sampleRename

Private Member Functions
void	recurse (std::map< std::string, std::shared_ptr< Sample > > &samples, DiskList &list, const std::vector< std::string > &hierarchy) const
	perform the recursive scanning of the directory tree
void	addSampleFile (std::map< std::string, std::shared_ptr< Sample > > &samples, const std::vector< std::string > &hierarchy, const std::string &path) const
	add the given file to the sample based on the hierarchy, creating the sample if necessary
std::string	findPathComponent (const std::vector< std::string > &hierarchy, int absSampleDepth, int relSampleDepth) const
	find the path component at the given depth

Private Attributes
int	m_relSampleDepth
	if this is negative it is the depth at which we take the sample name, counting from the end
int	m_absSampleDepth
	if m_relSampleDepth is not negative, it is the depth at which we take the sample name, counting from the first directory scanned
std::string	m_sampleName
	the value set by sampleName
std::size_t	m_minDepth
	the value set by minDepth
std::size_t	m_maxDepth
	the value set by maxDepth
std::regex	m_filePattern
	the value set by filePattern, converted to a regular expression
std::regex	m_directoryPattern
	the value set by directoryPattern, converted to a regular expression
std::regex	m_samplePattern
	the value set by samplePattern, converted to a regular expression
std::regex	m_samplePostfix
	the value set by samplePostfix, converted to a regular expression
bool	m_samplePostfixEmpty
	whether samplePostfix has been set to the empty string
std::vector< std::pair< std::regex, std::string > >	m_sampleRename
int	m_extraNameComponent
	the depth set with extraNameComponent, or 0 otherwise

Detailed Description

the class used for scanning local directories and file servers for samples

Originally these was a series of stand-alone function calls, but people kept asking for more and more options, making it unwieldy to call and to maintain. Instead we now have a single class containing all the possible parameters, which makes it easier to configure and extend.

The member functions all return *this, so that usage like this is possible:

ScanDir()
  .filePattern ("*.root*")
  .scan (sh, "/data");

Definition at line 38 of file ScanDir.h.

Member Typedef Documentation

SampleRenameIter

using SH::ScanDir::SampleRenameIter = std::vector<std::pair<std::regex,std::string> >::const_iterator

private

the list of entries from sampleRename

Definition at line 211 of file ScanDir.h.

Constructor & Destructor Documentation

ScanDir()

SH::ScanDir::ScanDir

(

)

standard constructor

Guarantee

strong

Failures

out of memory I

Definition at line 32 of file ScanDir.cxx.

    : m_relSampleDepth (-2), m_absSampleDepth (0),
      m_minDepth (0), m_maxDepth (-1),
      m_filePattern (RCU::glob_to_regexp ("*.root*")),
      m_directoryPattern (RCU::glob_to_regexp ("*")),
      m_samplePattern (RCU::glob_to_regexp ("*")),
      m_samplePostfix (RCU::glob_to_regexp ("")),
      m_samplePostfixEmpty (true),
      m_extraNameComponent (0)
  {}

Member Function Documentation

absSampleDepth()

ScanDir & SH::ScanDir::absSampleDepth

(

int

val_absSampleDepth

)

the index of the file hierarchy at which we gather the sample name.

this differs from sampleDepth in that negative numbers count up in the directory hierarchy from the top of where we scan, while sampleDepth starts counting from the back if the number is negative.

Definition at line 56 of file ScanDir.cxx.

  {
    m_relSampleDepth = 0;
    m_absSampleDepth = val_absSampleDepth;
    return *this;
  }

addSampleFile()

void SH::ScanDir::addSampleFile ( std::map< std::string, std::shared_ptr< Sample > > & samples, const std::vector< std::string > & hierarchy, const std::string & path ) const

private

add the given file to the sample based on the hierarchy, creating the sample if necessary

Guarantee

basic

Failures

out of memory II

Definition at line 259 of file ScanDir.cxx.

  {
    std::string sampleName;
 
    if (!m_sampleName.empty())
    {
      sampleName = m_sampleName;
    } else
    {
      sampleName = findPathComponent (hierarchy, m_absSampleDepth,
                      m_relSampleDepth);
      if (sampleName.empty())
    return;
 
      if (!m_samplePostfixEmpty)
      {
    bool done = false;
    for (std::size_t iter = 0, end = sampleName.size();
         iter != end && !done; ++ iter)
    {
      if (RCU::match_expr (m_samplePostfix, sampleName.substr (iter)))
      {
        if (iter == 0)
          throw std::runtime_error ("sample name matches entire postfix pattern: \"" + sampleName + "\"");
        sampleName.resize (iter);
        done = true;
      }
    }
      }
 
      if (m_extraNameComponent != 0)
      {
    std::string component = findPathComponent
      (hierarchy, m_extraNameComponent, m_extraNameComponent);
    if (component.empty())
      return;
    sampleName += "_" + component;
      }
 
      if (!RCU::match_expr (m_samplePattern, sampleName))
    return;
 
      {
    bool done = false;
    for (SampleRenameIter iter = m_sampleRename.begin(),
           end = m_sampleRename.end(); !done && iter != end; ++ iter)
    {
      if (RCU::match_expr (iter->first, sampleName))
      {
        sampleName = iter->second;
        done = true;
      }
    }
      }
    }
 
    auto iter = samples.find (sampleName);
    if (iter == samples.end())
    {
      auto sample = std::make_shared<SampleLocal> (sampleName);
      samples[sampleName] = sample;
      iter = samples.find (sampleName);
    }
    SampleLocal *sample = dynamic_cast<SampleLocal*>(iter->second.get());
    RCU_ASSERT (sample != 0);
    sample->add (path);
  }

directoryPattern()

ScanDir & SH::ScanDir::directoryPattern

(

const std::string &

val_directoryPattern

)

the pattern for directories to be visited

See also

directoryPatternRegex

Definition at line 111 of file ScanDir.cxx.

  {
    m_directoryPattern = RCU::glob_to_regexp (val_directoryPattern);
    return *this;
  }

directoryRegex()

ScanDir & SH::ScanDir::directoryRegex

(

const std::string &

val_directoryRegex

)

the regular expression for directories to be visited

See also

directoryPattern

Definition at line 120 of file ScanDir.cxx.

  {
    m_directoryPattern = val_directoryRegex;
    return *this;
  }

extraNameComponent()

ScanDir & SH::ScanDir::extraNameComponent

(

int

val_relSampleDepth

)

attach an extra name component to the sample based on a second component of the path

Precondition

val_relSampleDepth != 0

Definition at line 157 of file ScanDir.cxx.

  {
    RCU_REQUIRE (val_relSampleDepth != 0);
    m_extraNameComponent = val_relSampleDepth;
    return *this;
  }

filePattern()

ScanDir & SH::ScanDir::filePattern

(

const std::string &

val_filePattern

)

the pattern for files to be accepted

See also

filePatternRegex

Definition at line 93 of file ScanDir.cxx.

  {
    m_filePattern = RCU::glob_to_regexp (val_filePattern);
    return *this;
  }

fileRegex()

ScanDir & SH::ScanDir::fileRegex

(

const std::string &

val_fileRegex

)

the regular expression for files to be accepted

See also

filePattern

Definition at line 102 of file ScanDir.cxx.

  {
    m_filePattern = val_fileRegex;
    return *this;
  }

findPathComponent()

std::string SH::ScanDir::findPathComponent ( const std::vector< std::string > & hierarchy, int absSampleDepth, int relSampleDepth ) const

private

find the path component at the given depth

Returns

the path componenent, or NULL if it doesn't exist

Guarantee

strong

Failures

out of memory II

Definition at line 332 of file ScanDir.cxx.

  {
    std::string sampleName;
 
    int myindex = absSampleDepth+1;
    if (relSampleDepth < 0)
      myindex = relSampleDepth + hierarchy.size();
    // rationale: only apply the upper-bound check for a non-negative
    //   index; a negative index (e.g. absSampleDepth <= -2) is meant to
    //   climb up the hierarchy in the else branch below, and converting
    //   it to std::size_t here would wrap to SIZE_MAX and drop every
    //   file.
    if (myindex >= 0 && std::size_t (myindex) >= hierarchy.size())
      return sampleName;
    if (myindex > 0)
    {
      sampleName = hierarchy[myindex];
    } else
    {
      sampleName = hierarchy[0];
      while (sampleName.empty() ||
         sampleName[sampleName.size()-1] == '/' ||
         myindex < 0)
      {
    while (!sampleName.empty() && sampleName[sampleName.size()-1] == '/')
      sampleName.pop_back();
    if (sampleName.empty())
      return sampleName;
    if (myindex < 0)
    {
      std::string::size_type split = sampleName.rfind ('/');
      if (split == std::string::npos)
      {
        sampleName.clear ();
        return sampleName;
      }
      sampleName.resize (split);
      ++ myindex;
    }
    if (sampleName.empty())
      return sampleName;
      }
      std::string::size_type split = sampleName.rfind ('/');
      if (split != std::string::npos)
    sampleName = sampleName.substr (split + 1);
    }
    return sampleName;
  }

maxDepth()

ScanDir & SH::ScanDir::maxDepth

(

std::size_t

val_maxDepth

)

the maximum depth for files to make it into the sample

Definition at line 84 of file ScanDir.cxx.

  {
    m_maxDepth = val_maxDepth;
    return *this;
  }

minDepth()

ScanDir & SH::ScanDir::minDepth

(

std::size_t

val_minDepth

)

the minimum depth for files to make it into the sample

Definition at line 75 of file ScanDir.cxx.

  {
    m_minDepth = val_minDepth;
    return *this;
  }

recurse()

void SH::ScanDir::recurse ( std::map< std::string, std::shared_ptr< Sample > > & samples, DiskList & list, const std::vector< std::string > & hierarchy ) const

private

perform the recursive scanning of the directory tree

Guarantee

basic

Failures

out of memory III
i/o errors

Definition at line 207 of file ScanDir.cxx.

  {
    using namespace msgScanDir;
 
    ANA_MSG_DEBUG ("scanning directory: " << list.dirname());
    while (list.next())
    {
      std::unique_ptr<DiskList> sublist (list.openDirUnique());
 
      if (sublist.get() != 0)
      {
    if (!RCU::match_expr (m_directoryPattern, list.fileName()))
        {
          ANA_MSG_DEBUG ("directory does not match pattern, skipping directory " << list.path());
        } else if (hierarchy.size() > m_maxDepth)
        {
          ANA_MSG_DEBUG ("maxDepth exceeded, skipping directory " << list.path());
        } else if (hierarchy.size() >= 100)
        {
          // rationale: a hard sanity cap so that a directory symlink
          //   loop (or a link to an ancestor) does not recurse forever
          //   when the user has not set an explicit maxDepth.
          ANA_MSG_WARNING ("directory nesting exceeds 100 levels, not descending further (possible symlink loop): " << list.path());
        } else
    {
          ANA_MSG_DEBUG ("descending into directory " << list.path());
      std::vector<std::string> subhierarchy = hierarchy;
      subhierarchy.push_back (list.fileName());
      recurse (samples, *sublist, subhierarchy);
    }
      } else
      {
    if (hierarchy.size() > m_minDepth &&
        RCU::match_expr (m_filePattern, list.fileName()))
    {
          ANA_MSG_DEBUG ("adding file " << list.path());
      std::vector<std::string> subhierarchy = hierarchy;
      subhierarchy.push_back (list.fileName());
      addSampleFile (samples, subhierarchy, list.path());
    } else
        {
          ANA_MSG_DEBUG ("skipping file " << list.path());
        }
      }
    }
  }

sampleDepth()

ScanDir & SH::ScanDir::sampleDepth

(

int

val_sampleDepth

)

the index of the file hierarchy at which we gather the sample name.

this is positive when it starts counting from the top, and negative when it starts from the back, i.e. -1 uses the file name, 0 denotes the directory inside the top level directory

Definition at line 46 of file ScanDir.cxx.

  {
    m_relSampleDepth = val_sampleDepth;
    m_absSampleDepth = val_sampleDepth;
    return *this;
  }

sampleName()

ScanDir & SH::ScanDir::sampleName

(

const std::string &

val_sampleName

)

a single sample name into which all found files should be placed.

if set, this overrides all other naming methods.

Definition at line 66 of file ScanDir.cxx.

  {
    m_sampleName = val_sampleName;
    return *this;
  }

samplePattern()

ScanDir & SH::ScanDir::samplePattern

(

const std::string &

val_samplePattern

)

the pattern for samples to be accepted

Definition at line 129 of file ScanDir.cxx.

  {
    m_samplePattern = RCU::glob_to_regexp (val_samplePattern);
    return *this;
  }

samplePostfix()

ScanDir & SH::ScanDir::samplePostfix

(

const std::string &

val_samplePostfix

)

the pattern for the postfix to be stripped from the sampleName

Definition at line 138 of file ScanDir.cxx.

  {
    m_samplePostfix = RCU::glob_to_regexp (val_samplePostfix);
    m_samplePostfixEmpty = val_samplePostfix.empty();
    return *this;
  }

sampleRename()

ScanDir & SH::ScanDir::sampleRename	(	const std::string &	pattern,
		const std::string &	name )

rename any sample matching pattern to name

Definition at line 148 of file ScanDir.cxx.

  {
    m_sampleRename.push_back (std::pair<std::regex,std::string> (std::regex (RCU::glob_to_regexp (pattern)), name));
    return *this;
  }

scan() [1/2]

const ScanDir & SH::ScanDir::scan	(	SampleHandler &	sh,
		const std::string &	dir ) const

scan the given directory and put the created samples into the sample handler

Returns

*this

Guarantee

basic

Failures

out of memory III
i/o errors
duplicate samples

Definition at line 167 of file ScanDir.cxx.

  {
    DiskListLocal list (dir);
    scan (sh, list);
    return *this;
  }

scan() [2/2]

const ScanDir & SH::ScanDir::scan	(	SampleHandler &	sh,
		DiskList &	list ) const

scan the given directory and put the created samples into the sample handler

Returns

*this

Guarantee

basic

Failures

out of memory III
i/o errors
duplicate samples

Definition at line 187 of file ScanDir.cxx.

  {
    std::vector<std::string> hierarchy;
    hierarchy.push_back (list.dirname());
 
    std::map<std::string,std::shared_ptr<Sample>> samples;
    recurse (samples, list, hierarchy);
    for (const auto& sample : samples)
    {
      if (sample.second != nullptr)
      {
    sh.add (sample.second);
      }
    }
    return *this;
  }

scanEOS()

const ScanDir & SH::ScanDir::scanEOS	(	SampleHandler &	sh,
		const std::string &	eosDir ) const

scan the given directory in EOS and put the created samples into the sample handler

Returns

*this

Guarantee

basic

Failures

out of memory III
i/o errors
duplicate samples

Definition at line 177 of file ScanDir.cxx.

  {
    DiskListEOS list (eosDir);
    scan (sh, list);
    return *this;
  }

Member Data Documentation

m_absSampleDepth

int SH::ScanDir::m_absSampleDepth

private

if m_relSampleDepth is not negative, it is the depth at which we take the sample name, counting from the first directory scanned

Definition at line 170 of file ScanDir.h.

m_directoryPattern

std::regex SH::ScanDir::m_directoryPattern

private

the value set by directoryPattern, converted to a regular expression

Definition at line 192 of file ScanDir.h.

m_extraNameComponent

int SH::ScanDir::m_extraNameComponent

private

the depth set with extraNameComponent, or 0 otherwise

Definition at line 216 of file ScanDir.h.

m_filePattern

std::regex SH::ScanDir::m_filePattern

private

the value set by filePattern, converted to a regular expression

Definition at line 187 of file ScanDir.h.

m_maxDepth

std::size_t SH::ScanDir::m_maxDepth

private

the value set by maxDepth

Definition at line 182 of file ScanDir.h.

m_minDepth

std::size_t SH::ScanDir::m_minDepth

private

the value set by minDepth

Definition at line 178 of file ScanDir.h.

m_relSampleDepth

int SH::ScanDir::m_relSampleDepth

private

if this is negative it is the depth at which we take the sample name, counting from the end

Definition at line 164 of file ScanDir.h.

m_sampleName

std::string SH::ScanDir::m_sampleName

private

the value set by sampleName

Definition at line 174 of file ScanDir.h.

m_samplePattern

std::regex SH::ScanDir::m_samplePattern

private

the value set by samplePattern, converted to a regular expression

Definition at line 197 of file ScanDir.h.

m_samplePostfix

std::regex SH::ScanDir::m_samplePostfix

private

the value set by samplePostfix, converted to a regular expression

Definition at line 202 of file ScanDir.h.

m_samplePostfixEmpty

bool SH::ScanDir::m_samplePostfixEmpty

private

whether samplePostfix has been set to the empty string

Definition at line 207 of file ScanDir.h.

m_sampleRename

std::vector<std::pair<std::regex,std::string> > SH::ScanDir::m_sampleRename

private

Definition at line 212 of file ScanDir.h.

---

The documentation for this struct was generated from the following files:

• ScanDir.h
• ScanDir.cxx