SH::ScanDir Struct Reference

the class used for scanning local directories and file servers for samples More...

#include <ScanDir.h>

Collaboration diagram for SH::ScanDir:

Public Member Functions
	ScanDir ()
	standard constructor More...
const ScanDir &	scan (SampleHandler &sh, const std::string &dir) const
	scan the given directory and put the created samples into the sample handler More...
const ScanDir &	scanEOS (SampleHandler &sh, const std::string &eosDir) const
	scan the given directory in EOS and put the created samples into the sample handler More...
const ScanDir &	scan (SampleHandler &sh, DiskList &list) const
	scan the given directory and put the created samples into the sample handler More...
ScanDir &	sampleDepth (int val_sampleDepth)
	the index of the file hierarchy at which we gather the sample name. More...
ScanDir &	absSampleDepth (int val_absSampleDepth)
	the index of the file hierarchy at which we gather the sample name. More...
ScanDir &	sampleName (const std::string &val_sampleName)
	a single sample name into which all found files should be placed. More...
ScanDir &	minDepth (std::size_t val_minDepth)
	the minimum depth for files to make it into the sample More...
ScanDir &	maxDepth (std::size_t val_maxDepth)
	the maximum depth for files to make it into the sample More...
ScanDir &	filePattern (const std::string &val_filePattern)
	the pattern for files to be accepted More...
ScanDir &	fileRegex (const std::string &val_fileRegex)
	the regular expression for files to be accepted More...
ScanDir &	directoryPattern (const std::string &val_directoryPattern)
	the pattern for directories to be visited More...
ScanDir &	directoryRegex (const std::string &val_directoryRegex)
	the regular expression for directories to be visited More...
ScanDir &	samplePattern (const std::string &val_samplePattern)
	the pattern for samples to be accepted More...
ScanDir &	samplePostfix (const std::string &val_samplePostfix)
	the pattern for the postfix to be stripped from the sampleName More...
ScanDir &	sampleRename (const std::string &pattern, const std::string &name)
	rename any sample matching pattern to name More...
ScanDir &	extraNameComponent (int val_relSampleDepth)
	attach an extra name component to the sample based on a second component of the path More...

Private Types
typedef std::vector< std::pair< boost::regex, std::string > >::const_iterator	SampleRenameIter
	the list of entries from sampleRename More...

Private Member Functions
void	recurse (std::map< std::string, SamplePtr > &samples, DiskList &list, const std::vector< std::string > &hierarchy) const
	perform the recursive scanning of the directory tree More...
void	addSampleFile (std::map< std::string, SamplePtr > &samples, const std::vector< std::string > &hierarchy, const std::string &path) const
	add the given file to the sample based on the hierarchy, creating the sample if necessary More...
std::string	findPathComponent (const std::vector< std::string > &hierarchy, int absSampleDepth, int relSampleDepth) const
	find the path component at the given depth More...

Private Attributes
int	m_relSampleDepth
	if this is negative it is the depth at which we take the sample name, counting from the end More...
int	m_absSampleDepth
	if m_relSampleDepth is not negative, it is the depth at which we take the sample name, counting from the first directory scanned More...
std::string	m_sampleName
	the value set by sampleName More...
std::size_t	m_minDepth
	the value set by minDepth More...
std::size_t	m_maxDepth
	the value set by maxDepth More...
boost::regex	m_filePattern
	the value set by filePattern, converted to a regular expression More...
boost::regex	m_directoryPattern
	the value set by directoryPattern, converted to a regular expression More...
boost::regex	m_samplePattern
	the value set by samplePattern, converted to a regular expression More...
boost::regex	m_samplePostfix
	the value set by samplePostfix, converted to a regular expression More...
bool	m_samplePostfixEmpty
	whether samplePostfix has been set to the empty string More...
std::vector< std::pair< boost::regex, std::string > >	m_sampleRename
int	m_extraNameComponent
	the depth set with extraNameComponent, or 0 otherwise More...

Detailed Description

the class used for scanning local directories and file servers for samples

Originally these was a series of stand-alone function calls, but people kept asking for more and more options, making it unwieldy to call and to maintain. Instead we now have a single class containing all the possible parameters, which makes it easier to configure and extend.

The member functions all return *this, so that usage like this is possible:

ScanDir()
  .filePattern ("*.root*")
  .scan (sh, "/data");

Definition at line 37 of file ScanDir.h.

Member Typedef Documentation

SampleRenameIter

typedef std::vector<std::pair<boost::regex,std::string> >::const_iterator SH::ScanDir::SampleRenameIter

private

the list of entries from sampleRename

Definition at line 210 of file ScanDir.h.

Constructor & Destructor Documentation

ScanDir()

SH::ScanDir::ScanDir

(

)

standard constructor

Guarantee

strong

Failures

out of memory I

Definition at line 32 of file ScanDir.cxx.

    : m_relSampleDepth (-2), m_absSampleDepth (0),
      m_minDepth (0), m_maxDepth (-1),
      m_filePattern (RCU::glob_to_regexp ("*.root*")),
      m_directoryPattern (RCU::glob_to_regexp ("*")),
      m_samplePattern (RCU::glob_to_regexp ("*")),
      m_samplePostfix (RCU::glob_to_regexp ("")),
      m_samplePostfixEmpty (true),
      m_extraNameComponent (0)
  {}

Member Function Documentation

absSampleDepth()

ScanDir & SH::ScanDir::absSampleDepth

(

int

val_absSampleDepth

)

the index of the file hierarchy at which we gather the sample name.

this differs from sampleDepth in that negative numbers count up in the directory hierarchy from the top of where we scan, while sampleDepth starts counting from the back if the number is negative.

Definition at line 56 of file ScanDir.cxx.

  {
    m_relSampleDepth = 0;
    m_absSampleDepth = val_absSampleDepth;
    return *this;
  }

addSampleFile()

void SH::ScanDir::addSampleFile ( std::map< std::string, SamplePtr > &  samples, const std::vector< std::string > &  hierarchy, const std::string &  path  ) const

private

add the given file to the sample based on the hierarchy, creating the sample if necessary

Guarantee

basic

Failures

out of memory II

Definition at line 254 of file ScanDir.cxx.

  {
    std::string sampleName;
 
    if (!m_sampleName.empty())
    {
      sampleName = m_sampleName;
    } else
    {
      sampleName = findPathComponent (hierarchy, m_absSampleDepth,
                      m_relSampleDepth);
      if (sampleName.empty())
    return;
 
      if (!m_samplePostfixEmpty)
      {
    bool done = false;
    for (std::size_t iter = 0, end = sampleName.size();
         iter != end && !done; ++ iter)
    {
      if (RCU::match_expr (m_samplePostfix, sampleName.substr (iter)))
      {
        if (iter == 0)
          RCU_THROW_MSG ("sample name matches entire postfix pattern: \"" + sampleName + "\"");
        sampleName.resize (iter);
        done = true;
      }
    }
      }
 
      if (m_extraNameComponent != 0)
      {
    std::string component = findPathComponent
      (hierarchy, m_extraNameComponent, m_extraNameComponent);
    if (component.empty())
      return;
    sampleName += "_" + component;
      }
 
      if (!RCU::match_expr (m_samplePattern, sampleName))
    return;
 
      {
    bool done = false;
    for (SampleRenameIter iter = m_sampleRename.begin(),
           end = m_sampleRename.end(); !done && iter != end; ++ iter)
    {
      if (RCU::match_expr (iter->first, sampleName))
      {
        sampleName = iter->second;
        done = true;
      }
    }
      }
    }
 
    std::map<std::string,SamplePtr>::iterator iter
      = samples.find (sampleName);
    if (iter == samples.end())
    {
      SamplePtr sample (new SampleLocal (sampleName));
      samples[sampleName] = sample;
      iter = samples.find (sampleName);
    }
    SampleLocal *sample = dynamic_cast<SampleLocal*>(iter->second.get());
    RCU_ASSERT (sample != 0);
    sample->add (path);
  }

directoryPattern()

ScanDir & SH::ScanDir::directoryPattern

(

const std::string &

val_directoryPattern

)

the pattern for directories to be visited

See also

directoryPatternRegex

Definition at line 111 of file ScanDir.cxx.

  {
    m_directoryPattern = RCU::glob_to_regexp (val_directoryPattern);
    return *this;
  }

directoryRegex()

ScanDir & SH::ScanDir::directoryRegex

(

const std::string &

val_directoryRegex

)

the regular expression for directories to be visited

See also

directoryPattern

Definition at line 120 of file ScanDir.cxx.

  {
    m_directoryPattern = val_directoryRegex;
    return *this;
  }

extraNameComponent()

ScanDir & SH::ScanDir::extraNameComponent

(

int

val_relSampleDepth

)

attach an extra name component to the sample based on a second component of the path

Precondition

val_relSampleDepth != 0

Definition at line 157 of file ScanDir.cxx.

  {
    RCU_REQUIRE (val_relSampleDepth != 0);
    m_extraNameComponent = val_relSampleDepth;
    return *this;
  }

filePattern()

ScanDir & SH::ScanDir::filePattern

(

const std::string &

val_filePattern

)

the pattern for files to be accepted

See also

filePatternRegex

Definition at line 93 of file ScanDir.cxx.

  {
    m_filePattern = RCU::glob_to_regexp (val_filePattern);
    return *this;
  }

fileRegex()

ScanDir & SH::ScanDir::fileRegex

(

const std::string &

val_fileRegex

)

the regular expression for files to be accepted

See also

filePattern

Definition at line 102 of file ScanDir.cxx.

  {
    m_filePattern = val_fileRegex;
    return *this;
  }

findPathComponent()

std::string SH::ScanDir::findPathComponent ( const std::vector< std::string > &  hierarchy, int  absSampleDepth, int  relSampleDepth  ) const

private

find the path component at the given depth

Returns

the path componenent, or NULL if it doesn't exist

Guarantee

strong

Failures

out of memory II

Definition at line 328 of file ScanDir.cxx.

  {
    std::string sampleName;
 
    int myindex = absSampleDepth+1;
    if (relSampleDepth < 0)
      myindex = relSampleDepth + hierarchy.size();
    if (std::size_t (myindex) >= hierarchy.size())
      return sampleName;
    if (myindex > 0)
    {
      sampleName = hierarchy[myindex];
    } else
    {
      sampleName = hierarchy[0];
      while (sampleName.empty() ||
         sampleName[sampleName.size()-1] == '/' ||
         myindex < 0)
      {
    while (!sampleName.empty() && sampleName[sampleName.size()-1] == '/')
      sampleName.pop_back();
    if (sampleName.empty())
      return sampleName;
    if (myindex < 0)
    {
      std::string::size_type split = sampleName.rfind ('/');
      if (split == std::string::npos)
      {
        sampleName.clear ();
        return sampleName;
      }
      sampleName.resize (split);
      ++ myindex;
    }
    if (sampleName.empty())
      return sampleName;
      }
      std::string::size_type split = sampleName.rfind ('/');
      if (split != std::string::npos)
    sampleName = sampleName.substr (split + 1);
    }
    return sampleName;
  }

maxDepth()

ScanDir & SH::ScanDir::maxDepth

(

std::size_t

val_maxDepth

)

the maximum depth for files to make it into the sample

Definition at line 84 of file ScanDir.cxx.

  {
    m_maxDepth = val_maxDepth;
    return *this;
  }

minDepth()

ScanDir & SH::ScanDir::minDepth

(

std::size_t

val_minDepth

)

the minimum depth for files to make it into the sample

Definition at line 75 of file ScanDir.cxx.

  {
    m_minDepth = val_minDepth;
    return *this;
  }

recurse()

void SH::ScanDir::recurse ( std::map< std::string, SamplePtr > &  samples, DiskList &  list, const std::vector< std::string > &  hierarchy  ) const

private

perform the recursive scanning of the directory tree

Guarantee

basic

Failures

out of memory III
i/o errors

Definition at line 209 of file ScanDir.cxx.

  {
    using namespace msgScanDir;
 
    ANA_MSG_DEBUG ("scanning directory: " << list.dirname());
    while (list.next())
    {
      std::unique_ptr<DiskList> sublist (list.openDir());
 
      if (sublist.get() != 0)
      {
    if (!RCU::match_expr (m_directoryPattern, list.fileName()))
        {
          ANA_MSG_DEBUG ("directory does not match pattern, skipping directory " << list.path());
        } else if (hierarchy.size() > m_maxDepth)
        {
          ANA_MSG_DEBUG ("maxDepth exceeded, skipping directory " << list.path());
        } else
    {
          ANA_MSG_DEBUG ("descending into directory " << list.path());
      std::vector<std::string> subhierarchy = hierarchy;
      subhierarchy.push_back (list.fileName());
      recurse (samples, *sublist, subhierarchy);
    }
      } else
      {
    if (hierarchy.size() > m_minDepth &&
        RCU::match_expr (m_filePattern, list.fileName()))
    {
          ANA_MSG_DEBUG ("adding file " << list.path());
      std::vector<std::string> subhierarchy = hierarchy;
      subhierarchy.push_back (list.fileName());
      addSampleFile (samples, subhierarchy, list.path());
    } else
        {
          ANA_MSG_DEBUG ("skipping file " << list.path());
        }
      }
    }
  }

sampleDepth()

ScanDir & SH::ScanDir::sampleDepth

(

int

val_sampleDepth

)

the index of the file hierarchy at which we gather the sample name.

this is positive when it starts counting from the top, and negative when it starts from the back, i.e. -1 uses the file name, 0 denotes the directory inside the top level directory

Definition at line 46 of file ScanDir.cxx.

  {
    m_relSampleDepth = val_sampleDepth;
    m_absSampleDepth = val_sampleDepth;
    return *this;
  }

sampleName()

ScanDir & SH::ScanDir::sampleName

(

const std::string &

val_sampleName

)

a single sample name into which all found files should be placed.

if set, this overrides all other naming methods.

Definition at line 66 of file ScanDir.cxx.

  {
    m_sampleName = val_sampleName;
    return *this;
  }

samplePattern()

ScanDir & SH::ScanDir::samplePattern

(

const std::string &

val_samplePattern

)

the pattern for samples to be accepted

Definition at line 129 of file ScanDir.cxx.

  {
    m_samplePattern = RCU::glob_to_regexp (val_samplePattern);
    return *this;
  }

samplePostfix()

ScanDir & SH::ScanDir::samplePostfix

(

const std::string &

val_samplePostfix

)

the pattern for the postfix to be stripped from the sampleName

Definition at line 138 of file ScanDir.cxx.

  {
    m_samplePostfix = RCU::glob_to_regexp (val_samplePostfix);
    m_samplePostfixEmpty = val_samplePostfix.empty();
    return *this;
  }

sampleRename()

ScanDir & SH::ScanDir::sampleRename	(	const std::string &	pattern,
		const std::string &	name
	)

rename any sample matching pattern to name

Definition at line 148 of file ScanDir.cxx.

  {
    m_sampleRename.push_back (std::pair<boost::regex,std::string> (boost::regex (RCU::glob_to_regexp (pattern)), name));
    return *this;
  }

scan() [1/2]

const ScanDir & SH::ScanDir::scan	(	SampleHandler &	sh,
		const std::string &	dir
	)		const

scan the given directory and put the created samples into the sample handler

Returns

*this

Guarantee

basic

Failures

out of memory III
i/o errors
duplicate samples

Definition at line 167 of file ScanDir.cxx.

  {
    DiskListLocal list (dir);
    scan (sh, list);
    return *this;
  }

scan() [2/2]

const ScanDir & SH::ScanDir::scan	(	SampleHandler &	sh,
		DiskList &	list
	)		const

scan the given directory and put the created samples into the sample handler

Returns

*this

Guarantee

basic

Failures

out of memory III
i/o errors
duplicate samples

Definition at line 187 of file ScanDir.cxx.

  {
    std::vector<std::string> hierarchy;
    hierarchy.push_back (list.dirname());
 
    std::map<std::string,SamplePtr> samples;
    typedef std::map<std::string,SamplePtr>::iterator samplesIter;
    recurse (samples, list, hierarchy);
    for (samplesIter sample = samples.begin(), end = samples.end();
     sample != end; ++ sample)
    {
      if (sample->second.get() != 0)
      {
    sh.add (sample->second);
      }
    }
    return *this;
  }

scanEOS()

const ScanDir & SH::ScanDir::scanEOS	(	SampleHandler &	sh,
		const std::string &	eosDir
	)		const

scan the given directory in EOS and put the created samples into the sample handler

Returns

*this

Guarantee

basic

Failures

out of memory III
i/o errors
duplicate samples

Definition at line 177 of file ScanDir.cxx.

  {
    DiskListEOS list (eosDir);
    scan (sh, list);
    return *this;
  }

Member Data Documentation

m_absSampleDepth

int SH::ScanDir::m_absSampleDepth

private

if m_relSampleDepth is not negative, it is the depth at which we take the sample name, counting from the first directory scanned

Definition at line 169 of file ScanDir.h.

m_directoryPattern

boost::regex SH::ScanDir::m_directoryPattern

private

the value set by directoryPattern, converted to a regular expression

Definition at line 191 of file ScanDir.h.

m_extraNameComponent

int SH::ScanDir::m_extraNameComponent

private

the depth set with extraNameComponent, or 0 otherwise

Definition at line 215 of file ScanDir.h.

m_filePattern

boost::regex SH::ScanDir::m_filePattern

private

the value set by filePattern, converted to a regular expression

Definition at line 186 of file ScanDir.h.

m_maxDepth

std::size_t SH::ScanDir::m_maxDepth

private

the value set by maxDepth

Definition at line 181 of file ScanDir.h.

m_minDepth

std::size_t SH::ScanDir::m_minDepth

private

the value set by minDepth

Definition at line 177 of file ScanDir.h.

m_relSampleDepth

int SH::ScanDir::m_relSampleDepth

private

if this is negative it is the depth at which we take the sample name, counting from the end

Definition at line 163 of file ScanDir.h.

m_sampleName

std::string SH::ScanDir::m_sampleName

private

the value set by sampleName

Definition at line 173 of file ScanDir.h.

m_samplePattern

boost::regex SH::ScanDir::m_samplePattern

private

the value set by samplePattern, converted to a regular expression

Definition at line 196 of file ScanDir.h.

m_samplePostfix

boost::regex SH::ScanDir::m_samplePostfix

private

the value set by samplePostfix, converted to a regular expression

Definition at line 201 of file ScanDir.h.

m_samplePostfixEmpty

bool SH::ScanDir::m_samplePostfixEmpty

private

whether samplePostfix has been set to the empty string

Definition at line 206 of file ScanDir.h.

m_sampleRename

std::vector<std::pair<boost::regex,std::string> > SH::ScanDir::m_sampleRename

private

Definition at line 211 of file ScanDir.h.

---

The documentation for this struct was generated from the following files:

• ScanDir.h
• ScanDir.cxx