H5Utils::Merger Class Reference

#include <Merger.h>

Collaboration diagram for H5Utils::Merger:

Public Member Functions
	Merger (hsize_t mergeAxis=0, int chunkSize=-1, bool requireSameFormat=true, std::size_t bufferSize=-1, bool bufferInRows=false)
	Create the merger.
	~Merger ()
void	merge (H5::Group &target, const H5::Group &source)
	Merge a source group into a target group.
void	merge (H5::DataSet &target, const H5::DataSet &source)
	Merge a source dataset into a target dataset.
void	merge (auto &target, const auto &source)
	Merge a source file/group into a target file/group.
void	flush (H5::Group &dst)
	Write all accumulated histogram data to the output.

Protected Attributes
hsize_t	m_mergeAxis
	The axis to merge along.
int	m_chunkSize
	The chunk size to apply.
bool	m_requireSameFormat
	Whether to require the same group structure.
std::size_t	m_bufferSize
	The size of the buffer.
bool	m_measureBufferInRows
	Whether to measure the buffer in bytes or rows.
std::unique_ptr< H5Utils::hist::HistogramMerger >	m_histMerger
	Accumulator for UHI histogram groups.

Static Private Member Functions
static H5::Group	convert (const H5::Group &group)
	convert() is a no-op for a Group, used by the merge() template
static H5::Group	convert (const H5::H5File &file)
	convert() opens the root group of a File, used by the merge() template

Detailed Description

Definition at line 24 of file Merger.h.

Constructor & Destructor Documentation

Merger()

H5Utils::Merger::Merger	(	hsize_t	mergeAxis = 0,
		int	chunkSize = -1,
		bool	requireSameFormat = true,
		std::size_t	bufferSize = -1,
		bool	bufferInRows = false )

Create the merger.

Parameters

mergeAxis	The axis to merge along
chunkSize	The chunk size to apply. If negative then the value found in the input datasets will be used.
requireSameFormat	Require all input files to have the same groups and datasets.
bufferSize	The maximum size of the buffer to use while merging datasets
bufferInRows	Whether the buffer size is specified in rows or bytes

Definition at line 14 of file Merger.cxx.

                         :
    m_mergeAxis(mergeAxis),
    m_chunkSize(chunkSize),
    m_requireSameFormat(requireSameFormat),
    m_bufferSize(bufferSize),
    m_measureBufferInRows(bufferInRows),
    m_histMerger(std::make_unique<H5Utils::hist::HistogramMerger>()) {}

~Merger()

H5Utils::Merger::~Merger

(

)

Definition at line 27 of file Merger.cxx.

{}

Member Function Documentation

convert() [1/2]

H5::Group H5Utils::Merger::convert ( const H5::Group & group )

staticprivate

convert() is a no-op for a Group, used by the merge() template

Definition at line 122 of file Merger.cxx.

  {
    return group;
  }

convert() [2/2]

H5::Group H5Utils::Merger::convert ( const H5::H5File & file )

staticprivate

convert() opens the root group of a File, used by the merge() template

Definition at line 127 of file Merger.cxx.

  {
    return file.openGroup("/");
  }

flush()

void H5Utils::Merger::flush

(

H5::Group &

dst

)

Write all accumulated histogram data to the output.

Parameters

dst	The root group of the output file.

Definition at line 132 of file Merger.cxx.

  {
    m_histMerger->write(dst);
  }

merge() [1/3]

void H5Utils::Merger::merge ( auto & target, const auto & source )

inline

Merge a source file/group into a target file/group.

Parameters

target	The file or group to merge into
source	The file or group to merge from

Accepts any combination of H5::File and H5::Group. Each argument is converted to the H5::Group to actually operate on via convert() (a no-op for a Group, the root group for a File) before forwarding to the merge(Group&, const Group&) overload above.

Definition at line 71 of file Merger.h.

      {
        H5::Group targetGroup = convert(target);
        H5::Group sourceGroup = convert(source);
        merge(targetGroup, sourceGroup);
      }

merge() [2/3]

void H5Utils::Merger::merge	(	H5::DataSet &	target,
		const H5::DataSet &	source )

Merge a source dataset into a target dataset.

Parameters

target	The dataset to merge into
source	The dataset to merge from

Definition at line 107 of file Merger.cxx.

  {
    std::size_t bufferSize = m_bufferSize;
    if (m_measureBufferInRows) {
      // Need to calculate the actual buffer size
      std::size_t rowSize = getRowSize(source, m_mergeAxis);
      if (std::size_t(-1) / m_bufferSize < rowSize)
        std::overflow_error("Requested buffer would overflow the register!");
      bufferSize = rowSize * m_bufferSize;
    }
    mergeDatasets(target, source, m_mergeAxis, bufferSize);
  }

merge() [3/3]

void H5Utils::Merger::merge	(	H5::Group &	target,
		const H5::Group &	source )

Merge a source group into a target group.

Parameters

target	The group to merge into
source	The group to merge from

Definition at line 29 of file Merger.cxx.

  {
    // Check if this group was empty before we started
    bool isEmpty = target.getNumObjs() == 0;
 
    // Iterate through each child of the source group
    for (hsize_t ii = 0; ii < source.getNumObjs(); ++ii) {
      H5G_obj_t childType = source.getObjTypeByIdx(ii);
      std::string childName = source.getObjnameByIdx(ii);
      // Find the correct index in the target
      hsize_t targetIdx = 0;
      for (; targetIdx < target.getNumObjs(); ++targetIdx)
        if (target.getObjnameByIdx(targetIdx) == childName)
          break;
      bool found = targetIdx != target.getNumObjs();
      if (found) {
        // Make sure these are the same type!
        if (target.getObjTypeByIdx(targetIdx) != childType)
          throw std::invalid_argument(
              "Both target and source contain " + childName +
              " but they have different types!");
      }
      else if (m_requireSameFormat && !isEmpty) {
        throw std::invalid_argument(
            "Target and source have different formats!");
      }
      switch (childType) {
        case H5G_GROUP:
          {
            H5::Group sg = source.openGroup(childName);
            // UHI histogram groups are accumulated separately and written
            // once all source files have been processed.
            if (sg.attrExists("uhi_schema")) {
              m_histMerger->add(sg.getObjName(), sg);
              continue;
            }
            H5::Group tg = found ?
              target.openGroup(childName) :
              target.createGroup(sg.getObjName());
            try {
              merge(tg, sg);
            }
            catch (...) {
              std::cerr << "Encountered an error merging child " << childName << std::endl;
              throw;
            }
          }
          break;
        case H5G_DATASET:
          {
            H5::DataSet sd = source.openDataSet(childName);
            if (sd.getSpace().getSimpleExtentNdims() == 0) {
              std::cerr << "WARNING: skipping scalar '"
                        << childName << "'" << std::endl;
              break;
            }
            H5::DataSet td = found ?
              target.openDataSet(childName) :
              createDataSet(target, sd, m_mergeAxis, m_chunkSize);
            try {
              merge(td, sd);
            }
            catch (...) {
              std::cerr << "Encountered an error merging child " << childName << std::endl;
              throw;
            }
          }
          break;
        default:
          break;
      }
    } //> end loop over children
    // TODO - this did no check to see if target contained something source
    // didn't, this is probably fine though.
  } //> end function merge(group)

Member Data Documentation

m_bufferSize

std::size_t H5Utils::Merger::m_bufferSize

protected

The size of the buffer.

Definition at line 99 of file Merger.h.

m_chunkSize

int H5Utils::Merger::m_chunkSize

protected

The chunk size to apply.

Definition at line 95 of file Merger.h.

m_histMerger

std::unique_ptr<H5Utils::hist::HistogramMerger> H5Utils::Merger::m_histMerger

protected

Accumulator for UHI histogram groups.

Definition at line 103 of file Merger.h.

m_measureBufferInRows

bool H5Utils::Merger::m_measureBufferInRows

protected

Whether to measure the buffer in bytes or rows.

Definition at line 101 of file Merger.h.

m_mergeAxis

hsize_t H5Utils::Merger::m_mergeAxis

protected

The axis to merge along.

Definition at line 93 of file Merger.h.

m_requireSameFormat

bool H5Utils::Merger::m_requireSameFormat

protected

Whether to require the same group structure.

Definition at line 97 of file Merger.h.

---

The documentation for this class was generated from the following files:

• Merger.h
• Merger.cxx