H5Utils::Merger Class Reference

#include <Merger.h>

Collaboration diagram for H5Utils::Merger:

Public Member Functions
	Merger (hsize_t mergeAxis=0, int chunkSize=-1, bool requireSameFormat=true, std::size_t bufferSize=-1, bool bufferInRows=false)
	Create the merger.
	~Merger ()
void	merge (H5::Group &target, const H5::Group &source)
	Merge a source group into a target group.
void	merge (H5::DataSet &target, const H5::DataSet &source)
	Merge a source dataset into a target dataset.
H5::Group	createFrom (H5::H5Location &targetLocation, const H5::Group &source)
	Make a new group from information in a source group.
H5::DataSet	createFrom (H5::H5Location &targetLocation, const H5::DataSet &source)
	Make a new dataset from information in a source dataset.
void	flush (H5::Group &dst)
	Write all accumulated histogram data to the output.

Protected Attributes
hsize_t	m_mergeAxis
	The axis to merge along.
int	m_chunkSize
	The chunk size to apply.
bool	m_requireSameFormat
	Whether to require the same group structure.
std::size_t	m_bufferSize
	The size of the buffer.
bool	m_measureBufferInRows
	Whether to measure the buffer in bytes or rows.
std::unique_ptr< H5Utils::hist::HistogramMerger >	m_histMerger
	Accumulator for UHI histogram groups.

Detailed Description

Definition at line 24 of file Merger.h.

Constructor & Destructor Documentation

Merger()

H5Utils::Merger::Merger	(	hsize_t	mergeAxis = 0,
		int	chunkSize = -1,
		bool	requireSameFormat = true,
		std::size_t	bufferSize = -1,
		bool	bufferInRows = false )

Create the merger.

Parameters

mergeAxis	The axis to merge along
chunkSize	The chunk size to apply. If negative then the value found in the input datasets will be used.
requireSameFormat	Require all input files to have the same groups and datasets.
bufferSize	The maximum size of the buffer to use while merging datasets
bufferInRows	Whether the buffer size is specified in rows or bytes

Definition at line 14 of file Merger.cxx.

                         :
    m_mergeAxis(mergeAxis),
    m_chunkSize(chunkSize),
    m_requireSameFormat(requireSameFormat),
    m_bufferSize(bufferSize),
    m_measureBufferInRows(bufferInRows),
    m_histMerger(std::make_unique<H5Utils::hist::HistogramMerger>()) {}

~Merger()

H5Utils::Merger::~Merger

(

)

Definition at line 27 of file Merger.cxx.

{}

Member Function Documentation

createFrom() [1/2]

H5::DataSet H5Utils::Merger::createFrom	(	H5::H5Location &	targetLocation,
		const H5::DataSet &	source )

Make a new dataset from information in a source dataset.

Parameters

targetLocation	Where the new dataset will be created
source	The dataset to use to create the new dataset

Definition at line 131 of file Merger.cxx.

  {
    return createDataSet(targetLocation, source, m_mergeAxis, m_chunkSize);
  }

createFrom() [2/2]

H5::Group H5Utils::Merger::createFrom	(	H5::H5Location &	targetLocation,
		const H5::Group &	source )

Make a new group from information in a source group.

Parameters

targetLocation	Where the new group will be created
source	The group to use to create the new group

Definition at line 122 of file Merger.cxx.

  {
    H5::Group newGroup = targetLocation.createGroup(source.getObjName());
    merge(newGroup, source);
    return newGroup;
  }

flush()

void H5Utils::Merger::flush

(

H5::Group &

dst

)

Write all accumulated histogram data to the output.

Parameters

dst	The root group of the output file.

Definition at line 138 of file Merger.cxx.

  {
    m_histMerger->write(dst);
  }

merge() [1/2]

void H5Utils::Merger::merge	(	H5::DataSet &	target,
		const H5::DataSet &	source )

Merge a source dataset into a target dataset.

Parameters

target	The dataset to merge into
source	The dataset to merge from

Definition at line 107 of file Merger.cxx.

  {
    std::size_t bufferSize = m_bufferSize;
    if (m_measureBufferInRows) {
      // Need to calculate the actual buffer size
      std::size_t rowSize = getRowSize(source, m_mergeAxis);
      if (std::size_t(-1) / m_bufferSize < rowSize)
        std::overflow_error("Requested buffer would overflow the register!");
      bufferSize = rowSize * m_bufferSize;
    }
    mergeDatasets(target, source, m_mergeAxis, bufferSize);
  }

merge() [2/2]

void H5Utils::Merger::merge	(	H5::Group &	target,
		const H5::Group &	source )

Merge a source group into a target group.

Parameters

target	The group to merge into
source	The group to merge from

Definition at line 29 of file Merger.cxx.

  {
    // Check if this group was empty before we started
    bool isEmpty = target.getNumObjs() == 0;
 
    // Iterate through each child of the source group
    for (hsize_t ii = 0; ii < source.getNumObjs(); ++ii) {
      H5G_obj_t childType = source.getObjTypeByIdx(ii);
      std::string childName = source.getObjnameByIdx(ii);
      // Find the correct index in the target
      hsize_t targetIdx = 0;
      for (; targetIdx < target.getNumObjs(); ++targetIdx)
        if (target.getObjnameByIdx(targetIdx) == childName)
          break;
      bool found = targetIdx != target.getNumObjs();
      if (found) {
        // Make sure these are the same type!
        if (target.getObjTypeByIdx(targetIdx) != childType)
          throw std::invalid_argument(
              "Both target and source contain " + childName +
              " but they have different types!");
      }
      else if (m_requireSameFormat && !isEmpty) {
        throw std::invalid_argument(
            "Target and source have different formats!");
      }
      switch (childType) {
        case H5G_GROUP:
          {
            H5::Group sg = source.openGroup(childName);
            // UHI histogram groups are accumulated separately and written
            // once all source files have been processed.
            if (sg.attrExists("uhi_schema")) {
              m_histMerger->add(sg.getObjName(), sg);
              continue;
            }
            H5::Group tg = found ?
              target.openGroup(childName) :
              createFrom(target, sg);
            try {
              merge(tg, sg);
            }
            catch (...) {
              std::cerr << "Encountered an error merging child " << childName << std::endl;
              throw;
            }
          }
          break;
        case H5G_DATASET:
          {
            H5::DataSet sd = source.openDataSet(childName);
            if (sd.getSpace().getSimpleExtentNdims() == 0) {
              std::cerr << "WARNING: skipping scalar '"
                        << childName << "'" << std::endl;
              break;
            }
            H5::DataSet td = found ?
              target.openDataSet(childName) :
              createFrom(target, sd);
            try {
              merge(td, sd);
            }
            catch (...) {
              std::cerr << "Encountered an error merging child " << childName << std::endl;
              throw;
            }
          }
          break;
        default:
          break;
      }
    } //> end loop over children
    // TODO - this did no check to see if target contained something source
    // didn't, this is probably fine though.
  } //> end function merge(group)

Member Data Documentation

m_bufferSize

std::size_t H5Utils::Merger::m_bufferSize

protected

The size of the buffer.

Definition at line 93 of file Merger.h.

m_chunkSize

int H5Utils::Merger::m_chunkSize

protected

The chunk size to apply.

Definition at line 89 of file Merger.h.

m_histMerger

std::unique_ptr<H5Utils::hist::HistogramMerger> H5Utils::Merger::m_histMerger

protected

Accumulator for UHI histogram groups.

Definition at line 97 of file Merger.h.

m_measureBufferInRows

bool H5Utils::Merger::m_measureBufferInRows

protected

Whether to measure the buffer in bytes or rows.

Definition at line 95 of file Merger.h.

m_mergeAxis

hsize_t H5Utils::Merger::m_mergeAxis

protected

The axis to merge along.

Definition at line 87 of file Merger.h.

m_requireSameFormat

bool H5Utils::Merger::m_requireSameFormat

protected

Whether to require the same group structure.

Definition at line 91 of file Merger.h.

---

The documentation for this class was generated from the following files:

• Merger.h
• Merger.cxx