BasicGPUToAthenaImporter Class Reference

Standard tool to convert the GPU data representation back to CPU. More...

#include <BasicGPUToAthenaImporter.h>

Inheritance diagram for BasicGPUToAthenaImporter:

Collaboration diagram for BasicGPUToAthenaImporter:

Public Member Functions
	BasicGPUToAthenaImporter (const std::string &type, const std::string &name, const IInterface *parent)
virtual StatusCode	initialize () override
virtual StatusCode	convert (const EventContext &ctx, const CaloRecGPU::ConstantDataHolder &constant_data, CaloRecGPU::EventDataHolder &event_data, xAOD::CaloClusterContainer *cluster_collection) const override
virtual StatusCode	finalize () override
virtual	~BasicGPUToAthenaImporter ()=default

Protected Member Functions
void	record_times (const size_t event_num, const std::vector< size_t > &times) const
template<class ... Args>
void	record_times (const size_t event_num, const size_t &value) const
template<class ... Args>
void	record_times (const size_t event_num, const size_t &value, Args &&... args) const
void	print_times (const std::string &header, const size_t time_size) const

Protected Attributes
std::shared_mutex	m_timeMutex
	Mutex that is locked when recording times. More...
std::vector< size_t > m_times	ATLAS_THREAD_SAFE
	Vector to hold execution times to be recorded if necessary. More...
std::vector< size_t > m_eventNumbers	ATLAS_THREAD_SAFE
	Vector to hold the event numbers to be recorded if necessary. More...
Gaudi::Property< bool >	m_measureTimes
	If true, times are recorded to the file given by m_timeFileName. More...
Gaudi::Property< std::string >	m_timeFileName
	File to which times should be saved. More...

Private Member Functions
void	record_times_helper (const size_t) const
template<class Arg >
void	record_times_helper (const size_t index, Arg &&arg) const
template<class ... Args>
void	record_times_helper (size_t index, Args &&... args) const

Private Attributes
Gaudi::Property< bool >	m_keepGPUData {this, "KeepGPUData", true, "Keep GPU allocated data"}
	If true, do not delete the GPU data representation. More...
Gaudi::Property< bool >	m_useCPUPropertiesCalculation {this, "UseCPUClusterPropertiesCalculation", false, "Use CaloClusterKineHelper::calculateKine instead of GPU-calculated cluster properties"}
	if set to true, cluster properties are (re-)calculated using CaloClusterKineHelper::calculateKine. More...
SG::ReadHandleKey< CaloCellContainer >	m_cellsKey {this, "CellsName", "", "Name(s) of Cell Containers"}
	vector of names of the cell containers to use as input. More...
Gaudi::Property< std::string >	m_clusterSizeString {this, "ClusterSize", "Topo_420", "The size/type of the clusters"}
	Cluster size. Should be set accordingly to the threshold. More...
xAOD::CaloCluster::ClusterSize	m_clusterSize
Gaudi::Property< std::vector< int > >	m_missingCellsToFill {this, "MissingCellsToFill", {}, "Force fill these cells as disabled on empty containers."}
	Cell indices to fill as disabled cells (useful if the cell vector is always missing the same cells). More...
const CaloCell_ID *	m_calo_id {nullptr}
	Pointer to Calo ID Helper. More...
Gaudi::Property< bool >	m_saveUncalibrated {this, "SaveUncalibratedSignalState", true, "Use CaloClusterKineHelper::calculateKine instead of GPU-calculated cluster properties"}
	if set to true, the uncalibrated state is saved when importing the clusters. More...

Detailed Description

Standard tool to convert the GPU data representation back to CPU.

Author

Nuno Fernandes nuno..nosp@m.dos..nosp@m.santo.nosp@m.s.fe.nosp@m.rnand.nosp@m.es@c.nosp@m.ern.c.nosp@m.h

Date

30 May 2022

Definition at line 29 of file BasicGPUToAthenaImporter.h.

Constructor & Destructor Documentation

BasicGPUToAthenaImporter()

BasicGPUToAthenaImporter::BasicGPUToAthenaImporter	(	const std::string &	type,
		const std::string &	name,
		const IInterface *	parent
	)

Definition at line 25 of file BasicGPUToAthenaImporter.cxx.

                                                                                                                           :
  base_class(type, name, parent),
  CaloGPUTimed(this)
{
}

~BasicGPUToAthenaImporter()

virtual BasicGPUToAthenaImporter::~BasicGPUToAthenaImporter ( )

virtualdefault

Member Function Documentation

convert()

StatusCode BasicGPUToAthenaImporter::convert ( const EventContext &  ctx, const CaloRecGPU::ConstantDataHolder &  constant_data, CaloRecGPU::EventDataHolder &  event_data, xAOD::CaloClusterContainer *  cluster_collection  ) const

overridevirtual

Definition at line 90 of file BasicGPUToAthenaImporter.cxx.

{
 
 
  using clock_type = boost::chrono::thread_clock;
  auto time_cast = [](const auto & before, const auto & after)
  {
    return boost::chrono::duration_cast<boost::chrono::microseconds>(after - before).count();
  };
 
  const auto start = clock_type::now();
 
  SG::ReadHandle<CaloCellContainer> cell_collection(m_cellsKey, ctx);
  if ( !cell_collection.isValid() )
    {
      ATH_MSG_ERROR( " Cannot retrieve CaloCellContainer: " << cell_collection.name()  );
      return StatusCode::FAILURE;
    }
  const DataLink<CaloCellContainer> cell_collection_link (cell_collection.name(), ctx);
 
  ed.returnToCPU(!m_keepGPUData, true, true, false);
 
  const auto after_send = clock_type::now();
 
  std::vector<std::unique_ptr<CaloClusterCellLink>> cell_links;
 
  cell_links.reserve(ed.m_clusters->number);
 
  for (int i = 0; i < ed.m_clusters->number; ++i)
    {
      if (ed.m_clusters->seedCellID[i] >= 0)
        {
          cell_links.emplace_back(std::make_unique<CaloClusterCellLink>(cell_collection_link));
          cell_links.back()->reserve(256);
          //To be adjusted.
        }
      else
        {
          cell_links.emplace_back(nullptr);
          //The excluded clusters don't have any cells.
        }
    }
 
  const auto after_creation = clock_type::now();
  
  
  //cell_index is the actual cell index in the full set of cells (identifier hash)
  //cell_count is the cell position in the cell collection (what we want for the weight)
  const auto process_cell = [&](const int cell_index, const int cell_count)
  {
    const ClusterTag this_tag = ed.m_cell_state->clusterTag[cell_index];
    if (this_tag.is_part_of_cluster())
      {
        const int this_index = this_tag.cluster_index();
        const int32_t weight_pattern = this_tag.secondary_cluster_weight();
 
        float tempf = 1.0f;
 
        std::memcpy(&tempf, &weight_pattern, sizeof(float));
        //C++20 would give us bit cast to do this more properly.
        //Still, given how the bit pattern is created,
        //it should be safe.
 
        const float reverse_weight = tempf;
 
        const float this_weight = 1.0f - reverse_weight;
 
        if (cell_links[this_index])
          {
            cell_links[this_index]->addCell(cell_count, this_weight);
 
            if (cell_index == ed.m_clusters->seedCellID[this_index] && cell_links[this_index]->size() > 1)
              //Seed cells aren't shared,
              //so no need to check this on the other case.
              {
                CaloClusterCellLink::iterator begin_it = cell_links[this_index]->begin();
                CaloClusterCellLink::iterator back_it  = std::prev(cell_links[this_index]->end());
 
                const unsigned int first_idx = begin_it.index();
                const double first_wgt = begin_it.weight();
 
                begin_it.reindex(back_it.index());
                begin_it.reweight(back_it.weight());
 
                back_it.reindex(first_idx);
                back_it.reweight(first_wgt);
 
                //Of course, this is to ensure the first cell is the seed cell,
                //in accordance to the way some cluster properties
                //(mostly phi-related) are calculated.
              }
          }
 
        if (this_tag.is_shared_between_clusters())
          {
            const int other_index = this_tag.secondary_cluster_index();
            if (cell_links[other_index])
              {
                cell_links[other_index]->addCell(cell_count, reverse_weight);
              }
          }
      }
  };
 
  if (cell_collection->isOrderedAndComplete())
    //Fast path: cell indices within the collection and identifierHashes match!
    {
      for (int cell_index = 0; cell_index < NCaloCells; ++cell_index)
        {
          process_cell(cell_index, cell_index);
        }
    }
  else if (cell_collection->isOrdered() && m_missingCellsToFill.size() > 0)
    {
      size_t missing_cell_count = 0;
      for (int cell_index = 0; cell_index < NCaloCells; ++cell_index)
        {
          if (missing_cell_count < m_missingCellsToFill.size() && cell_index == m_missingCellsToFill[missing_cell_count])
            {
              ++missing_cell_count;
              continue;
            }
          process_cell(cell_index, cell_index - missing_cell_count);
        }
    }
  else
    //Slow path: be careful.
    {
      CaloCellContainer::const_iterator iCells = cell_collection->begin();
 
      for (int cell_count = 0; iCells != cell_collection->end(); ++iCells, ++cell_count)
        {
          const CaloCell * cell = (*iCells);
 
          //const int cell_index = m_calo_id->calo_cell_hash(cell->ID());
          const int cell_index = cell->caloDDE()->calo_hash();
                 
          process_cell(cell_index, cell_count);
        }
    }
  const auto after_cells = clock_type::now();
 
  std::vector<int> cluster_order(ed.m_clusters->number);
 
  std::iota(cluster_order.begin(), cluster_order.end(), 0);
 
  std::sort(cluster_order.begin(), cluster_order.end(), [&](const int a, const int b) -> bool
  {
    const bool a_valid = ed.m_clusters->seedCellID[a] >= 0;
    const bool b_valid = ed.m_clusters->seedCellID[b] >= 0;
    if (a_valid && b_valid)
      {
        return ed.m_clusters->clusterEt[a]
        > ed.m_clusters->clusterEt[b];
      }
    else if (a_valid)
      {
        return true;
      }
    else if (b_valid)
      {
        return false;
      }
    else
      {
        return b > a;
      }
  } );
 
  //Ordered by Et as in the default algorithm...
  //The fact that some invalid clusters
  //(with possibly trash values for Et)
  //can crop up is irrelevant since
  //we don't add those anyway:
  //the rest is still ordered like we want it to be.
 
  const auto after_sort = clock_type::now();
 
  cluster_container->clear();
  cluster_container->reserve(cell_links.size());
 
  for (size_t i = 0; i < cluster_order.size(); ++i)
    {
      const int cluster_index = cluster_order[i];
 
      if (cell_links[cluster_index] != nullptr && cell_links[cluster_index]->size() > 0)
        {
          xAOD::CaloCluster * cluster = new xAOD::CaloCluster();
          cluster_container->push_back(cluster);
 
          cluster->addCellLink(cell_links[cluster_index].release());
          cluster->setClusterSize(m_clusterSize);
          if (m_useCPUPropertiesCalculation)
            {
              CaloClusterKineHelper::calculateKine(cluster, false, true, true);
            }
          else
            {
              cluster->setE(ed.m_clusters->clusterEnergy[cluster_index]);
              cluster->setEta(ed.m_clusters->clusterEta[cluster_index]);
              cluster->setPhi(ed.m_clusters->clusterPhi[cluster_index]);
            }
 
          if (m_saveUncalibrated)
            {
              cluster->setRawE(cluster->calE());
              cluster->setRawEta(cluster->calEta());
              cluster->setRawPhi(cluster->calPhi());
              cluster->setRawM(cluster->calM());
            }
 
        }
 
    }
 
  const auto after_fill = clock_type::now();
 
  if (m_measureTimes)
    {
      record_times(ctx.evt(), time_cast(start, after_send),
                   time_cast(after_send, after_creation),
                   time_cast(after_creation, after_cells),
                   time_cast(after_cells, after_sort),
                   time_cast(after_sort, after_fill)
                  );
    }
 
  return StatusCode::SUCCESS;
 
}

finalize()

StatusCode BasicGPUToAthenaImporter::finalize ( )

overridevirtual

Definition at line 325 of file BasicGPUToAthenaImporter.cxx.

{
 
  if (m_measureTimes)
    {
      print_times("Transfer_from_GPU Cluster_Creation Cell_Adding Sorting Collection_Filling", 5);
    }
  return StatusCode::SUCCESS;
}

initialize()

StatusCode BasicGPUToAthenaImporter::initialize ( )

overridevirtual

Definition at line 33 of file BasicGPUToAthenaImporter.cxx.

{
  ATH_CHECK( m_cellsKey.initialize() );
 
  ATH_CHECK( detStore()->retrieve(m_calo_id, "CaloCell_ID") );
 
  auto get_option_from_string = [](const std::string & str, bool & failed)
  {
    failed = false;
    CRGPU_RECURSIVE_MACRO(
            CRGPU_CHEAP_STRING_TO_ENUM( str, xAOD::CaloCluster,
                                        SW_55ele,
                                        SW_35ele,
                                        SW_37ele,
                                        SW_55gam,
                                        SW_35gam,
                                        SW_37gam,
                                        SW_55Econv,
                                        SW_35Econv,
                                        SW_37Econv,
                                        SW_softe,
                                        Topo_420,
                                        Topo_633,
                                        SW_7_11,
                                        SuperCluster,
                                        Tower_01_01,
                                        Tower_005_005,
                                        Tower_fixed_area
                                      )
    )
    //I know Topological Clustering only supports a subset of those,
    //but this is supposed to be a general data exporting tool...
    else
      {
        //failed = true;
        return xAOD::CaloCluster::CSize_Unknown;
      }
  };
 
  bool size_failed = false;
  m_clusterSize = get_option_from_string(m_clusterSizeString, size_failed);
 
  if (m_clusterSize == xAOD::CaloCluster::CSize_Unknown)
    {
      ATH_MSG_ERROR("Invalid Cluster Size: " << m_clusterSizeString);
    }
 
  if (size_failed)
    {
      return StatusCode::FAILURE;
    }
 
  return StatusCode::SUCCESS;
}

print_times()

void CaloGPUTimed::print_times ( const std::string &  header, const size_t  time_size  ) const

inlineprotectedinherited

Definition at line 143 of file CaloGPUTimed.h.

  {
    std::shared_lock<std::shared_mutex> lock(m_timeMutex);
    
    if (m_timeFileName.size() == 0)
      {
        return;
      }
    
    std::vector<size_t> indices(m_eventNumbers.size());
    
    std::iota(indices.begin(), indices.end(), 0);
    std::sort(indices.begin(), indices.end(), [&](size_t a, size_t b)
    {
      return m_eventNumbers[a] < m_eventNumbers[b];
    }
             );
    std::ofstream out(m_timeFileName);
 
    out << "Event_Number Total " << header << "\n";
 
    for (const size_t idx : indices)
      {
        out << m_eventNumbers[idx] << " ";
 
        size_t total = 0;
 
        for (size_t i = 0; i < time_size; ++i)
          {
            total += m_times[idx * time_size + i];
          }
 
        out << total << " ";
 
        for (size_t i = 0; i < time_size; ++i)
          {
            out << m_times[idx * time_size + i] << " ";
          }
        out << "\n";
      }
 
    out << std::endl;
 
    out.close();
  }

record_times() [1/3]

template<class ... Args>

void CaloGPUTimed::record_times ( const size_t  event_num, const size_t &  value  ) const

inlineprotectedinherited

Definition at line 105 of file CaloGPUTimed.h.

  {
    const size_t time_size = 1;
 
    size_t old_size;
 
    {
      std::unique_lock<std::shared_mutex> lock(m_timeMutex);
      old_size = m_times.size();
      m_times.resize(old_size + time_size);
      m_eventNumbers.push_back(event_num);
    }
    {
      std::shared_lock<std::shared_mutex> lock(m_timeMutex);
      record_times_helper(old_size, value);
    }
  }

record_times() [2/3]

template<class ... Args>

void CaloGPUTimed::record_times ( const size_t  event_num, const size_t &  value, Args &&...  args  ) const

inlineprotectedinherited

Definition at line 124 of file CaloGPUTimed.h.

  {
    const size_t time_size = sizeof...(args) + 1;
 
    size_t old_size;
 
    {
      std::unique_lock<std::shared_mutex> lock(m_timeMutex);
      old_size = m_times.size();
      m_times.resize(old_size + time_size);
      m_eventNumbers.push_back(event_num);
    }
    {
      std::shared_lock<std::shared_mutex> lock(m_timeMutex);
      record_times_helper(old_size, value, std::forward<Args>(args)...);
    }
 
  }

record_times() [3/3]

void CaloGPUTimed::record_times ( const size_t  event_num, const std::vector< size_t > &  times  ) const

inlineprotectedinherited

Definition at line 86 of file CaloGPUTimed.h.

  {
    size_t old_size;
    {
      std::unique_lock<std::shared_mutex> lock(m_timeMutex);
      old_size = m_times.size();
      m_times.resize(old_size + times.size());
      m_eventNumbers.push_back(event_num);
    }
    {
      std::shared_lock<std::shared_mutex> lock(m_timeMutex);
      for (size_t i = 0; i < times.size(); ++i)
        {
          m_times[old_size + i] = times[i];
        }
    }
  }

record_times_helper() [1/3]

template<class Arg >

void CaloGPUTimed::record_times_helper ( const size_t  index, Arg &&  arg  ) const

inlineprivateinherited

Definition at line 70 of file CaloGPUTimed.h.

  {
    // coverity[missing_lock]
    m_times[index] = std::forward<Arg>(arg);
    
    //This is called within a function that holds the lock itself.
  }

record_times_helper() [2/3]

void CaloGPUTimed::record_times_helper ( const  size_t ) const

inlineprivateinherited

Definition at line 64 of file CaloGPUTimed.h.

  {
    //Do nothing
  }

record_times_helper() [3/3]

template<class ... Args>

void CaloGPUTimed::record_times_helper ( size_t  index, Args &&...  args  ) const

inlineprivateinherited

Definition at line 79 of file CaloGPUTimed.h.

  {
    (record_times_helper(index++, std::forward<Args>(args)), ...);
  }

Member Data Documentation

ATLAS_THREAD_SAFE [1/2]

std::vector<size_t> m_times CaloGPUTimed::ATLAS_THREAD_SAFE

mutableprotectedinherited

Vector to hold execution times to be recorded if necessary.

Definition at line 35 of file CaloGPUTimed.h.

ATLAS_THREAD_SAFE [2/2]

std::vector<size_t> m_eventNumbers CaloGPUTimed::ATLAS_THREAD_SAFE

mutableprotectedinherited

Vector to hold the event numbers to be recorded if necessary.

Definition at line 40 of file CaloGPUTimed.h.

m_calo_id

const CaloCell_ID* BasicGPUToAthenaImporter::m_calo_id {nullptr}

private

Pointer to Calo ID Helper.

Definition at line 76 of file BasicGPUToAthenaImporter.h.

m_cellsKey

SG::ReadHandleKey<CaloCellContainer> BasicGPUToAthenaImporter::m_cellsKey {this, "CellsName", "", "Name(s) of Cell Containers"}

private

vector of names of the cell containers to use as input.

Definition at line 62 of file BasicGPUToAthenaImporter.h.

m_clusterSize

xAOD::CaloCluster::ClusterSize BasicGPUToAthenaImporter::m_clusterSize

private

Definition at line 67 of file BasicGPUToAthenaImporter.h.

m_clusterSizeString

Gaudi::Property<std::string> BasicGPUToAthenaImporter::m_clusterSizeString {this, "ClusterSize", "Topo_420", "The size/type of the clusters"}

private

Cluster size. Should be set accordingly to the threshold.

Definition at line 65 of file BasicGPUToAthenaImporter.h.

m_keepGPUData

Gaudi::Property<bool> BasicGPUToAthenaImporter::m_keepGPUData {this, "KeepGPUData", true, "Keep GPU allocated data"}

private

If true, do not delete the GPU data representation.

Defaults to true.

Definition at line 51 of file BasicGPUToAthenaImporter.h.

m_measureTimes

Gaudi::Property<bool> CaloGPUTimed::m_measureTimes

protectedinherited

If true, times are recorded to the file given by m_timeFileName.

Defaults to false.

Definition at line 46 of file CaloGPUTimed.h.

m_missingCellsToFill

Gaudi::Property<std::vector<int> > BasicGPUToAthenaImporter::m_missingCellsToFill {this, "MissingCellsToFill", {}, "Force fill these cells as disabled on empty containers."}

private

Cell indices to fill as disabled cells (useful if the cell vector is always missing the same cells).

Definition at line 71 of file BasicGPUToAthenaImporter.h.

m_saveUncalibrated

Gaudi::Property<bool> BasicGPUToAthenaImporter::m_saveUncalibrated {this, "SaveUncalibratedSignalState", true, "Use CaloClusterKineHelper::calculateKine instead of GPU-calculated cluster properties"}

private

if set to true, the uncalibrated state is saved when importing the clusters.

Default is true.

Definition at line 81 of file BasicGPUToAthenaImporter.h.

m_timeFileName

Gaudi::Property<std::string> CaloGPUTimed::m_timeFileName

protectedinherited

File to which times should be saved.

Definition at line 50 of file CaloGPUTimed.h.

m_timeMutex

std::shared_mutex CaloGPUTimed::m_timeMutex

mutableprotectedinherited

Mutex that is locked when recording times.

Definition at line 32 of file CaloGPUTimed.h.

m_useCPUPropertiesCalculation

Gaudi::Property<bool> BasicGPUToAthenaImporter::m_useCPUPropertiesCalculation {this, "UseCPUClusterPropertiesCalculation", false, "Use CaloClusterKineHelper::calculateKine instead of GPU-calculated cluster properties"}

private

if set to true, cluster properties are (re-)calculated using CaloClusterKineHelper::calculateKine.

Else, the GPU-calculated values are used. Default is false.

Definition at line 57 of file BasicGPUToAthenaImporter.h.

---

The documentation for this class was generated from the following files:

• BasicGPUToAthenaImporter.h
• BasicGPUToAthenaImporter.cxx