TFCSONNXHandler Class Reference

Copyright (C) 2002-2024 CERN for the benefit of the ATLAS collaboration. More...

#include <TFCSONNXHandler.h>

Inheritance diagram for TFCSONNXHandler:

Collaboration diagram for TFCSONNXHandler:

Public Types
typedef std::map< std::string, std::map< std::string, double > >	NetworkInputs
	Format for network inputs. More...
typedef std::map< std::string, double >	NetworkOutputs
	Format for network outputs. More...

Public Member Functions
	TFCSONNXHandler (const std::string &inputFile)
	TFCSONNXHandler constructor. More...
	TFCSONNXHandler (const std::vector< char > &bytes)
	TFCSONNXHandler constructor. More...
	TFCSONNXHandler (const TFCSONNXHandler &copy_from)
	TFCSONNXHandler copy constructor. More...
NetworkOutputs	compute (NetworkInputs const &inputs) const override
	Function to pass values to the network. More...
void	writeNetToTTree (TTree &tree) override
	Save the network to a TTree. More...
std::vector< std::string >	getOutputLayers () const override
	List the names of the outputs. More...
void	deleteAllButNet () override
	Get rid of any memory objects that arn't needed to run the net. More...
template<typename Tin , typename Tout >
VNetworkBase::NetworkOutputs	computeTemplate (VNetworkBase::NetworkInputs const &inputs)
	VNetworkBase ()
	VNetworkBase default constructor. More...
	VNetworkBase (const std::string &inputFile)
	VNetworkBase constructor. More...
	VNetworkBase (const VNetworkBase &copy_from)
	VNetworkBase copy constructor. More...
virtual void	writeNetToTTree (TTree &tree)=0
	Save the network to a TTree. More...
void	writeNetToTTree (TFile &root_file, std::string const &tree_name=m_defaultTreeName)
	Save the network to a TTree. More...
void	writeNetToTTree (std::string const &root_name, std::string const &tree_name=m_defaultTreeName)
	Save the network to a TTree. More...
void	writeNetToTTree (TFile &root_file, std::string const &tree_name=m_defaultTreeName)
	Save the network to a TTree. More...
void	writeNetToTTree (std::string const &root_name, std::string const &tree_name=m_defaultTreeName)
	Save the network to a TTree. More...
bool	isFile () const
	Check if the argument inputFile is the path of a file on disk. More...
bool	msgLvl (const MSG::Level lvl) const
	Check whether the logging system is active at the provided verbosity level. More...
MsgStream &	msg () const
	Return a stream for sending messages directly (no decoration) More...
MsgStream &	msg (const MSG::Level lvl) const
	Return a decorated starting stream for sending messages. More...
MSG::Level	level () const
	Retrieve output level. More...
virtual void	setLevel (MSG::Level lvl)
	Update outputlevel. More...

Static Public Member Functions
static std::string	representNetworkInputs (NetworkInputs const &inputs, int maxValues=3)
	String representation of network inputs. More...
static std::string	representNetworkOutputs (NetworkOutputs const &outputs, int maxValues=3)
	String representation of network outputs. More...
static bool	isFile (std::string const &inputFile)
	Check if a string is the path of a file on disk. More...
static std::string	startMsg (MSG::Level lvl, const std::string &file, int line)
	Make a message to decorate the start of logging. More...

Static Public Attributes
static const std::string	m_defaultTreeName = "onnxruntime_session"
	Default name for the TTree to save in. More...

Protected Member Functions
virtual void	print (std::ostream &strm) const override
	Write a short description of this net to the string stream. More...
void	setupPersistedVariables () override
	Perform actions that prep data to create the net. More...
void	setupNet () override
	Perform actions that prepare network for use. More...
bool	isRootFile (std::string const &filename="") const
	Check if a string is possibly a root file path. More...
void	removePrefixes (NetworkOutputs &outputs) const
	Remove any common prefix from the outputs. More...
void	removePrefixes (std::vector< std::string > &output_names) const
	Remove any common prefix from the outputs. More...

Protected Attributes
std::string	m_inputFile
	Path to the file describing the network, including filename. More...

Private Member Functions
std::vector< char >	getSerializedSession (const std::string &tree_name=m_defaultTreeName)
	Return content of the proto (.onnx) file in memory. More...
std::vector< char >	readBytesFromTTree (TTree &tree)
	Retrieve the content of the proto file from a TTree. More...
void	writeBytesToTTree (TTree &tree, const std::vector< char > &bytes)
	Write the content of the proto file to a TTree as a branch. More...
void	readSerializedSession ()
	Do not persistify. More...
template<typename Tin , typename Tout >
NetworkOutputs	computeTemplate (NetworkInputs const &input)
	Do not persistify. More...
	ClassDefOverride (TFCSONNXHandler, 1)
	Do not persistify. More...
	ClassDef (VNetworkBase, 1)

Private Attributes
std::vector< char >	m_bytes
	Content of the proto file. More...
std::unique_ptr< Ort::Session >	m_session
	The network session itself. More...
std::vector< const char * >	m_inputNodeNames
	names that index the input nodes More...
std::vector< const char * >	m_outputNodeNames
	Do not persistify. More...
std::vector< std::vector< int64_t > >	m_inputNodeDims
	Do not persistify. More...
std::vector< std::vector< int64_t > >	m_outputNodeDims
	Do not persistify. More...
std::vector< int64_t >	m_outputNodeSize
	Do not persistify. More...
std::function< NetworkOutputs(NetworkInputs)>	m_computeLambda
	computeTemplate with apropreate types selected. More...
Ort::MemoryInfo	m_memoryInfo
	Do not persistify. More...
std::vector< std::string >	m_outputLayers
	Do not persistify. More...
std::string	m_nm
	Message source name. More...

Static Private Attributes
static boost::thread_specific_ptr< MsgStream > m_msg_tls	ATLAS_THREAD_SAFE
	Do not persistify! More...

Detailed Description

Copyright (C) 2002-2024 CERN for the benefit of the ATLAS collaboration.

Class for a neural network read in the ONNX format. Derived from the abstract base class VNetworkBase such that it can be used interchangably with it's sibling classes, TFCSSimpleLWTNNHandler and TFCSGANLWTNNHandler.

The TFCSNetworkFactory::Create function has VNetworkBase as its return type so that it can make a run-time decision about which derived class to use, based on the data or file presented. As such it's best not to create this directly, instead allow TFCSNetworkFactory::Create to create the appropreat network object so that new network formats can be accomidated by writing new subclasses of VNetworkBase.

A handler specific for an ONNX network

Inherits from the generic interface VNetworkBase, such that it cna be used interchangably with other network formats and libraries.

Definition at line 43 of file TFCSONNXHandler.h.

Member Typedef Documentation

NetworkInputs

typedef std::map<std::string, std::map<std::string, double> > VNetworkBase::NetworkInputs

inherited

Format for network inputs.

The doubles are the values to be passed into the network. Strings in the outer map identify the input node, which must corrispond to the names of the nodes as read from the description of the network found by the constructor. Strings in the inner map identify the part of the input node, for some networks these must be simple integers, in string form, as parts of nodes do not always have the ability to carry real string labels.

Definition at line 90 of file VNetworkBase.h.

NetworkOutputs

typedef std::map<std::string, double> VNetworkBase::NetworkOutputs

inherited

Format for network outputs.

The doubles are the values generated by the network. Strings identify which node this value came from, and when nodes have multiple values, are suffixed with a number to indicate which part of the node they came from. So in multi-value nodes the format becomes "<node_name>_<part_n>"

Definition at line 100 of file VNetworkBase.h.

Constructor & Destructor Documentation

TFCSONNXHandler() [1/3]

TFCSONNXHandler::TFCSONNXHandler ( const std::string &  inputFile )

explicit

TFCSONNXHandler constructor.

Calls setupPersistedVariables and setupNet.

Parameters

inputFile

file-path on disk (with file name) of a readable onnx file containing a proto format description of the network to be constructed.

Definition at line 24 of file TFCSONNXHandler.cxx.

    : VNetworkBase(inputFile) {
  ATH_MSG_INFO("Setting up from inputFile.");
  TFCSONNXHandler::setupPersistedVariables();
  TFCSONNXHandler::setupNet();
  ATH_MSG_DEBUG("Setup from file complete");
};

TFCSONNXHandler() [2/3]

TFCSONNXHandler::TFCSONNXHandler ( const std::vector< char > &  bytes )

explicit

TFCSONNXHandler constructor.

As this passes nothing to the super constructor the setupPersistedVariables will not be called.

Parameters

bytes

byte content of a .onnx file, (which are a subset if proto files). Allows TFCSONNXHandler objects to be created from data in memory, retrived rom any source. The bytes are not copied interally, and must remain in memory while the net is in use. (TODO check that assertion)

Definition at line 32 of file TFCSONNXHandler.cxx.

    : m_bytes(bytes) {
  ATH_MSG_INFO("Given onnx session bytes as input.");
  // The super constructor got no inputFile,
  // so it won't call setupNet itself
  TFCSONNXHandler::setupNet();
  ATH_MSG_DEBUG("Setup from session complete");
};

TFCSONNXHandler() [3/3]

TFCSONNXHandler::TFCSONNXHandler

(

const TFCSONNXHandler &

copy_from

)

TFCSONNXHandler copy constructor.

Will copy the variables taht would be generated by setupPersistedVariables and setupNet.

Parameters

copy_from

existing network that we are copying

Definition at line 41 of file TFCSONNXHandler.cxx.

    : VNetworkBase(copy_from) {
  ATH_MSG_DEBUG("TFCSONNXHandler copy construtor called");
  m_bytes = copy_from.m_bytes;
  // Cannot copy a session
  // m_session = copy_from.m_session;
  // But can read it from bytes
  readSerializedSession();
  m_inputNodeNames = copy_from.m_inputNodeNames;
  m_inputNodeDims = copy_from.m_inputNodeDims;
  m_outputNodeNames = copy_from.m_outputNodeNames;
  m_outputNodeDims = copy_from.m_outputNodeDims;
  m_outputLayers = copy_from.m_outputLayers;
};

Member Function Documentation

ClassDef()

VNetworkBase::ClassDef ( VNetworkBase  , 1    )

privateinherited

ClassDefOverride()

TFCSONNXHandler::ClassDefOverride ( TFCSONNXHandler  , 1    )

private

Do not persistify.

compute()

TFCSONNXHandler::NetworkOutputs TFCSONNXHandler::compute ( TFCSONNXHandler::NetworkInputs const &  inputs ) const

overridevirtual

Function to pass values to the network.

This function is used to actually run data through the loaded network and obtain results.

Parameters

inputs

values to be evaluated by the network

Returns

the output of the network

See also

VNetworkBase::NetworkInputs

VNetworkBase::NetworkOutputs

Implements VNetworkBase.

Definition at line 57 of file TFCSONNXHandler.cxx.

                                                                         {
  return m_computeLambda(inputs);
};

computeTemplate() [1/2]

template<typename Tin , typename Tout >

NetworkOutputs TFCSONNXHandler::computeTemplate ( NetworkInputs const &  input )

private

Do not persistify.

Computation template with adjustable types for input.

A lambda function will be used to make the correct type choice for the session/net used as a member variable during setupNet.

computeTemplate() [2/2]

template<typename Tin , typename Tout >

VNetworkBase::NetworkOutputs TFCSONNXHandler::computeTemplate

(

VNetworkBase::NetworkInputs const &

inputs

)

Definition at line 338 of file TFCSONNXHandler.cxx.

                                                                        {
  // working from
  // https://github.com/microsoft/onnxruntime-inference-examples/blob/main/c_cxx/squeezenet/main.cpp#L71
  //  and
  //  https://github.com/microsoft/onnxruntime-inference-examples/blob/main/c_cxx/MNIST/MNIST.cpp
  ATH_MSG_DEBUG("Setting up inputs for computation on ONNX network.");
  ATH_MSG_DEBUG("Input type " << typeid(Tin).name() << " output type "
                              << typeid(Tout).name());
 
  //  The inputs must be reformatted to the correct data structure.
  const size_t num_input_nodes = m_inputNodeNames.size();
  //  A pointer to all the nodes we will make
  //  Gonna keep the data in each node flat, becuase that's easier
  std::vector<std::vector<Tin>> input_values(num_input_nodes);
  std::vector<Ort::Value> node_values;
  // Non const values that will be needed at each step.
  std::string node_name;
  int n_dimensions, elements_in_node, key_number;
  size_t first_digit;
  // Move along the list of node names gathered in the constructor
  // we need both the node name, and the dimension
  // so we cannot itterate directly on the vector.
  ATH_MSG_DEBUG("Looping over " << num_input_nodes
                                << " input nodes of ONNX network.");
  for (size_t node_n = 0; node_n < m_inputNodeNames.size(); node_n++) {
    ATH_MSG_DEBUG("Node n = " << node_n);
    node_name = m_inputNodeNames[node_n];
    ATH_MSG_DEBUG("Node name " << node_name);
    // Get the shape of this node
    n_dimensions = m_inputNodeDims[node_n].size();
    ATH_MSG_DEBUG("Node dimensions " << n_dimensions);
    elements_in_node = 1;
    for (int dimension_len : m_inputNodeDims[node_n]) {
      elements_in_node *= dimension_len;
    };
    ATH_MSG_DEBUG("Elements in node " << elements_in_node);
    for (auto inp : inputs) {
      ATH_MSG_DEBUG("Have input named " << inp.first);
    };
    // Get the node content and remove any common prefix from the elements
    const std::map<std::string, double> node_inputs = inputs.at(node_name);
    std::vector<Tin> node_elements(elements_in_node);
 
    ATH_MSG_DEBUG("Found node named " << node_name << " with "
                                      << elements_in_node << " elements.");
    // Then the rest should be numbers from 0 up
    for (auto element : node_inputs){
      first_digit = element.first.find_first_of("0123456789");
      // if there is no digit, it's not an element
      if (first_digit < element.first.length()){
        key_number = std::stoi(element.first.substr(first_digit));
        node_elements[key_number] = element.second;
      }
    }
    input_values[node_n] = node_elements;
 
    ATH_MSG_DEBUG("Creating ort tensor n_dimensions = "
                  << n_dimensions
                  << ", elements_in_node = " << elements_in_node);
    // Doesn't copy data internally, so vector arguments need to stay alive
    Ort::Value node = Ort::Value::CreateTensor<Tin>(
        m_memoryInfo, input_values[node_n].data(), elements_in_node,
        m_inputNodeDims[node_n].data(), n_dimensions);
    // Problems with the string steam when compiling seperatly.
    // ATH_MSG_DEBUG("Created input node " << node << " from values " <<
    // input_values[node_n]);
 
    node_values.push_back(std::move(node));
  }
 
  ATH_MSG_DEBUG("Running computation on ONNX network.");
  // All inputs have been correctly formatted and the net can be run.
  auto output_tensors = m_session->Run(
      Ort::RunOptions{nullptr}, m_inputNodeNames.data(), &node_values[0],
      num_input_nodes, m_outputNodeNames.data(), m_outputNodeNames.size());
 
  ATH_MSG_DEBUG("Sorting outputs from computation on ONNX network.");
  // Finaly, the output must be rearanged in the expected format.
  TFCSONNXHandler::NetworkOutputs outputs;
  // as the output format is just a string to double map
  // the outputs will be keyed like "<node_name>_<part_n>"
  std::string output_name;
  const Tout *output_node;
  for (size_t node_n = 0; node_n < m_outputNodeNames.size(); node_n++) {
    // get a pointer to the data
    output_node = output_tensors[node_n].GetTensorMutableData<Tout>();
    ATH_MSG_VERBOSE("output node " << output_node);
    elements_in_node = m_outputNodeSize[node_n];
    node_name = m_outputNodeNames[node_n];
    // Does the GetTensorMutableData really always return a
    // flat array?
    // Likely yes, see use of memcopy on line 301 of
    // onnxruntime/core/languge_interop_ops/pyop/pyop.cc
    for (int part_n = 0; part_n < elements_in_node; part_n++) {
      ATH_MSG_VERBOSE("Node part " << part_n << " contains "
                                   << output_node[part_n]);
      // compose the output name
      output_name = node_name + "_" + std::to_string(part_n);
      outputs[output_name] = static_cast<double>(output_node[part_n]);
    }
  }
  removePrefixes(outputs);
  ATH_MSG_DEBUG("Returning outputs from computation on ONNX network.");
  return outputs;
};

deleteAllButNet()

void TFCSONNXHandler::deleteAllButNet ( )

overridevirtual

Get rid of any memory objects that arn't needed to run the net.

Minimise memory usage by deleting nay inputs that are no longer required to run the compute function. Doesn't actually do anything for this network type.

Implements VNetworkBase.

Definition at line 72 of file TFCSONNXHandler.cxx.

                                      {
  // As we don't copy the bytes, and the inputFile
  // is at most a name, nothing is needed here.
  ATH_MSG_DEBUG("Deleted nothing for ONNX.");
};

getOutputLayers()

std::vector< std::string > TFCSONNXHandler::getOutputLayers ( ) const

overridevirtual

List the names of the outputs.

Outputs are stored in an NetworkOutputs object which is indexed by strings. This function returns the list of all strings that will index the outputs.

Implements VNetworkBase.

Definition at line 67 of file TFCSONNXHandler.cxx.

                                                            {
  ATH_MSG_DEBUG("TFCSONNXHandler output layers requested.");
  return m_outputLayers;
};

getSerializedSession()

std::vector< char > TFCSONNXHandler::getSerializedSession ( const std::string &  tree_name = m_defaultTreeName )

private

Return content of the proto (.onnx) file in memory.

Get the session as a stream of bytes It's a vector<char> rather than a string becuase we need the guarantee that &bytes[0]+n == bytes[n] (string has this only after c++11). Also bytes may not be terminated by a null byte (which early strings required).

Definition at line 273 of file TFCSONNXHandler.cxx.

                                                                                {
  ATH_MSG_DEBUG("Getting serialized session for ONNX network.");
 
  if (this->isRootFile()) {
    ATH_MSG_INFO("Reading bytes from root file.");
    TFile tfile(this->m_inputFile.c_str(), "READ");
    TTree *tree = (TTree *)tfile.Get(tree_name.c_str());
    std::vector<char> bytes = this->readBytesFromTTree(*tree);
    ATH_MSG_DEBUG("Found bytes size " << bytes.size());
    return bytes;
  } else {
    ATH_MSG_INFO("Reading bytes from text file.");
    // see https://stackoverflow.com/a/50317432
    std::ifstream input(this->m_inputFile, std::ios::binary);
 
    std::vector<char> bytes((std::istreambuf_iterator<char>(input)),
                            (std::istreambuf_iterator<char>()));
 
    input.close();
    ATH_MSG_DEBUG("Found bytes size " << bytes.size());
    return bytes;
  }
};

isFile() [1/2]

bool VNetworkBase::isFile ( ) const

inherited

Check if the argument inputFile is the path of a file on disk.

Determines if the string that was passed to the constructor as inputFile corrisponds to tha path of a file that can be read on the disk.

Returns

is it a readable file on disk

Definition at line 117 of file VNetworkBase.cxx.

{ return isFile(m_inputFile); };

isFile() [2/2]

bool VNetworkBase::isFile ( std::string const &  inputFile )

staticinherited

Check if a string is the path of a file on disk.

Determines if a string corrisponds to tha path of a file that can be read on the disk.

Parameters

inputFile

name of the pottential file

Returns

is it a readable file on disk

Definition at line 119 of file VNetworkBase.cxx.

                                                    {
  if (FILE *file = std::fopen(inputFile.c_str(), "r")) {
    std::fclose(file);
    return true;
  } else {
    return false;
  };
};

isRootFile()

bool VNetworkBase::isRootFile ( std::string const &  filename = "" ) const

protectedinherited

Check if a string is possibly a root file path.

Just checks if the string ends in .root as there are almost no reliable rules for file paths.

Parameters

inputFile

name of the pottential file if blank, m_inputFile is used.

Returns

is it the path of a root file

Definition at line 101 of file VNetworkBase.cxx.

                                                             {
  const std::string *to_check = &filename;
  if (filename.length() == 0) {
    to_check = &this->m_inputFile;
    ATH_MSG_DEBUG("No file name given, so using m_inputFile, " << m_inputFile);
  };
  const std::string ending = ".root";
  const int ending_len = ending.length();
  const int filename_len = to_check->length();
  if (filename_len < ending_len) {
    return false;
  }
  return (0 ==
          to_check->compare(filename_len - ending_len, ending_len, ending));
};

level()

MSG::Level ISF_FCS::MLogging::level ( ) const

inlineinherited

Retrieve output level.

Definition at line 201 of file MLogging.h.

{ return msg().level(); }

msg() [1/2]

MsgStream & ISF_FCS::MLogging::msg ( ) const

inlineinherited

Return a stream for sending messages directly (no decoration)

Definition at line 231 of file MLogging.h.

                                      {
  MsgStream *ms = m_msg_tls.get();
  if (!ms) {
    ms = new MsgStream(Athena::getMessageSvc(), m_nm);
    m_msg_tls.reset(ms);
  }
  return *ms;
}

msg() [2/2]

MsgStream & ISF_FCS::MLogging::msg ( const MSG::Level  lvl ) const

inlineinherited

Return a decorated starting stream for sending messages.

Definition at line 240 of file MLogging.h.

                                                        {
  return msg() << lvl;
}

msgLvl()

bool ISF_FCS::MLogging::msgLvl ( const MSG::Level  lvl ) const

inlineinherited

Check whether the logging system is active at the provided verbosity level.

Definition at line 222 of file MLogging.h.

                                                     {
  if (msg().level() <= lvl) {
    msg() << lvl;
    return true;
  } else {
    return false;
  }
}

print()

void TFCSONNXHandler::print ( std::ostream &  strm ) const

overrideprotectedvirtual

Write a short description of this net to the string stream.

Specialised for ONNX to print the input and output nodes with their dimensions.

Parameters

strm	output parameter, to which the description will be written.

Reimplemented from VNetworkBase.

Definition at line 78 of file TFCSONNXHandler.cxx.

                                                  {
  if (m_inputFile.empty()) {
    strm << "Unknown network";
  } else {
    strm << m_inputFile;
  };
  strm << "\nHas input nodes (name:dimensions);\n";
  for (size_t inp_n = 0; inp_n < m_inputNodeNames.size(); inp_n++) {
    strm << "\t" << m_inputNodeNames[inp_n] << ":[";
    for (int dim : m_inputNodeDims[inp_n]) {
      strm << " " << dim << ",";
    };
    strm << "]\n";
  };
  strm << "\nHas output nodes (name:dimensions);\n";
  for (size_t out_n = 0; out_n < m_outputNodeNames.size(); out_n++) {
    strm << "\t" << m_outputNodeNames[out_n] << ":[";
    for (int dim : m_outputNodeDims[out_n]) {
      strm << " " << dim << ",";
    };
    strm << "]\n";
  };
};

readBytesFromTTree()

std::vector< char > TFCSONNXHandler::readBytesFromTTree ( TTree &  tree )

private

Retrieve the content of the proto file from a TTree.

If the ONNX file has been saved as a loose variable in a TTree this method will read it back into m_bytes.

Definition at line 297 of file TFCSONNXHandler.cxx.

                                                               {
  ATH_MSG_DEBUG("TFCSONNXHandler reading bytes from tree.");
  std::vector<char> bytes;
  char data;
  tree.SetBranchAddress("serialized_m_session", &data);
  for (int i = 0; tree.LoadTree(i) >= 0; i++) {
    tree.GetEntry(i);
    bytes.push_back(data);
  };
  ATH_MSG_DEBUG("TFCSONNXHandler read bytes from tree.");
  return bytes;
};

readSerializedSession()

void TFCSONNXHandler::readSerializedSession ( )

private

Do not persistify.

Using content of the proto (.onnx) file make a session.

The m_session variable is initialised from the m_bytes variable so that the net can be run. Requires that the m_bytes variable is retained while the net is used.

Definition at line 324 of file TFCSONNXHandler.cxx.

                                            {
  ATH_MSG_DEBUG("Transforming bytes to session.");
  Ort::Env env(ORT_LOGGING_LEVEL_WARNING, "test");
  Ort::SessionOptions opts;
  opts.SetInterOpNumThreads(1);
  opts.SetIntraOpNumThreads(1);
  // Prevent ONNX from spawning additional threads
  m_session =
      std::make_unique<Ort::Session>(env, m_bytes.data(), m_bytes.size(), opts);
  ATH_MSG_DEBUG("Transformed bytes to session.");
};

removePrefixes() [1/2]

void VNetworkBase::removePrefixes ( VNetworkBase::NetworkOutputs &  outputs ) const

protectedinherited

Remove any common prefix from the outputs.

Parameters

outputs

The outputs, changed in place.

Definition at line 151 of file VNetworkBase.cxx.

                                                                           {
  std::vector<std::string> output_layers;
  for (auto const &output : outputs)
    output_layers.push_back(output.first);
  const int length = GetPrefixLength(output_layers);
  for (std::string layer_name : output_layers) {
    // remove this output
    auto nodeHandle = outputs.extract(layer_name);
    // change the key
    nodeHandle.key() = layer_name.substr(length);
    // replace the output
    outputs.insert(std::move(nodeHandle));
  }
};

removePrefixes() [2/2]

void VNetworkBase::removePrefixes ( std::vector< std::string > &  output_names ) const

protectedinherited

Remove any common prefix from the outputs.

Parameters

outputs

The output names, changed in place.

Definition at line 144 of file VNetworkBase.cxx.

                                              {
  const int length = GetPrefixLength(output_names);
  for (long unsigned int i = 0; i < output_names.size(); i++)
    output_names[i] = output_names[i].substr(length);
};

representNetworkInputs()

std::string VNetworkBase::representNetworkInputs ( VNetworkBase::NetworkInputs const &  inputs, int  maxValues = 3  )

staticinherited

String representation of network inputs.

Create a string that summarises a set of network inputs. Gives basic dimensions plus a few values, up to the maxValues

Parameters

inputs	values to be evaluated by the network
maxValues	maximum number of values to include in the representaiton

Returns

string represetning the inputs

Definition at line 37 of file VNetworkBase.cxx.

                                                    {
  std::string representation =
      "NetworkInputs, outer size " + std::to_string(inputs.size());
  int valuesIncluded = 0;
  for (const auto &outer : inputs) {
    representation += "\n key->" + outer.first + "; ";
    for (const auto &inner : outer.second) {
      representation += inner.first + "=" + std::to_string(inner.second) + ", ";
      ++valuesIncluded;
      if (valuesIncluded > maxValues)
        break;
    };
    if (valuesIncluded > maxValues)
      break;
  };
  representation += "\n";
  return representation;
};

representNetworkOutputs()

std::string VNetworkBase::representNetworkOutputs ( VNetworkBase::NetworkOutputs const &  outputs, int  maxValues = 3  )

staticinherited

String representation of network outputs.

Create a string that summarises a set of network outputs. Gives basic dimensions plus a few values, up to the maxValues

Parameters

outputs	output of the network
maxValues	maximum number of values to include in the representaiton

Returns

string represetning the outputs

Definition at line 57 of file VNetworkBase.cxx.

                                                              {
  std::string representation =
      "NetworkOutputs, size " + std::to_string(outputs.size()) + "; \n";
  int valuesIncluded = 0;
  for (const auto &item : outputs) {
    representation += item.first + "=" + std::to_string(item.second) + ", ";
    ++valuesIncluded;
    if (valuesIncluded > maxValues)
      break;
  };
  representation += "\n";
  return representation;
};

setLevel()

void ISF_FCS::MLogging::setLevel ( MSG::Level  lvl )

virtualinherited

Update outputlevel.

Definition at line 105 of file MLogging.cxx.

                                    {
  lvl = (lvl >= MSG::NUM_LEVELS) ? MSG::ALWAYS
        : (lvl < MSG::NIL)       ? MSG::NIL
                                 : lvl;
  msg().setLevel(lvl);
}

setupNet()

void TFCSONNXHandler::setupNet ( )

overrideprotectedvirtual

Perform actions that prepare network for use.

Will be called in the streamer or class constructor after the inputs have been set (either automaically by the streamer or by setupPersistedVariables in the constructor). Does not delete any resources used.

Implements VNetworkBase.

Definition at line 112 of file TFCSONNXHandler.cxx.

                               {
  // From
  // https://gitlab.cern.ch/atlas/athena/-/blob/master/Control/AthOnnxruntimeUtils/AthOnnxruntimeUtils/OnnxUtils.h
  // m_session = AthONNX::CreateORTSession(inputFile);
  // This segfaults.
 
  // TODO; should I be using m_session_options? see
  // https://github.com/microsoft/onnxruntime-inference-examples/blob/2b42b442526b9454d1e2d08caeb403e28a71da5f/c_cxx/squeezenet/main.cpp#L71
  ATH_MSG_INFO("Setting up ONNX session.");
  this->readSerializedSession();
 
  // Need the type from the first node (which will be used to set
  // just set it to undefined to avoid not initialised warnings
  ONNXTensorElementDataType first_input_type =
      ONNX_TENSOR_ELEMENT_DATA_TYPE_UNDEFINED;
 
  // iterate over all input nodes
  ATH_MSG_DEBUG("Getting input nodes.");
  const int num_input_nodes = m_session->GetInputCount();
  Ort::AllocatorWithDefaultOptions allocator;
  for (int i = 0; i < num_input_nodes; i++) {
 
#if ORT_API_VERSION > 11
    Ort::AllocatedStringPtr node_names = m_session->GetInputNameAllocated(i, allocator);
    m_storeInputNodeNames.push_back(std::move(node_names));
    const char *input_name = m_storeInputNodeNames.back().get();
#else
    const char *input_name = m_session->GetInputName(i, allocator);
#endif
    m_inputNodeNames.push_back(input_name);
    ATH_MSG_VERBOSE("Found input node named " << input_name);
 
    Ort::TypeInfo type_info = m_session->GetInputTypeInfo(i);
 
    // For some reason unless auto is used as the return type
    // this causes a segfault once the loop ends....
    auto tensor_info = type_info.GetTensorTypeAndShapeInfo();
    if (i == 0)
      first_input_type = tensor_info.GetElementType();
    // Check the type has not changed
    if (tensor_info.GetElementType() != first_input_type) {
      ATH_MSG_ERROR("First type was " << first_input_type << ". In node " << i
                                      << " found type "
                                      << tensor_info.GetElementType());
      throw std::runtime_error("Networks with varying input types not "
                               "yet impelmented in TFCSONNXHandler.");
    };
 
    std::vector<int64_t> recieved_dimension = tensor_info.GetShape();
    ATH_MSG_VERBOSE("There are " << recieved_dimension.size()
                                 << " dimensions.");
    // This vector sometimes includes a symbolic dimension
    // which is represented by -1
    // A symbolic dimension is usually a conversion error,
    // from a numpy array with a shape like (None, 7),
    // in which case it's safe to treat it as having
    // dimension 1.
    std::vector<int64_t> dimension_of_node;
    for (int64_t node_dim : recieved_dimension) {
      if (node_dim < 1) {
        ATH_MSG_WARNING("Found symbolic dimension "
                        << node_dim << " in node named " << input_name
                        << ". Will treat this as dimension 1.");
        dimension_of_node.push_back(1);
      } else {
        dimension_of_node.push_back(node_dim);
      };
    };
    m_inputNodeDims.push_back(dimension_of_node);
  };
  ATH_MSG_DEBUG("Finished looping on inputs.");
 
  // Outputs
  // Store the type from the first node (which will be used to set
  // m_computeLambda)
  ONNXTensorElementDataType first_output_type =
      ONNX_TENSOR_ELEMENT_DATA_TYPE_UNDEFINED;
 
  // iterate over all output nodes
  int num_output_nodes = m_session->GetOutputCount();
  ATH_MSG_DEBUG("Getting " << num_output_nodes << " output nodes.");
  for (int i = 0; i < num_output_nodes; i++) {
#if ORT_API_VERSION > 11
    Ort::AllocatedStringPtr node_names = m_session->GetOutputNameAllocated(i, allocator);
    m_storeOutputNodeNames.push_back(std::move(node_names));
    const char *output_name = m_storeOutputNodeNames.back().get();
#else
    const char *output_name = m_session->GetOutputName(i, allocator);
#endif
    m_outputNodeNames.push_back(output_name);
    ATH_MSG_VERBOSE("Found output node named " << output_name);
 
    const Ort::TypeInfo type_info = m_session->GetOutputTypeInfo(i);
    auto tensor_info = type_info.GetTensorTypeAndShapeInfo();
    if (i == 0)
      first_output_type = tensor_info.GetElementType();
 
    // Check the type has not changed
    if (tensor_info.GetElementType() != first_output_type) {
      ATH_MSG_ERROR("First type was " << first_output_type << ". In node " << i
                                      << " found type "
                                      << tensor_info.GetElementType());
      throw std::runtime_error("Networks with varying output types not "
                               "yet impelmented in TFCSONNXHandler.");
    };
 
    const std::vector<int64_t> recieved_dimension = tensor_info.GetShape();
    ATH_MSG_VERBOSE("There are " << recieved_dimension.size()
                                 << " dimensions.");
    // Again, check for sybolic dimensions
    std::vector<int64_t> dimension_of_node;
    int node_size = 1;
    for (int64_t node_dim : recieved_dimension) {
      if (node_dim < 1) {
        ATH_MSG_WARNING("Found symbolic dimension "
                        << node_dim << " in node named " << output_name
                        << ". Will treat this as dimension 1.");
        dimension_of_node.push_back(1);
      } else {
        dimension_of_node.push_back(node_dim);
        node_size *= node_dim;
      };
    };
    m_outputNodeDims.push_back(dimension_of_node);
    m_outputNodeSize.push_back(node_size);
 
    // The outputs are treated as a flat vector
    for (int part_n = 0; part_n < node_size; part_n++) {
      // compose the output name
      std::string layer_name =
          std::string(output_name) + "_" + std::to_string(part_n);
      ATH_MSG_VERBOSE("Found output layer named " << layer_name);
      m_outputLayers.push_back(layer_name);
    }
  }
  ATH_MSG_DEBUG("Removing prefix from stored layers.");
  removePrefixes(m_outputLayers);
  ATH_MSG_DEBUG("Finished output nodes.");
 
  ATH_MSG_DEBUG("Setting up m_computeLambda with input type "
                << first_input_type << " and output type "
                << first_output_type);
  if (first_input_type == ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT &&
      first_output_type == ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT) {
    // gotta capture this in the lambda so it can access class methods
    m_computeLambda = [this](NetworkInputs const &inputs) {
      return computeTemplate<float, float>(inputs);
    };
  } else if (first_input_type == ONNX_TENSOR_ELEMENT_DATA_TYPE_DOUBLE &&
             first_output_type == ONNX_TENSOR_ELEMENT_DATA_TYPE_DOUBLE) {
    m_computeLambda = [this](NetworkInputs const &inputs) {
      return computeTemplate<double, double>(inputs);
    };
  } else {
    throw std::runtime_error("Haven't yet implemented that combination of "
                             "input and output types as a subclass of VState.");
  };
  ATH_MSG_DEBUG("Finished setting lambda function.");
};

setupPersistedVariables()

void TFCSONNXHandler::setupPersistedVariables ( )

overrideprotectedvirtual

Perform actions that prep data to create the net.

Will be called in the class constructor before calling setupNet, but not in the streamer. It sets any variables that the sreamer would persist when saving or loading to file.

Implements VNetworkBase.

Definition at line 102 of file TFCSONNXHandler.cxx.

                                              {
  ATH_MSG_DEBUG("Setting up persisted variables for ONNX network.");
  // depending which constructor was called,
  // bytes may already be filled
  if (m_bytes.empty()) {
    m_bytes = getSerializedSession();
  };
  ATH_MSG_DEBUG("Setup persisted variables for ONNX network.");
};

startMsg()

std::string ISF_FCS::MLogging::startMsg ( MSG::Level  lvl, const std::string &  file, int  line  )

staticinherited

Make a message to decorate the start of logging.

Print a message for the start of logging.

Definition at line 116 of file MLogging.cxx.

                                                                          {
  int col1_len = 20;
  int col2_len = 5;
  int col3_len = 10;
  auto last_slash = file.find_last_of('/');
  int path_len = last_slash == std::string::npos ? 0 : last_slash;
  int trim_point = path_len;
  int total_len = file.length();
  if (total_len - path_len > col1_len)
    trim_point = total_len - col1_len;
  std::string trimmed_name = file.substr(trim_point);
  const char *LevelNames[MSG::NUM_LEVELS] = {
      "NIL", "VERBOSE", "DEBUG", "INFO", "WARNING", "ERROR", "FATAL", "ALWAYS"};
  std::string level = LevelNames[lvl];
  std::string level_string = std::string("(") + level + ") ";
  std::stringstream output;
  output << std::setw(col1_len) << std::right << trimmed_name << ":"
         << std::setw(col2_len) << std::left << line << std::setw(col3_len)
         << std::right << level_string;
  return output.str();
}

VNetworkBase() [1/3]

VNetworkBase::VNetworkBase

VNetworkBase default constructor.

For use in streamers.

Definition at line 45 of file VNetworkBase.cxx.

: m_inputFile("unknown"){};

VNetworkBase() [2/3]

VNetworkBase::VNetworkBase

explicit

VNetworkBase constructor.

Only saves inputFile to m_inputFile; Inherting classes should call setupPersistedVariables and setupNet in constructor;

Parameters

inputFile

file-path on disk (with file name) of a readable file containing a description of the network to be constructed or the content of the file.

Definition at line 59 of file VNetworkBase.cxx.

    : m_inputFile(inputFile) {
  ATH_MSG_DEBUG("Constructor called with inputFile");
};

VNetworkBase() [3/3]

VNetworkBase::VNetworkBase

VNetworkBase copy constructor.

Does not call setupPersistedVariables or setupNet but will pass on m_inputFile. Inherting classes should do whatever they need to move the variables created in the setup functions.

Parameters

copy_from

existing network that we are copying

Definition at line 71 of file VNetworkBase.cxx.

  : MLogging(),
    m_inputFile (copy_from.m_inputFile)
{
};

writeBytesToTTree()

void TFCSONNXHandler::writeBytesToTTree ( TTree &  tree, const std::vector< char > &  bytes  )

private

Write the content of the proto file to a TTree as a branch.

The ONNX proto file is saved as a simple branch (no streamers involved).

Definition at line 310 of file TFCSONNXHandler.cxx.

                                                                      {
  ATH_MSG_DEBUG("TFCSONNXHandler writing bytes to tree.");
  char m_session_data;
  tree.Branch("serialized_m_session", &m_session_data,
              "serialized_m_session/B");
  for (Char_t here : bytes) {
    m_session_data = here;
    tree.Fill();
  };
  tree.Write();
  ATH_MSG_DEBUG("TFCSONNXHandler written bytes to tree.");
};

writeNetToTTree() [1/6]

void VNetworkBase::writeNetToTTree

Save the network to a TTree.

All data required to recreate the network object is saved into a TTree. The format is not specified.

Parameters

root_name	The path of the file to save inside.
tree_name	The name of the TTree to save inside.

Definition at line 196 of file VNetworkBase.cxx.

                                                               {
  ATH_MSG_DEBUG("Making or updating file name " << root_name);
  TFile root_file(root_name.c_str(), "UPDATE");
  this->writeNetToTTree(root_file, tree_name);
  root_file.Close();
};

writeNetToTTree() [2/6]

void VNetworkBase::writeNetToTTree ( std::string const &  root_name, std::string const &  tree_name = m_defaultTreeName  )

inherited

Save the network to a TTree.

All data required to recreate the network object is saved into a TTree. The format is not specified.

Parameters

root_name	The path of the file to save inside.
tree_name	The name of the TTree to save inside.

Definition at line 93 of file VNetworkBase.cxx.

                                                               {
  ATH_MSG_DEBUG("Making or updating file name " << root_name);
  TFile root_file(root_name.c_str(), "UPDATE");
  this->writeNetToTTree(root_file, tree_name);
  root_file.Close();
};

writeNetToTTree() [3/6]

void VNetworkBase::writeNetToTTree

Save the network to a TTree.

All data required to recreate the network object is saved into a TTree. The format is not specified.

Parameters

root_file	The file to save inside.
tree_name	The name of the TTree to save inside.

Definition at line 184 of file VNetworkBase.cxx.

                                                               {
  ATH_MSG_DEBUG("Making tree name " << tree_name);
  root_file.cd();
  const std::string title = "onnxruntime saved network";
  TTree tree(tree_name.c_str(), title.c_str());
  this->writeNetToTTree(tree);
  root_file.Write();
};

writeNetToTTree() [4/6]

void VNetworkBase::writeNetToTTree ( TFile &  root_file, std::string const &  tree_name = m_defaultTreeName  )

inherited

Save the network to a TTree.

All data required to recreate the network object is saved into a TTree. The format is not specified.

Parameters

root_file	The file to save inside.
tree_name	The name of the TTree to save inside.

Definition at line 83 of file VNetworkBase.cxx.

                                                               {
  ATH_MSG_DEBUG("Making tree name " << tree_name);
  root_file.cd();
  const std::string title = "onnxruntime saved network";
  TTree tree(tree_name.c_str(), title.c_str());
  this->writeNetToTTree(tree);
  root_file.Write();
};

writeNetToTTree() [5/6]

void TFCSONNXHandler::writeNetToTTree ( TTree &  tree )

overridevirtual

Save the network to a TTree.

All data required to recreate the network object is saved into a TTree. The format is not specified. Will still work even if deleteAllButNet has already been called.

Parameters

tree	The tree to save inside.

Implements VNetworkBase.

Definition at line 62 of file TFCSONNXHandler.cxx.

                                                 {
  ATH_MSG_DEBUG("TFCSONNXHandler writing net to tree.");
  this->writeBytesToTTree(tree, m_bytes);
};

writeNetToTTree() [6/6]

virtual void VNetworkBase::writeNetToTTree

Save the network to a TTree.

All data required to recreate the network object is saved into a TTree. The format is not specified.

Parameters

tree	The tree to save inside.

Member Data Documentation

ATLAS_THREAD_SAFE

boost::thread_specific_ptr<MsgStream> m_msg_tls ISF_FCS::MLogging::ATLAS_THREAD_SAFE

inlinestaticprivateinherited

Do not persistify!

MsgStream instance (a std::cout like with print-out levels)

Definition at line 215 of file MLogging.h.

m_bytes

std::vector<char> TFCSONNXHandler::m_bytes

private

Content of the proto file.

Definition at line 170 of file TFCSONNXHandler.h.

m_computeLambda

std::function<NetworkOutputs(NetworkInputs)> TFCSONNXHandler::m_computeLambda

private

computeTemplate with apropreate types selected.

Definition at line 307 of file TFCSONNXHandler.h.

m_defaultTreeName

const std::string VNetworkBase::m_defaultTreeName = "onnxruntime_session"

inlinestaticinherited

Default name for the TTree to save in.

Definition at line 173 of file VNetworkBase.h.

m_inputFile

std::string VNetworkBase::m_inputFile

protectedinherited

Path to the file describing the network, including filename.

Definition at line 245 of file VNetworkBase.h.

m_inputNodeDims

std::vector<std::vector<int64_t> > TFCSONNXHandler::m_inputNodeDims

private

Do not persistify.

dimension lengths in each named input node

Describes the shape of the input nodes.

See also

TFCSONNXHandler::m_inputNodeNames

Definition at line 276 of file TFCSONNXHandler.h.

m_inputNodeNames

std::vector<const char *> TFCSONNXHandler::m_inputNodeNames

private

names that index the input nodes

An ONNX network is capable of having two layers of labels, input node names, then labels within each node, but it's twin, LWTNN is not. LWTNN supports one list of nodes indexed by strings, and each input node may have more than one value, indexed by positive integers (list like), so this interfae only supports that more limited format.

Definition at line 234 of file TFCSONNXHandler.h.

m_memoryInfo

Ort::MemoryInfo TFCSONNXHandler::m_memoryInfo

private

Initial value:

= Ort::MemoryInfo::CreateCpu(
      OrtArenaAllocator, OrtMemTypeDefault)

Do not persistify.

Specifies memory behavior for vectors in ONNX.

Definition at line 312 of file TFCSONNXHandler.h.

m_nm

std::string ISF_FCS::MLogging::m_nm

privateinherited

Message source name.

Definition at line 211 of file MLogging.h.

m_outputLayers

std::vector<std::string> TFCSONNXHandler::m_outputLayers

private

Do not persistify.

Externally visible names that index the output.

Definition at line 318 of file TFCSONNXHandler.h.

m_outputNodeDims

std::vector<std::vector<int64_t> > TFCSONNXHandler::m_outputNodeDims

private

Do not persistify.

dimension lengths in each named output node

As the final output must be flat in each output node, this is for internal manipulations only.

See also

TFCSONNXHandler::m_inputNodeDims

Definition at line 284 of file TFCSONNXHandler.h.

m_outputNodeNames

std::vector<const char *> TFCSONNXHandler::m_outputNodeNames

private

Do not persistify.

the names that index the output nodes

An ONNX network is capable of having two layers of labels, input node names, then labels within each node, but it's twin, LWTNN is not. LWTNN supports one list of nodes indexed by strings, and each input node may have more than one value, indexed by positive integers (list like), so this interfae only supports that more limited format.

Definition at line 258 of file TFCSONNXHandler.h.

m_outputNodeSize

std::vector<int64_t> TFCSONNXHandler::m_outputNodeSize

private

Do not persistify.

total elements in each named output node

For internal use only, gives the total number of elements in the output nodes.

See also

TFCSONNXHandler::m_inputNodeDims

Definition at line 292 of file TFCSONNXHandler.h.

m_session

std::unique_ptr<Ort::Session> TFCSONNXHandler::m_session

private

The network session itself.

This is the object created by onnxruntime_cxx_api which contains information about the network and can run inputs through it.

Held as a unique pointer to prevent the need for manual memory management

Definition at line 210 of file TFCSONNXHandler.h.

---

The documentation for this class was generated from the following files:

• TFCSONNXHandler.h
• TFCSONNXHandler.cxx