Interface class for creating Onnx Runtime sessions. More...

#include <IOnnxRuntimeInferenceTool.h>

Inheritance diagram for AthOnnx::IOnnxRuntimeInferenceTool:

Collaboration diagram for AthOnnx::IOnnxRuntimeInferenceTool:

Public Member Functions
virtual void	setBatchSize (int64_t batchSize)=0
	set batch size. More...

virtual int64_t	getBatchSize (int64_t dataSize, int idx=0) const =0
	methods for determining batch size from the data size More...

template<typename T >
StatusCode	addInput (std::vector< Ort::Value > &inputTensors, std::vector< T > &data, unsigned idx=0, int64_t batchSize=-1) const
	add the input data to the input tensors More...

template<typename T >
StatusCode	addOutput (std::vector< Ort::Value > &outputTensors, std::vector< T > &data, unsigned idx=0, int64_t batchSize=-1) const
	add the output data to the output tensors More...

virtual StatusCode	inference (std::vector< Ort::Value > &inputTensors, std::vector< Ort::Value > &outputTensors) const =0
	perform inference More...

virtual void	printModelInfo () const =0

virtual void	print () const =0
	Print the state of the tool. More...

Protected Attributes
unsigned	m_numInputs

unsigned	m_numOutputs

std::vector< std::vector< int64_t > >	m_inputShapes

std::vector< std::vector< int64_t > >	m_outputShapes

Private Member Functions
template<typename T >
Ort::Value	createTensor (std::vector< T > &data, const std::vector< int64_t > &dataShape, int64_t batchSize) const

Detailed Description

Interface class for creating Onnx Runtime sessions.

Interface class for creating Onnx Runtime sessions. It is thread safe, supports models with various number of inputs and outputs, supports models with dynamic batch size, and usess . It defines a standardized procedure to perform Onnx Runtime inference. The procedure is as follows, assuming the tool m_onnxTool is created and initialized:

create input tensors from the input data: ```c++ std::vector<Ort::Value> inputTensors; std::vector<float> inputData_1; // The input data is filled by users, possibly from the event information. int64_t batchSize = m_onnxTool->getBatchSize(inputData_1.size(), 0); // The batch size is determined by the input data size to support dynamic batch size. m_onnxTool->addInput(inputTensors, inputData_1, 0, batchSize); std::vector<int64_t> inputData_2; // Some models may have multiple inputs. Add inputs one by one. int64_t batchSize_2 = m_onnxTool->getBatchSize(inputData_2.size(), 1); m_onnxTool->addInput(inputTensors, inputData_2, 1, batchSize_2); ```
create output tensors: ```c++ std::vector<Ort::Value> outputTensors; std::vector<float> outputData; // The output data will be filled by the onnx session. m_onnxTool->addOutput(outputTensors, outputData, 0, batchSize); ```

perform inference: ```c++ m_onnxTool->inference(inputTensors, outputTensors); ```

Model outputs will be automatically filled to outputData.

Author: Xiangyang Ju xju@c.nosp@m.ern..nosp@m.ch

Definition at line 47 of file IOnnxRuntimeInferenceTool.h.

Member Function Documentation

◆ addInput()

template<typename T >

StatusCode AthOnnx::IOnnxRuntimeInferenceTool::addInput	(	std::vector< Ort::Value > &	inputTensors,
		std::vector< T > &	data,
		unsigned	idx = `0`,
		int64_t	batchSize = `-1`
	)		const

add the input data to the input tensors

Parameters

inputTensors	the input tensor container
data	the input data
idx	the index of the input node
batchSize	the batch size

Returns: StatusCode::SUCCESS if the input data is added successfully

◆ addOutput()

template<typename T >

StatusCode AthOnnx::IOnnxRuntimeInferenceTool::addOutput	(	std::vector< Ort::Value > &	outputTensors,
		std::vector< T > &	data,
		unsigned	idx = `0`,
		int64_t	batchSize = `-1`
	)		const

add the output data to the output tensors

Parameters

outputTensors	the output tensor container
data	the output data
idx	the index of the output node
batchSize	the batch size

Returns: StatusCode::SUCCESS if the output data is added successfully

◆ createTensor()

template<typename T >

Ort::Value AthOnnx::IOnnxRuntimeInferenceTool::createTensor	(	std::vector< T > &	data,
		const std::vector< int64_t > &	dataShape,
		int64_t	batchSize
	)		const

private

◆ getBatchSize()

virtual int64_t AthOnnx::IOnnxRuntimeInferenceTool::getBatchSize	(	int64_t	dataSize,
		int	idx = `0`
	)		const

pure virtual

methods for determining batch size from the data size

Parameters

dataSize	the size of the input data, like std::vector<T>::size()
idx	the index of the input node

Returns: the batch size, which equals to dataSize / size of the rest dimensions.

Implemented in AthOnnx::OnnxRuntimeInferenceTool.

◆ inference()

virtual StatusCode AthOnnx::IOnnxRuntimeInferenceTool::inference	(	std::vector< Ort::Value > &	inputTensors,
		std::vector< Ort::Value > &	outputTensors
	)		const