AthXRT::DeviceMgmtSvc Class Reference

Service used for programming XRT compatible accelerators. More...

#include <DeviceMgmtSvc.h>

Inheritance diagram for AthXRT::DeviceMgmtSvc:

Collaboration diagram for AthXRT::DeviceMgmtSvc:

Classes
struct	AthClContext
	Struct to hold information about a context, as well as the devices, the program and XCLBIN file associated with the context. More...
struct	SystemInfo
struct	XclbinInfo
	Struct to hold information about an XCLBIN file, as well as the kernels it contains. More...

Public Member Functions
Interface inherited from @c IService
virtual StatusCode	initialize () override
	Initialise the service. More...
virtual StatusCode	finalize () override
	Finalise the service. More...
Interface inherited from @c AthXRT::IDeviceMgmtSvc
virtual const std::vector< IDeviceMgmtSvc::OpenCLHandle >	get_opencl_handles_by_kernel_name (const std::string &name) const override
	Get a list of OpenCL handles providing the specified kernel. More...
virtual const std::vector< std::shared_ptr< xrt::device > >	get_xrt_devices_by_kernel_name (const std::string &name) const override
	Get a list of XRT devices providing the specified kernel. More...

Private Member Functions
std::string	get_device_name (const cl::Device &device) const
	Get the name of a cl::device. More...
std::string	get_device_bdf (const cl::Device &device) const
	Get the BDF (bus:device:function) string of a cl::device. More...
StatusCode	inspect_devices (SystemInfo &si)
	Inspect the available devices and fill the SystemInfo structure. More...
StatusCode	inspect_xclbins (SystemInfo &si)
	Inspect the provided XCLBIN files and fill the SystemInfo structure. More...
StatusCode	pair_devices_and_xclbins (const SystemInfo &si)
	Pair devices and XCLBINs and create contexts. More...
StatusCode	program_devices ()
	Program the devices with the XCLBIN files and create contexts. More...
bool	is_xclbin_compatible_with_device (const DeviceMgmtSvc::XclbinInfo &xclbin_info, const cl::Device &device) const
	Helper function to check if an XCLBIN file is compatible with a device. More...

Private Attributes
std::vector< XclbinInfo >	m_xclbin_infos
	List of XCLBIN files info configured for the service. More...
std::vector< AthClContext >	m_ath_cl_contexts
	List of contexts configured for the service. More...
Service properties
Gaudi::Property< std::vector< std::string > >	m_xclbin_path_list
	The list of xclbin files to use. More...

Detailed Description

Service used for programming XRT compatible accelerators.

This service aims to be a central point for managing XRT compatible accelerators. It will enumerate the available devices, program them with the specified XCLBIN files and provide a way for client algorithms to request lists of devices by kernel name, either using native XRT API or OpenCL API.

Author

Quentin Berthet quent.nosp@m.in.b.nosp@m.erthe.nosp@m.t@ce.nosp@m.rn.ch

Definition at line 31 of file DeviceMgmtSvc.h.

Member Function Documentation

finalize()

StatusCode AthXRT::DeviceMgmtSvc::finalize ( )

overridevirtual

Finalise the service.

Definition at line 470 of file DeviceMgmtSvc.cxx.

                                   {
 
  // Finalise the base class.
  ATH_CHECK(Service::finalize());
 
  // Return gracefully.
  return StatusCode::SUCCESS;
}

get_device_bdf()

std::string AthXRT::DeviceMgmtSvc::get_device_bdf ( const cl::Device &  device ) const

private

Get the BDF (bus:device:function) string of a cl::device.

Parameters

device

The device to get the BDF from.

Returns

The BDF of the device as a string, or "error" in case of failure.

Definition at line 179 of file DeviceMgmtSvc.cxx.

                                                                    {
 
  cl_int err = CL_SUCCESS;
  std::string device_bdf;
  err = device.getInfo(CL_DEVICE_PCIE_BDF, &device_bdf);
  if (err != CL_SUCCESS) {
    ATH_MSG_ERROR("Failed to get device BDF");
    return std::string("error");
  }
  return device_bdf;
}

get_device_name()

std::string AthXRT::DeviceMgmtSvc::get_device_name ( const cl::Device &  device ) const

private

Get the name of a cl::device.

Parameters

device

The device to get the name from.

Returns

The name of the device as a string, or "error" in case of failure.

Definition at line 164 of file DeviceMgmtSvc.cxx.

                                                                     {
 
  cl_int err = CL_SUCCESS;
  std::string device_name;
  err = device.getInfo(CL_DEVICE_NAME, &device_name);
  if (err != CL_SUCCESS) {
    ATH_MSG_ERROR("Failed to get device name");
    return std::string("error");
  }
  return device_name;
}

get_opencl_handles_by_kernel_name()

const std::vector< IDeviceMgmtSvc::OpenCLHandle > AthXRT::DeviceMgmtSvc::get_opencl_handles_by_kernel_name ( const std::string &  name ) const

overridevirtual

Get a list of OpenCL handles providing the specified kernel.

Parameters

name	The name of the kernel to search for.

Returns

A vector of OpenCL handles structs providing the specified kernel.

Definition at line 501 of file DeviceMgmtSvc.cxx.

                                 {
 
  std::vector<IDeviceMgmtSvc::OpenCLHandle> handles;
 
  // Iterate over all contexts and check if the kernel name is in the list.
  // If so, add the context and program to the list of handles to return.
  for (const AthClContext &ath_cl_context : m_ath_cl_contexts) {
    if (std::find(ath_cl_context.xclbin_info.kernel_names.begin(),
                  ath_cl_context.xclbin_info.kernel_names.end(),
                  name) != ath_cl_context.xclbin_info.kernel_names.end()) {
      IDeviceMgmtSvc::OpenCLHandle handle = {ath_cl_context.context,
                                             ath_cl_context.program};
      handles.push_back(handle);
    }
  }
 
  return handles;
}

get_xrt_devices_by_kernel_name()

const std::vector< std::shared_ptr< xrt::device > > AthXRT::DeviceMgmtSvc::get_xrt_devices_by_kernel_name ( const std::string &  name ) const

overridevirtual

Get a list of XRT devices providing the specified kernel.

Parameters

name	The name of the kernel to search for.

Returns

A vector of XRT device handles providing the specified kernel.

Definition at line 480 of file DeviceMgmtSvc.cxx.

                                                                         {
 
  std::vector<std::shared_ptr<xrt::device>> devices;
 
  // Iterate over all contexts and check if the kernel name is in the list.
  // If so, add the device(s) to the list of devices to return.
  for (const AthClContext &ath_cl_context : m_ath_cl_contexts) {
    if (std::find(ath_cl_context.xclbin_info.kernel_names.begin(),
                  ath_cl_context.xclbin_info.kernel_names.end(),
                  name) != ath_cl_context.xclbin_info.kernel_names.end()) {
      for (const cl::Device &device : ath_cl_context.devices) {
        devices.push_back(std::make_shared<xrt::device>(
            xrt::opencl::get_xrt_device(device())));
      }
    }
  }
 
  return devices;
}

initialize()

StatusCode AthXRT::DeviceMgmtSvc::initialize ( )

overridevirtual

Initialise the service.

Initialize the service.

This function will inspect the available devices and XCLBIN files, pair them and create contexts based on the information gathered. It will then program the devices with the XCLBIN files and create contexts.

Returns

StatusCode::SUCCESS if the initialization was successful, or StatusCode::FAILURE otherwise.

Definition at line 446 of file DeviceMgmtSvc.cxx.

                                     {
 
  SystemInfo sys_info;
 
  // Inspect available device(s) and fill sys_info.
  ATH_CHECK(inspect_devices(sys_info));
 
  // Inspect provided XCLBINs to gather information about kernel(s)
  // into sys_info and a list of XclbinInfo.
  ATH_CHECK(inspect_xclbins(sys_info));
 
  // Now we can make a decision about the pairing of device(s)
  // and xclbin(s), and the number of required context(s) by
  // filling m_ath_cl_contexts.
  ATH_CHECK(pair_devices_and_xclbins(sys_info));
 
  // Program the devices with the XCLBINs and create contexts,
  // and programs based on m_ath_cl_contexts.
  ATH_CHECK(program_devices());
 
  // Return gracefully.
  return StatusCode::SUCCESS;
}

inspect_devices()

StatusCode AthXRT::DeviceMgmtSvc::inspect_devices ( SystemInfo &  si )

private

Inspect the available devices and fill the SystemInfo structure.

Find all available XRT compatible accelerator devices and group them by type.

Parameters

si	The SystemInfo structure to fill

Definition at line 18 of file DeviceMgmtSvc.cxx.

                                                        {
 
  std::vector<cl::Platform> platforms;
 
  ATH_CHECK(cl::Platform::get(&platforms) == CL_SUCCESS);
  si.device_count = 0;
  for (const cl::Platform &platform : platforms) {
 
    // Filter platforms by name. Currently AMD FPGA platform is
    // still referenced as "Xilinx".
    std::string platform_name;
    ATH_CHECK(platform.getInfo(CL_PLATFORM_NAME, &platform_name) == CL_SUCCESS);
    if (platform_name != "Xilinx") {
      ATH_MSG_WARNING("Skipping unsuported platform " << platform_name);
      continue;
    }
 
    // Get devices list for the platform and count total devices.
    std::vector<cl::Device> devices;
    ATH_CHECK(platform.getDevices(CL_DEVICE_TYPE_ACCELERATOR, &devices) ==
              CL_SUCCESS);
    ATH_MSG_DEBUG("Found XRT/OpenCL platform '" << platform_name << "' with "
                                                << devices.size()
                                                << " devices");
    si.device_count += devices.size();
 
    // Group devices by type, using the name property.
    // All similar devices should have the same name,
    // I.E: "xilinx_u250_gen3x16_xdma_shell_4_1"
    for (const cl::Device &device : devices) {
      const std::string device_name = get_device_name(device);
 
      // Check if we already have a device with the same name.
      bool found = false;
      for (std::vector<cl::Device> &list : si.device_types) {
        std::string list_device_name;
        ATH_CHECK(list[0].getInfo(CL_DEVICE_NAME, &list_device_name) ==
                  CL_SUCCESS);
        if (device_name == list_device_name) {
          found = true;
          list.push_back(device);
          break;
        }
      }
      if (!found) {
        std::vector<cl::Device> new_list = {device};
        si.device_types.push_back(new_list);
      }
    }
  }
 
  // We expect to have at least one device to program.
  if (si.device_count < 1) {
    // This should not be this catastrophic and we could fallback to
    // software implementation, but for now we will consider that an
    // accelerator have to be present if this service is configured.
    ATH_MSG_ERROR("No XRT device found");
    return StatusCode::FAILURE;
  } else {
    ATH_MSG_INFO("Found a total of "
                 << si.device_count << " AMD FPGA device(s) ("
                 << si.device_types.size() << " device type(s))");
  }
 
  return StatusCode::SUCCESS;
 
}  // DeviceMgmtSvc::inspect_devices()

inspect_xclbins()

StatusCode AthXRT::DeviceMgmtSvc::inspect_xclbins ( SystemInfo &  si )

private

Inspect the provided XCLBIN files and fill the SystemInfo structure.

Gather information using XRT native API about the XCLBIN files and the kernel(s) they contain and perform some basic sanity checks.

Parameters

si	The SystemInfo structure to fill

Definition at line 90 of file DeviceMgmtSvc.cxx.

                                                        {
 
  // We expect at least one XCLBIN file to be specified.
  if (m_xclbin_path_list.empty()) {
    ATH_MSG_ERROR("No XCLBIN list specified");
    return StatusCode::FAILURE;
  }
 
  // If there is more XCLBIN files to program than device(s), this
  // is probably an error (the opposite is ok).
  if (m_xclbin_path_list.size() > si.device_count) {
    ATH_MSG_ERROR(
        "More XCLBIN file(s) specified than "
        "devices type available ("
        << si.device_count << "): ");
    for (const std::string &xclbin_path : m_xclbin_path_list) {
      ATH_MSG_ERROR(xclbin_path);
    }
    return StatusCode::FAILURE;
  }
 
  // Inspect XCLBIN files.
  for (const std::string &xclbin_path : m_xclbin_path_list) {
 
    if (!std::filesystem::exists(xclbin_path)) {
      ATH_MSG_ERROR("XCLBIN file does not exist: " << xclbin_path);
      return StatusCode::FAILURE;
    }
 
    // Create a temporary XRT API XCLBIN object to use introspection
    // to gather some information. With this approach, we are loading
    // them twice from disk which is not optimal.
    DeviceMgmtSvc::XclbinInfo xclbin_info;
    try {
      xrt::xclbin xrt_xclbin(xclbin_path);
      xclbin_info.path = xclbin_path;
      xclbin_info.xsa_name = xrt_xclbin.get_xsa_name();
      xclbin_info.fpga_device_name = xrt_xclbin.get_fpga_device_name();
      xclbin_info.uuid = xrt_xclbin.get_uuid().to_string();
      for (const xrt::xclbin::kernel &kernel : xrt_xclbin.get_kernels()) {
        // Ensure that the kernel have a least one compute unit.
        // Having a kernel without compute unit is not a common use case,
        // but it is possible if a .xo with a kernel is linked in the
        // .xclbin, but the number of said kernel is set to 0.
        if (!kernel.get_cus().empty()) {
          xclbin_info.kernel_names.push_back(kernel.get_name());
        }
      }
    } catch (...) {
      ATH_MSG_ERROR("Could not create xrt::xclbin from " << xclbin_path);
      return StatusCode::FAILURE;
    }
    m_xclbin_infos.push_back(xclbin_info);
  }
 
  // Extract some more information from the XCLBIN collection:
  // The number of different XCLBIN files and the number of different
  // FPGA device names targeted by the XCLBIN files.
  std::set<std::string> uuids;
  std::set<std::string> fpga_device_names;
  for (const XclbinInfo &info : m_xclbin_infos) {
    uuids.insert(info.uuid);
    fpga_device_names.insert(info.fpga_device_name);
  }
  si.different_xclbin_count = uuids.size();
  si.different_xclbin_fpga_device_name = fpga_device_names.size();
 
  return StatusCode::SUCCESS;
 
}  // DeviceMgmtSvc::inspect_xclbin()

is_xclbin_compatible_with_device()

bool AthXRT::DeviceMgmtSvc::is_xclbin_compatible_with_device ( const DeviceMgmtSvc::XclbinInfo &  xclbin_info, const cl::Device &  device  ) const

private

Helper function to check if an XCLBIN file is compatible with a device.

Check if an XCLBIN is compatible with a device.

This is done by comparing the device name and the XSA name used in the XCLBIN file up to the second occurrence of the underscore character. This check is not a guarantee that the XCLBIN will work on the device, but mismatching XSA names are a strong indicator of incompatibility.

Parameters

xclbin_info	The XCLBIN to check compatibility with.
device	The device to check compatibility with.

Returns

True if the XCLBIN is compatible with the device, false otherwise.

Definition at line 226 of file DeviceMgmtSvc.cxx.

                                  {
 
  const std::string device_prefix =
      getPrefixUpToNthOccurrence(get_device_name(device), '_', 2);
  const std::string xsa_prefix =
      getPrefixUpToNthOccurrence(xclbin_info.xsa_name, '_', 2);
  if (device_prefix == xsa_prefix) {
    return true;
  } else {
    return false;
  }
}

pair_devices_and_xclbins()

StatusCode AthXRT::DeviceMgmtSvc::pair_devices_and_xclbins ( const SystemInfo &  si )

private

Pair devices and XCLBINs and create contexts.

This function will pair devices and XCLBINsd epending on the number of devices and XCLBINs, will attempt to program all provided XCLBINs according to the following rules:

• If we have only one FPGA type and one XCLBIN: Program all devices with the same XCLBIN and create one context.
• If we have only one FPGA type and multiple identical XCLBIN: Program the same number of devices that we have XCLBIN files, but put them in only one context as the XCLBIN will be identical for all programmed devices.
• If we have multiple type and multiple different XCLBIN, but all targeting the same device: Program all devices with a different XCLBIN and create one context per device/XCLBIN.
• If we have multiple type and multiple different XCLBIN, and the XCLBIN files target multiple device types: Program each device with a matching XCLBIN and create one context per device/XCLBIN. Some devices might be left un-programmed if no matching XCLBIN is found.

  Parameters

  si	The SystemInfo structure containing information about the devices and XCLBINs available on the system.

  Returns

  StatusCode::SUCCESS if the pairing was successful, or StatusCode::FAILURE otherwise.

Definition at line 261 of file DeviceMgmtSvc.cxx.

                                                                       {
 
  // Do we have only one FPGA type?
  if (si.device_types.size() == 1) {
 
    if (m_xclbin_infos.size() == 1) {
 
      // We have one or multiple device(s) of the same type and only have one
      // XCLBIN: Program all device(s) with the same XCLBIN and create one
      // context.
      ATH_MSG_DEBUG("Case 1: One or multiple identical device(s), one xclbin");
      DeviceMgmtSvc::AthClContext ath_cl_context;
      for (const cl::Device &device : si.device_types[0]) {
        ath_cl_context.devices.push_back(device);
      }
      ath_cl_context.xclbin_info = m_xclbin_infos[0];
      m_ath_cl_contexts.push_back(ath_cl_context);
 
    } else {
 
      if (si.different_xclbin_fpga_device_name > 1) {
 
        // This is an error: we only have one FPGA type but
        // XCLBIN files targeting multiple fpga.
        ATH_MSG_ERROR(
            "Specified XCLBINs target multiple device types, but only one "
            "device type is present");
        return StatusCode::FAILURE;
      }
 
      if (si.different_xclbin_count == 1) {
        // We have multiple device of the same type and multiple identical
        // xclbin: Program the same number of devices that we have XCLBIN files,
        // but put them in only one context as the XCLBIN will be identical for
        // all programmed devices. Some devices might be left un-programmed.
        ATH_MSG_DEBUG(
            "Case 2: Multiple identical devices, multiple identical xclbins");
        DeviceMgmtSvc::AthClContext ath_cl_context;
        for (std::size_t i = 0; i < m_xclbin_infos.size(); ++i) {
          ath_cl_context.devices.push_back(si.device_types[0][i]);
        }
        ath_cl_context.xclbin_info = m_xclbin_infos[0];
        m_ath_cl_contexts.push_back(ath_cl_context);
 
      } else {
 
        // We have multiple device of the same type and multiple different
        // XCLBIN, but all targeting the same device: Program all devices with a
        // differnt XCLBIN and create one context per device/XCLBIN. Some
        // devices might be left un-programmed.
        ATH_MSG_DEBUG(
            "Case 3: Multiple identical devices, multiple different XCLBIN "
            "files, but targeting the same device type");
        for (std::size_t i = 0; i < m_xclbin_infos.size(); ++i) {
          DeviceMgmtSvc::AthClContext ath_cl_context;
          ath_cl_context.xclbin_info = m_xclbin_infos[i];
          ath_cl_context.devices.push_back(si.device_types[0][i]);
          m_ath_cl_contexts.push_back(ath_cl_context);
        }
      }
    }
  } else {
 
    // More tricky (and probably an edge case): we have different
    // devices types. We will try to pair each device them with a
    // XCLBIN files based on the device name and XCLBIN XSA name,
    // and load them in separate contexts.
    ATH_MSG_DEBUG("Case 4: Multiple different devices, multiple xclbins");
    std::vector<XclbinInfo> unaffected_xclbin_infos(m_xclbin_infos);
    for (const std::vector<cl::Device> &device_type : si.device_types) {
      for (const cl::Device &device : device_type) {
        DeviceMgmtSvc::AthClContext ath_cl_context;
        ath_cl_context.devices.push_back(device);
 
        // Try to find a matching XCLBIN for this device.
        std::vector<XclbinInfo>::iterator iter;
        bool found = false;
        for (iter = unaffected_xclbin_infos.begin();
             iter != unaffected_xclbin_infos.end();) {
          if (is_xclbin_compatible_with_device(*iter, device)) {
            ath_cl_context.xclbin_info = *iter;
            iter = unaffected_xclbin_infos.erase(iter);
            found = true;
            break;
          } else {
            ++iter;
          }
        }
 
        // Only keep this combination if we found a matching XCLBIN.
        if (found) {
          m_ath_cl_contexts.push_back(ath_cl_context);
        } else {
          // If we did not find a matching XCLBIN, we will not program the
          // device. This is not an error, but we will report it.
          ATH_MSG_WARNING("No compatible XCLBIN found for device "
                          << get_device_name(device) << " ("
                          << get_device_bdf(device) << ")");
        }
      }
    }
 
    for (const XclbinInfo &xclbin_info : unaffected_xclbin_infos) {
      // Report XCLBIN files that were not affected to a device.
      // (This could happen for XCLBIN not compatible with any device.)
      ATH_MSG_WARNING(
          "No compatible device found for XCLBIN: " << xclbin_info.path);
    }
  }
 
  return StatusCode::SUCCESS;
 
}  // DeviceMgmtSvc::pair_devices_and_xclbins()

program_devices()

StatusCode AthXRT::DeviceMgmtSvc::program_devices ( )

private

Program the devices with the XCLBIN files and create contexts.

This function will program the devices with the XCLBIN files and create contexts based on the information in m_ath_cl_contexts. If an incompatible XCLBIN is found for a device, the initialization will fail.

Returns

StatusCode::SUCCESS if the programming was successful, or StatusCode::FAILURE otherwise.

Definition at line 381 of file DeviceMgmtSvc.cxx.

                                          {
 
  cl_int err = CL_SUCCESS;
 
  for (AthClContext &ath_cl_context : m_ath_cl_contexts) {
 
    // Create an OpenCL context for the device(s).
    ath_cl_context.context = std::make_shared<cl::Context>(
        ath_cl_context.devices, nullptr, nullptr, nullptr, &err);
    if (err != CL_SUCCESS) {
      ATH_MSG_ERROR("Failed to create cl::Context");
      return StatusCode::FAILURE;
    }
 
    // Load XCLBIN file from disk.
    std::ifstream file;
    try {
      file.open(ath_cl_context.xclbin_info.path.c_str(), std::ios::binary);
    } catch (...) {
      ATH_MSG_ERROR("Could not open " << ath_cl_context.xclbin_info.path
                                      << " for reading");
      return StatusCode::FAILURE;
    }
    std::vector<char> xclbin_buffer((std::istreambuf_iterator<char>(file)),
                                    std::istreambuf_iterator<char>());
 
    // Wrap XCLBIN data and size in a vector of cl::Program::Binaries.
    // If we program multiple devices, we need to provide the same
    // binary for each device.
    cl::Program::Binaries binary;
    for (std::size_t i = 0; i < ath_cl_context.devices.size(); ++i) {
      binary.push_back({xclbin_buffer.data(), xclbin_buffer.size()});
    }
 
    // Create a program from the XCLBIN binary.
    // Effectively loading the XCLBIN on the device(s).
    ath_cl_context.program = std::make_shared<cl::Program>(
        *ath_cl_context.context, ath_cl_context.devices, binary, nullptr, &err);
    if (err != CL_SUCCESS) {
      ATH_MSG_ERROR("Failed to create cl::Program");
      return StatusCode::FAILURE;
    }
 
    // Report what have been done.
    std::string bdfs = "";
    for (const cl::Device &device : ath_cl_context.devices) {
      bdfs += get_device_bdf(device);
      bdfs += " ";
    }
    ATH_MSG_INFO("Loaded " << ath_cl_context.xclbin_info.path << " on "
                           << ath_cl_context.devices.size() << " "
                           << get_device_name(ath_cl_context.devices[0])
                           << " device(s): " << bdfs);
  }
 
  return StatusCode::SUCCESS;
 
}  // DeviceMgmtSvc::program_devices

Member Data Documentation

m_ath_cl_contexts

std::vector<AthClContext> AthXRT::DeviceMgmtSvc::m_ath_cl_contexts

private

List of contexts configured for the service.

Definition at line 137 of file DeviceMgmtSvc.h.

m_xclbin_infos

std::vector<XclbinInfo> AthXRT::DeviceMgmtSvc::m_xclbin_infos

private

List of XCLBIN files info configured for the service.

Definition at line 125 of file DeviceMgmtSvc.h.

m_xclbin_path_list

Gaudi::Property<std::vector<std::string> > AthXRT::DeviceMgmtSvc::m_xclbin_path_list

private

Initial value:

{
      this,
      "XclbinPathsList",
      {},
      "The list of XCLBIN files to program on FPGAs"}

The list of xclbin files to use.

This is a list of paths to XCLBIN files to load on accelerator devices.

Definition at line 72 of file DeviceMgmtSvc.h.

---

The documentation for this class was generated from the following files:

• DeviceMgmtSvc.h
• DeviceMgmtSvc.cxx