The section we describe the SYCL explicit scaling language API and provide usage examples on Arctic Sound-based platform for multi-GPU and multi-stack execution.

Before you run an application, it is recommended to run the sycl-ls command to find out which devices are available on this platform, especially when the run is for a performance measure. This ensures the run is not taking a fallback path.

Intel GPUs are represented as SYCL GPU devices, and as root-devices. The discovery of root-devices is best with the sycl-ls tool, for example:

Note that sycl-ls shows devices from all platforms of all SYCL backends that are seen by SYCL runtime. Thus in the example above, there are GPUs corresponding to the single physical GPU (managed by either OpenCL or Level-Zero backend). root-devices.

One can use the environment variable ONEAPI_DEVICE_SELECTOR described in EnvironmentVariables.md

If there are multiple GPUs in a system then they will be seen as multiple different root-devices. On Linux these would be multiple SYCL root-devices of the same SYCL platform (representing Level-Zero driver). On Windows* these would appear as root-devices of multiple different SYCL platforms (Level-Zero drivers).

CreateMultipleRootDevices=N NEOReadDebugKeys=1 environment variables can be used to emulate multiple GPU cards. For example:

Intel® Data Center GPU Max 1350 or 1550 has 2 stacks. The root-device, corresponding to the whole GPU, can be partitioned to 2 sub-devices, each sub-device corresponding to a physical stack.

Each call to create_sub_devices will return exactly the same sub-devices and in the persistent order. To control what sub-devices are exposed by Level-Zero UMD one can use ZE_AFFINITY_MASK environment variable. Note that the partition_by_affinity_domain is the only partitioning supported for Intel GPUs.

Similarly, next_partitionable and numa are the only partitioning properties supported (both doing the same thing). CreateMultipleRootDevices=N NEOReadDebugKeys=1 environment variables can be used to emulate multiple stacks of a GPU.

Each sub-device (stack) can be further decomposed to a set of sub-sub-devices (Compute Slice). One can create a context associating with a sub-sub-device. In this scheme, the execution resource will be limited to the sub-sub-device, giving the program fine-grained control at compute slice level. The following code finds all sub-devices and sub-sub-devices of a device: