Cross-Vendor GPU Programming: Extending CUDA Beyond NVIDIA

Cross-platform languages, such as HIP [1], SYCL [15], oneAPI [18], and Kokkos [37] require developers to rewrite applications or frameworks entirely using their specific languages and programming models. Tools like HIPify [10] and SYCLomatic [24] aim to ease this burden by facilitating partial code conversion through source-to-source transformation. However, these tools face significant limitations. First, they cannot fully convert CUDA code, leaving developers to address gaps in compatibility manually. Second, achieving optimal performance across different GPUs and accelerators often entails maintaining separate code versions for each platform, undermining the goal of unified programming. Lastly, neither HIP nor SYCL supports inline PTX assembly, necessitating manual removal or conditional compilation using macros, further complicating code maintenance and portability.

As shown in Table 1, SCALE addresses these limitations by compiling CUDA directly for AMD GPUs without introducing a new programming model. It seamlessly converts inline PTX to LLVM Intermediate Representation (IR) and utilizes the appropriate clang backend to generate accelerator-specific machine code. Additionally, its clang-based compiler supports language extensions, enabling unified programming without requiring code rewrites or maintaining separate codebases.

ML compilers, such as TVM [7], OpenXLA [30], and MAX [23], introduce extension languages, such as TVM's tensor language, which require developers to modify existing machine learning frameworks. This task becomes particularly challenging, similar to unified platform solutions. Additionally, ML compilers rely on graph optimizers that translate high-level operator descriptions into machine code, often leveraging libraries like cuDNN. While these libraries enhance performance, their dependency can limit heterogeneity; the cross-platform capability is compromised if equivalent implementations are unavailable for a specific accelerator. Moreover, ML compilers primarily cater to AI and ML workloads, limiting their applicability in other critical domains where GPUs are essential. For instance, fields such as drug discovery, weather modeling, and computational fluid dynamics often lack the attention afforded to AI and ML despite their reliance on GPU acceleration. This narrow focus restricts the potential of ML compilers to address the full spectrum of GPU-based applications.

SCALE operates directly at the CUDA level, providing seamless support for all types of applications, including both ML and non-ML workloads, as Table 1 shows. Moreover, developing CUDA-X libraries from scratch—one of our key future goals—will enable SCALE to ensure seamless compatibility across all existing and future accelerators without requiring any code modifications.

clang [21, 39] provides robust support for CUDA compilation but depends on a different dialect than nvcc. More specifically, nvcc uses a split compilation model to separate host and device code early in the process. In contrast, clang employs a merged parsing approach where both host and device code must be semantically correct during each compilation step. While this approach enhances clang’s robustness for complex C++ edge cases, it introduces compatibility challenges for CUDA codebases written with nvcc in mind. Many real-world CUDA programs rely on nvcc-specific behaviors and semantics, leading to compilation failures or the need for significant modifications when using clang. This dialect discrepancy also affects frameworks like HIP, which inherit the LLVM-dialect CUDA model, making seamless compatibility with nvcc-based codebases difficult and further complicating cross-platform development. Remove your paper from the ACM proceedings. The paper will appear in our website. SCALE resolves the CUDA dialect issue by offering an nvcc frontend in its clang compiler. The nvcc frontend replicates NVIDIA's behavior for seamless compilation of existing CUDA programs, similar to LLVM's clang-cl for Microsoft Visual C++ compatibility.

Interception approaches, as explored in prior works [19, 32, 40], aim to enable cross-platform compatibility by intercepting CUDA calls and translating them into equivalent implementations for other accelerators. While effective in specific scenarios, these methods face critical limitations. First, these methods translate intercepted CUDA calls to equivalent functions on other accelerators, but if the target accelerator lacks an identical implementation, functionality or performance is degraded, ultimately hindering support for diverse GPU architectures and workloads. Second, these approaches are inherently fragile, relying on proprietary NVIDIA features, such as the undocumented CUDA export table [12, 31, 33], a crucial yet hidden data structure that stores function pointers for CUDA libraries. Due to the lack of official documentation, developers resort to unreliable hacks to replicate its behavior, making solutions prone to failure if NVIDIA modifies or removes this structure. Lastly, they introduce significant security vulnerabilities by employing techniques like LD_PRELOAD to override native CUDA libraries with custom implementations, bypassing standard library loading mechanisms. This practice is highly problematic in security-sensitive environments, as it exposes systems to potential vulnerabilities and unauthorized access.

SCALE overcomes these limitations, as shown in Table 1, by directly re-implementing CUDA's runtime and driver APIs for AMD GPUs using HSA.

SCALE's clang compiler enhances the GPU programming experience while providing robust support for heterogeneous accelerators through the following capabilities.

Inline PTX Assembly. SCALE supports CUDA's inline PTX assembly, enabling developers to incorporate low-level GPU instructions directly into their frameworks without using directly the CUDA API. This support ensures that performance-critical code sections can utilize hardware-specific optimizations while maintaining portability across platforms. SCALE handles several key features of PTX assembly by transforming them to AMD-specific equivalents. A critical aspect of SCALE is its handling of instructions like ballot and shuffle, which depend on GPU warp [3, 25] sizes—64 for some AMD GPUs and 32 for NVIDIA GPUs. SCALE introduces virtual warp size and cudaLaneMask data structure (described in more detail in § 3.4) to handle the warp size of the target hardware. It fully supports the shfl family of PTX instructions, enabling efficient data movement across threads within a warp, and handles PTX instructions that rely on specific integer lengths. Additionally, SCALE resolves inconsistencies in NVIDIA's nvcc handling of PTX asm inputs and outputs by establishing clear, predictable rules. Read-only inputs can be modified within the scope of the asm block, acting as local variables passed by value, while write-only outputs are treated as read-write, eliminating ambiguity between +r and =r constraints. Input or output types are determined by the expression type rather than the constraint letter, ensuring compliance with standard PTX rules for truncation, widening, or type-checking.

Compiler Warnings. By default, SCALE adopts clang’s stricter warning behavior, which is more rigorous than nvcc, meaning projects using -Werror may fail to compile unless the flag is adjusted or underlying code issues are resolved. Additionally, SCALE supports all standard clang++ flags alongside nvcc options, except where conflicts arise. The SCALE implementation of the CUDA runtime and driver APIs also applies [[nodiscard]] to error return codes, ensuring warnings for ignored CUDA API errors, which can be suppressed using -Wno-unused-result.

Language Extensions. SCALE enhances the CUDA programming model with optional language extensions that provide additional functionality and flexibility, empowering developers to write more expressive and efficient code while addressing the limitations of native CUDA. One notable extension is clang::loop_unroll, which provides explicit control over loop unrolling during compilation. This allows developers to optimize loops for performance-critical sections of code by specifying unrolling behavior directly, ensuring better utilization of GPU resources and improved runtime efficiency. Another feature is __builtin_provable(bool X), which serves as a powerful tool for static analysis and optimization. This build-in function enables code to opportunistically optimize for specific cases without incurring runtime branching overhead or requiring the propagation of such information throughout the program via templates.

CUDA Runtime and Driver APIs. SCALE re-implements the majority of CUDA Runtime and Driver APIs using Heterogeneous System Architecture (HSA) [38]. This API includes a class for managing device memory, streams, events, and synchronization, allowing developers to retain the familiar CUDA programming model while achieving compatibility with AMD GPUs.

CUDA-X libraries APIs. SCALE provides partial support for CUDA-X libraries like cuBLAS and cuSOLVER by mapping their functionality to AMD's ROCm equivalents (e.g., rocBLAS). While the current implementation leverages existing ROCm libraries, we plan to develop its own high-performance versions of these libraries to improve portability and extend support to other accelerators. This approach ensures that frameworks using popular CUDA libraries can transition to AMD GPUs with minimal effort.

To overcome the challenges of the CUDA dialect problem, SCALE introduces an nvcc mode within its clang compiler, referred to as a second compiler for simplicity. The nvcc-compatible clang frontend replicates the behavior of NVIDIA's nvcc compiler. This frontend ensures that CUDA programs written with nvcc-specific semantics and behaviors can be compiled directly without modification. By emulating nvcc’s behavior, this compiler handles features like inline PTX assembly, NVIDIA-specific pragmas, and intricate details of nvcc’s compilation pipeline, such as its handling of preprocessor directives and device-specific functions.

GPUs from different vendors have major architecture differences including warp [25] (or wavefront [3]) size and compute capability. Warp and wavefront sizes define how threads execute in parallel on different GPU architectures, creating a fundamental challenge for cross-platform execution. NVIDIA GPUs use warps of 32 threads, while some AMD GPUs rely on wavefronts of 64 threads. This difference affects thread scheduling, synchronization, and divergence handling, making it difficult to execute the same code efficiently on both architectures. Many CUDA applications are optimized for NVIDIA's 32-thread warps, assuming specific behaviors for operations like ballot and shuffle, which may not directly translate to AMD's 64-thread wavefronts. As a result, without proper adaptation, cross-platform execution can lead to incorrect results, inefficient thread utilization, or performance degradation. SCALE addresses this challenge using two techniques: (1) warp32 emulation, which logically splits a 64-lane wavefront into two 32-lane warps, preserving expected CUDA behavior for intra-warp operations such as shuffles. By leveraging (2) cudaLaneMask_t to extend warp-level operations to 64 lanes while allowing the compiler to issue warnings for patterns that assume a 32-lane warp.

NVIDIA's compute capability system allows developers to leverage hardware-specific features by enabling or disabling functionality through preprocessor directives. However, AMD GPUs use architecture identifiers like gfx1030, which fundamentally differ from NVIDIA's compute capability numbering scheme. Numeric comparisons on compute capability values, often used in CUDA projects, would malfunction if AMD identifiers were substituted directly, resulting in errors and incompatibilities. To bridge this gap, SCALE provides a separate CUDA installation directory for each AMD GPU target, mapping NVIDIA compute capability identifiers like sm_86 to the corresponding AMD architecture, such as gfx1030. This mapping ensures compatibility with existing build systems by allowing them to function as if targeting NVIDIA GPUs. The default mapping uses sm_86 to maximize compatibility with modern CUDA projects. By default, SCALE assigns AMD GPUs a compute capability number of at least sm_600000, ensuring backward compatibility while maintaining scalability for future architectures. However, developers are allowed to define different mappings.

SCALE has been rigorously tested on a broad set of open-source CUDA projects, including AMGX, FLAMEGPU2, ALIEN, GOMC, GPU JPEG2K, XGBoost, and Fais that do not support AMD GPU natively. Table 2 presents the execution time, the availability of ROCm support for running on AMD GPUs, and the application names used from the framework. The parentheses in the App. Name column indicates the specific steps or image numbers used in the evaluation. ALIEN and GPUJPEG do not provide standalone applications, so we ran their unit tests to ensure compatibility with SCALE. It is important to note that, for all frameworks, we executed both their unit tests and multiple applications to validate their functionality. However, we report only one representative application per framework due to space constraints.

SCALE supports state-of-the-art AMD GPU microarchitectures, including gfx900 (Vega 10, GCN 5.0), gfx1030 (Navi 21, RDNA 2.0), gfx1100 (Navi 31, RDNA 3.0), and the datacenter grade gfx942 (MI300X, CDNA3).

SCALE currently supports 47% of the CUDA 12.6 runtime API, 22% of the driver API, and 80% of the math API. While these percentages may seem modest, they can run thirteen real-world CUDA applications across multiple AMD GPU architectures. This success stems from the observation that, despite the extensive size of the CUDA API, only a small subset is commonly utilized in practice.

Figure 2 compares the execution time of CUDA applications compiled using our approach against their implementation using ROCm compiled with hipcc. This time includes all GPU operations: allocations, copies, kernel executions, and data frees. To better interpret our results, we analyze the execution time breakdown and find that SCALE generates highly optimized code, providing performance comparable to and, in some cases, surpassing native ROCm. These results highlight the effectiveness of our compiler in decreasing multiple code base issues without compromising performance. We also evaluated the execution of hashcat and LLaMA C++ which support AMD GPUs via ROCm. We observed that SCALE is not as performant as ROCm in these applications and are actively investigating the root cause. For Thrust, the ROCm implementation (rocThrust [2]) lacks the same benchmarks as the CUDA version, and due to time constraints, we were unable to adapt them for direct comparison. Lastly, for stdgpu, ROCm support remains experimental and fails to compile.