Performance of GPT-2 across System Programming Languages

GPT-2, developed by OpenAI, represents a seminal advancement in large language models (LLMs). Initially implemented in C by Andrej Karpathy as the open-source llm.c project1, this foundational work has subsequently inspired people porting it to various system programming languages, including C++2, Rust3, and Mojo4. This paper presents a comparative analysis of these implementations, using the llm.c project as a performance benchmark while acknowledging the potential impact of CPU-side optimizations. To ensure methodological rigor, identical input and output conditions specified by llm.c were maintained across all implementations, with performance evaluations conducted in a standardized CPU server environment. Given the shared LLVM back-end among the programming languages under examination, we accepted all optimization techniques when benchmarking the various language implementations, permitting the utilization of high-performance libraries native to each ecosystem. Our findings reveal that Rust, despite lacking native OpenMP support, achieved performance comparable to C when augmented with the Rayon parallelism library along with CBLAS5, which relied on Intel’s oneAPI Math Kernel Library (oneMKL)6. Conversely, Mojo’s auto-vectorization capabilities did not demonstrably outperform C in this specific context. When high-performance libraries were incorporated, Rust’s performance potential remained constrained by lack of performance-orientated libraries such as high-performance matrix multiplication or concepts such as expression templates, which proved to be one of the main driving factors of C++ performance with its use of the Eigen library7. This discrepancy suggests that implementing concepts from Eigen’s programming model in Rust could significantly enhance its performance capabilities for matrix-intensive computations. These results contribute valuable insights to the ongoing discourse on language selection for LLM implementation, highlighting both the current performance landscape and potential avenues for future optimization efforts across different programming ecosystems.