CertiK and Columbia University Introduce TensorLean, Discover 25 Bugs in AI GPU Kernels
Researchers from CertiK and Columbia University have developed TensorLean, a formal verification framework that uncovered 25 software bugs in production artificial intelligence GPU kernels. The findings highlight computational errors in systems used for AI training and inference, raising concerns about the reliability of highly optimized GPU software.
According to Announced on November 20, TensorLean applies mathematical proofs to verify whether GPU kernels perform calculations correctly. The researchers describe the project as the first formal verification of production AI kernels, targeting errors that conventional software testing methods may fail to identify.
TensorLean Identifies 25 Bugs in DeepSeek and FlashAttention
The research team examined more than 100 production GPU kernels running across several generations of graphics processing hardware. The investigation identified two bugs in DeepSeek kernels and another 23 in FlashAttention implementations.
FlashAttention is designed to accelerate calculations within AI models, while DeepSeek relies on optimized computational systems to handle complex workloads. Errors in these operations can compromise calculation accuracy even when the affected software continues running without an obvious failure.
TensorLean analyzes compiled Parallel Thread Execution (PTX) instructions to assess whether GPU operations satisfy predefined mathematical requirements. It also incorporates the Lean proof assistant to produce independently verifiable evidence that computations meet their specified correctness conditions.
CertiK co-founder Ronghui Gu emphasized that finding a software bug is only part of the verification process. Developers must also demonstrate that a corrected program satisfies its original requirements under the specified conditions.
AI Training Errors Could Create Significant Financial Costs
The research also highlights the potential financial consequences of undetected computational errors during large-scale AI training. When incorrect calculations propagate through subsequent operations, development teams may need to repeat portions of an expensive training process.
For a training operation costing $100 million, repeating just 1% of the work would result in an additional $1 million expense. The example illustrates how relatively small computational inaccuracies can translate into substantial costs for organizations operating large AI systems.
TensorLean focuses on verification challenges associated with GPU optimization techniques, including reduced-precision arithmetic, overlapping data transfers and variations in execution configurations. Although these approaches can improve processing efficiency, they may also introduce computational errors that conventional testing struggles to detect.
Formal Verification Designed for Optimized GPU Software
The framework allows developers to conduct verification again after modifying GPU kernels, helping determine whether performance improvements have affected computational correctness. It also supports reusable mathematical specifications, which can simplify verification across related computational operations.
Researchers aim to incorporate these checks into software development workflows, allowing engineering teams to evaluate computational accuracy alongside performance optimization. TensorLean's machine-checkable mathematical proofs are intended to provide additional evidence that optimized GPU programs satisfy their required specifications.
The project addresses a growing engineering challenge in AI infrastructure: improving processing performance without compromising the accuracy of underlying computations. By applying formal verification to production GPU kernels, the researchers seek to make correctness checks a more integrated part of large-scale AI software development.
Writer: Emily Carter
Crypto & Financial Markets Journalist
Emily Carter reports on cryptocurrency markets, digital asset developments, blockchain innovation, and the global forces shaping the future of finance. Her coverage focuses on market movements, institutional developments, regulatory shifts, and the evolving digital economy.
