uMaHF0G5M1jYL9t88qHEEkQggU6GJ5wTZlhvItt7
Bookmark

CertiK and Columbia University Introduce TensorLean, Discover 25 Bugs in AI GPU Kernels

CertiK and Columbia University developed TensorLean, a verification framework that uncovered 25 bugs in production AI GPU kernels.

CertiK and Columbia University introduce TensorLean, a framework identifying computational bugs in production artificial intelligence GPU kernels.

Researchers from CertiK and Columbia University have developed TensorLean, a formal verification framework that uncovered 25 software bugs in production artificial intelligence GPU kernels. The findings highlight computational errors in systems used for AI training and inference, raising concerns about the reliability of highly optimized GPU software.

According to Announced on November 20, TensorLean applies mathematical proofs to verify whether GPU kernels perform calculations correctly. The researchers describe the project as the first formal verification of production AI kernels, targeting errors that conventional software testing methods may fail to identify.

TensorLean Identifies 25 Bugs in DeepSeek and FlashAttention

The research team examined more than 100 production GPU kernels running across several generations of graphics processing hardware. The investigation identified two bugs in DeepSeek kernels and another 23 in FlashAttention implementations.

FlashAttention is designed to accelerate calculations within AI models, while DeepSeek relies on optimized computational systems to handle complex workloads. Errors in these operations can compromise calculation accuracy even when the affected software continues running without an obvious failure.

TensorLean analyzes compiled Parallel Thread Execution (PTX) instructions to assess whether GPU operations satisfy predefined mathematical requirements. It also incorporates the Lean proof assistant to produce independently verifiable evidence that computations meet their specified correctness conditions.

CertiK co-founder Ronghui Gu emphasized that finding a software bug is only part of the verification process. Developers must also demonstrate that a corrected program satisfies its original requirements under the specified conditions.

AI Training Errors Could Create Significant Financial Costs

The research also highlights the potential financial consequences of undetected computational errors during large-scale AI training. When incorrect calculations propagate through subsequent operations, development teams may need to repeat portions of an expensive training process.

For a training operation costing $100 million, repeating just 1% of the work would result in an additional $1 million expense. The example illustrates how relatively small computational inaccuracies can translate into substantial costs for organizations operating large AI systems.

TensorLean focuses on verification challenges associated with GPU optimization techniques, including reduced-precision arithmetic, overlapping data transfers and variations in execution configurations. Although these approaches can improve processing efficiency, they may also introduce computational errors that conventional testing struggles to detect.

Formal Verification Designed for Optimized GPU Software

The framework allows developers to conduct verification again after modifying GPU kernels, helping determine whether performance improvements have affected computational correctness. It also supports reusable mathematical specifications, which can simplify verification across related computational operations.

Researchers aim to incorporate these checks into software development workflows, allowing engineering teams to evaluate computational accuracy alongside performance optimization. TensorLean's machine-checkable mathematical proofs are intended to provide additional evidence that optimized GPU programs satisfy their required specifications.

The project addresses a growing engineering challenge in AI infrastructure: improving processing performance without compromising the accuracy of underlying computations. By applying formal verification to production GPU kernels, the researchers seek to make correctness checks a more integrated part of large-scale AI software development.


Writer: Emily Carter

Crypto & Financial Markets Journalist

Emily Carter reports on cryptocurrency markets, digital asset developments, blockchain innovation, and the global forces shaping the future of finance. Her coverage focuses on market movements, institutional developments, regulatory shifts, and the evolving digital economy.


Check out other news and articles on Google News

Disclaimer:


The articles published on hoka.news are intended to provide up-to-date information on various topics, including cryptocurrency and technology news. The content on our site is not intended as an invitation to buy, sell, or invest in any assets. We encourage readers to conduct their own research and evaluation before making any investment or financial decisions.
hoka.news is not responsible for any losses or damages that may arise from the use of information provided on this site. Investment decisions should be based on thorough research and advice from qualified financial advisors. Information on hoka.news may change without notice, and we do not guarantee the accuracy or completeness of the content published.