GPU-Accelerated Optimization of Transformer-Based Neural Networks for Real-Time Inference

Soutrik Mukherjee, Sangwhan Cha|March 30, 2026arXiv

Key Takeaway

Hybrid precision (FP32 for softmax/normalization, FP16 for linear layers) delivers 2x speedup with zero accuracy loss—a practical strategy for deploying transformers in latency-critical applications.

Summary

This paper optimizes transformer models (BERT and GPT-2) for fast GPU inference using mixed-precision techniques—keeping sensitive operations in full precision while using lower precision for others. The system achieves 64x speedup over CPU and sub-10ms latency while maintaining numerical accuracy and eliminating instability issues.

efficiency architecture

Key Terms

mixed-precision-training fp16-precision inference-latency bert-model gpt-2-architecture