0.860 ms
GPU inference performance
GPU AI Performance Lab
Reproducible inference lab
A hands-on inference optimization lab benchmarking PyTorch, ONNX Runtime, TensorRT, CUDA, and batch-scaling behavior on an NVIDIA RTX 3060 Laptop GPU.
Role and scope
Built benchmark scripts, ONNX export and validation flows, runtime comparisons, TensorRT Execution Provider experiments, report generation, and CUDA finance benchmarking.
PythonPyTorchONNX RuntimeTensorRTCUDAC++WSL2ResNet18YOLOv8n
3335 samples/s
Batch-16 throughput
+131.6%
TensorRT lift
7.25x
CUDA finance speedup
Case study
What this shows
The GPU AI Performance Lab compares inference and CUDA performance paths across real local GPU tooling and documents latency, throughput, validation, and backend tradeoffs.
Problem
AI applications need a practical understanding of latency, throughput, precision, export validation, and backend differences before models can be trusted in deployment workflows.
Environment
Benchmarks were run with an NVIDIA RTX 3060 Laptop GPU with 6 GB VRAM, WSL2 Ubuntu, CUDA, PyTorch, ONNX Runtime CUDA Execution Provider, and TensorRT Execution Provider paths.
ResNet18 inference
The lab exports ResNet18 to ONNX with dynamic batch support, validates ONNX output against PyTorch, and benchmarks FP32/FP16 paths across batch sizes.
Model-output validation
Output validation is part of the workflow, with maximum absolute differences treated as expected small CUDA numerical differences rather than ignored.
TensorRT execution
TensorRT EP FP16 improved ResNet18 batch-16 throughput by 131.6% versus PyTorch FP32 and improved batch-1 p50 latency versus ONNX Runtime CUDA FP32.
CUDA finance module
The repository also includes a C++/CUDA Monte Carlo option-pricing benchmark, reaching 7.25x speedup versus CPU at 10M simulated paths while documenting overhead at smaller workloads.
Interested in the implementation?