NVIDIA TensorRT 7’s Compiler Delivers Real-Time Inference for Smarter Human-to-AI Interactions

NVIDIA introduced inference software that developers everywhere can use to deliver conversational AI applications.

NVIDIA TensorRT 7—the seventh generation of the company’s inference software development kit—enables smarter human-to-AI interactions, enabling real-time engagement with applications such as voice agents, chatbots and recommendation engines. TensorRT 7 features a new deep learning compiler designed to automatically optimize and accelerate the complex recurrent and transformer-based neural networks needed for AI speech applications. 

“We have entered a new chapter in AI, where machines are capable of understanding human language in real time,” said NVIDIA founder and CEO Jensen Huang at his GTC China keynote. “TensorRT 7 helps make this possible, providing developers everywhere with the tools to build and deploy faster, smarter conversational AI services that allow more natural human-to-AI interaction.”

Importance of Recurrent Neural Networks

TensorRT 7 speeds up a growing universe of AI models that are being used to make predictions on time-series, sequence-data scenarios that use recurrent loop structures, called RNNs. In addition to being used for conversational AI speech networks, RNNs help with arrival time planning for cars or satellites, prediction of events in electronic medical records, financial asset forecasting and fraud detection.

With TensorRT’s new deep learning compiler, developers everywhere now have the ability to automatically optimize networks—such as bespoke automatic speech recognition networks, and WaveRNN and Tacotron 2 for text-to-speech—and to deliver the best possible performance and lowest latencies. 

The new compiler also optimizes transformer-based models like BERT for natural language processing.

Accelerating Inference from Edge to Cloud

TensorRT 7 can rapidly optimize, validate and deploy a trained neural network for inference by hyperscale data centers, embedded or automotive GPU platforms.

NVIDIA’s inference platform, which includes TensorRT, as well as several NVIDIA CUDA-X AI libraries and NVIDIA GPUs—delivers low-latency, high-throughput inference for applications beyond conversational AI, including image classification, fraud detection, segmentation, object detection and recommendation engines. Its capabilities are widely used by some of the leading enterprise and consumer technology companies, including Alibaba, American Express, Baidu, PayPal, Pinterest, Snap, Tencent and Twitter.

Availability

TensorRT 7 will be available in the coming days for development and deployment, without charge to members of the NVIDIA Developer program from the TensorRT webpage. The latest versions of plug-ins, parsers and samples are also available as open source from the TensorRT GitHub repository.

Sources: Press materials received from the company and additional information gleaned from the company’s website.