Friendli Inference

Friendli Inference: The Fastest LLM Inference Engine

What is Friendli Inference?

Friendli Inference is a highly optimized engine designed to accelerate the serving of Large Language Models (LLMs), significantly reducing costs by 50-90%. It stands out as the fastest LLM inference engine on the market, outperforming vLLM and TensorRT-LLM in performance testing.

How does Friendli Inference work?

Friendli Inference achieves its remarkable performance through several key technologies:

• Iteration Batching: This innovative batching technology efficiently handles concurrent generation requests, achieving up to tens of times higher LLM inference throughput compared to conventional batching while maintaining the same latency requirements. It is protected by patents in the US, Korea, and China.
• DNN Library: Friendli DNN Library comprises a set of optimized GPU kernels specifically designed for generative AI. This library enables faster LLM inference for various tensor shapes and data types, supports quantization, Mixture of Experts (MoE), and LoRA adapters.
• Friendli TCache: This intelligent caching system identifies and stores frequently used computational results, reducing the workload on GPUs by leveraging the cached results.
• Speculative Decoding: Friendli Inference natively supports speculative decoding, an optimization technique that speeds up LLM/LMM inference by making educated guesses on future tokens in parallel while generating the current token. This ensures identical model outputs at a fraction of the inference time.

Key Features and Benefits

• Significant Cost Savings: Reduce LLM serving costs by 50-90%.
• Multi-LoRA Serving: Simultaneously supports multiple LoRA models on fewer GPUs, even a single GPU.
• Wide Model Support: Supports a wide range of generative AI models, including quantized models and MoE.
• Groundbreaking Performance:
  • Up to 6x fewer GPUs required.
  • Up to 10.7x higher throughput.
  • Up to 6.2x lower latency.

Highlights

• Running Quantized Mixtral 8x7B on a Single GPU: Friendli Inference can run a quantized Mixtral-7x8B-instruct v0.1 model on a single NVIDIA A100 80GB GPU, achieving at least 4.1x faster response time and 3.8x ~ 23.8x higher token throughput compared to a baseline vLLM system.
• Quantized Llama 2 70B on Single GPU: Seamlessly run AWQ-ed LLMs, such as Llama 2 70B 4-bit, on a single A100 80 GB GPU, enabling efficient LLM deployment and remarkable efficiency gains without sacrificing accuracy.
• Even Faster TTFT with Friendli TCache: Friendli TCache optimizes Time to First Token (TTFT) by reusing recurring computations, delivering 11.3x to 23x faster TTFT compared to vLLM.

How to Use Friendli Inference

Friendli Inference offers three ways to run generative AI models:

1. Friendli Dedicated Endpoints: Build and run generative AI models on autopilot.
2. Friendli Container: Serve LLM and LMM inferences with Friendli Inference in your private environment.
3. Friendli Serverless Endpoints: Call the fast and affordable API for open-source generative AI models.

Why choose Friendli Inference?

Friendli Inference is the ideal solution for organizations looking to optimize the performance and cost-effectiveness of their LLM inference workloads. Its innovative technologies and wide range of features make it a powerful tool for deploying and scaling generative AI models.

Who is Friendli Inference for?

Friendli Inference is suitable for:

• Businesses deploying large language models.
• Researchers working with generative AI.
• Developers building AI-powered applications.

Best way to optimize LLM inference?

The best way to optimize LLM inference is to use Friendli Inference, which offers significant cost savings, high throughput, and low latency compared to other solutions.