Ollama v0.40.5 Pre-release: MLX Metal Kernels Ported to CUDA, OpenAI Compatibility Upgraded

Ollama's latest pre-release version v0.40.5 is now available. Although still in the Release Candidate (RC) stage, it brings several changes worth the attention of local LLM enthusiasts. From CUDA kernel porting to compatibility fixes for the OpenAI Responses API, this release signals Ollama's continued investment in cross-platform inference and ecosystem integration.

MLX's Metal Custom Kernels Ported to CUDA

The most technically significant change in this update is PR #18657, submitted by developer @dhiltgen: porting custom kernels originally written for Apple Metal in the MLX framework to the CUDA platform.

MLX is a machine learning framework introduced by Apple, optimized specifically for Apple Silicon's unified memory architecture. Its Metal kernels fully leverage the GPU characteristics of M-series chips. Porting this set of kernels to CUDA means the same computational logic can now run on NVIDIA GPUs, opening up new optimization space for Ollama's inference performance on non-Apple hardware.

This is not a simple API translation. Metal and CUDA differ significantly in memory management, thread scheduling, and instruction sets. Kernel-level porting requires re-adapting the underlying computational patterns. @dhiltgen also submitted PR #18993 to bump the MLX version number, indicating the team is maintaining both technical tracks in parallel.

For users running Ollama on NVIDIA GPUs, the actual benefit of this change depends on whether these kernels are integrated into the default inference path in future releases. But directionally, Ollama is narrowing the performance gap between different hardware platforms.

A bit of background on "custom kernels": In deep learning inference, a "kernel" refers to a parallel computing program executed on the GPU, typically responsible for operators such as matrix multiplication, attention computation, and normalization. General-purpose kernels provided by deep learning frameworks often sacrifice performance for compatibility, while "custom kernels" are hand-optimized versions tailored to specific hardware microarchitectures, able to fully utilize that hardware's vectorized instructions, shared memory, and warp scheduling capabilities to achieve higher throughput at the same compute power. The custom kernels MLX wrote for Apple Metal were hand-tuned specifically for the unified memory and GPU characteristics of M-series chips. After porting to CUDA, these operators need to be reimplemented on NVIDIA's SM architecture, memory hierarchy, and warp scheduling model—a typical "cross-backend kernel rewrite" whose value lies in bringing proven high-efficiency algorithms from the Apple ecosystem into the NVIDIA inference path.

OpenAI Compatibility Layer: Accepting Custom Tool Calls in Responses History

PR #18911 fixes a detail issue in ecosystem integration: the OpenAI-compatible interface can now accept custom tool calls in Responses history records.

Ollama has always positioned itself as an OpenAI API alternative for local models, with a large number of third-party tools and applications communicating with Ollama through OpenAI-format interfaces. When these applications use custom tools (rather than OpenAI's predefined tool types), the previous compatibility layer could encounter parsing issues when processing historical messages. This fix makes tool call passing smoother in multi-turn conversations.

For developers building Agents or complex toolchains, this change reduces debugging costs caused by interface incompatibilities.

About the OpenAI Responses API and tool calls: OpenAI's Responses API is a conversation interface designed for agent scenarios. Unlike the traditional Chat Completions interface, it models model output as a series of "items," including messages, function calls, function returns, etc., thereby naturally supporting multi-turn, multi-step tool call loops. Tool calls fall into two categories: one is OpenAI's built-in predefined tools (such as file retrieval and code interpreter), and the other is developer-defined function tools (custom tool calls), where the model generates structured parameters, the client executes them, and results are returned. For a compatibility layer to correctly reproduce this mechanism, it must fully preserve and parse the input-output structure of custom tool calls in history records; otherwise, multi-turn Agent workflows will break after the second round. Ollama fixing this essentially enables local models to more reliably serve as backends for OpenAI Agent frameworks.

Documentation, Payment Error Messages, and Version Maintenance

The remaining changes lean toward engineering maintenance. PR #18981 standardized the language in the model recommendation documentation; PR #18986 improved the cloud payment error message, submitted by new contributor @willietran—his first contribution to the Ollama project.

Overall, v0.40.5 is not a major feature release for ordinary users, but rather an iteration focused on low-level optimization and compatibility patches. For developers concerned about local inference performance, the follow-up progress of the CUDA kernel porting is worth tracking.

About RC (Release Candidate) and pre-release versions: Ollama v0.40.5 is a pre-release version in the Release Candidate (RC) stage. Such versions have completed major feature development and entered a validation period of "feature freeze, concentrated bug fixing." Their version numbers are usually the same as the official release, but with an rc or pre-release tag attached. The purpose of an RC is to let the community validate ahead of time, so that regression issues discovered can still be fixed before the official release. Therefore, stability is usually higher than nightly builds but lower than official releases. For production environments, RC versions may have minor interface behavior adjustments or dependency conflicts; for developers wanting to try out new kernels and new compatibility fixes, they can be used to assess the direction of upcoming official releases in advance. Ollama placing low-level changes like the CUDA kernel porting in an RC also fits its "validate first, then default" iteration rhythm.

Frequently Asked Questions

Can this version be used for daily use?

It is currently marked as Pre-release. It is recommended to wait for the official release before using it in production environments.

Will the CUDA kernel porting immediately improve inference speed?

Not necessarily. Kernel porting is foundational work. The actual performance improvement depends on when these kernels are integrated into the default inference pipeline.

Which scenarios are affected by the OpenAI compatibility fix?

It mainly affects third-party applications that use custom tool calls, especially Agent-type tools that require multi-turn tool calls.

One-line Summary

Ollama v0.40.5 pre-release ports MLX Metal kernels to CUDA and fixes the OpenAI compatibility layer.

Paragraph Summary

Ollama v0.40.5 pre-release is now available. Although still in the RC stage, it brings several low-level changes. The most technically significant is porting custom kernels written by the MLX framework for Apple Metal to CUDA, enabling the same computational logic to run on NVIDIA GPUs and opening space for cross-platform inference performance optimization. On the compatibility side, the OpenAI-compatible interface fixes the issue of accepting custom tool calls in Responses history, improving the integration experience for Agent-type third-party applications. It also includes engineering maintenance such as language standardization for model recommendation documentation and optimization of cloud payment error messages, with new contributor @willietran participating for the first time. Overall, this is not a major feature release for ordinary users, but primarily low-level optimization and compatibility patches. The follow-up integration progress of the CUDA kernels is worth watching.