Realtime API
A real-time interaction interface provided by OpenAI, built on a bidirectional streaming channel over persistent WebSocket connections, supporting low-latency scenarios such as voice conversation and real-time transcription.
Timeline (last 90 days)
v3.36.0 concentrated on fixing multiple boundary issues related to Realtime WebSocket, including cleanup of failed connection establishment, respect for native options, ending sync WebSocket recovery on interruption, respecting Translation receipt with timeout during terminal cleanup, preserving WebSocket close state across recovery and cancellation scenarios, and respecting native options for the legacy beta socket (PR #4121, #4129, #4132, #4117, #4120).
The article suggests: if a project is calling the Responses API via Vertex AI, using Realtime WebSocket and encountering issues such as lost close state after connection recovery or connections not being properly released after cancellation, or still relying on Pydantic v1 for request validation, it should upgrade to v3.36.0 as soon as possible.
Projects using the beta Realtime feature and running on websockets 13 should upgrade to v3.35.1 first.
v3.35.1 fixes the byte reading issue of the beta Realtime feature under websockets 13, corresponding to PR #4102, with commit hash 5766349.
Before upgrading, it is recommended to verify message sending and receiving on Realtime connections in a test environment, confirm the byte reading path works properly, and then push to production.
Developers using realtime audio streams or binary frames are advised to upgrade to v3.35.0 and run regression tests.
v3.35.0 added typed Live sideband events to the Realtime API (PR #4099).
A sideband connection is an event channel in the Realtime API that is independent of the main WebSocket session and is used to send outbound control-type events such as transcription text and tool-call status.
The modalities parameter of the Realtime API is used to declare the input/output modality types supported by the current session, such as text and audio.
v3.19.2 clarified the definition of the modality array in the Realtime API (#3954).
2 more timeline events
All Facts (14)
GPT-Realtime-2支持function calling（函数调用），允许模型在听到用户说话的同时决定调用哪个函数，无需等待完整的文本转录
85%UnverifiedA sideband connection is an event channel in the Realtime API that is independent of the main WebSocket session and is used to send outbound control-type events such as transcription text and tool-call status.
70%UnverifiedThe Realtime API launched by OpenAI at the end of 2024 supports WebRTC as a transport layer, with latency typically under 200ms.
70%UnverifiedThe article suggests: if a project is calling the Responses API via Vertex AI, using Realtime WebSocket and encountering issues such as lost close state after connection recovery or connections not being properly released after cancellation, or still relying on Pydantic v1 for request validation, it should upgrade to v3.36.0 as soon as possible.
50%Unverifiedv3.36.0 concentrated on fixing multiple boundary issues related to Realtime WebSocket, including cleanup of failed connection establishment, respect for native options, ending sync WebSocket recovery on interruption, respecting Translation receipt with timeout during terminal cleanup, preserving WebSocket close state across recovery and cancellation scenarios, and respecting native options for the legacy beta socket (PR #4121, #4129, #4132, #4117, #4120).
50%Unverifiedv3.35.1 fixes the byte reading issue of the beta Realtime feature under websockets 13, corresponding to PR #4102, with commit hash 5766349.
50%UnverifiedBefore upgrading, it is recommended to verify message sending and receiving on Realtime connections in a test environment, confirm the byte reading path works properly, and then push to production.
50%UnverifiedProjects using the beta Realtime feature and running on websockets 13 should upgrade to v3.35.1 first.
50%UnverifiedDevelopers using realtime audio streams or binary frames are advised to upgrade to v3.35.0 and run regression tests.
50%UnverifiedThe modalities parameter of the Realtime API is used to declare the input/output modality types supported by the current session, such as text and audio.
50%Unverifiedv3.19.2 clarified the definition of the modality array in the Realtime API (#3954).
50%UnverifiedOpenAI's Realtime API, ElevenLabs' low-latency mode, and Google's Live API all emphasize low-latency voice capabilities.
50%Unverifiedv3.35.0 added typed Live sideband events to the Realtime API (PR #4099).
50%UnverifiedThe 2024 DevDay introduced capabilities including the Realtime API and Vision Fine-tuning.
50%