Class RealtimeClientEvent

    • Constructor Detail

    • Method Detail

      • conversationItemCreate

         final Optional<ConversationItemCreateEvent> conversationItemCreate()

        Add a new Item to the Conversation's context, including messages, function calls, and function call responses. This event can be used both to populate a "history" of the conversation and to add new items mid-stream, but has the current limitation that it cannot populate assistant audio messages.

        If successful, the server will respond with a conversation.item.created event, otherwise an error event will be sent.

      • conversationItemDelete

         final Optional<ConversationItemDeleteEvent> conversationItemDelete()

        Send this event when you want to remove any item from the conversation history. The server will respond with a conversation.item.deleted event, unless the item does not exist in the conversation history, in which case the server will respond with an error.

      • conversationItemRetrieve

         final Optional<ConversationItemRetrieveEvent> conversationItemRetrieve()

        Send this event when you want to retrieve the server's representation of a specific item in the conversation history. This is useful, for example, to inspect user audio after noise cancellation and VAD. The server will respond with a conversation.item.retrieved event, unless the item does not exist in the conversation history, in which case the server will respond with an error.

      • conversationItemTruncate

         final Optional<ConversationItemTruncateEvent> conversationItemTruncate()

        Send this event to truncate a previous assistant message’s audio. The server will produce audio faster than realtime, so this event is useful when the user interrupts to truncate audio that has already been sent to the client but not yet played. This will synchronize the server's understanding of the audio with the client's playback.

        Truncating audio will delete the server-side text transcript to ensure there is not text in the context that hasn't been heard by the user.

        If successful, the server will respond with a conversation.item.truncated event.

      • inputAudioBufferAppend

         final Optional<InputAudioBufferAppendEvent> inputAudioBufferAppend()

        Send this event to append audio bytes to the input audio buffer. The audio buffer is temporary storage you can write to and later commit. In Server VAD mode, the audio buffer is used to detect speech and the server will decide when to commit. When Server VAD is disabled, you must commit the audio buffer manually.

        The client may choose how much audio to place in each event up to a maximum of 15 MiB, for example streaming smaller chunks from the client may allow the VAD to be more responsive. Unlike made other client events, the server will not send a confirmation response to this event.

      • outputAudioBufferClear

         final Optional<OutputAudioBufferClearEvent> outputAudioBufferClear()

        WebRTC Only: Emit to cut off the current audio response. This will trigger the server to stop generating audio and emit a output_audio_buffer.cleared event. This event should be preceded by a response.cancel client event to stop the generation of the current response. Learn more.

      • inputAudioBufferCommit

         final Optional<InputAudioBufferCommitEvent> inputAudioBufferCommit()

        Send this event to commit the user input audio buffer, which will create a new user message item in the conversation. This event will produce an error if the input audio buffer is empty. When in Server VAD mode, the client does not need to send this event, the server will commit the audio buffer automatically.

        Committing the input audio buffer will trigger input audio transcription (if enabled in session configuration), but it will not create a response from the model. The server will respond with an input_audio_buffer.committed event.

      • responseCancel

         final Optional<ResponseCancelEvent> responseCancel()

        Send this event to cancel an in-progress response. The server will respond with a response.done event with a status of response.status=cancelled. If there is no response to cancel, the server will respond with an error.

      • responseCreate

         final Optional<ResponseCreateEvent> responseCreate()

        This event instructs the server to create a Response, which means triggering model inference. When in Server VAD mode, the server will create Responses automatically.

        A Response will include at least one Item, and may have two, in which case the second will be a function call. These Items will be appended to the conversation history.

        The server will respond with a response.created event, events for Items and content created, and finally a response.done event to indicate the Response is complete.

        The response.create event includes inference configuration like instructions, and temperature. These fields will override the Session's configuration for this Response only.

      • sessionUpdate

         final Optional<SessionUpdateEvent> sessionUpdate()

        Send this event to update the session’s default configuration. The client may send this event at any time to update any field, except for voice. However, note that once a session has been initialized with a particular model, it can’t be changed to another model using session.update.

        When the server receives a session.update, it will respond with a session.updated event showing the full, effective configuration. Only the fields that are present are updated. To clear a field like instructions, pass an empty string.

      • asConversationItemCreate

         final ConversationItemCreateEvent asConversationItemCreate()

        Add a new Item to the Conversation's context, including messages, function calls, and function call responses. This event can be used both to populate a "history" of the conversation and to add new items mid-stream, but has the current limitation that it cannot populate assistant audio messages.

        If successful, the server will respond with a conversation.item.created event, otherwise an error event will be sent.

      • asConversationItemDelete

         final ConversationItemDeleteEvent asConversationItemDelete()

        Send this event when you want to remove any item from the conversation history. The server will respond with a conversation.item.deleted event, unless the item does not exist in the conversation history, in which case the server will respond with an error.

      • asConversationItemRetrieve

         final ConversationItemRetrieveEvent asConversationItemRetrieve()

        Send this event when you want to retrieve the server's representation of a specific item in the conversation history. This is useful, for example, to inspect user audio after noise cancellation and VAD. The server will respond with a conversation.item.retrieved event, unless the item does not exist in the conversation history, in which case the server will respond with an error.

      • asConversationItemTruncate

         final ConversationItemTruncateEvent asConversationItemTruncate()

        Send this event to truncate a previous assistant message’s audio. The server will produce audio faster than realtime, so this event is useful when the user interrupts to truncate audio that has already been sent to the client but not yet played. This will synchronize the server's understanding of the audio with the client's playback.

        Truncating audio will delete the server-side text transcript to ensure there is not text in the context that hasn't been heard by the user.

        If successful, the server will respond with a conversation.item.truncated event.

      • asInputAudioBufferAppend

         final InputAudioBufferAppendEvent asInputAudioBufferAppend()

        Send this event to append audio bytes to the input audio buffer. The audio buffer is temporary storage you can write to and later commit. In Server VAD mode, the audio buffer is used to detect speech and the server will decide when to commit. When Server VAD is disabled, you must commit the audio buffer manually.

        The client may choose how much audio to place in each event up to a maximum of 15 MiB, for example streaming smaller chunks from the client may allow the VAD to be more responsive. Unlike made other client events, the server will not send a confirmation response to this event.

      • asOutputAudioBufferClear

         final OutputAudioBufferClearEvent asOutputAudioBufferClear()

        WebRTC Only: Emit to cut off the current audio response. This will trigger the server to stop generating audio and emit a output_audio_buffer.cleared event. This event should be preceded by a response.cancel client event to stop the generation of the current response. Learn more.

      • asInputAudioBufferCommit

         final InputAudioBufferCommitEvent asInputAudioBufferCommit()

        Send this event to commit the user input audio buffer, which will create a new user message item in the conversation. This event will produce an error if the input audio buffer is empty. When in Server VAD mode, the client does not need to send this event, the server will commit the audio buffer automatically.

        Committing the input audio buffer will trigger input audio transcription (if enabled in session configuration), but it will not create a response from the model. The server will respond with an input_audio_buffer.committed event.

      • asResponseCancel

         final ResponseCancelEvent asResponseCancel()

        Send this event to cancel an in-progress response. The server will respond with a response.done event with a status of response.status=cancelled. If there is no response to cancel, the server will respond with an error.

      • asResponseCreate

         final ResponseCreateEvent asResponseCreate()

        This event instructs the server to create a Response, which means triggering model inference. When in Server VAD mode, the server will create Responses automatically.

        A Response will include at least one Item, and may have two, in which case the second will be a function call. These Items will be appended to the conversation history.

        The server will respond with a response.created event, events for Items and content created, and finally a response.done event to indicate the Response is complete.

        The response.create event includes inference configuration like instructions, and temperature. These fields will override the Session's configuration for this Response only.

      • asSessionUpdate

         final SessionUpdateEvent asSessionUpdate()

        Send this event to update the session’s default configuration. The client may send this event at any time to update any field, except for voice. However, note that once a session has been initialized with a particular model, it can’t be changed to another model using session.update.

        When the server receives a session.update, it will respond with a session.updated event showing the full, effective configuration. Only the fields that are present are updated. To clear a field like instructions, pass an empty string.

      • ofConversationItemCreate

         final static RealtimeClientEvent ofConversationItemCreate(ConversationItemCreateEvent conversationItemCreate)

        Add a new Item to the Conversation's context, including messages, function calls, and function call responses. This event can be used both to populate a "history" of the conversation and to add new items mid-stream, but has the current limitation that it cannot populate assistant audio messages.

        If successful, the server will respond with a conversation.item.created event, otherwise an error event will be sent.

      • ofConversationItemDelete

         final static RealtimeClientEvent ofConversationItemDelete(ConversationItemDeleteEvent conversationItemDelete)

        Send this event when you want to remove any item from the conversation history. The server will respond with a conversation.item.deleted event, unless the item does not exist in the conversation history, in which case the server will respond with an error.

      • ofConversationItemRetrieve

         final static RealtimeClientEvent ofConversationItemRetrieve(ConversationItemRetrieveEvent conversationItemRetrieve)

        Send this event when you want to retrieve the server's representation of a specific item in the conversation history. This is useful, for example, to inspect user audio after noise cancellation and VAD. The server will respond with a conversation.item.retrieved event, unless the item does not exist in the conversation history, in which case the server will respond with an error.

      • ofConversationItemTruncate

         final static RealtimeClientEvent ofConversationItemTruncate(ConversationItemTruncateEvent conversationItemTruncate)

        Send this event to truncate a previous assistant message’s audio. The server will produce audio faster than realtime, so this event is useful when the user interrupts to truncate audio that has already been sent to the client but not yet played. This will synchronize the server's understanding of the audio with the client's playback.

        Truncating audio will delete the server-side text transcript to ensure there is not text in the context that hasn't been heard by the user.

        If successful, the server will respond with a conversation.item.truncated event.

      • ofInputAudioBufferAppend

         final static RealtimeClientEvent ofInputAudioBufferAppend(InputAudioBufferAppendEvent inputAudioBufferAppend)

        Send this event to append audio bytes to the input audio buffer. The audio buffer is temporary storage you can write to and later commit. In Server VAD mode, the audio buffer is used to detect speech and the server will decide when to commit. When Server VAD is disabled, you must commit the audio buffer manually.

        The client may choose how much audio to place in each event up to a maximum of 15 MiB, for example streaming smaller chunks from the client may allow the VAD to be more responsive. Unlike made other client events, the server will not send a confirmation response to this event.

      • ofOutputAudioBufferClear

         final static RealtimeClientEvent ofOutputAudioBufferClear(OutputAudioBufferClearEvent outputAudioBufferClear)

        WebRTC Only: Emit to cut off the current audio response. This will trigger the server to stop generating audio and emit a output_audio_buffer.cleared event. This event should be preceded by a response.cancel client event to stop the generation of the current response. Learn more.

      • ofInputAudioBufferCommit

         final static RealtimeClientEvent ofInputAudioBufferCommit(InputAudioBufferCommitEvent inputAudioBufferCommit)

        Send this event to commit the user input audio buffer, which will create a new user message item in the conversation. This event will produce an error if the input audio buffer is empty. When in Server VAD mode, the client does not need to send this event, the server will commit the audio buffer automatically.

        Committing the input audio buffer will trigger input audio transcription (if enabled in session configuration), but it will not create a response from the model. The server will respond with an input_audio_buffer.committed event.

      • ofResponseCancel

         final static RealtimeClientEvent ofResponseCancel(ResponseCancelEvent responseCancel)

        Send this event to cancel an in-progress response. The server will respond with a response.done event with a status of response.status=cancelled. If there is no response to cancel, the server will respond with an error.

      • ofResponseCreate

         final static RealtimeClientEvent ofResponseCreate(ResponseCreateEvent responseCreate)

        This event instructs the server to create a Response, which means triggering model inference. When in Server VAD mode, the server will create Responses automatically.

        A Response will include at least one Item, and may have two, in which case the second will be a function call. These Items will be appended to the conversation history.

        The server will respond with a response.created event, events for Items and content created, and finally a response.done event to indicate the Response is complete.

        The response.create event includes inference configuration like instructions, and temperature. These fields will override the Session's configuration for this Response only.

      • ofSessionUpdate

         final static RealtimeClientEvent ofSessionUpdate(SessionUpdateEvent sessionUpdate)

        Send this event to update the session’s default configuration. The client may send this event at any time to update any field, except for voice. However, note that once a session has been initialized with a particular model, it can’t be changed to another model using session.update.

        When the server receives a session.update, it will respond with a session.updated event showing the full, effective configuration. Only the fields that are present are updated. To clear a field like instructions, pass an empty string.