9 Comments

Hey great post. I was curious about this too, and started exploring the network tab. They definitely use websockets. Make sure you click on preserve logs. Open a new tab without opening chatgpt yet, then go to inspect and make sure your preserve logs checkbox is checked. Now open chatgpt. You'll see 3 WS connections.

But you are right on the SSE & one direction communication I guess because that's what confused me lot and started googling and found your article. In the /ws logs, I don't see the stream of token. Not not even on XHR/Fetch. I see a POST request with my questions, I see incoming streaming of tokens on /ws, which didn't have the tokens explicitly. Not even encoded ones! I'm so confused

Expand full comment
author

I think the WS implementation may be more recent. I wrote this over a year ago!

Expand full comment

Yeah that is possible!

Expand full comment

Hi Theodor.

Loved your blog and explaination.

Can you please do a deep dive into websocket implementation too?

Side note: I am using enteprise chatgpt which is using HTTP/1.1. Here I am still seeing Server side streaming being used, while on personal chatgpt I think they are uisng Websockets with HTTP3. I tried a lot but not able to see the message chunks being received in network tab.

Expand full comment

I was looking at this as well but I don't think this tells the full story. In JS EventSource can only set up a GET request and they are sending a POST with message data. They might be using a custom implementation?

Expand full comment
author

What we do for rizzgpt.app is similar to what you’re describing. When I originally wrote this, I didn’t have the experience/context that you also needed the GET request.

Expand full comment

I'm not sure how OpenAI implements but there's a https://www.npmjs.com/package/@microsoft/fetch-event-source dependency that supports POST and GET the SSE at the same endpoint.

We have used SSE for a while in a ChatGPT like project but recently switched to websocket.

Expand full comment
author

Oh interesting! We just wrote this from scratch. Why do you prefer websockets?

Expand full comment

Mainly because the developer experience :D

Consider a scenario involving a complex workflow:

- Transmit certain data to the client.

- Perform specific tasks.

- Dispatch a different set of data to the client.

It would be much easier to implement such workflow in websocket.

Other things I concered are

- HTTP Overhead: Each SSE entails an individual HTTP request, which introduces some network overhead.

- library activity: In my project, I incorporated SSE using [sse-starlette](https://pypi.org/project/sse-starlette/) on the server side and [fetch-event-source](https://www.npmjs.com/package/@microsoft/fetch-event-source) on the client side. These libraries are relatively less popular and have seen less active maintenance compared to WebSocket libraries like [websockets](https://pypi.org/project/websockets/)

Expand full comment