Hey great post. I was curious about this too, and started exploring the network tab. They definitely use websockets. Make sure you click on preserve logs. Open a new tab without opening chatgpt yet, then go to inspect and make sure your preserve logs checkbox is checked. Now open chatgpt. You'll see 3 WS connections.
But you are right on the SSE & one direction communication I guess because that's what confused me lot and started googling and found your article. In the /ws logs, I don't see the stream of token. Not not even on XHR/Fetch. I see a POST request with my questions, I see incoming streaming of tokens on /ws, which didn't have the tokens explicitly. Not even encoded ones! I'm so confused
Can you please do a deep dive into websocket implementation too?
Side note: I am using enteprise chatgpt which is using HTTP/1.1. Here I am still seeing Server side streaming being used, while on personal chatgpt I think they are uisng Websockets with HTTP3. I tried a lot but not able to see the message chunks being received in network tab.
I was looking at this as well but I don't think this tells the full story. In JS EventSource can only set up a GET request and they are sending a POST with message data. They might be using a custom implementation?
What we do for rizzgpt.app is similar to what you’re describing. When I originally wrote this, I didn’t have the experience/context that you also needed the GET request.
Hey great post. I was curious about this too, and started exploring the network tab. They definitely use websockets. Make sure you click on preserve logs. Open a new tab without opening chatgpt yet, then go to inspect and make sure your preserve logs checkbox is checked. Now open chatgpt. You'll see 3 WS connections.
But you are right on the SSE & one direction communication I guess because that's what confused me lot and started googling and found your article. In the /ws logs, I don't see the stream of token. Not not even on XHR/Fetch. I see a POST request with my questions, I see incoming streaming of tokens on /ws, which didn't have the tokens explicitly. Not even encoded ones! I'm so confused
I think the WS implementation may be more recent. I wrote this over a year ago!
Yeah that is possible!
Hi Theodor.
Loved your blog and explaination.
Can you please do a deep dive into websocket implementation too?
Side note: I am using enteprise chatgpt which is using HTTP/1.1. Here I am still seeing Server side streaming being used, while on personal chatgpt I think they are uisng Websockets with HTTP3. I tried a lot but not able to see the message chunks being received in network tab.
I was looking at this as well but I don't think this tells the full story. In JS EventSource can only set up a GET request and they are sending a POST with message data. They might be using a custom implementation?
What we do for rizzgpt.app is similar to what you’re describing. When I originally wrote this, I didn’t have the experience/context that you also needed the GET request.
I'm not sure how OpenAI implements but there's a https://www.npmjs.com/package/@microsoft/fetch-event-source dependency that supports POST and GET the SSE at the same endpoint.
We have used SSE for a while in a ChatGPT like project but recently switched to websocket.
Oh interesting! We just wrote this from scratch. Why do you prefer websockets?
Mainly because the developer experience :D
Consider a scenario involving a complex workflow:
- Transmit certain data to the client.
- Perform specific tasks.
- Dispatch a different set of data to the client.
It would be much easier to implement such workflow in websocket.
Other things I concered are
- HTTP Overhead: Each SSE entails an individual HTTP request, which introduces some network overhead.
- library activity: In my project, I incorporated SSE using [sse-starlette](https://pypi.org/project/sse-starlette/) on the server side and [fetch-event-source](https://www.npmjs.com/package/@microsoft/fetch-event-source) on the client side. These libraries are relatively less popular and have seen less active maintenance compared to WebSocket libraries like [websockets](https://pypi.org/project/websockets/)