Better HTTP/1 connection reuse in Go 1.27
go.dev[1]: https://github.com/cipherdothost/x-go/blob/caa562ded82b8c417...
Not because of the copy itself, since a little copy is better than a little dependency, as they say, but because it was a pain to update everything when I inevitably found a bug in what I copied.
I can attest to the fact this is a Frequently Asked Question among people who are trying to Go as an HTTP client, so even though this feels somewhat icky, I can understand where it is coming from. However, I'm also missing, if I want to slam this connection closed, which is a reasonable thing to want, how do I do that without uselessly draining a socket I don't intend to reuse? Fake the "Connection: close" header into the Response object? Hope I can dig out an underlying socket of some sort under the concrete type of the Body.Response that I can bypass this code with? Seems like there should be a way to do that too.
> Note, however, that Transport will automatically try to read a Response Body to EOF asynchronously up to a conservative limit when a Body is closed.
Similar wording appears in a few other places; search for “asynchronously”.
I’d argue that implementing both TCP and IP from scratch is easier than implementing an HTTP library.
Decoder only reads X bytes (512? I don’t remember) and if the structure fits into these bytes exactly (and the multiplications of it) the newline leaves hanging there.
I always check for Decoder.More (this drains whitespace) after decoding.
Turns out that json.Encoder always appends a newline but (as GP said) json.Decoder will not attempt to read it if the JSON is complete at the power-of-two byte boundary (IIRC the buffer doubles for the first few iterations). The extra byte in the pipe when closed then causes reads and writes to go boom. We switched to using json.Marshal because it doesn't output a newline (sadly the encoded form isn't streamed as a result).
I still get twitchy around encoding/json as a result of that experience.
[1]: https://github.com/docker/docker/issues/14203#issuecomment-1...
https://cs.opensource.google/go/go/+/refs/tags/go1.27.2:src/...
What it won't do looking at the code is handle streamed responses over transfer-encoding:chunked which means that if the other end is doing
json.NewEncoder(responseWriter).Encode(&someStruct)
The standard library maintainers recognized this as a problem that there was no “better,” more convenient way to do it and in encoding/json/v2 they have UnmarshalRead for that use case: https://pkg.go.dev/encoding/json/v2#UnmarshalRead
Don't believe there was a more convenient way to achieve the same without writing our own Decoders/Encoders.
Does this mean I need to disable http2 for this to work or is that just a further proof that http2 isn't actually being used?
But of course, the only scenarios where you have prior knowledge are things like reverse proxies or custom made clients.
sock.write('HTTP/1.1 200 OK\r\ncontent-length: ' + f.size() + '\r\n\r\n')
sendfile(sock, f)
1. Suggest drainAndCloseBody // Team: this is reasonable
2. Implement drainAndCloseBodyWithTimer // Team: no problems here
3. then drainAsyncAndCloseBodyWithTimer // Team: a bit wordy but no problem here
4. Someone at google uses 3. with an http/2 connection // Team: this is where we draw the line?
I sympathize with the struggle of following golang's mission to be simple when you consider this common but tricky edge case. Iteration 3 makes sense but by gosh is it ugly ha.
I made a lot of money on PHP and more on Golang.
PHP had warts (variable order) but much like go it came with just about everything you wanted.
It isnt just the functionality - it's the fact that both of them had amazing documentation with the functionality. PHP manual was an amazing reference and the "comments" section always had a bit of solid wisdom for ya (and you needed it cause of the warts). Go's manual (and the blog) are such a fine resource that I rarely need to reach further than what it contains.
It's much easier to use what's included when the documentation is accessible and clear.
PHP's "batteries included" approach is probably a product of it being launched before package managers became ubiquitous (the PHP package manager Composer was only released in 2012), Go's is a conscious decision.
Huge security issues involved with it, see: https://portswigger.net/research/can-ai-do-novel-security-re...
It seems like an edge problem for an edge use case, but I guess At Go’s scale, it’s probably important.