Summary

http-client: Always include a path component when sending a proxy request

Metadata

Description

See this|issue from mitmproxy]. While debugging an issue I had to locally patch my mitmproxy to be able to get it to work...

Changes and comments

[2021-09-03 10:15:54 UTC] wasamasa wrote:

I suspect the same issue keeps me from using Burpsuite unless I enable "invisible proxying". This happens only for https:// URLs though, not http:// ones.

edit: Nevermind, this is its invisible proxying tolerating clients that don't implement CONNECT, so it's a separate issue.

[2021-09-03 10:15:54 UTC] wasamasa wrote:

1630657949125917

[2021-09-03 10:15:54 UTC] wasamasa wrote:

1630663386030514

[2021-09-03 16:25:25 UTC] sjamaan wrote:

Should be fixed with f3efa21fb, will make a release later after verifying that this fixes the situation under mitmproxy.

My interpretation of the RFCs is that the old version of mitmproxy was incorrect, but it's still fine to update the path to have a trailing slash, because it would be added by a well-behaving proxy anyway, so there should be no observable difference in that case.

[2021-09-06 08:21:42 UTC] sjamaan wrote:

Nope, that didn't fix it because uri-common has a hateful behaviour where it tries to "normalize" the path in a way that doesn't actually match the real situation.

This is not ideal for this particular piece of code, but it is helpful when trying to manipulate paths in a sane way. And since uri-common makes the promise/guarantee never to mutate the input unless you explicitly do so yourself, it can't really avoid doing this at the readout level (note that when you write an empty path it will normalize it too, so then it will have the right value). So while I'm not 100% sure we should keep it, it's probably still fine.

Anyway, I've worked around this by also checking for the empty faked out root path in 4c32441.

There's still some behaviour I want to investigate before making a new release - it looks like the client is disconnecting when it is receiving a 404. A quick study of the http-client code seems to indicate we're not doing it there, so maybe it's mitmproxy (or the upstream server even?) which is disconnecting.

[2021-09-10 07:14:37 UTC] sjamaan changed status from assigned to closed

[2021-09-10 07:14:37 UTC] sjamaan set resolution to fixed

[2021-09-10 07:14:37 UTC] sjamaan wrote:

Fixed in 1.2.1