http-client: Always include a path component when sending a proxy request

[sjamaan]

See ​this issue from mitmproxy. While debugging an issue I had to locally patch my mitmproxy to be able to get it to work...

[Vasilij Schneidermann]

I suspect the same issue keeps me from using Burpsuite unless I enable "invisible proxying". This happens only for ​https:// URLs though, not ​http:// ones.

edit: Nevermind, this is its invisible proxying tolerating clients that don't implement CONNECT, so it's a separate issue.

[sjamaan]

Should be fixed with f3efa21fb, will make a release later after verifying that this fixes the situation under mitmproxy.

My interpretation of the RFCs is that the old version of mitmproxy was incorrect, but it's still fine to update the path to have a trailing slash, because it would be added by a well-behaving proxy anyway, so there should be no observable difference in that case.

[sjamaan]

Nope, that didn't fix it because uri-common has a hateful behaviour where it tries to "normalize" the path in a way that doesn't actually match the real situation.

This is not ideal for this particular piece of code, but it is helpful when trying to manipulate paths in a sane way. And since uri-common makes the promise/guarantee never to mutate the input unless you explicitly do so yourself, it can't really avoid doing this at the readout level (note that when you write an empty path it will normalize it too, so then it will have the right value). So while I'm not 100% sure we should keep it, it's probably still fine.

Anyway, I've worked around this by also checking for the empty faked out root path in 4c32441.

There's still some behaviour I want to investigate before making a new release - it looks like the client is disconnecting when it is receiving a 404. A quick study of the http-client code seems to indicate we're not doing it there, so maybe it's mitmproxy (or the upstream server even?) which is disconnecting.

[sjamaan]

Fixed in 1.2.1