[guardian-dev] NetCipher and HttpURLConnection

[Josh Steiner]

Hi all,

I had a few questions about NetCipher. We wanted to get the official
Facebook SDK working with TOR and it uses the stock HttpURLConnection
stack, not Apache.

First off, is there a reason we support only Apache's http libs?
Other than "engineering time" of course.

Also, I notice that we include our own build of the jar in the
NetCipher libs/ (httpclientandroidlib-1.1.2.jar), is there a reason
for this rather than just using the one that ships in Android AOSP?

Thanks

-Josh

[Nathan of Guardian]

Post by Josh Steiner
Hi all,
I had a few questions about NetCipher. We wanted to get the official
Facebook SDK working with TOR and it uses the stock HttpURLConnection
stack, not Apache.
First off, is there a reason we support only Apache's http libs?
Other than "engineering time" of course.

HttpURLConnection does not support SOCKS proxying AFAIK.

Post by Josh Steiner
Also, I notice that we include our own build of the jar in the
NetCipher libs/ (httpclientandroidlib-1.1.2.jar), is there a reason
for this rather than just using the one that ships in Android AOSP?

Aosp's is out of date.

[Josh Steiner]

On Mon, Feb 10, 2014 at 5:28 PM, Nathan of Guardian

Post by Nathan of Guardian

Post by Josh Steiner
Hi all,
I had a few questions about NetCipher. We wanted to get the official
Facebook SDK working with TOR and it uses the stock HttpURLConnection
stack, not Apache.
First off, is there a reason we support only Apache's http libs?
Other than "engineering time" of course.

HttpURLConnection does not support SOCKS proxying AFAIK.

I think it does, but I havn't tried it:

http://developer.android.com/reference/java/net/HttpURLConnection.html

"By default, this class will connect directly to the origin server. It
can also connect via an HTTP or SOCKS proxy. To use a proxy,
useURL.openConnection(Proxy) when creating the connection."

Need to decide if it's smarter to port the Facebook SDK lib to Apache
or add a HttpURLConnection support to NetCipher. Obviously the latter
would be more flexible, but I'm not sure how much work that will be.

-j

[Nathan of Guardian]

Post by Josh Steiner
http://developer.android.com/reference/java/net/HttpURLConnection.html
"By default, this class will connect directly to the origin server. It
can also connect via an HTTP or SOCKS proxy. To use a proxy,
useURL.openConnection(Proxy) when creating the connection."

Hmm, great. Not sure why we missed that before, or perhaps it changed?

The main issue, and one we have a problem with right now with Apache, is
that we need to make sure DNS does not leak. This means no InetAddress
look ups before you connect, and that the SOCKS support is the right
kind for the remote DNS resolution support.

Post by Josh Steiner
Need to decide if it's smarter to port the Facebook SDK lib to Apache
or add a HttpURLConnection support to NetCipher. Obviously the latter
would be more flexible, but I'm not sure how much work that will be.

I think if we can ensure the SOCKS proxying works as expected, then the
latter is the best bet.

+n

[Josh Steiner]

On Mon, Feb 10, 2014 at 6:55 PM, Nathan of Guardian <

Post by Nathan of Guardian

Post by Josh Steiner
http://developer.android.com/reference/java/net/HttpURLConnection.html
"By default, this class will connect directly to the origin server. It
can also connect via an HTTP or SOCKS proxy. To use a proxy,
useURL.openConnection(Proxy) when creating the connection."

Hmm, great. Not sure why we missed that before, or perhaps it changed?
The main issue, and one we have a problem with right now with Apache, is
that we need to make sure DNS does not leak. This means no InetAddress
look ups before you connect, and that the SOCKS support is the right
kind for the remote DNS resolution support.

Interesting, I'll see what I can dig up on that. If HttpURLConnecction* is
better in this regard, would you prefer to drop Apache support or to keep
both as an option? It is nice to have both if you are integrating with 3rd
party libs where you can't control what they decided to use, but if its a
gaping security hole we might not want to endorse it.

Post by Nathan of Guardian

Post by Josh Steiner
Need to decide if it's smarter to port the Facebook SDK lib to Apache
or add a HttpURLConnection support to NetCipher. Obviously the latter
would be more flexible, but I'm not sure how much work that will be.

I think if we can ensure the SOCKS proxying works as expected, then the
latter is the best bet.
+n

[Mark Murphy]

Post by Josh Steiner
Interesting, I'll see what I can dig up on that. If HttpURLConnecction* is
better in this regard, would you prefer to drop Apache support or to keep
both as an option? It is nice to have both if you are integrating with 3rd
party libs where you can't control what they decided to use, but if its a
gaping security hole we might not want to endorse it.

FWIW, AFAIK, the implementation of HttpUrlConnection changed in Android
4.4 to be more or less a clone of Square's OkHTTP:

http://square.github.io/okhttp/

That was mostly for SPDY support, as I understand it.

I mention it, as the behavior of HttpUrlConection may have changed in
Android 4.4 with respect to things like SOCKS. Also, if you determine
that OkHTTP/4.4's HttpUrlConnection is significantly better, developers
can use OkHTTP on older Android devices to have an
HttpUrlConnection-compatible API.

[Josh Steiner]

Turns out SOCKS support is totally busted for HttpURLConnection* on
Android. We'll test OkHTTP to see if it works out better for us:

https://code.google.com/p/android/issues/detail?id=33122

(*pet peeve, why isn't it HTTPURLConnection, or even HttpUrlConnection?!?)

Post by Mark Murphy

Post by Josh Steiner
Interesting, I'll see what I can dig up on that. If HttpURLConnecction* is
better in this regard, would you prefer to drop Apache support or to keep
both as an option? It is nice to have both if you are integrating with 3rd
party libs where you can't control what they decided to use, but if its a
gaping security hole we might not want to endorse it.

FWIW, AFAIK, the implementation of HttpUrlConnection changed in Android
http://square.github.io/okhttp/
That was mostly for SPDY support, as I understand it.
I mention it, as the behavior of HttpUrlConection may have changed in
Android 4.4 with respect to things like SOCKS. Also, if you determine
that OkHTTP/4.4's HttpUrlConnection is significantly better, developers
can use OkHTTP on older Android devices to have an
HttpUrlConnection-compatible API.
--
Mark Murphy (a Commons Guy)
http://commonsware.com | http://github.com/commonsguy
http://commonsware.com/blog | http://twitter.com/commonsguy
_The Busy Coder's Guide to Android Development_: Version 5.5... And
Still Going Strong!
_______________________________________________
Guardian-dev mailing list
List info: https://lists.mayfirst.org/mailman/listinfo/guardian-dev
To Unsubscribe
Or visit: https://lists.mayfirst.org/mailman/options/guardian-dev/josh%40vitriolix.com

[Rod Hynes]

On Wed, Feb 12, 2014 at 11:01 AM,

Post by Josh Steiner
Interesting, I'll see what I can dig up on that. If HttpURLConnecction* is
better in this regard, would you prefer to drop Apache support or to keep
both as an option? It is nice to have both if you are integrating with 3rd
party libs where you can't control what they decided to use, but if its a
gaping security hole we might not want to endorse it.

This may be useful information:

Last year I evaluated Android HTTP client libraries looking for one that
worked with Tor via SOCKS to connect to a Tor Hidden Service (referenced by
.onion domain name). At the time, none of OkHttp, stock HttpUrlConnection,
or stock Apache HttpClient worked. I specifically wanted SOCKS since I did
not want to run an HTTP proxy in front of Tor's SOCKS (which I believe
would work).

Digging deeper, I believe that these libraries rely on the SOCKS support
built into Java's Socket class. There are a couple of issues with this:

1. To take advantage of remote host name resolution with the SOCKS support
built into Java's Socket class, the InetAddress you pass to
Socket.connect() ought to be constructed using createUnresolved[1].
Otherwise, the InetAddress instance resolves the host name using the system
resolver in its constructor. If I recall correctly, none of the above
libraries appeared to defer to the SOCKS proxy in this way, although they
do make use of the SOCKS support in the Socket class (for example, check
out this use in OkHttp: [2]).

2. Android's "Apache Harmony" Socket implementation differs from the
standard Java runtime in its SOCKS support. If you look what I believe to
be the Android code for Socket[3], it's totally different than the Oracle
Socket code[4] and it only supports SOCKS4, not SOCKS4a or SOCKS5. So
remote domain resolution with SOCKS is actually not supported at all with
Socket on Android (standard/Oracle Java does support SOCKS4a and SOCKS5
with remote resolution when createUnresolved() is used). What happens in
practice is that Android's Socket.connect() throws an exception when
presented with the unresolved InetAddress.

If you use one of these libraries and your system resolver can resolve the
host address, they will appear to work but you may be leaking DNS info
outside of Tor. If you need to resolve a .onion address you'll notice an
explicit error if they fail.

Ultimately, what I did (see: [5]) was use Apache HttpClient because it
allowed me to provide my own socket factory, in which I explicitly perform
the SOCKS4a protocol, sending the unresolved host name to Tor.

[1]
http://developer.android.com/reference/java/net/InetSocketAddress.html#createUnresolved%28java.lang.String,%20int%29
[2]
https://github.com/square/okhttp/blob/master/okhttp/src/main/java/com/squareup/okhttp/Connection.java#L87
[3]
https://android.googlesource.com/platform/libcore/+/master/luni/src/main/java/java/net/PlainSocketImpl.java
[4]
http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/java/net/SocksSocketImpl.java
[5]
https://github.com/rod-hynes/ploggy/blob/master/AndroidApp/src/ca/psiphon/ploggy/WebClient.java#L331