MITM With SSL?

[TechnikEmpire]

I'm not sure if this is a bug or I'm just missing something, I'm assuming the latter. Trying to run the Mitm test, nothing happens on SSL connections. The browser just spins forever waiting for a response and I see no error or other activity coming from the proxy. I'm coming to LittleProxy from my own transparent proxy, and first thing I see missing is the creation, installation or any use of a root CA by which the proxy can spoof certs for the sake of intercepting SSL. Am I just missing some code or documentation? Thanks in advance.

[jekh]

LittleProxy currently doesn't provide this configuration out-of-the-box. It does provide the MitmManager interface, which you can implement to customize your certificate generation as needed. It would be wonderful to get this functionality into LittleProxy, it just hasn't been done yet. There is talk of an extras package (see issue #173) to support this, and @ganskef is currently working on some SSL functionality, but nothing has been released yet. If you're willing to put something together, I'm sure everybody would love to see it!

[TechnikEmpire]

@jekh I did this in C# with my own proxy, working on doing it in C++ as we speak. It's a real pain in the ass to be honest, but basically you've gotta just use bouncy castle to generate a root CA and then you can peek SSL sockets to read the TLS hello/sni/etc and generate an in-memory cert store on the fly. The only problem is, I've been out of Java for a long time (since android 2 :| ) so I'm fumbling a bit trying to dust off the Java section in my memory. lol I'l see if I can put something together though.

[jekh]

I definitely understand that it is a huge pain in the ass... I am completely out of my element when it comes to SSL certificate spoofing. If it helps, the latest commit from @ganskef on his implementation (using bouncy castle) is at b2bd8fb. That PR has since been repurposed, but I believe he has moved the MITM code to https://github.com/ganskef/LittleProxy-mitm.

[TechnikEmpire]

Thanks for the links, I'll have a look tonight and see whats what. You're gonna wanna use ECDH for key exchange to balance security and performance. Anything else becomes so computationally expensive that user experience suffers to the point of it being unusable in any production scenario. TBH I'm out of my depth with this stuff too, I've just googled enough to make it work. :) lol Again I'll take a peek but for anyone coming across this thread:

https://github.com/dlundquist/sniproxy/blob/master/src/tls.c <- Code for parsing TLS hello for the sake of host discovery and hence upstream certificate spoofing.

Blog + Code for being a spoofing CA. Yes it's in C# but it's using bouncy castle and the difference between this and what it would look like in Java is just a matter of syntax. :)
http://blog.differentpla.net/blog/2013/03/24/bouncy-castle-being-a-certificate-authority/
https://github.com/rlipscombe/bouncy-castle-csharp

[jekh]

Thanks for the links, @TechnikEmpire. I'll take a look. In terms of performance, assuming it doesn't take "too" long to generate certs (i.e. a few seconds or less), we should be able to leverage caching to make sure that certs are only generated once per host, rather than generating a new cert with every request. I can help with that though, once we have the foundations laid.

[BeamLiu]

It is happy that I found similar discussion here, I did a fork from main stream, and implemented the CA signed feature, it is really nice feature, user only need install the LittleProxy generated CA to his system, then the boring security warning dialog won't come up again.

I am also using bouncy castle to generate the different website certificates in the fly, then sign the certificates using CA, at last, populate the private key and corresponding signed certificates into a keystore in the fly. I did a testing, there isn't much performance issue, my local few seconds could finish all the work of publishing a new certificate for a website, and there is a 200 capicity of LRU map to had certificates cache.

This change is very isolated with other codes, it just had a new implementation of MitmManager.java
@jekh , could u kindly take a look? Is there any possiblity for me to create a pull request to merge this feature? My code changes are here: master...BeamLiu:master

[jekh]

@BeamLiu Thank you! I've only been able to do a cursory glance so far, but I'm very excited about this. I'll be busy for the next few days but will have a deeper look as soon as I can. Hopefully in the meantime others like @ganskef will be able to review as well (he has done some work on this already and is much better-versed in SSL than I am).

[ganskef]

@BeamLiu Thank you! I would love it, to have a CA implementation in a LittleProxy release. Please have a look at CR #174 and ganskef/LittleProxy-mitm. I will have a closer look at your code - first thougts:

I use Guava instead of Apache LRU cache. But, I think caching is not really neccessary, since it benefits from LittleProxys connection cache. It's optionally in my implementation.
I use a PKCS12 key store, since Android is not implemeting Oracles jks format.
But, I've no success within Android and Netty SSL - unclear OOM errors - WIP. I consider to use Spongy Castle instead of Bouncy Castle.
I had noticed truncated SAN lists at developer.chrome.com for example. Therefore I create the dynamic cert for the host name instead of the CN/SAN from the upstream cert. This simplifies my response while offline, too. I've build a caching proxy for offline use for http and https. It's in my daily use with Mac OS X and Linux (and Android http only).

[BeamLiu]

Thanks @jekh , @ganskef , interesting, it seems we are doing similar, sorry for that I didn't had a search before my starting. Anyway, both of u could have a round of reviewing first if u have time, my use case is to intercept browser traffic in desktop, I was trying to modify less code as possible, so I just forked from master, and had code changes directly. I will take a look at @ganskef 's code too.

[ganskef]

Hello @BeamLiu, please have a look at CR #173. That is what I'm waiting
for. MITM seems not to be invited to be a part of LittleProxy core. I
think, it have to be core, since https is growing in the web (ArchLinux,
Debian, github, mdn, google, http2, spdy...).

Am 09.05.2015 um 12:09 schrieb BeamLiu:

Thanks @jekh https://github.com/jekh , @ganskef
https://github.com/ganskef , interesting, it seems we are doing
similar, sorry for that I didn't had a search before my starting.
Anyway, both of u could have a round of reviewing first if u have time,
my use case is to intercept browser traffic in desktop, I was trying to
modify less code as possible, so I just forked from master, and had code
changes directly. I will take a look at @ganskef
https://github.com/ganskef 's code too.

—
Reply to this email directly or view it on GitHub
#184 (comment).

[ganskef]

Hello @TechnikEmpire, in my experience it was not really neccessary to cache the SSLContext for the dynamic server certificate. I generate an 1024 bit RSA key pair using SHA1PRNG. Thoughts: 2048 takes much longer time on older CPUs. And for almost every client (every browser I've tried), 1024 is sufficient. With my 6 jear old notebook key generation is done in 50-500ms, once per host, up to 8 connections, handled by LittleProxy. If I use SSL caching, I can't notice it in my daily use.

[ganskef]

Hello @BeamLiu, I've installed your fork in Debian 8. All tests passes. I've run the Launcher with -port 9090 -ca_mitm. I've found a certificate and a key store in my home dir. Good.

Now I've tried to import littleproxy_root.cer as a Certificate Authority in both Chromium and Iceweasel (Mozilla Firefox) browsers, but it fails with this message:
The file contained one certificate, which was not imported:
LittleProxyRoot: Not a Certification Authority.

Whats wrong? My implementation exports a certificate which can be imported as Certificate Authority into the browser. I expect to use it to suppress the certificate warnings/failures in the browser.

[BeamLiu]

Hi @ganskef , according to https://wiki.mozilla.org/CA:Recommendations_for_Roots , I added the additional CA=true extension into the root certificate, please update the latest code and remove the old keystore and certificate in user home directory, it should work now. Thanks a lot. Later, I will also have a try in Linux, and the intecerption between android and server over https.

[ganskef]

Hello @BeamLiu, yepp it's working now like expected. But, this means it
fails with https://developer.chrome.com/ :-( .

The reason is, the Subject Alternative Name field in the upstream
certificate is truncated. I could not locate the reason.

My workaround is to create the fake certificate with the requested host
name CN=developer.chrome.com instead of CN=misc.google.com and SAN=
DNS-Name: misc.google.com
DNS-Name: *.888googler.com
DNS-Name: *.adgoogle.net
DNS-Name: *.baselinestudy.com
DNS-Name: *.chrome.com
DNS-Name: *.chromecast.com
DNS-Name: *.chromium.org
DNS-Name: *.developer.google.com
DNS-Name: *.developers.google.com
DNS-Name: *.feedburner.com
DNS-Name: *.flutterapp.com
DNS-Name: *.gbc.beatthatquote.com
DNS-Name: *.gerritcodereview.com
DNS-Name: *.glass-community.com
DNS-Name: *.google.org
DNS-Name: *.googleapps.com
DNS-Name: *.googlecompare.co.uk
DNS-Name: *.googleforveterans.com
DNS-Name: *.googletraveladservices.com
DNS-Name: *.hindiweb.com
DNS-Name: *.hwgo.com
DNS-Name: *.madewithcode.com
DNS-Name: *.mfg-inspector.com
DNS-Name: *.pagespeedmobilizer.com
DNS-Name: *.personfinder.google.org
DNS-Name: *.quickoffice.com
DNS-Name: *.quoteproxy.beatthatquote.com
DNS-Name: *.schemer.com
DNS-Name: *.screenwisetrendspanel.com
DNS-Name: *.shibboleth.tv
DNS-Name: *.staging.widevine.com
DNS-Name: *.support.google.com
DNS-Name: *.thinkwithgoogle.at
DNS-Name: *.thinkwithgoogle.ch
DNS-Name: *.thinkwithgoogle.co.il
DNS-Name: *.thinkwithgoogle.co.uk
DNS-Name: *.thinkwithgoogle.co.za
DNS-Name: *.thinkwithgoogle.com.au
DNS-Name: *.thinkwithgoogle.com.br
DNS-Name: *.thinkwithgoogle.com.hk
DNS-Name: *.thinkwithgoogle.de
DNS-Name: *.thinkwithgoogle.dk
DNS-Name: *.thinkwithgoogle.es
DNS-Name: *.thinkwithgoogle.fi
DNS-Name: *.thinkwithgoogle.fr
DNS-Name: *.thinkwithgoogle.it
DNS-Name: *.thinkwithgoogle.jp
DNS-Name: *.uat.widevine.com
DNS-Name: *.whatbrowser.org
DNS-Name: *.widevine.com
DNS-Name: *.womenwill.com
DNS-Name: *.xn--9trs65b.com
DNS-Name: *.youradmob.com
DNS-Name: *.youtubemobilesupport.com
DNS-Name: *.zagat.com
DNS-Name: 888googler.com
DNS-Name: adgoogle.net
DNS-Name: baselinestudy.com
DNS-Name: chrome.com
DNS-Name: chromecast.com
DNS-Name: chromium.org
DNS-Name: clickserve.dartsearch.net
DNS-Name: clickserve.eu.dartsearch.net
DNS-Name: clickserve.uk.dartsearch.net
DNS-Name: clickserve.us2.dartsearch.net
DNS-Name: clickserver.googleads.com
DNS-Name: cookiechoices.org
DNS-Name: feedburner.com
DNS-Name: flutterapp.com
DNS-Name: gbc.beatthatquote.com
DNS-Name: gerritcodereview.com
DNS-Name: glass-community.com
DNS-Name: google.domains
DNS-Name: google.org
DNS-Name: googleapps.com
DNS-Name: googlecompare.co.uk
DNS-Name: googletraveladservices.com
DNS-Name: hindiweb.com
DNS-Name: hwgo.com
DNS-Name: images.zagat.com
DNS-Name: madewithcode.com
DNS-Name: mfg-inspector.com
DNS-Name: n339.asp-cc.com
DNS-Name: pagespeedmobilizer.com
DNS-Name: quoteproxy.beatthatquote.com
DNS-Name: schemer.com
DNS-Name: screenwisetrendspanel.com
DNS-Name: thinkwithgoogle.at
DNS-Name: thinkwithgoogle.ch
DNS-Name: thinkwithgoogle.co.il
DNS-Name: thinkwithgoogle.co.uk
DNS-Name: thinkwithgoogle.co.za
DNS-Name: thinkwithgoogle.com.au
DNS-Name: thinkwithgoogle.com.br
DNS-Name: thinkwithgoogle.com.hk
DNS-Name: thinkwithgoogle.de
DNS-Name: thinkwithgoogle.dk
DNS-Name: thinkwithgoogle.es
DNS-Name: thinkwithgoogle.fi
DNS-Name: thinkwithgoogle.fr
DNS-Name: thinkwithgoogle.it
DNS-Name: thinkwithgoogle.jp
DNS-Name: whatbrowser.org
DNS-Name: womenwill.com
DNS-Name: www.cookiechoices.org
DNS-Name: xn--9trs65b.com
DNS-Name: youradmob.com
DNS-Name: youtubemobilesupport.com
DNS-Name: zagat.com

Am 14.05.2015 um 08:45 schrieb BeamLiu:

Hi @ganskef https://github.com/ganskef , according to
https://wiki.mozilla.org/CA:Recommendations_for_Roots , I added the
additional CA=true extension into the root certificate, please update
the latest code and remove the old keystore and certificate in user home
directory, it should work now. Thanks a lot. Later, I will also have a
try in Linux, and the intecerption between android and server over https.

—
Reply to this email directly or view it on GitHub
#184 (comment).

[BeamLiu]

hi @ganskef , yes, I tested here, the same, I believe the google certificate may have some special and lead to some potential bug in lower level library, however, the browser can recognize all the SAN. I had a testing, to publish 120+ SAN in my local, and then use this proxy, it seems everything works fine. Don't know what is special with the google certificate.

There should be some more bugs in lower level, maybe in netty, JDK, or LittleProxy itself, the reason I need MITM https interception is that I had some jobs are doing in Fiddler, and want to translate it to java program automation using LittleProxy, after my code is done, I am crazy that the original black box client can work well with Fiddler, but cannot establish SSL connection with LittleProxy, I am still doing investigation, I am also doubting it is some bugs in netty, as it failed in handshake stage, no workaround solution yet, so I believe some bugs are expected.