Unknown SSL protocol error with HTTPS proxy decryption

[galehar]

Since my company implemented HTTPS proxy decryption through bluecoat SSLVA, git 2.5.1 is unable to connect to the repository with https. I always get the error :

Unknown SSL protocol error in connection to ...

I did install the CA certificate from the company and I set http.sslCAInfo to point to it. The certificate is properly set, but I still get the Unkown SSL protocol error.

• successfully set certificate verify locations:
• CAfile: C:\Program Files\Git\mingw64/ssl/CompanyCA.pem

Setting http.sslVerify to false doesn't fix the problem, still the same issue.

I tried with curl to diagnose, and setting the CA certificate with the cacert option allows me to connect successfully, and so does using the insecure option.

I also tried with an old git version 1.9.4.msysgit.2, and with it, I've also been able to successfully connect by setting sslCAInfo properly or disabling sslVerify.

git 2.5.1 on the other hand doesn't work at all. I tried both 32 and 64 bit versions. This is on Windows 7 SP1 64 bits.

[shiftkey]

@galehar could you try cloning with all the diagnostics enabled? My psychic debugger thinks something network-related is involved:

GIT_TRACE=1; GIT_CURL_VERBOSE=1; git ...

Feel free to strip out sensitive stuff, but there should be some extra info in here to help us troubleshoot further.

[galehar]

Here is a trace of the connection with the sensitive stuff replaced.

git pull
14:17:18.090286 git.c:558 trace: exec: 'git-pull'
14:17:18.091287 run-command.c:347 trace: run_command: 'git-pull'
14:17:18.312309 git.c:348 trace: built-in: git 'rev-parse' '--parseopt' '--stuck-long' '--'
14:17:18.421320 git.c:348 trace: built-in: git 'rev-parse' '--git-dir'
14:17:18.493327 git.c:348 trace: built-in: git 'rev-parse' '--git-path' 'objects'
14:17:18.573335 git.c:348 trace: built-in: git 'rev-parse' '--is-bare-repository'
14:17:18.619339 git.c:348 trace: built-in: git 'rev-parse' '--show-toplevel'
14:17:18.668344 git.c:348 trace: built-in: git 'ls-files' '-u'
14:17:18.715349 git.c:348 trace: built-in: git 'symbolic-ref' '-q' 'HEAD'
14:17:18.869364 git.c:348 trace: built-in: git 'config' 'pull.ff'
14:17:18.914369 git.c:348 trace: built-in: git 'rev-parse' '-q' '--verify' 'HEAD'
14:17:18.956373 git.c:348 trace: built-in: git 'rev-parse' '--verify' 'HEAD'
14:17:18.999377 git.c:348 trace: built-in: git 'update-index' '-q' '--ignore-submodules' '--refresh'
14:17:19.076385 git.c:348 trace: built-in: git 'diff-files' '--quiet' '--ignore-submodules'
14:17:19.151393 git.c:348 trace: built-in: git 'diff-index' '--cached' '--quiet' '--ignore-submodules' 'HEAD' '--'
14:17:19.248402 git.c:348 trace: built-in: git 'rev-parse' '-q' '--git-dir'
14:17:19.575435 git.c:348 trace: built-in: git 'rev-parse' '-q' '--verify' 'HEAD'
14:17:19.616439 git.c:348 trace: built-in: git 'fetch' '--update-head-ok'
14:17:19.619439 run-command.c:347 trace: run_command: 'git-remote-https' 'origin' 'https://mygitserver.company.com/git/repository-name'

• Couldn't find host www.hostname.com in the _netrc file; using defaults
• timeout on name lookup is not supported
• Trying ip_of_proxy...
• Connected to autoconf (ip_of_proxy) port 3128 (#0)
• Establish HTTP proxy tunnel to mygitserver.company.com:443

  CONNECT mygitserver.company.com:443 HTTP/1.1
  Host: mygitserver.company.com:443
  User-Agent: git/2.5.1.windows.1
  Proxy-Connection: Keep-Alive

< HTTP/1.1 407 Proxy Authentication Required
< Proxy-Authenticate: NTLM
< Proxy-Authenticate: BASIC realm="Company_NTLM"
< Cache-Control: no-cache
< Pragma: no-cache
< Content-Type: text/html; charset=utf-8
< Proxy-Connection: close
< Set-Cookie: BCSI-CS-05ae2b15ff1376d7=2; Path=/
< Connection: close
< Content-Length: 1672
<

• Ignore 1672 bytes of response-body
• Connect me again please
• ALPN, offering http/1.1
• Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@strength
• successfully set certificate verify locations:
• CAfile: C:\Program Files\Git\mingw64/ssl/CompanyCA.pem
  CApath: none
• Unknown SSL protocol error in connection to mygitserver.company.com:443
• Closing connection 0
• Couldn't find host mygitserver.company.com in the _netrc file; using defaults
• timeout on name lookup is not supported
• Hostname autoconf was found in DNS cache
• Trying ip_of_proxy...
• Connected to autoconf (ip_of_proxy) port 3128 (Fixes for the new SDK #1)
• Establish HTTP proxy tunnel to mygitserver.company.com:443
• Proxy auth using NTLM with user 'ProxyUserName'

  CONNECT mygitserver.company.com:443 HTTP/1.1
  Host: mygitserver.company.com:443
  Proxy-Authorization: NTLM TlRMTVNTUAABAAAAB4IIogAAAAAAAAAAAAAAAAAAAAAGAbEdAAAADw==
  User-Agent: git/2.5.1.windows.1
  Proxy-Connection: Keep-Alive

[dscho]

Since 1.9.5 works, you could compare the verbose output of both...

[galehar]

Well, I did compare the verbose output, but I didn't learn anything from it. With git 1.9.4, I get "SSL connection using DHE-RSA-AES256-SHA" and with git 2.5.1: "Unknown SSL protocol error in connection to mygitserver.company.com:443". Besides, the problem is not related to https decryption. It has been disabled today, and I can connect with git 1.9.4 without having to mess with sslCAInfo or sslVerify. Still same error with git 2.5.1.
Also, I tried git 1.9.5, and I get the same error as with 2.5.1. Only git 1.9.4 works.

Here is the full log of the successful connection with git 1.9.4:

$ git fetch
trace: built-in: git 'fetch'
trace: run_command: 'git-remote-https' 'origin' 'https://mygitserver.company.com/git/repository-name'

• Couldn't find host mygitserver.company.com in the _netrc file; using defaults
• Adding handle: conn: 0x1e81a48
• Adding handle: send: 0
• Adding handle: recv: 0
• Curl_addHandleToPipeline: length: 1
• - Conn 0 (0x1e81a48) send_pipe: 1, recv_pipe: 0
• About to connect() to proxy autoconf port 3128 (#0)
• Trying ip_of_proxy...
• Connected to autoconf (ip_of_proxy) port 3128 (#0)
• Establish HTTP proxy tunnel to mygitserver.company.com:443

  CONNECT mygitserver.company.com:443 HTTP/1.1
  Host: mygitserver.company.com:443
  User-Agent: git/1.9.4.msysgit.2
  Proxy-Connection: Keep-Alive
  Pragma: no-cache

< HTTP/1.1 407 Proxy Authentication Required
< Proxy-Authenticate: NTLM
< Proxy-Authenticate: BASIC realm="Airbus_NTLM"
< Cache-Control: no-cache
< Pragma: no-cache
< Content-Type: text/html; charset=utf-8
< Proxy-Connection: close
< Set-Cookie: BCSI-CS-eef0d1994df46519=2; Path=/
< Connection: close
< Content-Length: 1672
<

• Ignore 1672 bytes of response-body
• Received HTTP code 407 from proxy after CONNECT
• Couldn't find host mygitserver.company.com in the _netrc file; using defaults
• Found bundle for host mygitserver.company.com: 0x1e64508
• About to connect() to proxy autoconf port 3128 (Fixes for the new SDK #1)
• Trying ip_of_proxy...
• Adding handle: conn: 0x1e93898
• Adding handle: send: 0
• Adding handle: recv: 0
• Curl_addHandleToPipeline: length: 1
• - Conn 0 (0x1e81a48) send_pipe: 1, recv_pipe: 0
• - Conn 1 (0x1e93898) send_pipe: 1, recv_pipe: 0
• Connected to autoconf (ip_of_proxy) port 3128 (Fixes for the new SDK #1)
• Establish HTTP proxy tunnel to mygitserver.company.com:443
• Proxy auth using NTLM with user 'ProxyUserName'

  CONNECT mygitserver.company.com:443 HTTP/1.1
  Host: mygitserver.company.com:443
  Proxy-Authorization: NTLM TlRMTVNTUAABAAAAt4II4gAAAAAAAAAAAAAAAAAAAAAGAbEdAAAADw
  ==
  User-Agent: git/1.9.4.msysgit.2
  Proxy-Connection: Keep-Alive
  Pragma: no-cache

< HTTP/1.1 407 Proxy Authentication Required
< Proxy-Authenticate: NTLM TlRMTVNTUAACAAAABAAEADgAAAA1goniglnfoWEAazAAAAAAAAAAA
JwAnAA8AAAABQLODgAAAA9FAFUAAgAEAEUAVQABABgARgBSADAALQBIAEkAUABFAC0AUAAwADIABAAcA
GUAdQAuAGEAaQByAGIAdQBzAC4AYwBvAHIAcAADADYAZgByADAALQBoAGkAcABlAC0AcAAwADIALgBlA
HUALgBhAGkAcgBiAHUAcwAuAGMAbwByAHAABQAWAGEAaQByAGIAdQBzAC4AYwBvAHIAcAAAAAAA
< Cache-Control: no-cache
< Pragma: no-cache
< Content-Type: text/html; charset=utf-8
< Proxy-Connection: Keep-Alive
< Set-Cookie: BCSI-CS-eef0d1994df46519=2; Path=/
< Connection: Keep-Alive
< Content-Length: 1723
<

• Ignore 1723 bytes of response-body
• TUNNEL_STATE switched to: 0
• Establish HTTP proxy tunnel to mygitserver.company.com:443
• Proxy auth using NTLM with user 'ProxyUserName'

  CONNECT mygitserver.company.com:443 HTTP/1.1
  Host: mygitserver.company.com:443
  Proxy-Authorization: NTLM TlRMTVNTUAADAAAAGAAYAIAAAAAYARgBmAAAAAAAAABYAAAADgAOAF
  gAAAAaABoAZgAAABAAEACwAQAANYKI4gYBsR0AAAAPMlKWX0k8kdZcXg8bfalwTW4AZwAxADIANgBjAD
  gAVABPAEwAWAAzADAAMAAwADgANgAzADcANgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADF6Ngb5akgVV
  EgQHSr5iu6AQEAAAAAAABYbl608/HQAbgPzHO2QWF6AAAAAAIABABFAFUAAQAYAEYAUgAwAC0ASABJAF
  AARQAtAFAAMAAyAAQAHABlAHUALgBhAGkAcgBiAHUAcwAuAGMAbwByAHAAAwA2AGYAcgAwAC0AaABpAH
  AAZQAtAHAAMAAyAC4AZQB1AC4AYQBpAHIAYgB1AHMALgBjAG8AcgBwAAUAFgBhAGkAcgBiAHUAcwAuAG
  MAbwByAHAACAAwADAAAAAAAAAAAQAAAAAgAACEt1DOFPJM4ledS5wPhxxN9Owm2QS6fh+YVRXucngZBQ
  oAEAAAAAAAAAAAAAAAAAAAAAAACQAAAAAAAAAAAAAAr9QCeUrjDrzmdBsMLt3l+Q==
  User-Agent: git/1.9.4.msysgit.2
  Proxy-Connection: Keep-Alive
  Pragma: no-cache

< HTTP/1.1 200 Connection established
<

• Proxy replied OK to CONNECT request
• successfully set certificate verify locations:
• CAfile: C:\Users\ProxyUserName\Documents\GitPortable\App\Git/bin/curl-ca-bundle.crt
  CApath: none
• SSL connection using DHE-RSA-AES256-SHA
• Server certificate:

•    subject: stripped certificate info
  

•    start date: 2014-10-15 00:00:00 GMT
  

•    expire date: 2015-10-15 23:59:59 GMT
  

•    subjectAltName: mygitserver.company.com matched
  

•    issuer: stripped issuer certificate info
  

•    SSL certificate verify ok.
  

  GET /git/repository-name/info/refs?service=git-upload-pack HTTP/1.1
  User-Agent: git/1.9.4.msysgit.2
  Host: mygitserver.company.com
  Accept: /
  Accept-Encoding: gzip
  Pragma: no-cache

< HTTP/1.1 401 Authorization Required

• Server nginx is not blacklisted
  < Server: nginx
  < Date: Fri, 18 Sep 2015 09:23:40 GMT
  < Content-Type: text/html; charset=iso-8859-1
  < Content-Length: 495
  < Connection: keep-alive
  < WWW-Authenticate: Basic realm="Git Authentication"
  <
• Connection Fixes for the new SDK #1 to host autoconf left intact
  trace: run_command: 'git credential-wincred get'
  trace: exec: 'git-credential-wincred' 'get'
  trace: run_command: 'git-credential-wincred' 'get'
• Couldn't find host mygitserver.company.com in the _netrc file; using defaults
• Found bundle for host mygitserver.company.com: 0x1e64508
• Re-using existing connection! (Fixes for the new SDK #1) with host autoconf
• Connected to autoconf (ip_of_proxy) port 3128 (Fixes for the new SDK #1)
• Adding handle: conn: 0x1e93898
• Adding handle: send: 0
• Adding handle: recv: 0
• Curl_addHandleToPipeline: length: 1
• - Conn 0 (0x1e81a48) send_pipe: 1, recv_pipe: 0
• - Conn 1 (0x1e93898) send_pipe: 1, recv_pipe: 0

  GET /git/repository-name/info/refs?service=git-upload-pack HTTP/1.1
  User-Agent: git/1.9.4.msysgit.2
  Host: mygitserver.company.com
  Accept: /
  Accept-Encoding: gzip
  Pragma: no-cache

< HTTP/1.1 401 Authorization Required

• Server nginx is not blacklisted
  < Server: nginx
  < Date: Fri, 18 Sep 2015 09:23:40 GMT
  < Content-Type: text/html; charset=iso-8859-1
  < Content-Length: 495
  < Connection: keep-alive
  < WWW-Authenticate: Basic realm="Git Authentication"
  <
• Ignoring the response-body
• Connection Fixes for the new SDK #1 to host autoconf left intact
• Issue another request to this URL: 'https://mygitserver.company.com/git/repository-name/info/refs?service=git-upload-pack'
• Couldn't find host mygitserver.company.com in the _netrc file; using defaults
• Found bundle for host mygitserver.company.com: 0x1e64508
• Re-using existing connection! (Fixes for the new SDK #1) with host autoconf
• Connected to autoconf (ip_of_proxy) port 3128 (Fixes for the new SDK #1)
• Adding handle: conn: 0x1e93898
• Adding handle: send: 0
• Adding handle: recv: 0
• Curl_addHandleToPipeline: length: 1
• - Conn 0 (0x1e81a48) send_pipe: 1, recv_pipe: 0
• - Conn 1 (0x1e93898) send_pipe: 1, recv_pipe: 0
• Server auth using Basic with user 'rlangella'

  GET /git/repository-name/info/refs?service=git-upload-pack HTTP/1.1
  Authorization: Basic cmxhbmdlbGxhOnJsYW5nZWxsYQ==
  User-Agent: git/1.9.4.msysgit.2
  Host: mygitserver.company.com
  Accept: /
  Accept-Encoding: gzip
  Pragma: no-cache

< HTTP/1.1 200 OK

• Server nginx is not blacklisted
  < Server: nginx
  < Date: Fri, 18 Sep 2015 09:23:41 GMT
  < Content-Type: application/x-git-upload-pack-advertisement
  < Transfer-Encoding: chunked
  < Connection: keep-alive
  < Expires: Fri, 01 Jan 1980 00:00:00 GMT
  < Pragma: no-cache
  < Cache-Control: no-cache, max-age=0, must-revalidate
  < X-Frame-Options: ALLOW
  < Front-End-Https: on
  < Strict-Transport-Security: max-age=15768000
  < X-Frame-Options: SAMEORIGIN
  < X-UA-Compatible: IE=EmulateIE8
  <
• Connection Fixes for the new SDK #1 to host autoconf left intact
  trace: run_command: 'git credential-wincred store'
  trace: exec: 'git-credential-wincred' 'store'
  trace: run_command: 'git-credential-wincred' 'store'
  trace: run_command: 'rev-list' '--objects' '--stdin' '--not' '--all' '--quiet'
  trace: run_command: 'rev-list' '--objects' '--stdin' '--not' '--all'
  trace: built-in: git 'rev-list' '--objects' '--stdin' '--not' '--all'
  trace: run_command: 'gc' '--auto'
  trace: built-in: git 'gc' '--auto'

[dscho]

Unknown SSL protocol error in connection to mygitserver.company.com:443

Without SSL, you certainly do not get this error with 1.9.5 and 2.5.1. What do you get instead?

[galehar]

What do you mean without SSL? I can't disable SSL, I can only connect to the repository via https. The https decryption module for the proxy has been disabled, so it's not messing with certificates anymore. I can set sslVerify to false to disable certificate verification, but I still have to use SSL.

[dscho]

What do you mean without SSL?

Well, I took the meaning of

Besides, the problem is not related to https decryption. It has been disabled today

too literal, I guess.

[dscho]

The best strategy from here would be to reduce the problem. If you can test with a host that does not need to go through the proxy, you could find out whether it is actually the proxy that does bad things. Try to think of ways to take out as much as possible, until you hit a spot where taking away one block "fixes" things. That way, you could identify the real culprits better and we have a chance of finding a solution for you.

[dscho]

@galehar any luck yet with the investigation?

[galehar]

Well, I tested from another network with a different proxy, and it worked fine. It was a different computer, but with the same OS and same git version.
I also tested another repository on github, and I could clone it without any problem.
So, in the end, it seems that the problem arises from a combination of factors from the proxy and the git repository (or maybe the reverse proxy in front of it?).
Unfortunately, I don't have access to neither, some I'm not sure how to procede from there. Maybe I could ask for repository or reverse proxy log. I might be able to try a network trace, but I'm not sure I'll be able to interpret the result.

[dscho]

You could test with curl on the machine that has the problem. And then you could also set up your own server and force the proxy to connect to it and see whether the problem still occurs.

It is possible that the proxy gets confused with WebDAV commands...

[soltmar]

Hi,

This is related to my original post here:

msysgit/msysgit#23 (comment)

I have exactly same issue.

Everything is working correctly in 1.9.0 and 1.9.5 git versions but not with latest one ( 2.5.3 )

I've got exactly the same CURL Verbose output as @galehar

Resulting in "Unknown SSL protocol error in connection to bitbucket:443" (Original output in the link
above)
I wasn't able to resolve that in any way.

Connection to my proxy is going through http (we don't have https proxy set).

I've got Env Vars set in this way:

HTTP_PROXY = http://user:pass@proxy.com:8080
HTTPS_PROXY = http://user:pass@proxy.com:8080

Please note http in HTTPS_PROXY.

This configuration works perfectly for 1.9 and 1.9.5 as mentioned on the beginning.

Any ideas ?

[dscho]

I tried with curl to diagnose, and setting the CA certificate with the cacert option allows me to connect successfully

Could you compare the verbose output of curl.exe with the verbose output of the git fetch call?