webmagic创始人回答:ebMagic默认的HttpClient只会用TLSv1去请求,对于某些只支持TLS1.2的站点(例如 https://juejin.im/) ,就会报错:

Posted 张氏码农之旅

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了webmagic创始人回答:ebMagic默认的HttpClient只会用TLSv1去请求,对于某些只支持TLS1.2的站点(例如 https://juejin.im/) ,就会报错:相关的知识,希望对你有一定的参考价值。

见官方文档解决方案:https://github.com/code4craft/webmagic/issues/701

webmagic创始人回答:

 code4craft commented on 29 Nov 2017 •

更新会在0.7.4版本发布。

临时适配方式,修改HttpClientGenerator中的buildSSLConnectionSocketFactory方法,

return new SSLConnectionSocketFactory(createIgnoreVerifySSL(), new String[]{"SSLv3", "TLSv1", "TLSv1.1", "TLSv1.2"},
                    null,
                    new DefaultHostnameVerifier())

 

WebMagic默认的HttpClient只会用TLSv1去请求,对于某些只支持TLS1.2的站点(例如 https://juejin.im/) ,就会报错:

javax.net.ssl.SSLException: Received fatal alert: protocol_version
	at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
	at sun.security.ssl.Alerts.getSSLException(Alerts.java:154)
	at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:2023)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1125)
	at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:394)
	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:353)
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:141)
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at us.codecraft.webmagic.downloader.HttpClientDownloader.download(HttpClientDownloader.java:85)

现在的修改方式是在HttpClientGenerator中构建SSLConnectionSocketFactory时加上支持。

 

以上是关于webmagic创始人回答:ebMagic默认的HttpClient只会用TLSv1去请求,对于某些只支持TLS1.2的站点(例如 https://juejin.im/) ,就会报错:的主要内容,如果未能解决你的问题,请参考以下文章

webmagic自定义存储(mysqlredis存储)

好东西WebMagic 0.7.3 版本发布,Java 爬虫框架

大三上寒假15天--第10天

WebMagic爬虫Demo

WebMagic快速入门

WebMagic