How to solve java.net.SocketTimeoutException:60000millis problem in HDFS

Posted 2020-07-16 白开水加糖

Many HDFS users encounter the following error when DFSClient ready file from a certain Data Node.

        "Java.net.SocketTimeoutException: 60000millis timeout while waiting for channel to be ready for read. ch"

The reason about this error is that the Data Node fails to communicate with DFSClient when DFSClient requests some block from that Data Node, which often occurs when the small cluster conntains too many files and the Data Node with such heavy burden cannot schedules a certain block within the limited time (for instance 60 sec). Then DFSClient waits untill this interval fires, than turn to request the other Data Node containing the same block.

If HDFS is used for in-time service, people wil find the 60 sec timeout is too long and they hope to shorten the interval making the DFSClient switch to other Data Node quickly. When I encounter is problem, I seach the Internet for solution. Depressed, No answeris correct.

Then I debug the source code, finding the timeout value is controlled by the config item "ipc.ping.interval". This item has the default value 60000 (millisec). You can add this config item into you"core-site.xml" file and it will take effert when DFSClient starts.