hadoop balancer配置

Posted 2022-04-08 yjt1993

hadoop版本：2.9.2

1、带宽的设置参数： dfs.datanode.balance.bandwidthPerSec   默认值 10m

2、datanode之间数据块的传输线程大小：dfs.datanode.max.transfer.threads 默认值4096

3、修改dfs.datanode.balance.max.concurrent.moves ，指定DataNode上同时用于balance待移动block的最大线程个数，这个值默认是50

如果dfs.datanode.balance.max.concurrent.moves 这个值设置不合理，balancer里面会报IO错误：(我这里设置的有点高了，达到了200)

java.io.IOException: Got error, status=ERROR, status message Not able to receive block 1144718791 from /10.x.x.x:53336 because threads quota is exceeded., block move is failed
at org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:118)
at org.apache.hadoop.hdfs.server.balancer.Dispatcher$PendingMove.receiveResponse(Dispatcher.java:431)
at org.apache.hadoop.hdfs.server.balancer.Dispatcher$PendingMove.dispatch(Dispatcher.java:372)
at org.apache.hadoop.hdfs.server.balancer.Dispatcher$PendingMove.access$3000(Dispatcher.java:230)
at org.apache.hadoop.hdfs.server.balancer.Dispatcher$1.run(Dispatcher.java:1053)

 

 

4、如果不想在datanode设置带宽，可以临时设置 hdfs dfsadmin -setBalancerBandwidth 20m

 

5、启动balancer的方法

(1)、start-balancer.sh -threshold 1

(2)、hadoop balancer -threshold  1

(3)、hadoop-daemon.sh start balancer -threshold 1