原创大数据基础之Kudukudu增加或删除目录/数据盘

Posted 2021-12-06 barneywill

kudu加减数据盘不能直接修改配置fs_data_dirs后重启，否则会报错：

Check failed: _s.ok() Bad status: Already present: FS layout already exists; not overwriting existing layout: FSManager roots already exist: /data0/kudu/data

官方解释如下：

When Kudu starts, it checks each configured data directory, expecting either for all to be initialized or for all to be empty. If a server fails to start with a log message like
then this precondition has failed. This could be because Kudu was configured with non-empty data directories on first startup, or because a previously-running, healthy Kudu process was restarted and at least one data directory was deleted or is somehow corrupted, perhaps because of a disk error. If in the latter situation, consult the Changing Directory Configurations documentation.

For higher read parallelism and larger volumes of storage per server, users may want to configure servers to store data in multiple directories on different devices. Once a server is started, users must go through the following steps to change the directory configuration.
Users can add or remove data directories to an existing master or tablet server via the kudu fs update_dirs tool. Data is striped across data directories, and when a new data directory is added, new data will be striped across the union of the old and new directories.
The tool can only run while the server is offline, so establish a maintenance window to update the server. The tool itself runs quickly, so this offline window should be brief, and as such, only the server to update needs to be offline. However, if the server is offline for too long (see the follower_unavailable_considered_failed_sec flag), the tablet replicas on it may be evicted from their Raft groups. To avoid this, it may be desirable to bring the entire cluster offline while performing the update.

 

操作过程为：

• 修改kudu配置（增加或删除目录/数据盘），但不重启kudu；
• 逐台操作tserver
  • 　　关闭一台tserver
  • 　　在这台tserver上执行 kudu fs update_dirs
  • 　　执行完成后启动tserver

增加数据盘后使用rebalance命令

kudu cluster rebalance

参考：
https://kudu.apache.org/releases/1.7.0/docs/troubleshooting.html#disk_issues
https://kudu.apache.org/releases/1.7.0/docs/administration.html#change_dir_config