安装CDH出现file /opt/cloudera/parcels/.flood/CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel...does not exist

Posted 2022-05-10 黄智霖-blog

使用Cloudera Manager采取parcel方式安装CDH的时候，部分agent节点出现异常：

Src file /opt/cloudera/parcels/.flood/CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel/CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel does not exist

查看对应agent的日志文件查看异常信息：

[root@cdh-01 parcel-cache]# tail -f /var/log/cloudera-scm-agent/cloudera-scm-agent.log

发现日志里有大量的如下输出：
 

20/May/2019 21:32:17 +0000] 8949 Thread-13 downloader   INFO     Fetching torrent: http://cdh-03.echatsoft.com:7180/cmf/parcel/download/CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel.torrent
[20/May/2019 21:32:17 +0000] 8949 Thread-13 downloader   INFO     Starting download of: http://cdh-03.echatsoft.com:7180/cmf/parcel/download/CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel
[20/May/2019 21:32:17 +0000] 8949 Thread-13 downloader   INFO     Failed adding torrent: file:///opt/cloudera/parcel-cache/CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel.torrent Already present torrent: CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel
[20/May/2019 21:32:17 +0000] 8949 Thread-13 downloader   INFO     Current state: CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel [totalDownloaded=2087665645 totalSize=2087665645 upload=322898691 state=seeding seed=['http://cdh-03.echatsoft.com:7180/cmf/parcel/download/CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel'] location=/opt/cloudera/parcels/.flood/CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel progress=1000000]
[20/May/2019 21:32:17 +0000] 8949 Thread-13 downloader   INFO     Completed download of http://cdh-03.echatsoft.com:7180/cmf/parcel/download/CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel code=200 state=downloaded
[20/May/2019 21:32:17 +0000] 8949 Thread-13 parcel_cache WARNING  No checksum in header, skipping verification
[20/May/2019 21:32:17 +0000] 8949 Thread-13 parcel_cache INFO     Unpacking /opt/cloudera/parcels/.flood/CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel/CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel into /opt/cloudera/parcels
[20/May/2019 21:32:17 +0000] 8949 Thread-13 downloader   ERROR    Failed op: Src file /opt/cloudera/parcels/.flood/CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel/CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel does not exist

也就是提示Src file /opt/cloudera/parcels/.flood/CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel/CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel不存在，但是parcel应该从server节点下载下来了才对，为什么提示不存在？想不通，于是停止了该agent服务：

[root@cdh-01 parcel-cache]# systemctl stop cloudera-scm-agent

然后清理该agent节点上的parcel缓存(目录默认在 /opt/cloudera/)：

然后重启该agent服务，跟踪完整的日志输出，以确认出现问题的地方(不用重启server节点 ，agent节点重启后会主动向server节点通信，然后开始安装的流程)：

[root@cdh-01 parcel-cache]# systemctl start cloudera-scm-agent
[root@cdh-01 parcel-cache]# tail -f /var/log/cloudera-scm-agent/cloudera-scm-agent.log

启动成功之后首先是Fetching torrent，然后开始下载：

[20/May/2019 19:51:18 +0000] 8949 Thread-13 downloader   INFO     Fetching torrent: http://cdh-03.echatsoft.com:7180/cmf/parcel/download/CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel.torrent
[20/May/2019 19:51:19 +0000] 8949 Thread-13 downloader   INFO     Starting download of: http://cdh-03.echatsoft.com:7180/cmf/parcel/download/CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel
......

[20/May/2019 19:51:31 +0000] 8949 Thread-13 downloader   INFO     Current state: CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel [totalDownloaded=94896207 totalSize=2087665645 upload=16947965 state=downloading seed=[] location=/opt/cloudera/parcels/.flood/CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel progress=45455]
[20/May/2019 19:51:35 +0000] 8949 Thread-13 downloader   INFO     Current state: CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel [totalDownloaded=95420495 totalSize=2087665645 upload=17018436 state=downloading seed=[] location=/opt/cloudera/parcels/.flood/CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel progress=45706]
[20/May/2019 19:51:37 +0000] 8949 Thread-13 downloader   INFO     Current state: CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel [totalDownloaded=101712030 totalSize=2087665645 upload=17018436 state=downloading seed=[] location=/opt/cloudera/parcels/.flood/CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel progress=48720]
[20/May/2019 19:51:39 +0000] 8949 Thread-13 downloader   INFO     Current state: CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel [totalDownloaded=115343360 totalSize=2087665645 upload=18874368 state=downloading seed=[] location=/opt/cloudera/parcels/.flood/CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel progress=55249]
[20/May/2019 19:51:41 +0000] 8949 Thread-13 downloader   INFO     Current state: CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel [totalDownloaded=116916224 totalSize=2087665645 upload=23068672 state=downloading seed=[] location=/opt/cloudera/parcels/.flood/CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel progress=56003]
......

下载成功之后，开始解压：

[20/May/2019 20:07:23 +0000] 8949 Thread-13 parcel_cache INFO     Unpacking /opt/cloudera/parcels/.flood/CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel/CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel into /opt/cloudera/parcels

这些流程都是正常进行的，下载也是成功了的，但是发现在解压一段时间之后就抛出了异常：
 

[20/May/2019 20:41:43 +0000] 8949 Thread-13 throttling_logger ERROR    Error while unpacking /opt/cloudera/parcels/.flood/CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel/CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel
Traceback (most recent call last):
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/parcel_cache.py", line 194, in cb
    unpack_parcel(op.path, unpack_dir, self.path)
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/parcel_cache.py", line 220, in unpack_parcel
    raise Exception("Untar failed with return code: %d, with tar output:\\n stdout: [%s],\\n stderr: [%s]" % (return_code, output[0], output[1]))
Exception: Untar failed with return code: 2, with tar output:
 stdout: [],
 stderr: [tar: CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/kudu/sbin-release/kudu-tserver: Wrote only 2048 of 10240 bytes
tar: CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/kudu/sbin-release/kudu-master: Cannot open: No space left on device
tar: CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/kudu/sbin-debug: Cannot mkdir: No space left on device
tar: CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/kudu/sbin-debug: Cannot mkdir: No space left on device
tar: CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/kudu/sbin-debug/kudu-tserver: Cannot open: No such file or directory
tar: CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/kudu/sbin-debug: Cannot mkdir: No space left on device
tar: CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/kudu/sbin-debug/kudu-master: Cannot open: No such file or directory
......

提示磁盘空间不足o(╯□╰)o。通过df命令查看，果然磁盘空间不够了，最后通过清理磁盘解决。

注：由于parcel文件较大（将近两个G），agent从server下载下来后还需要解压，然后再激活，所以我们要保证充足的磁盘空间，特别是用虚拟机搭建集群的同学。

创作打卡挑战赛 赢取流量/现金/CSDN周边激励大奖