spark集群详细搭建过程及遇到的问题解决

Posted 2020-08-19

（二）配置ssh无密码访问集群机器

master节点

[email protected]:/home# su - spark
[email protected]:~$ 
[email protected]:~$ ssh-keygen -t rsa
#一直enter键

[email protected]:~$ cd .ssh/
[email protected]:~/.ssh$ ls
id_rsa  id_rsa.pub
[email protected]:~/.ssh$  cat id_rsa.pub > authorized_keys
[email protected]:~/.ssh$ scp [email protected]:~/.ssh/id_rsa.pub ./master_rsa.pub

[email protected]:~/.ssh$ ls
authorized_keys  id_rsa  id_rsa.pub  known_hosts  master_rsa.pub
[email protected]:~/.ssh$ cat master_rsa.pub >>authorized_keys

worker1节点

与master节点执行过程相同

worker2节点

与master节点执行过程相同

执行完上述操作之后：

master节点：

[email protected]:~/.ssh$ scp [email protected]:~/.ssh/id_rsa.pub ./worker1_rsa.pub

注意是在./ssh目录下

[email protected]:~/.ssh$ cat worker1_rsa.pub >>authorized_keys 
[email protected]:~/.ssh$ scp [email protected]:~/.ssh/id_rsa.pub ./worker2_rsa.pub

[email protected]:~/.ssh$ cat worker2_rsa.pub >>authorized_keys

worker1节点

[email protected]:~/.ssh$ scp [email protected]:~/.ssh/id_rsa.pub ./worker2_rsa.pub
cat worker2_rsa.pub >>authorized_keys

worker2节点

[email protected]:~/.ssh$ scp [email protected]:~/.ssh/id_rsa.pub ./worker1_rsa.pub
cat worker1_rsa.pub >>authorized_keys

验证是否配置成功

从master节点分别登陆自身、worker1节点与worker2节点（可能第一次需要密码、退出后，第二次重新ssh则不需要密码）

[email protected]:~/.ssh$ ssh master

[email protected]:~/.ssh$ ssh worker1

[email protected]:~/.ssh$ ssh worker2

从worker1节点登陆自身、master节点与worker2节点（可能第一次需要密码、退出后，第二次重新ssh则不需要密码）

[email protected]:~/.ssh$ ssh master

[email protected]:~/.ssh$ ssh worker1

[email protected]:~/.ssh$ ssh worker2

从worker2节点登陆自身、master节点与worker1节点（可能第一次需要密码、退出后，第二次重新ssh则不需要密码）

[email protected]:~/.ssh$ ssh master

[email protected]:~/.ssh$ ssh worker1

[email protected]:~/.ssh$ ssh worker2

注意，登陆到其他节点后，一定要记得退出

下文中，将开始安装hadoop。。。

本文出自 “lefteva” 博客，请务必保留此出处http://lefteva.blog.51cto.com/11892835/1873526