0470-?????????Redhat7.4??????CDH5.16.1
Posted
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了0470-?????????Redhat7.4??????CDH5.16.1相关的知识，希望对你有一定的参考价值。
?????????oracle from ?????? ?????? tp????????? ever ?????? scp rom
Fayson???github??? https://github.com/fayson/cdhproject
?????????????????????????????????Hadoop????????????ID???gh_c4c535955d0f?????????????????????????????????
1.??????????????????
Cloudera???2018???11???29????????????CDH5.16.1??????????????????????????????Fayson??????????????????0466-CDH5.16.1???CM5.16.1????????????????????????????????????????????????Redhat7.4??????CDH5.16.1???Cloudera????????????????????????????????????4????????????
1.????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????Fayson?????????????????????CDH????????????????????????
2.??????Cloudera Manager???
3.??????Cloudera Manager???Web UI???????????????CDH?????????
4.??????????????????????????????HDFS???????????????MapReduce???Hive??????????????????????????????
???????????????????????????????????????
1.?????????????????????Redhat7.4
2.CDH???CM????????????5.16.1
3.??????root?????????????????????
4.???????????????CDH???CM????????????
2.????????????
2.1.hostname???hosts??????
??????????????????????????????????????????????????????IP?????????IP????????????????????????/etc/hosts????????????????????????/etc/hostname???????????????
???cm?????????172.31.13.38????????????
- hostname??????
/etc/hostname???????????????
ip-172-31-13-38.ap-southeast-1.compute.internal?????????????????????????????????????????????
[[email protected] ~]$ hostnamectl set-hostname ip-172-31-13-38.ap-southeast-1.compute.internal<font face="????????????" size=4 color=red >?????????????????????hostname???REDHAT6?????????
- hosts??????
/etc/hosts????????????:
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
172.31.13.38 ip-172-31-13-38.ap-southeast-1.compute.internal
172.31.11.232 ip-172-31-11-232.ap-southeast-1.compute.internal
172.31.11.9 ip-172-31-11-9.ap-southeast-1.compute.internal
172.31.13.166 ip-172-31-13-166.ap-southeast-1.compute.internal????????????????????????????????????????????????????????????????????????????????????4????????????hosts?????????
2.2.??????SELinux
?????????????????????setenforce 0 ??????????????????????????????shell?????????
[[email protected] ~]$ sh ssh_do_all.sh node.list "setenforce 0"
????????????????????????/etc/selinux/config???????????????
SELINUX=disabled
SELINUXTYPE=targeted??????????????????????????????????????????
sh ssh_do_all.sh node.list "cat /etc/selinux/config | grep SELINUX"
2.3.???????????????
????????????????????????systemctl stop firewalld?????????????????????shell???????????????????????????
[[email protected] ~]$ sh ssh_do_all.sh node.list "systemctl stop firewalld"
[[email protected] ~]$ sh ssh_do_all.sh node.list "systemctl disable firewalld"
[[email protected] ~]$ sh ssh_do_all.sh node.list "systemctl status firewalld"
2.4.??????????????????
???Redhat7.x?????????????????????????????????????????????chrony????????????????????????chrony???????????????ntp?????????ntp??????????????????????????????????????????cm???172.31.13.38?????????????????????ntp??????????????????3?????????????????????????????????
1.??????????????????chrony
[[email protected] shell]# sh ssh_do_all.sh node.list "yum -y remove chrony"
2.??????????????????ntp
[[email protected] shell]# sh ssh_do_all.sh node.list "yum -y install ntp"
3.cm?????????????????????????????????
[[email protected] shell]# vim /etc/ntp.conf
#server 0.rhel.pool.ntp.org iburst
#server 1.rhel.pool.ntp.org iburst
#server 2.rhel.pool.ntp.org iburst
#server 3.rhel.pool.ntp.org iburst
server 127.127.1.0 # local clock
fudge 127.127.1.0 stratum 10
4.??????????????????????????????cm???????????????
#server 0.rhel.pool.ntp.org iburst
#server 1.rhel.pool.ntp.org iburst
#server 2.rhel.pool.ntp.org iburst
#server 3.rhel.pool.ntp.org iburst
server 172.31.13.38
5.?????????????????????ntp??????
[[email protected] shell]# sh ssh_do_all.sh node.list "systemctl restart ntpd"
[[email protected] shell]# sh ssh_do_all.sh node.list "systemctl status ntpd"
[[email protected] shell]# sh ssh_do_all.sh node.list "systemctl enable ntpd"
6.??????????????????????????????????????????ntpq -p???????????????????????????????????????
[[email protected] shell]# sh ssh_do_all.sh node.list "ntpq -p"
????????????*????????????????????????
2.5.??????swap
??????????????????
[[email protected] shell]# sh ssh_do_all.sh node.list "echo vm.swappiness = 10 >> /etc/sysctl.conf"
[[email protected] shell]# sh ssh_do_all.sh node.list "sysctl vm.swappiness=10"
2.6.?????????????????????
?????????????????????
[[email protected] shell]# sh ssh_do_all.sh node.list "echo never > /sys/kernel/mm/transparent_hugepage/defrag "
[[email protected] shell]# sh ssh_do_all.sh node.list "echo never > /sys/kernel/mm/transparent_hugepage/enabled"
?????????????????????
????????????????????????/etc/rc.d/rc.local?????????
if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
echo never > /sys/kernel/mm/transparent_hugepage/enabled
fi
if test -f /sys/kernel/mm/transparent_hugepage/defrag; then
echo never > /sys/kernel/mm/transparent_hugepage/defrag
fi?????????????????????
2.7.??????????????????repo
<font face="????????????" size=4 color=red >Fayson?????????AWS??????????????????????????????????????????????????????????????????????????????????????????
- ??????????????????iso??????
[[email protected] ~]$ sudo mkdir /media/DVD1
[[email protected] ~]$ sudo mount -o loop
CentOS-7-x86_64-DVD-1611.iso /media/DVD1/
- ??????????????????repo
[[email protected] ~]$ sudo vim /etc/yum.repos.d/local_os.repo
[local_iso]
name=CentOS-$releasever - Media
baseurl=file:///media/DVD1
gpgcheck=0
enabled=1
[[email protected] ~]$ sudo yum repolist2.8.??????http??????
- ??????httpd??????
[[email protected] ~]$ yum -y install httpd- ???????????????httpd??????
[[email protected] ~]$ systemctl start httpd
[[email protected] ~]$ systemctl stop httpd- ?????????httpd??????????????????????????????repo?????????http?????????????????????????????????????????????
[[email protected] ~]$ sudo mkdir /var/www/html/iso
[[email protected] ~]$ sudo scp -r /media/DVD1/* /var/www/html/iso/
[[email protected] ~]$ sudo vim /etc/yum.repos.d/os.repo
[osrepo]
name=os_repo
baseurl=http://172.31.2.159/iso/
enabled=true
gpgcheck=false
[[email protected] ~]$ sudo yum repolist2.9.??????MariaDB
1.??????MariaDB
[[email protected] ~]# yum -y install mariadb
[[email protected] ~]# yum -y install mariadb-server
2.???????????????MariaDB
[[email protected] ~]# systemctl start mariadb
[[email protected] ~]# systemctl enable mariadb
[[email protected] ~]# /usr/bin/mysql_secure_installation
NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB
SERVERS IN PRODUCTION USE! PLEASE READ EACH STEP CAREFULLY!
In order to log into MariaDB to secure it, we???ll need the current
password for the root user. If you???ve just installed MariaDB, and
you haven???t set the root password yet, the password will be blank,
so you should just press enter here.
Enter current password for root (enter for none):
OK, successfully used password, moving on...
Setting the root password ensures that nobody can log into the MariaDB
root user without the proper authorisation.
Set root password? [Y/n] Y
New password:
Re-enter new password:
Password updated successfully!
Reloading privilege tables..
... Success!
By default, a MariaDB installation has an anonymous user, allowing anyone
to log into MariaDB without having to have a user account created for
them. This is intended only for testing, and to make the installation
go a bit smoother. You should remove them before moving into a
production environment.
Remove anonymous users? [Y/n] Y
... Success!
Normally, root should only be allowed to connect from ???localhost???. This
ensures that someone cannot guess at the root password from the network.
Disallow root login remotely? [Y/n] n
... skipping.
By default, MariaDB comes with a database named ???test??? that anyone can
access. This is also intended only for testing, and should be removed
before moving into a production environment.
Remove test database and access to it? [Y/n] Y
- Dropping test database...
... Success!
- Removing privileges on test database...
... Success!
Reloading the privilege tables will ensure that all changes made so far
will take effect immediately.
Reload privilege tables now? [Y/n] Y
... Success!
Cleaning up...
All done! If you???ve completed all of the above steps, your MariaDB
installation should now be secure.
Thanks for using MariaDB!
3.??????CM???Hive???????????????
[[email protected] ~]# mysql -u root -p
Enter password:
Welcome to the MariaDB monitor. Commands end with ; or g.
Your MariaDB connection id is 9
Server version: 5.5.56-MariaDB MariaDB Server
Copyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others.
Type ???help;??? or ???h??? for help. Type ???c??? to clear the current input statement.
MariaDB [(none)]>
create database metastore default character set utf8; CREATE USER ???hive???@???%??? IDENTIFIED BY ???password???; GRANT ALL PRIVILEGES ON metastore. * TO ???hive???@???%???; FLUSH PRIVILEGES; create database cm default character set utf8; CREATE USER ???cm???@???%??? IDENTIFIED BY ???password???; GRANT ALL PRIVILEGES ON cm. * TO ???cm???@???%???; FLUSH PRIVILEGES; create database am default character set utf8; CREATE USER ???am???@???%??? IDENTIFIED BY ???password???; GRANT ALL PRIVILEGES ON am. * TO ???am???@???%???; FLUSH PRIVILEGES; create database rm default character set utf8; CREATE USER ???rm???@???%??? IDENTIFIED BY ???password???; GRANT ALL PRIVILEGES ON rm. * TO ???rm???@???%???; FLUSH PRIVILEGES;
create database hue default character set utf8; CREATE USER ???hue???@???%??? IDENTIFIED BY ???password???; GRANT ALL PRIVILEGES ON hue. * TO ???hue???@???%???; FLUSH PRIVILEGES;
create database oozie default character set utf8; CREATE USER ???oozie???@???%??? IDENTIFIED BY ???password???; GRANT ALL PRIVILEGES ON oozie. * TO ???oozie???@???%???; FLUSH PRIVILEGES;
create database sentry default character set utf8; CREATE USER ???sentry???@???%??? IDENTIFIED BY ???password???; GRANT ALL PRIVILEGES ON sentry. * TO ???sentry???@???%???; FLUSH PRIVILEGES;
create database nav_ms default character set utf8; CREATE USER ???nav_ms???@???%??? IDENTIFIED BY ???password???; GRANT ALL PRIVILEGES ON nav_ms. * TO ???nav_ms???@???%???; FLUSH PRIVILEGES;
create database nav_as default character set utf8;
CREATE USER ???nav_as???@???%??? IDENTIFIED BY ???password???;
GRANT ALL PRIVILEGES ON nav_as. * TO ???nav_as???@???%???;
FLUSH PRIVILEGES;
- ??????jdbc??????
[[email protected] ~]# mkdir -p /usr/share/java/
[[email protected] ~]# mv mysql-connector-java-5.1.34.jar /usr/share/java/
[[email protected] ~]# cd /usr/share/java
[[email protected] java]# ln -s mysql-connector-java-5.1.34.jar mysql-connector-java.jar
[[email protected] java]# ll
total 940
-rwxrwxr-x. 1 root root 960372 Feb 1 08:31 mysql-connector-java-5.1.34.jar
lrwxrwxrwx 1 root root 31 Feb 2 00:52 mysql-connector-java.jar -> mysql-connector-java-5.1.34.jar
3.Cloudera Manager??????
3.1.????????????repo???
1.??????CM5.16.1???????????????????????????
http://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5.16.1/RPMS/x86_64/cloudera-manager-agent-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm
http://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5.16.1/RPMS/x86_64/cloudera-manager-daemons-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm
http://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5.16.1/RPMS/x86_64/cloudera-manager-server-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm
http://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5.16.1/RPMS/x86_64/cloudera-manager-server-db-2-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm
http://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5.16.1/RPMS/x86_64/enterprise-debuginfo-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm
http://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5.16.1/RPMS/x86_64/jdk-6u31-linux-amd64.rpm
http://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5.16.1/RPMS/x86_64/oracle-j2sdk1.7-1.7.0+update67-1.x86_64.rpm
2.??????CDH5.16.1???????????????????????????
http://archive.cloudera.com/cdh5/parcels/5.16.1/CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel
http://archive.cloudera.com/cdh5/parcels/5.16.1/CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel.sha1
http://archive.cloudera.com/cdh5/parcels/5.16.1/manifest.json
3.???Cloudera Manager???????????????7???rpm????????????????????????????????????????????????createrepo????????????rpm????????????
[[email protected] cm5.16.1]# ll
total 1019160
-rw-r--r-- 1 root root 9864584 Nov 27 14:40 cloudera-manager-agent-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm
-rw-r--r-- 1 root root 789872988 Nov 27 14:40 cloudera-manager-daemons-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm
-rw-r--r-- 1 root root 8704 Nov 27 14:40 cloudera-manager-server-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm
-rw-r--r-- 1 root root 10612 Nov 27 14:40 cloudera-manager-server-db-2-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm
-rw-r--r-- 1 root root 30604172 Nov 27 14:40 enterprise-debuginfo-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm
-rw-r--r-- 1 root root 71204325 Nov 27 14:40 jdk-6u31-linux-amd64.rpm
-rw-r--r-- 1 root root 142039186 Nov 27 14:40 oracle-j2sdk1.7-1.7.0+update67-1.x86_64.rpm
[[email protected] cm5.16.1]# createrepo .
Spawning worker 0 with 2 pkgs
Spawning worker 1 with 2 pkgs
Spawning worker 2 with 2 pkgs
Spawning worker 3 with 1 pkgs
Workers Finished
Saving Primary metadata
Saving file lists metadata
Saving other metadata
Generating sqlite DBs
Sqlite DBs complete
4.??????Web?????????
?????????cdh5.16.1/cm5.16.1???????????????/var/www/html?????????, ????????????????????????HTTP????????????rpm??????
[[email protected] ~]# mv cm5.16.1/ cdh5.16.1/ /var/www/html/
?????????????????????????????????
5.??????Cloudera Manager???repo???
[[email protected] ~]# vim /etc/yum.repos.d/cm.repo
[cmrepo]
name = cm_repo
baseurl = http://172.31.13.38/cm5.16.1
enable = true
gpgcheck = false
[[email protected] yum.repos.d]# yum repolist
Loaded plugins: amazon-id, rhui-lb, search-disabled-repos
repo id repo name status
cmrepo cm_repo 7
rhui-REGION-client-config-server-7/x86_64 Red Hat Update Infrastructure 2.0 Client C 1
rhui-REGION-rhel-server-releases/7Server/x86_64 Red Hat Enterprise Linux Server 7 (RPMs) 20,668
rhui-REGION-rhel-server-rh-common/7Server/x86_64 Red Hat Enterprise Linux Server 7 RH Commo 233
repolist: 20,909
6.????????????JDK
yum -y install oracle-j2sdk1.7-1.7.0+update67-1
3.2.??????Cloudera Manager Server
1.??????yum??????Cloudera Manager Server
yum -y install cloudera-manager-server
2.??????????????????
[[email protected] yum.repos.d]# /usr/share/cmf/schema/scm_prepare_database.sh mysql cm cm password
JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera
Verifying that we can write to /etc/cloudera-scm-server
Creating SCM configuration file in /etc/cloudera-scm-server
Executing: /usr/java/jdk1.7.0_67-cloudera/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/usr/share/cmf/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.
[ main] DbCommandExecutor INFO Successfully connected to database.
All done, your SCM database is configured correctly!
3.??????Cloudera Manager Server
[[email protected] ~]# systemctl start cloudera-scm-server
4.????????????????????????
[[email protected] ~]# netstat -lnpt | grep 7180
tcp 0 0 0.0.0.0:7180 0.0.0.0:* LISTEN 3331/java
5.??????http://13.250.14.115:7180/cmf/login??????CM
4.CDH??????
4.1.CDH??????????????????
1.admin/admin?????????CM
2.??????license?????????????????????
3.??????60?????????????????????
4.??????????????????
5.????????????ip??????????????????????????????????????????????????????
6.??????????????????
????????????????????????CDH????????????????????????????????????????????????Cloudera???????????????????????????????????????????????????CDH???????????????????????????
7.??????parcel?????????????????????????????????,?????????-????????????????????????????????????
http://172.31.13.38/cdh5.16.1???????????????????????????
???????????????????????????????????????????????????????????????????????????http???CDH5.16.1???????????????????????????????????????http???????????????????????????????????????????????????????????????
8.?????????????????????????????????cm???http??????
9.??????????????????????????????????????????jdk
10.??????????????????????????????????????????????????????????????????????????????????????????
11.??????????????????????????????????????????ssh????????????
12.?????????????????????????????????????????????Cloudera Manager?????????????????????
13.??????????????????????????????????????????cdh???????????????
14.?????????????????????????????????????????????????????????????????????????????????
???????????????????????????????????????
4.2.????????????????????????
1.???????????????????????????
2.?????????????????????????????????????????????
3.????????????????????????????????????????????????????????????
4.??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
5.?????????????????????????????????????????????
6.????????????
7.?????????????????????CM??????????????????
5.Kudu??????
CDH???5.13?????????????????????Kudu??????????????????????????????Parcel???csd?????????CDH5.13????????????Kudu1.5???CDH5.16.1????????????Kudu1.7???
1.??????CM??????Kudu1.7
??????Kudu??????
2.??????Master???Tablet Server
3.???????????????????????????????????????Master??????Tablet??????????????????????????????(fs_data_dir)???????????????????????????????????????????????????????????????Kudu?????????
4.??????Kudu??????
5.????????????
6.??????Impala
??????????????????Impala???Hue
?????????CDH5.16.1+Kudu1.7???????????????
???????????????????????????????????????Fayson?????????????????????????????????????????????????????????Fayson??????????????????
???CENTOS7.2??????CDH5.10???Kudu1.2(???)???
???CENTOS7.2??????CDH5.10???Kudu1.2(???)???
???CENTOS6.5??????CDH5.12.1(???)???
???CENTOS6.5??????CDH5.12.1(???)???
???CDH?????????????????????
???CentOS6.5??????CDH5.13???
??????????????????????????????????????????????????????????????????????????????
????????????Hadoop????????????????????????????????????Hadoop?????????????????????????????????
????????????????????????????????????????????????????????????????????????Hadoop??????
<div align=center><img src="https://img-blog.csdnimg.cn/20181124184403645.png"/></div>;
以上是关于0470-?????????Redhat7.4??????CDH5.16.1的主要内容，如果未能解决你的问题，请参考以下文章