CDH安装配置第二篇:CDH安装的前期配置

Posted yaowentao

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了CDH安装配置第二篇:CDH安装的前期配置相关的知识,希望对你有一定的参考价值。

本篇介绍cdh安装之前需要的一些必要配置,当然这些配置也可以用shell脚本来配置。在安装之前请先配置好yum源,在文中用的统一都是阿里源。

 

系统:centos7

主机3台:

master.com.cn 192.168.1.110

salver1.com.cn 192.168.1.111

salver2.com.cn 192.168.1.112

yum服务器:yum.com.cn 192.168.1.100。这里如果不知道怎么配置的朋友,可以参考第一篇:https://www.cnblogs.com/yaowentao/p/12296329.html

 

基本配置

1. 主机名在系统安装的配置完毕,如果没有配置,可以进系统在修改

1 $ vi /etc/hostname

2. 配置host文件

1 $ vi /etc/hosts
2 192.168.1.100 yum.com.cn
3 192.168.1.110 master.com.cn
4 192.168.1.111 salver1.com.cn
5 192.168.1.112 salver2.com.cn

3. 关闭防火墙,selinux。配置完后重启

1 $ sudo systemctl disable firewalld        #停止防火墙
2 $ sudo systemctl stop firewalld           #禁止开机启动防火墙
3 
4 $ vi /etc/selinux/config
5 SELINUX=disable                #关闭selinux

 

4. 配置ntp服务与ssh免密码登陆配置,请参考我的另外2篇文档

 

 

 

5. 如果有本地的yum源,就配置。如果没有就使用公网的yum源。

5.1 配置yum 本地库,也可以下载官网的repo文件,一个是从网上下载,一个是从本地yum库下载
1 [cloudera-repo]
2 name=cloudera-repo
3 baseurl=http://192.168.10.225/cloudera-repos/cm6/6.2.1/redhat7/yum/
4 enabled=1
5 gpgcheck=0
5.2 网上yum源,直接下载即可
1 $ sudo wget https://archive.cloudera.com/cm6/6.2.1/redhat7/yum/cloudera-manager.repo -P /etc/yum.repos.d/ 

6. 安装cdh专用jdk(需要在所有主机上都要安装)

1 $ sudo yum -y install oracle-j2sdk1.8
技术图片

7. 配置本地的mysql数据库,便于存放cdh服务器的一些配置,以及hadoop组件中需要的一些配置,如hive的源数据库。这里一般推荐用mysql,当然oracle等其他的数据库也可以。

  1. 直接用yum源安装
  2. 1 $ wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm
    2 $ sudo rpm -ivh mysql-community-release-el7-5.noarch.rpm
    3 $ sudo yum update
    4 $ sudo yum install mysql-server
    5 $ sudo systemctl start mysqld

    技术图片
  3. 配置mysql数据,并进行初始化,设置root密码
  4. $vi /etc/my.cnf
    
    [mysqld]
    datadir=/var/lib/mysql
    socket=/var/lib/mysql/mysql.sock
    transaction-isolation = READ-COMMITTED
    # Disabling symbolic-links is recommended to prevent assorted security risks;
    # to do so, uncomment this line:
    symbolic-links = 0
    
    
    key_buffer_size = 32M
    max_allowed_packet = 32M
    thread_stack = 256K
    thread_cache_size = 64
    query_cache_limit = 8M
    query_cache_size = 64M
    query_cache_type = 1
    
    
    max_connections = 550
    #expire_logs_days = 10
    #max_binlog_size = 100M
    
    
    #log_bin should be on a disk with enough free space.
    #Replace /var/lib/mysql/mysql_binary_log with an appropriate path for your
    #system and chown the specified folder to the mysql user.
    log_bin=/var/lib/mysql/mysql_binary_log
    
    
    #In later versions of MySQL, if you enable the binary log and do not set
    #a server_id, MySQL will not start. The server_id must be unique within
    #the replicating group.
    server_id=1
    
    
    binlog_format = mixed
    
    
    read_buffer_size = 2M
    read_rnd_buffer_size = 16M
    sort_buffer_size = 8M
    join_buffer_size = 8M
    
    
    # InnoDB settings
    innodb_file_per_table = 1
    innodb_flush_log_at_trx_commit  = 2
    innodb_log_buffer_size = 64M
    innodb_buffer_pool_size = 4G
    innodb_thread_concurrency = 8
    innodb_flush_method = O_DIRECT
    innodb_log_file_size = 512M
    
    
    [mysqld_safe]
    log-error=/var/log/mysqld.log
    pid-file=/var/run/mysqld/mysqld.pid
    
    
    sql_mode=STRICT_ALL_TABLES
  5. 配置开机自启动

  6. 1 $ sudo systemctl enable mysqld
    2 $ sudo systemctl start mysqld
  7. 初始化mysql,设置root密码
  8.  1 $ sudo /usr/bin/mysql_secure_installation
     2 [...]
     3 Enter current password for root (enter for none):
     4 OK, successfully used password, moving on...
     5 [...]
     6 Set root password? [Y/n] Y
     7 New password:
     8 Re-enter new password:
     9 Remove anonymous users? [Y/n] Y
    10 [...]
    11 Disallow root login remotely? [Y/n] N
    12 [...]
    13 Remove test database and access to it [Y/n] Y
    14 [...]
    15 Reload privilege tables now? [Y/n] Y
    16 All done!
    技术图片

     

     

     

  9. 安装mysql的jdbc

  10. 1 $ wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz
    2 $ tar zxvf mysql-connector-java-5.1.46.tar.gz
    3 
    4 $ sudo mkdir -p /usr/share/java/
    5 $ cd mysql-connector-java-5.1.46
    6 $ sudo cp mysql-connector-java-5.1.46-bin.jar /usr/share/java/mysql-connector-java.jar
  11. 给root授远程登陆权限
  12. 1 mysql> GRANT ALL PRIVILEGES ON *.* TO root@% IDENTIFIED BY 123456 WITH GRANT OPTION;
    2 mysql> FLUSH PRIVILEGES;
  13. 创建cm服务,hadoop组件需要的库。用户统一用root。
  14.  1 CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
     2 GRANT ALL ON scm.* TO root@% IDENTIFIED BY 123456;
     3 CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
     4 GRANT ALL ON amon.* TO root@% IDENTIFIED BY 123456;
     5 CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
     6 GRANT ALL ON rman.* TO root@% IDENTIFIED BY 123456;
     7 CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
     8 GRANT ALL ON hue.* TO root@% IDENTIFIED BY 123456;
     9 CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
    10 GRANT ALL ON metastore.* TO root@% IDENTIFIED BY 123456;
    11 CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
    12 GRANT ALL ON sentry.* TO root@% IDENTIFIED BY 123456;
    13 CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
    14 GRANT ALL ON nav.* TO root@% IDENTIFIED BY 123456;
    15 CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
    16 GRANT ALL ON navms.* TO root@% IDENTIFIED BY 123456;
    17 CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
    18 GRANT ALL ON oozie.* TO root@% IDENTIFIED BY 123456;

    技术图片

  15.  安装cm服务器文件

  16. 1 master:
    2 $ sudo yum -y install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server
    3 
    技术图片

     

     

    4 salver:
    5 $ sudo yum -y install cloudera-manager-daemons cloudera-manager-agent
    技术图片 技术图片
  17. scm初始化
    1.   
      1 $ sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm root
      技术图片
  18. 完成以上配置,就能正常登陆cm服务器的前台了。下面进入服务器的配置。请看第三篇。
    1.   启动cm服务
      1 $ sudo systemctl start cloudera-scm-server
      2 $ sudo tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log
      技术图片

启动成功: 

技术图片

 

 

各位小伙伴,对以上如有问题,请在下方留言沟通。以上都是我亲自测试通过安装成功的。

 

转发,请注明出处!

以上是关于CDH安装配置第二篇:CDH安装的前期配置的主要内容,如果未能解决你的问题,请参考以下文章

CDH6.2.0搭建步骤

CDH6.2.0搭建步骤

CDH6.2.0搭建步骤

离线安装Cloudera Manager5.3.4与CDH5.3.4

大数据平台CDH6.1.0 安装配置

hadoop(cdh版)下载安装即配置