在线安装CM集群

Posted 百里登风

tags:

篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了在线安装CM集群相关的知识，希望对你有一定的参考价值。

https://www.cloudera.com/documentation/manager/5-0-x/Cloudera-Manager-Installation-Guide/cm5ig_install_path_B.html

前期准备工作不多说了

先给每个节点配置镜像源，我这里以节点2为例

 

[cloudera-manager]
# Packages for Cloudera Manager, Version 5, on RedHat or CentOS 6 x86_64                 
name=Cloudera Manager
baseurl=https://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.13.0/
gpgkey =https://archive.cloudera.com/cm5/redhat/6/x86_64/cm/RPM-GPG-KEY-cloudera    
gpgcheck =1

 

 卸载centos6.5自带的jdk后就安装jdk,也是3个节点都安装

 

 

 

 

 在节点1安装server

sudo yum install cloudera-manager-daemons cloudera-manager-server

 

 

 在每个节点都安装agent

sudo yum install cloudera-manager-agent cloudera-manager-daemons

 

 在每个节点 vim /etc/cloudera-scm-agent/config.ini

 

 

下载rpm包到电脑上

 

 上传到cm1节点

 

sudo yum --nogpgcheck localinstall cloudera-cdh-5-0.x86_64.rpm

 

 

 

sudo rpm --import http://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/RPM-GPG-KEY-cloudera

 

 

 

sudo yum install avro-tools crunch flume-ng hadoop-hdfs-fuse hadoop-hdfs-nfs3 hadoop-httpfs 
hbase-solr hive-hbase hive-webhcat hue-beeswax hue-hbase hue-impala hue-pig hue-plugins hue-rdbms 
hue-search hue-spark hue-sqoop hue-zookeeper impala impala-shell kite llama mahout oozie pig pig-udf-datafu search sentry solr-mapreduce spark-python sqoop sqoop2 whirr

 

 

 

 

安装完之后，在cm1节点启动server服务

 

 在3个节点都启动agent服务

 

 

 

 

 

但是没有能打开

查看日志

in org.springframework.beans.factory.support.DefaultListableBeanFactory@1fe903d5: defining beans [commandLineConfigurationBean,entityManagerFactoryBean,com.cloudera.server.cmf.TrialState,com.cloudera.server.cmf.TrialManager,com.cloudera.cmf.crypto.LicenseLoader]; root of factory hierarchy
2018-04-11 20:19:50,074 ERROR main:com.cloudera.server.cmf.Main: Server failed.
org.springframework.beans.factory.BeanCreationException: Error creating bean with name \'com.cloudera.server.cmf.TrialState\': Cannot resolve reference to bean \'entityManagerFactoryBean\' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name \'entityManagerFactoryBean\': FactoryBean threw exception on object creation; nested exception is javax.persistence.PersistenceException: [PersistenceUnit: cmf.server] Unable to build EntityManagerFactory
    at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:328)
    at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:106)
    at org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:616)
    at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:148)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1003)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:907)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:485)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
    at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:293)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:290)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:192)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:585)
    at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:895)
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:425)
    at com.cloudera.server.cmf.Main.bootstrapSpringContext(Main.java:392)
    at com.cloudera.server.cmf.Main.<init>(Main.java:242)
    at com.cloudera.server.cmf.Main.main(Main.java:216)
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name \'entityManagerFactoryBean\': FactoryBean threw exception on object creation; nested exception is javax.persistence.PersistenceException: [PersistenceUnit: cmf.server] Unable to build EntityManagerFactory
    at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:149)
    at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:102)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1440)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:247)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:192)
    at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:322)
    ... 17 more
Caused by: javax.persistence.PersistenceException: [PersistenceUnit: cmf.server] Unable to build EntityManagerFactory
    at org.hibernate.ejb.Ejb3Configuration.buildEntityManagerFactory(Ejb3Configuration.java:924)
    at org.hibernate.ejb.Ejb3Configuration.buildEntityManagerFactory(Ejb3Configuration.java:899)
    at org.hibernate.ejb.HibernatePersistence.createEntityManagerFactory(HibernatePersistence.java:59)
    at javax.persistence.Persistence.createEntityManagerFactory(Persistence.java:63)
    at com.cloudera.enterprise.CommonMain.setupHibernate(CommonMain.java:183)
    at com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean.getObject(EntityManagerFactoryBean.java:114)
    at com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean.getObject(EntityManagerFactoryBean.java:65)
    at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:142)
    ... 22 more
Caused by: org.hibernate.service.classloading.spi.ClassLoadingException: HHH010003: JDBC Driver class not found: com.mysql.jdbc.Driver
    at org.hibernate.service.jdbc.connections.internal.C3P0ConnectionProvider.configure(C3P0ConnectionProvider.java:142)
    at org.hibernate.service.internal.StandardServiceRegistryImpl.configureService(StandardServiceRegistryImpl.java:75)
    at org.hibernate.service.internal.AbstractServiceRegistryImpl.initializeService(AbstractServiceRegistryImpl.java:159)
    at org.hibernate.service.internal.AbstractServiceRegistryImpl.getService(AbstractServiceRegistryImpl.java:131)
    at org.hibernate.engine.jdbc.internal.JdbcServicesImpl.buildJdbcConnectionAccess(JdbcServicesImpl.java:223)
    at org.hibernate.engine.jdbc.internal.JdbcServicesImpl.configure(JdbcServicesImpl.java:89)
    at org.hibernate.service.internal.StandardServiceRegistryImpl.configureService(StandardServiceRegistryImpl.java:75)
    at org.hibernate.service.internal.AbstractServiceRegistryImpl.initializeService(AbstractServiceRegistryImpl.java:159)
    at org.hibernate.service.internal.AbstractServiceRegistryImpl.getService(AbstractServiceRegistryImpl.java:131)
    at org.hibernate.cfg.Configuration.buildTypeRegistrations(Configuration.java:1797)
    at org.hibernate.cfg.Configuration.buildSessionFactory(Configuration.java:1755)
    at org.hibernate.ejb.EntityManagerFactoryImpl.<init>(EntityManagerFactoryImpl.java:96)
    at org.hibernate.ejb.Ejb3Configuration.buildEntityManagerFactory(Ejb3Configuration.java:914)
    ... 29 more
Caused by: org.hibernate.service.classloading.spi.ClassLoadingException: Unable to load class [com.mysql.jdbc.Driver]
    at org.hibernate.service.classloading.internal.ClassLoaderServiceImpl.classForName(ClassLoaderServiceImpl.java:149)
    at org.hibernate.service.jdbc.connections.internal.C3P0ConnectionProvider.configure(C3P0ConnectionProvider.java:139)
    ... 41 more
Caused by: java.lang.ClassNotFoundException: Could not load requested class : com.mysql.jdbc.Driver
    at org.hibernate.service.classloading.internal.ClassLoaderServiceImpl$AggregatedClassLoader.findClass(ClassLoaderServiceImpl.java:296)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:247)
    at org.hibernate.service.classloading.internal.ClassLoaderServiceImpl.classForName(ClassLoaderServiceImpl.java:146)
    ... 42 more
[root@cm1 cloudera-scm-server]# pwd
/var/log/cloudera-scm-server
[root@cm1 cloudera-scm-server]# cat cloudera-scm-server.log

 

 

没有安装数据库的驱动

sudo yum install mysql-server

 

 

 

 

 

 我们登录进mysql

create database amon DEFAULT CHARACTER SET utf8;
grant all on amon.* TO \'amon\'@\'%\' IDENTIFIED BY \'amon_password\';
grant all on *.* TO \'amon\'@\'cm1\' IDENTIFIED BY \'amon_password\';
grant all on amon.* TO \'amon\'@\'localhost\' IDENTIFIED BY \'amon_password\';


create database rman DEFAULT CHARACTER SET utf8;
grant all on rman.* TO \'rman\'@\'%\' IDENTIFIED BY \'rman_password\';
grant all on *.* TO \'rman\'@\'cm1\' IDENTIFIED BY \'rman_password\';
grant all on rman.* TO \'rman\'@\'localhost\' IDENTIFIED BY \'rman_password\';

create database nav DEFAULT CHARACTER SET utf8;
grant all on nav.* TO \'nav\'@\'%\' IDENTIFIED BY \'nav_password\';
grant all on *.* TO \'nav\'@\'cm1\' IDENTIFIED BY \'nav_password\';
grant all on nav.* TO \'nav\'@\'localhost\' IDENTIFIED BY \'nav_password\';

create database hive DEFAULT CHARACTER SET utf8;
grant all on hive.* TO \'hive\'@\'%\' IDENTIFIED BY \'hive_password\';
grant all on *.* TO \'hive\'@\'cm1\' IDENTIFIED BY \'hive_password\';
grant all on hive.* TO \'hive\'@\'localhost\' IDENTIFIED BY \'hive_password\';


create database scm DEFAULT CHARACTER SET utf8;
grant all on scm.* TO \'scm\'@\'%\' IDENTIFIED BY \'scm_password\';
grant all on *.* TO \'scm\'@\'cm1\' IDENTIFIED BY \'scm_password\';
grant all on scm.* TO \'scm\'@\'localhost\' IDENTIFIED BY \'scm_password\';

 

create database oozie DEFAULT CHARACTER SET utf8;
grant all on oozie.* TO \'oozie\'@\'localhost\' IDENTIFIED BY \'oozie_password\';
grant all on *.* TO \'oozie\'@\'cm1\' IDENTIFIED BY \'oozie_password\';
grant all on oozie.* TO \'oozie\'@\'%\' IDENTIFIED BY \'oozie_password\';


create database hue DEFAULT CHARACTER SET utf8;
grant all on hue.* TO \'hue\'@\'localhost\' IDENTIFIED BY \'hue_password\';
grant all on *.* TO \'hue\'@\'cm1\' IDENTIFIED BY \'hue_password\';
grant all on hue.* TO \'hue\'@\'%\' IDENTIFIED BY \'hue_password\';
flush privileges;


flush privileges;

 

mysql> create database amon DEFAULT CHARACTER SET utf8;
Query OK, 1 row affected (0.00 sec)

mysql> grant all on amon.* TO \'amon\'@\'%\' IDENTIFIED BY \'amon_password\';
Query OK, 0 rows affected (0.00 sec)

mysql> grant all on *.* TO \'amon\'@\'cm1\' IDENTIFIED BY \'amon_password\';
Query OK, 0 rows affected (0.00 sec)

mysql> grant all on amon.* TO \'amon\'@\'localhost\' IDENTIFIED BY \'amon_password\';
Query OK, 0 rows affected (0.00 sec)

mysql> create database rman DEFAULT CHARACTER SET utf8;
Query OK, 1 row affected (0.00 sec)

mysql> grant all on rman.* TO \'rman\'@\'%\' IDENTIFIED BY \'rman_password\';
Query OK, 0 rows affected (0.00 sec)

mysql> grant all on *.* TO \'rman\'@\'cm1\' IDENTIFIED BY \'rman_password\';
Query OK, 0 rows affected (0.00 sec)

mysql> grant all on rman.* TO \'rman\'@\'localhost\' IDENTIFIED BY \'rman_password\';
Query OK, 0 rows affected (0.00 sec)

mysql> create database nav DEFAULT CHARACTER SET utf8;
Query OK, 1 row affected (0.00 sec)

mysql> grant all on nav.* TO \'nav\'@\'%\' IDENTIFIED BY \'nav_password\';
Query OK, 0 rows affected (0.00 sec)

mysql> grant all on *.* TO \'nav\'@\'cm1\' IDENTIFIED BY \'nav_password\';
Query OK, 0 rows affected (0.00 sec)

mysql> grant all on nav.* TO \'nav\'@\'localhost\' IDENTIFIED BY \'nav_password\';
Query OK, 0 rows affected (0.00 sec)

mysql> create database metastore DEFAULT CHARACTER SET utf8;
Query OK, 1 row affected (0.00 sec)

mysql> grant all on hive.* TO \'hive\'@\'%\' IDENTIFIED BY \'hive_password\';
Query OK, 0 rows affected (0.00 sec)

mysql> grant all on *.* TO \'hive\'@\'cm1\' IDENTIFIED BY \'hive_password\';
Query OK, 0 rows affected (0.00 sec)

mysql> grant all on *.* TO \'hive\'@\'cm1\' IDENTIFIED BY \'hive_password\';
Query OK, 0 rows affected (0.00 sec)

mysql>flush privileges;

 

 在节点1执行

sudo /usr/share/cmf/schema/scm_prepare_database.sh mysql scm scm scm_password

 

把服务重启一下server (节点1)  agent（3个节点）

 

打开这个地址http://192.168.86.141:7180/

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

下一步如果安装失败了

而且出现这情况

 

 就这样解决

 

 如果是出现cloudemanager安装时出现8475 MainThread agent ERROR Heartbeating to 出门cm1:7180 failed这样的问题

 就这样解决

 

 

 

 

 

 

 

 

 

 遇到这情况就重新运行一下

 

 

 

用find / -name cm_guid查找 cm_guid文件

然后把他删除了（3个节点都这样）

 

 

 

 

遇到这种问题

 

 

 

查看agent日志

 

 Failed to connect to previous supervisor

 

以上是关于在线安装CM集群的主要内容，如果未能解决你的问题，请参考以下文章

CDH在线安装详细步骤(使用官方yum源)

Ubuntu14.04用apt在线/离线安装CDH5.1.2[Apache Hadoop 2.3.0]

Centos7 安装CDH6.3.2 集群

Cloudera Manager环境准备

Cloudera Manager环境准备

重装一次CM的坑爹记录