Hadoop 配置

Posted iAthena

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了Hadoop 配置相关的知识,希望对你有一定的参考价值。

1/增加hadoop 的用户;

sudo useradd -m hadoop -s /bin/bash

创建了可以登陆的 hadoop 用户,并使用 /bin/bash 作为 shell。

设置密码:sudo passwd hadoop

2/可为 hadoop 用户增加管理员权限 

sudo adduser hadoop sudo

更新源


   2  sudo apt update

3/安装ssh服务器,配置无密码登陆

 3  sudo apt-get install openssh-server4  ssh localhost 5  cd ~/.ssh/ 6  ssh-keygen -t rsa  7  cat ./id_rsa.pub >> ./authorized_keys

测试:

8  ssh localhost

一般选择下载最新的稳定版本,即下载 “stable” 下的 hadoop-2.x.y.tar.gz 这个格式的文件,这是编译好的,另一个包含 src 的则是 Hadoop 源代码,需要进行编译才可使用。

验证安装文件

   33  cat hadoop-2.7.2.tar.gz.mds |grep 'MD5'   34  head -n 6 hadoop-2.7.2.tar.gz.mds    35  md5sum hadoop-2.7.2.tar.gz |tr "a-z" "A-Z"

将 Hadoop 安装至 /usr/local/ 中:

sudo tar -zxf ~/packages/hadoop-2.7.2.tar.gz -C /usr/local
sudo tar -zxf ~/下载/hadoop-2.6.0.tar.gz -C /usr/local    # 解压到/usr/local中cd /usr/local/sudo mv ./hadoop-2.7.2/ ./hadoop            # 将文件夹名改为hadoopsudo chown -R hadoop:hadoop ./hadoop       # 修改文件权限

测试:

cd hadoop./bin/hadoop version


  1. [email protected]:/usr/local/hadoop$ ./bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar
An example program must be given as the first argument.

Valid program names are:

  aggregatewordcount: An Aggregate based map/reduce program that counts the words in the input files.

  aggregatewordhist: An Aggregate based map/reduce program that computes the histogram of the words in the input files.

  bbp: A map/reduce program that uses Bailey-Borwein-Plouffe to compute exact digits of Pi.

  dbcount: An example job that count the pageview counts from a database.

  distbbp: A map/reduce program that uses a BBP-type formula to compute exact bits of Pi.

  grep: A map/reduce program that counts the matches of a regex in the input.

  join: A job that effects a join over sorted, equally partitioned datasets

  multifilewc: A job that counts words from several files.

  pentomino: A map/reduce tile laying program to find solutions to pentomino problems.

  pi: A map/reduce program that estimates Pi using a quasi-Monte Carlo method.

  randomtextwriter: A map/reduce program that writes 10GB of random textual data per node.

  randomwriter: A map/reduce program that writes 10GB of random data per node.

  secondarysort: An example defining a secondary sort to the reduce.

  sort: A map/reduce program that sorts the data written by the random writer.

  sudoku: A sudoku solver.

  teragen: Generate data for the terasort

  terasort: Run the terasort

  teravalidate: Checking results of terasort

  wordcount: A map/reduce program that counts the words in the input files.

  wordmean: A map/reduce program that counts the average length of the words in the input files.

  wordmedian: A map/reduce program that counts the median length of the words in the input files.

  wordstandarddeviation: A map/reduce program that counts the standard deviation of the length of the words in the input files.

  1. ./bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar wordcount README.txt ~/RESULT

注意,Hadoop 默认不会覆盖结果文件,因此再次运行上面实例会提示出错,需要先将 ./output 删除。






以上是关于Hadoop 配置的主要内容,如果未能解决你的问题,请参考以下文章

Hadoop 安装

Linux虚拟机上Hadoop基础环境搭建

hadoop安装及hive安装

大数据练习环境部署 - Hadoop集群部署

Hadoop概念学习系列之Hadoop集群动态增加新节点或删除已有某节点及复制策略导向

Centos6下安装Hadoop2.6 问题总结