[Nutch]Hadoop澶氭満瀹屽叏鍒嗗竷寮忔ā寮廻adoop閰嶇疆

Posted 2020-07-15

鏍囩锛?/p>

1. 涓嬭浇hadoop

浣跨敤濡備笅鍛戒护锛?/p>

wget http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-1.2.1/hadoop-1.2.1.tar.gz

浣跨敤濡備笅鍛戒护瑙ｅ帇锛?/p>

tar -zxvf hadoop-1.2.1.tar.gz

2. 閰嶇疆hadoop

灏唄adoop鐨刡in鐩綍鍔犲叆鍒伴厤缃枃浠朵腑锛屼娇鐢ㄥ涓嬪懡浠わ細

vim .bashrc

娣诲姞濡備笅琛岋細

export PATH=/home/hadoop/workspace/hadoop-1.2.1/bin:$PATH

濡備笅锛?

浣跨敤濡備笅鍛戒护浣垮叾鐢熸晥锛?/p>

source  .bashrc

楠岃瘉hadoop璺緞锛屼娇鐢ㄥ涓嬪懡浠わ細

which hadoop

濡備笅锛?

3.1 淇敼masters閰嶇疆鏂囦欢

灏唋ocalhost淇敼涓篽ost1锛?/p>

vim conf/masters

3.2 淇敼slaves閰嶇疆鏂囦欢

鍒犻櫎localhost锛屾坊鍔爃ost1,锛宧ost2锛宧ost3锛?/p>

vim conf/slaves

3.3 淇敼core-site.xml

浣跨敤濡備笅鍛戒护锛?/p>

vim conf/core-site.xml

娣诲姞濡備笅鍐呭锛?/p>

<property>
<name>fs.default.name</name>
<value>hdfs://host1:9000</value>
</property>

濡備笅锛?

3.4 淇敼hdfs-site.xml

浣跨敤濡備笅鍛戒护锛?/p>

vim conf/hdfs-site.xml

娣诲姞濡備笅鍐呭锛?/p>

<property>
  <name>dfs.name.dir</name>
  <value>/home/hadoop/dfs/filesystem/name</value>
</property>
<property>
  <name>dfs.data.dir</name>
  <value>/home/hadoop/dfs/filesystem/data</value>
</property>
<property>
  <name>dfs.replication</name>
  <value>3</value>
</property>

濡備笅锛?

3.5 淇敼mapred-site.xml鏂囦欢

浣跨敤濡備笅鍛戒护锛?/p>

vim conf/mapred-site.xml

娣诲姞濡備笅鍐呭锛?/p>

<property>
  <name>mapred.job.tracker</name>
  <value>host1:9001</value>
</property>
<property> 
  <name>mapred.tasktracker.map.tasks.maximum</name>
  <value>4</value>
</property> 
<property> 
  <name>mapred.tasktracker.reduce.tasks.maximum</name>
  <value>4</value>
</property>
<property>
  <name>mapred.system.dir</name>
  <value>/home/hadoop/mapreduce/system</value>
</property>
<property>
  <name>mapred.local.dir</name>
  <value>/home/hadoop/mapreduce/local</value>
</property>

濡備笅锛?

3.6 淇敼hadoop-env.sh鏂囦欢

浣跨敤vim鎵撳紑conf鐩綍涓嬮潰鐨刪adoop-env.sh鏂囦欢锛?/p>

vim conf/hadoop-env.sh

鍦ㄩ噷闈㈤厤缃甁AVA_HOME锛屽姞鍏ュ涓嬪唴瀹?

export JAVA_HOME=/usr/lib/jvm/java-8-oracle

濡傚浘锛?

scp -r /home/hadoop/workspace/hadoop-1.2.1 hadoop@host2:/home/hadoop/workspace/hadoop-1.2.1

濡備笅锛?

浣跨敤濡備笅鍛藉悕澶嶅埗鍒颁富鏈篽ost3锛?/p>

scp -r /home/hadoop/workspace/hadoop-1.2.1  hadoop@host3:/home/hadoop/workspace/hadoop-1.2.1

濡備笅锛?

5. 鏍煎紡鍖栧悕绉拌妭鐐瑰苟鍚姩闆嗙兢

浣跨敤濡備笅鍛戒护锛?/p>

hadoop namenode -format

濡備笅锛?

6. 鍚姩闆嗙兢

浣跨敤濡備笅鍛戒护锛?/p>

start-all.sh

濡備笅锛?

鍦╤ost2涓婁娇鐢╦ps鍙互鐪嬪埌濡備笅杩涚▼锛?

鍦╤ost3涓婁娇鐢╦ps鍙互鐪嬪埌濡備笅杩涚▼锛?

8. 鏌ョ湅web绠＄悊椤甸潰