Hadoop+HBase+Spark+Hive鐜鎼缓

Posted

tags:

篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了Hadoop+HBase+Spark+Hive鐜鎼缓相关的知识，希望对你有一定的参考价值。

鏍囩锛?a href='http://www.mamicode.com/so/1/vax' title='vax'>vax   ice   shift   tmp   鐗圭偣   php   base   lse   win   

 

鎶€鏈垎浜浘鐗? class=
鏉ㄨ禑蹇窇
绠€涔︿綔鑰?/span>
2018-09-24 10:24

鎽樿锛氬ぇ鏁版嵁闂ㄦ杈冮珮锛屼粎浠呯幆澧冪殑鎼缓鍙兘灏辫鑰楄垂鎴戜滑澶ч噺鐨勭簿鍔涳紝鏈枃鎬荤粨浜嗕綔鑰呮槸濡備綍鎼缓澶ф暟鎹幆澧冪殑锛堝崟鏈虹増鍜岄泦缇ょ増锛夛紝甯屾湜鑳藉府鍔╁寮熷濡逛滑鏇村揩鍦拌蛋涓婂ぇ鏁版嵁瀛︿範涔嬭矾銆?/p>

0. 鍑嗗瀹夎鍖?/h1>

鏈枃鎵€闇€鐨勭郴缁熼暅鍍忋€佸ぇ鏁版嵁杞欢瀹夎鍖呫€佸紑鍙戠幆澧冭蒋浠跺畨瑁呭寘绛夐兘鍙互鍦ㄦ垜鐨勭櫨搴︿簯鐩樹腑涓嬭浇銆?br>閾炬帴锛?a href="https://pan.baidu.com/s/1yYx8DoW3Sus2USmpzGVG4w" rel="nofollow" target="_blank">绯荤粺闀滃儚鍜屽悇绉嶅ぇ鏁版嵁杞欢
瀵嗙爜锛歯2cn

1. Windows涓嬪畨瑁匲buntu鍙岀郴缁?/h1>

Hadoop绛夊ぇ鏁版嵁寮€婧愭鏋舵槸涓嶆敮鎸乄indows绯荤粺鐨勶紝鎵€浠ラ渶瑕佸厛瀹夎涓€涓狶inux鍙岀郴缁熴€傚綋鐒讹紝濡傛灉浣犳湁涓€鍙板崟鐙殑鐢佃剳鐢ㄦ潵瀹夎Ubuntu绯荤粺锛屽氨涓嶉渶瑕佸畨瑁呭弻绯荤粺浜嗐€?/p>

鍙岀郴缁熷畨瑁?br>璇峰弬鑰冨畨瑁呮寚鍗楋細
绗竴姝ワ細鍒跺畾绯荤粺鍚姩U鐩?/a>
绗簩姝ワ細瀹夎鍙岀郴缁?/a>

2. 鎼缓Hadoop骞冲彴

Hadoop鏄疉pache 鍏徃寮€鍙戠殑涓€娆惧彲闈犵殑銆佸彲鎵╁睍鎬х殑銆佸垎甯冨紡璁＄畻鐨勫紑婧愯蒋浠躲€備互Hadoop鍒嗗竷寮忔枃浠剁郴缁燂紙HDFS锛夊拰鍒嗗竷寮忚繍绠楃紪绋嬫鏋讹紙MapReduce锛変负鏍稿績锛屽厑璁稿湪闆嗙兢鏈嶅姟鍣ㄤ笂浣跨敤绠€鍗曠殑缂栫▼妯″瀷瀵瑰ぇ鏁版嵁闆嗚繘琛屽垎甯冨紡澶勭悊銆備笅闈紝璇疯窡鐫€浣滆€呬竴姝ユ鎼缓鑷繁鐨凥adoop骞冲彴鍚с€?/p>

2.1鏇存柊婧?/h2>

鍦╞ash缁堢涓繍琛屽涓媠hell鎸囦护锛岃缃畆oot鐢ㄦ埛瀵嗙爜锛屽苟鍒囨崲鍒皉oot鐢ㄦ埛

#璁剧疆root瀵嗙爜
sudo passwd
#鍒囨崲鍒皉oot鐢ㄦ埛
su root

鏇存柊婧?/p> 

apt-get update

瀹夎vim缂栬瘧鍣?/p> 

apt-get install vim

澶囦唤鍘熷鐨勫畼鏂规簮

cp /etc/apt/sources.list /etc/apt/sources.list.bak

鍒犻櫎鍘熷鐨勫畼鏂规簮

rm /etc/apt/sources.list

杩愯濡備笅shell鍛戒护锛岄噸鏂板垱寤簊ources.list鏂囦欢

vim /etc/apt/sources.list

鎸?i 杩涘叆vim鐨勭紪杈戞ā寮忥紝澶嶅埗涓嬮潰鐨勬竻鍗庢簮鍒皊ources.list鏂囦欢涓紝鐒跺悗鎸?esc 閫€鍑虹紪杈戞ā寮忥紝鏈€鍚庤緭鍏?+wq 锛屾寜鍥炶溅淇濆瓨锛堜篃鍙互鎸塻hift + zz 杩涜淇濆瓨锛夈€?/p> 

# 榛樿娉ㄩ噴浜嗘簮鐮侀暅鍍忎互鎻愰珮 apt update 閫熷害锛屽鏈夐渶瑕佸彲鑷鍙栨秷娉ㄩ噴
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ artful main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ artful main restricted universe multiverse
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ artful-updates main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ artful-updates main restricted universe multiverse
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ artful-backports main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ artful-backports main restricted universe multiverse
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ artful-security main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ artful-security main restricted universe multiverse

# 棰勫彂甯冭蒋浠舵簮锛屼笉寤鸿鍚敤
# deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ artful-proposed main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ artful-proposed main restricted universe multiverse

杩愯濡備笅shell鍛戒护锛屽畬鎴愭簮鐨勬洿鏂?/p> 

apt-get update

2.2 瀹夎SSH銆侀厤缃甋SH鏃犲瘑鐮佺櫥闄?/h3>

Ubuntu 榛樿宸插畨瑁呬簡 SSH client锛屾澶栬繕闇€瑕佸畨瑁?SSH server锛?/p> 

sudo apt-get install openssh-server

瀹夎鍚庯紝淇敼sshd_config閰嶇疆

vim /etc/ssh/sshd_config

鍦ㄦ枃浠朵腑璁剧疆濡備笅灞炴€э細锛堟寜 / 鍙互杩涘叆鎼滅储妯″紡锛屾寜esc閫€鍑烘悳绱㈡ā寮忥級

PubkeyAuthentication yes
PermitRootLogin yes

閲嶅惎ssh鏈嶅姟

sudo /etc/init.d/ssh restart

閲嶅惎鍚庯紝鍙互浣跨敤濡備笅鍛戒护鐧婚檰鏈満锛屼絾姝ゆ椂闇€瑕佸瘑鐮佹墠鑳界櫥闄嗭細

ssh localhost

棣栧厛閫€鍑哄垰鎵嶇殑 ssh锛屽氨鍥炲埌浜嗘垜浠師鍏堢殑缁堢绐楀彛锛岀劧鍚庡埄鐢?ssh-keygen 鐢熸垚瀵嗛挜锛屽苟灏嗗瘑閽ュ姞鍏ュ埌鎺堟潈涓細

exit                           # 閫€鍑哄垰鎵嶇殑 ssh localhost
cd ~/.ssh/                     # 鑻ユ病鏈夎鐩綍锛岃鍏堟墽琛屼竴娆sh localhost
ssh-keygen -t rsa              # 浼氭湁鎻愮ず锛岄兘鎸夊洖杞﹀氨鍙互
cat ./id_rsa.pub >> ./authorized_keys  # 鍔犲叆鎺堟潈

鍦?Linux 绯荤粺涓紝~ 浠ｈ〃鐨勬槸鐢ㄦ埛鐨勪富鏂囦欢澶癸紙root鐢ㄦ埛渚嬪锛夛紝鍗?鈥?home/鐢ㄦ埛鍚嶁€?杩欎釜鐩綍锛屽浣犵殑鐢ㄦ埛鍚嶄负 ubuntu锛屽垯 ~ 灏变唬琛?鈥?home/ubuntu/鈥濄€?濡傛灉鏄痳oot鐢ㄦ埛鍒檦浠ｈ〃/root锛屾澶栵紝鍛戒护涓殑 # 鍚庨潰鐨勬枃瀛楁槸娉ㄩ噴锛屽彧闇€瑕佽緭鍏ュ墠闈㈠懡浠ゅ嵆鍙€?/p>

姝ゆ椂鍐嶇敤 ssh localhost 鍛戒护锛屾棤闇€杈撳叆瀵嗙爜灏卞彲浠ョ洿鎺ョ櫥闄嗕簡銆?/p>

2.3 瀹夎JAVA鐜

Java鐜鎺ㄨ崘浣跨敤 Oracle 鐨?JDK锛岄鍏堬紝鍑嗗濂芥枃浠?span class="Apple-converted-space"> jdk-8u162-linux-x64.tar.gz锛岀劧鍚庡皢鏂囦欢绉诲埌/usr/local鐩綍涓嬶細

mv jdk-8u162-linux-x64.tar.gz /usr/local

瑙ｅ帇鏂囦欢

tar -zxvf jdk-8u162-linux-x64.tar.gz

閲嶅懡鍚嶆枃浠跺す涓簀ava

mv jdk-1.8.0_162 java

鐢╲im鎵撳紑/etc/profile鏂囦欢锛圠inux涓嬮厤缃郴缁熺幆澧冨彉閲忕殑鏂囦欢锛?/p> 

vim /etc/profile

鎸塱杩涘叆缂栬緫妯″紡锛屽湪鏂囦欢鏈熬娣诲姞濡備笅JAVA鐜鍙橀噺

export JAVA_HOME=/usr/local/java
export JRE_HOME=/usr/local/java/jre
export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib:$JRE_HOME/lib
export PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin

娣诲姞鐜鍙橀噺鍚庯紝缁撴灉濡備笅鍥炬墍绀猴紝鎸?esc 閫€鍑虹紪杈戞ā寮忥紝鐒跺悗杈撳叆:+wq 锛屾寜鍥炶溅淇濆瓨锛堜篃鍙互鎸塻hift + zz 杩涜淇濆瓨锛夈€?/p>

 
鎶€鏈垎浜浘鐗? src=
閰嶇疆鐜鍙橀噺

鏈€鍚庯紝闇€瑕佽璇ョ幆澧冨彉閲忕敓鏁堬紝鎵ц濡備笅浠ｇ爜锛?/p> 

source /etc/profile

妫€楠孞AVA鏄惁瀹夎鎴愬姛

echo $JAVA_HOME     # 妫€楠屽彉閲忓€?java -version
java
javac

濡傛灉璁剧疆姝ｇ‘鐨勮瘽锛宩ava -version 浼氳緭鍑?java 鐨勭増鏈俊鎭紝java 鍜?javac 浼氳緭鍑哄懡浠ょ殑浣跨敤鎸囧銆?/p>

2.4 瀹夎Hadoop

涓嬭浇 hadoop-2.7.6.tar.gz 鏂囦欢锛岀劧鍚庡皢鏂囦欢绉诲埌/usr/local鐩綍涓?/span>

mv hadoop-2.7.6.tar.gz /usr/local

瑙ｅ帇

tar -zxvf hadoop-2.7.6.tar.gz

鏂囦欢澶归噸鍛藉悕涓篽adoop

mv hadoop-2.7.6 hadoop

閰嶇疆鐜鍙橀噺锛屾墦寮€鏂囦欢/etc/profile锛屾坊鍔犲涓婬adoop鐜鍙橀噺

export HADOOP_HOME=/usr/local/hadoop
export PATH=$PATH:/usr/local/hadoop/bin:/usr/local/hadoop/sbin

鍚屾牱锛岄渶瑕佽璇ョ幆澧冨彉閲忕敓鏁堬紝鎵ц濡備笅浠ｇ爜锛?/p> 

source /etc/profile

杈撳叆濡備笅鍛戒护鏉ユ鏌?Hadoop 鏄惁鍙敤锛屾垚鍔熷垯浼氭樉绀?Hadoop 鐗堟湰淇℃伅锛?/p> 

hadoop version

2.5 Hadoop閰嶇疆锛堝崟鏈轰吉鍒嗗竷寮忔ā寮忥級

Hadoop 鍙互鍦ㄥ崟鑺傜偣涓婁互浼垎甯冨紡鐨勬柟寮忚繍琛岋紝Hadoop 杩涚▼浠ュ垎绂荤殑 Java 杩涚▼鏉ヨ繍琛岋紝鑺傜偣鏃綔涓?NameNode 涔熶綔涓?DataNode锛屽悓鏃讹紝璇诲彇鐨勬槸 HDFS 涓殑鏂囦欢銆?/p>

淇敼閰嶇疆鏂囦欢 core-site.xml (gedit /usr/local/hadoop/etc/hadoop/core-site.xml)锛屽皢褰撲腑鐨?/p> 

<configuration>
</configuration>

淇敼涓轰笅闈㈤厤缃細

<configuration>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>file:/usr/local/hadoop/tmp</value>
        <description>Abase for other temporary directories.</description>
    </property>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
    </property>
</configuration>

鍚屾牱鐨勶紝淇敼閰嶇疆鏂囦欢 hdfs-site.xml(gedit /usr/local/hadoop/etc/hadoop/hdfs-site.xml)锛?/p> 

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>file:/usr/local/hadoop/tmp/dfs/name</value>
    </property>
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>file:/usr/local/hadoop/tmp/dfs/data</value>
    </property>
</configuration>

淇敼鏂囦欢 hadoop-env.sh (gedit /usr/local/hadoop/etc/hadoop/hadoop-env.sh)锛屽湪鏂囦欢寮€濮嬪娣诲姞Hadoop鍜孞ava鐜鍙橀噺銆?/p> 

export JAVA_HOME=/usr/local/java
export HADOOP_HOME=/usr/local/hadoop
export PATH=$PATH:/usr/local/hadoop/bin
Hadoop閰嶇疆鏂囦欢璇存槑

Hadoop 鐨勮繍琛屾柟寮忔槸鐢遍厤缃枃浠跺喅瀹氱殑锛堣繍琛?Hadoop 鏃朵細璇诲彇閰嶇疆鏂囦欢锛夛紝浼垎甯冨紡鍙渶瑕侀厤缃?fs.defaultFS 鍜?dfs.replication 灏卞彲浠ヨ繍琛岋紙瀹樻柟鏁欑▼濡傛锛夛紝涓嶈繃鑻ユ病鏈夐厤缃?hadoop.tmp.dir 鍙傛暟锛屽垯榛樿浣跨敤鐨勪复鏃剁洰褰曚负 /tmp/hadoo-hadoop锛岃€岃繖涓洰褰曞湪閲嶅惎鏃舵湁鍙兘琚郴缁熸竻鐞嗘帀锛屽鑷村繀椤婚噸鏂版墽琛?format 鎵嶈銆傛墍浠ユ垜浠繘琛屼簡璁剧疆锛屽悓鏃朵篃鎸囧畾 dfs.namenode.name.dir 鍜?dfs.datanode.data.dir锛屽惁鍒欏湪鎺ヤ笅鏉ョ殑姝ラ涓彲鑳戒細鍑洪敊銆?/p>

閰嶇疆瀹屾垚鍚庯紝鎵ц NameNode 鐨勬牸寮忓寲:

/usr/local/hadoop/bin/hdfs namenode -format

鍚姩hadoop

./usr/local/hadoop/sbin/start-all.sh

鎴愬姛鍚姩鍚庯紝杩愯jps鍛戒护

source /etc/profile
jps

濡傛灉瀹夎鎴愬姛锛屽垯浼氬嚭鐜板涓嬪涓嬭繘绋?/p>

 
鎶€鏈垎浜浘鐗? src=
hadoop鎴愬姛鍚姩鍚庣殑杩涚▼

鎴愬姛鍚姩鍚庯紝鍙互璁块棶 Web 鐣岄潰 http://localhost:50070 鏌ョ湅 NameNode 鍜?Datanode 淇℃伅锛岃繕鍙互鍦ㄧ嚎鏌ョ湅 HDFS 涓殑鏂囦欢銆?/span>

 

2.6 Hadoop閰嶇疆锛堥泦缇ゆā寮忥級

2.6.1 璁剧疆闈欐€両P锛堜互涓昏妭鐐逛负渚嬶級

缂栬緫鏂囦欢/etc/network/interfaces

vim /etc/network/interfaces

鍦ㄦ枃浠跺悗闈㈡坊鍔犲涓嬮厤缃俊鎭紙eth0鏄綉鍗″悕锛岄渶瑕佹牴鎹疄闄呮儏鍐垫洿鏀癸級

auto eth0 #缃戝崱鍚?iface eth0 inet static
address 192.168.1.2 #闈欐€?span class="hljs-selector-tag">IP锛堝彲鏍规嵁瀹為檯鎯呭喌鑷敱璁剧疆锛?netmask 255.255.255.0 #瀛愮綉鎺╃爜
gateway 192.168.1.1 #缃戝叧
dns-nameservers 192.168.1.1 #DNS鏈嶅姟鍣ㄥ湴鍧€锛屼笌缃戝叧鐩稿悓鍗冲彲

缂栬緫鏂囦欢/etc/resolve.conf

vim /etc/resolve.conf

鍦ㄦ枃浠朵腑娣诲姞濡備笅閰嶇疆淇℃伅

nameserver 192.168.1.1

姝ns鍦ㄧ郴缁熼噸鍚悗浼氬け鏁堬紝缂栬緫鏂囦欢/etc/resolvconf/resolv.conf.d/base

vim /etc/resolvconf/resolv.conf.d/base

娣诲姞濡備笅鍐呭锛屼粠鑰屾案涔呬繚瀛楧NS閰嶇疆

nameserver 192.168.1.1

杩愯濡備笅鍛戒护閲嶅惎缃戠粶

/etc/init.d/networking restart

濡傛灉閲嶅惎鍚庢棤鏁堬紝鍒欓噸鍚郴缁熴€?/p>

閲嶅惎鍚庡鏋滃彂鐜版壘涓嶅埌缃戝崱锛屽垯鍚敤绯荤粺鎵樼缃戝崱

vim /etc/NetworkManager/NetworkManager.conf

淇敼

managed=false

涓?/p> 

managed=true

杩愯濡備笅鍛戒护閲嶅惎缃戠粶

/etc/init.d/networking restart

濡傛灉閲嶅惎鍚庢棤鏁堬紝鍒欓噸鍚郴缁熴€?/p>

2.6.2 閰嶇疆hosts鏂囦欢(姣忓彴涓绘満閮借閰嶇疆)

淇敼涓绘満鍚?/p> 

vim /etc/hostname

鎻愮ず锛氫富鑺傜偣璁剧疆涓簃aster锛屼粠鑺傜偣璁剧疆涓簊lave1銆乻lave2绛夌瓑銆?/p>

缂栬緫鏂囦欢/etc/hosts

vim /etc/hosts

灏嗕互涓嬫暟鎹鍒惰繘鍏ラ泦缇ょ殑鍚勪釜涓绘満涓?/p> 

192.168.1.2     master
192.168.1.11    slave1

娉ㄦ剰锛氳嫢鍐嶅鍔犱竴涓粠鏈猴紝鍒欐坊鍔爏lave2鐨勪俊鎭?/p>

浣跨敤浠ヤ笅鎸囦护鍦╩aster涓绘満涓繘琛屾祴璇曪紝鍙娇鐢ㄧ被浼兼寚浠ゅ湪slave1涓婃祴璇曪細

ping slave1

濡傛灉ping鐨勯€氾紝璇存槑缃戠粶杩炴帴姝ｅ父锛屽惁鍒欒妫€鏌ョ綉缁滆繛鎺ユ垨鑰匢P淇℃伅鏄惁姝ｇ‘銆?/p>

2.6.3 SSH鏃犲瘑鐮佺櫥闄嗚妭鐐癸紙master涓婇厤缃級

杩欎釜鎿嶄綔鏄璁?master 鑺傜偣鍙互鏃犲瘑鐮?SSH 鐧婚檰鍒板悇涓?slave 鑺傜偣涓娿€?/p>

棣栧厛鐢熸垚 master 鑺傜偣鐨勫叕鍖欙紝鍦?master鑺傜偣鐨勭粓绔腑鎵ц锛堝洜涓烘敼杩囦富鏈哄悕锛屾墍浠ヨ繕闇€瑕佸垹鎺夊師鏈夌殑鍐嶉噸鏂扮敓鎴愪竴娆★級锛?/p> 

cd ~/.ssh               # 濡傛灉娌℃湁璇ョ洰褰曪紝鍏堟墽琛屼竴娆sh localhost
rm ./id_rsa*            # 鍒犻櫎涔嬪墠鐢熸垚鐨勫叕鍖欙紙濡傛灉鏈夛級
ssh-keygen -t rsa       # 涓€鐩存寜鍥炶溅灏卞彲浠?

璁?master 鑺傜偣闇€鑳芥棤瀵嗙爜 SSH 鏈満锛屽湪 master 鑺傜偣涓婃墽琛岋細

cat ./id_rsa.pub >> ./authorized_keys

瀹屾垚鍚庡彲鎵ц ssh master 楠岃瘉涓€涓嬶紙鍙兘闇€瑕佽緭鍏?yes锛屾垚鍔熷悗鎵ц exit 杩斿洖鍘熸潵鐨勭粓绔級銆傛帴鐫€鍦?master 鑺傜偣灏嗕笂鍏寵浼犺緭鍒?slave1鑺傜偣锛?/p> 

scp ~/.ssh/id_rsa.pub [email protected]:/root/

scp 鏄?secure copy 鐨勭畝鍐欙紝鐢ㄤ簬鍦?Linux 涓嬭繘琛岃繙绋嬫嫹璐濇枃浠讹紝绫讳技浜?cp 鍛戒护锛屼笉杩?cp 鍙兘鍦ㄦ湰鏈轰腑鎷疯礉銆傛墽琛?scp 鏃朵細瑕佹眰杈撳叆 slave1 涓?root 鐢ㄦ埛鐨勫瘑鐮併€?/p>

鎺ョ潃鍦?slave1 鑺傜偣涓婏紝灏?ssh 鍏寵鍔犲叆鎺堟潈

mkdir /root/.ssh       # 濡傛灉涓嶅瓨鍦ㄨ鏂囦欢澶归渶鍏堝垱寤猴紝鑻ュ凡瀛樺湪鍒欏拷鐣?cat /root/id_rsa.pub >> /root/.ssh/authorized_keys
rm /root/id_rsa.pub    # 鐢ㄥ畬灏卞彲浠ュ垹鎺変簡

濡傛灉鏈夊叾浠?slave 鑺傜偣锛屼篃瑕佹墽琛屽皢 master 鍏寵浼犺緭鍒?slave 鑺傜偣銆佸湪 slave 鑺傜偣涓婂姞鍏ユ巿鏉冭繖涓ゆ銆?/p>

杩欐牱锛屽湪 master 鑺傜偣涓婂氨鍙互鏃犲瘑鐮?SSH 鍒板悇涓?slave 鑺傜偣浜嗭紝鍙湪 master 鑺傜偣涓婃墽琛屽涓嬪懡浠よ繘琛屾楠岋細

ssh root@slave1

濡傛灉涓嶉渶瑕佸瘑鐮侊紝鍒欓厤缃垚鍔熴€?/p>

2.6.4 淇敼Hadoop閰嶇疆鏂囦欢锛坢aster涓婇厤缃級

淇敼閰嶇疆鏂囦欢 core-site.xml (gedit /usr/local/hadoop/etc/hadoop/core-site.xml)锛屽皢褰撲腑鐨?/p> 

<configuration>
</configuration>

淇敼涓轰笅闈㈤厤缃細

<configuration>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>file:/usr/local/hadoop/tmp</value>
        <description>Abase for other temporary directories.</description>
    </property>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://master:9000</value>
    </property>
</configuration>

鍚屾牱鐨勶紝淇敼閰嶇疆鏂囦欢 hdfs-site.xml(gedit /usr/local/hadoop/etc/hadoop/hdfs-site.xml)锛?/p> 

<configuration>
        <property>
                <name>dfs.namenode.secondary.http-address</name>
                <value>Master:50090</value>
        </property>
        <property>
                <name>dfs.replication</name>
                <value>2</value>
        </property>
        <property>
                <name>dfs.namenode.name.dir</name>
                <value>file:/usr/local/hadoop/tmp/dfs/name</value>
        </property>
        <property>
                <name>dfs.datanode.data.dir</name>
                <value>file:/usr/local/hadoop/tmp/dfs/data</value>
        </property>
</configuration>

淇敼鏂囦欢 mapred-site.xml 锛堝彲鑳介渶瑕佸厛閲嶅懡鍚嶏紝榛樿鏂囦欢鍚嶄负 mapred-site.xml.template锛夛紝鐒跺悗閰嶇疆淇敼濡備笅锛?/p> 

<configuration>
        <property>
                <name>mapreduce.framework.name</name>
                <value>yarn</value>
        </property>
        <property>
                <name>mapreduce.jobhistory.address</name>
                <value>master:10020</value>
        </property>
        <property>
                <name>mapreduce.jobhistory.webapp.address</name>
                <value>master:19888</value>
        </property>
</configuration>

閰嶇疆yarn-site.xml(gedit /usr/local/hadoop/etc/hadoop/yarn-site.xml)

<configuration>
<property>
     <name>yarn.resourcemanager.hostname</name>
     <value>master</value>
</property>
<property>
     <name>yarn.nodemanager.resource.memory-mb</name>
     <value>10240</value>
</property>
<property>
     <name>yarn.nodemanager.aux-services</name>
     <value>mapreduce_shuffle</value>
</property>
</configuration>

淇敼鏂囦欢 hadoop-env.sh (gedit /usr/local/hadoop/etc/hadoop/hadoop-env.sh)锛屽湪鏂囦欢寮€濮嬪娣诲姞Hadoop鍜孞ava鐜鍙橀噺銆?/p> 

export JAVA_HOME=/usr/local/java
export HADOOP_HOME=/usr/local/hadoop
export PATH=$PATH:/usr/local/hadoop/bin

閰嶇疆slaves(gedit /usr/local/hadoop/etc/hadoop/slaves)锛屽垹闄ら粯璁ょ殑localhost锛屽鍔犱粠鑺傜偣:

slave1

娉ㄦ剰锛氳嫢鍐嶅鍔犱竴涓粠鏈猴紝鍐嶆坊鍔爏lave2

閰嶇疆濂藉悗锛屽皢 master 涓婄殑 /usr/local/hadoop 鏂囦欢澶瑰鍒跺埌鍚勪釜鑺傜偣涓娿€?/p> 

sudo rm -rf /usr/local/hadoop/tmp     # 鍒犻櫎 Hadoop 涓存椂鏂囦欢
sudo rm -rf /usr/local/hadoop/logs   # 鍒犻櫎鏃ュ織鏂囦欢
scp -r /usr/local/hadoop slave1:/usr/local

娉ㄦ剰锛氭瘡鍙颁粠鏈轰笂闇€瑕侀厤缃瓾adoop鐨勭幆澧冨彉閲?/p>

鍦╩aster鑺傜偣涓婂惎鍔╤adoop

./usr/local/hadoop/bin/hdfs namenode -format
./usr/local/hadoop/sbin/start-all.sh

鎴愬姛鍚姩鍚庯紝杩愯jps鍛戒护

source /etc/profile
jps

濡傛灉瀹夎鎴愬姛锛宮aster鑺傜偣浼氭湁NameNode杩涚▼锛宻lave鑺傜偣浼氭湁DataNode杩涚▼銆?/p>

鎴愬姛鍚姩鍚庯紝鍙互璁块棶 Web 鐣岄潰 http://master:50070 鏌ョ湅 NameNode 鍜?Datanode 淇℃伅锛岃繕鍙互鍦ㄧ嚎鏌ョ湅 HDFS 涓殑鏂囦欢銆?/span>

3. 瀹夎HBase鏁版嵁搴?/h1>

HBase鏄竴涓垎甯冨紡鐨勩€侀潰鍚戝垪鐨勫紑婧愭暟鎹簱,婧愪簬Google鐨勪竴绡囪鏂囥€夿igTable锛氫竴涓粨鏋勫寲鏁版嵁鐨勫垎甯冨紡瀛樺偍绯荤粺銆嬨€侶Base浠ヨ〃鐨勫舰寮忓瓨鍌ㄦ暟鎹紝琛ㄦ湁琛屽拰鍒楃粍鎴愶紝鍒楀垝鍒嗕负鑻ュ共涓垪鏃?鍒楃皣(column family)銆傛浜嗚ВHBase鐨勫畼鏂硅祫璁紝璇疯闂?a href="https://hbase.apache.org/" rel="nofollow" target="_blank">HBase瀹樻柟缃戠珯銆侶Base鐨勮繍琛屾湁涓夌妯″紡锛氬崟鏈烘ā寮忋€佷吉鍒嗗竷寮忔ā寮忋€佸垎甯冨紡妯″紡銆?br>鍗曟満妯″紡锛氬湪涓€鍙拌绠楁満涓婂畨瑁呭拰浣跨敤HBase锛屼笉娑夊強鏁版嵁鐨勫垎甯冨紡瀛樺偍锛涗吉鍒嗗竷寮忔ā寮忥細鍦ㄤ竴鍙拌绠楁満涓婃ā鎷熶竴涓皬鐨勯泦缇わ紱鍒嗗竷寮忔ā寮忥細浣跨敤澶氬彴璁＄畻鏈哄疄鐜扮墿鐞嗘剰涔変笂鐨勫垎甯冨紡瀛樺偍銆傝繖閲屽嚭浜庡涔犵洰鐨勶紝鎴戜滑鍙噸鐐硅璁轰吉鍒嗗竷寮忔ā寮忋€?/p>

3.1. HBase瀹夎

涓嬭浇 hbase-2.0.0-bin.tar.gz 鏂囦欢锛屽苟灏嗘枃浠剁Щ鍒?usr/local鐩綍涓?/span>

mv hbase-2.0.0-bin.tar.gz /usr/local

瑙ｅ帇

tar -zxvf hbase-2.0.0-bin.tar.gz

鏂囦欢澶归噸鍛藉悕

mv hbase-2.0.0 hbase

灏唄base涓嬬殑bin鐩綍娣诲姞鍒皃ath涓紝杩欐牱锛屽惎鍔╤base灏辨棤闇€鍒?usr/local/hbase鐩綍涓嬶紝澶уぇ鐨勬柟渚夸簡hbase鐨勪娇鐢ㄣ€傛暀绋嬩笅闈㈢殑閮ㄥ垎杩樻槸鍒囨崲鍒颁簡/usr/local/hbase鐩綍鎿嶄綔锛屾湁鍔╀簬鍒濆鑰呯悊瑙ｈ繍琛岃繃绋嬶紝鐔熺粌涔嬪悗鍙互涓嶅繀鍒囨崲銆?br>缂栬緫/etc/profile鏂囦欢

vim /etc/profile

鍦?etc/profile鏂囦欢灏捐娣诲姞濡備笅鍐呭锛?/p> 

export HBASE_HOME=/usr/local/hbase
export PATH=$HBASE_HOME/bin:$PATH
export HBASE_MANAGES_ZK=true

缂栬緫瀹屾垚鍚庯紝鎸?esc 閫€鍑虹紪杈戞ā寮忥紝鐒跺悗杈撳叆:+wq 锛屾寜鍥炶溅淇濆瓨锛堜篃鍙互鎸塻hift + zz 杩涜淇濆瓨锛夛紝鏈€鍚庡啀鎵цsource鍛戒护浣夸笂杩伴厤缃湪褰撳墠缁堢绔嬪嵆鐢熸晥锛屽懡浠ゅ涓嬶細

source /etc/profile

鏌ョ湅HBase鐗堟湰锛岀‘瀹歨base瀹夎鎴愬姛,鍛戒护濡備笅锛?/p> 

hbase version

3.2. HBase浼垎甯冩ā寮忛厤缃?/h2>

閰嶇疆/usr/local/hbase/conf/hbase-site.xml锛屾墦寮€骞剁紪杈慼base-site.xml锛屽懡浠ゅ涓嬶細

gedit /usr/local/hbase/conf/hbase-site.xml

鍦ㄥ惎鍔℉Base鍓嶉渶瑕佽缃睘鎬base.rootdir锛岀敤浜庢寚瀹欻Base鏁版嵁鐨勫瓨鍌ㄤ綅缃紝鍥犱负濡傛灉涓嶈缃殑璇濓紝hbase.rootdir榛樿涓?tmp/hbase-${user.name},杩欐剰鍛崇潃姣忔閲嶅惎绯荤粺閮戒細涓㈠け鏁版嵁銆傛澶勮缃负HBase瀹夎鐩綍涓嬬殑hbase-tmp鏂囦欢澶瑰嵆锛?usr/local/hbase/hbase-tmp锛?娣诲姞閰嶇疆濡備笅锛?/p> 

<configuration>
        <property>
                <name>hbase.rootdir</name>
                <value>hdfs://localhost:9000/hbase</value>
        </property>
        <property>
                <name>hbase.cluster.distributed</name>
                <value>true</value>
        </property>
</configuration>

鎵撳紑鏂囦欢锛坓edit /usr/local/hbase/conf/hbase-env.sh锛夋坊鍔爅ava鐜鍙橀噺

export JAVA_HOME=/usr/local/java
export HBASE_HOME=/usr/local/hbase
export PATH=$PATH/usr/local/hbase/bin

3.3 HBase闆嗙兢妯″紡閰嶇疆

淇敼master鑺傜偣鐨勯厤缃枃浠秇base-site.xml(gedit /usr/local/hbase/conf/hbase-site.xml)

   <configuration>
        <property>
                <name>hbase.rootdir</name>
                <value>hdfs://master:9000/hbase</value>
        </property>
        <property>
                <name>hbase.cluster.distributed</name>
                <value>true</value>
        </property>
        <property>
                <name>hbase.zookeeper.quorum</name>
                <value>master,slave1</value>
        </property>
        <property>
                <name>hbase.temp.dir</name>
                <value>/usr/local/hbase/tmp</value>
        </property>
        <property>
                <name>hbase.zookeeper.property.dataDir</name>
                <value>/usr/local/hbase/tmp/zookeeper</value>
        </property>
        <property>
                <name>hbase.master.info.port</name>
                <value>16010</value>
        </property>
</configuration>

娉ㄦ剰锛氳嫢鍐嶅鍔犱竴涓粠鏈猴紝hbase.zookeeper.quorum 娣诲姞slave2

淇敼閰嶇疆鏂囦欢regionservers(gedit /usr/local/hbase/conf/regionservers)锛屽垹闄ら噷闈㈢殑localhosts,鏀逛负锛?/p> 

master
slave1

鑻ュ啀澧炲姞涓€涓粠鏈猴紝娣诲姞slave2

浼犻€丠base鑷冲叾瀹僺lave鑺傜偣(浠庢満涓嶉渶涓嬭浇瀹夎鍖咃紝鐢变富鏈轰紶閫佽繃鍘诲嵆鍙紝浠庢満鐜鍙橀噺闇€瑕侀厤缃?锛屽嵆灏嗛厤缃ソ鐨刪base鏂囦欢澶逛紶閫佸埌鍚勪釜鑺傜偣瀵瑰簲浣嶇疆涓婏細

scp -r /usr/local/hbase [email protected]:/usr/local/

娉ㄦ剰锛氭瘡鍙颁粠鏈轰笂闇€瑕侀厤缃瓾Base鐨勭幆澧冨彉閲?/p>

3.4 娴嬭瘯杩愯

棣栧厛鍒囨崲鐩綍鑷矵Base瀹夎鐩綍/usr/local/hbase锛涘啀鍚姩HBase銆傚懡浠ゅ涓嬶細

/usr/local/hadoop/sbin/start-all.sh  #鍚姩hadoop锛屽鏋滃凡鍚姩锛屽垯涓嶇敤鎵ц璇ュ懡浠?/usr/local/hbase/start-hbase.sh     #鍚姩hbase
hbase shell                           #杩涘叆hbase shell锛屽鏋滃彲浠ヨ繘鍏ヨ鏄嶩Base瀹夎鎴愬姛浜?

鍋滄HBase杩愯,鍛戒护濡備笅锛?/p> 

bin/stop-hbase.sh

濡傛灉hbase鍚姩鎴愬姛锛屽垯浣跨敤jps鍛戒护浼氬嚭鐜板涓嬭繘绋?/p>

 
鎶€鏈垎浜浘鐗? src=
涓昏妭鐐硅繘绋?/div>

 
鎶€鏈垎浜浘鐗? src=
浠庤妭鐐硅繘绋?/div>

4. 瀹夎Spark鍐呭瓨璁＄畻寮曟搸

Apache Spark 鏄竴涓柊鍏寸殑澶ф暟鎹鐞嗛€氱敤寮曟搸锛屾彁渚涗簡鍒嗗竷寮忕殑鍐呭瓨鎶借薄銆係park 鏈€澶х殑鐗圭偣灏辨槸蹇紝鍙瘮 Hadoop MapReduce 鐨勫鐞嗛€熷害蹇?100 鍊嶃€係park鍩轰簬Hadoop鐜锛孒adoop YARN涓篠park鎻愪緵璧勬簮璋冨害妗嗘灦锛孒adoop HDFS涓篠park鎻愪緵搴曞眰鐨勫垎甯冨紡鏂囦欢瀛樺偍銆?/p>

4.1. Spark瀹夎

Spark鐨勫畨瑁呰繃绋嬭緝涓虹畝鍗曪紝鍦ㄥ凡瀹夎濂?Hadoop 鐨勫墠鎻愪笅锛岀粡杩囩畝鍗曢厤缃嵆鍙娇鐢紝棣栧厛涓嬭浇 spark-2.3.0-bin-hadoop2.7.tgz 鏂囦欢锛屽苟灏嗘枃浠剁Щ鍒?usr/local鐩綍涓?/span>

mv spark-2.3.0-bin-hadoop2.7.tgz /usr/local

瑙ｅ帇

cd /usr/local
tar -zxvf spark-2.3.0-bin-hadoop2.7.tgz

鏂囦欢澶归噸鍛藉悕

mv spark-2.3.0 spark

缂栬緫/etc/profile鏂囦欢锛屾坊鍔犵幆澧冨彉閲?/p> 

vim /etc/profile

鍦?etc/profile鏂囦欢灏捐娣诲姞濡備笅鍐呭锛?/p> 

export SPARK_HOME=/usr/local/spark
export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin

缂栬緫瀹屾垚鍚庯紝淇濆瓨閫€鍑猴紝鍐嶆墽琛宻ource鍛戒护浣夸笂杩伴厤缃湪褰撳墠缁堢绔嬪嵆鐢熸晥锛屽懡浠ゅ涓嬶細

source /etc/profile

4.2. Spark鍗曟満閰嶇疆

閰嶇疆鏂囦欢spark-env.sh

cd /usr/local/spark
cp ./conf/spark-env.sh.template ./conf/spark-env.sh

缂栬緫spark-env.sh鏂囦欢(vim ./conf/spark-env.sh)锛屽湪绗竴琛屾坊鍔犱互涓嬮厤缃俊鎭?

export JAVA_HOME=/usr/local/java
export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop
export HADOOP_HDFS_HOME=/usr/local/hadoop
export SPARK_HOME=/usr/local/spark
export SPARK_DIST_CLASSPATH=$(/usr/local/hadoop/bin/hadoop classpath)
SPARK_MASTER_WEBUI_PORT=8079

4.3. Spark闆嗙兢閰嶇疆

鍦╩aster涓婇厤缃枃浠秙park-env.sh

cd /usr/local/spark
cp ./conf/spark-env.sh.template ./conf/spark-env.sh

缂栬緫spark-env.sh鏂囦欢(vim ./conf/spark-env.sh)锛屽湪绗竴琛屾坊鍔犱互涓嬮厤缃俊鎭?

export JAVA_HOME=/usr/local/java
export SCALA_HOME=/usr/local/scala
export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop
export HADOOP_HDFS_HOME=/usr/local/hadoop
export SPARK_HOME=/usr/local/spark
export SPARK_MASTER_IP=master
export SPARK_MASTER_PORT=7077
export SPARK_MASTER_HOST=master
export SPARK_WORKER_CORES=2
export SPARK_WORKER_PORT=8901
export SPARK_WORKER_INSTANCES=1
export SPARK_WORKER_MEMORY=2g
export SPARK_DIST_CLASSPATH=$(/usr/local/hadoop/bin/hadoop classpath)
export SPARK_MASTER_WEBUI_PORT=8079

淇濆瓨骞跺埛鏂伴厤缃細

source spark-env.sh

閰嶇疆浠庢満鍒楄〃锛?/p> 

cp slaves.template slaves
gedit slaves

鍦ㄦ渶鍚庡姞涓婏細

master
slave1

鎶婁富鏈虹殑spark鏂囦欢澶瑰鍒跺埌浠庢満锛屽鍒惰剼鏈涓嬶細

scp -r /usr/local/spark [email protected]:/usr/local

娉ㄦ剰锛氭瘡鍙颁粠鏈轰笂闇€瑕侀厤缃甋park鐨勭幆澧冨彉閲?/p>

4.4 楠岃瘉Spark瀹夎鍜岄厤缃?/h2>

閫氳繃杩愯Spark鑷甫鐨勭ず渚嬶紝楠岃瘉Spark鏄惁瀹夎鎴愬姛銆?/p> 

cd /usr/local/spark
./sbin/start-all.sh
bin/run-example SparkPi 2>&1 | grep "Pi is"

杩愯缁撴灉濡備笅鍥炬墍绀猴紝鍙互寰楀埌蟺 鐨?14浣嶅皬鏁拌繎浼煎€硷細

 
鎶€鏈垎浜浘鐗? src=
Spark Pi璁＄畻缁撴灉

鍦ㄤ富鏈虹殑娴忚鍣ㄨ緭鍏?a href="http://master:8079/" rel="nofollow" target="_blank">http://master:8079锛堥泦缇ゆā寮忥級鎴栬€?a href="http://localhost:8079/" rel="nofollow" target="_blank">http://localhost:8079(鍗曟満妯″紡)灏卞彲浠ョ湅鍒版湁涓や釜鑺傜偣鍦╯park闆嗙兢涓娿€?/span>

 

5. 瀹夎hive

Hive鏄竴涓灦鏋勫湪Hadoop涔嬩笂鐨勬暟鎹粨搴撳熀纭€宸ュ叿锛岀敤鏉ュ鐞嗙粨鏋勫寲鏁版嵁锛屼负澶ф暟鎹煡璇㈠拰鍒嗘瀽鎻愪緵鏂逛究銆傛渶鍒濓紝Hive鏄敱Facebook寮€鍙戯紝鍚庢潵鐢盇pache杞欢鍩洪噾浼氬紑鍙戯紝骞朵綔涓鸿繘涓€姝ュ皢瀹冧綔涓哄悕涔変笅Apache Hive涓轰竴涓紑婧愰」鐩€侶ive 涓嶆槸涓€涓叧绯绘暟鎹簱锛屼篃涓嶆槸涓€涓璁＄敤浜庤仈鏈轰簨鍔″锛圤LTP锛夊疄鏃舵煡璇㈠拰琛岀骇鏇存柊鐨勮瑷€銆傜畝鍗曠殑璇达紝Hive灏辨槸鍦℉adoop涓婃灦浜嗕竴灞係QL鎺ュ彛锛屽彲浠ュ皢SQL缈昏瘧鎴怣apReduce鍘籋adoop涓婃墽琛岋紝杩欐牱灏变娇寰楁暟鎹紑鍙戝拰鍒嗘瀽浜哄憳寰堟柟渚跨殑浣跨敤SQL鏉ュ畬鎴愭捣閲忔暟鎹殑缁熻鍜屽垎鏋愶紝鑰屼笉蹇呬娇鐢ㄧ紪绋嬭瑷€寮€鍙慚apReduce閭ｄ箞楹荤儲銆?/p>

5.1. Hive瀹夎

涓嬭浇 apache-hive-1.2.2-bin.tar.gz 鏂囦欢锛屽苟灏嗘枃浠剁Щ鍒?usr/local鐩綍涓?/span>

mv apache-hive-1.2.2-bin.tar.gz /usr/local

瑙ｅ帇

tar -zxvf apache-hive-1.2.2-bin.tar.gz

鏂囦欢澶归噸鍛藉悕

mv apache-hive-1.2.2 hive

缂栬緫/etc/profile鏂囦欢锛岄厤缃幆澧冨彉閲?/p> 

vim /etc/profile

鍦?etc/profile鏂囦欢灏捐娣诲姞濡備笅鍐呭锛?/p> 

export HIVE_HOME=/usr/local/hive
export PATH=$PATH:$HIVE_HOME/bin

缂栬緫瀹屾垚鍚庯紝淇濆瓨閫€鍑猴紝鍐嶆墽琛宻ource鍛戒护浣夸笂杩伴厤缃湪褰撳墠缁堢绔嬪嵆鐢熸晥锛屽懡浠ゅ涓嬶細

source /etc/profile

5.2. 瀹夎骞堕厤缃甅ySQL

鎴戜滑閲囩敤MySQL鏁版嵁搴撲繚瀛楬ive鐨勫厓鏁版嵁锛岃€屼笉鏄噰鐢℉ive鑷甫鐨刣erby鏉ュ瓨鍌ㄥ厓鏁版嵁銆倁buntu涓婱ysql鐨勫畨瑁呮瘮杈冪畝鍗曪紝鐩存帴杩愯濡備笅鍛戒护銆傚湪瀹夎杩囩▼涓紝浼氳姹傞厤缃敤鎴峰悕鍜屽瘑鐮侊紝杩欎釜涓€瀹氳璁颁綇銆?/p> 

apt-get install mysql-server

鍚姩骞剁櫥闄唌ysql shell

service mysql start
mysql -u root -p  #鐧婚檰shell鐣岄潰

鏂板缓hive鏁版嵁搴?/p> 

#杩欎釜hive鏁版嵁搴撲笌hive-site.xml涓璴ocalhost:3306/hive鐨刪ive瀵瑰簲锛岀敤鏉ヤ繚瀛榟ive鍏冩暟鎹?mysql> create database hive;

灏唄ive鏁版嵁搴撶殑瀛楃缂栫爜璁剧疆涓簂atin1锛堥噸瑕侊級

mysql> alter database hive character set latin1;

5.3. Hive閰嶇疆

淇敼/usr/local/hive/conf涓嬬殑hive-site.xml锛屾墽琛屽涓嬪懡浠わ細

cd /usr/local/hive/conf
mv hive-default.xml.template hive-default.xml

涓婇潰鍛戒护鏄皢hive-default.xml.template閲嶅懡鍚嶄负hive-default.xml锛岀劧鍚庯紝浣跨敤vim缂栬緫鍣ㄦ柊寤轰竴涓厤缃枃浠秇ive-site.xml锛屽懡浠ゅ涓嬶細

cd /usr/local/hive/conf
vim hive-site.xml

鍦╤ive-site.xml涓坊鍔犲涓嬮厤缃俊鎭紝鍏朵腑锛歎SERNAME鍜孭ASSWORD鏄疢ySQL鐨勭敤鎴峰悕鍜屽瘑鐮併€?/p> 

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:mysql://localhost:3306/hive?createDatabaseIfNotExist=true</value>
    <description>JDBC connect string for a JDBC metastore</description>
  </property>
  <property>
    <name>javax.jdo.option.ConnectionDriverName</name>
    <value>com.mysql.jdbc.Driver</value>
    <description>Driver class name for a JDBC metastore</description>
  </property>
  <property>
    <name>javax.jdo.option.ConnectionUserName</name>
    <value>USERNAME</value>
    <description>username to use against metastore database</description>
  </property>
  <property>
    <name>javax.jdo.option.ConnectionPassword</name>
    <value>PASSWORD</value>
    <description>password to use against metastore database</description>
  </property>
</configuration>

鐒跺悗锛屾寜閿洏涓婄殑鈥淓SC鈥濋敭閫€鍑簐im缂栬緫鐘舵€侊紝鍐嶈緭鍏?wq锛屼繚瀛樺苟閫€鍑簐im缂栬緫鍣ㄣ€傜敱浜嶩ive鍦ㄨ繛鎺ySQL鏃堕渶瑕?a href="https://dev.mysql.com/downloads/connector/j/" rel="nofollow" target="_blank">JDBC椹卞姩锛屾墍浠ラ鍏堥渶瑕佷笅杞藉搴旂増鏈殑椹卞姩锛岀劧鍚庡皢椹卞姩绉诲姩鍒?usr/local/hive/lib涓€?/p> 

#瑙ｅ帇
tar -zxvf mysql-connector-java-5.1.47.tar.gz
#灏唌ysql-connector-java-5.1.47.tar.gz鎷疯礉鍒?usr/local/hive/lib鐩綍涓?cp mysql-connector-java-5.1.47/mysql-connector-java-5.1.47-bin.jar /usr/local/hive/lib

鍚姩hive锛堝惎鍔╤ive涔嬪墠锛岃鍏堝惎鍔╤adoop闆嗙兢锛夈€?/p> 

./usr/local/hadoop/sbin/start-all.sh #鍚姩hadoop锛屽鏋滃凡缁忓惎鍔紝鍒欎笉鐢ㄦ墽琛岃鍛戒护
hive  #鍚姩hive

5.4. Spark鍜孒ive鐨勬暣鍚?/h2>

Hive鐨勮绠楀紩鎿庨粯璁や负MapReduce锛屽鏋滄兂瑕佺敤Spark浣滀负Hive鐨勮绠楀紩鎿庯紝鍙互鍙傝€冩枃绔?a href="https://www.jianshu.com/p/b373ee023e51" target="_blank">缂栬瘧Spark婧愮爜鏀寔Hive骞堕儴缃?/a>

6. 缁撹

鏈枃浠嬬粛浜嗗ぇ鏁版嵁鐜鐨勬惌寤鸿繃绋嬶紝鏃ㄥ湪璁╁寮熷濡逛滑灏戣俯鍧戙€傚悗闈紝浣滆€呭皢涓哄ぇ瀹跺鐚浣曞埄鐢↗ava+Scala鏉ュ紑鍙戝ぇ鏁版嵁搴旂敤銆傚鏋滆寰楁枃绔犳湁鐢ㄧ殑璇濓紝鍒繕璁扮偣璧炲摝O(鈭鈭?O~锛?/p>



以上是关于Hadoop+HBase+Spark+Hive鐜鎼缓的主要内容，如果未能解决你的问题，请参考以下文章

Hadoop相关项目Hive-Pig-Spark-Storm-HBase-Sqoop

Mac下hadoop，hive, hbase，spark单机环境搭建

大数据(hadoop,hive,hbase,spark,flume等)各技术间的关系

Todo转载全栈工程师-Hadoop, HBase, Hive, Spark

大数据学习系列之七 ----- Hadoop+Spark+Zookeeper+HBase+Hive集

《hadoop 集群搭建spark安装Hbase安装Hive安装Kafka安装》