CentOS 7涓嬫渶鏂扮増(6.2.4)ELK+Filebeat+Log4j鏃ュ織闆嗘垚鐜鎼缓瀹屾暣鎸囧崡

Posted 2020-11-10

鏍囩锛?a href='http://www.mamicode.com/so/1/get' title='get'>get   oca   pop   褰卞搷   ase   exception   spec   

鐜板湪鐨勫叕鍙哥敱浜庣粷澶ч儴鍒嗛」鐩兘閲囩敤鍒嗗竷寮忔灦鏋勶紝寰堟棭灏遍噰鐢‥LK浜嗭紝鍙笉杩囨渶杩戝洜涓洪澶栫殑宸ヤ綔闇€瑕侊紝浠旂粏鐨勭爺绌朵簡鍒嗗竷寮忕郴缁熶腑锛屾€庝箞鏍风殑鏃ュ織瑙勮寖鍜屾灦鏋勬墠鏄悎鐞嗗拰鑳藉鏈夋晥鎻愰珮闂鎺掓煡鏁堢巼鐨勩€傜粡杩囦粩缁嗙殑鍒嗘瀽鍜岀爺绌讹紝纭畾涓嬮潰鐨勬灦鏋勫簲璇ユ槸姣旇緝鍚堢悊鐨勪箣涓€锛團ilebeat涔熸敮鎸佺洿鎺ュ啓鍒癊S锛夈€傚叧浜嶦LK鐨勫悇绉嶆灦鏋勪互鍙婁紭缂虹偣锛屽彲浠ュ弬鑰僪ttps://www.ibm.com/developerworks/cn/opensource/os-cn-elk-filebeat/index.html銆?/p>

 

棣栧厛锛屼笉绠℃槸涓嶆槸鍒嗗竷寮忔灦鏋勶紝閮藉簲璇ヨВ鍐宠姹傛棩蹇椾笂涓嬫枃鍏宠仈鐨勯棶棰橈紝杩欏彲浠ラ€氳繃Log4j鑷甫鐨凬DC鏉ュ疄鐜般€備互Spring MVC涓轰緥锛屽彲浠ヤ娇鐢℉andlerInterceptor鍦℉andlerInterceptor涓缃甆DC.push(SessionBeanUtil.getSessionKey(request).substring(0, 8) + "_" + path + "_" + formatter.format(new Date()));锛屽湪postHandle涓璑DC.pop();銆?/p>

鍏舵锛屽浜庡垎甯冨紡璇锋眰锛岃繕瑕佽В鍐冲叏灞€鐨勮姹傛棩蹇椾笂涓嬫枃鍏宠仈鐨勯棶棰橈紝杩欓渶瑕佷緷璧栦笌鍏蜂綋鐨凴PC妗嗘灦鏉ュ疄鐜帮紝浠ubbo涓轰緥锛屽彲浠ラ€氳繃filter鍦ㄨ姹傜鍜屾湇鍔＄璁剧疆銆?/p>

绗笁锛屾湇鍔＄殑璋冪敤閾惧簲璇ュ彲浠ョ畻鍋氭槸鏃ュ織妗嗘灦鐨勮寖鐣达紝杩欏彲浠ラ€氳繃zipkin闆嗘垚鏉ュ疄鐜般€?/p>

鍦ㄥ疄鐜颁笂锛屽浜庢湇鍔＄殑璋冪敤閾惧彲浠ヤ娇鐢ㄥ崟鐙殑ES瀛樺偍銆佷篃鍙互鍦ㄤ竴涓ぇ鐨凟S闆嗙兢涓瓨鍌ㄤ负涓€涓猧ndex銆?/p>

浠嶦LK 6.0寮€濮嬶紝瀵逛簬婧愭槸log4j鐨勫満鏅紝涓嶅啀寤鸿浣跨敤logstash-input-log4j锛岃€屾槸浣跨敤Filebeat浣滀负搴旂敤绔殑鏃ュ織浠ｇ悊锛屽叿浣撳彲浠ュ弬鑰僪ttps://www.elastic.co/guide/en/logstash/current/plugins-inputs-log4j.html銆?/p>

鎵€浠ワ紝鍦ㄦ湰鏂囩殑鎼缓杩囩▼涓紝绗旇€呴噰鐢ㄧ殑涔熸槸Filebeat鐨勬柟妗堛€?/p>

鐜璇存槑涓庡噯澶?/h3>

Filebeat涓€鑸畨瑁呭湪浜х敓鏃ュ織鐨勬湇鍔″櫒锛岃繖閲宼omcat鍦╳indows 10 x64涓嬶紝鎵€浠ilebeat涔熼渶瑕佸畨瑁呭湪windows涓嬨€?/p>

CentOS 7.4 64浣?ip 192.168.230.128锛孍LK瀹夎鍦╟entos涓嬶紝缁熶竴瀹夎鍦?usr/local/app鐩綍涓嬨€?/p>

鐗堟湰涓哄畨瑁呮椂鐨勬渶鏂扮増鏈細

Elasticsearch 6.2.4

Kibana 6.2.4

Logstash 6.2.4

浠巋ttps://www.elastic.co/cn/downloads涓嬭浇骞惰В鍘嬶紝濡備笅锛?/p>

鍥犱负ES涓昏閫氳繃restful api瀵瑰鎻愪緵鏈嶅姟锛屾墍浠ヤ竴鑸畨瑁匛S鏃堕『甯﹀畨瑁卐lasticsearch-head锛屽畠鎻愪緵浜唚eb鎺у埗鍙般€俥lasticsearch-head閫氳繃婧愮爜鐨勬柟寮忔墭绠″湪git涓婏紝鎵€浠ラ渶瑕佸畨瑁呬笅git锛屽悓鏃秂lasticsearch-head鏄竴涓猲odejs搴旂敤锛屾墍浠ヨ繕闇€瑕佸叿鏈塶ode銆?/p>

鎵€浠ュ湪姝ｅ紡寮€濮嬪墠锛屾垜浠渶瑕佺‘淇漜entos涓嬩笅鍒楀熀纭€璁炬柦宸茬粡鍏峰锛?/p>

1銆乬it宸茬粡瀹夎銆倅um install git

2銆佸鏋滃笇鏈涙簮鐮佸畨瑁卬odejs锛岀‘淇漡cc婊¤冻node鐨勮姹傦紝鎴栬€呯洿鎺ヤ娇鐢ㄥ凡缁忕紪璇戝ソ鐨勭増鏈紝 绗旇€呯洿鎺ヤ娇鐢ㄧ紪璇戝ソ鐨勭増鏈紝鍙粠https://nodejs.org/en/download/releases/閫夋嫨鍏蜂綋鐨勭増鏈紝鍩虹杩愯鐜鏈€濂戒笉瑕佷娇鐢ㄦ渶鏂扮増鏈紝杩欓噷鎴戜滑浣跨敤v4.9銆備笅杞借В鍘嬪悗鍗冲彲鐢ㄣ€?/p>

3銆佸洜涓篹lk涓嶈兘浣跨敤root杩愯锛屾墍浠ラ渶瑕佹柊寤轰竴涓猠lk鐢ㄦ埛骞惰缃幆澧冨彉閲忋€?/p>

groupadd elk

useradd -g elk elk

璁剧疆elk鐢ㄦ埛鐨勭幆澧冨彉閲忥細

su - elk

vim .bash_profile  澧炲姞濡備笅锛?/p>

export NODE_HOME=/usr/local/app/node-v4.9.1-linux-x64
PATH=$NODE_HOME/bin:$PATH:$HOME/.local/bin:$HOME/bin
export NODE_PATH=$NODE_HOME/lib/node_modules
export PATH
4銆佷笅杞紼LK濂椾欢鏈韩骞惰В鍘嬶細

cd /usr/local/app

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.2.4.tar.gz

wget https://artifacts.elastic.co/downloads/kibana/kibana-6.2.4-linux-x86_64.tar.gz

wget https://artifacts.elastic.co/downloads/logstash/logstash-6.2.4.tar.gz

wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.2.4-linux-x86_64.tar.gz

git clone https://github.com/elastic/elasticsearch-head.git 

6銆佸洜涓篍S鍜宭ogstash閮芥槸java搴旂敤锛屾墍浠ョ‘淇濆畨瑁呬簡JDK 1.8锛屽彲浠ヤ娇鐢╟entos鑷甫鐨刼penjdk锛屼笉杩囩敓浜х幆澧冧笅涓€鑸笉寤鸿浣跨敤openjdk锛屾澶勪粎涓轰簡娴嬭瘯鏂逛究銆?/p>

ELK鐜鎼缓

鍥犱负浣跨敤鐨勬槸vmware锛屾€诲叡鍐呭瓨鎵嶅垎閰嶄簡1G锛屾墍浠ヤ负浜嗛伩鍏峅OM锛屽ES浠ュ強logstash鍐呭瓨鍧囪繘琛屼簡璋冩暣锛岄檺鍒朵负256M銆?/p>

ES瀹夎

鍥犱负es闇€瑕侀潪root鐢ㄦ埛杩愯锛屾墍浠ュ浜嶦S鐩稿叧鐨勬墍鏈夋搷浣滃潎鍦╡lk鐢ㄦ埛涓嬭繍琛屻€?/p>

su - elk

鏇存敼ES閰嶇疆锛?/p>

[[email protected] ~]$ cd /usr/local/app/elasticsearch-6.2.4/config/
[[email protected] config]$ ll
鎬荤敤閲?16
-rw-rw----. 1 elk elk 2976 6鏈? 1 16:22 elasticsearch.yml
-rw-rw----. 1 elk elk 2771 6鏈? 2 13:54 jvm.options
-rw-rw----. 1 elk elk 5091 4鏈? 13 04:33 log4j2.properties
[[email protected] config]$ pwd
/usr/local/app/elasticsearch-6.2.4/config

vim elasticsearch.yml

# 纭繚涓嬪垪鍙傛暟琚纭缃細

cluster.name: logger   # ES闆嗙兢鐨勫悕瀛?/p>

node.name: node-1

path.data: /usr/local/app/elasticsearch-6.2.4/data

path.logs: /usr/local/app/elasticsearch-6.2.4/log

bootstrap.memory_lock: false   # 瀵逛簬闈炰笓鐢‥S锛屽缓璁缃负false锛岄粯璁や负true
bootstrap.system_call_filter: false

network.host: 0.0.0.0  # 鏀寔杩滅▼璁块棶

http.port: 9200 # restful api璁块棶鎺ュ彛

http.cors.enabled: true      #鍏佽ES head璺ㄥ煙璁块棶
http.cors.allow-origin: "*"   #鍏佽ES head璺ㄥ煙璁块棶

vim jvm.options # JVM鍙傛暟鍦ㄨ繖涓枃浠朵腑璁剧疆锛屽綋鐒跺懡浠よ涔熷彲浠?/p>

-Xms256m
-Xmx256m

涓婅堪閰嶇疆璋冩暣瀹屾垚鍚庯紝灏卞彲浠ュ惎鍔‥S浜嗐€?/p>

[[email protected] bin]$ nohup ./elasticsearch &

[[email protected] bin]$ ps axu | grep elas

windows涓嬫祻瑙堝櫒璁块棶涓媓ttp://192.168.230.128:9200/

杩欐牱锛孍S灏卞畨瑁呭ソ浜嗐€?/p>

ES head瀹夎

闇€瑕佹敞鎰忕殑鏄紝铏界劧ES head鍙互璁や负鏄痚s鐨勬彃浠讹紝浣嗘槸瀹冧笉鑳芥斁鍦?ES_HOME/plugins鐩綍涓嬶紝鍥犱负瀹冨苟涓嶇鍚圗S鎻掍欢鐨勮鑼冿紝鍚﹀垯ES鍚姩浼氬け璐ャ€?/p>

[[email protected] elasticsearch-head]$ pwd
/usr/local/app/elasticsearch-head
[[email protected] elasticsearch-head]$ npm config set registry https://registry.npm.taobao.org

npm install

Please report this full log at https://github.com/Medium/phantomjs

npm ERR! Darwin 15.0.0

npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "install"

npm ERR! node v4.4.3

npm ERR! npm  v3.10.9

npm ERR! code ELIFECYCLE

npm ERR! [email protected] install: `node install.js`

npm ERR! Exit status 1

npm ERR! 

npm ERR! Failed at the [email protected] install script 鈥榥ode install.js

濡傛灉鍑虹幇涓婅堪閿欒锛屽垯鎵ц涓嬪垪鍚嶇О锛?/p>

npm install phantomjs-prebuilt@2.1.14 --ignore-scripts

鐒跺悗閲嶆柊鎵ц

npm install

杩欐牱ES head灏卞畨瑁呭ソ浜嗐€備笉瑕佹€ョ潃鍚姩锛侊紒锛?/p>

濡傛灉鍙槸鏈満璁块棶锛屼笅闈㈢殑閰嶇疆淇敼涓嶆槸蹇呴』鐨勩€傚鏋滆鍏朵粬鏈哄櫒璁块棶锛屽垯闇€瑕佷慨鏀癸紝涓€鑸潵璇达紝鍙鏄湇鍔″櫒搴旂敤锛岄兘鏄€氳繃杩滅▼璁块棶鐨勩€?/p>

[[email protected] elasticsearch-head]$ vim Gruntfile.js

鎼滅储server锛屽湪鍏秓ptions瀵硅薄灞炴€т笅澧炲姞涓€涓猦ostname灞炴€э紝鍊间负"*"锛屽涓嬶細

鐜板湪灏卞彲浠ュ惎鍔╡s head浜嗭紝濡備笅锛?/p>

 璁块棶涓媓ttp://192.168.230.128:9100/鍚э紝濡備笅锛?/p>

 

logstash瀹夎

logstash鍜宬ibana鍙互浣跨敤root鐢ㄦ埛鏉ュ惎鍔ㄣ€?/p>

鍒涘缓涓€涓猯ogstash閰嶇疆鏂囦欢锛屾瘮濡俵ogstash-es.conf锛岄厤缃粠filebeat璇诲彇鏁版嵁婧愶紝杈撳嚭鍒癳s锛屼负浜嗙畝鍖栬捣瑙侊紝蹇界暐杩囨护鍣紙瀹為檯鐢熶骇涓紝涓€鑸渶瑕侀厤缃繃婊ゅ櫒瀵规棩蹇楄繘琛岃鑼冨寲澶勭悊鍜屽垎绫伙級

cd /usr/local/app/logstash-6.2.4/config

[[email protected] config]$ vim logstash-es.conf

input {
  stdin { }
  beats {
    port => 5000
    ssl => false
  }
}
output {
    elasticsearch {
        action => "index"
        hosts => "127.0.0.1:9200"
        index  => "logstash-%{+YYYY-MM}"
    }
    stdout { codec=> rubydebug }
}

涓轰簡娴嬭瘯鏂逛究锛屽悓鏃跺紑鍚帶鍒跺彴杈撳叆鍜岃緭鍑恒€?/p>

[[email protected] config]$ vim jvm.options   #璁剧疆鏈€澶氫娇鐢?56m鍐呭瓨

-Xms256m
-Xmx256m

鍚姩logstash銆?/p>

闅忎究杈撳叆涓瓧绗︿覆娴嬭瘯涓嬶紝濡備笂锛岀湅涓媓ttp://192.168.230.128:9100/

杩欐牱logstash瀵逛簬鍐欏叆es鍜屼粠鎺у埗鍙拌緭鍏ョ殑閰嶇疆灏辨纭簡锛宖ilebeat杩橀渶瑕佺瓑鎴戜滑鍚庨潰楠岃瘉銆傛垜浠厛瀹屾垚Kibana鐨勫畨瑁呬笌閰嶇疆銆?/p>

Kibana瀹夎

kibana涔熸槸涓猲odejs搴旂敤銆傞鍏堟潵淇敼kibana鐨勯厤缃細

[[email protected] app]$ cd kibana-6.2.4-linux-x86_64/config/
[[email protected] config]$ ll
鎬荤敤閲?8
-rw-r--r--. 1 zhjh256 zhjh256 4647 6鏈? 2 12:32 kibana.yml
[[email protected] config]$ vim kibana.yml  # 纭繚涓嬪垪閰嶇疆姝ｇ‘

server.port: 5601

server.host: "192.168.230.128"

elasticsearch.url: "http://localhost:9200"

涓婅堪閰嶇疆瀹屾垚鍚庯紝灏卞彲浠ュ惎鍔ㄤ簡銆?/p>

 

璁块棶涓媓ttp://192.168.230.128:5601銆?/p>

绗竴娆¤闂殑鏃跺€欙紝浼氳姹傝缃甀ndex Pattern锛屽洜涓烘垜浠湪logstash-es.conf涓缃负logstash-%{+YYYY-MM}锛屾墍浠ヨ缃负logstash-*灏卞彲浠ヤ簡銆?/p>

Discover鏄富瑕佺殑鏌ヨ浜や簰鐣岄潰锛屽涓嬫墍绀猴細

 

鎼滅储涓嬶紝鎴戜滑鍒氭墠鍦ㄦ帶鍒跺彴杈撳叆鐨勬祴璇曟枃瀛楋細

 

鏈夋椂鍊欏湪璁块棶discover鐨勬椂鍊欙紝鎻愮ずno results found锛屽涓嬫墍绀猴細

 杩欓€氬父鏄敱浜庨粯璁ょ殑鏌ヨ鏃堕棿鑼冨洿澶煭鐨勫師鍥狅紝鍙互閫氳繃鍙充笂瑙掔殑TimeRange鏉ヨ缃煡璇㈢殑鏃堕棿鑼冨洿銆?/p>

鍒拌繖閲岋紝ELK鐨勭幆澧冩惌寤轰笌鍩烘湰閰嶇疆灏卞畬鎴愪簡銆?/p>

鏇村鐨勯厤缃笌浼樺寲鍙傝鍚勫畼鏂规枃妗ttps://www.elastic.co/guide/index.html銆傚悗缁湁鏃堕棿鐨勮瘽锛岀瑪鑰呬細鍐嶅啓涓€绡囥€?/p>

Filebeat瀹夎涓庨厤缃?/h4>

閲囩敤Filebeat浣滀负婧愮浠ｇ悊涔嬪悗锛屽噯纭殑璇达紝璺焞og4j宸茬粡娌℃湁鍏崇郴浜嗐€傛墍浠ヨ繖閲屽亣璁捐鑰呯煡閬搇og4j鐨勯厤缃紝鐢熸垚鐨勬枃浠跺湪d:httxlogs鐩綍銆?/p>

鍥犱负windows涓婩ilebeat鐨勫惎鍔ㄨ剼鏈槸浣跨敤powershell鑴氭湰缂栧啓鐨勶紝鎵€浠ョ‘淇濆畨瑁呬簡ps锛寃indows 10涓嬭嚜甯︺€?/p>

浠巋ttps://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.2.4-windows-x86_64.zip涓嬭浇windows鐗堟湰鐨刦ilebeat銆俧ilebeat鍙互瀹夎鍦ㄤ换浣曠洰褰曪紝杩欓噷浠:chrome涓嬭浇filebeat涓轰緥銆?/p>

鎵撳紑閰嶇疆鏂囦欢D:chrome涓嬭浇filebeatfilebeat.yml锛岀‘淇濅笅鍒楄缃纭細

filebeat.prospectors:
- type: log
  enabled: true  #鍚敤閰嶇疆
  paths:
    - D:httxlogs*  #璁剧疆鐩戞帶璺緞
#output.elasticsearch:   #绂佺敤ES鍐欏叆
output.logstash:            #鍚敤logstash鍐欏叆
  # The Logstash hosts
  hosts: ["192.168.230.128:5000"]

瀹夎涓庡惎鍔╢ilebeat銆?/p>

鎵撳紑powershell锛屾墽琛屽涓嬪懡浠わ細

PS C:Usersadmin> cd 鈥楧:chrome涓嬭浇filebeat鈥?PS D:chrome涓嬭浇filebeat> .install-service-filebeat.ps1

瀹夊叏璀﹀憡
璇峰彧杩愯浣犱俊浠荤殑鑴氭湰銆傝櫧鐒舵潵鑷?Internet 鐨勮剼鏈細鏈変竴瀹氱殑鐢ㄥ锛屼絾姝よ剼鏈彲鑳戒細鎹熷潖浣犵殑璁＄畻鏈恒€傚鏋滀綘淇′换姝よ剼鏈紝璇蜂娇鐢?Unblock-File cmdlet 鍏佽杩愯璇ヨ剼鏈紝鑰屼笉鏄剧ず姝よ鍛婃秷鎭€傛槸鍚﹁杩愯 D:chrome涓嬭浇filebeatinstall-service-filebeat.ps1?
[D] 涓嶈繍琛?D)  [R] 杩愯涓€娆?R)  [S] 鏆傚仠(S)  [?] 甯姪 (榛樿鍊间负鈥淒鈥?: R

Status   Name               DisplayName
------   ----               -----------
Stopped  filebeat           filebeat


PS D:chrome涓嬭浇filebeat> Start-Service filebeat
PS D:chrome涓嬭浇filebeat> Stop-Service filebeat

filebeat鐨勬棩蹇楀湪C:ProgramDatafilebeatlogs鐩綍涓嬶紝杩欐槸鍐欐鐨勶紝涓嶇煡閬撳摢閲屽彲浠ユ洿鏀广€?/p>

filebeat浼氬畾鏈熻緭鍑烘棩蹇椾互鍙婇亣鍒扮殑寮傚父淇℃伅銆?/p>

鏈€鍚庯紝鎴戜滑鍐嶅洖鍒発ibana鎺у埗鍙帮紝鐪嬩笅log4j鐩稿叧鐨勬棩蹇楋紝濡備笅锛?/p>

宸﹁竟鎺у埗瑕佹樉绀哄摢浜涘垪锛屾瘮濡傛樉绀烘潵婧愬拰涓绘満锛?/p>

 

 鍒版涓烘锛屾暣涓狤LK+log4j鐨勯泦鎴愭湰韬氨瀹屾垚浜嗐€備絾鏄杈惧埌楂樻晥鍙敤鐨勬楠わ紝涓嬪垪闂杩橀渶瑕佽繘涓€姝ョ爺绌讹細

1銆乫ilebeat璇诲彇鏂囦欢浼间箮鏄互琛屼负鍗曚綅锛岃繖鍦╡xception鍫嗘爤涓嬫槸涓嶅彲鎺ュ彈鐨勶紝闇€瑕乴ogstash鎴栬€呭叾浠栨柟寮忎簩娆″鐞嗐€?/p>

2銆乪s head鐨勬帶鍒跺彴搴旇鏉ヨ浣滀负es鐨勭洃鎺ц繕鍙互锛屼絾鏄綔涓烘棩蹇椾氦浜掑熀鏈笂娌′粈涔堢敤锛屾晥鏋滃お宸簡銆俴ibana杩樹笉閿欙紝鍙笉杩噁ilter涓嶆槸璧风爜搴旇鍥哄畾涔堬紵锛?/p>

3銆乪lk鐨勮缁嗛厤缃互鍙婁紭鍖栥€?/p>

4銆乫ilebeat鏃ュ織涓嚭鐜?sync.go:105: ERR Failed to publish events (host: 10.172.0.165:5044:10200), caused by: write tcp "锛屼絾鏄笉褰卞搷鏃ュ織鐨勫彂閫併€?/p>

 

鍙傝€冿細

https://serverfault.com/questions/911440/filebeat-cant-connect-to-logstash-on-another-server

https://elasticsearch.cn/question/3157

以上是关于CentOS 7涓嬫渶鏂扮増(6.2.4)ELK+Filebeat+Log4j鏃ュ織闆嗘垚鐜鎼缓瀹屾暣鎸囧崡的主要内容，如果未能解决你的问题，请参考以下文章

CentOS 7安装部署ELK 6.2.4

CentOS7搭建ELK-6.2.4版本

鎵嬫妸鎵嬫暀浣犲湪Windows鐜涓嬪崌绾

细做ELK

铻嶄簯鍗虫椂閫氳SDK闆嗘垚 -- 鍥藉唴鍘傚晢鎺ㄩ€侀泦鎴愯俯鍧戠瘒(Android骞冲彴)