markdown 卡夫卡有用的命令

Posted 2021-05-06

# Kafka

### get size for all topics on worker
```sh
 for topic in $(find   /opt/cloudera/data/*/kafka/ -maxdepth 1 -mindepth 1 -type d -exec basename {} \;| awk -F '-' '{print $1}'| sort |uniq | grep -v consumer_offsets);do

du -csh /opt/cloudera/data/*/kafka/${topic}*| grep total| awk '{print "topic '$topic' "$1}'
done
```
 
### get config for topic
```sh
 kafka-configs --describe --zookeeper hdp-m-01:2181,hdp-m-02:2181,hdp-m-03:2181/kafka  --entity-type topics --entity-name  adrequests
```
### get config adjustments for all topics
```sh
 kafka-configs --describe --zookeeper hdp-m-01:2181,hdp-m-02:2181,hdp-m-03:2181/kafka  --entity-type topics
```

### create topic
```sh
kafka-topics --create --zookeeper hdp-m-01:2181,hdp-m-02:2181,hdp-m-03:2181/kafka --replication-factor 3 --partitions 50 evnt_v1_rtb
```
### delete topic
```sh
kafka-topics --delete --zookeeper hdp-m-01:2181,hdp-m-02:2181,hdp-m-03:2181/kafka --topic events_adreqs
```

### change topic retention
```sh
# 1 day = 86400000, 7 day = 604800000
kafka-topics --zookeeper hdp-m-01:2181,hdp-m-02:2181,hdp-m-03:2181/kafka --alter --topic evnt_v1_areq --config retention.ms=21600000
```

### describe kafka topic
```sh
kafka-topics --describe --zookeeper hdp-m-01:2181,hdp-m-02:2181,hdp-m-03:2181/kafka evnt_v1_opps
```

### list kafka topics
```sh
kafka-topics --list --zookeeper hdp-m-01:2181/kafka
```

### describe all kafka topics
```sh
kafka-topics --describe --zookeeper hdp-m-01:2181,hdp-m-02:2181,hdp-m-03:2181/kafka
```

### consuming massages from the command line
```sh
kafka-console-consumer evnt_v1_imprs --zookeeper hdp-m-01:2181/kafka
```

### find events with empty network_id for example
```sh
timeout 30 kafka-console-consumer evnt_v1_opps --zookeeper hdp-m-01:2181/kafka 2>&1 | grep hostname  | jq 'select(.network_id == null)'
```

### rebalance cluster

https://www.cloudera.com/documentation/kafka/latest/topics/kafka_performance.html#concept_exp_hzk_br_unique_1


### more info
https://archive.cloudera.com/kafka/kafka/2/kafka-0.9.0-kafka2.0.1/ops.html

### retention policy
http://www.allprogrammingtutorials.com/tutorials/configuring-messages-retention-time-in-kafka.php