Memo Kafka

Table des matières
Configure multiple listener
advertised.listeners: PLAINTEXT_WAN://external.wan:9094,PLAINTEXT_LAN://192.168.2.91:9092
listeners=PLAINTEXT_WAN://:9094,PLAINTEXT_LAN://:9092
listener.security.protocol.map: PLAINTEXT_LAN:PLAINTEXT,PLAINTEXT_WAN:PLAINTEXT
inter.broker.listener.name=PLAINTEXT_LAN
List topics
kafka-topics.sh --list --zookeeper localhost:2181
Create a topic
kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 4 --topic test_topic
Delete a topic
kafka-topics.sh --zookeeper localhost:2181 --delete --topic test_topic
Get detailled information
kafka-topics.sh --zookeeper localhost:2181 --describe --topic syslog
Topic:syslog PartitionCount:4 ReplicationFactor:1 Configs:retention.ms=3600000,retention.bytes=1000000000
Topic: syslog Partition: 0 Leader: 0 Replicas: 0 Isr: 0
Topic: syslog Partition: 1 Leader: 0 Replicas: 0 Isr: 0
Topic: syslog Partition: 2 Leader: 0 Replicas: 0 Isr: 0
Topic: syslog Partition: 3 Leader: 0 Replicas: 0 Isr: 0
Define retention policy of a topic
Time policy
Example for 1 hour :
kafka-configs.sh --zookeeper localhost:2181 --alter --entity-type topics --add-config retention.ms=3600000 --entity-name syslog
Size policy
Example for 1 Gigabyte :
kafka-configs.sh --zookeeper localhost:2181 --alter --entity-type topics --add-config retention.bytes=1000000000 --entity-name syslog
Push a file content of messages to Kafka
(One message per line)
kafka-console-producer.sh --broker-list localhost:9092 --topic test_topic < file.log
Reading messages from a topic
From begining (only 10 msg):
kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic syslog --from-beginning --property print.key=true --max-messages 10
Last messages (only 10 msg):
kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic syslog --property print.key=true --max-messages 10
Kafkacat version
From begining (only 10 msg):
kafkacat -q -C -b 127.0.0.1 -t syslog -o beginning -K '|' -c 10
Last messages (only 10 msg):
kafkacat -q -b 127.0.0.1 -t syslog -o end -K '|' -c 10
Delete a consumer group
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --delete --group connect-syslog-test
Mirrormaker topic1 to topic2
/usr/bin/kafkacat -b localhost:9092 -K: -G consumer_grp -t topicSrc | /usr/bin/kafkacat -b localhost:9092 -t topicDst -K: -P > /home/kafka/kafka/kafka-mirror.log 2>&1'