Memo Kafka

technotes

Table des matières

Configure multiple listener

advertised.listeners:           PLAINTEXT_WAN://external.wan:9094,PLAINTEXT_LAN://192.168.2.91:9092
listeners=PLAINTEXT_WAN://:9094,PLAINTEXT_LAN://:9092
listener.security.protocol.map: PLAINTEXT_LAN:PLAINTEXT,PLAINTEXT_WAN:PLAINTEXT
inter.broker.listener.name=PLAINTEXT_LAN

List topics

kafka-topics.sh --list --zookeeper localhost:2181

Create a topic

kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 4 --topic test_topic

Delete a topic

kafka-topics.sh --zookeeper localhost:2181 --delete --topic test_topic

Get detailled information

kafka-topics.sh --zookeeper localhost:2181  --describe --topic syslog
Topic:syslog  PartitionCount:4        ReplicationFactor:1     Configs:retention.ms=3600000,retention.bytes=1000000000
        Topic: syslog Partition: 0    Leader: 0       Replicas: 0     Isr: 0
        Topic: syslog Partition: 1    Leader: 0       Replicas: 0     Isr: 0
        Topic: syslog Partition: 2    Leader: 0       Replicas: 0     Isr: 0
        Topic: syslog Partition: 3    Leader: 0       Replicas: 0     Isr: 0

Define retention policy of a topic

Time policy

Example for 1 hour :

kafka-configs.sh --zookeeper localhost:2181 --alter --entity-type topics --add-config retention.ms=3600000 --entity-name syslog

Size policy

Example for 1 Gigabyte :

kafka-configs.sh --zookeeper localhost:2181 --alter --entity-type topics --add-config retention.bytes=1000000000 --entity-name syslog

Push a file content of messages to Kafka

(One message per line)

kafka-console-producer.sh --broker-list localhost:9092 --topic test_topic < file.log

Reading messages from a topic

From begining (only 10 msg):

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic syslog --from-beginning --property print.key=true --max-messages 10

Last messages (only 10 msg):

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic syslog --property print.key=true --max-messages 10

Kafkacat version

From begining (only 10 msg):

kafkacat -q -C -b 127.0.0.1 -t syslog -o beginning -K '|' -c 10

Last messages (only 10 msg):

kafkacat -q -b 127.0.0.1 -t syslog -o end -K '|' -c 10

Delete a consumer group

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --delete --group connect-syslog-test

Mirrormaker topic1 to topic2

/usr/bin/kafkacat -b localhost:9092 -K: -G consumer_grp -t topicSrc | /usr/bin/kafkacat -b localhost:9092 -t topicDst -K: -P > /home/kafka/kafka/kafka-mirror.log 2>&1'
kafka memo snippet"

Related