kafka日常管理和参数配置

定义内存中累积的消息数量阈值，达到该值时会强制将日志刷写到磁盘‌

log.flush.interval.messages=10000

设置内存中日志保留的时间阈值（毫秒）

log.flush.interval.ms=1000

控制消息最大保留时长（默认7天）

log.retention.hours=168

限定单个分区日志总大小（示例值为80GB）

log.retention.bytes=85899345920

定义单个日志段文件大小上限（默认1GB）

log.segment.bytes=1073741824

设置日志清理任务执行周期（默认5分钟）

log.retention.check.interval.ms=300000

消息最大保留时长（默认7天）

log.retention.hours=168

日志清理任务执行周期（默认5分钟）

log.retention.check.interval.ms=300000

✅ 一、retention.ms

作用范围：主题级别（topic-level）或日志段文件（log segment）

含义：

控制 Kafka 保留消息的时间。一条消息写入 Kafka 后，只要没有超过这个时间，即使消费者没有消费，消息也不会被删除。

retention.ms（普通删除策略）

Kafka 中 retention.ms 控制对启用 delete 策略主题的消息保留时长。消息会在达到该时间后被删除：

默认值： 604800000（7 天，单位是毫秒）

应用场景：所有正常的消息保留策略

相关配置：

log.retention.hours（老版本用这个，等价于 retention.ms）

log.retention.bytes（可以限制日志文件大小而非时间）

例子：

retention.ms=86400000 # 1 天后自动清理消息

✅ 二、delete.retention.ms

作用范围：压缩（compacted）主题

含义：

控制 Kafka 对于启用 log compaction 的 topic，在标记某条消息为“可删除”之后，实际删除操作延迟的时间。

delete.retention.ms 专门用于开启了 compact 策略的主题，用于控制被标记为删除（即 tombstone）的 key 的保留时长：

默认值： 86400000（1 天）

应用场景：针对设置了 cleanup.policy=compact 的 topic

注意：并不是普通消息的保留时间，而是：

一条消息的 key 被写入新版本（或 tombstone，即 value=null）之后，Kafka 会等 delete.retention.ms 这么久，才会真正物理删除老的或 null 的记录。

————————————————

例如：

cleanup.policy=compact

delete.retention.ms=3600000 # 压缩清理后，1小时后才会真正删除

参数应用范围用途典型用途默认值

retention.ms 所有 topic 消息保留时间控制普通消息多久被清理 7 天

delete.retention.ms compact topic 被标记删除的 key 的实际删除延迟控制 tombstone 或旧 key 被真正删除前的等待时间 1 天

在线调整参数

kafka-topics.sh --describe -bootstrap-server localhost:9092 --topic dma-noa-auto-lane-change-gps-calc

Topic: dma-noa-auto-lane-change-gps-calc TopicId: GeV-i6MhSHm_fOvqVia2_A PartitionCount: 6 ReplicationFactor: 2 Configs: flush.ms=1000,segment.bytes=1073741824,flush.messages=10000,max.message.bytes=10485760,retention.bytes=85899345920

Topic: dma-noa-auto-lane-change-gps-calc Partition: 0 Leader: 2 Replicas: 2,1 Isr: 2,1

Topic: dma-noa-auto-lane-change-gps-calc Partition: 1 Leader: 1 Replicas: 1,0 Isr: 0,1

Topic: dma-noa-auto-lane-change-gps-calc Partition: 2 Leader: 0 Replicas: 0,2 Isr: 0,2

Topic: dma-noa-auto-lane-change-gps-calc Partition: 3 Leader: 2 Replicas: 2,0 Isr: 0,2

Topic: dma-noa-auto-lane-change-gps-calc Partition: 4 Leader: 1 Replicas: 1,2 Isr: 2,1

Topic: dma-noa-auto-lane-change-gps-calc Partition: 5 Leader: 0 Replicas: 0,1 Isr: 0,1

kafka-configs.sh -bootstrap-server localhost:9092 --entity-type topics --entity-name dma-noa-auto-lane-change-gps-calc --alter --add-config retention.ms=432000000

I have no name!@prod-kafka-0:/$ kafka-configs.sh -bootstrap-server localhost:9092 --entity-type topics --entity-name dma-noa-auto-lane-change-gps-calc --alter --add-config retention.ms=432000000

Completed updating config for topic dma-noa-auto-lane-change-gps-calc.

I have no name!@prod-kafka-0:/$ kafka-topics.sh --describe -bootstrap-server localhost:9092 --topic dma-noa-auto-lane-change-gps-calc

Topic: dma-noa-auto-lane-change-gps-calc TopicId: GeV-i6MhSHm_fOvqVia2_A PartitionCount: 6 ReplicationFactor: 2 Configs: flush.ms=1000,segment.bytes=1073741824,retention.ms=432000000,flush.messages=10000,max.message.bytes=10485760,retention.bytes=85899345920

Topic: dma-noa-auto-lane-change-gps-calc Partition: 0 Leader: 2 Replicas: 2,1 Isr: 2,1

Topic: dma-noa-auto-lane-change-gps-calc Partition: 1 Leader: 1 Replicas: 1,0 Isr: 0,1

Topic: dma-noa-auto-lane-change-gps-calc Partition: 2 Leader: 0 Replicas: 0,2 Isr: 0,2

Topic: dma-noa-auto-lane-change-gps-calc Partition: 3 Leader: 2 Replicas: 2,0 Isr: 0,2

Topic: dma-noa-auto-lane-change-gps-calc Partition: 4 Leader: 1 Replicas: 1,2 Isr: 2,1

Topic: dma-noa-auto-lane-change-gps-calc Partition: 5 Leader: 0 Replicas: 0,1 Isr: 0,1

[2025-09-22 11:27:58,823] INFO Processing override for entityPath: topics/dma-noa-auto-lane-change-gps-calc with config: Map(retention.ms -> 432000000) (kafka.server.ZkConfigManager)

查看消息

kafka-topics.sh --bootstrap-server localhost:9092 --list

用户认证消息查看：

kafka-topics.sh --bootstrap-server localhost:9092 --command-config /tmp/client.properties --list

security.protocol=SASL_PLAINTEXT

sasl.mechanism=PLAIN

sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="brokerUser" password="55c45d7759-lm5xw";

kafka-ui

- bootstrapServers: kafka.kafka-test:9092

name: kafka-test

properties:

"security.protocol": SASL_PLAINTEXT

"sasl.mechanism": PLAIN

"sasl.jaas.config": org.apache.kafka.common.security.plain.PlainLoginModule required username="brokerUser" password="*****";

brokerUser 55c45d7759-lm5xw

kafka-topics.sh --bootstrap-server kafka.kafka-test:9092 --list

kafka.kafka-test:9092

kafka-consumer-groups.sh --describe --bootstrap-server localhost:9092 --all-topics --group dma10v-group-active-event-monitor-10v

GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID

dma10v-group-active-event-monitor-10v active-event-monitor-10v 3 1272385 1272385 0 consumer-dma10v-group-active-event-monitor-10v-10-19e3da9b-0423-4fc9-a0ba-21cb624af829 /172.20.14.86 consumer-dma10v-group-active-event-monitor-10v-10

dma10v-group-active-event-monitor-10v active-event-monitor-10v 7 1282938 1282938 0 consumer-dma10v-group-active-event-monitor-10v-10-361fe72f-82e1-4edb-92a4-08ce735aa43c /172.20.13.87 consumer-dma10v-group-active-event-monitor-10v-10

dma10v-group-active-event-monitor-10v active-event-monitor-10v 2 1647122 1647122 0 consumer-dma10v-group-active-event-monitor-10v-10-0d2d5115-f109-4a3b-8134-7c92254d6983 /172.20.14.89 consumer-dma10v-group-active-event-monitor-10v-10

dma10v-group-active-event-monitor-10v active-event-monitor-10v 0 919169 919169 0 consumer-dma10v-group-active-event-monitor-10v-10-095c6e40-51fe-4811-84d2-e10ccb8dc7d7 /172.20.13.205 consumer-dma10v-group-active-event-monitor-10v-10

dma10v-group-active-event-monitor-10v active-event-monitor-10v 5 1179426 1179426 0 consumer-dma10v-group-active-event-monitor-10v-10-1d0a1570-839e-4d3c-bde0-b5ff5a6eba97 /172.20.14.84 consumer-dma10v-group-active-event-monitor-10v-10

dma10v-group-active-event-monitor-10v active-event-monitor-10v 1 1189697 1189697 0 consumer-dma10v-group-active-event-monitor-10v-10-0bdd219c-e922-4a67-adc3-e78d9bbef2ff /172.20.13.209 consumer-dma10v-group-active-event-monitor-10v-10

dma10v-group-active-event-monitor-10v active-event-monitor-10v 9 1010422 1010422 0 consumer-dma10v-group-active-event-monitor-10v-10-50e0cb89-06ce-4f17-81d5-db8430a7e675 /172.20.13.207 consumer-dma10v-group-active-event-monitor-10v-10

dma10v-group-active-event-monitor-10v active-event-monitor-10v 8 1293150 1293150 0 consumer-dma10v-group-active-event-monitor-10v-10-4daaaa3a-5459-44d1-acde-bdc209a9f29b /172.20.14.22 consumer-dma10v-group-active-event-monitor-10v-10

dma10v-group-active-event-monitor-10v active-event-monitor-10v 10 1324679 1324679 0 consumer-dma10v-group-active-event-monitor-10v-10-5302a09d-586b-455a-a361-b84ce3ed8c7c /172.20.13.208 consumer-dma10v-group-active-event-monitor-10v-10

dma10v-group-active-event-monitor-10v active-event-monitor-10v 4 1028147 1028147 0 consumer-dma10v-group-active-event-monitor-10v-10-1b7a0d96-2095-4bf9-911f-3b575e6e7938 /172.20.14.85 consumer-dma10v-group-active-event-monitor-10v-10

dma10v-group-active-event-monitor-10v active-event-monitor-10v 11 966066 966066 0 consumer-dma10v-group-active-event-monitor-10v-10-7542012f-9c38-47e0-8826-9915bb91e27f /172.20.14.18 consumer-dma10v-group-active-event-monitor-10v-10

dma10v-group-active-event-monitor-10v active-event-monitor-10v 6 1120030 1120030 0 consumer-dma10v-group-active-event-monitor-10v-10-334d461a-8ba1-4d92-a5ed-cc2683d370a5 /172.20.13.211 consumer-dma10v-group-active-event-monitor-10v-10

kafka-consumer-groups.sh --describe --bootstrap-server localhost:9092 --topic active-event-monitor-10v --group dma10v-group-active-event-monitor-10v

active-event-monitor-10v

kafka-topics.sh --describe -bootstrap-server localhost:9092 --topic active-event-monitor-10v

Topic: active-event-monitor-10v TopicId: JMe0rPCvTPqGZfIu4dYt8g PartitionCount: 12 ReplicationFactor: 2 Configs: flush.ms=1000,segment.bytes=1073741824,flush.messages=10000,max.message.bytes=10485760,retention.bytes=85899345920

Topic: active-event-monitor-10v Partition: 0 Leader: 2 Replicas: 2,1 Isr: 1,2

Topic: active-event-monitor-10v Partition: 1 Leader: 1 Replicas: 1,0 Isr: 1,0

Topic: active-event-monitor-10v Partition: 2 Leader: 0 Replicas: 0,2 Isr: 2,0

Topic: active-event-monitor-10v Partition: 3 Leader: 2 Replicas: 2,0 Isr: 2,0

Topic: active-event-monitor-10v Partition: 4 Leader: 1 Replicas: 1,2 Isr: 1,2

Topic: active-event-monitor-10v Partition: 5 Leader: 0 Replicas: 0,1 Isr: 1,0

Topic: active-event-monitor-10v Partition: 6 Leader: 2 Replicas: 2,1 Isr: 2,1

Topic: active-event-monitor-10v Partition: 7 Leader: 1 Replicas: 1,0 Isr: 1,0

Topic: active-event-monitor-10v Partition: 8 Leader: 0 Replicas: 0,2 Isr: 2,0

Topic: active-event-monitor-10v Partition: 9 Leader: 2 Replicas: 2,0 Isr: 2,0

Topic: active-event-monitor-10v Partition: 10 Leader: 1 Replicas: 1,2 Isr: 1,2

Topic: active-event-monitor-10v Partition: 11 Leader: 0 Replicas: 0,1 Isr: 1,0

I have no name!@prod-kafka-0:/bitnami/kafka/data$ du -sh active-event-monitor-10v*

76M active-event-monitor-10v-1

74M active-event-monitor-10v-11

78M active-event-monitor-10v-2

76M active-event-monitor-10v-3

72M active-event-monitor-10v-5

76M active-event-monitor-10v-7

76M active-event-monitor-10v-8

78M active-event-monitor-10v-9

I have no name!@prod-kafka-0:/bitnami/kafka/data$ du -shc active-event-monitor-10v*

76M active-event-monitor-10v-1

74M active-event-monitor-10v-11

79M active-event-monitor-10v-2

76M active-event-monitor-10v-3

73M active-event-monitor-10v-5

76M active-event-monitor-10v-7

76M active-event-monitor-10v-8

78M active-event-monitor-10v-9

603M total

kafka-configs.sh -bootstrap-server localhost:9092 --entity-type topics --entity-name active-event-monitor-10v --alter --add-config retention.ms=432000000

Topic: active-event-monitor-10v TopicId: JMe0rPCvTPqGZfIu4dYt8g PartitionCount: 12 ReplicationFactor: 2 Configs: flush.ms=1000,segment.bytes=1073741824,retention.ms=432000000,flush.messages=10000,max.message.bytes=10485760,retention.bytes=85899345920

kafka-configs.sh -bootstrap-server localhost:9092 --entity-type topics --entity-name active-event-monitor-10v --alter --add-config delete.retention.ms=60000

[2025-09-23 06:57:43,295] INFO Processing notification(s) to /config/changes (kafka.common.ZkNodeChangeNotificationListener)

[2025-09-23 06:57:43,296] INFO Processing override for entityPath: topics/active-event-monitor-10v with config: Map(retention.ms -> 432000000, delete.retention.ms -> 60000) (kafka.server.ZkConfigManager)

[2025-09-23 06:57:43,309] INFO Processing notification(s) to /config/changes (kafka.common.ZkNodeChangeNotificationListener

kafka-configs.sh -bootstrap-server localhost:9092 --entity-type topics --entity-name file-blocks --alter --add-config retention.ms=432000000

kafka-configs.sh -bootstrap-server localhost:9092 --entity-type topics --entity-name dar-data-analysis-source-10v --alter --add-config retention.ms=432000000

kafka-configs.sh -bootstrap-server localhost:9092 --entity-type topics --entity-name dr-file-info2 --alter --add-config retention.ms=432000000

kafka-configs.sh -bootstrap-server localhost:9092 --entity-type topics --entity-name dr-append-rename-file --alter --add-config retention.ms=345600000

kafka-configs.sh -bootstrap-server localhost:9092 --entity-type topics --entity-name ar-file-result-sync --alter --add-config retention.ms=345600000

kafka-configs.sh -bootstrap-server localhost:9092 --entity-type topics --entity-name dr-append-file-info --alter --add-config retention.ms=345600000

kafka-configs.sh -bootstrap-server localhost:9092 --entity-type topics --entity-name dr-append-file-info --describe

Dynamic configs for topic dr-append-file-info are:

retention.ms=432000000 sensitive=false synonyms={DYNAMIC_TOPIC_CONFIG:retention.ms=432000000}

max.message.bytes=10485760 sensitive=false synonyms={DYNAMIC_TOPIC_CONFIG:max.message.bytes=10485760, STATIC_BROKER_CONFIG:message.max.bytes=10485760, DEFAULT_CONFIG:message.max.bytes=1048588}

dar-data-analysis-source-10v

file-blocks

认证的消息查询

kafka-topics.sh --bootstrap-server localhost:9092 --command-config /tmp/client.properties --list

dr-append-file-info

dr-append-rename-file

dr-file-info2

file-blocks

(责任编辑：liangzh)

搜索

热门标签:

kafka日常管理和参数配置