问题:
1. Kafka的哪些信息记录在Zookeeper中
2. Consumer Group消费的每个Partition的Offset信息存放在什么位置3. Topic的每个Partition存放在哪个Broker上的信息存放在哪里
4. Producer跟Zookeeper究竟有没有关系?没有关系!!!
//consumers、config、brokers、controller_epoch都是Kafka相关的znode [zk: localhost:2181(CONNECTED) 0] ls / [abc, admin, zookeeper, consumers, config, controller, storm, brokers, controller_epoch] [zk: localhost:2181(CONNECTED) 1] ls ///consumers目录下是所有的consumer group的定义 [zk: localhost:2181(CONNECTED) 2] ls /consumers [test-consumer-group, topic-p6-r2-consumer-group, learn.topic.consumers.group, learn.topic.p8.r2.consumers.group] [zk: localhost:2181(CONNECTED) 3] ls learn.topic.p8.r2.consumers.group Command failed: java.lang.IllegalArgumentException: Path must start with / character ///consumer group下游offsets、owners和ids的定义 [zk: localhost:2181(CONNECTED) 4] ls /consumers/learn.topic.p8.r2.consumers.group [offsets, owners, ids] ///offsets下面是topic的定义 [zk: localhost:2181(CONNECTED) 5] ls /consumers/learn.topic.p8.r2.consumers.group/offsets [learn.topic.p8.r2] ///topic下面是partition的定义,learn.topic.p8.r2表示8个partition,2个副本 ///partition中的offset的设置在znode的data中,而不是子znode中 [zk: localhost:2181(CONNECTED) 6] ls /consumers/learn.topic.p8.r2.consumers.group/offsets/learn.topic.p8.r2 [3, 2, 1, 0, 7, 6, 5, 4] [zk: localhost:2181(CONNECTED) 7] ls /consumers/learn.topic.p8.r2.consumers.group/offsets/learn.topic.p8.r2/2 [] [zk: localhost:2181(CONNECTED) 8] ls /consumers/learn.topic.p8.r2.consumers.group/offsets/learn.topic.p8.r2/3 [] [zk: localhost:2181(CONNECTED) 9] ls /consumers/learn.topic.p8.r2.consumers.group/offsets/learn.topic.p8.r2/4 [] ///consumer group属于哪个topic所有,代码中不需要显式指定 ///consumer group消费哪个topic,kafka自动将topic设置为这个consumer group的owner [zk: localhost:2181(CONNECTED) 19] ls /consumers/learn.topic.p8.r2.consumers.group/owners [learn.topic.p8.r2] [zk: localhost:2181(CONNECTED) 20] ls /consumers/learn.topic.p8.r2.consumers.group/owners/learn.topic.p8.r2 [3, 2, 1, 0, 7, 6, 5, 4] [zk: localhost:2181(CONNECTED) 21] ls /consumers/learn.topic.p8.r2.consumers.group/owners/learn.topic.p8.r2/3 [] ///后面是broker的ID [zk: localhost:2181(CONNECTED) 23] ls /consumers/learn.topic.p8.r2.consumers.group/ids [learn.topic.p8.r2.consumers.group_TP-A383-D-1427853446455-efa1e47b] [zk: localhost:2181(CONNECTED) 24] ls /consumers/learn.topic.p8.r2.consumers.group/ids/learn.topic.p8.r2.consumers.group_TP-A383-D-1427853446455-efa1e47b [] [zk: localhost:2181(CONNECTED) 10] ls /brokers [topics, ids] [zk: localhost:2181(CONNECTED) 11] ls /brokers/topics [topic_p10_r3, topic-p6-r2, test_many_partitions, test, topic_p10_r1, learn.topic.p8.r2, learn.topic, topic-3-replica-1-part] [zk: localhost:2181(CONNECTED) 12] ls /brokers/topics/learn.topic.p8.r2 [partitions] [zk: localhost:2181(CONNECTED) 13] ls /brokers/topics/learn.topic.p8.r2/partitions [3, 2, 1, 0, 7, 6, 5, 4] [zk: localhost:2181(CONNECTED) 14] ls /brokers/topics/learn.topic.p8.r2/partitions/3 [state] [zk: localhost:2181(CONNECTED) 15] ls /brokers/topics/learn.topic.p8.r2/partitions/3/state [] [zk: localhost:2181(CONNECTED) 16] ls /brokers/ids [2, 1, 0] [zk: localhost:2181(CONNECTED) 17] ls /brokers/ids/2 [] [zk: localhost:2181(CONNECTED) 18]
每个Consumer Group的每个Partition的Offset信息是作为值存放在Partition节点上的,而不是作为子节点,如下所示,4704是3号Partition已经消费的Offset数
[zk: localhost:2181(CONNECTED) 28] get /consumers/learn.topic.p8.r2.consumers.group/offsets/learn.topic.p8.r2/3 4704 cZxid = 0x1bf14 ctime = Tue Mar 31 06:25:39 EDT 2015 mZxid = 0x1c658 mtime = Tue Mar 31 22:03:48 EDT 2015 pZxid = 0x1bf14 cversion = 0 dataVersion = 177 aclVersion = 0 ephemeralOwner = 0x0 dataLength = 4 numChildren = 0
Topic的每个Partition存放在哪个Broker上的信息存放在哪里?
[zk: localhost:2181(CONNECTED) 30] ls /brokers [topics, ids] [zk: localhost:2181(CONNECTED) 31] ls /brokers/topics [topic_p10_r3, topic-p6-r2, test_many_partitions, test, topic_p10_r1, learn.topic.p8.r2, learn.topic, topic-3-replica-1-part] [zk: localhost:2181(CONNECTED) 32] ls /brokers/topics/learn.topic.p8.r2 [partitions] [zk: localhost:2181(CONNECTED) 33] ls /brokers/topics/learn.topic.p8.r2/partitions [3, 2, 1, 0, 7, 6, 5, 4] [zk: localhost:2181(CONNECTED) 34] ls /brokers/topics/learn.topic.p8.r2/partitions/3 [state] [zk: localhost:2181(CONNECTED) 35] ls /brokers/topics/learn.topic.p8.r2/partitions/3/state [] /// [zk: localhost:2181(CONNECTED) 36] get /brokers/topics/learn.topic.p8.r2/partitions/3/state {"controller_epoch":42,"leader":0,"version":1,"leader_epoch":2,"isr":[0,1]} cZxid = 0x1bee8 ctime = Tue Mar 31 05:34:16 EDT 2015 mZxid = 0x1c247 mtime = Tue Mar 31 21:48:01 EDT 2015 pZxid = 0x1bee8 cversion = 0 dataVersion = 5 aclVersion = 0 ephemeralOwner = 0x0 dataLength = 75 numChildren = 0
每个Broker的信息:包括host和port
//三个broker [zk: localhost:2181(CONNECTED) 38] ls /brokers/ids [2, 1, 0] [zk: localhost:2181(CONNECTED) 39] ls /brokers/ids/2 [] ///host和port信息 [zk: localhost:2181(CONNECTED) 40] get /brokers/ids/2 {"jmx_port":-1,"timestamp":"1427852973061","host":"192.168.26.140","version":1,"port":9094} cZxid = 0x1c257 ctime = Tue Mar 31 21:49:33 EDT 2015 mZxid = 0x1c257 mtime = Tue Mar 31 21:49:33 EDT 2015 pZxid = 0x1c257 cversion = 0 dataVersion = 0 aclVersion = 0 ephemeralOwner = 0x14c72a6b3610003 dataLength = 91 numChildren = 0
每个Topic的Partion存放在哪个或者哪些Broker上,信息存放在
///每个partition的leaderpartition和follower partition信息 [zk: localhost:2181(CONNECTED) 53] get /brokers/topics/learn.topic.p8.r2 {"version":1,"partitions":{"4":[2,1],"5":[0,2],"6":[1,2],"1":[2,0],"0":[1,2],"2":[0,1],"7":[2,0],"3":[1,0]}}
相关推荐
kafka集群zookeeper集群整合配置 本人亲自验证
Kafka、Zookeeper集群配置Demo
ELK+Filebeat+Kafka+ZooKeeper构建日志分析平台,架构图解
kafka+zookeeper.zip
kafka(端口9091-9093)-zookeeper(端口2181-2183)集群配置
Kafka、Zookeeper集群配置Demo
zookeeper+kafka.zip windows安装kafka和zookeeperzookeeper+kafka.zip windows安装kafka和zookeeper
zookeeper配置、集群部署 kafka配置、集群部署 Window平台下
kafka配置文件zookeeper参数.md
ZooKeeper是一个开放源码的分布式应用程序协调服务,它包含一个简单的原语集,分布式应用程序可以基于它实现同步服务,配置维护和命名服务等。 Kafka目前主要作为一个分布式的发布订阅式的消息系统使用 本资源包括...
kafka zookeeper 执行文件 kafka zookeeper 执行文件 kafka zookeeper 执行文件 kafka zookeeper 执行文件 kafka zookeeper 执行文件 kafka zookeeper 执行文件 kafka zookeeper 执行文件
storm,kafka,zookeeper jar包 storm,kafka,zookeeper jar包
kafka zookeeper 安装包
centos7下kafka和zookeeper和Redis集群搭建
Linux下Kafka自带zookeeper安装步骤单点/集群整合
小白学习Linux下的Kafka+Zookeeper使用以及配置的文档和压缩包
kafka+zookeeper高可用集群搭建shell使用脚本,直接在linux运行就行
kafka+zookeeper集群安装详细教程,有需要的可以下载,并实际操作一下。
kubernetes 部署单机版Kafka、zookeeper
kafka和zookeeper安装包_linux