kafka迁移与扩容工具使用
参考官网site:https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-6.ReassignPartitionsTool
说明:
当我们对kafka集群扩容时,需要满足2点要求:
- 将指定topic迁移到集群内新增的node上。
- 将topic的指定partition迁移到新增的node上。
1. 迁移topic到新增的node上
--topics-to-move-json-file migration-push-token-topic.json --broker-list "104,105,106" --generate
脚本migration-push-token-topic.json文件内容如下:
{
"topics":
[
{
"topic": "push-token-topic"
}
],
"version":1
}
生成分配partitions的json脚本:
Current partition replica assignment
{"version":1,"partitions":[{"topic":"cluster-switch-topic","partition":10,"replicas":[8]},{"topic":"cluster-switch-topic","partition":5,"replicas":[4]},{"topic":"cluster-switch-topic","partition":3,"replicas":[5]},{"topic":"cluster-switch-topic","partition":4,"replicas":[5]},{"topic":"cluster-switch-topic","partition":9,"replicas":[5]},{"topic":"cluster-switch-topic","partition":1,"replicas":[5]},{"topic":"cluster-switch-topic","partition":11,"replicas":[4]},{"topic":"cluster-switch-topic","partition":7,"replicas":[5]},{"topic":"cluster-switch-topic","partition":2,"replicas":[4]},{"topic":"cluster-switch-topic","partition":0,"replicas":[4]},{"topic":"cluster-switch-topic","partition":6,"replicas":[4]},{"topic":"cluster-switch-topic","partition":8,"replicas":[4]}]}
重新分配parttions的json脚本如下:
migration-topic-cluster-switch-topic.json
{"version":1,"partitions":[{"topic":"cluster-switch-topic","partition":10,"replicas":[5]},{"topic":"cluster-switch-topic","partition":5,"replicas":[4]},{"topic":"cluster-switch-topic","partition":4,"replicas":[5]},{"topic":"cluster-switch-topic","partition":3,"replicas":[4]},{"topic":"cluster-switch-topic","partition":9,"replicas":[4]},{"topic":"cluster-switch-topic","partition":1,"replicas":[4]},{"topic":"cluster-switch-topic","partition":11,"replicas":[4]},{"topic":"cluster-switch-topic","partition":7,"replicas":[4]},{"topic":"cluster-switch-topic","partition":2,"replicas":[5]},{"topic":"cluster-switch-topic","partition":0,"replicas":[5]},{"topic":"cluster-switch-topic","partition":6,"replicas":[5]},{"topic":"cluster-switch-topic","partition":8,"replicas":[5]}]}
lizhitao@localhost:$ bin/kafka-reassign-partitions.sh --zookeeper 192.168.2.225:2183/config/mobile/mq/mafka01 --reassignment-json-file migration-topic-cluster-switch-topic.json --execute
2.topic修改(replicats-factor)副本个数
lizhitao@localhost:$ ./bin/kafka-reassign-partitions.sh --zookeeper 192.168.2.225:2183/config/mobile/mq/mafka
--reassignment-json-file replicas-update-push-token-topic.json --execute
假如初始时push-token-topic为一个副本,为了提高可用性,需要改为2副本模式。
脚本replicas-push-token-topic.json文件内容如下:
{
"partitions":
[
{
"topic": "log.mobile_nginx",
"partition": 0,
"replicas": [101,102,104]
},
{
"topic": "log.mobile_nginx",
"partition": 1,
"replicas": [102,103,106]
},
{
"topic": "xxxx",
"partition": 数字,
"replicas": [数组]
}
],
"version":1
}
3.topic的分区扩容用法
a.先扩容分区数量,脚本如下:
例如:push-token-topic初始分区数量为12,目前到增加到15个
lizhitao@localhost:$ ./bin/kafka-topics.sh --zookeeper 192.168.2.225:2183/config/mobile/mq/mafka --alter --partitions 15 --topic push-token-topic
b.设置topic分区副本
lizhitao@localhost:$ ./bin/kafka-reassign-partitions.sh --zookeeper 192.168.2.225:2183/config/mobile/mq/mafka
--reassignment-json-file partitions-extension-push-token-topic.json --execute
脚本partitions-extension-push-token-topic.json文件内容如下:
- {
- "partitions":
- [
- {
- "topic": "push-token-topic",
- "partition": 12,
- "replicas": [101,102]
- },
- {
- "topic": "push-token-topic",
- "partition": 13,
- "replicas": [103,104]
- },
- {
- "topic": "push-token-topic",
- "partition": 14,
- "replicas": [105,106]
- }
- ],
- "version":1
- }
相关推荐
Apache Kafka实战.pdf..
This book is here to help you get familiar with Apache Kafka and to solve your challenges related to the consumption of millions of messages in publisher-subscriber architectures. It is aimed at ...
Apache Kafka Apache Kafka Apache Kafka Apache Kafka
Spring for Apache Kafka API。 Spring for Apache Kafka 开发文档。
Apache Kafka源码剖析 PDF较大,分5份上传!一起解压即可。
Apache Kafka is a popular distributed streaming platform that acts as a messaging queue or an enterprise messaging system. It lets you publish and subscribe to a stream of records and process them in ...
Apache Kafka 官方文档中文版,Apache Kafka 官方文档中文版
Building Data Streaming Applications with Apache Kafka 英文azw3 本资源转载自网络,如有侵权,请联系上传者或csdn删除 本资源转载自网络,如有侵权,请联系上传者或csdn删除
Learning Apache Kafka Second Edition provides you with step-by-step, practical examples that help you take advantage of the real power of Kafka and handle hundreds of megabytes of messages per second ...
apache kafka技术内幕 和 apacke kafka源码分析2本PDF 电子书 网盘下载
Kafka是一个对于像Hadoop的一样的日志数据和离线分析系统,但又要求实时处理的限制,这是一个可行的解决方案。 1.Kafka集群包含一个或多个服务器,这种服务器被称为broker 2.Partition是物理上的概念,每个Topic...
Streaming Architecture New Designs Using Apache Kafka and MapR Streams
。Apache Kafka Cookbook(PACKT,2015)
with no previous knowledge of Apache Kafka. As the book progresses, the difficulty level increases. The second half of this cookbook is about configuration; this is advanced material for those who ...
Apache Kafka(带书签)
Kafka 是由 LinkedIn 开发的一个分布式的消息系统, 使用 Scala 编写, 它以可水平扩展和高吞吐率而被广泛使用。 目前越来越多的开源分布式处理系统如Cloudera、 Apache Storm、Spark 都支持与 Kafka 集成。
Apache Kafka实战epu.rar
kafka官方文档有关解释|
Learning Apache Kafka 英文版 第二版