- 浏览: 2486872 次
- 性别:
- 来自: 成都
文章分类
最新评论
-
nation:
你好,在部署Mesos+Spark的运行环境时,出现一个现象, ...
Spark(4)Deal with Mesos -
sillycat:
AMAZON Relatedhttps://www.godad ...
AMAZON API Gateway(2)Client Side SSL with NGINX -
sillycat:
sudo usermod -aG docker ec2-use ...
Docker and VirtualBox(1)Set up Shared Disk for Virtual Box -
sillycat:
Every Half an Hour30 * * * * /u ...
Build Home NAS(3)Data Redundancy -
sillycat:
3 List the Cron Job I Have>c ...
Build Home NAS(3)Data Redundancy
Kafka Cluster 2019(6) 3 Nodes Cluster on CentOS7
Here is my machines:
192.168.56.110 rancher-home
192.168.56.111 rancher-worker1
192.168.56.112 rancher-worker2
Need JAVA ENV JDK8, 11, 12
> sudo yum install git
> git clone https://github.com/gcuisinier/jenv.git ~/.jenv
> echo 'export PATH="$HOME/.jenv/bin:$PATH"' >> ~/.bash_profile
> echo 'eval "$(jenv init -)"' >> ~/.bash_profile
> . ~/.bash_profile
Check version
> jenv --version
jenv 0.5.2-12-gdcbfd48
Download JDK 8, 11, 12 from Official website
https://www.oracle.com/technetwork/java/javase/downloads/jdk12-downloads-5295953.html
jdk-11.0.4_linux-x64_bin.tar.gz
jdk-12.0.2_linux-x64_bin.tar.gz
jdk-8u221-linux-x64.tar.gz
Unzip all of these files and place in working directory, link to /opt directory
> tar zxvf jdk-11.0.4_linux-x64_bin.tar.gz
> tar zxvf jdk-12.0.2_linux-x64_bin.tar.gz
> tar zxvf jdk-8u221-linux-x64.tar.gz
> mv jdk-11.0.4 ~/tool/
> mv jdk-12.0.2 ~/tool/
> mv jdk1.8.0_221 ~/tool/
> sudo ln -s /home/carl/tool/jdk1.8.0_221 /opt/jdk-1.8.0
> sudo ln -s /home/carl/tool/jdk-11.0.4 /opt/jdk-11.0.4
> sudo ln -s /home/carl/tool/jdk-12.0.2 /opt/jdk-12.0.2
Add to JENV
> jenv add /opt/jdk-1.8.0
> jenv add /opt/jdk-11.0.4
> jenv add /opt/jdk-12.0.2
Check the installed versions
> jenv versions
11.0
11.0.4
12.0
12.0.2
1.8
1.8.0.221
oracle64-11.0.4
oracle64-12.0.2
oracle64-1.8.0.221
Try to set global to 11
> jenv global 11.0
Check the java version
> java -version
java version "11.0.4" 2019-07-16 LTS
Java(TM) SE Runtime Environment 18.9 (build 11.0.4+10-LTS)
Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.4+10-LTS, mixed mode)
Install Latest Zookeeper
Find the latest version from here
https://www.apache.org/dyn/closer.cgi/zookeeper/
I download the latest version
> wget http://mirrors.ocf.berkeley.edu/apache/zookeeper/zookeeper-3.5.5/apache-zookeeper-3.5.5-bin.tar.gz
Unzip and place in the working directory
> tar zxvf apache-zookeeper-3.5.5-bin.tar.gz
> mv apache-zookeeper-3.5.5-bin ~/tool/zookeeper-3.5.5
> sudo ln -s /home/carl/tool/zookeeper-3.5.5 /opt/zookeeper-3.5.5
> sudo ln -s /opt/zookeeper-3.5.5 /opt/zookeeper
Add this to the path
> vi ~/.bash_profile
PATH=$PATH:/opt/zookeeper/bin
ZOO_MY_ID=1
export ZOO_MY_ID
> . ~/.bash_profile
Prepare the configuration for the Zookeeper Cluster
> cd /opt/zookeeper
> mkdir data
> cp conf/zoo_sample.cfg conf/zoo.cfg
> vi conf/zoo.cfg
server.1=rancher-home:2888:3888
server.2=rancher-worker1:2888:3888
server.3=rancher-worker2:2888:3888
dataDir=/opt/zookeeper/data
> vi /opt/zookeeper/data/myid
1
Do the similar to other 2 machines, just change the myid from 1 to 2 or 3.
Start the service on 3 machines
> /opt/zookeeper/bin/zkServer.sh start /opt/zookeeper/conf/zoo.cfg
Check the status on all the nodes
> zkServer.sh status conf/zoo.cfg
/home/carl/.jenv/shims/java
ZooKeeper JMX enabled by default
Using config: conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: follower
> zkServer.sh status conf/zoo.cfg
/home/carl/.jenv/shims/java
ZooKeeper JMX enabled by default
Using config: conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: leader
One Server will be the leader, other 2 servers will be the followers.
Find the latest Kafka
https://kafka.apache.org/downloads
My current latest version is 2.12-2.3.0
> wget https://www-us.apache.org/dist/kafka/2.3.0/kafka_2.12-2.3.0.tgz
Unzip and place it in the right directories
> tar zxvf kafka_2.12-2.3.0.tgz
> mv kafka_2.12-2.3.0 ~/tool/
> sudo ln -s /home/carl/tool/kafka_2.12-2.3.0 /opt/kafka-2.3.0
> sudo ln -s /opt/kafka-2.3.0 /opt/kafka
Configure on 3 machines
> cd /opt/kafka
> vi config/server.properties
On the first Node
broker.id=1
zookeeper.connect=rancher-home:2181,rancher-worker1:2181,rancher-worker2:2181
On the second Node
broker.id=2
zookeeper.connect=rancher-home:2181,rancher-worker1:2181,rancher-worker2:2181
On the Third Node
broker.id=3
zookeeper.connect=rancher-home:2181,rancher-worker1:2181,rancher-worker2:2181
Start the Kafka Service on 3 Nodes
> nohup /opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties &
3 Servers are started successfully.
Verify from Clients
Create One Topic
> bin/kafka-topics.sh --create --zookeeper rancher-home:2181,rancher-worker1:2181,rancher-worker2:2181 --replication-factor 2 --partitions 2 --topic cluster1
Created topic cluster1.
Producer
> bin/kafka-console-producer.sh --broker-list rancher-home:9092,rancher-worker1:9092,rancher-worker2:9092 --topic cluster1
>hello
>sillycat
>yiyikang
>where is the pokemon game?
Consumer
> bin/kafka-console-consumer.sh --bootstrap-server rancher-home:9092,rancher-worker1:9092,rancher-worker2:9092 --topic cluster1 --from-beginning
sillycat
hello
yiyikang
where is the pokemon game?
Here is my machines:
192.168.56.110 rancher-home
192.168.56.111 rancher-worker1
192.168.56.112 rancher-worker2
Need JAVA ENV JDK8, 11, 12
> sudo yum install git
> git clone https://github.com/gcuisinier/jenv.git ~/.jenv
> echo 'export PATH="$HOME/.jenv/bin:$PATH"' >> ~/.bash_profile
> echo 'eval "$(jenv init -)"' >> ~/.bash_profile
> . ~/.bash_profile
Check version
> jenv --version
jenv 0.5.2-12-gdcbfd48
Download JDK 8, 11, 12 from Official website
https://www.oracle.com/technetwork/java/javase/downloads/jdk12-downloads-5295953.html
jdk-11.0.4_linux-x64_bin.tar.gz
jdk-12.0.2_linux-x64_bin.tar.gz
jdk-8u221-linux-x64.tar.gz
Unzip all of these files and place in working directory, link to /opt directory
> tar zxvf jdk-11.0.4_linux-x64_bin.tar.gz
> tar zxvf jdk-12.0.2_linux-x64_bin.tar.gz
> tar zxvf jdk-8u221-linux-x64.tar.gz
> mv jdk-11.0.4 ~/tool/
> mv jdk-12.0.2 ~/tool/
> mv jdk1.8.0_221 ~/tool/
> sudo ln -s /home/carl/tool/jdk1.8.0_221 /opt/jdk-1.8.0
> sudo ln -s /home/carl/tool/jdk-11.0.4 /opt/jdk-11.0.4
> sudo ln -s /home/carl/tool/jdk-12.0.2 /opt/jdk-12.0.2
Add to JENV
> jenv add /opt/jdk-1.8.0
> jenv add /opt/jdk-11.0.4
> jenv add /opt/jdk-12.0.2
Check the installed versions
> jenv versions
11.0
11.0.4
12.0
12.0.2
1.8
1.8.0.221
oracle64-11.0.4
oracle64-12.0.2
oracle64-1.8.0.221
Try to set global to 11
> jenv global 11.0
Check the java version
> java -version
java version "11.0.4" 2019-07-16 LTS
Java(TM) SE Runtime Environment 18.9 (build 11.0.4+10-LTS)
Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.4+10-LTS, mixed mode)
Install Latest Zookeeper
Find the latest version from here
https://www.apache.org/dyn/closer.cgi/zookeeper/
I download the latest version
> wget http://mirrors.ocf.berkeley.edu/apache/zookeeper/zookeeper-3.5.5/apache-zookeeper-3.5.5-bin.tar.gz
Unzip and place in the working directory
> tar zxvf apache-zookeeper-3.5.5-bin.tar.gz
> mv apache-zookeeper-3.5.5-bin ~/tool/zookeeper-3.5.5
> sudo ln -s /home/carl/tool/zookeeper-3.5.5 /opt/zookeeper-3.5.5
> sudo ln -s /opt/zookeeper-3.5.5 /opt/zookeeper
Add this to the path
> vi ~/.bash_profile
PATH=$PATH:/opt/zookeeper/bin
ZOO_MY_ID=1
export ZOO_MY_ID
> . ~/.bash_profile
Prepare the configuration for the Zookeeper Cluster
> cd /opt/zookeeper
> mkdir data
> cp conf/zoo_sample.cfg conf/zoo.cfg
> vi conf/zoo.cfg
server.1=rancher-home:2888:3888
server.2=rancher-worker1:2888:3888
server.3=rancher-worker2:2888:3888
dataDir=/opt/zookeeper/data
> vi /opt/zookeeper/data/myid
1
Do the similar to other 2 machines, just change the myid from 1 to 2 or 3.
Start the service on 3 machines
> /opt/zookeeper/bin/zkServer.sh start /opt/zookeeper/conf/zoo.cfg
Check the status on all the nodes
> zkServer.sh status conf/zoo.cfg
/home/carl/.jenv/shims/java
ZooKeeper JMX enabled by default
Using config: conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: follower
> zkServer.sh status conf/zoo.cfg
/home/carl/.jenv/shims/java
ZooKeeper JMX enabled by default
Using config: conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: leader
One Server will be the leader, other 2 servers will be the followers.
Find the latest Kafka
https://kafka.apache.org/downloads
My current latest version is 2.12-2.3.0
> wget https://www-us.apache.org/dist/kafka/2.3.0/kafka_2.12-2.3.0.tgz
Unzip and place it in the right directories
> tar zxvf kafka_2.12-2.3.0.tgz
> mv kafka_2.12-2.3.0 ~/tool/
> sudo ln -s /home/carl/tool/kafka_2.12-2.3.0 /opt/kafka-2.3.0
> sudo ln -s /opt/kafka-2.3.0 /opt/kafka
Configure on 3 machines
> cd /opt/kafka
> vi config/server.properties
On the first Node
broker.id=1
zookeeper.connect=rancher-home:2181,rancher-worker1:2181,rancher-worker2:2181
On the second Node
broker.id=2
zookeeper.connect=rancher-home:2181,rancher-worker1:2181,rancher-worker2:2181
On the Third Node
broker.id=3
zookeeper.connect=rancher-home:2181,rancher-worker1:2181,rancher-worker2:2181
Start the Kafka Service on 3 Nodes
> nohup /opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties &
3 Servers are started successfully.
Verify from Clients
Create One Topic
> bin/kafka-topics.sh --create --zookeeper rancher-home:2181,rancher-worker1:2181,rancher-worker2:2181 --replication-factor 2 --partitions 2 --topic cluster1
Created topic cluster1.
Producer
> bin/kafka-console-producer.sh --broker-list rancher-home:9092,rancher-worker1:9092,rancher-worker2:9092 --topic cluster1
>hello
>sillycat
>yiyikang
>where is the pokemon game?
Consumer
> bin/kafka-console-consumer.sh --bootstrap-server rancher-home:9092,rancher-worker1:9092,rancher-worker2:9092 --topic cluster1 --from-beginning
sillycat
hello
yiyikang
where is the pokemon game?
发表评论
-
Update Site will come soon
2021-06-02 04:10 1610I am still keep notes my tech n ... -
Stop Update Here
2020-04-28 09:00 261I will stop update here, and mo ... -
NodeJS12 and Zlib
2020-04-01 07:44 430NodeJS12 and Zlib It works as ... -
Docker Swarm 2020(2)Docker Swarm and Portainer
2020-03-31 23:18 310Docker Swarm 2020(2)Docker Swar ... -
Docker Swarm 2020(1)Simply Install and Use Swarm
2020-03-31 07:58 322Docker Swarm 2020(1)Simply Inst ... -
Traefik 2020(1)Introduction and Installation
2020-03-29 13:52 294Traefik 2020(1)Introduction and ... -
Portainer 2020(4)Deploy Nginx and Others
2020-03-20 12:06 380Portainer 2020(4)Deploy Nginx a ... -
Private Registry 2020(1)No auth in registry Nginx AUTH for UI
2020-03-18 00:56 374Private Registry 2020(1)No auth ... -
Docker Compose 2020(1)Installation and Basic
2020-03-15 08:10 327Docker Compose 2020(1)Installat ... -
VPN Server 2020(2)Docker on CentOS in Ubuntu
2020-03-02 08:04 397VPN Server 2020(2)Docker on Cen ... -
Buffer in NodeJS 12 and NodeJS 8
2020-02-25 06:43 335Buffer in NodeJS 12 and NodeJS ... -
NodeJS ENV Similar to JENV and PyENV
2020-02-25 05:14 415NodeJS ENV Similar to JENV and ... -
Prometheus HA 2020(3)AlertManager Cluster
2020-02-24 01:47 360Prometheus HA 2020(3)AlertManag ... -
Serverless with NodeJS and TencentCloud 2020(5)CRON and Settings
2020-02-24 01:46 293Serverless with NodeJS and Tenc ... -
GraphQL 2019(3)Connect to MySQL
2020-02-24 01:48 208GraphQL 2019(3)Connect to MySQL ... -
GraphQL 2019(2)GraphQL and Deploy to Tencent Cloud
2020-02-24 01:48 391GraphQL 2019(2)GraphQL and Depl ... -
GraphQL 2019(1)Apollo Basic
2020-02-19 01:36 275GraphQL 2019(1)Apollo Basic Cl ... -
Serverless with NodeJS and TencentCloud 2020(4)Multiple Handlers and Running wit
2020-02-19 01:19 264Serverless with NodeJS and Tenc ... -
Serverless with NodeJS and TencentCloud 2020(3)Build Tree and Traverse Tree
2020-02-19 01:19 259Serverless with NodeJS and Tenc ... -
Serverless with NodeJS and TencentCloud 2020(2)Trigger SCF in SCF
2020-02-19 01:18 252Serverless with NodeJS and Tenc ...
相关推荐
CentOS7网络配置,安装JDK,安装Kafka,订阅发布以及测试
centos 离线服务器安装kafka
尚硅谷大数据技术之Kafka(2019新版)源码笔记,视频bilibili上有
CentOS7下编译的Kafka Manager 1.3.3.17源码生成的文件,可以直接下载使用,自己编译很耗时
kafka在centos7上搭建过程,包括jdk配置,环境变量配置,zookeepr搭建,以及使用springboot集成kafka的小demo用例
centos 7.4安装kafka 一、 关闭防火墙 二、 禁止selinux 三、 更新系统 四、 安装JDK 五、 配置Java环境变量 六、 安装kafka 七、 安装zookeeper 八、 配置kafka 九、 测试kafka 十、 把zookeeper配置成服务 十一、...
kafka-2.3.0-3.x86_64.rpm for centos7 Default locations binaries: /opt/kafka data: /var/lib/kafka logs: /var/log/kafka configs: /etc/kafka, /etc/sysconfig/kafka
文档中列出了一般的从虚拟机安装CENTOS,到配置基础环境,网络配置等,安装KAFKA环境,多BROKER配置的步骤,每步参考的链接,一些应该会用到的命令,还有我走过的一些坑~希望对大家有帮助
how to integrate Kafka with Camus, Apache Camel, Amazon Cloud, and so on.Chapter 6, Kafka Integrations, provides a short introduction to both Storm and Hadoop and discusses how Kafka integration works...
卡夫卡集群要部署此kafka系统,您需要一个docker swarm集群,在集群的管理器节点上运行以下命令: sudo docker stack deploy --compose-file=docker-compose.yml microservice就是这样!您已经部署了一个kafka集群!
CentOS7 - hyperledger fabric1.1 - 2+6多机部署(kafka共识)配置文件
Centos7安装Kafka.doc
centos8安装kafka_2.11-2.4.1.docx
Centos7kafka2.3.03.zip
kafka-cluster-infra:Terraform和Ansible安装Kafka集群
1、kafka-docker-compose集群安装 2、kafka配置详解
CentOS7下安装Kafka集群.docx
CentOS7安装kafka-manager详细文档
Kafka Streams is a library designed to allow for easy stream processing of data flowing into a Kafka cluster. Stream processing has become one of the biggest needs for companies over the last few ...
本文详细记录了Centos下kafka集群环境部署过程,完全可以做实操手册。在此分享,希望能帮助到有用到的朋友!