Flume 集群搭建 ,配置了2个sink,负载均衡
三台服务器,分别是
192.168.134.131 master
192.168.134.132 datanodea
192.168.134.133 datanodeb
集群的模式是这样的:
Master
|
=====================
|| ||
DataNodeA DataNodeB
这是master
#agent agent.channels = channel agent.sources = source agent.sinks = node1 node2 agent.sinkgroups = g1 agent.sinkgroups.g1.sinks = node1 node2 agent.sinkgroups.g1.processor.type = load_balance agent.sinkgroups.g1.processor.backoff = true agent.sinkgroups.g1.processor.selector = round_robin agent.sinkgroups.g1.processor.selector.maxTimeOut=10000 #channel agent.channels.channel.type = memory agent.channels.channel.capacity = 1000000 agent.channels.channel.transactionCapacity = 1000000 agent.channels.channel.keep-alive = 10 #source agent.sources.source.channels = channel agent.sources.source.type = avro agent.sources.source.bind = master agent.sources.source.port = 41414 agent.sources.source.threads = 5 #sink #node1 agent.sinks.node1.channel = channel agent.sinks.node1.type = avro agent.sinks.node1.hostname = datanodea agent.sinks.node1.port = 41414 #node2 agent.sinks.node2.channel = channel agent.sinks.node2.type = avro agent.sinks.node2.hostname = datanodeb agent.sinks.node2.port = 41414
这是datanodea
#agent agent.channels = ch1 ch2 agent.sources = source agent.sinks = elasticsearch file agent.source.source.selector.type = replicating #channel agent.channels.ch1.type = memory agent.channels.ch1.capacity = 1000000 agent.channels.ch1.transactionCapacity = 1000000 agent.channels.ch1.keep-alive = 10 agent.channels.ch2.type = memory agent.channels.ch2.capacity = 1000000 agent.channels.ch2.transactionCapacity = 1000000 agent.channels.ch2.keep-alive = 10 #source agent.sources.source.channels = ch1 ch2 agent.sources.source.type = avro agent.sources.source.bind = datanodea agent.sources.source.port = 41414 agent.sources.source.threads = 5 #sink agent.sinks.file.channel = ch1 agent.sinks.file.type = file_roll agent.sinks.file.sink.directory = /opt/flume/data agent.sinks.file.sink.serializer = TEXT agent.sinks.elasticsearch.channel = ch2 agent.sinks.elasticsearch.type = elasticsearch agent.sinks.elasticsearch.hostNames = master:9300 agent.sinks.elasticsearch.indexName = flume_index agent.sinks.elasticsearch.indexType = flume_type agent.sinks.elasticsearch.clusterName = elasticsearch agent.sinks.elasticsearch.batchSize = 1 agent.sinks.elasticsearch.serializer = org.apache.flume.sink.elasticsearch.ElasticSearchDynamicSerializer
这是datanodeb
#agent agent.channels = ch1 ch2 agent.sources = source agent.sinks = elasticsearch file agent.source.source.selector.type = replicating #channel agent.channels.ch1.type = memory agent.channels.ch1.capacity = 1000000 agent.channels.ch1.transactionCapacity = 1000000 agent.channels.ch1.keep-alive = 10 agent.channels.ch2.type = memory agent.channels.ch2.capacity = 1000000 agent.channels.ch2.transactionCapacity = 1000000 agent.channels.ch2.keep-alive = 10 #source agent.sources.source.channels = ch1 ch2 agent.sources.source.type = avro agent.sources.source.bind = datanodeb agent.sources.source.port = 41414 agent.sources.source.threads = 5 #sink agent.sinks.file.channel = ch1 agent.sinks.file.type = file_roll agent.sinks.file.sink.directory = /opt/flume/data agent.sinks.file.sink.serializer = TEXT agent.sinks.elasticsearch.channel = ch2 agent.sinks.elasticsearch.type = elasticsearch agent.sinks.elasticsearch.hostNames = master:9300 agent.sinks.elasticsearch.indexName = flume_index agent.sinks.elasticsearch.indexType = flume_type agent.sinks.elasticsearch.clusterName = elasticsearch agent.sinks.elasticsearch.batchSize = 1 agent.sinks.elasticsearch.serializer = org.apache.flume.sink.elasticsearch.ElasticSearchDynamicSerializer
相关推荐
flume1.5.2集群搭建与使用文档 最新版本
flume集群环境搭建,详细讲解,图文并茂,包括flume信息监控和众多文章链接
hadoop12下载flume安装包wget http://mirrors.tuna.tsinghua.edu.cn/apache/flume/1.7.0/ap
1、内容概要:Hadoop+Spark+Hive+HBase+Oozie+Kafka+Flume+Flink+Elasticsearch+Redash等大数据集群及组件搭建指南(详细搭建步骤+实践过程问题总结)。 2、适合人群:大数据运维、大数据相关技术及组件初学者。 3、...
Flume是一个分布式、可靠和高可用性的数据采集工具,用于将大量数据从各种源采集到Hadoop生态系统中进行处理。在大型互联网企业的数据处理任务中,Flume被广泛应用。 Flume逻辑上分三层架构:agent,collector,...
Hadoop&Hbase&Kafka&ZooKeeper&Mysql8.0&Spark&Flume集群搭建文档内涵软件包链接 Anaconda
自己整理的Hadoop环境的一些安装,和一些简单的使用,其中包括Hadoop、hbase、hive、mysql、zookeeper、Kafka、flume。都是一些简单的安装步骤和使用,只在自己的虚拟机(Linux centOS7)上使用测试过。按照步骤一步...
Hadoop 分布式集群搭建 Hadoop由Apache基金会开发的分布式系统基础架构,是利用集群对大量数据进行分布式处理和存储的软件框架。用户可以轻松地在Hadoop集群上开发和运行处理海量数据的应用程序。Hadoop有高可靠,...
CentOS集群搭建、Hadoop集群搭建 配置免密 连接外网,Hive安装 Zookeeper搭建 Kafka scala flume安装 Spark搭建及启动
cdh5.5.4 集群搭建 【自动化脚本+hadoop-ha,yarn-ha,zk,hbase,hive,flume,kafka,spark】全套高可用环境搭建,还有自动化启动脚本。只需要复制粘贴命令,就可以完成。3台机器。相关资源可以留言发邮件,我发资料。cdh...
文档包括hadoop完全分布式、高可用集群搭建教程以及filnk,flume,hive,mysql,spark集群spark高可集群,sqoop的搭建教程以及需要的各种环境安装包以及要使用的相关软件。
包括了:Hadoop2.7.3、Spark2.0、Zookeeper3.4.8、Hive、Hbase、Kafka2.10、Flume的环境搭建
大数据综合实验环境搭建(3个集群、Zookeeper、Hive、HBase) 本资源摘要信息主要对大数据综合实验环境搭建进行了详细的介绍,涵盖了 JDK 的安装和配置、免密码登录的设置、Zookeeper 的安装、Hive 和 HBase 的安装...
Hadoop由Apache基金会开发的分布式系统基础架构,是利用集群对大量数据进行分布式处理和存储的软件框架。...此外,Hadoop还包括了Hive,Hbase,ZooKeeper,Pig,Avro,Sqoop,Flume,Mahout等项目。
按照文档操作可安装7个节点的大数据集群,包括hadoop,hive,hbase,spark,tez,flume,kafka等等,不技术自动化运维及监控
apache-flume-1.8.0-bin.tar.gz apache-phoenix-4.13.0-HBase-1.3-bin.tar.gz hadoop-2.7.4.tar.gz hbase-1.3.1-bin.tar.gz jdk-8u144-linux-x64.tar.gz kafka_2.12-1.0.0.tgz scala-2.12.4.tar.gz scala-2.12.4.tgz...
搭建Hadoop集群,并使用flume+kafka+storm+hbase实现日志抓取分析,使用一个主节点master、两个slave节点
分布式消息队列Kafka的基本架构和集群搭建过程,以及使用Java语言实现客户端API的详细过程。 第三部分(第8~9章):详细讲解了企业大数据处理的两个实际应用案例,分别是基于Druid构建多维数据分析平台和基于JMX...
大数据组件 详细安装步骤(linux配置 hadoop集群搭建 hive flume kafka spark zk 搭建安装)