flume+kafka+hdfs详解 |
您所在的位置：网站首页 › flume采集kafka数据写入hdfs › flume+kafka+hdfs详解 |
flume+kafka+hdfs详解
|
flume+kafka+hdfs详解
原创
jethai 2015-10-03 19:26:55 博主文章分类：Hadoop/CDH ©著作权 文章标签 flume+kafka+hdfs 文章分类 Hive 大数据 ©著作权归作者所有：来自51CTO博客作者jethai的原创作品，请联系作者获取转载授权，否则将追究法律责任flume架构图
单节点flume配置 flume-1.4.0 启动flume bin/flume-ng agent --conf ./conf -f conf/flume-conf.properties -Dflume.root.logger=DEBUG,console -n agent-n表示配置文件中agent的名字 agent.sources = r1 agent.sinks = s1 agent.channels = c1 agent.sources.r1.channels = c1 agent.sinks.s1.channel = c1 #Describe/configure the sources agent.sources.r1.type = exec agent.sources.r1.command = tail -F /home/flume/loginfo #Use a channel which buffers events in memory agent.channels.c1.type = memory agent.channels.c1.capacity = 1000 #Event agent.channels.c1.transactionCapacity = 100 agent.sinks.s1.type = logger
flume-1.4.0 + kafka-0.7.2+hdfs flume配置 agent.sources = r1 agent.sinks = s_kafka s_hdfs agent.channels = c_kafka c_hdfs agent.sources.r1.channels = c_kafka c_hdfs agent.sources.r1.type = exec #下面这个脚本tail某个日志 agent.sources.r1.command = tail -F /home/flume/loginfo agent.channels.c_kafka.type = memory agent.channels.c_hdfs.type = memory agent.sinks.s_kafka.type = com.sink.FirstkafkaSink agent.sinks.s_kafka.channel = c_kafka #kafka需要连接zk，写入broker数据 agent.sinks.s_kafka.zkconnect = localhost:2181 agent.sinks.s_kafka.topic = test agent.sinks.s_kafka.serializer.class = kafka.serializer.StringEncoder agent.sinks.s_kafka.metadata.broker.list = localhost:9092 #配置文件server.properties agent.sinks.s_kafka.custom.encoding = UTF-8 agent.sinks.s_hdfs.type = hdfs agent.sinks.s_hdfs.channel = c_hdfs #默认端口8020 agent.sinks.s_hdfs.hdfs.path = hdfs://localhost:9000/root/source agent.sinks.s_hdfs.hdfs.filePrefix = events- agent.sinks.s_hdfs.hdfs.fileType = DataStream agent.sinks.s_hdfs.hdfs.writeFormat = Text agent.sinks.s_hdfs.hdfs.rollCount = 30 #达到某一数值记录生成文件 agent.sinks.s_hdfs.hdfs.rollSize = 0 agent.sinks.s_hdfs.hdfs.rollInterval = 0 agent.sinks.s_hdfs.hdfs.useLocalTimeStamp = true agent.sinks.s_hdfs.hdfs.idleTimeout = 51 agent.sinks.s_hdfs.hdfs.threadsPoolSize = 2Flume内置channel,source,sink汇总http://www.iteblog.com/archives/948 打赏 赞 收藏 评论 分享 举报上一篇：多线程同步基础 下一篇：java读取Excel文件 |
【本文地址】
今日新闻 |
推荐新闻 |