{"id":5216,"date":"2022-10-05T11:22:05","date_gmt":"2022-10-05T03:22:05","guid":{"rendered":"http:\/\/www.menglanglang.cn\/?p=5216"},"modified":"2022-10-05T11:22:07","modified_gmt":"2022-10-05T03:22:07","slug":"%e5%a4%a7%e6%95%b0%e6%8d%ae%e5%ad%a6%e4%b9%a0%e5%88%9d%e7%ba%a7%e5%85%a5%e9%97%a8%e6%95%99%e7%a8%8b%ef%bc%88%e5%8d%81%e5%9b%9b%ef%bc%89-flume-1-9-x-%e5%8d%95%e8%8a%82%e7%82%b9","status":"publish","type":"post","link":"http:\/\/www.menglanglang.cn\/?p=5216","title":{"rendered":"\u5927\u6570\u636e\u5b66\u4e60\u521d\u7ea7\u5165\u95e8\u6559\u7a0b\uff08\u5341\u56db\uff09 \u2014\u2014 Flume 1.9.x \u5355\u8282\u70b9\u7684\u5b89\u88c5\u3001\u914d\u7f6e\u3001\u542f\u52a8\u548c\u6d4b\u8bd5"},"content":{"rendered":"\n

\u200b<\/p>\n\n\n\n

\u5b98\u7f51\u4ecb\u7ecd\u5982\u4e0b\uff1a<\/p>\n\n\n\n

Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data. It has a simple and flexible architecture based on streaming data flows. It is robust and fault tolerant with tunable reliability mechanisms and many failover and recovery mechanisms. It uses a simple extensible data model that allows for online analytic application.<\/p>

Flume \u662f\u4e00\u79cd\u5206\u5e03\u5f0f\u7684\u3001\u53ef\u9760\u7684\u4e14\u53ef\u7528\u7684\u670d\u52a1\uff0c\u53ef\u4ee5\u9ad8\u6548\u5730\u6536\u96c6\u3001\u805a\u5408\u548c\u79fb\u52a8\u5927\u91cf\u65e5\u5fd7\u6570\u636e\u3002\u5b83\u5177\u6709\u57fa\u4e8e\u6d41\u6570\u636e\u6d41\u7684\u7b80\u5355\u7075\u6d3b\u7684\u4f53\u7cfb\u7ed3\u6784\u3002\u5b83\u5177\u6709\u5065\u58ee\u6027\u548c\u5bb9\u9519\u6027\uff0c\u5177\u6709\u53ef\u8c03\u7684\u53ef\u9760\u6027\u673a\u5236\u548c\u8bb8\u591a\u6545\u969c\u8f6c\u79fb\u548c\u6062\u590d\u673a\u5236\u3002\u5b83\u4f7f\u7528\u4e00\u4e2a\u7b80\u5355\u7684\u53ef\u6269\u5c55\u6570\u636e\u6a21\u578b\uff0c\u5141\u8bb8\u5728\u7ebf\u5206\u6790\u5e94\u7528\u3002<\/p><\/blockquote>\n\n\n\n

\"\"\/<\/p>\n\n\n\n

\u672c\u7bc7\u4e3b\u8981\u7b80\u5355\u4ecb\u7ecd\u4e0b Flume \u7684\u5b89\u88c5\u3001\u914d\u7f6e\u3001\u542f\u52a8\u548c\u6d4b\u8bd5\u3002<\/p>\n\n\n\n

\u7b2c\u4e00\u6b65\uff1a\u4e0a\u4f20\u5b89\u88c5\u5305<\/h3>\n\n\n\n

\u4e0a\u4f20 flume \u5305\u5230\u6d4b\u8bd5\u670d\u52a1\u5668\uff0c\u8fd9\u91cc\u6d4b\u8bd5\u76f4\u63a5\u4e0a\u4f20\u5230 home \u76ee\u5f55\u4e0b\u3002<\/p>\n\n\n\n

\u7b2c\u4e8c\u6b65\uff1a\u89e3\u538b\u5b89\u88c5\u5305<\/h3>\n\n\n\n

# cd \/home<\/p>

# tar -zxvf apache-flume-1.9.0-bin.tar.gz<\/p><\/blockquote>\n\n\n\n

\u7b2c\u4e09\u6b65\uff1a\u91cd\u547d\u540d\u76ee\u5f55<\/h3>\n\n\n\n

\u7531\u4e8e\u89e3\u538b\u7684 flume \u5b89\u88c5\u5305\u76ee\u5f55\u540d\u8f83\u957f\uff0c\u8fd9\u91cc\u4fee\u6539\u77ed\u4e00\u4e9b\uff08\u5f3a\u8feb\u75c7\uff09<\/p>\n\n\n\n

# mv apache-flume-1.9.0-bin flume-1.9.0<\/p><\/blockquote>\n\n\n\n

\u7b2c\u56db\u6b65\uff1a\u914d\u7f6e\u73af\u5883\u53d8\u91cf<\/h3>\n\n\n\n

\u4fee\u6539\u5f53\u671f\u7528\u6237\u914d\u7f6e\u6587\u4ef6.bash_profile<\/p>\n\n\n\n

# vi ~\/.bash_profile<\/p><\/blockquote>\n\n\n\n

\u5728\u914d\u7f6e\u6700\u540e\u6dfb\u52a0\u5982\u4e0b\u914d\u7f6e\uff1a<\/p>\n\n\n\n

export FLUME_HOME=\/home\/flume-1.9.0
export PATH=$PATH:$FLUME\/bin<\/p><\/blockquote>\n\n\n\n

\u6267\u884c\u547d\u4ee4\u4f7f\u5f97\u914d\u7f6e\u7acb\u523b\u751f\u6548\uff1a<\/p>\n\n\n\n

# source ~\/.bash_profile<\/p><\/blockquote>\n\n\n\n

\u7b2c\u4e94\u6b65\uff0c\u521b\u5efa\u81ea\u5b9a\u4e49\u914d\u7f6e<\/h3>\n\n\n\n

\u5728 Flume \u7684\u5b89\u88c5\u76ee\u5f55\u4e0b\uff0c\u521b\u5efa\u4e2a\u81ea\u5b9a\u4e49\u914d\u7f6e\u6587\u4ef6\u76ee\u5f55 defconf<\/p>\n\n\n\n

# cd \/home\/flume-1.9.0<\/p>

# mkdir defconf<\/p><\/blockquote>\n\n\n\n

\u521b\u5efa\u81ea\u5b9a\u4e49\u914d\u7f6e\u6587\u4ef6 netcat2logger\uff0c\u914d\u7f6e\u5982\u4e0b\uff1a<\/p>\n\n\n\n

# example.conf: A single-node Flume configuration<\/p>

# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1<\/p>

# Describe\/configure the source
a1.sources.r1.type = netcat
#a1.sources.r1.bind = localhost<\/p>

a1.sources.r1.bind = node19
a1.sources.r1.port = 44444<\/p>

# Describe the sink
a1.sinks.k1.type = logger<\/p>

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100<\/p>

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1<\/p><\/blockquote>\n\n\n\n

\u7b2c\u516d\u6b65\uff1a\u542f\u52a8 Flume<\/h3>\n\n\n\n

# flume-ng agent --conf conf --conf-file defconf\/netcat2logger --name a1 -Dflume.root.logger=INFO,console<\/p><\/blockquote>\n\n\n\n

\u53ef\u4ee5\u770b\u5230\u8f93\u51fa\u65e5\u5fd7\u5982\u4e0b\uff1a<\/p>\n\n\n\n

\"\"\/<\/p>\n\n\n\n

\u7b2c\u4e03\u6b65\uff1a\u6d4b\u8bd5<\/h3>\n\n\n\n

\u5728\u5b89\u88c5 flume \u7684\u8282\u70b9\u673a\u5668\u4e0a\uff0c\u4f7f\u7528\u4e0b\u9762\u547d\u4ee4\u53d1\u9001\u4fe1\u606f\uff1a<\/p>\n\n\n\n

# telnet node19 44444<\/p><\/blockquote>\n\n\n\n

\"\"\/<\/p>\n\n\n\n

\u53ef\u4ee5\u770b\u5230 flume \u76d1\u542c\u7684\u7aef\u53e3\u6536\u5230\u4e86\u53d1\u9001\u7684\u4fe1\u606f\uff0c\u5e76\u6253\u5370\u5230\u4e86\u63a7\u5236\u53f0\uff1a<\/p>\n\n\n\n

\"\"\/<\/p>\n\n\n\n

\u7b2c\u516b\u6b65\uff1a\u505c\u6b62 Flume<\/h3>\n\n\n\n

\u6309 Ctrl+C \u6216\u8005 Ctrl+Z \u5373\u53ef\u505c\u6b62\u3002<\/p>\n\n\n\n

\u5e38\u89c1\u9519\u8bef\uff1a<\/h3>\n\n\n\n

\u9519\u8bef\u4e00\uff1a<\/strong>\u5728\u67e5\u770b Flume \u7248\u672c\u65f6\uff0c\u63d0\u793a\u201cUnsupported major.minor version 52.0\u201d\u3002<\/p>\n\n\n\n

[root@node19 flume-1.9.0]# flume-ng version
Exception in thread \"main\" java.lang.UnsupportedClassVersionError: org\/apache\/flume\/tools\/GetJavaProperty : Unsupported major.minor version 52.0
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
        at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:482)
Error: Could not find or load main class org.apache.flume.tools.GetJavaProperty
Flume 1.9.0
Source code repository: https:\/\/git-wip-us.apache.org\/repos\/asf\/flume.git
Revision: d4fcab4f501d41597bc616921329a4339f73585e
Compiled by fszabo on Mon Dec 17 20:45:25 CET 2018
From source with checksum 35db629a3bda49d23e9b3690c80737f9<\/p><\/blockquote>\n\n\n\n

\u89e3\u51b3\u65b9\u6cd5\uff1a<\/strong>\u8bf4\u660e JDK \u7248\u672c\u592a\u4f4e\u5bfc\u81f4\uff0c\u53ef\u4ee5\u628a hbase \u548c hadoop \u4e2d\u7684 export JAVA_HOME \u4fee\u6539\uff0c\u4e00\u822c\u90fd\u9700\u8981\u5347\u9ad8 JDK \u7248\u672c\u3002<\/p>\n\n\n\n

\u9519\u8bef\u4e8c\uff1a<\/strong>Error: Could not find or load main class org.apache.flume.tools.GetJavaProperty<\/p>\n\n\n\n

\u627e\u4e0d\u5230\u6216\u65e0\u6cd5\u52a0\u8f7d\u4e3b\u7c7b org.apache.flume.tools.GetJavaProperty\u3002\u5982\u679c\u7cfb\u7edf\u91cc\u5b89\u88c5\u4e86 hbase\uff0c\u4f1a\u51fa\u73b0\u9519\u8bef\uff1a\u627e\u4e0d\u5230\u6216\u65e0\u6cd5\u52a0\u8f7d\u4e3b\u7c7b org.apache.flume.tools.GetJavaProperty\u3002\u5982\u679c\u6ca1\u6709\u5b89\u88c5hbase\uff0c\u8fd9\u4e00\u6b65\u53ef\u4ee5\u7565\u8fc7\u3002<\/p>\n\n\n\n

\u89e3\u51b3\u65b9\u6cd5\uff1a<\/strong><\/p>\n\n\n\n

\u65b9\u6cd5\u4e00\uff1a\u5c06 hbase \u7684 hbase.env.sh \u7684\u8fd9\u4e00\u884c\u914d\u7f6e\u6ce8\u91ca\u6389\uff0c\u5373\u5728 export \u524d\u52a0\u4e00\u4e2a#\u3002<\/p>\n\n\n\n

#export HBASE_CLASSPATH=\/home\/hadoop\/hbase\/conf<\/p><\/blockquote>\n\n\n\n

\u65b9\u6cd5\u4e8c\uff1a\u5c06 HBASE_CLASSPATH \u6539\u4e3a JAVA_CLASSPATH\uff0c\u914d\u7f6e\u5982\u4e0b\uff1a<\/p>\n\n\n\n

export JAVA_CLASSPATH=.:$JAVA_HOME\/lib\/dt.jar:$JAVA_HOME\/lib\/tools.jar<\/p><\/blockquote>\n\n\n\n

Flume \u73af\u5883\u7b80\u5355\u7684\u64cd\u4f5c\u5199\u5230\u8fd9\u91cc\uff0cGood Luck\uff01 \u200b<\/p>\n\n\n\n

\u539f\u6587\u94fe\u63a5\uff1ahttps:\/\/blog.csdn.net\/tzhuwb\/article\/details\/127170336<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"

\u200b \u5b98\u7f51\u4ecb\u7ecd\u5982\u4e0b\uff1a Flume is a distributed, reliable, and availab […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[880],"tags":[881,714,877,183,214,199],"class_list":["post-5216","post","type-post","status-publish","format-standard","hentry","category-flume","tag-flume","tag-714","tag-877","tag-183","tag-214","tag-199"],"_links":{"self":[{"href":"http:\/\/www.menglanglang.cn\/index.php?rest_route=\/wp\/v2\/posts\/5216","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/www.menglanglang.cn\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.menglanglang.cn\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.menglanglang.cn\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/www.menglanglang.cn\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=5216"}],"version-history":[{"count":1,"href":"http:\/\/www.menglanglang.cn\/index.php?rest_route=\/wp\/v2\/posts\/5216\/revisions"}],"predecessor-version":[{"id":5217,"href":"http:\/\/www.menglanglang.cn\/index.php?rest_route=\/wp\/v2\/posts\/5216\/revisions\/5217"}],"wp:attachment":[{"href":"http:\/\/www.menglanglang.cn\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=5216"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.menglanglang.cn\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=5216"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.menglanglang.cn\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=5216"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}