flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "lizhenmxcz@163.com" <lizhenm...@163.com>
Subject flume hive sink not work
Date Thu, 22 Oct 2015 02:27:12 GMT

hi all:
I use flume to import data from syslog to hive,but encount the follow errors. 

2015-10-22 10:05:05,115 (SinkRunner-PollingRunner-DefaultSinkProcessor) [WARN - org.apache.flume.sink.hive.HiveSink.drainOneBatch(HiveSink.java:324)]
k2 : Failed connecting to EndPoint {metaStoreUri='thrift://bigdata1:9083', database='dnsdb',
table='dns_request', partitionVals=[] }
org.apache.flume.sink.hive.HiveWriter$ConnectException: Failed connecting to EndPoint {metaStoreUri='thrift://bigdata1:9083',
database='dnsdb', table='dns_request', partitionVals=[] }
        at org.apache.flume.sink.hive.HiveWriter.<init>(HiveWriter.java:99)
        at org.apache.flume.sink.hive.HiveSink.getOrCreateWriter(HiveSink.java:344)
        at org.apache.flume.sink.hive.HiveSink.drainOneBatch(HiveSink.java:296)
        at org.apache.flume.sink.hive.HiveSink.process(HiveSink.java:254)
        at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
        at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.flume.sink.hive.HiveWriter$ConnectException: Failed connecting to EndPoint
{metaStoreUri='thrift://bigdata1:9083', database='dnsdb', table='dns_request', partitionVals=[]
}
        at org.apache.flume.sink.hive.HiveWriter.newConnection(HiveWriter.java:380)
        at org.apache.flume.sink.hive.HiveWriter.<init>(HiveWriter.java:86)
        ... 6 more
Caused by: java.util.concurrent.TimeoutException
        at java.util.concurrent.FutureTask.get(FutureTask.java:201)
        at org.apache.flume.sink.hive.HiveWriter.timedCall(HiveWriter.java:431)
        at org.apache.flume.sink.hive.HiveWriter.newConnection(HiveWriter.java:373)
        ... 7 more


my configuration is:


a1.sources = r1
a1.channels = c1 c2
a1.sinks = k1 k2

a1.sources.r1.type = syslogudp
a1.sources.r1.port = 514
a1.sources.r1.host = 192.168.55.246

a1.sources.r1.channels = c1 c2
a1.sources.r1.interceptors = i1
a1.sources.r1.interceptors.i1.type = regex_extractor
a1.sources.r1.interceptors.i1.regex = Dns(.*)\\[
a1.sources.r1.interceptors.i1.serializers = t1
a1.sources.r1.interceptors.i1.serializers.t1.name = type

a1.sources.r1.selector.type = multiplexing
a1.sources.r1.selector.header = type
a1.sources.r1.selector.mapping.Request = c1
a1.sources.r1.selector.mapping.Answer = c2

a1.sinks.k2.type = hive
a1.sinks.k2.channel = c1
a1.sinks.k2.hive.metastore = thrift://bigdata1:9083
a1.sinks.k2.hive.database = dnsdb
a1.sinks.k2.hive.table = dns_request
a1.sinks.k2.hive.partiton = %Y,%m,%d,%H
a1.sinks.k2.hive.txnsPerBatchAsk = 2
a1.sinks.k2.batchSize = 10
a1.sinks.k2.serializer = delimited
a1.sinks.k2.serializer.delimiter = ,
a1.sinks.k2.serializer.fieldnames = timepoint,random,sip,dip,spt,type,name

a1.sinks.k1.type = hive
a1.sinks.k1.channel = c2
a1.sinks.k1.hive.metastore = thrift://bigdata1:9083
a1.sinks.k1.hive.database = Dnsdb
a1.sinks.k1.hive.table = dns_answer
a1.sinks.k1.hive.partiton = %Y,%m,%d,%H
a1.sinks.k1.hive.txnsPerBatchAsk = 2
a1.sinks.k1.batchSize = 10
a1.sinks.k1.serializer = delimited
a1.sinks.k1.serializer.delimiter = ,
a1.sinks.k1.serializer.fieldnames = timepoint,random,sip,dip,dpt,name,nosuchname,typemax,typecname,typeaddr,authservername,additionalrecords

help me please,thanks.



lizhenmxcz@163.com
Mime
View raw message