flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "lizhenmxcz@163.com" <lizhenm...@163.com>
Subject Re: Re: flume hive sink not work
Date Thu, 22 Oct 2015 06:23:18 GMT

telnet say "nane or service not known",but the port is listening 
tcp        0      0 0.0.0.0:9083                0.0.0.0:*                   LISTEN      32119/java


lizhenmxcz@163.com
 
From: iain wright
Date: 2015-10-22 14:13
To: user
Subject: Re: flume hive sink not work
Are you able to telnet to bigdata1 on port 9083?
On Oct 21, 2015 7:27 PM, "lizhenmxcz@163.com" <lizhenmxcz@163.com> wrote:

hi all:
I use flume to import data from syslog to hive,but encount the follow errors. 

2015-10-22 10:05:05,115 (SinkRunner-PollingRunner-DefaultSinkProcessor) [WARN - org.apache.flume.sink.hive.HiveSink.drainOneBatch(HiveSink.java:324)]
k2 : Failed connecting to EndPoint {metaStoreUri='thrift://bigdata1:9083', database='dnsdb',
table='dns_request', partitionVals=[] }
org.apache.flume.sink.hive.HiveWriter$ConnectException: Failed connecting to EndPoint {metaStoreUri='thrift://bigdata1:9083',
database='dnsdb', table='dns_request', partitionVals=[] }
        at org.apache.flume.sink.hive.HiveWriter.<init>(HiveWriter.java:99)
        at org.apache.flume.sink.hive.HiveSink.getOrCreateWriter(HiveSink.java:344)
        at org.apache.flume.sink.hive.HiveSink.drainOneBatch(HiveSink.java:296)
        at org.apache.flume.sink.hive.HiveSink.process(HiveSink.java:254)
        at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
        at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.flume.sink.hive.HiveWriter$ConnectException: Failed connecting to EndPoint
{metaStoreUri='thrift://bigdata1:9083', database='dnsdb', table='dns_request', partitionVals=[]
}
        at org.apache.flume.sink.hive.HiveWriter.newConnection(HiveWriter.java:380)
        at org.apache.flume.sink.hive.HiveWriter.<init>(HiveWriter.java:86)
        ... 6 more
Caused by: java.util.concurrent.TimeoutException
        at java.util.concurrent.FutureTask.get(FutureTask.java:201)
        at org.apache.flume.sink.hive.HiveWriter.timedCall(HiveWriter.java:431)
        at org.apache.flume.sink.hive.HiveWriter.newConnection(HiveWriter.java:373)
        ... 7 more


my configuration is:


a1.sources = r1
a1.channels = c1 c2
a1.sinks = k1 k2

a1.sources.r1.type = syslogudp
a1.sources.r1.port = 514
a1.sources.r1.host = 192.168.55.246

a1.sources.r1.channels = c1 c2
a1.sources.r1.interceptors = i1
a1.sources.r1.interceptors.i1.type = regex_extractor
a1.sources.r1.interceptors.i1.regex = Dns(.*)\\[
a1.sources.r1.interceptors.i1.serializers = t1
a1.sources.r1.interceptors.i1.serializers.t1.name = type

a1.sources.r1.selector.type = multiplexing
a1.sources.r1.selector.header = type
a1.sources.r1.selector.mapping.Request = c1
a1.sources.r1.selector.mapping.Answer = c2

a1.sinks.k2.type = hive
a1.sinks.k2.channel = c1
a1.sinks.k2.hive.metastore = thrift://bigdata1:9083
a1.sinks.k2.hive.database = dnsdb
a1.sinks.k2.hive.table = dns_request
a1.sinks.k2.hive.partiton = %Y,%m,%d,%H
a1.sinks.k2.hive.txnsPerBatchAsk = 2
a1.sinks.k2.batchSize = 10
a1.sinks.k2.serializer = delimited
a1.sinks.k2.serializer.delimiter = ,
a1.sinks.k2.serializer.fieldnames = timepoint,random,sip,dip,spt,type,name

a1.sinks.k1.type = hive
a1.sinks.k1.channel = c2
a1.sinks.k1.hive.metastore = thrift://bigdata1:9083
a1.sinks.k1.hive.database = Dnsdb
a1.sinks.k1.hive.table = dns_answer
a1.sinks.k1.hive.partiton = %Y,%m,%d,%H
a1.sinks.k1.hive.txnsPerBatchAsk = 2
a1.sinks.k1.batchSize = 10
a1.sinks.k1.serializer = delimited
a1.sinks.k1.serializer.delimiter = ,
a1.sinks.k1.serializer.fieldnames = timepoint,random,sip,dip,dpt,name,nosuchname,typemax,typecname,typeaddr,authservername,additionalrecords

help me please,thanks.



lizhenmxcz@163.com
Mime
View raw message