flume-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "hailin (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (FLUME-2996) Flume process shutdown automaticly
Date Sat, 24 Sep 2016 10:15:20 GMT

    [ https://issues.apache.org/jira/browse/FLUME-2996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15518810#comment-15518810
] 

hailin edited comment on FLUME-2996 at 9/24/16 10:14 AM:
---------------------------------------------------------

hi bellow is the configure file content:

a1.sources = r1
a1.sinks = sink_etl_product sink_kinesis_stream
a1.channels = channel_etl_product channel_kinesis_stream

#source setting
#a1.sources.r1.type = http
a1.sources.r1.type = flume.source.HTTPSource
a1.sources.r1.port = 6558
a1.sources.r1.dataSize = 1
a1.sources.r1.selector.type = multiplexing
a1.sources.r1.selector.header = event_type
a1.sources.r1.selector.mapping.etl_product = channel_etl_product channel_kinesis_stream

#handler setting
a1.sources.r1.handler = flume.source.BlobExtHandler
a1.sources.r1.handler.gzip = true
a1.sources.r1.handler.maxBlobLength = 100000

a1.channels.channel_etl_product.type = file
a1.channels.channel_etl_product.checkpointDir = ./checkpoint/product
a1.channels.channel_etl_product.dataDirs = ./data/product
a1.channels.channel_etl_product.checkpointInterval = 10

#Kinesis Firehose
a1.sinks.sink_etl_mix.type = flume.sink.KinesisFirehoseSink
a1.sinks.sink_etl_mix.channel = channel_etl_product
a1.sinks.sink_etl_mix.streamName = product_flumeFirehoseStream
a1.sinks.sink_etl_mix.endpoint = {firehose_endpoint}


flume.source.HTTPSource,flume.source.BlobExtHandler,flume.sink.KinesisFirehoseSink, these
3 Java class contains our bussiness logic. 
what I want is that the flume will transfer data into AWS Kinesis Firehose.
The application works fine on AWS EC2 linux instance. What I meet is flume shutdown automaticly
after several hours , the process disappear when run "ps -ef | grep flume".

I check the flume log, no errors found, seems it shutdown normally:
24 Sep 2016 05:53:59,391 INFO  [agent-shutdown-hook] (org.apache.flume.lifecycle.LifecycleSupervisor.stop:79)
 - Stopping lifecycle supervisor 10
24 Sep 2016 05:53:59,394 INFO  [agent-shutdown-hook] (org.apache.flume.channel.file.FileChannel.stop:321)
 - Stopping FileChannel channel_etl_product { dataDirs: [./data/product] }...
24 Sep 2016 05:53:59,395 INFO  [agent-shutdown-hook] (org.apache.flume.channel.file.EventQueueBackingStoreFile.beginCheckpoint:230)
 - Start checkpoint for ./checkpoint/product/checkpoint, elements to sync = 0
24 Sep 2016 05:53:59,401 INFO  [agent-shutdown-hook] (org.apache.flume.channel.file.EventQueueBackingStoreFile.checkpoint:255)
 - Updating checkpoint metadata: logWriteOrderID: 1474685940834, queueSize: 0, queueHead:
0
24 Sep 2016 05:53:59,407 INFO  [agent-shutdown-hook] (org.apache.flume.channel.file.Log.writeCheckpoint:1034)
 - Updated checkpoint for file: ./data/product/log-3 position: 6674 logWriteOrderID: 1474685940834
24 Sep 2016 05:53:59,407 INFO  [agent-shutdown-hook] (org.apache.flume.channel.file.Log.shutdownWorker:857)
 - Attempting to shutdown background worker.
24 Sep 2016 05:53:59,409 INFO  [agent-shutdown-hook] (org.apache.flume.channel.file.LogFile$Writer.close:370)
 - Closing ./data/product/log-3
24 Sep 2016 05:53:59,409 INFO  [agent-shutdown-hook] (org.apache.flume.channel.file.LogFile$RandomReader.close:504)
 - Closing RandomReader ./data/product/log-2
24 Sep 2016 05:53:59,414 INFO  [agent-shutdown-hook] (org.apache.flume.channel.file.LogFile$RandomReader.close:504)
 - Closing RandomReader ./data/product/log-3
24 Sep 2016 05:53:59,420 INFO  [agent-shutdown-hook] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:150)
 - Component type: CHANNEL, name: channel_etl_product stopped

please let me know if need more information.


thanks


was (Author: hehailin):
hi bellow is the configure file content:

a1.sources = r1
a1.sinks = sink_etl_product sink_kinesis_stream
a1.channels = channel_etl_product channel_kinesis_stream

#source setting
#a1.sources.r1.type = http
a1.sources.r1.type = flume.source.HTTPSource
a1.sources.r1.port = 6558
a1.sources.r1.dataSize = 1
a1.sources.r1.selector.type = multiplexing
a1.sources.r1.selector.header = event_type
a1.sources.r1.selector.mapping.etl_product = channel_etl_product channel_kinesis_stream

#handler setting
a1.sources.r1.handler = flume.source.BlobExtHandler
a1.sources.r1.handler.gzip = true
a1.sources.r1.handler.maxBlobLength = 100000

a1.channels.channel_etl_product.type = file
a1.channels.channel_etl_product.checkpointDir = ./checkpoint/product
a1.channels.channel_etl_product.dataDirs = ./data/product
a1.channels.channel_etl_product.checkpointInterval = 10

#Kinesis Firehose
a1.sinks.sink_etl_mix.type = flume.sink.KinesisFirehoseSink
a1.sinks.sink_etl_mix.channel = channel_etl_product
a1.sinks.sink_etl_mix.streamName = product_flumeFirehoseStream
a1.sinks.sink_etl_mix.endpoint = {firehose_endpoint}


flume.source.HTTPSource,flume.source.BlobExtHandler,flume.sink.KinesisFirehoseSink, these
3 Java class contains our bussiness logic. 
what I want is that the flume will transfer data into AWS Kinesis Firehose.
The application works fine on AWS EC2 linux instance. What I meet is flume shutdown automaticly
after several hours , the process disappear when run "ps -ef | grep flume".

please let me know if need more information.


thanks

> Flume process shutdown automaticly
> ----------------------------------
>
>                 Key: FLUME-2996
>                 URL: https://issues.apache.org/jira/browse/FLUME-2996
>             Project: Flume
>          Issue Type: Bug
>         Environment: Amazon Linux
>            Reporter: hailin
>
> hi guys,
> I install flume from apache-flume-1.6.0-bin.tar.gz, and startup flume with some my custom
configure file with cmd line like this: 
> bin/flume-ng agent -Dflume.monitoring.type=http -Dflume.monitoring.port=41414 -c conf
-f conf/flume.conf --name a1 -Dflume.root.logger=INFO,DAILY 
> it works fine without any issue, but after several hours when I check the process, I
found it was shutdown automaticly. Bellow is the flume log information:
> .....
> 24 Sep 2016 05:53:59,391 INFO  [agent-shutdown-hook] (org.apache.flume.lifecycle.LifecycleSupervisor.stop:79)
 - Stopping lifecycle supervisor 10
> 24 Sep 2016 05:53:59,394 INFO  [agent-shutdown-hook] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:150)
 - Component type: SINK, name: sink_kinesis_stream stopped
> 24 Sep 2016 05:53:59,394 INFO  [agent-shutdown-hook] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:156)
 - Shutdown Metric for type: SINK, name: sink_kinesis_stream. sink.start.time == 1474685921906
> ....
> Can anyone help me fix it? How can I do to stop this issue as our bussiness already block
because of this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message