flume-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "67287685 (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (FLUME-3233) log roll to a new name,the flume collect repeat
Date Thu, 03 May 2018 06:17:00 GMT

     [ https://issues.apache.org/jira/browse/FLUME-3233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

67287685 updated FLUME-3233:
----------------------------
    Description: 
we use the flume 1.7 to collect our log send to kafka. 

our java application  compressed the log at every day 00:00,the log name change aaa.log ->
aaa.20180503.tmp -> aaa.20180503.log.gz,the change while complete less than 1 second time.

this have a problem ,when have some data into log at 23:59:59,the log while compress fast,the
flume not collect the new data,the log is become zip file,so some data will lost.

To solve this problem,we Modify the java application and flume

1.modify java application

the log is compress delay 5 minutes,aaa.log --> aaa.20180503.tmp -> after 5 min ->
aaa.20180503.log.gz

the flume have enough time complete collect the data .

 

2.modify flume 1.7

use inode judge the file uniqueness,not use inode + filename ,when use inode + filename,the
flume while repeate collect when the aaa.log change to aaa.20180503.tmp.this two file have
the same inode,we will not repeat collect the file.

 

Please help to see if our plan is feasible,thanks every .

 

  was:
we use the flume 1.7 to collect our log send to kafka. 

our java application  compressed the log at every day 00:00,the log name change aaa.log ->
aaa.20180503.tmp -> aaa.20180503.log.gz,the change while complete less than 1 second time.

this have a problem ,when have some data into log at 23:59:59,the log while compress fast,the
flume while not collect the new data,the log is become zip file,the data is lost.

To solve this problem,we Modify the java application and flume

1.modify java application

the log is compress delay 5 minutes,aaa.log --> aaa.20180503.tmp -> after 5 min ->
aaa.20180503.log.gz,the flume while complete collect the data ,not lost.

 

2.modify flume 1.7

use inode judge the file uniqueness,not use inode + filename ,when use inode + filename,the
flume while repeate collect when the aaa.log --> aaa.20180503.tmp,this two file have the
same inode

 

Please help to see if our plan is feasible,thanks every .

 


> log roll to a new name,the flume collect repeat
> -----------------------------------------------
>
>                 Key: FLUME-3233
>                 URL: https://issues.apache.org/jira/browse/FLUME-3233
>             Project: Flume
>          Issue Type: Bug
>    Affects Versions: 1.7.0
>            Reporter: 67287685
>            Priority: Major
>
> we use the flume 1.7 to collect our log send to kafka. 
> our java application  compressed the log at every day 00:00,the log name change aaa.log ->
aaa.20180503.tmp -> aaa.20180503.log.gz,the change while complete less than 1 second time.
> this have a problem ,when have some data into log at 23:59:59,the log while compress
fast,the flume not collect the new data,the log is become zip file,so some data will lost.
> To solve this problem,we Modify the java application and flume
> 1.modify java application
> the log is compress delay 5 minutes,aaa.log --> aaa.20180503.tmp -> after 5 min
-> aaa.20180503.log.gz
> the flume have enough time complete collect the data .
>  
> 2.modify flume 1.7
> use inode judge the file uniqueness,not use inode + filename ,when use inode + filename,the
flume while repeate collect when the aaa.log change to aaa.20180503.tmp.this two file have
the same inode,we will not repeat collect the file.
>  
> Please help to see if our plan is feasible,thanks every .
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@flume.apache.org
For additional commands, e-mail: issues-help@flume.apache.org


Mime
View raw message