Mailing-List: contact user-help@flume.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flume.apache.org
Received-SPF: pass (athena.apache.org: domain of hshreedharan@cloudera.com
 designates 209.85.216.54 as permitted sender)
Date: Fri, 07 Nov 2014 09:59:12 -0800 (PST)
MIME-Version: 1.0
Message-Id: <1415383151698.7d2e7e63@Nodemailer>
In-Reply-To: 
 <CAJmzdXkS9O3rsrb_rhu-_xhTatjhOFNFEdM4_gD+eK28=8+xog@mail.gmail.com>
References: 
 <CAJmzdXkS9O3rsrb_rhu-_xhTatjhOFNFEdM4_gD+eK28=8+xog@mail.gmail.com>
From: "Hari Shreedharan" <hshreedharan@cloudera.com>
To: user@flume.apache.org
Cc: user@flume.apache.org
Subject: Re: File channels creating many large files
Content-Type: multipart/alternative;
 boundary="----Nodemailer-0.5.0-?=_1-1415383152133"

------Nodemailer-0.5.0-?=_1-1415383152133
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

Flume will leave at least 2 files per data directory. Once you have enough =
events to cause 2 files to be created, there will be at least 2 per dir. =
You can use maxFileSize parameter to control the size of these files.


Thanks,
Hari

On Fri, Nov 7, 2014 at 10:25 AM, Jeff Lord <jlord@cloudera.com> wrote:

> Guy,
> What version of flume is this=3F
> -Jeff
> On Fri, Nov 7, 2014 at 1:19 AM, Needham, Guy <Guy.Needham@virginmedia.co.=
uk>
> wrote:
>>  Hi all,
>>
>> I have a configuration with a file channel configured such that:
>>
>> a1.channels.ch1.type =3D file
>> a1.channels.ch1.checkpointDir =3D /hadoop/user/flume/channels/checkpoint=

>> a1.channels.ch1.dataDirs =3D /hadoop/user/flume/channels/data
>> a1.channels.ch1.capacity =3D 100000
>> a1.channels.ch1.transactionCapacity =3D 5000
>>
>> It's been running since October 28th with no issues, but when I looked
>> today in /hadoop/user/flume/channels/data I saw that the file channel =
was
>> building up large files which had been processed and not deleting them:
>>
>> [rdd@hadoop-kn-p2-m01 flume]$ ls -lh channels/data/
>> total 1.6G
>> -rw-r----- 1 rdd rdd 1.5G Oct 28 16:10 log-1
>> -rw-r----- 1 rdd rdd   47 Oct 28 16:10 log-1.meta
>> -rw-r----- 1 rdd rdd  72M Oct 31 16:28 log-2
>> -rw-r----- 1 rdd rdd   47 Oct 31 16:29 log-2.meta
>> It seems like for each day that data landed (we're still in testing so
>> data not landing constantly) a data file has been created but not =
deleted
>> when reading was completed.
>> Is this expected behaviour=3F Is there a way to stop large files =
building up
>> and still use the file channel=3F
>> Regards,
>> Guy Needham | Data Discovery
>> Virgin Media | Enterprise Data, Design & Management
>> Bartley Wood Business Park, Hook, Hampshire RG27 9UP
>> D 01256 75 3362
>> I welcome VSRE emails. Learn more at *http://vsre.info/*
>> <http://vsre.info/>
>>
>>
>>
>>
>> --------------------------------------------------------------------
>> Save Paper - Do you really need to print this e-mail=3F
>>
>> Visit www.virginmedia.com for more information, and more fun.
>>
>> This email and any attachments are or may be confidential and legally
>> privileged
>> and are sent solely for the attention of the addressee(s). If you have
>> received this
>> email in error, please delete it from your system: its use, disclosure =
or
>> copying is
>> unauthorised. Statements and opinions expressed in this email may not
>> represent
>> those of Virgin Media. Any representations or commitments in this email =
are
>> subject to contract.
>>
>> Registered office: Media House, Bartley Wood Business Park, Hook,
>> Hampshire, RG27 9UP
>> Registered in England and Wales with number 2591237
>>
------Nodemailer-0.5.0-?=_1-1415383152133
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: quoted-printable


<span id=3D=22mailbox-conversation=22>Flume will leave at least 2 files per=
 data directory. Once you have enough events to cause 2 files to be created=
, there will be at least 2 per dir. You can use maxFileSize parameter to =
control the size of these files.</span><div class=3D=22mailbox=5Fsignature=
=22>
<br>

Thanks,
Hari</div>
<br><br><div class=3D=22gmail=5Fquote=22><p>On Fri, Nov 7, 2014 at 10:25 AM=
, Jeff Lord <span dir=3D=22ltr=22>&lt;<a href=3D=22mailto:jlord@cloudera.=
com=22 target=3D=22=5Fblank=22>jlord@cloudera.com</a>&gt;</span> =
wrote:<br></p><blockquote class=3D=22gmail=5Fquote=22 style=3D=22margin:0 0=
 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;=22>
<div dir=3D=22ltr=22>Guy,<div><br></div>
<div>What version of flume is this=3F</div>
<div><br></div>
<div>-Jeff</div>
</div>
<div class=3D=22gmail=5Fextra=22>
<br><div class=3D=22gmail=5Fquote=22>On Fri, Nov 7, 2014 at 1:19 AM, =
Needham, Guy <span dir=3D=22ltr=22>&lt;<a href=3D=22mailto:Guy.=
Needham@virginmedia.co.uk=22>Guy.Needham@virginmedia.co.uk</a>&gt;</span> =
wrote:<br><blockquote class=3D=22gmail=5Fquote=22 style=3D=22margin:0 0 0 .=
8ex;border-left:1px #ccc solid;padding-left:1ex=22>


<div>
<font face=3D=22Tahoma=22><span style=3D=22font-size:10pt=22>
<div>Hi all,</div>
<div>=C2=A0</div>
<div>I have a configuration with a file channel configured such that: =
</div>
<div>=C2=A0</div>
<div>a1.channels.ch1.type =3D file</div>
<div>a1.channels.ch1.checkpointDir =3D /hadoop/user/flume/channels/checkpoi=
nt</div>
<div>a1.channels.ch1.dataDirs =3D /hadoop/user/flume/channels/data</div>
<div>a1.channels.ch1.capacity =3D 100000</div>
<div>a1.channels.ch1.transactionCapacity =3D 5000</div>
<div>=C2=A0</div>
<div>It's been running since October 28th with no issues, but when I looked=
 today in /hadoop/user/flume/channels/data I saw that the file channel was =
building up large files which had been processed and not deleting =
them:</div>
<div>=C2=A0</div>
<div>[rdd@hadoop-kn-p2-m01 flume]$ ls -lh channels/data/</div>
<div>total 1.6G</div>
<div>-rw-r----- 1 rdd rdd 1.5G Oct 28 16:10 log-1</div>
<div>-rw-r----- 1 rdd rdd=C2=A0=C2=A0 47 Oct 28 16:10 log-1.meta</div>
<div>-rw-r----- 1 rdd rdd=C2=A0 72M Oct 31 16:28 log-2</div>
<div>-rw-r----- 1 rdd rdd=C2=A0=C2=A0 47 Oct 31 16:29 log-2.meta</div>
<div style=3D=22margin-top:5pt;margin-bottom:5pt=22>It seems like for each =
day that data landed (we're still in testing so data not landing =
constantly) a data file has been created but not deleted when reading was =
completed.</div>
<div style=3D=22margin-top:5pt;margin-bottom:5pt=22>Is this expected =
behaviour=3F Is there a way to stop large files building up and still use =
the file channel=3F</div>
<div style=3D=22margin-top:5pt;margin-bottom:5pt=22>Regards, </div>
<div style=3D=22margin-top:5pt;margin-bottom:5pt=22>Guy Needham | Data =
Discovery<br>

Virgin Media | Enterprise Data, Design &amp; Management<br>

Bartley Wood Business Park, Hook, Hampshire RG27 9UP<br>

D 01256 75 3362 </div>
<div style=3D=22margin-top:5pt;margin-bottom:5pt=22>I welcome VSRE emails. =
Learn more at <a href=3D=22http://vsre.info/=22><font =
color=3D=22blue=22><u>http://vsre.info/</u></font></a> </div>
<div>=C2=A0</div>
<div>=C2=A0</div>
</span></font>
<p><br>
--------------------------------------------------------------------<br>
Save Paper - Do you really need to print this e-mail=3F</p>

<p>Visit <a href=3D=22http://www.virginmedia.com=22>www.virginmedia.com</a>=
 for more information, and more fun.</p>

<p>This email and any attachments are or may be confidential and legally =
privileged<br>
and are sent solely for the attention of the addressee(s). If you have =
received this<br>
email in error, please delete it from your system: its use, disclosure or =
copying is<br>
unauthorised. Statements and opinions expressed in this email may not =
represent<br>
those of Virgin Media. Any representations or commitments in this email =
are<br>
subject to contract. </p>

<p>Registered office: Media House, Bartley Wood Business Park, Hook, =
Hampshire, RG27 9UP<br>
Registered in England and Wales with number 2591237</p>
</div>

</blockquote>
</div>
<br></div>
</blockquote></div><br>
------Nodemailer-0.5.0-?=_1-1415383152133--