Mailing-List: contact user-help@flume.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flume.apache.org
Received-SPF: pass (nike.apache.org: domain of brock@cloudera.com designates
 209.85.160.51 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <50586446.3000100@pubmatic.com>
References: <50519FD8.9000501@pubmatic.com> <50586446.3000100@pubmatic.com>
From: Brock Noland <brock@cloudera.com>
Date: Tue, 18 Sep 2012 09:37:01 -0500
Message-ID: 
 <CAFukC=7_J=TVY8VMY9uDHJ=3d39CdNyvCiDznMYvnGPQEc8AJw@mail.gmail.com>
Subject: Re: HDFS file rolling behaviour
To: user@flume.apache.org
Content-Type: multipart/alternative; boundary=047d7b16051f0b0c3c04c9fad47b

--047d7b16051f0b0c3c04c9fad47b
Content-Type: text/plain; charset=ISO-8859-1

If you have not increased the OS number of open files limit, you should.
The default limit of 1024 is too low for nearly every modern application.

In regards to the rolling, can you paste you config and describe in more
detail the unexpected behavior you are seeing?

Brock

On Tue, Sep 18, 2012 at 7:08 AM, Jagadish Bihani <
jagadish.bihani@pubmatic.com> wrote:

>  Hi
>
> Does anybody know about  the issue mentioned in the following mail?
>
>
> Update: I have seen following behaviour now even for time based rolling.
> By time based rolling I would expect: That single file should be created
> after x seconds.
> But in my case some n files are created after every x seconds.
> Is it something to do with HDFS batch size?
>
> Regards,
> Jagadish
>
>
>
> -------- Original Message --------  Subject: HDFS file rolling behaviour  Date:
> Thu, 13 Sep 2012 14:26:56 +0530  From: Jagadish Bihani
> <jagadish.bihani@pubmatic.com> <jagadish.bihani@pubmatic.com>  To:
> user@flume.apache.org
>
> Hi
>
> I use two flume agents:
> 1. flume_agent 1 which is a source with (exec source -file channel -avro
> sink)
> 2. flume_agent 2 which is a dest with (avro source -file channel - HDFS
> sink)
>
> I have observed that for HDFS sink with rolling by *file size/number of
> events* it
> creates a lot of simultaneous connections to source's avro sink. But
> while rolling by *time interval* it does it *one by one* i.e. opens 1
> HDFS file write to
> it and then close it.  I expect for other rolling intervals too same thing
> should happen
> i.e.  first open file and if x number of events are written to it then
> roll it and open another
> and so on.
>
> In my case my data ingestion works fine with "time" based rolling but in
> other
> cases due to the above behaviour I get exceptions like:
> -- too many open files
> -- timeout related exceptions for file channel and few more exceptions.
>
> I can increase the values of the parameters giving exceptions but I dont
> know what
> adverse effects it may have.
>
> Can somebody throw some light on the rolling based on file size/number of
> events ?
>
> Regards,
> Jagadish
>
>
>
>


-- 
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/

--047d7b16051f0b0c3c04c9fad47b
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

If you have not increased the OS number of open files limit, you should. Th=
e default limit of 1024 is too low for nearly every modern application.<div=
><br></div><div>In regards to the rolling, can you paste you config and des=
cribe in more detail the unexpected behavior you are seeing?<br>

<div><br></div><div>Brock</div><div><br><div class=3D"gmail_quote">On Tue, =
Sep 18, 2012 at 7:08 AM, Jagadish Bihani <span dir=3D"ltr">&lt;<a href=3D"m=
ailto:jagadish.bihani@pubmatic.com" target=3D"_blank">jagadish.bihani@pubma=
tic.com</a>&gt;</span> wrote:<br>

<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">
 =20
   =20
 =20
  <div bgcolor=3D"#FFFFFF" text=3D"#000000">
    Hi<br>
    <br>
    Does anybody know about=A0 the issue mentioned in the following mail?<b=
r>
    <br>
    <br>
    Update: I have seen following behaviour now even for time based
    rolling.<br>
    By time based rolling I would expect: That single file should be
    created after x seconds.<br>
    But in my case some n files are created after every x seconds.<br>
    Is it something to do with HDFS batch size?<br>
    <br>
    Regards,<br>
    Jagadish<div><div class=3D"h5"><br>
    <div><br>
      <br>
      -------- Original Message --------
      <table border=3D"0" cellpadding=3D"0" cellspacing=3D"0">
        <tbody>
          <tr>
            <th align=3D"RIGHT" nowrap valign=3D"BASELINE">Subject:
            </th>
            <td>HDFS file rolling behaviour</td>
          </tr>
          <tr>
            <th align=3D"RIGHT" nowrap valign=3D"BASELINE">Date: </th>
            <td>Thu, 13 Sep 2012 14:26:56 +0530</td>
          </tr>
          <tr>
            <th align=3D"RIGHT" nowrap valign=3D"BASELINE">From: </th>
            <td>Jagadish Bihani <a href=3D"mailto:jagadish.bihani@pubmatic.=
com" target=3D"_blank">&lt;jagadish.bihani@pubmatic.com&gt;</a></td>
          </tr>
          <tr>
            <th align=3D"RIGHT" nowrap valign=3D"BASELINE">To: </th>
            <td><a href=3D"mailto:user@flume.apache.org" target=3D"_blank">=
user@flume.apache.org</a></td>
          </tr>
        </tbody>
      </table>
      <br>
      <br>
     =20
      Hi<br>
      <br>
      I use two flume agents:<br>
      1. flume_agent 1 which is a source with (exec source -file channel
      -avro sink)<br>
      2. flume_agent 2 which is a dest with (avro source -file channel -
      HDFS sink)<br>
      <br>
      I have observed that for HDFS sink with rolling by <b>file
        size/number of events</b> it<br>
      creates a lot of simultaneous connections to source&#39;s avro sink.
      But<br>
      while rolling by <b>time interval</b> it does it <b>one by one</b>
      i.e. opens 1 HDFS file write to <br>
      it and then close it.=A0 I expect for other rolling intervals too
      same thing should happen<br>
      i.e.=A0 first open file and if x number of events are written to it
      then roll it and open another<br>
      and so on. <br>
      <br>
      In my case my data ingestion works fine with &quot;time&quot; based r=
olling
      but in other <br>
      cases due to the above behaviour I get exceptions like:<br>
      -- too many open files<br>
      -- timeout related exceptions for file channel and few more
      exceptions. <br>
      <br>
      I can increase the values of the parameters giving exceptions but
      I dont know what<br>
      adverse effects it may have.<br>
      <br>
      Can somebody throw some light on the rolling based on file
      size/number of events ?<br>
      <br>
      Regards,<br>
      Jagadish<br>
      <br>
      <br>
    </div>
    <br>
  </div></div></div>

</blockquote></div><br><br clear=3D"all"><div><br></div>-- <br>Apache MRUni=
t - Unit testing MapReduce - <a href=3D"http://incubator.apache.org/mrunit/=
" target=3D"_blank">http://incubator.apache.org/mrunit/</a><br>
</div></div>

--047d7b16051f0b0c3c04c9fad47b--