Mailing-List: contact chukwa-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: chukwa-dev@incubator.apache.org
Received-SPF: pass (athena.apache.org: domain of luangsay@gmail.com designates
 209.85.214.182 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAOTz1wCPBQuQ=5V=wgVeWNzMpY6DFbqQ+H5jaDL+L1HeJfCw3Q@mail.gmail.com>
References: 
 <CAOTz1wDB+8mhZ+KNwh-GxnUGHPTako+G7hrCP08GiCxVpqwDWQ@mail.gmail.com>
	<CADfbTpFRzqhcErb_srY_kChATvo9NnU+mvYLQQxNt1qqVHkiww@mail.gmail.com>
	<CAOTz1wCPBQuQ=5V=wgVeWNzMpY6DFbqQ+H5jaDL+L1HeJfCw3Q@mail.gmail.com>
Date: Sun, 21 Apr 2013 17:05:48 +0200
Message-ID: 
 <CAOTz1wB1XmBd7jGR58sw_mY6JExnn6wsMG76cjrVLzR2zRL=xw@mail.gmail.com>
Subject: Re: Creating a new adaptor: FileTailingAdaptor that would not cut
 lines
From: Luangsay Sourygna <luangsay@gmail.com>
To: chukwa-dev@incubator.apache.org
Content-Type: multipart/alternative; boundary=089e015368c88be8e204dae049f2

--089e015368c88be8e204dae049f2
Content-Type: text/plain; charset=ISO-8859-1

Here is the Jira I opened:
https://issues.apache.org/jira/browse/CHUKWA-686

Writting the Junit tests, I discovered a small "error" in the classes
CharFileTailingAdaptorUTF8 and CharFileTailingAdaptorUTF8NewLineEscaped.
When we create the chunk, all the buffer is passed to the constructor,
meaning that the chunk will get both the usefull data and the useless data:
      ChunkImpl event = new ChunkImpl(type, toWatch.getAbsolutePath(),
                buffOffsetInFile + bytesUsed, buf, this);

I think we should only pass the usefull part of the data, just like this:
      ChunkImpl event = new ChunkImpl(type,
              toWatch.getAbsolutePath(), buffOffsetInFile
              + bytesUsed, Arrays.copyOf(buf, bytesUsed), this);

Although it does not seem a real issue because the method hasNext() of
AbstractProcessor.class ensures we only process the usefull part, I see two
reasons to fix this:
- it makes CharFileTailingAdaptorUTF8 fails some of my unit tests
(TestFileTailingAdaptorPreserveLines.testDontBreakLines() for instance)
that should not fail for this adaptor.
- we send data on the network for nothing. Since the useless part only
represents less than a line, it is not usually a big deal: we only transfer
a few bytes for nothing. However, a customer of mine has a log file with
lines as long as 300 kB (I know, quite strange for a "log file"...) so in
that case I think the fix is worth it.

Regards,

Sourygna


On Fri, Apr 19, 2013 at 9:01 PM, Luangsay Sourygna <luangsay@gmail.com>wrote:

> Well, log4j socket adaptor may be great if you control the software that
> generates logs.
> That is not usually my case: customers don't really like having to install
> a Chukwa agents
> on their production servers so I don't want to think about telling them to
> change the log system
> of their software.
>
> As for partial line when log files rotate, I don't think this is something
> Chukwa should manage (what
> is more: how could Chukwa be aware there is a problem?).
> To my view, this would be an error of the "logrotate" system. As far as I
> know, RFA and DRFA log4j
> appenders handle quite well the rotation.
>
> Regards,
>
> Sourygna
>
>
> On Fri, Apr 19, 2013 at 8:17 AM, Eric Yang <eric818@gmail.com> wrote:
>
>> I think the best solution is to use Log4j socket appender and Chukwa log4j
>> socket adaptor to get the full entry of the log without worry about line
>> feed.  However, this solution only works with program that is written in
>> Java, and does not keep a copy of existing log file on disk.
>>
>> I think your proposal is a good idea to solve tailing text file and only
>> line delimited entry will be send.  How do we handle partial line and log
>> file has rotated?
>>
>> regards,
>> Eric
>>
>> On Thu, Apr 18, 2013 at 11:33 AM, Luangsay Sourygna <luangsay@gmail.com
>> >wrote:
>>
>> > Hi all,
>> >
>> > FileTailingAdaptor is great to tail log files and send them to Hadoop.
>> > However, last line of the chunk is usually cut which leads to some
>> errors.
>> >
>> > I know that we can use CharFileTailingAdaptorUTF8 to solve such problem.
>> > Nonetheless, this adaptor calls the MapProcessor.process() method for
>> every
>> > line in each chunk, thus slowing a lot the Demux phase.
>> >
>> > I suggest creating a new adaptor that would mix the benefits of the two
>> > adaptors: the (Demux) speed of FileTailingAdaptor and
>> > the preservation of lines from CharFileTailingAdaptorUTF8.
>> >
>> > The implementation of the extractRecords() would be:
>> > - "for loop" on the buffer, starting from the end of the buffer and
>> going
>> > backward
>> > - if we find a separator, save the offset and exit the loop
>> > - rest of method would be similar to CharFileTailingAdaptorUTF8.
>> >
>> > Could you guys please tell me what do you think about it?
>> > How do you currently manage the "lines cut" with Chukwa?
>> >
>> > Regards,
>> >
>> > Sourygna
>> >
>>
>
>

--089e015368c88be8e204dae049f2--