Return-Path: 
 <user-return-5521-apmail-hadoop-user-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 731A3EB33
	for <apmail-hadoop-user-archive@minotaur.apache.org>;
 Wed, 27 Feb 2013 02:40:27 +0000 (UTC)
Received: (qmail 50927 invoked by uid 500); 27 Feb 2013 02:40:22 -0000
Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org
Received: (qmail 50824 invoked by uid 500); 27 Feb 2013 02:40:22 -0000
Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:user-help@hadoop.apache.org>
List-Unsubscribe: <mailto:user-unsubscribe@hadoop.apache.org>
List-Post: <mailto:user@hadoop.apache.org>
List-Id: <user.hadoop.apache.org>
Reply-To: user@hadoop.apache.org
Delivered-To: mailing list user@hadoop.apache.org
Received: (qmail 50816 invoked by uid 99); 27 Feb 2013 02:40:22 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Feb 2013 02:40:22 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of azuryyyu@gmail.com designates
 209.85.223.181 as permitted sender)
Received: from [209.85.223.181] (HELO mail-ie0-f181.google.com)
 (209.85.223.181)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Feb 2013 02:40:17 +0000
Received: by mail-ie0-f181.google.com with SMTP id 17so80284iea.12
        for <user@hadoop.apache.org>; Tue, 26 Feb 2013 18:39:56 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20120113;
        h=mime-version:x-received:in-reply-to:references:date:message-id
         :subject:from:to:content-type;
        bh=BdPZ8t6sY9TEBr8vUmGGuWDesJ67UyFUcEEl9xGrqyU=;
        b=iFEK/NPpVvStuLZFy4BpPWmwskrZI4oy3dxEUBiz6Jy1fK1mx0zhI/CAoAfFEu6/qa
         hgFIHZAOy0/OH+SoNGfQjq5ndC6usXh0IzNC5JG8bDB5Ekuu6nVg39NUYiFGg2LesT9J
         5lcoiLS1Pug+At1AZkz+xx8D0xI/K6u9FaYHy87LLrbNbbzhNWuQGDDOCtHPyM6ARdQ3
         O7N1ff2+uCcLNJy+H9WprGXFXaqRTSyfo+j9G4b5bEZqbWu4nSiluQolcRjjG/ClZiLW
         CcN3aJtS1FEk/qxRZLlWf/39YF6VOo75He9QDL5YE09Whzwk0WtpGC1Q3jodm3Le45KP
         yaBA==
MIME-Version: 1.0
X-Received: by 10.50.42.168 with SMTP id p8mr232154igl.106.1361932796706; Tue,
 26 Feb 2013 18:39:56 -0800 (PST)
Received: by 10.64.26.70 with HTTP; Tue, 26 Feb 2013 18:39:56 -0800 (PST)
In-Reply-To: 
 <CAPxQDUjqB+qWm0VuJF_xi4rm8LcWbPnRw4GGJjnv6tMAkhoQPg@mail.gmail.com>
References: 
 <CAPxQDUjqB+qWm0VuJF_xi4rm8LcWbPnRw4GGJjnv6tMAkhoQPg@mail.gmail.com>
Date: Wed, 27 Feb 2013 10:39:56 +0800
Message-ID: 
 <CALr1C9qn8k5w72yZHa-Q3_m_+rM=eRB9giQBZ6s2i6XpwpRXgA@mail.gmail.com>
Subject: Re: Concatenate adjacent lines with hadoop
From: Azuryy Yu <azuryyyu@gmail.com>
To: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=14dae934108d8e45c504d6abb0d5
X-Virus-Checked: Checked by ClamAV on apache.org

--14dae934108d8e45c504d6abb0d5
Content-Type: text/plain; charset=ISO-8859-1

That's easy, in your example,

Map output key: FIELD-N ; Map output value: just original value.
In the reduece: if there is  LOGTAG<TAB> in the value, then this is the
first log entry. if not, this is a splitted log entry. just get a sub
string and concat with the first log entry.

Am I explain clearly?



On Wed, Feb 27, 2013 at 9:36 AM, Matthieu Labour <matthieu@actionx.com>wrote:

> Hi
>
> Please find below the issue I need to solve. Thank you in advance for your
> help/ tips.
>
> I have log files where sometimes log lines are splited (this happens when
> the log line exceeds a specific length)
>
> Dec 16 21:47:20 d.14b48e47-abf2-403e-8a1a-04e821a42cb6 app[web.3]
> LOGTAG<TAB>FIELD-0<TAB>....<TAB>FIELD-MAX
> Dec 16 21:47:20 d.14b48e47-abf2-403e-8a1a-04e821a42cb6 app[web.3]
> LOGTAG<TAB>FIELD-0<TAB>....<TAB>FIELD-N      <======= log line is being
> splitted
> Dec 16 21:47:20 d.14b48e47-abf2-403e-8a1a-04e821a42cb6 app[web.3]
> FIELD-N<TAB>FIELD-N+1 .....FIELD-MAX
>
> Can I "reconcile"/ "concatenate" splited log lines with a hadoop map
> reduce job?
>
> On other words, using a map reduce job, can I concatenate the 2 following
> adjacent lines (provided that I 'detect' them)
>
> Dec 16 21:47:20 d.14b48e47-abf2-403e-8a1a-04e821a42cb6 app[web.3]
> LOGTAG<TAB>FIELD-0<TAB>....<TAB>FIELD-N      <======= log line is being
> splitted
> Dec 16 21:47:20 d.14b48e47-abf2-403e-8a1a-04e821a42cb6 app[web.3]
> FIELD-N<TAB>FIELD-N+1 .....FIELD-MAX
>
> into
>
> Dec 16 21:47:20 d.14b48e47-abf2-403e-8a1a-04e821a42cb6 app[web.3]
> LOGTAG<TAB>FIELD-0<TAB>....<TAB>FIELD-N<TAB>FIELD-N+1 .....FIELD-MAX
>
> Thank you!
>

--14dae934108d8e45c504d6abb0d5
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>That&#39;s easy, in your example,</div><div>=A0</div>=
<div>Map output key: FIELD-N=A0; Map output value: just original value.</di=
v><div>In the reduece: if there is=A0 LOGTAG&lt;TAB&gt; in the value, then =
this is the first log entry. if not, this is a splitted log entry. just get=
 a sub string and concat with the first log entry.</div>
<div>=A0</div><div>Am I explain clearly?</div><div>=A0</div></div><div clas=
s=3D"gmail_extra"><br><br><div class=3D"gmail_quote">On Wed, Feb 27, 2013 a=
t 9:36 AM, Matthieu Labour <span dir=3D"ltr">&lt;<a href=3D"mailto:matthieu=
@actionx.com" target=3D"_blank">matthieu@actionx.com</a>&gt;</span> wrote:<=
br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">Hi<div><br></div><div>Please find below the =
issue I need to solve. Thank you in advance for your help/ tips.</div><div>
<br></div><div>I have log files where sometimes log lines are splited (this=
 happens when the log line exceeds a specific length)</div>
<div><br></div><div>Dec 16 21:47:20 d.14b48e47-abf2-403e-8a1a-04e821a42cb6 =
app[web.3] LOGTAG&lt;TAB&gt;FIELD-0&lt;TAB&gt;....&lt;TAB&gt;FIELD-MAX</div=
><div><div>Dec 16 21:47:20 d.14b48e47-abf2-403e-8a1a-04e821a42cb6 app[web.3=
] LOGTAG&lt;TAB&gt;FIELD-0&lt;TAB&gt;....&lt;TAB&gt;FIELD-N =A0 =A0 =A0&lt;=
=3D=3D=3D=3D=3D=3D=3D log line is being splitted</div>

<div>Dec 16 21:47:20 d.14b48e47-abf2-403e-8a1a-04e821a42cb6 app[web.3] FIEL=
D-N&lt;TAB&gt;FIELD-N+1 .....FIELD-MAX</div></div><div><br></div><div>Can I=
 &quot;reconcile&quot;/ &quot;concatenate&quot; splited log lines with a ha=
doop map reduce job?</div>

<div><br></div><div>On other words, using a map reduce job, can I concatena=
te the 2 following adjacent lines (provided that I &#39;detect&#39; them)=
=A0</div><div><br></div><div><div>Dec 16 21:47:20 d.14b48e47-abf2-403e-8a1a=
-04e821a42cb6 app[web.3] LOGTAG&lt;TAB&gt;FIELD-0&lt;TAB&gt;....&lt;TAB&gt;=
FIELD-N =A0 =A0 =A0&lt;=3D=3D=3D=3D=3D=3D=3D log line is being splitted</di=
v>

<div>Dec 16 21:47:20 d.14b48e47-abf2-403e-8a1a-04e821a42cb6 app[web.3] FIEL=
D-N&lt;TAB&gt;FIELD-N+1 .....FIELD-MAX</div></div><div><br></div><div>into<=
/div><div><br></div><div>Dec 16 21:47:20 d.14b48e47-abf2-403e-8a1a-04e821a4=
2cb6 app[web.3] LOGTAG&lt;TAB&gt;FIELD-0&lt;TAB&gt;....&lt;TAB&gt;FIELD-N&l=
t;TAB&gt;FIELD-N+1 .....FIELD-MAX</div>

<div><br></div><div>Thank you!</div>
</blockquote></div><br></div>

--14dae934108d8e45c504d6abb0d5--