Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: local policy)
Message-ID: <541BF232.3010608@vesseltracker.com>
Date: Fri, 19 Sep 2014 11:06:58 +0200
From: Georgi Ivanov <ivanov@vesseltracker.com>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:31.0) Gecko/20100101 Thunderbird/31.1.1
MIME-Version: 1.0
To: user@hadoop.apache.org
Subject: Re: Re-sampling time data with MR job. Ideas
References: <541BE68B.4080603@vesseltracker.com>
 <CALKHTG_BGGvrCzNZbH7_AvcR9trfwKU50BciVJeHXX1tQDMc=Q@mail.gmail.com>
In-Reply-To: 
 <CALKHTG_BGGvrCzNZbH7_AvcR9trfwKU50BciVJeHXX1tQDMc=Q@mail.gmail.com>
Content-Type: multipart/alternative;
 boundary="------------030108060904020205060007"

This is a multi-part message in MIME format.
--------------030108060904020205060007
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit

Hi Mirko,
Thanks for the reply.

Lets assume i have a record every 1 second for every given entity.

entity_id | timestamp | data

1 , 2014-01-01 12:13:01 - i want this
..some more for different entity
1 , 2014-01-01 12:13:02
1 , 2014-01-01 12:13:03
1 , 2014-01-01 12:13:04
1 , 2014-01-01 12:13:05
........
1 , 2014-01-01 12:23:01 - I want this
1 , 2014-01-01 12:23:02

The problem is that in reality this is not coming sorted by entity_id , 
timestamp
so i can't filter in the mapper .
The mapper will get different entity_id's and based on the input split.


Georgi

On 19.09.2014 10:34, Mirko Kämpf wrote:
> Hi Georgi,
>
> I would already emit the new time stamp (with resolution 10 min) in 
> the mapper. This allows you to (pre)aggregate the data already in the 
> mapper and you have less traffic during the shuffle & sort stage. 
> Changing the resolution means you have to aggregate the individual 
> entities or do you still need all individual entities and just want to 
> translate the timestamp to another resolution (5s => 10 min)?
>
> Cheers,
> Mirko
>
>
>
>
> 2014-09-19 9:17 GMT+01:00 Georgi Ivanov <ivanov@vesseltracker.com 
> <mailto:ivanov@vesseltracker.com>>:
>
>     Hello,
>     I have time related data like this :
>     entity_id, timestamp , data
>
>     The resolution of the data is something like 5 seconds.
>     I want to extract the data with 10 minutes resolution.
>
>     So what i can do is :
>     Just emit everything in the mapper as data is not sorted there .
>     Emit only every 10 minutes from reducer. The reducer is receiving
>     data sorted by entity_id,timestamp pair (secondary sorting)
>
>     This will work fine, but it will take forever, since i have to
>     process TB's of data.
>     Also the data emitted to the reducer will be huge( as i am not
>     filtering in map phase at all) and the number of reducers is much
>     smaller than the number of mappers.
>
>     Are there any better ideas how to do this ?
>
>     Georgi
>
>


--------------030108060904020205060007
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit

<html>
  <head>
    <meta content="text/html; charset=utf-8" http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    <div class="moz-cite-prefix">Hi Mirko,<br>
      Thanks for the reply.<br>
      <br>
      Lets assume i have a record every 1 second for every given entity.<br>
      <br>
      entity_id | timestamp | data<br>
      <br>
      1 , 2014-01-01 12:13:01 - i want this<br>
      ..some more for different entity<br>
      1 , 2014-01-01 12:13:02<br>
      1 , 2014-01-01 12:13:03<br>
      1 , 2014-01-01 12:13:04<br>
      1 , 2014-01-01 12:13:05<br>
      ........<br>
      1 , 2014-01-01 12:23:01 - I want this<br>
      1 , 2014-01-01 12:23:02<br>
      <br>
      The problem is that in reality this is not coming sorted by
      entity_id , timestamp <br>
      so i can't filter in the mapper .<br>
      The mapper will get different entity_id's and based on the input
      split.<br>
      <br>
      <br>
      <br>
      Georgi<br>
      <br>
      On 19.09.2014 10:34, Mirko Kämpf wrote:<br>
    </div>
    <blockquote
cite="mid:CALKHTG_BGGvrCzNZbH7_AvcR9trfwKU50BciVJeHXX1tQDMc=Q@mail.gmail.com"
      type="cite">
      <div dir="ltr">Hi Georgi,
        <div><br>
        </div>
        <div>I would already emit the new time stamp (with resolution 10
          min) in the mapper. This allows you to (pre)aggregate the data
          already in the mapper and you have less traffic during the
          shuffle &amp; sort stage. Changing the resolution means you
          have to aggregate the individual entities or do you still need
          all individual entities and just want to translate the
          timestamp to another resolution (5s =&gt; 10 min)?</div>
        <div><br>
        </div>
        <div>Cheers,</div>
        <div>Mirko</div>
        <div><br>
        </div>
        <div><br>
        </div>
        <div><br>
        </div>
      </div>
      <div class="gmail_extra"><br>
        <div class="gmail_quote">2014-09-19 9:17 GMT+01:00 Georgi Ivanov
          <span dir="ltr">&lt;<a moz-do-not-send="true"
              href="mailto:ivanov@vesseltracker.com" target="_blank">ivanov@vesseltracker.com</a>&gt;</span>:<br>
          <blockquote class="gmail_quote" style="margin:0 0 0
            .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello,<br>
            I have time related data like this :<br>
            entity_id, timestamp , data<br>
            <br>
            The resolution of the data is something like 5 seconds.<br>
            I want to extract the data with 10 minutes resolution.<br>
            <br>
            So what i can do is :<br>
            Just emit everything in the mapper as data is not sorted
            there .<br>
            Emit only every 10 minutes from reducer. The reducer is
            receiving data sorted by entity_id,timestamp pair (secondary
            sorting)<br>
            <br>
            This will work fine, but it will take forever, since i have
            to process TB's of data.<br>
            Also the data emitted to the reducer will be huge( as i am
            not filtering in map phase at all) and the number of
            reducers is much smaller than the number of mappers.<br>
            <br>
            Are there any better ideas how to do this ?<span
              class="HOEnZb"><font color="#888888"><br>
                <br>
                Georgi<br>
              </font></span></blockquote>
        </div>
        <br>
      </div>
    </blockquote>
    <br>
  </body>
</html>

--------------030108060904020205060007--