Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: local policy)
Message-ID: <541BE68B.4080603@vesseltracker.com>
Date: Fri, 19 Sep 2014 10:17:15 +0200
From: Georgi Ivanov <ivanov@vesseltracker.com>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:31.0) Gecko/20100101 Thunderbird/31.1.1
MIME-Version: 1.0
To: "user@hadoop.apache.org" <user@hadoop.apache.org>
Subject: Re-sampling time data with MR job. Ideas
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit

Hello,
I have time related data like this :
entity_id, timestamp , data

The resolution of the data is something like 5 seconds.
I want to extract the data with 10 minutes resolution.

So what i can do is :
Just emit everything in the mapper as data is not sorted there .
Emit only every 10 minutes from reducer. The reducer is receiving data 
sorted by entity_id,timestamp pair (secondary sorting)

This will work fine, but it will take forever, since i have to process 
TB's of data.
Also the data emitted to the reducer will be huge( as i am not filtering 
in map phase at all) and the number of reducers is much smaller than the 
number of mappers.

Are there any better ideas how to do this ?

Georgi