hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From daemeon reiydelle <daeme...@gmail.com>
Subject Re: Can I configure multiple M/Rs and normal processes to one workflow?
Date Wed, 04 Feb 2015 19:50:11 GMT
Null map step (at a guess?), 3 step reduce. No problem. Suspect 3 may be
rather long running?



*.......*






*“Life should not be a journey to the grave with the intention of arriving
safely in apretty and well preserved body, but rather to skid in broadside
in a cloud of smoke,thoroughly used up, totally worn out, and loudly
proclaiming “Wow! What a Ride!” - Hunter ThompsonDaemeon C.M. ReiydelleUSA
(+1) 415.501.0198London (+44) (0) 20 8144 9872*

On Tue, Feb 3, 2015 at 6:44 PM, 임정택 <kabhwan@gmail.com> wrote:

> Hello all.
>
> We're periodically scan HBase tables to aggregate statistic information,
> and store it to MySQL.
>
> We have 3 kinds of CP (kind of data source), each has one Channel and one
> Article table.
> (Channel : Article is 1:N relation.)
>
> All CPs table schema are different a bit, so in order to aggregate we
> should apply different logics, with joining Channel and Article.
>
> I've thought about workflow like this, but I wonder it can make sense.
>
> 1. run single process which initializes MySQL by creating table, deleting
> row, etc.
> 2. run 3 M/Rs simultaneously to aggregate statistic information for each
> CP, and insert rows  per Channel to MySQL.
> 3. run single process which finalizes whole aggregation - runs aggregation
> query from MySQL to insert new row to MySQL, rolling table, etc.
>
> Definitely 1,2,3 should be run in a row.
>
> Any helps are really appreciated!
> Thanks.
>
> Regards.
> Jungtaek Lim (HeartSaVioR)
>

Mime
View raw message