hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 임정택 <kabh...@gmail.com>
Subject Can I configure multiple M/Rs and normal processes to one workflow?
Date Wed, 04 Feb 2015 02:44:51 GMT
Hello all.

We're periodically scan HBase tables to aggregate statistic information,
and store it to MySQL.

We have 3 kinds of CP (kind of data source), each has one Channel and one
Article table.
(Channel : Article is 1:N relation.)

All CPs table schema are different a bit, so in order to aggregate we
should apply different logics, with joining Channel and Article.

I've thought about workflow like this, but I wonder it can make sense.

1. run single process which initializes MySQL by creating table, deleting
row, etc.
2. run 3 M/Rs simultaneously to aggregate statistic information for each
CP, and insert rows  per Channel to MySQL.
3. run single process which finalizes whole aggregation - runs aggregation
query from MySQL to insert new row to MySQL, rolling table, etc.

Definitely 1,2,3 should be run in a row.

Any helps are really appreciated!

Jungtaek Lim (HeartSaVioR)

View raw message