Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of
 publicnetworkservices@gmail.com designates 209.85.223.181 as permitted
 sender)
MIME-Version: 1.0
Date: Tue, 5 Mar 2013 15:35:01 -0800
Message-ID: 
 <CAPwWofEEsg0P1j3-Gwb8neUik-a5k-4oNM_X+Whrngaw8y4HKg@mail.gmail.com>
Subject: Execution handover in map/reduce pipeline
From: Public Network Services <publicnetworkservices@gmail.com>
To: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=f46d0447f3821bd34004d735ec9d

--f46d0447f3821bd34004d735ec9d
Content-Type: text/plain; charset=ISO-8859-1

Hi...

I have an application that processes large amounts of proprietary
binary-encoded text data in the following sequence

   1. Gets a URL to a file or a directory as input
   2. Reads the list of the binary files found under the input URL
   3. Extracts the text data from each of those files
   4. Saves the text data into new files
   5. Informs the application about newly extracted files
   6. Processes each of the extracted text files
   7. Submits the processing results to a proprietary data repository

This whole processing is highly CPU-intensive and can be partially
parallelized, so I am thinking of trying Hadoop for achieving higher
performance.

So, assuming that all the above take place in HDFS (including the input URL
being an HDFS one), a MapReduce implementation could use

   - A lightweight non-Hadoop thread to kick-start the execution flow, i.e.
   implement step 1
   - A Mapper that would implement steps 2-4
   - A Reducer that would implement step 5 (receive the notifications)
   - A Mapper that would implement step 6
   - A Reducer that would implement step 7

The first mapper (for steps 2-4) will probably need to do its processing in
a single, non-parallelized step.

My question is, how is the first reducer going to hand over execution to
the second mapper, once done?

Or, is there a better way of implementing the above scenario?

Thanks!

--f46d0447f3821bd34004d735ec9d
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi...<div><br></div><div>I have an application that processes large amounts=
 of proprietary binary-encoded text data in the following sequence</div><di=
v><ol><li>Gets a URL to a file or a directory as input</li><li>Reads the li=
st of the binary files found under the input URL</li>
<li>Extracts the text data from each of those files</li><li>Saves the text =
data into new files</li><li>Informs the application about newly extracted f=
iles</li><li>Processes each of the extracted text files</li><li>Submits the=
 processing results to a proprietary data repository</li>
</ol><div>This whole processing is highly CPU-intensive and can be partiall=
y parallelized, so I am thinking of trying Hadoop for achieving higher perf=
ormance.</div></div><div><br></div><div>So, assuming that all the above tak=
e place in HDFS (including the input URL being an HDFS one), a MapReduce im=
plementation could use</div>
<div><ul><li>A lightweight non-Hadoop thread to kick-start the execution fl=
ow, i.e. implement step 1</li><li>A Mapper that would implement steps 2-4</=
li><li>A Reducer that would implement step 5 (receive the notifications)</l=
i>
<li>A Mapper that would implement step 6</li><li>A Reducer that would imple=
ment step 7</li></ul><div>The first mapper (for steps 2-4) will probably ne=
ed to do its processing in a single, non-parallelized step.</div><div><br>
</div><div>My question is, how is the first reducer going to hand over exec=
ution to the second mapper, once done?</div><div><br></div><div>Or, is ther=
e a better way of implementing the above scenario?</div><div></div></div>
<div><br></div><div>Thanks!</div><div><br></div>

--f46d0447f3821bd34004d735ec9d--