hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Owen O'Malley" <omal...@apache.org>
Subject Re: JVM Spawning
Date Wed, 03 Sep 2008 04:27:10 GMT
On Tue, Sep 2, 2008 at 9:13 PM, Ryan LeCompte <lecompte@gmail.com> wrote:

> I see... so there really isn't a way for me to test a map/reduce
> program using a single node without incurring the overhead of
> upping/downing JVM's... My input is broken up into 5 text files.... is
> there a way I could start the job such that it only uses 1 map to
> process the whole thing? I guess I'd have to concatenate the files
> into 1 file and somehow turn off splitting?

There is a MultipleFileInputFormat, but it is less useful than it should be,
but it is a good
place to start. If you defining a MultipleFileInputFormat that reads text
files should be pretty easy and it will give you a single map for your job.
Otherwise, yes, you'll need to make a single file and ask for a single map.

-- Owen

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message