hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sandy <snickerdoodl...@gmail.com>
Subject Is Hadoop Really the right framework for me?
Date Thu, 10 Jul 2008 21:47:21 GMT

I have been posting on the forums for a couple of weeks now, and I really
appreciate all the help that I've been receiving. I am fairly new to Java,
and even newer to the Hadoop framework. While I am sufficiently impressed
with the Hadoop, quite a bit of the underlying functionality is masked to
the user (which, while I understand is the point of a Map Reduce Framework,
can be a touch frustrating for someone who is still trying to learn their
way around), and the documentation is sometimes difficult to navigate. I
have been thusfar unable to sufficiently find an answer to this question on
my own.

My goal is to implement a fairly simple map reduce algorithm. My question
is, "Is Hadoop really the right framework to use for this algorithm?"

I have one very large file containing multiple lines of text. I want to
assign a mapper job to each line. Furthermore, the mapper needs to be able
to know what line it is processing. If we were thinking about this in terms
of the Word Count Example, let's say we have a modification where we want to
just see where the words came from, rather than just the count of the words.

For this example, we have the file:

Hello World
Hello Hadoop
Goodbye Hadoop

I want to assign a mapper to each line. Each mapper will emit a word and its
corresponding line number. For this example, we would have three mappers,
(call them m1, m2, and m3). Each mapper will emit the following:

m1 emits:
<"Hello", 1> <"World", 1>

m2 emits:
<"Hello", 2> <"Hadoop", 2>

m3 emits:
<"Goodbye",3> <"Hadoop", 3>

My reduce function will count the number of words based on the -instances-
of line numbers they have, which is necessary, because I wish to use the
line numbers for another purpose.

I have tried Hadoop Pipes, and the Hadoop Python interface. I am now looking
at the Java interface, and am still puzzled how quite to implement this,
mainly because I don't see how to assign mappers to lines of files, rather
than to files themselves. From what I can see from the documentation, Hadoop
seems to be more suitable for applications that deal multiple files rather
than multiple lines. I want it to be able to spawn for any input file, a
number of mappers corresponding to the number of lines. There can be a cap
on the number of mappers spawned (e.g. 128) so that if the number of lines
exceed the number of mappers, then the mappers can concurrently process
lines until all lines are exhausted. I can't see a straightfoward way to do
this using the Hadoop framework.

Please keep in mind that I cannot put each line in its own separate file;
the number of lines in my file is sufficiently large that this is really not
a good idea.

Given this information, is Hadoop really the right framework to use? If not,
could you please suggest alternative frameworks? I am currently looking at
Skynet and Erlang, though I am not too familiar with either.

I would appreciate any feedback. Thank you for your time.



  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message