hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Boyu Zhang <boyuzhan...@gmail.com>
Subject Questions About Passing Parameters to Hadoop Job
Date Sun, 22 Nov 2009 20:21:23 GMT
Dear All,

I am implementing an algorithm that read a data file(.txt file,
approximately 90MB), compare each line of the data file with each line of a
specific samples file(.txt file, approximately 20MB). To do this, I need to
pass each line of the samples file as parameters to map-reduce job. And they
are large, in a sense.

My current way is that I use the job.set and job.get to set and retrieve
these lines as configurations. But it is not efficient at all!

Could anyone help me with an alternative solution? Thanks a million!

Boyu Zhang
University of Delaware

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message