hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Doug Meil <doug.m...@explorysmedical.com>
Subject RE: Question about MapReduce
Date Mon, 19 Oct 2009 14:34:03 GMT
Hi there-

I didn't see the option in the thread yet which seems pretty straightforward:

When setting up the job:

    Job job = new Job(conf, "my job");

    conf.setStrings("param", "param1");

And then in the map method:

      String paramVal = context.getConfiguration().get("param");

This is using Hadoop .20 syntax, the previous verison had a 'configure' method you had to

-----Original Message-----
From: Something Something [mailto:luckyguy2050@yahoo.com] 
Sent: Thursday, October 15, 2009 2:31 PM
To: hbase-user@hadoop.apache.org; Hadoop
Subject: Re: Question about MapReduce

1) I don't think TableInputFormat is useful in this case.  Looks like it's used for scanning
columns from a single HTable.
2) TableMapReduceUtil - same problem.  Seems like this works with just one table.
3) JV recommended NLineInputFormat, but my parameters are not in a file.  They come from multiple
files and are in memory.

I guess what I am looking for is something like... InMemoryInputFormat... similar to FileInputFormat
& DbInputFormat.  There's no such class right now.

Worse comes to worst, I can write the parameters into a flat file, and use FileInputFormat
- but that will slow down this process considerably.  Is there no other way?

From: Mark Vigeant <mark.vigeant@riskmetrics.com>
To: "hbase-user@hadoop.apache.org" <hbase-user@hadoop.apache.org>
Sent: Thu, October 15, 2009 7:21:40 AM
Subject: RE: Question about MapReduce

There is a tableInputFormat class in org.apache.hadoop.hbase.mapreduce.TableInputFormat

Also, if you want to use TableMapReduceUtil you probably want to have your mapper function
extend TableMapper.

Check out the javadocs for more info: http://hadoop.apache.org/hbase/docs/current/api/index.html

-----Original Message-----
From: Something Something [mailto:luckyguy2050@yahoo.com] 
Sent: Thursday, October 15, 2009 1:37 AM
To: general@hadoop.apache.org; hbase-user@hadoop.apache.org
Subject: Re: Question about MapReduce

If the answer is...


I apologize for the spam.  If this isn't the right way, please let me know.  Thanks.

--- On Wed, 10/14/09, Something Something <luckyguy2050@yahoo.com> wrote:

From: Something Something <luckyguy2050@yahoo.com>
Subject: Question about MapReduce
To: general@hadoop.apache.org, hbase-user@hadoop.apache.org
Date: Wednesday, October 14, 2009, 10:18 PM

I would like to start a Map-Reduce job that does not read data from an input file or from
a database.  I would like to pass 3 arguments to the Mapper & Reducer to work on.  Basically,
these arguments are keys on the 3 different tables on HBase.

In other words, I don't want to use FileInputFormat or DbInputFormat because everything I
need is already on HBase. 

How can I do this?  Please let me know.  Thanks.


View raw message