hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From cliff palmer <palmercl...@gmail.com>
Subject Re: return a parameter using Map only
Date Thu, 23 Sep 2010 11:33:22 GMT
Shi, you can certainly use one of the in-memory file systems like mcached,
but it's a lot of work to set that up just to avoid two i/o operations.

On Wed, Sep 22, 2010 at 5:06 PM, Shi Yu <shiyu@uchicago.edu> wrote:

> Dear Hadoopers,
> I am stuck at a probably very simple problem but can't figure it out. In
> the Hadoop Map/Reduce framework, I want to search a huge file (which is
> generated by another Reduce task) for a unique line of record (a <String,
> double> value actually). That record is expected to be passed to another
> function. I have read the previous post about using Mapper only output to
> HBase (
> http://www.mail-archive.com/hbase-user@hadoop.apache.org/msg06579.html)
> and another post (
> http://www.mail-archive.com/hbase-user@hadoop.apache.org/msg07337.html).
> They are both very interesting, however, I am still confused about how to
> circle away from writing to HBase, but to use the returned record directly
> from memory? I guess my problem doesn't need a reducer, so basically
> load-balance the search task via multiple Mappers. I want to have something
> like this
>   class myClass
>          method seekResultbyMapper (string toSearch, path reduceFile)
>               call Map(a,b)
>               do some simple calculation
>               return <String, double> result
>    class anotherClass
> <String, double>  para  =  myClass.seekResultbyMapper (c,d)
> I don't know whether this is doable (maybe it is not a valid style in
> Map/Reduce framework)? How to implement it using JAVA API? Thanks for any
> suggestion in advance.
> Best Regards,
> Shi
> --
> Postdoctoral Scholar
> Institute for Genomics and Systems Biology
> Department of Medicine, the University of Chicago
> Knapp Center for Biomedical Discovery
> 900 E. 57th St. Room 10148
> Chicago, IL 60637, US
> Tel: 773-702-6799

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message