incubator-hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward J. Yoon" <edwardy...@apache.org>
Subject Re: Custom partitioner in pig
Date Tue, 17 Mar 2009 08:49:41 GMT
Hi,

Interesting. FYI, We're use the Hbase. Check this out --
http://wiki.apache.org/hama/Architecture#head-29381b028f7a92606e6a3a59722e1ca084a91ab8

I think there is no way to sequentially collect the blocks at once.
Iterative jobs or Hbase will be needed.

If you have more good idea, pls let us know, too.
Thanks.

-- 
Best Regards, Edward J. Yoon
edwardyoon@apache.org
http://blog.udanax.org

On Tue, Mar 17, 2009 at 1:08 AM, Mridul Muralidharan
<mridulm@yahoo-inc.com> wrote:
>
> Hi,
>
>  My goal here is slightly different - and might not very 'nicely' fit into
> pig.
> Idea is to implement dense block matrix multiplication - and an efficient
> way to do that is to pair up the blocks as required in the partitioner
> itself (as opposed to partitioning for a single table as the jira seems to
> suggest).
>
> Once this is supported, matrix mult becomes just block division udf, custom
> partitioning, block multiplication of the partitioned blocks, summation of
> the results to get result.
> Hama guys are attempting something similar, though I did not see too much as
> 'code'.
>
>
> So assuming there is no way to do this currently in pig, I will need to
> investigate other options I guess.
>
> Thanks,
> Mridul
>
> Alan Gates wrote:
>>
>> Not yet, but we've had other requests for it.
>>  https://issues.apache.org/jira/browse/PIG-282
>>
>> Alan.
>>
>> On Mar 12, 2009, at 4:49 AM, Mridul Muralidharan wrote:
>>
>>> Hi,
>>>
>>>  Is there a way to specify or write a custom partitioner in pig ?
>>> Not split - partition data in a specific way - for some custom job.
>>>
>>> Thanks,
>>> Mridul
>>
>
>

Mime
View raw message