incubator-hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Jungblut (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HAMA-476) Splitter doesn't work correctly
Date Wed, 16 Nov 2011 16:33:51 GMT

    [ https://issues.apache.org/jira/browse/HAMA-476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13151305#comment-13151305
] 

Thomas Jungblut commented on HAMA-476:
--------------------------------------

bq. To split sequencefile as user requested size, there's no way to avoid read/write records.
I think we have to use just blockSize.

Correct, we have to split via the blocks.

bq. Unlike MapReduce, we are unable to queuing tasks when exceeds cluster capacity (I have
no idea at the moment).

There is no idea to have, we have to restrict more tasks than the cluster capacity. In YARN
this issue is even worse, because you don't know the capacity.


bq. From what I discovered so far, the first one ideally can be achieved by applying tiling
strategy. Then we can provide wrapper classes for user to access according to range requested.

How is this tiling gonna work without rewriting sequence files?
                
> Splitter doesn't work correctly
> -------------------------------
>
>                 Key: HAMA-476
>                 URL: https://issues.apache.org/jira/browse/HAMA-476
>             Project: Hama
>          Issue Type: Bug
>          Components: bsp
>    Affects Versions: 0.3.0
>            Reporter: Edward J. Yoon
>            Assignee: Edward J. Yoon
>             Fix For: 0.4.0
>
>
> - To split sequencefile as user requested size, there's no way to avoid read/write records.
I think we have to use just blockSize. 
> - Unlike MapReduce, we are unable to queuing tasks when exceeds cluster capacity (I have
no idea at the moment).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message