incubator-hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ChiaHung Lin (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HAMA-258) Design a input and output system
Date Fri, 04 Nov 2011 08:17:00 GMT

    [ https://issues.apache.org/jira/browse/HAMA-258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13143820#comment-13143820
] 

ChiaHung Lin commented on HAMA-258:
-----------------------------------

The current patch supports contiguous access mode. In addition to this, our framework also
needs the ability to have random access to the underlying data. These can be achieved with
the expression of data layout. 

Idea is borrowed from mpi i/o. A file is consisted of file type(s), which is a template/ pattern
in describing data layout, and displacement, an absolute position from which the first file
type begins. File type is consisted of elementary type, which is the basic construct unit
(e.g. byte), and holes that define non-accessible area. Suppose there are 3 processes (or
tasks). The first process holds a file type containing 1 elementary type and 5 holes (total
size is 6 elementary types) with its elementary type sitting at the first position (e.g. array[0]).
The second holds 2 elementary types and 4 holes where elementary types stay at the 2nd and
3rd position (array[1] and array[2]). The third holds 3 elementary types and 3 holes with
elementary type positions at the 4th, 5th, and 6th (array[3], array[4], array[5]). Holes occupy
places where elementary type left marking the data is non-accessible. 

The whole file thus can be expressed with the composition of three processes/ tasks. And each
process has a view to the part that it want to access. Therefore, an contiguous split can
be expressed by constructing how many times an elementary type (e.g. byte) would repeat (n
x byte). For non contiguous access (random access), a process can specify elementary type
and the layout (file type) to describe data it wants to access. 

The benefit, according to mpi i/o report, is its design goal favours common usage patterns
(90%) corresponded to real world requirements. 
                
> Design a input and output system
> --------------------------------
>
>                 Key: HAMA-258
>                 URL: https://issues.apache.org/jira/browse/HAMA-258
>             Project: Hama
>          Issue Type: New Feature
>          Components: bsp
>    Affects Versions: 0.3.0
>            Reporter: Edward J. Yoon
>            Assignee: Edward J. Yoon
>             Fix For: 0.4.0
>
>         Attachments: HAMA-258_improved.patch, IONoInput.patch, io_v01.patch, io_v02.patch,
io_v03.patch, io_v04.patch
>
>
> This issue will handle the input and output system with data splitter.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message