hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Binglin Chang (Updated) (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-3086) Supporting range scan using TFile, TotalOrderPartitioner and partition index
Date Tue, 11 Oct 2011 14:03:12 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-3086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Binglin Chang updated MAPREDUCE-3086:

    Affects Version/s:
               Status: Patch Available  (was: Open)

preliminary patch for 0.20
> Supporting range scan using TFile, TotalOrderPartitioner and partition index
> ----------------------------------------------------------------------------
>                 Key: MAPREDUCE-3086
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3086
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>    Affects Versions:
>            Reporter: Binglin Chang
>            Assignee: Binglin Chang
> Hive/HBase already has similar or more powerful functionality, but using hive/hbase is
overkill or inconvenient for some cases, so add some lightweight utility classes to only support
range scan should be reasonable. The utility classes include:
> # InputFormat supporting range scan: Indexed(Text|Binary)InputFormat
>   The input directory for IndexInputFormat should contain one partition index and many
tfiles, each tfile store a certain range of keys, not overlapping with other tfiles, the boundaries
are stored in partition index.
>   Add 4 jobconfs: mapred.indexed(text|binary)inputformat.key.(start|end), indicate range
scan parameters. 
>   For a mapreduce job using IndexedInputFormat, IndexedInputFormat.getSplits filter out
tfiles which are not in the scan range using partition index
>   IndexedInputFormat do not support multi directory & splitting in single file, these
can be added in future.
> # Tool to convert data of other format into IndexedInputForamt: TotalOrderIndexBuilder
>   If the input data is already total order partitioned and is tfile format, just add
partition index to input directory
>   Or run InputSampler to generate partiton index, then run mapreduce job with TotalOrder
partitioner to generate tfile backed data, finally move partition index to output directory.

> # Client tool to scan/search indexed data directory

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message