accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ACCUMULO-3602) BatchScanner optimization for AccumuloInputFormat
Date Wed, 08 Apr 2015 23:43:12 GMT

    [ https://issues.apache.org/jira/browse/ACCUMULO-3602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14486343#comment-14486343
] 

ASF GitHub Bot commented on ACCUMULO-3602:
------------------------------------------

Github user keith-turner commented on the pull request:

    https://github.com/apache/accumulo/pull/25#issuecomment-91069467
  
    I was discussing the use case I mentioned offline w/ @ctubbsii.  This use case was a large
number of ranges that can not be generated by a function.  We determined that function could
handle this case well by storing the ranges somewhere else beside the job conf.  For example
could do the following.
    
     * Store 10,000,000 sorted ranges in file in distributed cache (assume thousands of tablets)
     * Using the provided function, each mapper opens the file and reads the ranges for the
tablet its working on.
     * The ranges returned by the function are used to initialize the batch scanner for each
mapper.
    
    It seems like all of the use cases that the current implementation satisfies could also
be satisfied with the functor implementation.  
    
    If this PR does not follow the functor approach discussed, thats ok w/ me.   I can open
up a follow on issue to record the discussion if its not pursued here.


> BatchScanner optimization for AccumuloInputFormat
> -------------------------------------------------
>
>                 Key: ACCUMULO-3602
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-3602
>             Project: Accumulo
>          Issue Type: Improvement
>          Components: client
>    Affects Versions: 1.6.1, 1.6.2
>            Reporter: Eugene Cheipesh
>            Assignee: Eugene Cheipesh
>              Labels: performance
>             Fix For: 1.7.0
>
>
> Currently {{AccumuloInputFormat}} produces a split for reach {{Range}} specified in the
configuration. Some table indexing schemes, for instance z-order geospacial index, produce
large number of small ranges resulting in large number of splits. This is specifically a concern
when using {{AccumuloInputFormat}} as a source for Spark RDD where each Split is mapped to
an RDD partition.
> Large number of small RDD partitions leads to poor parallism on read and high overhead
on processing. A desirable alternative is to group ranges by tablet into a single split and
use {{BatchScanner}} to produce the records. Grouping by tablets is useful because it represents
Accumulos attempt to distributed stored records and can be influance by the user through table
splits.
> The grouping functionality already exists in the internal {{TabletLocator}} class. 
> Current proposal is to modify {{AbstractInputFormat}} such that it generates either {{RangeInputSplit}}
or {{MultiRangeInputSplit}} based on a new setting in {{InputConfigurator}}.  {{AccumuloInputFormat}}
would then be able to inspect the type of the split and instantiate an appropriate reader.
> The functinality of {{TabletLocator}} should be exposed as a public API in 1.7 as it
is useful for optimizations.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message