accumulo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From echeipesh <...@git.apache.org>
Subject [GitHub] accumulo pull request: ACCUMULO-3602 BatchScanner optimization for...
Date Wed, 08 Apr 2015 15:34:29 GMT
Github user echeipesh commented on a diff in the pull request:

    https://github.com/apache/accumulo/pull/25#discussion_r27981389
  
    --- Diff: core/src/main/java/org/apache/accumulo/core/client/mapred/AbstractInputFormat.java
---
    @@ -629,32 +676,37 @@ public float getProgress() throws IOException {
             }
             for (Map.Entry<KeyExtent,List<Range>> extentRanges : tserverBin.getValue().entrySet())
{
               Range ke = extentRanges.getKey().toDataRange();
    -          for (Range r : extentRanges.getValue()) {
    -            if (autoAdjust) {
    -              // divide ranges into smaller ranges, based on the tablets
    -              RangeInputSplit split = new RangeInputSplit(tableName, tableId, ke.clip(r),
new String[] {location});
    -
    -              split.setOffline(tableConfig.isOfflineScan());
    -              split.setIsolatedScan(tableConfig.shouldUseIsolatedScanners());
    -              split.setUsesLocalIterators(tableConfig.shouldUseLocalIterators());
    -              split.setMockInstance(mockInstance);
    -              split.setFetchedColumns(tableConfig.getFetchedColumns());
    -              split.setPrincipal(principal);
    -              split.setToken(token);
    -              split.setInstanceName(instance.getInstanceName());
    -              split.setZooKeepers(instance.getZooKeepers());
    -              split.setAuths(auths);
    -              split.setIterators(tableConfig.getIterators());
    -              split.setLogLevel(logLevel);
    -
    -              splits.add(split);
    -            } else {
    -              // don't divide ranges
    -              ArrayList<String> locations = splitsToAdd.get(r);
    -              if (locations == null)
    -                locations = new ArrayList<String>(1);
    -              locations.add(location);
    -              splitsToAdd.put(r, locations);
    +          if (batchScan) {
    +            // group ranges by tablet to be read by a BatchScanner
    +            ArrayList<Range> clippedRanges = new ArrayList<Range>();
    +            for(Range r: extentRanges.getValue())
    --- End diff --
    
    The intent of the change is to keep splits local and batch access to them to reduce overhead.
This implies clipping the ranges. The alternative could be a split with two locations that
has 90%+ of ranges on vs the other location. As I understand the scheduler at that point has
no weight for locality preference.
    
    On other hand having option to not clip would make the options more logically consistent
and justify having number of threads per batch reader as configuration. 
    
    I personally think the first option breaks the least eggs, what do you think?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message