drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From parthchandra <...@git.apache.org>
Subject [GitHub] drill pull request #723: DRILL-5207: Improve Parquet Scan pipelining.
Date Sat, 28 Jan 2017 00:44:01 GMT
Github user parthchandra commented on a diff in the pull request:

    https://github.com/apache/drill/pull/723#discussion_r98271891
  
    --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/columnreaders/AsyncPageReader.java
---
    @@ -282,89 +356,119 @@ public synchronized void setValuesRead(long valuesRead) {
           this.valuesRead = valuesRead;
         }
     
    -    public long getDiskScanTime() {
    +    public synchronized long getDiskScanTime() {
           return diskScanTime;
         }
     
    -    public void setDiskScanTime(long diskScanTime) {
    +    public synchronized void setDiskScanTime(long diskScanTime) {
           this.diskScanTime = diskScanTime;
         }
    -  }
     
    +  }
     
    -  private class AsyncPageReaderTask implements Callable<ReadStatus> {
    +  private class AsyncPageReaderTask implements Callable<Boolean> {
     
         private final AsyncPageReader parent = AsyncPageReader.this;
    +    private final LinkedBlockingQueue<ReadStatus> queue;
    +    private final String name;
     
    -    public AsyncPageReaderTask() {
    +    public AsyncPageReaderTask(String name, LinkedBlockingQueue<ReadStatus> queue)
{
    +      this.name = name;
    +      this.queue = queue;
         }
     
    -    @Override public ReadStatus call() throws IOException {
    +    @Override
    +    public Boolean call() throws IOException {
           ReadStatus readStatus = new ReadStatus();
     
    -      String oldname = Thread.currentThread().getName();
    -      String name = parent.parentColumnReader.columnChunkMetaData.toString();
    -      Thread.currentThread().setName(name);
    -
           long bytesRead = 0;
           long valuesRead = 0;
    +      final long totalValuesRead = parent.totalPageValuesRead;
           Stopwatch timer = Stopwatch.createStarted();
     
    +      final long totalValuesCount = parent.parentColumnReader.columnChunkMetaData.getValueCount();
    +
    +      // if we are done, just put a marker object in the queue and we are done.
    +      logger.trace("[{}]: Total Values COUNT {}  Total Values READ {} ", name, totalValuesCount,
totalValuesRead);
    +      if (totalValuesRead >= totalValuesCount) {
    +        try {
    +          queue.put(ReadStatus.EMPTY);
    +        } catch (InterruptedException e) {
    +          Thread.currentThread().interrupt();
    +          // Do nothing.
    +        }
    +        return true;
    +      }
    +
           DrillBuf pageData = null;
    +      timer.reset();
           try {
             long s = parent.dataReader.getPos();
             PageHeader pageHeader = Util.readPageHeader(parent.dataReader);
    -        long e = parent.dataReader.getPos();
    -        if (logger.isTraceEnabled()) {
    -          logger.trace("[{}]: Read Page Header : ReadPos = {} : Bytes Read = {} ", name,
s, e - s);
    -        }
    +        //long e = parent.dataReader.getPos();
    --- End diff --
    
    I kept it there because it is useful to debug. I commented it out because I added logging
at the trace level for performance measurement and this was creating a lot of noise in the
logs. I'd like to keep it around in case we ever need it again.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message