Mailing-List: contact reviews-help@impala.incubator.apache.org; run by ezmlm
Precedence: bulk
Message-Id: <201702230030.v1N0U3JM017293@ip-10-146-233-104.ec2.internal>
Date: Thu, 23 Feb 2017 00:30:03 +0000
From: "Alex Behm (Code Review)" <gerrit@cloudera.org>
To: impala-cr@cloudera.com, reviews@impala.incubator.apache.org
CC: Marcel Kornacker <marcel@cloudera.com>
Reply-To: alex.behm@cloudera.com
Subject: =?UTF-8?Q?=5BImpala-ASF-CR=5D_IMPALA-3905=3A_Implements_HdfsScanner=3A=3AGetNext=28=29_for_text_scans=2E=0A?=
In-Reply-To: <gerrit.1487090124000.Id193aa223434d7cc40061a42f81bbb29dcd0404b@gerrit.cloudera.org>
References: <gerrit.1487090124000.Id193aa223434d7cc40061a42f81bbb29dcd0404b@gerrit.cloudera.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Content-Disposition: inline
User-Agent: Gerrit/2.12.7
archived-at: Thu, 23 Feb 2017 00:30:07 -0000

Alex Behm has posted comments on this change.

Change subject: IMPALA-3905: Implements HdfsScanner::GetNext() for text scans.
......................................................................


Patch Set 2:

(12 comments)

http://gerrit.cloudera.org:8080/#/c/6000/2/be/src/exec/hdfs-text-scanner.cc
File be/src/exec/hdfs-text-scanner.cc:

Line 64:       batch_start_ptr_(NULL),
> use nullptr throughout
Done


Line 273:   MemPool* pool = row_batch->tuple_data_pool();
> please consider getting rid of this variable and referring to the rowbatch 
Done


Line 440:       eos_ = true;
> advance scan_state_?
Done. I think it makes sense to not allow GetNext() once eos() is true because there does not
seem to be a real use case for allowing that, and disallowing it helps simplify the logic in GetNext().
Imo, we should have that contract for ExecNode::GetNext() as well, but that's another story.


Line 445:         << "FE should have generated SNAPPY_BLOCKED instead.";
> why not check this when creating the stream?
Good idea. Done.


Line 446:     RETURN_IF_ERROR(UpdateDecompressor(stream_->file_desc()->file_compression));
> why is this needed, given that we just started (ie, why not move it into in
Moved into InitNewRange()


Line 459:     eos_ = true;
> scan_state_ transition?
Done


Line 469: Status HdfsTextScanner::CommitRows(RowBatch* row_batch, int num_rows) {
> how about coalescing that with hdfsscannode::commitrows? at the very least 
The main problem is that the HdfsScanner::CommitRows() may add the batch to the row-batch queue and allocate a new batch. But for the new scanners we want to manually add batches to the queue after getting another batch with GetNext().

I merged this CommitRows() with HdfsScanner::CommitRows() and added a flag to control the queue-adding behavior. It's slightly awkward, let me know if you have a better idea.


Line 485:     ExprContext::FreeLocalAllocations(entry.second);
> why call this here?
Added a comment. Did you have place in mind? We can probably move it somewhere else, but not sure what side-effects it may have.


http://gerrit.cloudera.org:8080/#/c/6000/2/be/src/exec/hdfs-text-scanner.h
File be/src/exec/hdfs-text-scanner.h:

Line 119:   /// regardless if whether it passed the conjuncts.
> regardless of whether
Done


Line 124:   /// Only valid to call in scan state FIRST_TUPLE_FOUND or PAST_SCAN_RANGE.
> thanks for introducing that state variable, it's much clearer than a bunch 
Thanks!


Line 135:   Status CommitRows(RowBatch* row_batch, int num_rows);
> in/out params go last
Done


Line 140:   /// otherwise it will just read num_bytes. If we are reading compressed text, num_bytes
> if we are reading compressed text, should this even be called (or why is th
Added comment.

This functions internally handles uncompressed text, streaming compressed text (e.g., gzip), and non-streaming compressed text. See FillByteBufferCompressedFile() and FillByteBufferCompressedStream().

In addition, this function is overridden by the LZO scanner to reuse much of the scanning logic in here.


-- 
To view, visit http://gerrit.cloudera.org:8080/6000
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Id193aa223434d7cc40061a42f81bbb29dcd0404b
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Alex Behm <alex.behm@cloudera.com>
Gerrit-Reviewer: Alex Behm <alex.behm@cloudera.com>
Gerrit-Reviewer: Marcel Kornacker <marcel@cloudera.com>
Gerrit-HasComments: Yes