Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id D1B70200D36 for ; Mon, 6 Nov 2017 13:50:00 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id D02F6160BEC; Mon, 6 Nov 2017 12:50:00 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 468BB1609E0 for ; Mon, 6 Nov 2017 13:50:00 +0100 (CET) Received: (qmail 47552 invoked by uid 500); 6 Nov 2017 12:49:59 -0000 Mailing-List: contact issues-help@carbondata.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@carbondata.apache.org Delivered-To: mailing list issues@carbondata.apache.org Received: (qmail 47543 invoked by uid 99); 6 Nov 2017 12:49:59 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Nov 2017 12:49:59 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 5B17FDFBC7; Mon, 6 Nov 2017 12:49:59 +0000 (UTC) From: jackylk To: issues@carbondata.apache.org Reply-To: issues@carbondata.apache.org References: In-Reply-To: Subject: [GitHub] carbondata pull request #1470: [CARBONDATA-1572] Support streaming ingest an... Content-Type: text/plain Message-Id: <20171106124959.5B17FDFBC7@git1-us-west.apache.org> Date: Mon, 6 Nov 2017 12:49:59 +0000 (UTC) archived-at: Mon, 06 Nov 2017 12:50:01 -0000 Github user jackylk commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1470#discussion_r149070335 --- Diff: integration/spark-common/src/main/scala/org/apache/carbondata/spark/rdd/CarbonScanRDD.scala --- @@ -210,8 +247,18 @@ class CarbonScanRDD( inputMetricsStats.initBytesReadCallback(context, inputSplit) val iterator = if (inputSplit.getAllSplits.size() > 0) { val model = format.getQueryModel(inputSplit, attemptContext) - val reader = { - if (vectorReader) { + val reader: RecordReader[Void, Object] = { + if (inputSplit.isStream) { + DataTypeUtil.setDataTypeConverter(new SparkDataTypeConverterImpl) + val inputFormat = new CarbonStreamInputFormat + val streamReader = inputFormat.createRecordReader(inputSplit, attemptContext) + .asInstanceOf[CarbonStreamRecordReader] + streamReader.setVectorReader(vectorReader) + model.setStatisticsRecorder( + CarbonTimeStatisticsFactory.createExecutorRecorder(model.getQueryId)) + streamReader.setQueryModel(model) --- End diff -- It is better to put `model` in constructor of `CarbonStreamRecordReader` ---