Mailing-List: contact issues-help@drill.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@drill.apache.org
Date: Fri, 12 May 2017 22:11:04 +0000 (UTC)
From: "ASF GitHub Bot (JIRA)" <jira@apache.org>
To: issues@drill.apache.org
Message-ID: <JIRA.13068328.1493705857000.201275.1494627064148@Atlassian.JIRA>
In-Reply-To: <JIRA.13068328.1493705857000@Atlassian.JIRA>
References: <JIRA.13068328.1493705857000@Atlassian.JIRA> <JIRA.13068328.1493705857757@jira-lw-us.apache.org>
Subject: [jira] [Commented] (DRILL-5459) Extend physical operator test
 framework to test mini plans consisting of multiple operators
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Fri, 12 May 2017 22:11:08 -0000


    [ https://issues.apache.org/jira/browse/DRILL-5459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16008799#comment-16008799 ] 

ASF GitHub Bot commented on DRILL-5459:
---------------------------------------

Github user jinfengni commented on a diff in the pull request:

    https://github.com/apache/drill/pull/823#discussion_r116333318
  
    --- Diff: exec/java-exec/src/test/java/org/apache/drill/exec/physical/unit/MiniPlanUnitTestBase.java ---
    @@ -0,0 +1,439 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + * <p/>
    + * http://www.apache.org/licenses/LICENSE-2.0
    + * <p/>
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.drill.exec.physical.unit;
    +
    +import com.google.common.base.Preconditions;
    +import com.google.common.collect.Lists;
    +import mockit.NonStrictExpectations;
    +import org.apache.drill.DrillTestWrapper;
    +import org.apache.drill.common.expression.SchemaPath;
    +import org.apache.drill.exec.physical.base.PhysicalOperator;
    +import org.apache.drill.exec.physical.impl.BatchCreator;
    +import org.apache.drill.exec.physical.impl.ScanBatch;
    +import org.apache.drill.exec.record.BatchSchema;
    +import org.apache.drill.exec.record.MaterializedField;
    +import org.apache.drill.exec.record.RecordBatch;
    +import org.apache.drill.exec.record.VectorAccessible;
    +import org.apache.drill.exec.rpc.NamedThreadFactory;
    +import org.apache.drill.exec.store.RecordReader;
    +import org.apache.drill.exec.store.dfs.DrillFileSystem;
    +import org.apache.drill.exec.store.parquet.ParquetDirectByteBufferAllocator;
    +import org.apache.drill.exec.store.parquet.ParquetReaderUtility;
    +import org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader;
    +import org.apache.drill.exec.util.TestUtilities;
    +import org.apache.hadoop.fs.Path;
    +import org.apache.parquet.hadoop.CodecFactory;
    +import org.apache.parquet.hadoop.ParquetFileReader;
    +import org.apache.parquet.hadoop.metadata.ParquetMetadata;
    +
    +import java.util.ArrayList;
    +import java.util.Collections;
    +import java.util.HashMap;
    +import java.util.Iterator;
    +import java.util.List;
    +import java.util.Map;
    +import java.util.concurrent.ExecutorService;
    +import java.util.concurrent.Executors;
    +
    +import static org.apache.drill.exec.physical.unit.TestMiniPlan.fs;
    +
    +/**
    + * A MiniPlanUnitTestBase extends PhysicalOpUnitTestBase, to construct MiniPlan (aka plan fragment).
    + * in the form of physical operator tree, and verify both the expected schema and output row results.
    + * Steps to construct a unit:
    + * 1. Call PopBuilder / ScanPopBuilder to construct the MiniPlan
    + * 2. Create a MiniPlanTestBuilder, and specify the expected schema and base line values, or if there
    + * is no batch expected.
    + */
    +
    +public class MiniPlanUnitTestBase extends PhysicalOpUnitTestBase {
    --- End diff --
    
    Thanks for the suggestion. I'll leave it as future improvement, as it requires refactor the new class as well as the existing physical operator test framework. 


> Extend physical operator test framework to test mini plans consisting of multiple operators
> -------------------------------------------------------------------------------------------
>
>                 Key: DRILL-5459
>                 URL: https://issues.apache.org/jira/browse/DRILL-5459
>             Project: Apache Drill
>          Issue Type: Improvement
>          Components: Tools, Build & Test
>            Reporter: Jinfeng Ni
>            Assignee: Jinfeng Ni
>              Labels: ready-to-commit
>
> DRILL-4437 introduced a unit test framework to test a non-scan physical operator. A JSON reader is implicitly used to specify the inputs to the physical operator under test. 
> There are needs to extend such unit test framework for two scenarios.
> 1. We need a way to test scan operator with different record readers. Drill supports a variety of data source, and it's important to make sure every record reader work properly according to the protocol defined.
> 2. We need a way to test a so-called mini-plan (aka plan fragment) consisting of multiple non-scan operators. 
> For the 2nd need, an alternative is to leverage SQL statement and query planner. However, such approach has a direct dependency on query planner; 1) any planner change may impact the testcase and lead to a different plan, 2) it's not always easy job to force the planner to get a desired plan fragment for testing.
> In particular, it would be good to have a relatively easy way to specify a mini-plan with a couple of targeted physical operators. 
> This JIRA is created to track the work to extend the unit test framework in DRILL-4437.
>  
> Related work: DRILL-5318 introduced a sub-operator test fixture, which mainly targeted to test at sub-operator level. The framework in DRILL-4437 and the extension would focus on operator level, or multiple operator levels, where execution would go through RecordBatch's API call. 
> Same as DRILL-4437, we are going to use mockit to mock required objects such fragment context, operator context etc. 


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)