Mailing-List: contact dev-help@drill.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@drill.apache.org
From: Mehant Baid <baid.mehant@gmail.com>
Subject: Moving directory based pruning to fire earlier
To: dev@drill.apache.org
Message-ID: <56535BF8.8010208@gmail.com>
Date: Mon, 23 Nov 2015 10:33:28 -0800
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:38.0)
 Gecko/20100101 Thunderbird/38.3.0
MIME-Version: 1.0
Content-Type: multipart/alternative;
 boundary="------------040400040604030304030505"

--------------040400040604030304030505
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit

As part of DRILL-3996 <https://issues.apache.org/jira/browse/DRILL-3996> 
Jinfeng mentioned that he plans to move the directory based pruning rule 
earlier than column based pruning. I want to expand on that a little, 
provide the motivation and gather thoughts/ feedback.

Currently both the directory based pruning and the column based pruning 
is fired in the same planning phase and are based on Drill logical rels. 
This is not optimal in the case where data is organized in such a way 
that both directory based pruning and column based pruning can be 
applied (when the data is organized with a nested directory structure 
plus the individual files contain partition columns). As part of 
creating the Drill logical scan we read the footers of all the files 
involved. If the directory based pruning rule is fired earlier (rule to 
fire based on calcite logical rels) then we will be able to prune out 
unnecessary directories and save the work of reading the footers of 
these files.

Thanks
Mehant


--------------040400040604030304030505--