Mailing-List: contact dev-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hive.apache.org
Date: Mon, 6 Jan 2014 19:24:51 +0000 (UTC)
From: "Eric Chu (JIRA)" <jira@apache.org>
To: hive-dev@hadoop.apache.org
Message-ID: <JIRA.12687216.1388772478050.48746.1389036291826@arcas>
In-Reply-To: <JIRA.12687216.1388772478050@arcas>
References: <JIRA.12687216.1388772478050@arcas>
Subject: [jira] [Commented] (HIVE-6134) Merging small files based on file
 size only works for CTAS queries
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HIVE-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13863292#comment-13863292 ] 

Eric Chu commented on HIVE-6134:
--------------------------------

Thanks [~ashutoshc] for pointing out the concatenate command. However, I think the ability to merge files for a table partition is orthogonal to supporting hive.merge.mapfiles, hive.merge.mapredfiles, and hive.merge.smallfiles.avgsize for "regular" queries (i.e., that don't result in a new table). Even if we have the optimal number of files at input for each partition, users querying over a large number of partitions with just SELECT FROM WHERE clauses will result in a large number of small output files, and there will be negative sides effects such as Hue timeout, the next job will have a large number of mappers, etc.

Can someone explain why the properties are supported only for queries with move tasks? Was it just a matter of scoping, or is there some reason that makes this inappropriate for queries without a move task? We are considering adding this support on our own and would like to get some insights on the original design considerations. Thanks!


> Merging small files based on file size only works for CTAS queries
> ------------------------------------------------------------------
>
>                 Key: HIVE-6134
>                 URL: https://issues.apache.org/jira/browse/HIVE-6134
>             Project: Hive
>          Issue Type: Bug
>    Affects Versions: 0.8.0, 0.10.0, 0.11.0, 0.12.0
>            Reporter: Eric Chu
>
> According to the documentation, if we set hive.merge.mapfiles to true, Hive will launch an additional MR job to merge the small output files at the end of a map-only job when the average output file size is smaller than hive.merge.smallfiles.avgsize. Similarly, by setting hive.merge.mapredfiles to true, Hive will merge the output files of a map-reduce job. 
> My expectation is that this is true for all MR queries. However, my observation is that this is only true for CTAS queries. In GenMRFileSink1.java, HIVEMERGEMAPFILES and HIVEMERGEMAPREDFILES are only used if ((ctx.getMvTask() != null) && (!ctx.getMvTask().isEmpty())). So, for a regular SELECT query that doesn't have move tasks, these properties are not used.
> Is my understanding correct and if so, what's the reasoning behind the logic of not supporting this for regular SELECT queries? It seems to me that this should be supported for regular SELECT queries as well. One scenario where this hits us hard is when users try to download the result in HUE, and HUE times out b/c there are thousands of output files. The workaround is to re-run the query as CTAS, but it's a significant time sink.


--
This message was sent by Atlassian JIRA
(v6.1.5#6160)