hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Navis (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-3387) meta data file size exceeds limit
Date Fri, 17 Aug 2012 03:50:37 GMT

    [ https://issues.apache.org/jira/browse/HIVE-3387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13436544#comment-13436544
] 

Navis commented on HIVE-3387:
-----------------------------

Configurations set by "set" command is not propagated to JobConf for MR job. It's just used
inside of hive.

In above case you mentioned, value of "mapreduce.jobtracker.split.metainfo.maxsize" applied
to hadoop is 10M(default) which is 1/10 of your expectation. If you change mapred-site.xml,
it would not occur.

I also think there should be a way to change properties of JobConf. But some permission things
should be preceded before that.
                
> meta data file size exceeds limit
> ---------------------------------
>
>                 Key: HIVE-3387
>                 URL: https://issues.apache.org/jira/browse/HIVE-3387
>             Project: Hive
>          Issue Type: Bug
>    Affects Versions: 0.7.1
>            Reporter: Alexander Alten-Lorenz
>             Fix For: 0.9.1
>
>
> The cause is certainly that we use an array list instead of a set structure in the split
locations API. Looks like a bug in Hive's CombineFileInputFormat.
> Reproduce:
> Set mapreduce.jobtracker.split.metainfo.maxsize=100000000 when submitting the Hive query.
Run a big hive query that write data into a partitioned table. Due to the large number of
splits, you encounter an exception on the job submitted to Hadoop and the exception said:
> meta data size exceeds 100000000.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message