hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nagarjuna Vissarapu <nagarjuna.v...@gmail.com>
Subject Re: dynamic partitioning issues in hive 0.13.0
Date Tue, 02 Dec 2014 02:11:45 GMT
Can you please check you mysql service which stores metada
On Dec 2, 2014 4:18 AM, "Viral Bajaria" <viral.bajaria@gmail.com> wrote:

> Hi,
>
> I have been running into 2 issues with dynamic partitioning queries.
>
> The query runs fine and I get errors like:
>
> *Failed with exception
> MetaException(message:javax.jdo.JDODataStoreException: Insert of object
> "org.apache.hadoop.hive.metastore.model.MPartition@5bbd373e" using
> statement "INSERT INTO `PARTITIONS`
> (`PART_ID`,`LAST_ACCESS_TIME`,`TBL_ID`,`SD_ID`,`CREATE_TIME`,`PART_NAME`)
> VALUES (?,?,?,?,?,?)" failed : Deadlock found when trying to get lock; try
> restarting transaction*
>
> Detailed exception: https://gist.github.com/vbajaria/aead8eb4597f31df4807
>
> Why would there be a deadlock on adding a new partition info to the
> metastore ? It seems weird that would even happen given that the partition
> did not even exist prior to this query run.
>
> I am thinking it could be an issue because I have 100s of such queries
> running and the INDEX on that table might be causing the issue ?
>
>
> The 2nd issue is more painful. After the 1st error mentioned above, I have
> a process which figures out what failed and then re-runs the data for that
> month. After the query finishes successfully, it ends up adding only 1
> partition instead of partition for each day in that month. If I run the
> query manually and look at the data returned then it returns data for all
> partitions, yet for some reason dynamic partition does not ad all. The
> query is an "INSERT OVERWRITE" query.
>
> In the end I have to drop all partitions for that month before re-running
> the query and then everything works fine. Without that the partitions that
> were missed on first attempt don't show up.
>
> The missing partitions are in itself a mystery.
>
> Any points on how to debug this ?
>
> Thanks,
> Viral
>
>
>

Mime
View raw message