hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mithun Radhakrishnan (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-6409) FileOutputCommitterContainer::commitJob() cancels delegation tokens too early.
Date Wed, 12 Feb 2014 00:04:19 GMT

     [ https://issues.apache.org/jira/browse/HIVE-6409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Mithun Radhakrishnan updated HIVE-6409:
---------------------------------------

    Attachment: HIVE-6409.trunk.patch

Removed the finally{} block from commitJob(). If commit succeeds, there are no exceptions,
and the tokens are cancelled. If commit fails, the cancel is avoided, and abortJob() takes
care of cancelling the token.

I've also removed an annoying typo. I would also like to remove the following  catch blocks:
{code}
catch(TException e){} 
{code}

TException isA MetaException, so this is redundant. The error handling is identical in both
cases. I'd like to remove this cruft, if it's alright.

> FileOutputCommitterContainer::commitJob() cancels delegation tokens too early.
> ------------------------------------------------------------------------------
>
>                 Key: HIVE-6409
>                 URL: https://issues.apache.org/jira/browse/HIVE-6409
>             Project: Hive
>          Issue Type: Bug
>          Components: HCatalog
>    Affects Versions: 0.12.0, 0.13.0
>            Reporter: Mithun Radhakrishnan
>            Assignee: Mithun Radhakrishnan
>         Attachments: HIVE-6409.trunk.patch
>
>
> When HCatalog's FileOutputCommitterContainer::commitJob() is run, it calls the underlying
OutputCommitter and then attempts to register partitions in HCatalog.
> If the commit fails (for example, because of HIVE-4996), commitJob() cancels delegation
tokens retrieved from HCatalog before the exception is rethrown.
> {code}
> java.io.IOException: java.lang.reflect.InvocationTargetException
>     at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.commitJob(PigOutputCommitter.java:185)
>     at
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:249)
>     at
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:212)
>     at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>     at java.lang.Thread.run(Thread.java:722)
> Caused by: java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>     at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:601)
>     at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.commitJob(PigOutputCommitter.java:183)
>     ... 5 more
> Caused by: org.apache.hcatalog.common.HCatException : 2006 : Error adding
> partition to metastore. Cause :
> MetaException(message:java.lang.RuntimeException: commitTransaction was called
> but openTransactionCalls = 0. This probably indicates that there are unbalanced
> calls to openTransaction/commitTransaction)
>     at
> org.apache.hcatalog.mapreduce.FileOutputCommitterContainer.registerPartitions(FileOutputCommitterContainer.java:712)
> {code}
> The problem is that this happens before abortJob() has had a chance to run, thus yielding
the following error:
> {code}
> MetaException(message:Could not connect to meta store using any of the URIs provided.
Most recent failure: org.apache.thrift.transport.TTransportException: Peer indicated failure:
DIGEST-MD5: IO error acquiring password
> 	at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:190)
> 	at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:266)
> 	at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
> 	at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
> 	at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
> 	at java.security.AccessController.doPrivileged(Native Method)
> ...
> {code}
> I'll have a patch out that only cancels delegation tokens if the commitJob() has succeeded.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message