hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Peter Vary (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-16357) Failed folder creation when creating a new table is reported incorrectly
Date Tue, 01 Aug 2017 15:29:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-16357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16109081#comment-16109081
] 

Peter Vary commented on HIVE-16357:
-----------------------------------

[~spena]: The original issue is the following with the original code:
- When the table creation is failed
- DBNotificationListener will be triggered with incomplete data
- DBNotificationListener will throw a new exception which will mask the original from the
user, making it harder to find the root problem

The patch fixes it by adding the check inside the DBNotificationListener, thus avoiding throwing
the second exception. Also the patch makes sure that if there is any further exceptions thrown
by the DBNotificationListener, or any other Listener configured by the user, they are catched,
logged and not propagated further.

The rationale behind this solution is the following:
- Listeners can be configured by the user, so they can have different usages. For example
one listener might only collect failed table creation events. In this case the listeners should
be notified on every event, and the listener should decide which event to handle, and which
event to omit.
- DBNotificationListener on the other hand is not interested in the failed events, so it should
skip these events.
- Also we thought, that listeners are configured by the users, so their code can be unstable,
buggy. With this in mind, it would be good be sure that they do not affect each other. So
if there is an error in one listener then the other listeners should still be notified.

We might not be aware of every usage patterns of the Listeners, or might overcomplicate this
Listener architecture. What do you think [~spena]? Is it worth to be prepared to these use-cases?

Thanks,
Peter

> Failed folder creation when creating a new table is reported incorrectly
> ------------------------------------------------------------------------
>
>                 Key: HIVE-16357
>                 URL: https://issues.apache.org/jira/browse/HIVE-16357
>             Project: Hive
>          Issue Type: Bug
>          Components: Metastore
>    Affects Versions: 2.3.0, 3.0.0
>            Reporter: Barna Zsombor Klara
>            Assignee: Barna Zsombor Klara
>            Priority: Minor
>             Fix For: 3.0.0
>
>         Attachments: HIVE-16357.01.patch, HIVE-16357.02.patch, HIVE-16357.03.patch, HIVE-16357.04.patch
>
>
> If the directory for a Hive table could not be created, them the HMS will throw a metaexception:
> {code}
>  if (tblPath != null) {
>           if (!wh.isDir(tblPath)) {
>             if (!wh.mkdirs(tblPath, true)) {
>               throw new MetaException(tblPath
>                   + " is not a directory or unable to create one");
>             }
>             madeDir = true;
>           }
>         }
> {code}
> However in the finally block we always try to call the DbNotificationListener, which
in turn will also throw an exception because the directory is missing, overwriting the initial
exception with a FileNotFoundException.
> Actual stacktrace seen by the caller:
> {code}
> 2017-04-03T05:58:00,128 ERROR [pool-7-thread-2] metastore.RetryingHMSHandler: MetaException(message:java.lang.RuntimeException:
java.io.FileNotFoundException: File file:/.../0 does not exist)
> 	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newMetaException(HiveMetaStore.java:6074)
> 	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:1496)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:498)
> 	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
> 	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
> 	at com.sun.proxy.$Proxy28.create_table_with_environment_context(Unknown Source)
> 	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:11125)
> 	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:11109)
> 	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> 	at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
> 	at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:422)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> 	at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118)
> 	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: File file:/.../0
does not exist
> 	at org.apache.hive.hcatalog.listener.DbNotificationListener$FileIterator.<init>(DbNotificationListener.java:203)
> 	at org.apache.hive.hcatalog.listener.DbNotificationListener.onCreateTable(DbNotificationListener.java:137)
> 	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1463)
> 	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:1482)
> 	... 20 more
> Caused by: java.io.FileNotFoundException: File file:/.../0 does not exist
> 	at org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:429)
> 	at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1515)
> 	at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1555)
> 	at org.apache.hadoop.fs.ChecksumFileSystem.listStatus(ChecksumFileSystem.java:574)
> 	at org.apache.hadoop.fs.FilterFileSystem.listStatus(FilterFileSystem.java:243)
> 	at org.apache.hadoop.fs.ProxyFileSystem.listStatus(ProxyFileSystem.java:195)
> 	at org.apache.hadoop.fs.FilterFileSystem.listStatus(FilterFileSystem.java:243)
> 	at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1515)
> 	at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1555)
> 	at org.apache.hive.hcatalog.listener.DbNotificationListener$FileIterator.<init>(DbNotificationListener.java:200)
> 	... 23 more
> {code}
> The original exception should be thrown back to the caller while any exception from the
finally block should be caught/handled/logged out locally.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message