hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aihua Xu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-16426) Query cancel: improve the way to handle files
Date Tue, 18 Apr 2017 14:57:41 GMT

    [ https://issues.apache.org/jira/browse/HIVE-16426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15972854#comment-15972854
] 

Aihua Xu commented on HIVE-16426:
---------------------------------

The change looks good to me. +1

> Query cancel: improve the way to handle files
> ---------------------------------------------
>
>                 Key: HIVE-16426
>                 URL: https://issues.apache.org/jira/browse/HIVE-16426
>             Project: Hive
>          Issue Type: Improvement
>            Reporter: Yongzhi Chen
>            Assignee: Yongzhi Chen
>         Attachments: HIVE-16426.1.patch
>
>
> 1. Add data structure support to make it is easy to check query cancel status.
> 2. Handle query cancel more gracefully. Remove possible file leaks caused by query cancel
as shown in following stack:
> {noformat}
> 2017-04-11 09:57:30,727 WARN  org.apache.hadoop.hive.ql.exec.Utilities: [HiveServer2-Background-Pool:
Thread-149]: Failed to clean-up tmp directories.
> java.io.InterruptedIOException: Call interrupted
>         at org.apache.hadoop.ipc.Client.call(Client.java:1496)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1439)
>         at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
>         at com.sun.proxy.$Proxy20.delete(Unknown Source)
>         at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.delete(ClientNamenodeProtocolTranslatorPB.java:535)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:606)
>         at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256)
>         at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
>         at com.sun.proxy.$Proxy21.delete(Unknown Source)
>         at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:2059)
>         at org.apache.hadoop.hdfs.DistributedFileSystem$13.doCall(DistributedFileSystem.java:675)
>         at org.apache.hadoop.hdfs.DistributedFileSystem$13.doCall(DistributedFileSystem.java:671)
>         at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>         at org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:671)
>         at org.apache.hadoop.hive.ql.exec.Utilities.clearWork(Utilities.java:277)
>         at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:463)
>         at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:142)
>         at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
>         at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
>         at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1978)
>         at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1691)
>         at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1423)
>         at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1207)
>         at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1202)
>         at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:238)
>         at org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:88)
>         at org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:303)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
>         at org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:316)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
> {noformat}
> 3. Add checkpoints to related file operations to improve response time for query cancelling.




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message