hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aihua Xu <...@cloudera.com>
Subject Re: Review Request 58456: Query cancel: improve the way to handle files
Date Tue, 18 Apr 2017 14:56:48 GMT


> On April 14, 2017, 5:43 p.m., Aihua Xu wrote:
> > service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java
> > Lines 399 (patched)
> > <https://reviews.apache.org/r/58456/diff/1/?file=1692689#file1692689line399>
> >
> >     I'm not exactly following what we are doing here. Not sure how background thread
gets closed later.
> >     
> >     Otherwise, the other changes look good.
> 
> Yongzhi Chen wrote:
>     The background thread will complete the task or gracefully closed with the guidance
of the cancel status. Our current cancel design majorly follows the pattern that cancel command
set the cancel status, the working thread(background thread) check the cancel status and decide
to quit or continue. The backgroundHandle.cancel(true) does not follow the pattern and cause
some conflicts. The following warning log is caused by this:
>     2017-04-11 09:57:30,727 WARN  org.apache.hadoop.hive.ql.exec.Utilities: [HiveServer2-Background-Pool:
Thread-149]: Failed to clean-up tmp directories.
>     java.io.InterruptedIOException: Call interrupted

I see. Got it.


- Aihua


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58456/#review172009
-----------------------------------------------------------


On April 14, 2017, 1:14 p.m., Yongzhi Chen wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58456/
> -----------------------------------------------------------
> 
> (Updated April 14, 2017, 1:14 p.m.)
> 
> 
> Review request for hive, Aihua Xu, Chaoyu Tang, and Sergio Pena.
> 
> 
> Bugs: HIVE-16426
>     https://issues.apache.org/jira/browse/HIVE-16426
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> 1. Use threadlocal variable to store cancel state to make it is accessible without being
passed around by parameters. 
> 2. Add checkpoints for file operations.
> 3. Remove backgroundHandle.cancel to avoid failed file cleanup because of the interruption.
By what I observed that the method seems not very effective for scheduled operation, for example,
the on going HMS API calls.
> 
> 
> Diffs
> -----
> 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java a80004662068eb2391c0dd7062f77156b222375b

>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java b0657f01d4482dc8bb8dc180e5e7deffbdb533e6

>   ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java 7a113bf8e5c4dd8c2c486741a5ebc7b8940e746b

>   service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 04fc0a17c93120b8f6e6d7c36e4d70631d56baca

> 
> 
> Diff: https://reviews.apache.org/r/58456/diff/1/
> 
> 
> Testing
> -------
> 
> Manually tested.
> 
> 
> Thanks,
> 
> Yongzhi Chen
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message