hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gabor Szadovszky <gabor.szadovs...@cloudera.com>
Subject Re: Review Request 51895: HIVE-14714 - Finishing Hive on Spark causes "java.io.IOException: Stream closed"
Date Thu, 15 Sep 2016 09:41:04 GMT


> On Sept. 15, 2016, 9:20 a.m., Barna Zsombor Klara wrote:
> > spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java, line
687
> > <https://reviews.apache.org/r/51895/diff/1/?file=1498559#file1498559line687>
> >
> >     Since we have 2 redirectors maybe also log out which one we are in.

The log contains the name of the actual thread (stdout-redir-1 or stderr-redir-1).


> On Sept. 15, 2016, 9:20 a.m., Barna Zsombor Klara wrote:
> > spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java, line
695
> > <https://reviews.apache.org/r/51895/diff/1/?file=1498559#file1498559line695>
> >
> >     Since we have 2 redirectors maybe also log out which one we are in.

The log contains the name of the actual thread (stdout-redir-1 or stderr-redir-1).


> On Sept. 15, 2016, 9:20 a.m., Barna Zsombor Klara wrote:
> > spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java, line
708
> > <https://reviews.apache.org/r/51895/diff/1/?file=1498559#file1498559line708>
> >
> >     Wouldn't lineBuilder.indexOf(String.valueOf('\n')) work as well?

We have to search for '\n' all the time so it might worth having a bit more complex code for
performance. What do you think?


> On Sept. 15, 2016, 9:20 a.m., Barna Zsombor Klara wrote:
> > spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java, line
671
> > <https://reviews.apache.org/r/51895/diff/1/?file=1498559#file1498559line671>
> >
> >     What would happen if the child process is killed while we are inside this while
loop (so after the BufferedReader#ready check)? Wouldn't we get a stream closed exception
on line 674?

It is a good point. I'll add a code part to the catch clause of run() so in any case we will
flush the lines from the buffer.


- Gabor


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51895/#review149035
-----------------------------------------------------------


On Sept. 14, 2016, 4:54 p.m., Gabor Szadovszky wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51895/
> -----------------------------------------------------------
> 
> (Updated Sept. 14, 2016, 4:54 p.m.)
> 
> 
> Review request for hive, Chaoyu Tang, Naveen Gangam, and Barna Zsombor Klara.
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> HIVE-14714 - Finishing Hive on Spark causes "java.io.IOException: Stream closed"
> 
> 
> Diffs
> -----
> 
>   spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java e8ca42aa22f0b312e009bea19e39adc8bd31e2b4

> 
> Diff: https://reviews.apache.org/r/51895/diff/
> 
> 
> Testing
> -------
> 
> As the modification result is related to logging and the spark job submission it would
require too much efforts to create unit tests.
> 
> Tested manually by "highjacking" $SPARK_HOME/bin/spark-submit script to reproduce the
following scenarios:
> - The submit process does not exit after the RemoteDriver stopped
>   - Generating some output for less time than the actual redirector timeout
>   - Generating output for more time than the actual redirector timeout
> - The submit process ends properly after the RemoteDriver stopped
> 
> Expected behavior: After ending the actual session the client exits immediately (beeline).
All the stdout/stderr of the RemoteDriver are captured properly in the hive.log until the
redirector timeout.
> 
> 
> Thanks,
> 
> Gabor Szadovszky
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message