hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gabor Szadovszky <gabor.szadovs...@cloudera.com>
Subject Re: Review Request 51895: HIVE-14714 - Finishing Hive on Spark causes "java.io.IOException: Stream closed"
Date Thu, 15 Sep 2016 15:27:35 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51895/
-----------------------------------------------------------

(Updated Sept. 15, 2016, 3:27 p.m.)


Review request for hive, Chaoyu Tang, Naveen Gangam, and Barna Zsombor Klara.


Changes
-------

Fix of unit test failure: Previously there was a 10s delay after invoking protocol.endSession.
After my fix there is no such big delay if there are no text printed on the streams of the
process therefore, the unit tests failed as the session was not closed properly between the
tests. Added code to wait for the ending of the session.


Repository: hive-git


Description
-------

HIVE-14714 - Finishing Hive on Spark causes "java.io.IOException: Stream closed"


Diffs (updated)
-----

  spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java e8ca42aa22f0b312e009bea19e39adc8bd31e2b4


Diff: https://reviews.apache.org/r/51895/diff/


Testing
-------

As the modification result is related to logging and the spark job submission it would require
too much efforts to create unit tests.

Tested manually by "highjacking" $SPARK_HOME/bin/spark-submit script to reproduce the following
scenarios:
- The submit process does not exit after the RemoteDriver stopped
  - Generating some output for less time than the actual redirector timeout
  - Generating output for more time than the actual redirector timeout
- The submit process ends properly after the RemoteDriver stopped

Expected behavior: After ending the actual session the client exits immediately (beeline).
All the stdout/stderr of the RemoteDriver are captured properly in the hive.log until the
redirector timeout.


Thanks,

Gabor Szadovszky


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message