hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Xuefu Zhang <xzh...@cloudera.com>
Subject Re: Review Request 58865: HIVE-16552: Limit the number of tasks a Spark job may contain
Date Wed, 03 May 2017 04:29:19 GMT


> On May 3, 2017, 3:35 a.m., Rui Li wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkTask.java
> > Lines 132 (patched)
> > <https://reviews.apache.org/r/58865/diff/3/?file=1705971#file1705971line132>
> >
> >     I think the log is unnecessary because the failure should already be logged
in the monitor

This is not new code.


> On May 3, 2017, 3:35 a.m., Rui Li wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkTask.java
> > Lines 135 (patched)
> > <https://reviews.apache.org/r/58865/diff/3/?file=1705971#file1705971line135>
> >
> >     Same as above. Can we consolidate the logs a bit?

Jobmonitor prints it on console, while the log here is written to hive.log.


> On May 3, 2017, 3:35 a.m., Rui Li wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/RemoteSparkJobMonitor.java
> > Lines 104 (patched)
> > <https://reviews.apache.org/r/58865/diff/3/?file=1705972#file1705972line104>
> >
> >     Maybe I was being misleading. I mean we can compute the total task only once
when the job first reaches RUNNING state, i.e. in the "if (!running)". At this point, the
total count is determined and won't change.

Yeah. However, I'd like to keep the state transition to running first before breaking up and
returning rc=4. In fact, if we lose the transition, Hive actually goes into an instable state.
What you said was what I tried in first place.


- Xuefu


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58865/#review173689
-----------------------------------------------------------


On May 2, 2017, 6:49 p.m., Xuefu Zhang wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58865/
> -----------------------------------------------------------
> 
> (Updated May 2, 2017, 6:49 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-16552
>     https://issues.apache.org/jira/browse/HIVE-16552
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> See JIRA description
> 
> 
> Diffs
> -----
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 84398c6 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkTask.java 32a7730 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/RemoteSparkJobMonitor.java
dd73f3e 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/SparkJobMonitor.java 0b224f2

> 
> 
> Diff: https://reviews.apache.org/r/58865/diff/3/
> 
> 
> Testing
> -------
> 
> Test locally
> 
> 
> Thanks,
> 
> Xuefu Zhang
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message