hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eugene Koifman" <ekoif...@hortonworks.com>
Subject Re: Review Request 22329: HIVE-7190. WebHCat launcher task failure can cause two concurent user jobs to run
Date Fri, 13 Jun 2014 17:29:39 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22329/#review45630
-----------------------------------------------------------

Ship it!


Ship It!

- Eugene Koifman


On June 12, 2014, 12:04 a.m., Ivan Mitic wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22329/
> -----------------------------------------------------------
> 
> (Updated June 12, 2014, 12:04 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> Approach in the patch is similar to what Oozie does to handle this situation. Specifically,
all child map jobs get tagged with the launcher MR job id. On launcher task restart, launcher
queries RM for the list of jobs that have the tag and kills them. After that it moves on to
start the same child job again. Again, similarly to what Oozie does, a new templeton.job.launch.time
property is introduced that captures the launcher job submit timestamp and later used to reduce
the search window when RM is queried. 
> 
> To validate the patch, you will need to add webhcat shim jars to templeton.libjars as
now webhcat launcher also has a dependency on hadoop shims. 
> 
> I have noticed that in case of the SqoopDelegator webhcat currently does not set the
MR delegation token when optionsFile flag is used. This also creates the problem in this scenario.
This looks like something that should be handled via a separate Jira.
> 
> 
> Diffs
> -----
> 
>   hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/HiveDelegator.java
23b1c4f 
>   hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/JarDelegator.java
41b1dc5 
>   hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/LauncherDelegator.java
04a5c6f 
>   hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/PigDelegator.java
04e061d 
>   hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/SqoopDelegator.java
adcd917 
>   hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/tool/JobSubmissionConstants.java
a6355a6 
>   hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/tool/LaunchMapper.java
556ee62 
>   hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/tool/TempletonUtils.java
fff4b68 
>   hcatalog/webhcat/svr/src/test/java/org/apache/hive/hcatalog/templeton/tool/TestTempletonUtils.java
8b46d38 
>   shims/0.20S/src/main/java/org/apache/hadoop/mapred/WebHCatJTShim20S.java d3552c1 
>   shims/0.23/src/main/java/org/apache/hadoop/mapred/WebHCatJTShim23.java 5a728b2 
>   shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java 299e918 
> 
> Diff: https://reviews.apache.org/r/22329/diff/
> 
> 
> Testing
> -------
> 
> I have validated that MR, Pig and Hive jobs do get tagged appropriately. I have also
validated that previous child jobs do get killed on RM failover/task failure.
> 
> 
> Thanks,
> 
> Ivan Mitic
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message