hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brock Noland" <br...@cloudera.com>
Subject Re: Review Request 26988: HIVE-8568 : Add HS2 API to fetch Job IDs for a given query
Date Sun, 26 Oct 2014 00:37:12 GMT


> On Oct. 26, 2014, 12:32 a.m., Brock Noland wrote:
> > Nice work! This looks great! The only thing I think we need to change is the execution
engine level. It's possible for a single HS2 to have both MR and Tez (and Spark) tasks running.
For example, in seperate sessions I can set the execution engine and run some queries in MR
and some in tez. Thus I think each job id needs a paired execution engine.

Actually I don't think this is true since we are getting the job ids for a specific session
not for all sessions....


- Brock


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26988/#review58542
-----------------------------------------------------------


On Oct. 23, 2014, 6:31 a.m., Mohit Sabharwal wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26988/
> -----------------------------------------------------------
> 
> (Updated Oct. 23, 2014, 6:31 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-8568
>     https://issues.apache.org/jira/browse/HIVE-8568
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> HIVE-8568 : Add HS2 API to fetch Job IDs for a given query
> 
> This patch adds a Thrift call to fetch Job IDs corresponding to all 
> running MR tasks. The response includes a list of job IDs and an
> enum indicating the execution engine. Throws an exception if no
> jobIDs were fetched.
> 
> Currently only returns MR job IDs. Support for Tez job IDs is left 
> as a todo item for a future commit.
> 
> The client may need to make the call multiple times since the jobs
> may not have started running. Exposed RunningJob associated with 
> ExecDriver as a public method, so that the Driver can access the
> jobIDs corresponding to all running tasks.
> 
> 
> Diffs
> -----
> 
>   itests/hive-unit-hadoop2/src/test/java/org/apache/hive/TestThriftGetJobIDs.java PRE-CREATION

>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java e25450531a71ef4ae4c6d9ea1788e618189a17cb

>   ql/src/java/org/apache/hadoop/hive/ql/DriverContext.java c7d3b6652f89cf7b6507f35176962ff3287d112d

>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 4e3df75c614fe7232e670201f2560c7ccd1db41c

>   ql/src/java/org/apache/hadoop/hive/ql/thrift/JobIDSet.java PRE-CREATION 
>   service/if/TCLIService.thrift 4024bb3f412440fb7533f2e2d8ebc9a7cdc0776d 
>   service/src/gen/thrift/gen-cpp/TCLIService.h 030475b25188c5d2494da4de0bd6edc1ae807eca

>   service/src/gen/thrift/gen-cpp/TCLIService.cpp 209ce63ae1ffd593de81e8e0a8e73218afe3cd79

>   service/src/gen/thrift/gen-cpp/TCLIService_server.skeleton.cpp 988bb4c11ddb717f585e0ba2fb4773ec5fff77e6

>   service/src/gen/thrift/gen-cpp/TCLIService_types.h f32dc3c90caedba86d943a9295a2f246a7b0ec90

>   service/src/gen/thrift/gen-cpp/TCLIService_types.cpp 326d25b8b7d814f7bbdfab7dde805be4834493dc

>   service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TCLIService.java
54851b8d513179e3618ee5a974941bb6a72378b6 
>   service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TExecutionEngine.java
PRE-CREATION 
>   service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetJobIDsReq.java
PRE-CREATION 
>   service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetJobIDsResp.java
PRE-CREATION 
>   service/src/gen/thrift/gen-php/TCLIService.php d2462967c4ee40c46bdeb6c8e24e22e63f3567e3

>   service/src/gen/thrift/gen-py/TCLIService/TCLIService-remote f6ff43f021524adb1a179595bfdd9260e98bff28

>   service/src/gen/thrift/gen-py/TCLIService/TCLIService.py ebc65746ccd7e58c0878f426b429274d4b59ed0b

>   service/src/gen/thrift/gen-py/TCLIService/ttypes.py 6cd64d0386f1e7c73eee7d9868c387c4942a5f9f

>   service/src/gen/thrift/gen-rb/t_c_l_i_service.rb fd1ca9aa13f3db170caf310ebb1ee1bac9f70b63

>   service/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb c731544888f7480b0b1af70440ce1697e8597c12

>   service/src/java/org/apache/hive/service/cli/CLIService.java f5751f1305d7dd4c1f74af5a3a4f94f018b7a38f

>   service/src/java/org/apache/hive/service/cli/CLIServiceClient.java 3155c238ff688bfea16b0aaeea950599bb659b5b

>   service/src/java/org/apache/hive/service/cli/EmbeddedCLIServiceClient.java 9cad5be198c063115a8e90c67b1c2fd910ca8bc6

>   service/src/java/org/apache/hive/service/cli/ICLIService.java c9cc1f4da56f1cd10f6348ea2b9e17e203b87664

>   service/src/java/org/apache/hive/service/cli/operation/Operation.java acb95cb015395f4a1a9280c3b0c719228e584df7

>   service/src/java/org/apache/hive/service/cli/operation/OperationManager.java a57b6e5d322ac312636c19633cee44f711b653df

>   service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 8cabf7ee2945296774d31925a2bce46a7320d668

>   service/src/java/org/apache/hive/service/cli/session/HiveSession.java 6359a5b879928e8726017520f9a733d6b11decd4

>   service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java fa28a6b6a4acb61d8b442ed13b0421e1e0f13368

>   service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java a0a6e183bbd05cd61ba97f187d66b286c145969c

>   service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java 1af45398b895cd7616c5627d318422e14b81e734

>   service/src/test/org/apache/hive/service/cli/thrift/ThriftCLIServiceTest.java 630cfc9124abf7a8871b613b967141d0447eb18e

> 
> Diff: https://reviews.apache.org/r/26988/diff/
> 
> 
> Testing
> -------
> 
> Added unit test that issues async execute statements and follows
> it up with getJobID calls.
> 
> Did not add to ThriftCLIServiceTest since test needs a miniMR cluster
> to run jobs that generate jobIDs.
> 
> 
> Thanks,
> 
> Mohit Sabharwal
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message