hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Szehon Ho <sze...@cloudera.com>
Subject Re: Review Request 42134: More information to user on GetOperationStatus in Hive Server2 when query is still executing
Date Thu, 03 Mar 2016 00:50:20 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42134/#review121751
-----------------------------------------------------------



This requirement is very similar to webui's display of operations, can you look at QueryDisplay
and see if it can be enhanced and combined (rather than repeating all this container classes).
 That class also gives you concurrency protection for free as well as marking start/end time
for free ..


ql/src/java/org/apache/hadoop/hive/ql/Driver.java (line 2009)
<https://reviews.apache.org/r/42134/#comment183563>

    This looks risky in terms of concurrency.  I dont see much concurrency protection.. Driver
is used by one thread and getStatus is called by another , I believe.



ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java (line 412)
<https://reviews.apache.org/r/42134/#comment183561>

    Is it possible to change Task class so that we just keep around state member variable,
instead of individual booleans for done, queued, etc?
    
    Then it would be the other way, isDone would just return the state == DONE, etc.



service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java (line 132)
<https://reviews.apache.org/r/42134/#comment183559>

    Can we put this under onNewState, check the state if its a RUNNING state and set the time
there, instead of calling everywhere.
    
    Same for markOperationCompletedTime(), check if state is terminal state and set the time
there.
    
    Same comment across all operations..


- Szehon Ho


On March 2, 2016, 11:44 a.m., Akshay Goyal wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42134/
> -----------------------------------------------------------
> 
> (Updated March 2, 2016, 11:44 a.m.)
> 
> 
> Review request for hive and Amareshwari Sriramadasu.
> 
> 
> Bugs: HIVE-4570
>     https://issues.apache.org/jira/browse/HIVE-4570
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> Driver maintains list of running and runnable tasks although that info is not exposed
outside. It's kept locally in the driver's execute method. We can add Driver.getTaskStatuses()
to return status on all tasks (both running and runnable). Similarly, start and completion
times for operations.
> 
> Proposed changes are :
> 
> struct TGetOperationStatusResp {
>   1: required TStatus status
>   2: optional TOperationState operationState
> 
>   // If operationState is ERROR_STATE, then the following fields may be set
>   // sqlState as defined in the ISO/IEF CLI specification
>   3: optional string sqlState
> 
>   // Internal error code
>   4: optional i32 errorCode
> 
>   // Error message
>   5: optional string errorMessage
> 
>   // List of statuses of sub tasks
>   6: optional string taskStatus
> 
>   // When was the operation started
>   7: optional i64 operationStarted
>   // When was the operation completed
>   8: optional i64 operationCompleted
> 
> }
> 
> 
> Diffs
> -----
> 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 3253146 
>   ql/src/java/org/apache/hadoop/hive/ql/TaskStatus.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java e199e5e 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java b184b4e 
>   service-rpc/if/TCLIService.thrift 0aa9d13 
>   service/src/java/org/apache/hive/service/cli/OperationStatus.java e45b828 
>   service/src/java/org/apache/hive/service/cli/operation/GetCatalogsOperation.java 8868ec1

>   service/src/java/org/apache/hive/service/cli/operation/GetColumnsOperation.java 35b6c52

>   service/src/java/org/apache/hive/service/cli/operation/GetFunctionsOperation.java 8db2e62

>   service/src/java/org/apache/hive/service/cli/operation/GetSchemasOperation.java d6f6280

>   service/src/java/org/apache/hive/service/cli/operation/GetTableTypesOperation.java
a09b39a 
>   service/src/java/org/apache/hive/service/cli/operation/GetTablesOperation.java 740b851

>   service/src/java/org/apache/hive/service/cli/operation/GetTypeInfoOperation.java 2a0fec2

>   service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java f5a9771

>   service/src/java/org/apache/hive/service/cli/operation/Operation.java 22f725c 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 100dc6a 
>   service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 8dff264 
>   service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java 5f01165

>   service/src/test/org/apache/hive/service/cli/CLIServiceTest.java e78181a 
> 
> Diff: https://reviews.apache.org/r/42134/diff/
> 
> 
> Testing
> -------
> 
> Test cases added to validate task status and operation start and end times.
> 
> Task status json seen with one stage is: [{"taskId":"Stage-1","externalHandle":"job_local1121055131_0136","taskState":"FINISHED_STATE"}]
> 
> And TaskStatus toString() returns Stage-1/job_local1121055131_0136/FINISHED_STATE for
the same.
> 
> 
> Thanks,
> 
> Akshay Goyal
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message