hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rajat Khandelwal <rajatgupt...@gmail.com>
Subject Re: Review Request 42134: More information to user on GetOperationStatus in Hive Server2 when query is still executing
Date Mon, 07 Mar 2016 14:42:44 GMT


> On March 3, 2016, 6:20 a.m., Szehon Ho wrote:
> > This requirement is very similar to webui's display of operations, can you look
at QueryDisplay and see if it can be enhanced and combined (rather than repeating all this
container classes).  That class also gives you concurrency protection for free as well as
marking start/end time for free ..

Tasks are maintained by the driver, and they are only sent to query display when it is started.
So Query display can not track other states, e.g. QUEUED, INITIALIZED etc. I don't think query
display is the way to solve this. Please suggest some alternative.

I'll be taking up this issue, and hence I've created a new review request. Please review at
https://reviews.apache.org/r/44453/diff/1#index_header


> On March 3, 2016, 6:20 a.m., Szehon Ho wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java, line 412
> > <https://reviews.apache.org/r/42134/diff/5/?file=1276514#file1276514line412>
> >
> >     Is it possible to change Task class so that we just keep around state member
variable, instead of individual booleans for done, queued, etc?
> >     
> >     Then it would be the other way, isDone would just return the state == DONE,
etc.

Fixed.


> On March 3, 2016, 6:20 a.m., Szehon Ho wrote:
> > service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java, line 135
> > <https://reviews.apache.org/r/42134/diff/5/?file=1276527#file1276527line135>
> >
> >     Can we put this under onNewState, check the state if its a RUNNING state and
set the time there, instead of calling everywhere.
> >     
> >     Same for markOperationCompletedTime(), check if state is terminal state and
set the time there.
> >     
> >     Same comment across all operations..

Fixed.


- Rajat


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42134/#review121751
-----------------------------------------------------------


On March 2, 2016, 5:14 p.m., Akshay Goyal wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42134/
> -----------------------------------------------------------
> 
> (Updated March 2, 2016, 5:14 p.m.)
> 
> 
> Review request for hive and Amareshwari Sriramadasu.
> 
> 
> Bugs: HIVE-4570
>     https://issues.apache.org/jira/browse/HIVE-4570
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> Driver maintains list of running and runnable tasks although that info is not exposed
outside. It's kept locally in the driver's execute method. We can add Driver.getTaskStatuses()
to return status on all tasks (both running and runnable). Similarly, start and completion
times for operations.
> 
> Proposed changes are :
> 
> struct TGetOperationStatusResp {
>   1: required TStatus status
>   2: optional TOperationState operationState
> 
>   // If operationState is ERROR_STATE, then the following fields may be set
>   // sqlState as defined in the ISO/IEF CLI specification
>   3: optional string sqlState
> 
>   // Internal error code
>   4: optional i32 errorCode
> 
>   // Error message
>   5: optional string errorMessage
> 
>   // List of statuses of sub tasks
>   6: optional string taskStatus
> 
>   // When was the operation started
>   7: optional i64 operationStarted
>   // When was the operation completed
>   8: optional i64 operationCompleted
> 
> }
> 
> 
> Diffs
> -----
> 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 3253146 
>   ql/src/java/org/apache/hadoop/hive/ql/TaskStatus.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java e199e5e 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java b184b4e 
>   service-rpc/if/TCLIService.thrift 0aa9d13 
>   service/src/java/org/apache/hive/service/cli/OperationStatus.java e45b828 
>   service/src/java/org/apache/hive/service/cli/operation/GetCatalogsOperation.java 8868ec1

>   service/src/java/org/apache/hive/service/cli/operation/GetColumnsOperation.java 35b6c52

>   service/src/java/org/apache/hive/service/cli/operation/GetFunctionsOperation.java 8db2e62

>   service/src/java/org/apache/hive/service/cli/operation/GetSchemasOperation.java d6f6280

>   service/src/java/org/apache/hive/service/cli/operation/GetTableTypesOperation.java
a09b39a 
>   service/src/java/org/apache/hive/service/cli/operation/GetTablesOperation.java 740b851

>   service/src/java/org/apache/hive/service/cli/operation/GetTypeInfoOperation.java 2a0fec2

>   service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java f5a9771

>   service/src/java/org/apache/hive/service/cli/operation/Operation.java 22f725c 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 100dc6a 
>   service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 8dff264 
>   service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java 5f01165

>   service/src/test/org/apache/hive/service/cli/CLIServiceTest.java e78181a 
> 
> Diff: https://reviews.apache.org/r/42134/diff/
> 
> 
> Testing
> -------
> 
> Test cases added to validate task status and operation start and end times.
> 
> Task status json seen with one stage is: [{"taskId":"Stage-1","externalHandle":"job_local1121055131_0136","taskState":"FINISHED_STATE"}]
> 
> And TaskStatus toString() returns Stage-1/job_local1121055131_0136/FINISHED_STATE for
the same.
> 
> 
> Thanks,
> 
> Akshay Goyal
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message