Mailing-List: contact dev-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hive.apache.org
Content-Type: multipart/alternative;
 boundary="===============0650156353740282261=="
MIME-Version: 1.0
Subject: Re: Review Request 58085: HIVE-16061: Some of console output is not
 printed to the beeline console
From: Peter Vary <pvary@cloudera.com>
To: Vihang Karajgaonkar <vihang@cloudera.com>, hive <dev@hive.apache.org>,
 Aihua Xu <axu@cloudera.com>, Peter Vary <pvary@cloudera.com>
Date: Fri, 31 Mar 2017 08:28:59 -0000
Message-ID: <20170331082859.26975.60040@reviews-vm2.apache.org>
Auto-Submitted: auto-generated
Sender: Peter Vary <noreply@reviews.apache.org>
References: <20170330204143.31069.6304@reviews-vm2.apache.org>
In-Reply-To: <20170330204143.31069.6304@reviews-vm2.apache.org>
Reply-To: Peter Vary <pvary@cloudera.com>
archived-at: Fri, 31 Mar 2017 08:29:06 -0000

--===============0650156353740282261==
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit


> On March 30, 2017, 8:41 p.m., Aihua Xu wrote:
> > common/src/java/org/apache/hadoop/hive/common/LogUtils.java
> > Lines 52 (patched)
> > <https://reviews.apache.org/r/58085/diff/1/?file=1681072#file1681072line52>
> >
> >     Any suggestions? 
> >     
> >     Seems that is what it does. Can't think of anything else.

// Constants of the key strings for the log4j MDC.

Maybe?


> On March 30, 2017, 8:41 p.m., Aihua Xu wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/Driver.java
> > Lines 1376-1380 (original)
> > <https://reviews.apache.org/r/58085/diff/1/?file=1681073#file1681073line1376>
> >
> >     We were using OperationLog incorrectly before. This log should not be for direct write, but we should be able to write to it through LOG.info/debug(). As you can see, there is already LOG.debug("Waiting to acquire compile lock: " + command); above.
> >     
> >     With this patch, it will write this message at DEBUG level. 
> >     
> >     The loglevels in operationLog (execution, verbose) are actually to control which classes are allowed to output logs. It's very confusion though.

When we used OperationLog for direct write, we might thought that there are some messages we would like to show the user regardless of the log settings. Like the MR deprecation warning. I am not saying that it was good, merely stating that it was the case, and we will not be able to archive this after this change :D


I was thinking about HiveCLI/BeeLine compatibility.
HiveCLI uses SessionState._console, to print out information to the user. There are some cases when we use _console.getInfoStream() to write directly to the HiveCLI console. Also there are some places where we use SessionState.out to print a message to the HiveCLI user. I am not sure how can print these messages to the BeeLine user, when we do not want to show the other log messages of the given class to him.


> On March 30, 2017, 8:41 p.m., Aihua Xu wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/log/LogDivertAppender.java
> > Lines 209 (patched)
> > <https://reviews.apache.org/r/58085/diff/1/?file=1681077#file1681077line209>
> >
> >     Not really. HIVE_SERVER2_LOGGING_OPERATION_LOG_LOCATION is used to configure logLocation here (the base location). So it's still used.

Yeah, you are right. I missed that :)


- Peter


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58085/#review170638
-----------------------------------------------------------


On March 30, 2017, 9:01 p.m., Aihua Xu wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58085/
> -----------------------------------------------------------
> 
> (Updated March 30, 2017, 9:01 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: hive-16061
>     https://issues.apache.org/jira/browse/hive-16061
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> HIVE-16061: Some of console output is not printed to the beeline console
> 
> 
> Diffs
> -----
> 
>   common/src/java/org/apache/hadoop/hive/common/LogUtils.java 01b2e7c 
>   itests/hive-unit/src/test/java/org/apache/hive/service/cli/operation/TestOperationLoggingLayout.java e344e0f 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java d981119 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/TaskRunner.java a596e92 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 1945163 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java 591ea97 
>   ql/src/java/org/apache/hadoop/hive/ql/log/LogDivertAppender.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java 08d0544 
>   ql/src/java/org/apache/hadoop/hive/ql/session/OperationLog.java 18216f2 
>   ql/src/test/results/clientpositive/beeline/drop_with_concurrency.q.out 993329e 
>   ql/src/test/results/clientpositive/beeline/escape_comments.q.out 2a05e53 
>   service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java 8f08c2e 
>   service/src/java/org/apache/hive/service/cli/operation/LogDivertAppender.java eaf1acb 
>   service/src/java/org/apache/hive/service/cli/operation/Operation.java 11a820f 
>   service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 3f8f68e 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java f41092e 
> 
> 
> Diff: https://reviews.apache.org/r/58085/diff/2/
> 
> 
> Testing
> -------
> 
> Test is done locally. You can see the beeline output now. 
> 
> 0: jdbc:hive2://localhost:10000> select t1.key from src t1 join src t2 on t1.key=t2.key limit 10;
> INFO  : Compiling command(queryId=axu_20170330125216_0de3bbf7-60f5-476d-b7eb-9861891d2961): select t1.key from src t1 join src t2 on t1.key=t2.key limit 10
> INFO  : Semantic Analysis Completed
> INFO  : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:t1.key, type:string, comment:null)], properties:null)
> INFO  : Completed compiling command(queryId=axu_20170330125216_0de3bbf7-60f5-476d-b7eb-9861891d2961); Time taken: 0.219 seconds
> INFO  : Concurrency mode is disabled, not creating a lock manager
> INFO  : Executing command(queryId=axu_20170330125216_0de3bbf7-60f5-476d-b7eb-9861891d2961): select t1.key from src t1 join src t2 on t1.key=t2.key limit 10
> WARN  : Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
> INFO  : Query ID = axu_20170330125216_0de3bbf7-60f5-476d-b7eb-9861891d2961
> INFO  : Total jobs = 1
> INFO  : Starting task [Stage-4:MAPREDLOCAL] in serial mode
> INFO  : Execution completed successfully
> INFO  : MapredLocal task succeeded
> INFO  : Launching Job 1 out of 1
> INFO  : Starting task [Stage-3:MAPRED] in serial mode
> INFO  : Number of reduce tasks is set to 0 since there's no reduce operator
> INFO  : Starting Job = job_local1894165710_0002, Tracking URL = http://localhost:8080/
> INFO  : Kill Command = /Users/axu/Documents/workspaces/tools/hadoop/hadoop-2.6.0/bin/hadoop job  -kill job_local1894165710_0002
> INFO  : Hadoop job information for Stage-3: number of mappers: 0; number of reducers: 0
> INFO  : 2017-03-30 12:52:21,788 Stage-3 map = 0%,  reduce = 0%
> ERROR : Ended Job = job_local1894165710_0002 with errors
> ERROR : FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
> INFO  : MapReduce Jobs Launched: 
> INFO  : Stage-Stage-3:  HDFS Read: 0 HDFS Write: 0 FAIL
> INFO  : Total MapReduce CPU Time Spent: 0 msec
> 
> 
> Thanks,
> 
> Aihua Xu
> 
>


--===============0650156353740282261==--