hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "jiraposter@reviews.apache.org (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-2139) Enables HiveServer to accept -hiveconf option
Date Fri, 15 Jul 2011 23:50:01 GMT

    [ https://issues.apache.org/jira/browse/HIVE-2139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13066312#comment-13066312
] 

jiraposter@reviews.apache.org commented on HIVE-2139:
-----------------------------------------------------



bq.  On 2011-06-27 20:51:48, Carl Steinbach wrote:
bq.  > I ran into some problems compiling this patch. Please verify that it builds using
the ant command 'ant clean package'.

I see. I added a dependency on commons-cli in hive common but didn't specify it explicitly.
I've updated ivy dependencies for this.


bq.  On 2011-06-27 20:51:48, Carl Steinbach wrote:
bq.  > common/src/java/org/apache/hadoop/hive/common/cli/HiveCli.java, line 42
bq.  > <https://reviews.apache.org/r/958/diff/1/?file=21667#file21667line42>
bq.  >
bq.  >     The name of this class is likely to generate confusion. Maybe change it to CommonCliOpts,
or something else?

done


bq.  On 2011-06-27 20:51:48, Carl Steinbach wrote:
bq.  > common/src/java/org/apache/hadoop/hive/common/cli/HiveCli.java, line 29
bq.  > <https://reviews.apache.org/r/958/diff/1/?file=21667#file21667line29>
bq.  >
bq.  >     The HiveMetaStore and HiveServer imports are unnecessary.

weird, "organize imports" adds this each time I run it, I removed it manually and it's fine.


bq.  On 2011-06-27 20:51:48, Carl Steinbach wrote:
bq.  > metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java, line 91
bq.  > <https://reviews.apache.org/r/958/diff/1/?file=21668#file21668line91>
bq.  >
bq.  >     We should try to avoid making the metastore dependent on ql. There's already
an open ticket (HIVE-850) that covers the task of moving SessionState to common. Looks like
now may be a good time to do this.

I've refactored the log initialization code in order to allow reuse (moved into common). Took
a look at doing more like 850, but it looks like that's going to be a much class to tease
apart.


bq.  On 2011-06-27 20:51:48, Carl Steinbach wrote:
bq.  > bin/ext/hiveserver.sh, line 31
bq.  > <https://reviews.apache.org/r/958/diff/1/?file=21665#file21665line31>
bq.  >
bq.  >     Does this mean that $HIVE_PORT takes precedence over another port specified
using the -p switch? If so then I think the reverse makes more sense.
bq.  >     
bq.  >     Also, in order to make the precedence explicit, I think it would be good to
move this logic to the HiveCli class, e.g. explicitly call System.getenv("HIVE_PORT") from
HiveCli.

excellent idea, I didn't think to move the check into the code. done.


bq.  On 2011-06-27 20:51:48, Carl Steinbach wrote:
bq.  > metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java, line 3347
bq.  > <https://reviews.apache.org/r/958/diff/1/?file=21668#file21668line3347>
bq.  >
bq.  >     cli.processHiveConf() copies all of the -hiveconf properties into the list of
SystemProperties, which I agree we want to do before initializing the logging system, but
subsequently we need to make sure that these same key/val properties are are also registered
as HiveConf values so that they have the opportunity to override values specified in hive-default.xml
and hive-site.xml. A similar trick is done for the CLI via the OptionsProcessor process_stage1()
and process_stage2() methods.

I've refactored a bit to allow this. It's tricky given that you need access to HiveConf, which
is buried in a couple places. I like the way it's come out in general, however we can't push
it into common (the shared cli code) given it would add a dependency on hiveconf.


bq.  On 2011-06-27 20:51:48, Carl Steinbach wrote:
bq.  > service/src/java/org/apache/hadoop/hive/service/HiveServer.java, line 76
bq.  > <https://reviews.apache.org/r/958/diff/1/?file=21669#file21669line76>
bq.  >
bq.  >     hive.metastore.server.[min|max].threads already exists. We should add similar
properties for controlling the min/max number of threads for HiveServer.

Happy to do it - but would you mind adding a jira assigned to me? Would like to limit the
scope creep on this one.


bq.  On 2011-06-27 20:51:48, Carl Steinbach wrote:
bq.  > service/src/java/org/apache/hadoop/hive/service/HiveServer.java, line 612
bq.  > <https://reviews.apache.org/r/958/diff/1/?file=21669#file21669line612>
bq.  >
bq.  >     Need to read the -hiveconf properties into the SessionState's HiveConf.

is this a bug in the existing code? -- in HiveServerHandler constructor, both the call to
super and the constructor itself seem to be creating "new HiveConf" object. See how I've addressed
this in the updated patch.


bq.  On 2011-06-27 20:51:48, Carl Steinbach wrote:
bq.  > service/src/java/org/apache/hadoop/hive/service/HiveServer.java, line 616
bq.  > <https://reviews.apache.org/r/958/diff/1/?file=21669#file21669line616>
bq.  >
bq.  >     We should also set options.maxWorkerThreads. Looks like the default value for
Thrift is Integer.MAX_VALUE.

added


- Patrick


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/958/#review919
-----------------------------------------------------------


On 2011-06-24 22:12:48, Patrick Hunt wrote:
bq.  
bq.  -----------------------------------------------------------
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/958/
bq.  -----------------------------------------------------------
bq.  
bq.  (Updated 2011-06-24 22:12:48)
bq.  
bq.  
bq.  Review request for hive and Carl Steinbach.
bq.  
bq.  
bq.  Summary
bq.  -------
bq.  
bq.  This patch updates HiveServer and HiveMetastore to add proper cli handling - similar
to that used in CliDriver (ie GnuParser).
bq.  
bq.  There's a common HiveCli class that's used by both main classes.
bq.  
bq.  I've attempted to make the cli's backward compatible with the prior command line processing.
Notice I've "deprecated" (via warnings, but the code still runs) if the old style CLI usage
is used.
bq.  
bq.  commands such as the following now work as expected:
bq.  
bq.  bin/hive --service hiveserver -t 200 -p 12000 --hiveconf hive.root.logger=DEBUG,console
bq.  
bq.  as does the following which generates usage information:
bq.  
bq.  bin/hive --service hiveserver -h
bq.  
bq.  Note: HiveMetastore as not initializing log4j, I updated the code to do similar to HiveServer
(otw the hiveconf hive.root.logger option above didn't work).
bq.  
bq.  
bq.  This addresses bug HIVE-2139.
bq.      https://issues.apache.org/jira/browse/HIVE-2139
bq.  
bq.  
bq.  Diffs
bq.  -----
bq.  
bq.    bin/ext/hiveserver.sh b5edce4 
bq.    bin/ext/metastore.sh db15f6e 
bq.    common/src/java/org/apache/hadoop/hive/common/cli/HiveCli.java PRE-CREATION 
bq.    metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java bc58bd5 
bq.    service/src/java/org/apache/hadoop/hive/service/HiveServer.java ea04be9 
bq.  
bq.  Diff: https://reviews.apache.org/r/958/diff
bq.  
bq.  
bq.  Testing
bq.  -------
bq.  
bq.  I couldn't find any tests for these changes, so I verfied the changes manually.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Patrick
bq.  
bq.



> Enables HiveServer to accept -hiveconf option
> ---------------------------------------------
>
>                 Key: HIVE-2139
>                 URL: https://issues.apache.org/jira/browse/HIVE-2139
>             Project: Hive
>          Issue Type: Improvement
>          Components: CLI
>         Environment: Linux + CDH3u0 (Hive 0.7.0+27.1-2~lucid-cdh3)
>            Reporter: Kazuki Ohta
>            Assignee: Patrick Hunt
>         Attachments: HIVE-2139.patch
>
>
> Currently, I'm trying to test HiveHBaseIntegration on HiveServer. But it doesn't seem
to accept -hiveconf command.
> {code}
> hive --service hiveserver -hiveconf hbase.zookeeper.quorum=hdp0,hdp1,hdp2
> Starting Hive Thrift Server
> java.lang.NumberFormatException: For input string: "-hiveconf"
>         at java.lang.NumberFormatException.forInputString(NumberFormatException.java:48)
>         at java.lang.Integer.parseInt(Integer.java:449)
>         at java.lang.Integer.parseInt(Integer.java:499)
>         at org.apache.hadoop.hive.service.HiveServer.main(HiveServer.java:382)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
> {code}
> Therefore, you need to throw the query like "set hbase.zookeeper.quorum=hdp0,hdp1,hdp2"
everytime. It's not convenient for separating the configuration between server-side and client-side.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message