hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "alanwake (Jira)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-23896) hiveserver2 not listening on any port, am i miss some configurations?
Date Thu, 23 Jul 2020 00:57:00 GMT

     [ https://issues.apache.org/jira/browse/HIVE-23896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

alanwake updated HIVE-23896:
----------------------------
    Description: 
 

 

i try deply hive 3.1.2 on k8s.  it was worked on version 2.3.2.

metastore node and postgres node are ok, but hiveserver look like i miss some important configuration
properties?
{code:java}
 {code}
 

 

 
{code:java}
[root@master hive]# ./get.sh 
NAME                         READY   STATUS    RESTARTS   AGE   IP             NODE      
            NOMINATED NODE   READINESS GATES
hive-7bd48747d4-5zjmh        1/1     Running   0          56s   10.244.3.110   node03.51.local
  <none>           <none>
metastore-66b58f9f76-6wsxj   1/1     Running   0          56s   10.244.3.109   node03.51.local
  <none>           <none>
postgres-57794b99b7-pqxwm    1/1     Running   0          56s   10.244.2.241   node02.51.local
  <none>           <none>NAME        TYPE        CLUSTER-IP       EXTERNAL-IP
  PORT(S)                           AGE   SELECTOR
hive        NodePort    10.108.40.17     <none>        10002:30626/TCP,10000:31845/TCP
  56s   app=hive
metastore   ClusterIP   10.106.159.220   <none>        9083/TCP                    
     56s   app=metastore
postgres    ClusterIP   10.108.85.47     <none>        5432/TCP                    
     56s   app=postgres

{code}
 

 
{code:java}
[root@master hive]# kubectl logs hive-7bd48747d4-5zjmh -n=hive
Configuring core
 - Setting hadoop.proxyuser.hue.hosts=*
 - Setting fs.defaultFS=hdfs://namenode.hadoop.svc.cluster.local:9000
 - Setting hadoop.http.staticuser.user=root
 - Setting hadoop.proxyuser.hue.groups=*
Configuring hdfs
 - Setting dfs.namenode.datanode.registration.ip-hostname-check=false
 - Setting dfs.webhdfs.enabled=true
 - Setting dfs.permissions.enabled=false
Configuring yarn
 - Setting yarn.timeline-service.enabled=true
 - Setting yarn.resourcemanager.system-metrics-publisher.enabled=true
 - Setting yarn.resourcemanager.store.class=org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore
 - Setting yarn.log.server.url=http://historyserver.hadoop.svc.cluster.local:8188/applicationhistory/logs/
 - Setting yarn.resourcemanager.fs.state-store.uri=/rmstate
 - Setting yarn.timeline-service.generic-application-history.enabled=true
 - Setting yarn.log-aggregation-enable=true
 - Setting yarn.resourcemanager.hostname=resourcemanager.hadoop.svc.cluster.local
 - Setting yarn.resourcemanager.resource.tracker.address=resourcemanager.hadoop.svc.cluster.local:8031
 - Setting yarn.timeline-service.hostname=historyserver.hadoop.svc.cluster.local
 - Setting yarn.resourcemanager.scheduler.address=resourcemanager.hadoop.svc.cluster.local:8030
 - Setting yarn.resourcemanager.address=resourcemanager.hadoop.svc.cluster.local:8032
 - Setting yarn.nodemanager.remote-app-log-dir=/app-logs
 - Setting yarn.resourcemanager.recovery.enabled=true
Configuring httpfs
Configuring kms
Configuring mapred
Configuring hive
 - Setting datanucleus.autoCreateSchema=false
 - Setting javax.jdo.option.ConnectionPassword=hive
 - Setting hive.metastore.uris=thrift://metastore:9083
 - Setting javax.jdo.option.ConnectionURL=jdbc:postgresql://metastore/metastore
 - Setting javax.jdo.option.ConnectionUserName=hive
 - Setting javax.jdo.option.ConnectionDriverName=org.postgresql.Driver
Configuring for multihomed network
[1/100] check for metastore:9083...
[1/100] metastore:9083 is not available yet
[1/100] try in 5s once again ...
[2/100] check for metastore:9083...
[2/100] metastore:9083 is not available yet
[2/100] try in 5s once again ...
[3/100] check for metastore:9083...
[3/100] metastore:9083 is not available yet
[3/100] try in 5s once again ...
[4/100] check for metastore:9083...
[4/100] metastore:9083 is not available yet
[4/100] try in 5s once again ...
[5/100] metastore:9083 is available.
mkdir: `/tmp': File exists
2020-07-22 07:15:33: Starting HiveServer2
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop-3.2.1/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Hive Session ID = c86e1a7f-e18c-4c04-a8ba-1179c235fdce
Hive Session ID = 6f036ed3-5041-4479-ad99-eebc193a110a
...
...
...
Hive Session ID = xxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxxxxxxx
# quit after print about 20 lines {code}
 
----
 

 
{code:java}
[root@master hive]# kubectl exec -it hive-7bd48747d4-5zjmh -n=hive /bin/bash
root@hive-7bd48747d4-5zjmh:/opt# ps -ef > 1.txt
root@hive-7bd48747d4-5zjmh:/opt# cat 1.txt 
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 07:15 ?        00:00:00 /bin/sh -c startup.sh
root       269     1  0 07:15 ?        00:00:00 /bin/bash /usr/local/bin/startup.sh
root       471   269  3 07:15 ?        00:00:05 /usr/lib/jvm/java-8-openjdk-amd64//bin/java
-Dproc_jar -Dproc_hiveserver2 -Dlog4j.configurationFile=hive-log4j2.properties -Djava.util.logging.config.file=/opt/hive/conf/parquet-logging.properties
-Dyarn.log.dir=/opt/hadoop-3.2.1/logs -Dyarn.log.file=hadoop.log -Dyarn.home.dir=/opt/hadoop-3.2.1
-Dyarn.root.logger=INFO,console -Djava.library.path=/opt/hadoop-3.2.1/lib/native -Xmx256m
-Dhadoop.log.dir=/opt/hadoop-3.2.1/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/opt/hadoop-3.2.1
-Dhadoop.id.str=root -Dhadoop.root.logger=INFO,console -Dhadoop.policy.file=hadoop-policy.xml
-Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /opt/hive/lib/hive-service-3.1.2.jar
org.apache.hive.service.server.HiveServer2 --hiveconf hive.server2.enable.doAs=false
root       555     0  0 07:17 pts/0    00:00:00 /bin/bash
root       561   555  0 07:18 pts/0    00:00:00 ps -ef

root@hive-7bd48747d4-5zjmh:/opt# netstat -lptn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program
name    
root@hive-7bd48747d4-5zjmh:/opt# 

{code}
 

 

 

  was:
 

 

i try deply hive 3.1.2 on k8s.  it was worked on version 2.3.2.

metastore node and postgres node are ok, but hiveserver look like i miss some important configuration
properties?
{code:java}
 {code}
 

 

 
{code:java}

[root@master hive]# ./get.sh 
NAME                         READY   STATUS    RESTARTS   AGE   IP             NODE      
            NOMINATED NODE   READINESS GATES
hive-7bd48747d4-5zjmh        1/1     Running   0          56s   10.244.3.110   node03.51.local
  <none>           <none>
metastore-66b58f9f76-6wsxj   1/1     Running   0          56s   10.244.3.109   node03.51.local
  <none>           <none>
postgres-57794b99b7-pqxwm    1/1     Running   0          56s   10.244.2.241   node02.51.local
  <none>           <none>NAME        TYPE        CLUSTER-IP       EXTERNAL-IP
  PORT(S)                           AGE   SELECTOR
hive        NodePort    10.108.40.17     <none>        10002:30626/TCP,10000:31845/TCP
  56s   app=hive
metastore   ClusterIP   10.106.159.220   <none>        9083/TCP                    
     56s   app=metastore
postgres    ClusterIP   10.108.85.47     <none>        5432/TCP                    
     56s   app=postgres

{code}
 

 
{code:java}
[root@master hive]# kubectl logs hive-7bd48747d4-5zjmh -n=hive
Configuring core
 - Setting hadoop.proxyuser.hue.hosts=*
 - Setting fs.defaultFS=hdfs://namenode.hadoop.svc.cluster.local:9000
 - Setting hadoop.http.staticuser.user=root
 - Setting hadoop.proxyuser.hue.groups=*
Configuring hdfs
 - Setting dfs.namenode.datanode.registration.ip-hostname-check=false
 - Setting dfs.webhdfs.enabled=true
 - Setting dfs.permissions.enabled=false
Configuring yarn
 - Setting yarn.timeline-service.enabled=true
 - Setting yarn.resourcemanager.system-metrics-publisher.enabled=true
 - Setting yarn.resourcemanager.store.class=org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore
 - Setting yarn.log.server.url=http://historyserver.hadoop.svc.cluster.local:8188/applicationhistory/logs/
 - Setting yarn.resourcemanager.fs.state-store.uri=/rmstate
 - Setting yarn.timeline-service.generic-application-history.enabled=true
 - Setting yarn.log-aggregation-enable=true
 - Setting yarn.resourcemanager.hostname=resourcemanager.hadoop.svc.cluster.local
 - Setting yarn.resourcemanager.resource.tracker.address=resourcemanager.hadoop.svc.cluster.local:8031
 - Setting yarn.timeline-service.hostname=historyserver.hadoop.svc.cluster.local
 - Setting yarn.resourcemanager.scheduler.address=resourcemanager.hadoop.svc.cluster.local:8030
 - Setting yarn.resourcemanager.address=resourcemanager.hadoop.svc.cluster.local:8032
 - Setting yarn.nodemanager.remote-app-log-dir=/app-logs
 - Setting yarn.resourcemanager.recovery.enabled=true
Configuring httpfs
Configuring kms
Configuring mapred
Configuring hive
 - Setting datanucleus.autoCreateSchema=false
 - Setting javax.jdo.option.ConnectionPassword=hive
 - Setting hive.metastore.uris=thrift://metastore:9083
 - Setting javax.jdo.option.ConnectionURL=jdbc:postgresql://metastore/metastore
 - Setting javax.jdo.option.ConnectionUserName=hive
 - Setting javax.jdo.option.ConnectionDriverName=org.postgresql.Driver
Configuring for multihomed network
[1/100] check for metastore:9083...
[1/100] metastore:9083 is not available yet
[1/100] try in 5s once again ...
[2/100] check for metastore:9083...
[2/100] metastore:9083 is not available yet
[2/100] try in 5s once again ...
[3/100] check for metastore:9083...
[3/100] metastore:9083 is not available yet
[3/100] try in 5s once again ...
[4/100] check for metastore:9083...
[4/100] metastore:9083 is not available yet
[4/100] try in 5s once again ...
[5/100] metastore:9083 is available.
mkdir: `/tmp': File exists
2020-07-22 07:15:33: Starting HiveServer2
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop-3.2.1/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Hive Session ID = c86e1a7f-e18c-4c04-a8ba-1179c235fdce
Hive Session ID = 6f036ed3-5041-4479-ad99-eebc193a110a
{code}
 
----
 

 
{code:java}
[root@master hive]# kubectl exec -it hive-7bd48747d4-5zjmh -n=hive /bin/bash
root@hive-7bd48747d4-5zjmh:/opt# ps -ef > 1.txt
root@hive-7bd48747d4-5zjmh:/opt# cat 1.txt 
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 07:15 ?        00:00:00 /bin/sh -c startup.sh
root       269     1  0 07:15 ?        00:00:00 /bin/bash /usr/local/bin/startup.sh
root       471   269  3 07:15 ?        00:00:05 /usr/lib/jvm/java-8-openjdk-amd64//bin/java
-Dproc_jar -Dproc_hiveserver2 -Dlog4j.configurationFile=hive-log4j2.properties -Djava.util.logging.config.file=/opt/hive/conf/parquet-logging.properties
-Dyarn.log.dir=/opt/hadoop-3.2.1/logs -Dyarn.log.file=hadoop.log -Dyarn.home.dir=/opt/hadoop-3.2.1
-Dyarn.root.logger=INFO,console -Djava.library.path=/opt/hadoop-3.2.1/lib/native -Xmx256m
-Dhadoop.log.dir=/opt/hadoop-3.2.1/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/opt/hadoop-3.2.1
-Dhadoop.id.str=root -Dhadoop.root.logger=INFO,console -Dhadoop.policy.file=hadoop-policy.xml
-Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /opt/hive/lib/hive-service-3.1.2.jar
org.apache.hive.service.server.HiveServer2 --hiveconf hive.server2.enable.doAs=false
root       555     0  0 07:17 pts/0    00:00:00 /bin/bash
root       561   555  0 07:18 pts/0    00:00:00 ps -ef

root@hive-7bd48747d4-5zjmh:/opt# netstat -lptn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program
name    
root@hive-7bd48747d4-5zjmh:/opt# 

{code}
 

 

 


> hiveserver2 not listening on any port, am i miss some configurations?
> ---------------------------------------------------------------------
>
>                 Key: HIVE-23896
>                 URL: https://issues.apache.org/jira/browse/HIVE-23896
>             Project: Hive
>          Issue Type: Bug
>          Components: HiveServer2
>    Affects Versions: 3.1.2
>         Environment: hive: 3.1.2
> hadoop: 3.2.1, standalone, url: hdfs://namenode.hadoop.svc.cluster.local:9000
> {quote}$ $HADOOP_HOME/bin/hadoop fs -mkdir /tmp
>  $ $HADOOP_HOME/bin/hadoop fs -mkdir /user/hive/warehouse
> {quote}
> hadoop commands  are workable in the hiveserver node(POD).
>  
>            Reporter: alanwake
>            Priority: Blocker
>
>  
>  
> i try deply hive 3.1.2 on k8s.  it was worked on version 2.3.2.
> metastore node and postgres node are ok, but hiveserver look like i miss some important
configuration properties?
> {code:java}
>  {code}
>  
>  
>  
> {code:java}
> [root@master hive]# ./get.sh 
> NAME                         READY   STATUS    RESTARTS   AGE   IP             NODE 
                 NOMINATED NODE   READINESS GATES
> hive-7bd48747d4-5zjmh        1/1     Running   0          56s   10.244.3.110   node03.51.local
  <none>           <none>
> metastore-66b58f9f76-6wsxj   1/1     Running   0          56s   10.244.3.109   node03.51.local
  <none>           <none>
> postgres-57794b99b7-pqxwm    1/1     Running   0          56s   10.244.2.241   node02.51.local
  <none>           <none>NAME        TYPE        CLUSTER-IP       EXTERNAL-IP
  PORT(S)                           AGE   SELECTOR
> hive        NodePort    10.108.40.17     <none>        10002:30626/TCP,10000:31845/TCP
  56s   app=hive
> metastore   ClusterIP   10.106.159.220   <none>        9083/TCP               
          56s   app=metastore
> postgres    ClusterIP   10.108.85.47     <none>        5432/TCP               
          56s   app=postgres
> {code}
>  
>  
> {code:java}
> [root@master hive]# kubectl logs hive-7bd48747d4-5zjmh -n=hive
> Configuring core
>  - Setting hadoop.proxyuser.hue.hosts=*
>  - Setting fs.defaultFS=hdfs://namenode.hadoop.svc.cluster.local:9000
>  - Setting hadoop.http.staticuser.user=root
>  - Setting hadoop.proxyuser.hue.groups=*
> Configuring hdfs
>  - Setting dfs.namenode.datanode.registration.ip-hostname-check=false
>  - Setting dfs.webhdfs.enabled=true
>  - Setting dfs.permissions.enabled=false
> Configuring yarn
>  - Setting yarn.timeline-service.enabled=true
>  - Setting yarn.resourcemanager.system-metrics-publisher.enabled=true
>  - Setting yarn.resourcemanager.store.class=org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore
>  - Setting yarn.log.server.url=http://historyserver.hadoop.svc.cluster.local:8188/applicationhistory/logs/
>  - Setting yarn.resourcemanager.fs.state-store.uri=/rmstate
>  - Setting yarn.timeline-service.generic-application-history.enabled=true
>  - Setting yarn.log-aggregation-enable=true
>  - Setting yarn.resourcemanager.hostname=resourcemanager.hadoop.svc.cluster.local
>  - Setting yarn.resourcemanager.resource.tracker.address=resourcemanager.hadoop.svc.cluster.local:8031
>  - Setting yarn.timeline-service.hostname=historyserver.hadoop.svc.cluster.local
>  - Setting yarn.resourcemanager.scheduler.address=resourcemanager.hadoop.svc.cluster.local:8030
>  - Setting yarn.resourcemanager.address=resourcemanager.hadoop.svc.cluster.local:8032
>  - Setting yarn.nodemanager.remote-app-log-dir=/app-logs
>  - Setting yarn.resourcemanager.recovery.enabled=true
> Configuring httpfs
> Configuring kms
> Configuring mapred
> Configuring hive
>  - Setting datanucleus.autoCreateSchema=false
>  - Setting javax.jdo.option.ConnectionPassword=hive
>  - Setting hive.metastore.uris=thrift://metastore:9083
>  - Setting javax.jdo.option.ConnectionURL=jdbc:postgresql://metastore/metastore
>  - Setting javax.jdo.option.ConnectionUserName=hive
>  - Setting javax.jdo.option.ConnectionDriverName=org.postgresql.Driver
> Configuring for multihomed network
> [1/100] check for metastore:9083...
> [1/100] metastore:9083 is not available yet
> [1/100] try in 5s once again ...
> [2/100] check for metastore:9083...
> [2/100] metastore:9083 is not available yet
> [2/100] try in 5s once again ...
> [3/100] check for metastore:9083...
> [3/100] metastore:9083 is not available yet
> [3/100] try in 5s once again ...
> [4/100] check for metastore:9083...
> [4/100] metastore:9083 is not available yet
> [4/100] try in 5s once again ...
> [5/100] metastore:9083 is available.
> mkdir: `/tmp': File exists
> 2020-07-22 07:15:33: Starting HiveServer2
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in [jar:file:/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in [jar:file:/opt/hadoop-3.2.1/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> Hive Session ID = c86e1a7f-e18c-4c04-a8ba-1179c235fdce
> Hive Session ID = 6f036ed3-5041-4479-ad99-eebc193a110a
> ...
> ...
> ...
> Hive Session ID = xxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxxxxxxx
> # quit after print about 20 lines {code}
>  
> ----
>  
>  
> {code:java}
> [root@master hive]# kubectl exec -it hive-7bd48747d4-5zjmh -n=hive /bin/bash
> root@hive-7bd48747d4-5zjmh:/opt# ps -ef > 1.txt
> root@hive-7bd48747d4-5zjmh:/opt# cat 1.txt 
> UID        PID  PPID  C STIME TTY          TIME CMD
> root         1     0  0 07:15 ?        00:00:00 /bin/sh -c startup.sh
> root       269     1  0 07:15 ?        00:00:00 /bin/bash /usr/local/bin/startup.sh
> root       471   269  3 07:15 ?        00:00:05 /usr/lib/jvm/java-8-openjdk-amd64//bin/java
-Dproc_jar -Dproc_hiveserver2 -Dlog4j.configurationFile=hive-log4j2.properties -Djava.util.logging.config.file=/opt/hive/conf/parquet-logging.properties
-Dyarn.log.dir=/opt/hadoop-3.2.1/logs -Dyarn.log.file=hadoop.log -Dyarn.home.dir=/opt/hadoop-3.2.1
-Dyarn.root.logger=INFO,console -Djava.library.path=/opt/hadoop-3.2.1/lib/native -Xmx256m
-Dhadoop.log.dir=/opt/hadoop-3.2.1/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/opt/hadoop-3.2.1
-Dhadoop.id.str=root -Dhadoop.root.logger=INFO,console -Dhadoop.policy.file=hadoop-policy.xml
-Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /opt/hive/lib/hive-service-3.1.2.jar
org.apache.hive.service.server.HiveServer2 --hiveconf hive.server2.enable.doAs=false
> root       555     0  0 07:17 pts/0    00:00:00 /bin/bash
> root       561   555  0 07:18 pts/0    00:00:00 ps -ef
> root@hive-7bd48747d4-5zjmh:/opt# netstat -lptn
> Active Internet connections (only servers)
> Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program
name    
> root@hive-7bd48747d4-5zjmh:/opt# 
> {code}
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message