hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tarik Yilmaz (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-14798) MSCK REPAIR TABLE throws null pointer exception
Date Fri, 24 Feb 2017 08:04:44 GMT

    [ https://issues.apache.org/jira/browse/HIVE-14798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15882181#comment-15882181
] 

Tarik Yilmaz commented on HIVE-14798:
-------------------------------------

We had the same problem too. Probably, this problem is irrelevant with Apache Hive because
our solution removes Google's Hive build and install Apache Hive.

{code}
#!/bin/bash
# Copyright 2017 Metglobal, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

HIVE_VERSION=2.1.1
ROLE=$(/usr/share/google/get_metadata_value attributes/dataproc-role)

cd /tmp

# Download Apache Hive build
wget http://www-eu.apache.org/dist/hive/hive-${HIVE_VERSION}/apache-hive-${HIVE_VERSION}-bin.tar.gz

# Extract Hive package
tar -xf apache-hive-${HIVE_VERSION}-bin.tar.gz

# Delete temporary Hive archive
rm -rf apache-hive-${HIVE_VERSION}-bin.tar.gz

rm -rf apache-hive-${HIVE_VERSION}-bin/conf
rm apache-hive-${HIVE_VERSION}-bin/lib/guava-14.0.1.jar

# Install Guava >= 15.0 for GoogleHadoopFileSystem
wget http://central.maven.org/maven2/com/google/guava/guava/19.0/guava-19.0.jar
mv guava-19.0.jar apache-hive-${HIVE_VERSION}-bin/lib/


if [[ "${ROLE}" == 'Master' ]]; then
	# Stop the current services
	service hive-metastore stop
	service hive-server2 stop
fi

# Remove current Hive
rm -rf /usr/lib/hive

# Install Apache Hive
mv apache-hive-${HIVE_VERSION}-bin /usr/lib/hive

# Create symlink for required remote files
ln -s /etc/hive/conf /usr/lib/hive/conf
ln -s /usr/lib/zookeeper/zookeeper.jar /usr/lib/hive/lib/zookeeper.jar
ln -s /usr/share/java/mysql.jar /usr/lib/hive/lib/mysql.jar

# Create symlink for Hive libraries
ln -s /usr/lib/hive/lib/hive-accumulo-handler-${HIVE_VERSION}.jar /usr/lib/hive/lib/hive-accumulo-handler.jar
ln -s /usr/lib/hive/lib/hive-ant-${HIVE_VERSION}.jar /usr/lib/hive/lib/hive-ant.jar
ln -s /usr/lib/hive/lib/hive-beeline-${HIVE_VERSION}.jar /usr/lib/hive/lib/hive-beeline.jar
ln -s /usr/lib/hive/lib/hive-cli-${HIVE_VERSION}.jar /usr/lib/hive/lib/hive-cli.jar
ln -s /usr/lib/hive/lib/hive-common-${HIVE_VERSION}.jar /usr/lib/hive/lib/hive-common.jar
ln -s /usr/lib/hive/lib/hive-contrib-${HIVE_VERSION}.jar /usr/lib/hive/lib/hive-contrib.jar
ln -s /usr/lib/hive/lib/hive-exec-${HIVE_VERSION}.jar /usr/lib/hive/lib/hive-exec.jar
ln -s /usr/lib/hive/lib/hive-hbase-handler-${HIVE_VERSION}.jar /usr/lib/hive/lib/hive-hbase-handler.jar
ln -s /usr/lib/hive/lib/hive-hplsql-${HIVE_VERSION}.jar /usr/lib/hive/lib/hive-hplsql.jar
ln -s /usr/lib/hive/lib/hive-hwi-${HIVE_VERSION}.jar /usr/lib/hive/lib/hive-hwi.jar
ln -s /usr/lib/hive/lib/hive-jdbc-${HIVE_VERSION}.jar /usr/lib/hive/lib/hive-jdbc.jar
ln -s /usr/lib/hive/lib/hive-llap-client-${HIVE_VERSION}.jar /usr/lib/hive/lib/hive-llap-client.jar
ln -s /usr/lib/hive/lib/hive-llap-common-${HIVE_VERSION}.jar /usr/lib/hive/lib/hive-llap-common.jar
ln -s /usr/lib/hive/lib/hive-llap-ext-client-${HIVE_VERSION}.jar /usr/lib/hive/lib/hive-llap-ext-client.jar
ln -s /usr/lib/hive/lib/hive-llap-server-${HIVE_VERSION}.jar /usr/lib/hive/lib/hive-llap-server.jar
ln -s /usr/lib/hive/lib/hive-llap-tez-${HIVE_VERSION}.jar /usr/lib/hive/lib/hive-llap-tez.jar
ln -s /usr/lib/hive/lib/hive-metastore-${HIVE_VERSION}.jar /usr/lib/hive/lib/hive-metastore.jar
ln -s /usr/lib/hive/lib/hive-orc-${HIVE_VERSION}.jar /usr/lib/hive/lib/hive-orc.jar
ln -s /usr/lib/hive/lib/hive-serde-${HIVE_VERSION}.jar /usr/lib/hive/lib/hive-serde.jar
ln -s /usr/lib/hive/lib/hive-service-${HIVE_VERSION}.jar /usr/lib/hive/lib/hive-service.jar
ln -s /usr/lib/hive/lib/hive-service-rpc-${HIVE_VERSION}.jar /usr/lib/hive/lib/hive-service-rpc.jar
ln -s /usr/lib/hive/lib/hive-shims-0.23-${HIVE_VERSION}.jar /usr/lib/hive/lib/hive-shims-0.23.jar
ln -s /usr/lib/hive/lib/hive-shims-${HIVE_VERSION}.jar /usr/lib/hive/lib/hive-shims.jar
ln -s /usr/lib/hive/lib/hive-shims-common-${HIVE_VERSION}.jar /usr/lib/hive/lib/hive-shims-common.jar
ln -s /usr/lib/hive/lib/hive-shims-scheduler-${HIVE_VERSION}.jar /usr/lib/hive/lib/hive-shims-scheduler.jar
ln -s /usr/lib/hive/lib/hive-storage-api-${HIVE_VERSION}.jar /usr/lib/hive/lib/hive-storage-api.jar
ln -s /usr/lib/hive/lib/hive-testutils-${HIVE_VERSION}.jar /usr/lib/hive/lib/hive-testutils.jar

if [[ "${ROLE}" == 'Master' ]]; then
	# Start Hive services again
	service hive-metastore start
	service hive-server2 start
fi

{code}

> MSCK REPAIR TABLE throws null pointer exception
> -----------------------------------------------
>
>                 Key: HIVE-14798
>                 URL: https://issues.apache.org/jira/browse/HIVE-14798
>             Project: Hive
>          Issue Type: Bug
>          Components: Metastore
>    Affects Versions: 2.1.0
>            Reporter: Anbu Cheeralan
>
> MSCK REPAIR TABLE statement throws null pointer exception in Hive 2.1
> I have tested the same against external/internal tables created both in HDFS and in Google
Cloud.
> The error shown in beeline/sql client 
> Error: Error while processing statement: FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask (state=08S01,code=1)
> Hive Logs:
> 2016-09-20T17:28:00,717 ERROR [HiveServer2-Background-Pool: Thread-92]: metadata.HiveMetaStoreChecker
(:()) - java.lang.NullPointerException
> 2016-09-20T17:28:00,717 WARN  [HiveServer2-Background-Pool: Thread-92]: exec.DDLTask
(:()) - Failed to run metacheck: 
> org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NullPointerException
>         at org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.getAllLeafDirs(HiveMetaStoreChecker.java:444)
>         at org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.getAllLeafDirs(HiveMetaStoreChecker.java:388)
>         at org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.findUnknownPartitions(HiveMetaStoreChecker.java:309)
>         at org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.checkTable(HiveMetaStoreChecker.java:285)
>         at org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.checkTable(HiveMetaStoreChecker.java:230)
>         at org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker.checkMetastore(HiveMetaStoreChecker.java:109)
>         at org.apache.hadoop.hive.ql.exec.DDLTask.msck(DDLTask.java:1814)
>         at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:403)
>         at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197)
>          at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
>         at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1858)
>         at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1562)
>         at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1313)
>         at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1084)
>         at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1077)
>         at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:235)
>         at org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:90)
>         at org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:299)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:422)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
>         at org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:312)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
>         at java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1011)
>         at java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006)
>         at org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker$1.call(HiveMetaStoreChecker.java:432)
>         at org.apache.hadoop.hive.ql.metadata.HiveMetaStoreChecker$1.call(HiveMetaStoreChecker.java:418)
>         ... 4 more
> Here are the steps to recreate this issue:
> use default;
> DROP TABLE IF EXISTS repairtable;
> CREATE TABLE repairtable(col STRING) PARTITIONED BY (p1 STRING, p2 STRING);
> MSCK REPAIR TABLE default.repairtable;



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message