Return-Path: Delivered-To: apmail-hive-user-archive@www.apache.org Received: (qmail 36388 invoked from network); 10 Jan 2011 04:48:03 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 10 Jan 2011 04:48:03 -0000 Received: (qmail 50992 invoked by uid 500); 10 Jan 2011 04:48:02 -0000 Delivered-To: apmail-hive-user-archive@hive.apache.org Received: (qmail 50946 invoked by uid 500); 10 Jan 2011 04:48:00 -0000 Mailing-List: contact user-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hive.apache.org Delivered-To: mailing list user@hive.apache.org Received: (qmail 50938 invoked by uid 99); 10 Jan 2011 04:47:59 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Jan 2011 04:47:59 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [207.97.245.131] (HELO smtp131.iad.emailsrvr.com) (207.97.245.131) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Jan 2011 04:47:53 +0000 Received: from localhost (localhost.localdomain [127.0.0.1]) by smtp53.relay.iad1a.emailsrvr.com (SMTP Server) with ESMTP id 41E4358295; Sun, 9 Jan 2011 23:47:31 -0500 (EST) X-Virus-Scanned: OK Received: by smtp53.relay.iad1a.emailsrvr.com (Authenticated sender: adarsh.sharma-AT-orkash.com) with ESMTPSA id A6F6D58229 for ; Sun, 9 Jan 2011 23:47:29 -0500 (EST) Message-ID: <4D2A9050.6040304@orkash.com> Date: Mon, 10 Jan 2011 10:21:28 +0530 From: Adarsh Sharma User-Agent: Thunderbird 2.0.0.22 (X11/20090625) MIME-Version: 1.0 To: user@hive.apache.org Subject: Re: Hive/Hbase Integration Error References: <4D259AAA.5020606@orkash.com> <9CC5CBBD-97B1-4302-B443-0E23C943440F@fb.com> <4D26AA69.9060200@orkash.com> In-Reply-To: Content-Type: multipart/alternative; boundary="------------090204000906080206060207" X-Virus-Checked: Checked by ClamAV on apache.org This is a multi-part message in MIME format. --------------090204000906080206060207 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Viral Bajaria wrote: > Hi Adarsh, > > Can you provide some more details as follows. I have had issues before > on communicating between my HMaster etc. but the initial issue was > ACL(s) and my zookeeper settings were a little messed up too (I can't > recollect what was wrong with zookeeper). I would suggest that you get > things up and running with default zookeeper settings before playing > around with them. You could also run a custom java program which > connects to hbase to make sure the issue is with hbase settings and > has nothing to do with Hive settings. > Thanks Viral, below is my specifications : You are absolutely right. > 1) Are you running Namenode, HMaster, Hive on the same machine ? *I am running Namenode, HMaster, Hive on the same machine. We have a cluster of 10 servers and :- 1 Server act as ( Namenode , JobTracker, HMaster , Hive-Node containing Hive-0.6.0 package ). Would this affects. 8 Servers act as ( Datanode, TaskTracker, HregionServers ). Out of these 8 Servers I set three ( sometime five ) servers IP as Zookeeper Nodes. Would I need zookeeper.quorum.property set to separate servers or this configuration is right.* > 2) Are you able to access the HMaster through the Web UI ? I think the > default port is 60010 (or you can check in your hbase-site.xml) > *I am able to connect Hbase through Web Ui and also able to create tables in it. But error occurs when Hive/Hbase Integration comes into Play.* > Thanks, > Viral > > On Thu, Jan 6, 2011 at 9:53 PM, Adarsh Sharma > > wrote: > > John Sichi wrote: >> Here is what you need to do: >> >> 1) Use svn to check out the source for Hive 0.6 >> > I download Hive-0.6.0 source code with the command > > svn co http://svn.apache.org/repos/asf/hive/branches/branch-0.6/ > hive-0.6.0 > > > >> 2) In your checkout, replace the HBase 0.20.3 jars with the ones from 0.20.6 >> > Replace hbase-0.20.3.jar,hbase-0.20.3.test.jar by hbase-0.20.6.jar > and hbase-0.20.6.test jars in Hive-0.6.0/hbase-handler/lib folder > >> 3) Build Hive 0.6 from source >> > Then Build the hive package by *ant -Dhadoop.version=0.20.0 > package *command > Am I doing something wrong. > > I want to know why it occurs in hive.log > > 2011-01-05 15:19:36,783 ERROR DataNucleus.Plugin (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires "org.eclipse.core.resources" but it cannot be resolved. > > > With Best Regards > > Adarsh Sharma > > > >> 4) Use your new Hive build >> > > And use this new Hive build but I am sorry but the error remains > the same. > > >> JVS >> >> On Jan 6, 2011, at 2:34 AM, Adarsh Sharma wrote: >> >> >>> Dear all, >>> >>> I am sorry I am posting this message again but I can't able to locate the root cause after googled a lot. >>> >>> I am trying Hive/Hbase Integration from the past 2 days. I am facing the below issue while creating external table in Hive. >>> >>> I am using hadoop-0.20.2, hbase-0.20.6, hive-0.6.0 ( Mysql as metstore ) and java-1.6.0_20. Hbase-0.20.3 is also checked. >>> >>> Problem arises when I issue the below command : >>> >>> hive> CREATE TABLE hive_hbasetable_k(key int, value string) >>> > STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' >>> > WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:val") >>> > TBLPROPERTIES ("hbase.table.name " = "hivehbasek"); >>> >>> >>> FAILED: Error in metadata: MetaException(message:org.apache.hadoop.hbase.MasterNotRunningException >>> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:374) >>> at org.apache.hadoop.hbase.client.HBaseAdmin.(HBaseAdmin.java:72) >>> at org.apache.hadoop.hive.hbase.HBaseStorageHandler.getHBaseAdmin(HBaseStorageHandler.java:64) >>> at org.apache.hadoop.hive.hbase.HBaseStorageHandler.preCreateTable(HBaseStorageHandler.java:159) >>> at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:275) >>> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:394) >>> at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:2126) >>> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:166) >>> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107) >>> at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55) >>> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:633) >>> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:506) >>> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:384) >>> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138) >>> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197) >>> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:302) >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>> at java.lang.reflect.Method.invoke(Method.java:597) >>> at org.apache.hadoop.util.RunJar.main(RunJar.java:156) >>> FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask >>> >>> >>> It seems my HMaster is not Running but I checked from IP:60010 that it is running and I am able to create,insert tables in Hbase Properly. >>> >>> Below is the contents of my hive.log : >>> >>> 2011-01-05 15:19:36,783 ERROR DataNucleus.Plugin (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires "org.eclipse.core.resources" but it cannot be resolved. >>> 2011-01-05 15:19:36,783 ERROR DataNucleus.Plugin (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires "org.eclipse.core.resources" but it cannot be resolved. >>> 2011-01-05 15:19:36,785 ERROR DataNucleus.Plugin (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires "org.eclipse.core.runtime" but it cannot be resolved. >>> 2011-01-05 15:19:36,785 ERROR DataNucleus.Plugin (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires "org.eclipse.core.runtime" but it cannot be resolved. >>> 2011-01-05 15:19:36,786 ERROR DataNucleus.Plugin (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires "org.eclipse.text" but it cannot be resolved. >>> 2011-01-05 15:19:36,786 ERROR DataNucleus.Plugin (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires "org.eclipse.text" but it cannot be resolved. >>> 2011-01-05 15:20:12,185 WARN zookeeper.ClientCnxn (ClientCnxn.java:run(967)) - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@561279c8 >>> java.net.ConnectException: Connection refused >>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >>> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) >>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933) >>> 2011-01-05 15:20:12,188 WARN zookeeper.ClientCnxn (ClientCnxn.java:cleanup(1001)) - Ignoring exception during shutdown input >>> java.nio.channels.ClosedChannelException >>> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638) >>> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360) >>> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999) >>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970) >>> 2011-01-05 15:20:12,188 WARN zookeeper.ClientCnxn (ClientCnxn.java:cleanup(1006)) - Ignoring exception during shutdown output >>> java.nio.channels.ClosedChannelException >>> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649) >>> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368) >>> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004) >>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970) >>> 2011-01-05 15:20:12,621 WARN zookeeper.ClientCnxn (ClientCnxn.java:run(967)) - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@799dbc3b >>> >>> Please help me, as i am not able to solve this problem. >>> >>> Also I want to add one more thing that my hadoop Cluster is of 9 nodes and 8 nodes act as Datanodes,Tasktrackers and Regionservers. >>> >>> Among these nodes is set zookeeper.quorum.property to have 5 Datanodes. I don't know the number of servers needed for Zookeeper in fully distributed mode. >>> >>> >>> Best Regards >>> >>> Adarsh Sharma >>> >>> >>> >>> >> > > --------------090204000906080206060207 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Viral Bajaria wrote:
Hi Adarsh,
 
Can you provide some more details as follows. I have had issues before on communicating between my HMaster etc. but the initial issue was ACL(s) and my zookeeper settings were a little messed up too (I can't recollect what was wrong with zookeeper). I would suggest that you get things up and running with default zookeeper settings before playing around with them. You could also run a custom java program which connects to hbase to make sure the issue is with hbase settings and has nothing to do with Hive settings.
 
Thanks Viral, below is my specifications :

You are absolutely right.

1) Are you running Namenode, HMaster, Hive on the same machine ?


I am running Namenode, HMaster, Hive on the same machine.

We have a cluster of 10 servers and :-

1 Server act as ( Namenode , JobTracker, HMaster , Hive-Node containing Hive-0.6.0 package ). Would this affects.

8 Servers act as ( Datanode, TaskTracker, HregionServers ).

Out of these 8 Servers I set three ( sometime five ) servers IP  as Zookeeper Nodes. Would I need zookeeper.quorum.property set to separate servers or this configuration is right.
2) Are you able to access the HMaster through the Web UI ? I think the default port is 60010 (or you can check in your hbase-site.xml)
 

I am able to connect Hbase through Web Ui and also able to create tables in it. But error occurs when Hive/Hbase Integration comes into Play.
Thanks,
Viral
 
On Thu, Jan 6, 2011 at 9:53 PM, Adarsh Sharma <adarsh.sharma@orkash.com> wrote:
John Sichi wrote: 
Here is what you need to do:

1) Use svn to check out the source for Hive 0.6
I download Hive-0.6.0 source code with the command

 svn co http://svn.apache.org/repos/asf/hive/branches/branch-0.6/ hive-0.6.0 



2) In your checkout, replace the HBase 0.20.3 jars with the ones from 0.20.6
Replace hbase-0.20.3.jar,hbase-0.20.3.test.jar by hbase-0.20.6.jar and hbase-0.20.6.test jars in Hive-0.6.0/hbase-handler/lib folder 

3) Build Hive 0.6 from source
Then Build the hive package by ant -Dhadoop.version=0.20.0 package command
Am I doing something wrong.

I want to know why it occurs in hive.log

2011-01-05 15:19:36,783 ERROR DataNucleus.Plugin (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires "org.eclipse.core.resources" but it cannot be resolved.


With Best Regards

Adarsh Sharma
    

4) Use your new Hive build

And use this new Hive build but I am sorry but the error remains the same. 


JVS

On Jan 6, 2011, at 2:34 AM, Adarsh Sharma wrote:

  
Dear all,

I am sorry I am posting this message again but I can't able to locate the root cause after googled a lot.

I am trying Hive/Hbase Integration from the past 2 days. I am facing the below issue while creating external table in Hive.

I am using hadoop-0.20.2, hbase-0.20.6, hive-0.6.0 ( Mysql as metstore ) and java-1.6.0_20. Hbase-0.20.3 is also checked.

Problem arises when I issue the below command :

hive> CREATE TABLE hive_hbasetable_k(key int, value string)
    > STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
    > WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:val")
    > TBLPROPERTIES ("hbase.table.name" = "hivehbasek");


FAILED: Error in metadata: MetaException(message:org.apache.hadoop.hbase.MasterNotRunningException
        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:374)
        at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:72)
        at org.apache.hadoop.hive.hbase.HBaseStorageHandler.getHBaseAdmin(HBaseStorageHandler.java:64)
        at org.apache.hadoop.hive.hbase.HBaseStorageHandler.preCreateTable(HBaseStorageHandler.java:159)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:275)
        at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:394)
        at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:2126)
        at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:166)
        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
        at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
        at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:633)
        at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:506)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:384)
        at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:302)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask


It seems my HMaster is not Running but I checked from IP:60010 that it is running and I am able to create,insert tables in Hbase Properly.

Below is the contents of my hive.log :

  2011-01-05 15:19:36,783 ERROR DataNucleus.Plugin (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires "org.eclipse.core.resources" but it cannot be resolved.
 2011-01-05 15:19:36,783 ERROR DataNucleus.Plugin (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires "org.eclipse.core.resources" but it cannot be resolved.
 2011-01-05 15:19:36,785 ERROR DataNucleus.Plugin (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires "org.eclipse.core.runtime" but it cannot be resolved.
 2011-01-05 15:19:36,785 ERROR DataNucleus.Plugin (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires "org.eclipse.core.runtime" but it cannot be resolved.
 2011-01-05 15:19:36,786 ERROR DataNucleus.Plugin (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires "org.eclipse.text" but it cannot be resolved.
 2011-01-05 15:19:36,786 ERROR DataNucleus.Plugin (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires "org.eclipse.text" but it cannot be resolved.
 2011-01-05 15:20:12,185 WARN  zookeeper.ClientCnxn (ClientCnxn.java:run(967)) - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@561279c8
 java.net.ConnectException: Connection refused
       at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
       at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
       at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
 2011-01-05 15:20:12,188 WARN  zookeeper.ClientCnxn (ClientCnxn.java:cleanup(1001)) - Ignoring exception during shutdown input
 java.nio.channels.ClosedChannelException
       at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
       at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
       at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
       at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
 2011-01-05 15:20:12,188 WARN  zookeeper.ClientCnxn (ClientCnxn.java:cleanup(1006)) - Ignoring exception during shutdown output
 java.nio.channels.ClosedChannelException
       at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
       at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
       at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
       at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
 2011-01-05 15:20:12,621 WARN  zookeeper.ClientCnxn (ClientCnxn.java:run(967)) - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@799dbc3b
 
  Please help me, as i am not able to solve this problem.
 
 Also I want to add one more thing that my hadoop Cluster is of 9 nodes and 8 nodes act as Datanodes,Tasktrackers and Regionservers.
 
 Among these nodes is set zookeeper.quorum.property to have 5 Datanodes. I don't know the number of servers needed for Zookeeper in fully distributed mode.
 
 
 Best Regards

 Adarsh Sharma



--------------090204000906080206060207--