hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Prasad Chakka <pra...@facebook.com>
Subject Re: Still can't get metastore working with mysql
Date Sat, 24 Jan 2009 23:02:25 GMT
On the other hand, you can generate SQL schema using "generate-schem" ant target in hive/metastore
and use the config parameters that Joydeep sent.


________________________________
From: Prasad Chakka <prasad@facebook.com>
Date: Sat, 24 Jan 2009 15:00:00 -0800
To: <hive-user@hadoop.apache.org>
Conversation: Still can't get metastore working with mysql
Subject: Re: Still can't get metastore working with mysql

Well, if you had just used the defaults that are in the hive source and changed just the JDBC
params, that is what will happen. The parameters that you are using comes from internal Facebook
settings and they are tuned for production. Checking for correctness of schema every time
hive connects to metastore is inefficient in production cluster.


________________________________
From: Josh Ferguson <josh@besquared.net>
Reply-To: <hive-user@hadoop.apache.org>
Date: Sat, 24 Jan 2009 14:55:21 -0800
To: <hive-user@hadoop.apache.org>
Subject: Re: Still can't get metastore working with mysql

Joydeep Sen Sarma sent them out in the list last month when I was first trying to get this
working. I tried to modify them appropriately.

This whole "lazy" creation of schemas and tables seems like a really bad way to do things.
My configuration shouldn't be different for something that is up and running and something
that needs to be created. I should just be able to define my connection parameters and then
have a command that migrates my database into the proper state for use. Trying to figure out
what magical set of esoteric configuration parameters will tell this black box library to
create my schema is really frustrating.

Josh F.

On Jan 24, 2009, at 2:26 PM, Prasad Chakka wrote:

Try setting fixedDataStore to false.

 BTB, where did you get the initial values from the config you posted? These values are for
the store that is already running in production and you don't expect any changes to the schema.



________________________________
From: Josh Ferguson <josh@besquared.net>
 Reply-To: <hive-user@hadoop.apache.org>
 Date: Sat, 24 Jan 2009 14:14:14 -0800
 To: <hive-user@hadoop.apache.org>
 Subject: Re: Still can't get metastore working with mysql

 That got me a little further, now I'm getting this error:

 FAILED: Error in metadata: MetaException(message:Got exception: javax.jdo.JDODataStoreException
Required table missing : "`SERDES`" in Catalog "" Schema "". JPOX requires this table to perform
its persistence operations. Either your MetaData is incorrect, or you need to enable "org.jpox.autoCreateTables")
 FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask

 In my hive-default.xml I turned on autoCreateTables and autoCreateSchema and it's still throwing
this error back to me? Is there something else I need to do?

 Josh Ferguson

 On Jan 24, 2009, at 1:28 PM, Prasad Chakka wrote:


Down load the jar from here http://dev.mysql.com/downloads/connector/j/5.1.html




________________________________
From: Prasad Chakka <prasad@facebook.com>
  Reply-To: <hive-user@hadoop.apache.org>
  Date: Sat, 24 Jan 2009 13:13:35 -0800
  To: <hive-user@hadoop.apache.org>
  Subject: Re: Still can't get metastore working with mysql

  JPOX/Hive doesn't yet download the relevant client libraries based on the class name :)
Drop mysql client jar in hive/lib.  Also try setting org.jpox.autoCreateSchema to true.



________________________________
From: Josh Ferguson <josh@besquared.net>
  Reply-To: <hive-user@hadoop.apache.org>
  Date: Sat, 24 Jan 2009 13:06:26 -0800
  To: <hive-user@hadoop.apache.org>
  Subject: Re: Still can't get metastore working with mysql

  So I'm not really sure what the state of the database should be, I have a database called
hive and the user has permissions to access it. I'm not sure which mysql client libraries
need to be in the classpath, I assumed it was all rolled into the jpox stuff. Here is the
complete error:

  09/01/24 16:02:16 DEBUG exec.DDLTask: org.apache.hadoop.hive.ql.metadata.HiveException:
MetaException(message:Got exception: javax.jdo.JDOFatalInternalException Error creating transactional
connection factory)
  at org.apache.hadoop.hive.ql.metadata.Hive.getTablesByPattern(Hive.java:446)
  at org.apache.hadoop.hive.ql.metadata.Hive.getAllTables(Hive.java:433)
  at org.apache.hadoop.hive.ql.exec.DDLTask.showTables(DDLTask.java:330)
  at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:127)
  at org.apache.hadoop.hive.ql.Driver.run(Driver.java:212)
  at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:174)
  at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:207)
  at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  at java.lang.reflect.Method.invoke(Method.java:597)
  at org.apache.hadoop.util.RunJar.main(RunJar.java:165)
  at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
  at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
  at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
  at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)
  Caused by: MetaException(message:Got exception: javax.jdo.JDOFatalInternalException Error
creating transactional connection factory)
  at org.apache.hadoop.hive.metastore.MetaStoreUtils.logAndThrowMetaException(MetaStoreUtils.java:543)
  at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTables(HiveMetaStoreClient.java:451)
  at org.apache.hadoop.hive.ql.metadata.Hive.getTablesByPattern(Hive.java:444)
  ... 16 more


  On Jan 24, 2009, at 12:58 PM, Prasad Chakka wrote:



There could be many reasons for it. Could you post the stacktrace for the JDOFatal?

   Some reasons could be
   That mysql client libs are in the classpath.
   That there is a database called 'hive' in mysql
   That there is schema already created in that database (otherwise set autoCreateSchema to
true) etc...

   Check documentation for the many jpox options



________________________________
From: Josh Ferguson <josh@besquared.net>
   Reply-To: <hive-user@hadoop.apache.org>
   Date: Sat, 24 Jan 2009 12:48:27 -0800
   To: <hive-user@hadoop.apache.org>
   Subject: Still can't get metastore working with mysql

   Ok I still can't manage to get the metastore working with mysql after
   all this time trying. The error is:

   FAILED: Error in metadata: MetaException(message:Got exception:
   javax.jdo.JDOFatalInternalException Error creating transactional
   connection factory)
   FAILED: Execution Error, return code 1 from
   org.apache.hadoop.hive.ql.exec.DDLTask

   The relevant config is:

   <!-- Hadoop Setup -->
   <property>
      <name>hadoop.bin.path</name>
      <value>/usr/local/hadoop/bin/hadoop</value>
      <description>Path to hadoop binary. Assumes that by default we are
   executing from hive</description>
   </property>

   <property>
      <name>hadoop.config.dir</name>
      <value>/usr/local/hadoop/conf</value>
      <description>Path to hadoop configuration. Again assumes that by
   default we are executing from hive/</description>
   </property>

   <!-- Hive Execution Parameters -->
   <property>
      <name>hive.exec.scratchdir</name>
      <value>/tmp/hive-${user.name}</value>
      <description>Scratch space for Hive jobs</description>
   </property>

   <property>
      <name>javax.jdo.option.ConnectionURL</name>
      <value>jdbc:mysql://localhost/hive</value>
   </property>

   <property>
      <name>javax.jdo.option.ConnectionDriverName</name>
      <value>com.mysql.jdbc.Driver</value>
   </property>

   <property>
      <name>javax.jdo.option.ConnectionUserName</name>
      <value>******</value>
   </property>

   <property>
      <name>javax.jdo.option.ConnectionPassword</name>
      <value>******</value>
   </property>

   <property>
      <name>org.jpox.autoCreateSchema</name>
      <value>false</value>
   </property>

   <property>
      <name>org.jpox.fixedDatastore</name>
      <value>true</value>
   </property>

   <property>
      <name>hive.metastore.usefilestore</name>
      <value>false</value>
   </property>

   <property>
      <name>hive.metastore.checkForDefaultDb</name>
      <value>false</value>
   </property>

   <property>
      <name>hive.metastore.rawstore.impl</name>
      <value>org.apache.hadoop.hive.metastore.ObjectStore</value>
      <description>Name of the class that implements
   org.apache.hadoop.hive.metastore.rawstore interface. This class is
   used to store and retrieval of raw metadata objects such as table,
   database</description>
   </property>

   <property>
      <name>hive.metastore.local</name>
      <value>true</value>
      <description>controls whether to connect to remove metastore server
   or open a new metastore server in Hive Client JVM</description>
   </property>



















Mime
View raw message