hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward Capriolo" <edlinuxg...@gmail.com>
Subject Hive questions about the meta db
Date Thu, 02 Oct 2008 00:04:07 GMT
I have been working with Hive for the past week. The ability to wrap
an SQL like tool over HDFS as very powerful. Now that I am comfortable
with the
concept, I am looking at an implementation of it.

Currently I have a three node cluster for testing hadoop1, hadoop2,
and hadoop3. I have hive installed on hadoop1 and derby is working as
the metastore
on the local filesystem. I am not able to run more then one instance
of Hive. That makes sense because hive probably wants exclusive access
to the meta_db.

This is a big downside as I can only run 1 job at once. These are the
solutions I am looking at:

Different hive instances different warehouse directories on hdfs.
instance1 /users/hive/warehouse1
instance2 /users/hive/warehouse2
I could for example install one copy of hive on each server.
Upside: I can now execute three jobs at once.
Downside: I have three separate warehouses. Even though they live on
hdfs together they are unaware of each other.

Option 2: Always use external tables, create my own schema 'replication' system
In this case the layout and install is the same.
instance1 /users/hive/warehouse1
instance2 /users/hive/warehouse2

Also if any instance creates a table, it should create the table
outside the warehouse
Now I need some external process that runs 'create external table
/users/hive/shared/table1' on all the other nodes. This way
all nodes can query the table. I am really not woried about table
mutations once that data goes in the tables will almost never be

IIRC meta_db might be able to be stored on hdfs in a future version.
Am I over thinking something, have I missed a way to execute multiple
hive queries at once?

View raw message