tajo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "JaeHwa Jung (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (TAJO-289) HCatalogStore supports SELECT statement
Date Fri, 08 Nov 2013 16:33:18 GMT

     [ https://issues.apache.org/jira/browse/TAJO-289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

JaeHwa Jung updated TAJO-289:
-----------------------------

    Attachment: TAJO-289.patch

I'm so glad to upload this patch.
>From now on, Tajo can connect HiveMetastore and can get lots of data stored on HDFS through
HiveMetaStore. Of course, there were some limitation as follows:

- current version support just text file format.
- current version cannot support compression type.

But, I'll implement unsupported functions time soon.

If you want to use HiveMetaStore as CatalogServer, you have to start up HiveMetaStore server
as follows:
{code:xml}
$HIVE_HOME/bin/hive --service metastore
{code}
 
Or you can start up HiveMetaStore server through HCatalog as follows:
{code:xml}
$HCATALOG_HOME/sbin/hcat_server.sh start
{code}
If you start up HCatalog server, HCatalogServer start up HiveMetaStore server actually.

If you start up HiveMetaStore server, you have to update catalog-site.xml as follows:
{code:xml}
  <property>
    <name>tajo.catalog.store.class</name>
    <value>org.apache.tajo.catalog.store.HCatalogStore</value>
  </property>
  <property>
    <name>tajo.catalog.uri</name>
    <value>thrift://localhost:10001</value>
  </property>
{code}
'tajo.catalog.url' is HiveMetaStore sever uri. You must specify your address on this property.


And I borrowed JobConf from CDH and I removed static resource files declaratives. If JobConf
use static resource files (mapred-default.xml, mapred-site.xml), TajoWorker made error message
as folllows:
{code:xml}
Service:org.apache.tajo.worker.TajoWorkerManagerService is started.
2013-11-08 23:44:03,411 INFO  worker.TaskRunnerManager (TaskRunnerManager.java:run(139)) -
FinishedQueryMasterTaskCleanThread started: expire interval minutes = 720
2013-11-08 23:44:03,419 ERROR service.CompositeService (CompositeService.java:start(72)) -
Error starting services org.apache.tajo.worker.TajoWorker
org.jboss.netty.channel.ChannelException: Failed to bind to: 0.0.0.0/0.0.0.0:8082
	at org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:272)
	at org.apache.tajo.pullserver.TajoPullServerService.start(TajoPullServerService.java:237)
	at org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68)
	at org.apache.tajo.worker.TajoWorker.start(TajoWorker.java:257)
	at org.apache.tajo.worker.TajoWorker.startWorker(TajoWorker.java:125)
	at org.apache.tajo.worker.TajoWorker.main(TajoWorker.java:659)
Caused by: java.net.BindException: Address already in use
	at sun.nio.ch.Net.bind(Native Method)
	at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:124)
	at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
	at org.jboss.netty.channel.socket.nio.NioServerBoss$RegisterTask.run(NioServerBoss.java:193)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:366)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:290)
	at org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
	at java.lang.Thread.run(Thread.java:680)
{code}
After I modified JobConf, TajoWorker start up always successfully. 

Finally, my development environments are as follows:
- hadoop 2.0.x-alpha
- hive-0.11.0  
- hcatalog-0.5.0

> HCatalogStore supports SELECT statement
> ---------------------------------------
>
>                 Key: TAJO-289
>                 URL: https://issues.apache.org/jira/browse/TAJO-289
>             Project: Tajo
>          Issue Type: Sub-task
>          Components: catalog
>    Affects Versions: 0.8-incubating
>            Reporter: JaeHwa Jung
>            Assignee: JaeHwa Jung
>             Fix For: 0.8-incubating
>
>         Attachments: TAJO-289.patch
>
>
> I designed Tajo to use Hive metastore with HCatalog.
> In this case, Tajo has to have an interface to connect to HCatalog.
> And I think that this connect is very useful to connect to another catalogs as follows:
>  some different Hive catalogs, HBase catalogs
> So, I named this interface as CatalogDriver. It will has some properties as follows:
> - catalog namespace name
> - catalog URI
> - catalog driver class



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message