tajo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jihoon Son (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (TAJO-16) Enable Tajo catalog to access Hive metastore.
Date Mon, 11 Nov 2013 06:09:18 GMT

    [ https://issues.apache.org/jira/browse/TAJO-16?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13818718#comment-13818718
] 

Jihoon Son commented on TAJO-16:
--------------------------------

Thanks for your patch.
After applying your patch and adding the above catalog configuration, I met the following
errors.
{code}
tajo> \d
2013-11-11 14:56:32,710 INFO  rpc.RpcConnectionPool (RpcConnectionPool.java:closeConnection(78))
- CloseConnection [class org.apache.tajo.ipc.TajoMasterClientProtocol,localhost/127.0.0.1:26002,false]
2013-11-11 14:56:32,713 ERROR rpc.RpcProtos (BlockingRpcClient.java:exceptionCaught(205))
- RPC Exception:Exception [org.apache.tajo.ipc.TajoMasterClientProtocol(127.0.0.1:26002)]:
org.apache.tajo.rpc.RetriesExhaustedException: Failed after attempts=3, exceptions:
java.lang.NullPointerException
java.lang.NullPointerException
java.lang.NullPointerException

	at org.apache.tajo.rpc.ServerCallable.withRetries(ServerCallable.java:91)
	at org.apache.tajo.catalog.AbstractCatalogClient.getAllTableNames(AbstractCatalogClient.java:73)
	at org.apache.tajo.master.TajoMasterClientService$TajoMasterClientProtocolServiceHandler.getTableList(TajoMasterClientService.java:273)
	at org.apache.tajo.ipc.TajoMasterClientProtocol$TajoMasterClientProtocolService$2.callBlockingMethod(TajoMasterClientProtocol.java:244)
	at org.apache.tajo.rpc.BlockingRpcServer$ServerHandler.messageReceived(BlockingRpcServer.java:91)
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
	at org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:70)
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
	at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
	at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
	at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
	at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:679)
{code}
The master log is as follows.
{code}
2013-11-11 14:55:36,538 INFO  rm.TajoWorkerResourceManager (TajoWorkerResourceManager.java:run(209))
- WorkerMonitor start
2013-11-11 14:55:36,540 INFO  rm.TajoWorkerResourceManager (TajoWorkerResourceManager.java:run(278))
- WorkerResourceAllocationThread start
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:{TAJO_HOME}/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:{HADOOP_HOME}/share/hadoop/common/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
2013-11-11 14:55:41,584 INFO  mortbay.log (NativeMethodAccessorImpl.java:invoke0(?)) - Logging
to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2013-11-11 14:55:41,626 INFO  webapp.HttpServer (HttpServer.java:start(333)) - Jetty bound
to port 26080
2013-11-11 14:55:41,626 INFO  mortbay.log (NativeMethodAccessorImpl.java:invoke0(?)) - jetty-6.1.14
2013-11-11 14:55:45,426 INFO  mortbay.log (NativeMethodAccessorImpl.java:invoke0(?)) - Started
SelectChannelConnector@0.0.0.0:26080
2013-11-11 14:55:45,436 INFO  master.TajoMaster (TajoMaster.java:checkAndInitializeSystemDirectories(201))
- Tajo Root Directory: hdfs://localhost:7020/tajo
2013-11-11 14:55:46,031 INFO  master.TajoMaster (TajoMaster.java:checkAndInitializeSystemDirectories(206))
- FileSystem (hdfs://localhost:7020) is initialized.
2013-11-11 14:55:46,093 INFO  master.TajoMaster (TajoMaster.java:checkAndInitializeSystemDirectories(226))
- Tajo Warehouse dir: hdfs://localhost:7020/tajo/warehouse
2013-11-11 14:55:46,095 INFO  master.TajoMaster (TajoMaster.java:checkAndInitializeSystemDirectories(235))
- Staging dir: hdfs://localhost:7020/tajo/warehouse
2013-11-11 14:55:46,102 WARN  storage.AbstractStorageManager (AbstractStorageManager.java:<init>(85))
- does not support block metadata. ('dfs.datanode.hdfs-blocks-metadata.enabled')
2013-11-11 14:55:46,135 INFO  service.AbstractService (AbstractService.java:init(81)) - Service:Dispatcher
is inited.
2013-11-11 14:55:46,138 INFO  catalog.CatalogServer (CatalogServer.java:init(108)) - Catalog
Store Class: org.apache.tajo.catalog.store.HCatalogStore
2013-11-11 14:55:46,141 INFO  service.AbstractService (AbstractService.java:init(81)) - Service:org.apache.tajo.catalog.CatalogServer
is inited.
2013-11-11 14:55:46,141 INFO  service.AbstractService (AbstractService.java:init(81)) - Service:org.apache.tajo.master.GlobalEngine
is inited.
2013-11-11 14:55:46,142 INFO  event.AsyncDispatcher (AsyncDispatcher.java:register(153)) -
Registering class org.apache.tajo.master.querymaster.QueryJobEvent$Type for class org.apache.tajo.master.querymaster.QueryJobManager$QueryJobManagerEventHandler
2013-11-11 14:55:46,143 INFO  service.AbstractService (AbstractService.java:init(81)) - Service:Dispatcher
is inited.
2013-11-11 14:55:46,143 INFO  service.AbstractService (AbstractService.java:init(81)) - Service:org.apache.tajo.master.querymaster.QueryJobManager
is inited.
2013-11-11 14:55:46,143 INFO  service.AbstractService (AbstractService.java:init(81)) - Service:org.apache.tajo.master.TajoMasterClientService
is inited.
2013-11-11 14:55:46,143 INFO  service.AbstractService (AbstractService.java:init(81)) - Service:org.apache.tajo.master.TajoMasterService
is inited.
2013-11-11 14:55:46,143 INFO  service.AbstractService (AbstractService.java:init(81)) - Service:org.apache.tajo.master.TajoMaster
is inited.
2013-11-11 14:55:46,143 INFO  master.TajoMaster (TajoMaster.java:init(177)) - Tajo Master
is initialized.
2013-11-11 14:55:46,143 INFO  master.TajoMaster (TajoMaster.java:start(412)) - TajoMaster
startup
2013-11-11 14:55:46,143 INFO  service.AbstractService (AbstractService.java:start(94)) - Service:Dispatcher
is started.
2013-11-11 14:55:46,201 INFO  rpc.NettyServerBase (NettyServerBase.java:start(106)) - Rpc
(CatalogProtocol) listens on /127.0.0.1:26005
2013-11-11 14:55:46,201 INFO  catalog.CatalogServer (CatalogServer.java:start(156)) - Catalog
Server startup (127.0.0.1:26005)
2013-11-11 14:55:46,201 INFO  service.AbstractService (AbstractService.java:start(94)) - Service:org.apache.tajo.catalog.CatalogServer
is started.
2013-11-11 14:55:46,247 INFO  service.AbstractService (AbstractService.java:start(94)) - Service:org.apache.tajo.master.GlobalEngine
is started.
2013-11-11 14:55:46,247 INFO  service.AbstractService (AbstractService.java:start(94)) - Service:Dispatcher
is started.
2013-11-11 14:55:46,247 INFO  service.AbstractService (AbstractService.java:start(94)) - Service:org.apache.tajo.master.querymaster.QueryJobManager
is started.
2013-11-11 14:55:46,256 INFO  rpc.NettyServerBase (NettyServerBase.java:start(106)) - Rpc
(TajoMasterClientProtocol) listens on /127.0.0.1:26002
2013-11-11 14:55:46,256 INFO  master.TajoMasterClientService (TajoMasterClientService.java:start(90))
- Instantiated TajoMasterClientService at /127.0.0.1:26002
2013-11-11 14:55:46,257 INFO  service.AbstractService (AbstractService.java:start(94)) - Service:org.apache.tajo.master.TajoMasterClientService
is started.
2013-11-11 14:55:46,265 INFO  rpc.NettyServerBase (NettyServerBase.java:start(106)) - Rpc
(TajoMasterProtocol) listens on /127.0.0.1:26001
2013-11-11 14:55:46,265 INFO  master.TajoMasterService (TajoMasterService.java:start(72))
- Instantiated TajoMasterService at /127.0.0.1:26001
2013-11-11 14:55:46,266 INFO  service.AbstractService (AbstractService.java:start(94)) - Service:org.apache.tajo.master.TajoMasterService
is started.
2013-11-11 14:55:46,266 INFO  service.AbstractService (AbstractService.java:start(94)) - Service:org.apache.tajo.master.TajoMaster
is started.
2013-11-11 14:55:46,318 WARN  conf.Configuration (Configuration.java:warnOnceIfDeprecated(817))
- fs.default.name is deprecated. Instead, use fs.defaultFS
2013-11-11 14:55:49,754 INFO  rm.TajoWorkerResourceManager (TajoWorkerResourceManager.java:workerHeartbeat(487))
- TajoWorker:host:localhost, port=28093,28091,28092,50461, slots=1024:1:1, used=0:0:0 added
in live TajoWorker list
2013-11-11 14:56:22,035 INFO  hive.metastore (HiveMetaStoreClient.java:open(244)) - Trying
to connect to metastore with URI thrift://localhost:10001
2013-11-11 14:56:22,049 WARN  hive.metastore (HiveMetaStoreClient.java:open(291)) - Failed
to connect to the MetaStore Server...
2013-11-11 14:56:22,049 INFO  hive.metastore (HiveMetaStoreClient.java:open(322)) - Waiting
1 seconds before next connection attempt.
2013-11-11 14:56:23,049 INFO  hive.metastore (HiveMetaStoreClient.java:open(244)) - Trying
to connect to metastore with URI thrift://localhost:10001
2013-11-11 14:56:23,050 WARN  hive.metastore (HiveMetaStoreClient.java:open(291)) - Failed
to connect to the MetaStore Server...
2013-11-11 14:56:23,050 INFO  hive.metastore (HiveMetaStoreClient.java:open(322)) - Waiting
1 seconds before next connection attempt.
2013-11-11 14:56:24,051 INFO  hive.metastore (HiveMetaStoreClient.java:open(244)) - Trying
to connect to metastore with URI thrift://localhost:10001
2013-11-11 14:56:24,052 WARN  hive.metastore (HiveMetaStoreClient.java:open(291)) - Failed
to connect to the MetaStore Server...
2013-11-11 14:56:24,052 INFO  hive.metastore (HiveMetaStoreClient.java:open(322)) - Waiting
1 seconds before next connection attempt.
2013-11-11 14:56:25,631 INFO  hive.metastore (HiveMetaStoreClient.java:open(244)) - Trying
to connect to metastore with URI thrift://localhost:10001
2013-11-11 14:56:25,631 WARN  hive.metastore (HiveMetaStoreClient.java:open(291)) - Failed
to connect to the MetaStore Server...
2013-11-11 14:56:25,631 INFO  hive.metastore (HiveMetaStoreClient.java:open(322)) - Waiting
1 seconds before next connection attempt.
2013-11-11 14:56:26,631 INFO  hive.metastore (HiveMetaStoreClient.java:open(244)) - Trying
to connect to metastore with URI thrift://localhost:10001
2013-11-11 14:56:26,632 WARN  hive.metastore (HiveMetaStoreClient.java:open(291)) - Failed
to connect to the MetaStore Server...
2013-11-11 14:56:26,632 INFO  hive.metastore (HiveMetaStoreClient.java:open(322)) - Waiting
1 seconds before next connection attempt.
2013-11-11 14:56:27,633 INFO  hive.metastore (HiveMetaStoreClient.java:open(244)) - Trying
to connect to metastore with URI thrift://localhost:10001
2013-11-11 14:56:27,634 WARN  hive.metastore (HiveMetaStoreClient.java:open(291)) - Failed
to connect to the MetaStore Server...
2013-11-11 14:56:27,634 INFO  hive.metastore (HiveMetaStoreClient.java:open(322)) - Waiting
1 seconds before next connection attempt.
2013-11-11 14:56:29,693 INFO  hive.metastore (HiveMetaStoreClient.java:open(244)) - Trying
to connect to metastore with URI thrift://localhost:10001
2013-11-11 14:56:29,694 WARN  hive.metastore (HiveMetaStoreClient.java:open(291)) - Failed
to connect to the MetaStore Server...
2013-11-11 14:56:29,694 INFO  hive.metastore (HiveMetaStoreClient.java:open(322)) - Waiting
1 seconds before next connection attempt.
2013-11-11 14:56:30,694 INFO  hive.metastore (HiveMetaStoreClient.java:open(244)) - Trying
to connect to metastore with URI thrift://localhost:10001
2013-11-11 14:56:30,695 WARN  hive.metastore (HiveMetaStoreClient.java:open(291)) - Failed
to connect to the MetaStore Server...
2013-11-11 14:56:30,695 INFO  hive.metastore (HiveMetaStoreClient.java:open(322)) - Waiting
1 seconds before next connection attempt.
2013-11-11 14:56:31,695 INFO  hive.metastore (HiveMetaStoreClient.java:open(244)) - Trying
to connect to metastore with URI thrift://localhost:10001
2013-11-11 14:56:31,696 WARN  hive.metastore (HiveMetaStoreClient.java:open(291)) - Failed
to connect to the MetaStore Server...
2013-11-11 14:56:31,696 INFO  hive.metastore (HiveMetaStoreClient.java:open(322)) - Waiting
1 seconds before next connection attempt.
2013. 11. 11 오후 2:56:32 org.jboss.netty.channel.DefaultChannelPipeline
경고: An exception was thrown by a user handler while handling an exception event ([id:
0xbe6bfd26, /127.0.0.1:39838 => /127.0.0.1:26002] EXCEPTION: org.apache.tajo.rpc.RemoteCallException:
Remote call error occurs when TajoMasterClientProtocolService.getTableListis called:)
org.apache.tajo.rpc.RemoteException: org.apache.tajo.rpc.RemoteCallException: Remote call
error occurs when TajoMasterClientProtocolService.getTableListis called:
	at org.apache.tajo.rpc.BlockingRpcServer$ServerHandler.exceptionCaught(BlockingRpcServer.java:117)
	at org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:60)
	at org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:377)
	at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525)
	at org.jboss.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:48)
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
	at org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:70)
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
	at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
	at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
	at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
	at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:679)
Caused by: org.apache.tajo.rpc.RemoteCallException: Remote call error occurs when TajoMasterClientProtocolService.getTableListis
called:
	at org.apache.tajo.rpc.BlockingRpcServer$ServerHandler.messageReceived(BlockingRpcServer.java:94)
	... 16 more
Caused by: org.apache.tajo.rpc.RetriesExhaustedException: Failed after attempts=3, exceptions:
java.lang.NullPointerException
java.lang.NullPointerException
java.lang.NullPointerException

	at org.apache.tajo.rpc.ServerCallable.withRetries(ServerCallable.java:91)
	at org.apache.tajo.catalog.AbstractCatalogClient.getAllTableNames(AbstractCatalogClient.java:73)
	at org.apache.tajo.master.TajoMasterClientService$TajoMasterClientProtocolServiceHandler.getTableList(TajoMasterClientService.java:273)
	at org.apache.tajo.ipc.TajoMasterClientProtocol$TajoMasterClientProtocolService$2.callBlockingMethod(TajoMasterClientProtocol.java:244)
	at org.apache.tajo.rpc.BlockingRpcServer$ServerHandler.messageReceived(BlockingRpcServer.java:91)
	... 16 more
2013-11-11 14:56:33,317 ERROR hive.metastore (HiveMetaStoreClient.java:close(346)) - Unable
to shutdown local metastore client
org.apache.thrift.transport.TTransportException: Cannot write to null outputStream
	at org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:142)
	at org.apache.thrift.protocol.TBinaryProtocol.writeI32(TBinaryProtocol.java:163)
	at org.apache.thrift.protocol.TBinaryProtocol.writeMessageBegin(TBinaryProtocol.java:91)
	at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62)
	at com.facebook.fb303.FacebookService$Client.send_shutdown(FacebookService.java:431)
	at com.facebook.fb303.FacebookService$Client.shutdown(FacebookService.java:425)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.close(HiveMetaStoreClient.java:343)
	at org.apache.hcatalog.common.HiveClientCache$CacheableHiveMetaStoreClient.tearDown(HiveClientCache.java:316)
	at org.apache.hcatalog.common.HiveClientCache$CacheableHiveMetaStoreClient.finalize(HiveClientCache.java:331)
	at java.lang.ref.Finalizer.invokeFinalizeMethod(Native Method)
	at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:101)
	at java.lang.ref.Finalizer.access$100(Finalizer.java:32)
	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:190)
{code}

> Enable Tajo catalog to access Hive metastore.
> ---------------------------------------------
>
>                 Key: TAJO-16
>                 URL: https://issues.apache.org/jira/browse/TAJO-16
>             Project: Tajo
>          Issue Type: Improvement
>            Reporter: JaeHwa Jung
>            Assignee: JaeHwa Jung
>             Fix For: 0.8-incubating
>
>         Attachments: TAJO-16.patch, TAJO-16.pdf, TAJO-16_2.pdf
>
>
> Hive has been widely used in this area. Many users have maintained lots of big tables
through Hive metastore. It would be great to enable Tajo catalog to access Hive metastore.
Hive metastore already provides thrift API. We could employ it.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message