hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nick Dimiduk <ndimi...@gmail.com>
Subject Re: s3n with hbase
Date Sat, 01 Nov 2014 00:40:16 GMT
Please don't do this. S3 is not a strongly consistent filesystem. HBase
will not be happy there. Better to run on HDFS and to snapshots/copytable
backup, restore to S3.

On Fri, Oct 31, 2014 at 4:53 PM, Khaled Elmeleegy <kdiaa@hotmail.com> wrote:

> Hi,
>
> I am trying to use hbase with s3, using s3n, but I get the below errors,
> when starting the master. I am testing this in a pseudo distributed mode on
> my laptop.
> I've also set hbase.rootdir to s3n://
> kdiaa-hbase.s3-us-west-2.amazonaws.com:80/root, where the corresponding
> bucket and directory are already created on s3. I've also set
> fs.s3n.awsAccessKeyId, and fs.s3n.awsSecretAccessKey to the appropriate
> values in hbase-site.xml
>
> So, I must be missing something. Any advice is appreciated.
>
>
>
> 2014-10-31 16:47:15,312 WARN  [master:172.16.209.239:60000]
> httpclient.RestS3Service: Response '/root' - Unexpected response code 404,
> expected 200
> 2014-10-31 16:47:15,349 WARN  [master:172.16.209.239:60000]
> httpclient.RestS3Service: Response '/root_%24folder%24' - Unexpected
> response code 404, expected 200
> 2014-10-31 16:47:15,420 WARN  [master:172.16.209.239:60000]
> httpclient.RestS3Service: Response '/' - Unexpected response code 404,
> expected 200
> 2014-10-31 16:47:15,420 WARN  [master:172.16.209.239:60000]
> httpclient.RestS3Service: Response '/' - Received error response with XML
> message
> 2014-10-31 16:47:15,601 FATAL [master:172.16.209.239:60000]
> master.HMaster: Unhandled exception. Starting shutdown.
> org.apache.hadoop.fs.s3.S3Exception:
> org.jets3t.service.S3ServiceException: S3 GET failed for '/' XML Error
> Message: <?xml version="1.0"
> encoding="UTF-8"?><Error><Code>NoSuchBucket</Code><Message>The
specified
> bucket does not exist</Message><BucketName>
> kdiaa-hbase.s3-us-west-2.amazonaws.com
> </BucketName><RequestId>1589CC5DB70ED750</RequestId><HostId>cb2ZGGlNkxtf5fredweXt/wxJlAHLkioUJC86pkh0JxQfBJ1CMYoZuxHU1g+CnTB</HostId></Error>
>         at
> org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.handleServiceException(Jets3tNativeFileSystemStore.java:245)
>         at
> org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.list(Jets3tNativeFileSystemStore.java:181)
>         at
> org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.list(Jets3tNativeFileSystemStore.java:158)
>         at
> org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.list(Jets3tNativeFileSystemStore.java:151)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
>         at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
>         at org.apache.hadoop.fs.s3native.$Proxy9.list(Unknown Source)
>         at
> org.apache.hadoop.fs.s3native.NativeS3FileSystem.getFileStatus(NativeS3FileSystem.java:432)
>         at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1397)
>         at
> org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:439)
>         at
> org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:147)
>         at
> org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:128)
>         at
> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:802)
>         at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:615)
>         at java.lang.Thread.run(Thread.java:695)
> Caused by: org.jets3t.service.S3ServiceException: S3 GET failed for '/'
> XML Error Message: <?xml version="1.0"
> encoding="UTF-8"?><Error><Code>NoSuchBucket</Code><Message>The
specified
> bucket does not exist</Message><BucketName>
> kdiaa-hbase.s3-us-west-2.amazonaws.com
> </BucketName><RequestId>1589CC5DB70ED750</RequestId><HostId>cb2ZGGlNkxtf5fredweXt/wxJlAHLkioUJC86pkh0JxQfBJ1CMYoZuxHU1g+CnTB</HostId></Error>
>         at
> org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:424)
>         at
> org.jets3t.service.impl.rest.httpclient.RestS3Service.performRestGet(RestS3Service.java:686)
>         at
> org.jets3t.service.impl.rest.httpclient.RestS3Service.listObjectsInternal(RestS3Service.java:1083)
>         at
> org.jets3t.service.impl.rest.httpclient.RestS3Service.listObjectsChunkedImpl(RestS3Service.java:1053)
>         at
> org.jets3t.service.S3Service.listObjectsChunked(S3Service.java:1333)
>         at
> org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.list(Jets3tNativeFileSystemStore.java:168)
>         ... 17 more
> 2014-10-31 16:47:15,604 INFO  [master:172.16.209.239:60000]
> master.HMaster: Aborting
> 2014-10-31 16:47:15,612 DEBUG [master:172.16.209.239:60000]
> master.HMaster: Stopping service threads
> 2014-10-31 16:47:15,612 INFO  [master:172.16.209.239:60000]
> ipc.RpcServer: Stopping server on 60000
> 2014-10-31 16:47:15,612 INFO  [RpcServer.listener,port=60000]
> ipc.RpcServer: RpcServer.listener,port=60000: stopping
> 2014-10-31 16:47:15,619 INFO  [master:172.16.209.239:60000]
> master.HMaster: Stopping infoServer
> 2014-10-31 16:47:15,633 INFO  [RpcServer.responder] ipc.RpcServer:
> RpcServer.responder: stopped
> 2014-10-31 16:47:15,633 INFO  [RpcServer.responder] ipc.RpcServer:
> RpcServer.responder: stopping
> 2014-10-31 16:47:15,660 INFO  [master:172.16.209.239:60000] mortbay.log:
> Stopped SelectChannelConnector@0.0.0.0:60010
> 2014-10-31 16:47:15,804 INFO  [master:172.16.209.239:60000]
> zookeeper.ZooKeeper: Session: 0x149689a7dd80000 closed
> 2014-10-31 16:47:15,804 INFO  [main-EventThread] zookeeper.ClientCnxn:
> EventThread shut down
> 2014-10-31 16:47:15,804 INFO  [master:172.16.209.239:60000]
> master.HMaster: HMaster main thread exiting
> 2014-10-31 16:47:15,804 ERROR [main] master.HMasterCommandLine: Master
> exiting
> java.lang.RuntimeException: HMaster Aborted
>         at
> org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:194)
>         at
> org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:135)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>         at
> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
>         at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2803)
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message