hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Purtell <apurt...@apache.org>
Subject Re: s3n with hbase
Date Fri, 07 Nov 2014 18:16:10 GMT
And note this is any file, potentially, table descriptors, what have you.

S3 isn't a filesystem, we can't pretend it is one.

On Fri, Nov 7, 2014 at 10:13 AM, Andrew Purtell <apurtell@apache.org> wrote:

> Admittedly it's been *years* since I experimented with pointing a HBase
> root at a s3 or s3n filesystem, but my (dated) experience is it could take
> some time for newly written objects to show up in a bucket. The write will
> have completed and the file will be closed, but upon immediate open attempt
> the client would get 500s from the S3 service. HBase expects to be able to
> read back files it has just written during flushing, compaction, and the
> like. When S3 confounds this expectation, and it did so for me quite often,
> the RegionServer aborts.
>
>
> On Thu, Nov 6, 2014 at 11:04 PM, Nick Dimiduk <ndimiduk@gmail.com> wrote:
>
>> I have not created an exhaustive checklist of the DFS feature requirements
>> of HBase. Would be an informative exercise and could be used to drive
>> toward broader support. My understanding is that these are fundamental
>> problems in the semantics of S3. As I mentioned earlier, I believe the
>> Azue
>> folks have been through this list in order to support HBase on their
>> service.
>>
>> On Sat, Nov 1, 2014 at 4:29 PM, Khaled Elmeleegy <kdiaa@hotmail.com>
>> wrote:
>>
>> > True. HDFS has strong consistency model, whereas S3 has a more relaxed
>> > consistency model -- eventual consistency. Do we know though, what's
>> about
>> > eventual consistency that breaks HBase? Is it a fundamental problem
>> that we
>> > can't fix or it's just an implementation issue that could be fixed. If
>> it's
>> > the later, then I think it's good to weight the cost of fixing this
>> problem
>> > against the benifit of being able to deploy on S3 directly. So any
>> insights
>> > there would be highly appreciated.
>> > Best,Khaled
>> >
>> > > From: ndimiduk@gmail.com
>> > > Date: Sat, 1 Nov 2014 12:36:40 -0700
>> > > Subject: Re: s3n with hbase
>> > > To: user@hbase.apache.org
>> > >
>> > > It's a reliability/stability problem. the S3 implementation of the FS
>> > > doesn't provide the characteristics we rely on because S3 doesn't have
>> > > these characteristics. It may be that there are improvements to be
>> made
>> > in
>> > > the s3 or s3n drivers, but I believe there's a fundamental difference
>> in
>> > > the semantics of S3 compared to HDFS. Agreed that this would be
>> > incredibly
>> > > useful deployment model. I've heard you can run HBase directly against
>> > the
>> > > Azue equivalent of S3, if you're willing to try a different cloud.
>> > Haven't
>> > > tried it myself.
>> > >
>> > > On Sat, Nov 1, 2014 at 12:21 PM, Khaled Elmeleegy <kdiaa@hotmail.com>
>> > wrote:
>> > >
>> > > > So, is it a performance problem? or a correctness problem, e.g.
>> reads
>> > may
>> > > > have stale values? or a reliability problem, where not strongly
>> > consistent
>> > > > file system can drive HBase unstable introducing failures?
>> > > > I can live with a performance hit, given the simplicity of running
>> > > > directly on S3. Obviously, I can't take a reliability hit though.
>> > > >
>> > > > > From: ndimiduk@gmail.com
>> > > > > Date: Fri, 31 Oct 2014 17:40:16 -0700
>> > > > > Subject: Re: s3n with hbase
>> > > > > To: user@hbase.apache.org
>> > > > >
>> > > > > Please don't do this. S3 is not a strongly consistent filesystem.
>> > HBase
>> > > > > will not be happy there. Better to run on HDFS and to
>> > snapshots/copytable
>> > > > > backup, restore to S3.
>> > > > >
>> > > > > On Fri, Oct 31, 2014 at 4:53 PM, Khaled Elmeleegy <
>> kdiaa@hotmail.com
>> > >
>> > > > wrote:
>> > > > >
>> > > > > > Hi,
>> > > > > >
>> > > > > > I am trying to use hbase with s3, using s3n, but I get the
below
>> > > > errors,
>> > > > > > when starting the master. I am testing this in a pseudo
>> distributed
>> > > > mode on
>> > > > > > my laptop.
>> > > > > > I've also set hbase.rootdir to s3n://
>> > > > > > kdiaa-hbase.s3-us-west-2.amazonaws.com:80/root, where the
>> > > > corresponding
>> > > > > > bucket and directory are already created on s3. I've also
set
>> > > > > > fs.s3n.awsAccessKeyId, and fs.s3n.awsSecretAccessKey to
the
>> > appropriate
>> > > > > > values in hbase-site.xml
>> > > > > >
>> > > > > > So, I must be missing something. Any advice is appreciated.
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > 2014-10-31 16:47:15,312 WARN  [master:172.16.209.239:60000]
>> > > > > > httpclient.RestS3Service: Response '/root' - Unexpected
response
>> > code
>> > > > 404,
>> > > > > > expected 200
>> > > > > > 2014-10-31 16:47:15,349 WARN  [master:172.16.209.239:60000]
>> > > > > > httpclient.RestS3Service: Response '/root_%24folder%24'
-
>> > Unexpected
>> > > > > > response code 404, expected 200
>> > > > > > 2014-10-31 16:47:15,420 WARN  [master:172.16.209.239:60000]
>> > > > > > httpclient.RestS3Service: Response '/' - Unexpected response
>> code
>> > 404,
>> > > > > > expected 200
>> > > > > > 2014-10-31 16:47:15,420 WARN  [master:172.16.209.239:60000]
>> > > > > > httpclient.RestS3Service: Response '/' - Received error
response
>> > with
>> > > > XML
>> > > > > > message
>> > > > > > 2014-10-31 16:47:15,601 FATAL [master:172.16.209.239:60000]
>> > > > > > master.HMaster: Unhandled exception. Starting shutdown.
>> > > > > > org.apache.hadoop.fs.s3.S3Exception:
>> > > > > > org.jets3t.service.S3ServiceException: S3 GET failed for
'/' XML
>> > Error
>> > > > > > Message: <?xml version="1.0"
>> > > > > > encoding="UTF-8"?><Error><Code>NoSuchBucket</Code><Message>The
>> > > > specified
>> > > > > > bucket does not exist</Message><BucketName>
>> > > > > > kdiaa-hbase.s3-us-west-2.amazonaws.com
>> > > > > >
>> > > >
>> >
>> </BucketName><RequestId>1589CC5DB70ED750</RequestId><HostId>cb2ZGGlNkxtf5fredweXt/wxJlAHLkioUJC86pkh0JxQfBJ1CMYoZuxHU1g+CnTB</HostId></Error>
>> > > > > >         at
>> > > > > >
>> > > >
>> >
>> org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.handleServiceException(Jets3tNativeFileSystemStore.java:245)
>> > > > > >         at
>> > > > > >
>> > > >
>> >
>> org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.list(Jets3tNativeFileSystemStore.java:181)
>> > > > > >         at
>> > > > > >
>> > > >
>> >
>> org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.list(Jets3tNativeFileSystemStore.java:158)
>> > > > > >         at
>> > > > > >
>> > > >
>> >
>> org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.list(Jets3tNativeFileSystemStore.java:151)
>> > > > > >         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> > Method)
>> > > > > >         at
>> > > > > >
>> > > >
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> > > > > >         at
>> > > > > >
>> > > >
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > > > > >         at java.lang.reflect.Method.invoke(Method.java:597)
>> > > > > >         at
>> > > > > >
>> > > >
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
>> > > > > >         at
>> > > > > >
>> > > >
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
>> > > > > >         at org.apache.hadoop.fs.s3native.$Proxy9.list(Unknown
>> > Source)
>> > > > > >         at
>> > > > > >
>> > > >
>> >
>> org.apache.hadoop.fs.s3native.NativeS3FileSystem.getFileStatus(NativeS3FileSystem.java:432)
>> > > > > >         at
>> > org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1397)
>> > > > > >         at
>> > > > > >
>> > > >
>> >
>> org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:439)
>> > > > > >         at
>> > > > > >
>> > > >
>> >
>> org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:147)
>> > > > > >         at
>> > > > > >
>> > > >
>> >
>> org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:128)
>> > > > > >         at
>> > > > > >
>> > > >
>> >
>> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:802)
>> > > > > >         at
>> > org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:615)
>> > > > > >         at java.lang.Thread.run(Thread.java:695)
>> > > > > > Caused by: org.jets3t.service.S3ServiceException: S3 GET
failed
>> > for '/'
>> > > > > > XML Error Message: <?xml version="1.0"
>> > > > > > encoding="UTF-8"?><Error><Code>NoSuchBucket</Code><Message>The
>> > > > specified
>> > > > > > bucket does not exist</Message><BucketName>
>> > > > > > kdiaa-hbase.s3-us-west-2.amazonaws.com
>> > > > > >
>> > > >
>> >
>> </BucketName><RequestId>1589CC5DB70ED750</RequestId><HostId>cb2ZGGlNkxtf5fredweXt/wxJlAHLkioUJC86pkh0JxQfBJ1CMYoZuxHU1g+CnTB</HostId></Error>
>> > > > > >         at
>> > > > > >
>> > > >
>> >
>> org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:424)
>> > > > > >         at
>> > > > > >
>> > > >
>> >
>> org.jets3t.service.impl.rest.httpclient.RestS3Service.performRestGet(RestS3Service.java:686)
>> > > > > >         at
>> > > > > >
>> > > >
>> >
>> org.jets3t.service.impl.rest.httpclient.RestS3Service.listObjectsInternal(RestS3Service.java:1083)
>> > > > > >         at
>> > > > > >
>> > > >
>> >
>> org.jets3t.service.impl.rest.httpclient.RestS3Service.listObjectsChunkedImpl(RestS3Service.java:1053)
>> > > > > >         at
>> > > > > >
>> > org.jets3t.service.S3Service.listObjectsChunked(S3Service.java:1333)
>> > > > > >         at
>> > > > > >
>> > > >
>> >
>> org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.list(Jets3tNativeFileSystemStore.java:168)
>> > > > > >         ... 17 more
>> > > > > > 2014-10-31 16:47:15,604 INFO  [master:172.16.209.239:60000]
>> > > > > > master.HMaster: Aborting
>> > > > > > 2014-10-31 16:47:15,612 DEBUG [master:172.16.209.239:60000]
>> > > > > > master.HMaster: Stopping service threads
>> > > > > > 2014-10-31 16:47:15,612 INFO  [master:172.16.209.239:60000]
>> > > > > > ipc.RpcServer: Stopping server on 60000
>> > > > > > 2014-10-31 16:47:15,612 INFO  [RpcServer.listener,port=60000]
>> > > > > > ipc.RpcServer: RpcServer.listener,port=60000: stopping
>> > > > > > 2014-10-31 16:47:15,619 INFO  [master:172.16.209.239:60000]
>> > > > > > master.HMaster: Stopping infoServer
>> > > > > > 2014-10-31 16:47:15,633 INFO  [RpcServer.responder]
>> ipc.RpcServer:
>> > > > > > RpcServer.responder: stopped
>> > > > > > 2014-10-31 16:47:15,633 INFO  [RpcServer.responder]
>> ipc.RpcServer:
>> > > > > > RpcServer.responder: stopping
>> > > > > > 2014-10-31 16:47:15,660 INFO  [master:172.16.209.239:60000]
>> > > > mortbay.log:
>> > > > > > Stopped SelectChannelConnector@0.0.0.0:60010
>> > > > > > 2014-10-31 16:47:15,804 INFO  [master:172.16.209.239:60000]
>> > > > > > zookeeper.ZooKeeper: Session: 0x149689a7dd80000 closed
>> > > > > > 2014-10-31 16:47:15,804 INFO  [main-EventThread]
>> > zookeeper.ClientCnxn:
>> > > > > > EventThread shut down
>> > > > > > 2014-10-31 16:47:15,804 INFO  [master:172.16.209.239:60000]
>> > > > > > master.HMaster: HMaster main thread exiting
>> > > > > > 2014-10-31 16:47:15,804 ERROR [main] master.HMasterCommandLine:
>> > Master
>> > > > > > exiting
>> > > > > > java.lang.RuntimeException: HMaster Aborted
>> > > > > >         at
>> > > > > >
>> > > >
>> >
>> org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:194)
>> > > > > >         at
>> > > > > >
>> > > >
>> >
>> org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:135)
>> > > > > >         at
>> > org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>> > > > > >         at
>> > > > > >
>> > > >
>> >
>> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
>> > > > > >         at
>> > > > org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2803)
>> > > > > >
>> > > > > >
>> > > >
>> > > >
>> >
>> >
>>
>
>
>
> --
> Best regards,
>
>    - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>



-- 
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message