Return-Path: X-Original-To: apmail-hbase-user-archive@www.apache.org Delivered-To: apmail-hbase-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id EED8617F54 for ; Fri, 31 Oct 2014 23:54:23 +0000 (UTC) Received: (qmail 91107 invoked by uid 500); 31 Oct 2014 23:54:22 -0000 Delivered-To: apmail-hbase-user-archive@hbase.apache.org Received: (qmail 91031 invoked by uid 500); 31 Oct 2014 23:54:21 -0000 Mailing-List: contact user-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hbase.apache.org Delivered-To: mailing list user@hbase.apache.org Received: (qmail 91015 invoked by uid 99); 31 Oct 2014 23:54:21 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Oct 2014 23:54:21 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=5.0 tests=RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of kdiaa@hotmail.com designates 157.55.1.153 as permitted sender) Received: from [157.55.1.153] (HELO DUB004-OMC2S14.hotmail.com) (157.55.1.153) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Oct 2014 23:53:55 +0000 Received: from DUB128-W67 ([157.55.1.138]) by DUB004-OMC2S14.hotmail.com over TLS secured channel with Microsoft SMTPSVC(7.5.7601.22751); Fri, 31 Oct 2014 16:53:54 -0700 X-TMN: [FvMMvWRFYXfuorsWxWX6yJsoiEV0zrjK] X-Originating-Email: [kdiaa@hotmail.com] Message-ID: From: Khaled Elmeleegy To: "user@hbase.apache.org" Subject: s3n with hbase Date: Fri, 31 Oct 2014 16:53:54 -0700 Importance: Normal Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginalArrivalTime: 31 Oct 2014 23:53:54.0576 (UTC) FILETIME=[EDB77900:01CFF565] X-Virus-Checked: Checked by ClamAV on apache.org Hi=2C=0A= =0A= I am trying to use hbase with s3=2C using s3n=2C but I get the below errors= =2C when starting the master. I am testing this in a pseudo distributed mod= e on my laptop.=0A= I've also set hbase.rootdir to s3n://kdiaa-hbase.s3-us-west-2.amazonaws.com= :80/root=2C where the corresponding bucket and directory are already create= d on s3. I've also set fs.s3n.awsAccessKeyId=2C and=A0fs.s3n.awsSecretAcces= sKey to the appropriate values in=A0hbase-site.xml=0A= =0A= So=2C I must be missing something. Any advice is appreciated.=0A= =0A= =0A= =0A= 2014-10-31 16:47:15=2C312 WARN =A0[master:172.16.209.239:60000] httpclient.= RestS3Service: Response '/root' - Unexpected response code 404=2C expected = 200=0A= 2014-10-31 16:47:15=2C349 WARN =A0[master:172.16.209.239:60000] httpclient.= RestS3Service: Response '/root_%24folder%24' - Unexpected response code 404= =2C expected 200=0A= 2014-10-31 16:47:15=2C420 WARN =A0[master:172.16.209.239:60000] httpclient.= RestS3Service: Response '/' - Unexpected response code 404=2C expected 200= =0A= 2014-10-31 16:47:15=2C420 WARN =A0[master:172.16.209.239:60000] httpclient.= RestS3Service: Response '/' - Received error response with XML message=0A= 2014-10-31 16:47:15=2C601 FATAL [master:172.16.209.239:60000] master.HMaste= r: Unhandled exception. Starting shutdown.=0A= org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException:= S3 GET failed for '/' XML Error Message: NoSuchBucketThe specified bucket does= not existkdiaa-hbase.s3-us-west-2.amazonaws.com1589CC5DB70ED750cb2ZGGlNkxtf5fredweX= t/wxJlAHLkioUJC86pkh0JxQfBJ1CMYoZuxHU1g+CnTB=0A= =A0 =A0 =A0 =A0 at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStor= e.handleServiceException(Jets3tNativeFileSystemStore.java:245)=0A= =A0 =A0 =A0 =A0 at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStor= e.list(Jets3tNativeFileSystemStore.java:181)=0A= =A0 =A0 =A0 =A0 at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStor= e.list(Jets3tNativeFileSystemStore.java:158)=0A= =A0 =A0 =A0 =A0 at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStor= e.list(Jets3tNativeFileSystemStore.java:151)=0A= =A0 =A0 =A0 =A0 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Meth= od)=0A= =A0 =A0 =A0 =A0 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethod= AccessorImpl.java:39)=0A= =A0 =A0 =A0 =A0 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Delegati= ngMethodAccessorImpl.java:25)=0A= =A0 =A0 =A0 =A0 at java.lang.reflect.Method.invoke(Method.java:597)=0A= =A0 =A0 =A0 =A0 at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke= Method(RetryInvocationHandler.java:186)=0A= =A0 =A0 =A0 =A0 at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke= (RetryInvocationHandler.java:102)=0A= =A0 =A0 =A0 =A0 at org.apache.hadoop.fs.s3native.$Proxy9.list(Unknown Sourc= e)=0A= =A0 =A0 =A0 =A0 at org.apache.hadoop.fs.s3native.NativeS3FileSystem.getFile= Status(NativeS3FileSystem.java:432)=0A= =A0 =A0 =A0 =A0 at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1= 397)=0A= =A0 =A0 =A0 =A0 at org.apache.hadoop.hbase.master.MasterFileSystem.checkRoo= tDir(MasterFileSystem.java:439)=0A= =A0 =A0 =A0 =A0 at org.apache.hadoop.hbase.master.MasterFileSystem.createIn= itialFileSystemLayout(MasterFileSystem.java:147)=0A= =A0 =A0 =A0 =A0 at org.apache.hadoop.hbase.master.MasterFileSystem.(M= asterFileSystem.java:128)=0A= =A0 =A0 =A0 =A0 at org.apache.hadoop.hbase.master.HMaster.finishInitializat= ion(HMaster.java:802)=0A= =A0 =A0 =A0 =A0 at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:= 615)=0A= =A0 =A0 =A0 =A0 at java.lang.Thread.run(Thread.java:695)=0A= Caused by: org.jets3t.service.S3ServiceException: S3 GET failed for '/' XML= Error Message: NoSu= chBucketThe specified bucket does not existkdiaa-hbase.s3-us-west-2.amazonaws.com1589CC5= DB70ED750cb2ZGGlNkxtf5fredweXt/wxJlAHLkioUJC86pkh0JxQfB= J1CMYoZuxHU1g+CnTB=0A= =A0 =A0 =A0 =A0 at org.jets3t.service.impl.rest.httpclient.RestS3Service.pe= rformRequest(RestS3Service.java:424)=0A= =A0 =A0 =A0 =A0 at org.jets3t.service.impl.rest.httpclient.RestS3Service.pe= rformRestGet(RestS3Service.java:686)=0A= =A0 =A0 =A0 =A0 at org.jets3t.service.impl.rest.httpclient.RestS3Service.li= stObjectsInternal(RestS3Service.java:1083)=0A= =A0 =A0 =A0 =A0 at org.jets3t.service.impl.rest.httpclient.RestS3Service.li= stObjectsChunkedImpl(RestS3Service.java:1053)=0A= =A0 =A0 =A0 =A0 at org.jets3t.service.S3Service.listObjectsChunked(S3Servic= e.java:1333)=0A= =A0 =A0 =A0 =A0 at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStor= e.list(Jets3tNativeFileSystemStore.java:168)=0A= =A0 =A0 =A0 =A0 ... 17 more=0A= 2014-10-31 16:47:15=2C604 INFO =A0[master:172.16.209.239:60000] master.HMas= ter: Aborting=0A= 2014-10-31 16:47:15=2C612 DEBUG [master:172.16.209.239:60000] master.HMaste= r: Stopping service threads=0A= 2014-10-31 16:47:15=2C612 INFO =A0[master:172.16.209.239:60000] ipc.RpcServ= er: Stopping server on 60000=0A= 2014-10-31 16:47:15=2C612 INFO =A0[RpcServer.listener=2Cport=3D60000] ipc.R= pcServer: RpcServer.listener=2Cport=3D60000: stopping=0A= 2014-10-31 16:47:15=2C619 INFO =A0[master:172.16.209.239:60000] master.HMas= ter: Stopping infoServer=0A= 2014-10-31 16:47:15=2C633 INFO =A0[RpcServer.responder] ipc.RpcServer: RpcS= erver.responder: stopped=0A= 2014-10-31 16:47:15=2C633 INFO =A0[RpcServer.responder] ipc.RpcServer: RpcS= erver.responder: stopping=0A= 2014-10-31 16:47:15=2C660 INFO =A0[master:172.16.209.239:60000] mortbay.log= : Stopped SelectChannelConnector@0.0.0.0:60010=0A= 2014-10-31 16:47:15=2C804 INFO =A0[master:172.16.209.239:60000] zookeeper.Z= ooKeeper: Session: 0x149689a7dd80000 closed=0A= 2014-10-31 16:47:15=2C804 INFO =A0[main-EventThread] zookeeper.ClientCnxn: = EventThread shut down=0A= 2014-10-31 16:47:15=2C804 INFO =A0[master:172.16.209.239:60000] master.HMas= ter: HMaster main thread exiting=0A= 2014-10-31 16:47:15=2C804 ERROR [main] master.HMasterCommandLine: Master ex= iting=0A= java.lang.RuntimeException: HMaster Aborted=0A= =A0 =A0 =A0 =A0 at org.apache.hadoop.hbase.master.HMasterCommandLine.startM= aster(HMasterCommandLine.java:194)=0A= =A0 =A0 =A0 =A0 at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HM= asterCommandLine.java:135)=0A= =A0 =A0 =A0 =A0 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70= )=0A= =A0 =A0 =A0 =A0 at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(Se= rverCommandLine.java:126)=0A= =A0 =A0 =A0 =A0 at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java= :2803)=0A= =0A= =