hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Collins <chris_j_coll...@yahoo.com>
Subject Re: example usage of s3 file system
Date Wed, 29 Aug 2012 15:44:08 GMT
No problem.  Didnt get anything from the issues email alias.  Seems that also the url that
Haavard pointed to probably also needs to say what jets3t jar file to include.  People probably
have a tendency to want to use the latest version (which is 9) where you seem to have to use
7.x to get hadoop not to give a method invocation exception.  I am guessing it cant be that
important though cause for the life of me I cant understand how it ever worked for native
(well at least doing a mkdir).

C
On Aug 29, 2012, at 5:48 AM, Harsh J <harsh@cloudera.com> wrote:

> Many thanks for taking this upstream with a fix Chris!
> 
> On Wed, Aug 29, 2012 at 12:51 PM, Chris Collins
> <chris_j_collins@yahoo.com> wrote:
>> Thanks Haavard, I am aware of that page but I am not sure why you are pointing me
to it.  This really looks like a bug where Jets3tNativeFileSystemStore is parsing a response
from jets3t.  its looking for ResponseCode=404 but actually getting ResponseCode: 404.  I
dont see how it ever worked looking back through the versions of release code.
>> 
>> I took a copy of the s3native package and made a fix and seems to get around the
issue.
>> 
>> I have reported it to the issues email alias, I will see if there is actually any
interest in this problem.
>> 
>> Cheers
>> 
>> C
>> On Aug 29, 2012, at 12:11 AM, Håvard Wahl Kongsgård <haavard.kongsgaard@gmail.com>
wrote:
>> 
>>> see also
>>> 
>>> http://wiki.apache.org/hadoop/AmazonS3
>>> 
>>> On Tue, Aug 28, 2012 at 9:14 AM, Chris Collins
>>> <chris_j_collins@yahoo.com> wrote:
>>>> Hi I am trying to use the Hadoop filesystem abstraction with S3 but in my
tinkering I am not having a great deal of success.  I am particularly interested in the ability
to mimic a directory structure (since s3 native doesnt do it).
>>>> 
>>>> Can anyone point me to some good example usage of Hadoop FileSystem with
s3?
>>>> 
>>>> I created a few directories using transit and AWS S3 console for test.  Doing
a liststatus of the bucket returns a FileStatus object of the directory created but if I try
to do a liststatus of that path I am getting a 404:
>>>> 
>>>> org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException:
Request Error. HEAD '/aaaa' on Host ....
>>>> 
>>>> Probably not the best list to look for help, any clues appreciated.
>>>> 
>>>> C
>>> 
>>> 
>>> 
>>> --
>>> Håvard Wahl Kongsgård
>>> Faculty of Medicine &
>>> Department of Mathematical Sciences
>>> NTNU
>>> 
>>> http://havard.security-review.net/
>> 
> 
> 
> 
> -- 
> Harsh J


Mime
View raw message