impala-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Armstrong <tarmstr...@cloudera.com>
Subject Re: Impala-Alluxio
Date Wed, 14 Dec 2016 15:47:01 GMT
See https://wiki.apache.org/hadoop/AmazonS3 and S3AFilesystem.java in the
Hadoop source tree.

It looks like Alluxio also has a similar filesystem implementation. I don't
know anything about Alluxio or how well it works in practce, just giving
pointers in case you are interested in developing the support.

On Tue, Dec 13, 2016 at 10:01 PM, 黄志 <huangzhi19921023@gmail.com> wrote:

> Hi, I am interested in this, I saw Impala use boto3 to implement S3Client,
> but Alluxio only offer Java API,how do we use it?and where is S3AFileSystem?
>
> 在 2016年12月9日星期五 UTC+8上午1:34:42,Tim Armstrong写道:
>>
>> It looks like Alluxio provides some kind of Hadoop Filesystem API
>> wrapper: http://www.alluxio.org/docs/master/en/File-System-
>> API.html#hadoop-api. I don't know if it's been upstreamed into Apache
>> Hadoop.
>>
>> We do use similar Hadoop Filesystem API adapters to provide S3 and Isilon
>> support without massive code changes so it seems likely that it's possible
>> to do the same for Alluxio. Our experience with S3 and Isilon was that it
>> required additional work to get it to work and perform well because
>> different filesystems have different quirks even if they implement the same
>> interface.
>>
>> It would require at least some small code changes to switch it on (e.g.
>> you can grep for S3AFilesystem to see how we turned on S3 support). Please
>> let us know if you experiment with it and have any success (or no success).
>>
>> - Tim
>>
>> On Thu, Dec 8, 2016 at 9:07 AM, Jim Apple <jba...@cloudera.com> wrote:
>>
>>> I don't know that Impala has been tested with Alluxio.
>>>
>>> This mailing list is deprecated; please use
>>> d...@impala.incubator.apache.org
>>>
>>>
>>> On Wednesday, December 7, 2016 at 11:08:59 PM UTC-8,
>>> andong....@gmail.com wrote:
>>>>
>>>> Hi
>>>> How are you configure the impala on the alluxio;
>>>>
>>>> Alluxio: Open Source Memory Speed Virtual Distributed Storage
>>>> Alluxio: http://alluxio.org/
>>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Impala Dev" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to impala-dev+...@cloudera.org.
>>>
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Impala Dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to impala-dev+unsubscribe@cloudera.org.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message