impala-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 黄志 <huangzhi19921...@gmail.com>
Subject Re: Impala-Alluxio
Date Thu, 15 Dec 2016 06:15:45 GMT
Hi, I  see the fe module uses hadoop-aws.jar to support S3, and I see the 
wrapper class in it, can I use the same way to use Alluxio? and is there 
any need to add code in Impala?

在 2016年12月15日星期四 UTC+8下午1:07:02,黄志写道:
>
> I see it!Thanks,I want to know how Impala uses S3, I did't see the code in 
> Impala about choosing s3 and invoke s3's API
>
> 在 2016年12月14日星期三 UTC+8下午11:47:33,Tim Armstrong写道:
>>
>> See https://wiki.apache.org/hadoop/AmazonS3 and S3AFilesystem.java in 
>> the Hadoop source tree.
>>
>> It looks like Alluxio also has a similar filesystem implementation. I 
>> don't know anything about Alluxio or how well it works in practce, just 
>> giving pointers in case you are interested in developing the support.
>>
>> On Tue, Dec 13, 2016 at 10:01 PM, 黄志 <huangzhi...@gmail.com> wrote:
>>
>>> Hi, I am interested in this, I saw Impala use boto3 to implement 
>>> S3Client, but Alluxio only offer Java API,how do we use it?and where is 
>>> S3AFileSystem?
>>>
>>> 在 2016年12月9日星期五 UTC+8上午1:34:42,Tim Armstrong写道:
>>>>
>>>> It looks like Alluxio provides some kind of Hadoop Filesystem API 
>>>> wrapper: 
>>>> http://www.alluxio.org/docs/master/en/File-System-API.html#hadoop-api. 
>>>> I don't know if it's been upstreamed into Apache Hadoop.
>>>>
>>>> We do use similar Hadoop Filesystem API adapters to provide S3 and 
>>>> Isilon support without massive code changes so it seems likely that it's

>>>> possible to do the same for Alluxio. Our experience with S3 and Isilon was

>>>> that it required additional work to get it to work and perform well because

>>>> different filesystems have different quirks even if they implement the same

>>>> interface.
>>>>
>>>> It would require at least some small code changes to switch it on (e.g. 
>>>> you can grep for S3AFilesystem to see how we turned on S3 support). Please

>>>> let us know if you experiment with it and have any success (or no success).
>>>>
>>>> - Tim
>>>>
>>>> On Thu, Dec 8, 2016 at 9:07 AM, Jim Apple <jba...@cloudera.com> wrote:
>>>>
>>>>> I don't know that Impala has been tested with Alluxio.
>>>>>
>>>>> This mailing list is deprecated; please use 
>>>>> d...@impala.incubator.apache.org
>>>>>
>>>>>
>>>>> On Wednesday, December 7, 2016 at 11:08:59 PM UTC-8, 
>>>>> andong....@gmail.com wrote:
>>>>>>
>>>>>> Hi
>>>>>> How are you configure the impala on the alluxio;
>>>>>>
>>>>>> Alluxio: Open Source Memory Speed Virtual Distributed Storage
>>>>>> Alluxio: http://alluxio.org/
>>>>>>
>>>>> -- 
>>>>> You received this message because you are subscribed to the Google 
>>>>> Groups "Impala Dev" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send

>>>>> an email to impala-dev+...@cloudera.org.
>>>>>
>>>>
>>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "Impala Dev" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to impala-dev+...@cloudera.org.
>>>
>>
>>
Mime
  • Unnamed multipart/mixed (inline, None, 0 bytes)
View raw message