hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Elliot West <tea...@gmail.com>
Subject Re: Using s3 as warehouse on emr
Date Fri, 22 Jan 2016 13:11:11 GMT
Related to this, might it be better to use the s3a protocol instead of s3n?
https://wiki.apache.org/hadoop/AmazonS3

Additionally, can anyone advise when EMRFS is required when storing Hive
tables in S3?
http://docs.aws.amazon.com/ElasticMapReduce/latest/ManagementGuide/emr-overview-arch.html#emr-arch-storage

On 22 January 2016 at 12:52, Zsolt Tóth <toth.zsolt.bme@gmail.com> wrote:

> Hi,
>
> I'd like to use S3 as the hive warehouse on my emr 4.x cluster.
> I've set hive.metastore.warehouse.dir=s3n://testbucket/hive_warehouse and
> fs.s3.impl=org.apache.hadoop.fs.s3native.NativeS3FileSystem (not sure if
> this is needed) in the hive-site.xml on the master node. Double checked the
> "set -v" output, the properties are correct.
>
> When I run a command like "create table test1 (x String);" in Hive CLI, it
> is created in the default warehouse dir (/user/hive/warehouse/) instead of
> s3n://...
>
> What am I missing here?
>
> Thanks!
> Zsolt
>

Mime
View raw message