impala-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "anujphadke (Code Review)" <ger...@cloudera.org>
Subject [Impala-CR](cdh5-trunk) IMPALA-1850: Allow fs.defaultFS to be set to a non-HDFS filesystem
Date Wed, 20 Apr 2016 19:40:18 GMT
anujphadke has posted comments on this change.

Change subject: IMPALA-1850: Allow fs.defaultFS to be set to a non-HDFS filesystem
......................................................................


Patch Set 20:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/1121/20/be/src/util/hdfs-util.cc
File be/src/util/hdfs-util.cc:

Line 75:   return strncmp(path, "s3a://", 6) == 0;
> shouldn't we do the same default_fs check here?  how are both of these case
Yes, I will make the change here too.

I am not sure how to create a setup where I can generate different combinations of qualified
and unqualified paths for s3 and hdfs to verify the API change

So I hardcoded different combinations of FS paths and set the configured_defaultFs to s3a/hdfs
and verified this

I0420 12:29:25.167100 25288 exec-env.cc:425] Configured defaultFS: hdfs://localhost:20500
I0420 12:29:25.167109 25288 exec-env.cc:426] hdfs://test : 1
I0420 12:29:25.167112 25288 exec-env.cc:427] test : 1
I0420 12:29:25.167114 25288 exec-env.cc:428] testFile : 1
I0420 12:29:25.167115 25288 exec-env.cc:429] s3a://test : 0
I0420 12:29:25.167116 25288 exec-env.cc:431] Configured defaultFS: s3a://
I0420 12:29:25.167117 25288 exec-env.cc:432] hdfs://test : 1
I0420 12:29:25.167119 25288 exec-env.cc:433] test : 0
I0420 12:29:25.167120 25288 exec-env.cc:434] testFile : 0
I0420 12:29:25.167121 25288 exec-env.cc:435] s3a://test : 0. 

On my private build before running the tests , I also added logs where these functions were
invoked to check the return value. It worked fine since for s3 the paths were always appended
with the s3a://


-- 
To view, visit http://gerrit.cloudera.org:8080/1121
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I2f45bef6c94ece634045acb906d12591587ccfed
Gerrit-PatchSet: 20
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: anujphadke <aphadke@cloudera.com>
Gerrit-Reviewer: Alex Behm <alex.behm@cloudera.com>
Gerrit-Reviewer: Casey Ching <casey@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dhecht@cloudera.com>
Gerrit-Reviewer: Henry Robinson <henry@cloudera.com>
Gerrit-Reviewer: Juan Yu <jyu@cloudera.com>
Gerrit-Reviewer: Sailesh Mukil <sailesh@cloudera.com>
Gerrit-Reviewer: anujphadke <aphadke@cloudera.com>
Gerrit-HasComments: Yes

Mime
View raw message