impala-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Piyush Narang <p.nar...@criteo.com>
Subject Re: Using Impala with a federated HDFS setup
Date Wed, 06 Dec 2017 22:02:08 GMT
Thanks a ton Tim.

-- Piyush


From: Tim Armstrong <tarmstrong@cloudera.com>
Reply-To: "user@impala.apache.org" <user@impala.apache.org>
Date: Wednesday, December 6, 2017 at 4:58 PM
To: "user@impala.apache.org" <user@impala.apache.org>
Subject: Re: Using Impala with a federated HDFS setup

Hi Piyush,

 I opened a new JIRA to track the feature: https://issues.apache.org/jira/browse/IMPALA-6287<https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_IMPALA-2D6287&d=DwMFaQ&c=nxfEpP1JWHVKAq835DW4mA&r=3Ka-O_qIfLiCDaGELmIN3BcChZatNdPOwe36odQXFYo&m=Q6LUDgSZqoREcltq-zZoI7IGacUvW30CJegQebTeuzM&s=FKMdRh_Eyx4UJFSAiT8O-92zvCMsyVmrG6m1vsl31Vs&e=>.
- Tim

On Tue, Dec 5, 2017 at 7:07 AM, Piyush Narang <p.narang@criteo.com<mailto:p.narang@criteo.com>>
wrote:
Thanks Tim. Yeah I was able to get it to work by pointing directly to the hdfs:// URL for
the data we want to query. For our initial Impala experiments we should be able to proceed.
Do you know if there’s a jira tracking the long term fix? (or is that going to continue
to be Impala-77?). Based on our initial experiments, we might be open to helping out with
the fix, we can chime in on the jira in the next couple of weeks as well.

Thanks,

-- Piyush


From: Tim Armstrong <tarmstrong@cloudera.com<mailto:tarmstrong@cloudera.com>>
Reply-To: "user@impala.apache.org<mailto:user@impala.apache.org>" <user@impala.apache.org<mailto:user@impala.apache.org>>
Date: Monday, December 4, 2017 at 8:35 PM
To: "user@impala.apache.org<mailto:user@impala.apache.org>" <user@impala.apache.org<mailto:user@impala.apache.org>>
Subject: Re: Using Impala with a federated HDFS setup

Hi Piyush,
  You're right that we don't support ViewFileSystem at the moment. It looks like IMPALA-77
was resolved by failing more gracefully for viewfs.

I suspect it just needs some targeted code changes and testing - the logic for different filesystems
is mostly the same with some tweaks. I'm not sure when this will happen - it will likely largely
depend on when someone steps forward to do it.
I believe pointing directly to the hdfs:// URLs should work.
 - Tim

On Thu, Nov 30, 2017 at 11:39 AM, Piyush Narang <p.narang@criteo.com<mailto:p.narang@criteo.com>>
wrote:
Hi folks,

Our company is looking to experiment with setting up Impala for some of our adhoc query workloads.
I was working on setting up Impala to test things out and I ran into the following errors
on startup, “Currently configured default filesystem: ViewFileSystem. fs.defaultFS (viewfs://root)
is not supported.”. Noticed that this has been implemented as part of this jira: https://issues.apache.org/jira/browse/IMPALA-77<https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_IMPALA-2D77&d=DwMFaQ&c=nxfEpP1JWHVKAq835DW4mA&r=3Ka-O_qIfLiCDaGELmIN3BcChZatNdPOwe36odQXFYo&m=qT5IbyXEu2Sy58jS9NeuJSnIOLpbD0hRJb4qI6t5_IY&s=GMdxzTI4bDF8FUK-J5SzOIyWrzk-S6NsEH0rVKTHKLg&e=>.
Is this still currently not on the Impala roadmap? Are there any possible workarounds for
users with federated namenodes?

The data we want to query as of now resides in one namenode’s namespace. A potential workaround
for us might be to just expose that hdfs:// mount point directly. Not sure if anyone’s tried
this and what kinds of issues they’ve run into.

Thanks,
Piyush


Mime
View raw message