Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id B870A200C36 for ; Fri, 10 Mar 2017 08:09:27 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id B716F160B79; Fri, 10 Mar 2017 07:09:27 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id DB829160B69 for ; Fri, 10 Mar 2017 08:09:26 +0100 (CET) Received: (qmail 23577 invoked by uid 500); 10 Mar 2017 07:09:26 -0000 Mailing-List: contact user-help@impala.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@impala.incubator.apache.org Delivered-To: mailing list user@impala.incubator.apache.org Received: (qmail 23566 invoked by uid 99); 10 Mar 2017 07:09:25 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Mar 2017 07:09:25 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 8A8181A7ACD for ; Fri, 10 Mar 2017 07:09:25 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 4.005 X-Spam-Level: **** X-Spam-Status: No, score=4.005 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HK_RANDOM_ENVFROM=0.626, HK_RANDOM_FROM=0.999, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001, WEIRD_PORT=0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id xKPXz6LY2uvv for ; Fri, 10 Mar 2017 07:09:24 +0000 (UTC) Received: from mail-oi0-f51.google.com (mail-oi0-f51.google.com [209.85.218.51]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 4B3A25F252 for ; Fri, 10 Mar 2017 07:09:23 +0000 (UTC) Received: by mail-oi0-f51.google.com with SMTP id 62so47716352oih.2 for ; Thu, 09 Mar 2017 23:09:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=yuJNVTRlkxOD/wSOyB8THcJS/OgM/dd04SgVj3X1Z3o=; b=CDsOj0ZD/N+QRFXX1nY5j/KNX27awiiU1DJ6tn6CWG9nqWjrZLv06R0v/QMhS4OmBH BA4NSaWUj/TUNbqK1AiEa26BT/155f2c0HT8b5yjTSzn96kE8zXQz8v/jxnfzNZ5B4+S lF9lFf1OAY8nIZJnddHOqd4lfQ9nqPkX/Ewz/8OJwXMAa6hrVG3Ta2gikJqkTrl0gFOB gfmdtHaoXdXh2IHAGI4zxhssutkntCCvkbf7MqLQjk3VsjjvD0sJAiB4HcdJFDatYotm efta8AxbG3rqv8PlzufgCbcC7LcHqqDKH9HRqXo7qjurjql56EVPCnM9Vi+nXNXcxh68 8MDQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=yuJNVTRlkxOD/wSOyB8THcJS/OgM/dd04SgVj3X1Z3o=; b=C5j16hISMauizkhhRLfycb6R5fWrTzNo6tQQSUrdnA6HWaYFHwa6AsSoOj3oqd6ywb u8VrzJn7alA0vjCxvlP8XX/1D2/+/7OuJPGE1USKoCwJazycP0LI1oi8sXewGSjgVYWp 1+vSVIgnykQl+OrvZ9otfHqxQ5j8c0hoFeKgJy8OdZf+0xUS4K2MrID93gDCyOvEBTgL gT2orT64s7SmyR5iinKUH9bPqNJA4mcPDPpKSLBgGPyVjxsPvkCirDukaaAXNiYAgrfT qfrz4R1Ih8mOLjei12JufWmuA3hpjG40G0eXgnrAtWrqThPbZrtxdGhNLVn3J/lIp7fH wmJg== X-Gm-Message-State: AMke39mlpEWmlUjVBokkRPkt5XUxxqMmu1cNhHY+gWhdSFTfflyAW9uOCDJXn7LMtyUkoNPeA0iGz+38LWN7UA== X-Received: by 10.202.83.5 with SMTP id h5mr7953063oib.19.1489129762113; Thu, 09 Mar 2017 23:09:22 -0800 (PST) MIME-Version: 1.0 Received: by 10.74.14.73 with HTTP; Thu, 9 Mar 2017 23:09:21 -0800 (PST) In-Reply-To: References: From: =?UTF-8?B?5L+K5p2w6ZmI?= Date: Fri, 10 Mar 2017 15:09:21 +0800 Message-ID: Subject: Re: Impala Failed to read file from HDFS To: user@impala.incubator.apache.org Content-Type: multipart/alternative; boundary=001a113d2cf67ecea1054a5b0a35 archived-at: Fri, 10 Mar 2017 07:09:27 -0000 --001a113d2cf67ecea1054a5b0a35 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Plus: In my root directory I found user/hive/warehouse/parquet_data.db/test/2.parquet. So it seems impalad is manipulating on local file system. How do I configure this? 2017-03-10 15:03 GMT+08:00 =E4=BF=8A=E6=9D=B0=E9=99=88 = : > Thanks from quick reply:) > > 1.parquet is always in the hdfs. I also did following command for you > reference, please note the URI which is start with file:. It looks weird. > > [bdpe30-cjj:21000] > use parquet_data; > Query: use parquet_data > [bdpe30-cjj:21000] > load data inpath "hdfs:///data/2.parquet" into table > test; > Query: load data inpath "hdfs:///data/2.parquet" into table test > +----------------------------------------------------------+ > | summary | > +----------------------------------------------------------+ > | Loaded 1 file(s). Total files in destination location: 2 | > +----------------------------------------------------------+ > Fetched 1 row(s) in 0.50s > [bdpe30-cjj:21000] > select count(*) from test; > Query: select count(*) from test > Query submitted at: 2017-03-10 07:14:45 (Coordinator: > http://bdpe30-cjj:25000) > Query progress can be monitored at: http://bdpe30-cjj:25000/query_ > plan?query_id=3D5d4ecce7d21182cc:e2dd7f5700000000 > WARNINGS: > Failed to open HDFS file *file:*/user/hive/warehouse/ > parquet_data.db/test/1.parquet > Error(2): No such file or directory > > > It seems like the load operation read data from hdfs, but not put into > right place for query. Also the impalad seems access the file in local fi= le > system. > > > 2017-03-10 14:48 GMT+08:00 Jeszy : > >> Hello, >> >> Sounds like Impala expected 1.parquet to be in the folder, but it wasn't= . >> You probably forgot to do 'refresh ' after altering data from >> the outside. >> >> HTH >> >> On Fri, Mar 10, 2017 at 7:30 AM, =E4=BF=8A=E6=9D=B0=E9=99=88 wrote: >> > Hi, >> > I'm using latest impala built from github, and setup impala cluster >> with >> > 2-nodes like below: >> > node-1: statestored, catalogd, namenode,datanode. >> > node-2: impalad, datanode. >> > >> > Then I created database and table, loaded data from external parquet >> file >> > into table. Everything was OK, but when I executed a query it failed >> with >> > following message: >> > >> > Failed to open HDFS file >> > file:/user/hive/warehouse/parquet_data.db/test/1.parquet >> > Error(2): No such file or directory >> > >> > But I can still =E2=80=98desc test=E2=80=99. Anyone met with this? Tha= nks in advanced. >> > >> > >> > >> > -- >> > Thanks & Best Regards >> > > > > -- > Thanks & Best Regards > --=20 Thanks & Best Regards --001a113d2cf67ecea1054a5b0a35 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Plus:

In my root directory I found=C2= =A0user/hive/warehouse/parquet_data.db/test/2.parquet. So it seems im= palad is manipulating on local file system.=C2=A0 How do I configure this?= =C2=A0

2017-03-10 15:03 GMT+08:00 =E4=BF=8A=E6=9D=B0=E9=99=88 <cjjnjust@= gmail.com>:
Thanks from quick reply:)

1.parquet is always in the h= dfs. I also did following command for you reference, please note the URI wh= ich is start with file:. It looks weird.

[bdp= e30-cjj:21000] > use parquet_data;
Query: use parquet_data
[bdpe30-cjj:21000] > load data inpath "hdfs:///data/2.parque= t" into table test;
Query: load data inpath "hdfs:///da= ta/2.parquet" into table test
+-----------------------------= -----------------------------+
| summary =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0|
+-----------------------------------------------= -----------+
| Loaded 1 file(s). Total files in destination locat= ion: 2 |
+--------------------------------------------------= --------+
Fetched 1 row(s) in 0.50s
[bdpe30-cjj:21000] = > select count(*) from test;
Query: select count(*) from test<= /div>
Query submitted at: 2017-03-10 07:14:45 (Coordinator: http://bdpe30-cjj:25000)
WARNINGS:
F= ailed to open HDFS file file:/user/hive/warehouse/parquet_data.db/test/1.parquet
Error(2= ): No such file or directory


It seems like the load operation read data from hdfs, but not put in= to right place for query. Also the impalad seems access the file in local f= ile system.=C2=A0


2017-03-= 10 14:48 GMT+08:00 Jeszy <jeszyb@gmail.com>:
Hello,

Sounds like Impala expected 1.parquet to be in the folder, but it wasn'= t.
You probably forgot to do 'refresh <table>' after altering da= ta from
the outside.

HTH

On Fri, Mar 10, 2017 at 7:30 AM, =E4=BF=8A=E6=9D=B0=E9=99=88 <cjjnjust@gmail.com> wr= ote:
> Hi,
> I'm using latest impala built from github,=C2=A0 and setup impala = cluster with
> 2-nodes like below:
> node-1: statestored, catalogd, namenode,datanode.
> node-2: impalad, datanode.
>
> Then I created database and table, loaded data from external parquet f= ile
> into table. Everything was OK, but when I executed a query it failed w= ith
> following message:
>
> Failed to open HDFS file
> file:/user/hive/warehouse/parquet_data.db/test/1.parquet
> Error(2): No such file or directory
>
> But I can still =E2=80=98desc test=E2=80=99. Anyone met with this? Tha= nks in advanced.
>
>
>
> --
> Thanks & Best Regards



--
=
Thanks & Best Regards



--
=
Thanks &a= mp; Best Regards
--001a113d2cf67ecea1054a5b0a35--