From user-return-710-archive-asf-public=cust-asf.ponee.io@impala.apache.org Wed Mar 7 18:35:46 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 219C7180676 for ; Wed, 7 Mar 2018 18:35:45 +0100 (CET) Received: (qmail 31202 invoked by uid 500); 7 Mar 2018 17:35:44 -0000 Mailing-List: contact user-help@impala.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@impala.apache.org Delivered-To: mailing list user@impala.apache.org Received: (qmail 31192 invoked by uid 99); 7 Mar 2018 17:35:44 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Mar 2018 17:35:44 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 2332E1A0183 for ; Wed, 7 Mar 2018 17:35:44 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.879 X-Spam-Level: * X-Spam-Status: No, score=1.879 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=cloudera.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id 1GknW3foTdqL for ; Wed, 7 Mar 2018 17:35:42 +0000 (UTC) Received: from mail-qt0-f171.google.com (mail-qt0-f171.google.com [209.85.216.171]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 586285F2EC for ; Wed, 7 Mar 2018 17:35:42 +0000 (UTC) Received: by mail-qt0-f171.google.com with SMTP id y6so3592129qtm.7 for ; Wed, 07 Mar 2018 09:35:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cloudera.com; s=google; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=CxA8qgq8mSS/iHQ61ZGvN1ayJgfSngEQqzVWrIzq8ac=; b=QzOiOEorLKb3EEgeMLzZ3xxvtcGZB2193T0pAkszNnTbdMpgxwMvdPweX1EJh9A3ip XkFHhl/NVgx1znRNacETq+RWefwozsi8RDnyug195m224LDa9XcwKPESmFG7v++JSKjb l7sBzdAeNqAqLDkNL44Dpp0MbabGh066GKsS7l1uQZGWvcAInFX7yVv2UOd+w4046GRy iRQXiaqqTyURXSnGMoltmw7BXBsqV27AZ9Sb30r4o4N6XLivhN81QSIalXMESFbO/2QU jv6YbxYto/+vvE8YYMHoA7SiQbZIoR99bUC1dhTQj9faX4ktYWOeNNkXnEjNtL2jTzPw Xx3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=CxA8qgq8mSS/iHQ61ZGvN1ayJgfSngEQqzVWrIzq8ac=; b=eQvCHAwl/DKoqH7B4ljYQOfbFzXwYQwFTRjqBhscC89XoLRDdWWjtlLQPQKazpfhdp vfo90uFBgK8IXqbeBh4lvVdGlUvlgoABAhbs7WPWN+jt2gJt/2YlH4IZahV74hdw+AAx NtUqi1vfvIY352yHequOzCIlP8DxabIiKrY2cIqgv0lNhEQ7WKMLbQ/i3t5UBzWZ9//R klYztKVtzgI7qiXsGXSvj1OjMYLJI21Jngd4VTA++0/tjrsSEnb22v4/PvrNVk43Sgd9 QW9foArcIIKbDMPcvx3Qqx00s4j1foZTzO86FMD42oSYysToIExr29d9zaXUrJDIJhCw 4Y5Q== X-Gm-Message-State: AElRT7HkMJl4CXsBUBsV2cx2LME+0f4/HQ6hr0NIEAZPDqFOLy8pgHHv WEMgc1osqEbGbwuWshWCtV1pSJma5rSd4NR7ldQnbbGN X-Google-Smtp-Source: AG47ELueWCfegmp5DLRdOu7eXF1shZ8Si1Py0y/Ek8LtW7+68/XedoeGvQGfS9jbOOQLaqHeKjtj34Ko2mVW58zitCM= X-Received: by 10.200.45.35 with SMTP id n32mr35425993qta.92.1520444141831; Wed, 07 Mar 2018 09:35:41 -0800 (PST) MIME-Version: 1.0 Received: by 10.237.36.103 with HTTP; Wed, 7 Mar 2018 09:35:01 -0800 (PST) In-Reply-To: <0972EE4B-CEBF-43B4-AAF1-E6A92C5CC07D@criteo.com> References: <0972EE4B-CEBF-43B4-AAF1-E6A92C5CC07D@criteo.com> From: Vuk Ercegovac Date: Wed, 7 Mar 2018 09:35:01 -0800 Message-ID: Subject: Re: Debugging ClassNotFound errors while trying to use Hive UDFs To: user@impala.apache.org Content-Type: multipart/alternative; boundary="001a1143ca74f9ad900566d5fc89" --001a1143ca74f9ad900566d5fc89 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable An issue that we're working on is described here: https://issues.apache.org/jira/browse/IMPALA-6215. For your scenario, does the ClassNotFoundException come up deterministically for the complex query or just once in a while? When you test this query, is it the only query running on the system? If you don't mind sharing the query so that I can develop a repro, that would be appreciated. On Wed, Mar 7, 2018 at 2:57 AM, Piyush Narang wrote: > Hi folks, > > > > I=E2=80=99m running into a strange ClassNotFound error while trying to us= e one of > my Hive UDFs in Impala. > > I=E2=80=99ve defined the UDF: > > > create function device_type(string, string, string) returns string > location 'my/path/to/my-udf-SNAPSHOT.jar' symbol=3D'com.criteo.hadoop. > hive.udf.UDFUserAgentToDeviceType'; > > > > I try to use it in simple queries and it works fine: > > > select device_type("a", "b", "c"); > > Query: select device_type("a", "b", "c") > > =E2=80=A6 > > +----------------------------------------+ > > | device_type('a', 'b', 'c') | > > +----------------------------------------+ > > | Mobile - Other | > > +----------------------------------------+ > > Fetched 1 row(s) in 0.01s > > > select ua_device_family, ua_browser_family, ua_os_family, > device_type(ua_device_family, ua_browser_family, ua_os_family) from > bi_arbitrage_full limit 1; > > Query: select ua_device_family, ua_browser_family, ua_os_family, > device_type(ua_device_family, ua_browser_family, ua_os_family) from > bi_arbitrage_full limit 1 > > =E2=80=A6 > > +------------------+-------------------+--------------+----- > -----------------------------------------------------------------------+ > > | ua_device_family | ua_browser_family | ua_os_family | > device_type(ua_device_family, ua_browser_family, ua_os_family) | > > +------------------+-------------------+--------------+----- > -----------------------------------------------------------------------+ > > | iPhone | mobile safari | iOS | > iPhone > | > > +------------------+-------------------+--------------+----- > -----------------------------------------------------------------------+ > > Fetched 1 row(s) in 0.73s > > > > Now when I try to run a more complex query (same database) > > I get an error: > > WARNINGS: ImpalaRuntimeException: Unable to find class. > > CAUSED BY: ClassNotFoundException: com.criteo.hadoop.hive.udf. > UDFUserAgentToDeviceType > > > > I turned up the log level on the coordinator to debug. I see the calls > being made to load the UDF and the use of the UDF in the query but no > details on the ClassNotFound. > > > > Has anyone run into a similar issue? We=E2=80=99re running Impala 2.11.0-= cdh5.14.0 > RELEASE (build d68206561bce6b26762d62c01a78e6cd27aa7690) so I think we > should hopefully be clear of this really old bug - > https://issues.apache.org/jira/browse/IMPALA-695. > > Thanks, > > > > -- Piyush > > > --001a1143ca74f9ad900566d5fc89 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
An issue that we're working on is described here:=C2= =A0https://is= sues.apache.org/jira/browse/IMPALA-6215.

For your scenario= , does the ClassNotFoundException come up deterministically for the complex= query or just
once in a while? When you test this query, is it t= he only query running on the system?

If you don= 9;t mind sharing the query so that I can develop a repro, that would be app= reciated.

On Wed, Mar 7, 2018 at 2:57 AM, Piyush Narang &= lt;p.narang@criteo= .com> wrote:

Hi folks,=

=C2=A0

I=E2=80=99m running= into a strange ClassNotFound error while trying to use one of my Hive UDFs= in Impala.

I=E2=80=99ve define= d the UDF:

> create functio= n device_type(string, string, string) returns string location 'my/path/= to/my-udf-SNAPSHOT.jar' symbol=3D'com.criteo.hadoop.hive.= udf.UDFUserAgentToDeviceType';

=C2=A0

I try to use it in = simple queries and it works fine:

> select device_= type("a", "b", "c");

Query: select devic= e_type("a", "b", "c")

=E2=80=A6=

+------------------= ----------------------+

| device_type('= a', 'b', 'c') |

+------------------= ----------------------+

| Mobile - Other=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 |=

+------------------= ----------------------+

Fetched 1 row(s) in= 0.01s

> select ua_devi= ce_family, ua_browser_family, ua_os_family, device_type(ua_device_family, u= a_browser_family, ua_os_family) from bi_arbitrage_full limit 1;

Query: select ua_de= vice_family, ua_browser_family, ua_os_family, device_type(ua_device_family,= ua_browser_family, ua_os_family) from bi_arbitrage_full limit 1<= /u>

=E2=80=A6=

+------------------= +-------------------+--------------+-----------------------------= -----------------------------------------------+

| ua_device_family = | ua_browser_family | ua_os_family | device_type(ua_device_family, ua_brows= er_family, ua_os_family) |

+------------------= +-------------------+--------------+-----------------------------= -----------------------------------------------+

| iPhone=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 | mobile safari=C2=A0= =C2=A0=C2=A0=C2=A0 | iOS=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 | iPhone=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 |

+------------------= +-------------------+--------------+-----------------------------= -----------------------------------------------+

Fetched 1 row(s) in= 0.73s

=C2=A0

Now when I try to r= un a more complex query (same database)

I get an error:<= /u>

WARNINGS: ImpalaRun= timeException: Unable to find class.

CAUSED BY: ClassNot= FoundException: com.criteo.hadoop.hive.udf.UDFUserAgentToDeviceType=

=C2=A0

I turned up the log= level on the coordinator to debug. I see the calls being made to load the = UDF and the use of the UDF in the query but no details on the ClassNotFound= .

=C2=A0

Has anyone run into= a similar issue? We=E2=80=99re running Impala 2.11.0-cdh5.14.0 RELEASE (bu= ild d68206561bce6b26762d62c01a78e6cd27aa7690) so I think we should hop= efully be clear of this really old bug - https://issues.apache.org/jira/browse/IMPALA-695.

Thanks,

=

=C2=A0

--=C2=A0Piyush

=C2=A0


--001a1143ca74f9ad900566d5fc89--