spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From maropu <...@git.apache.org>
Subject [GitHub] spark pull request: [SPARK-6747] [SQL] Throw an AnalysisException ...
Date Tue, 07 Jul 2015 00:37:41 GMT
GitHub user maropu opened a pull request:

    https://github.com/apache/spark/pull/7248

    [SPARK-6747] [SQL] Throw an AnalysisException when unsupported Java list types used in
Hive UDF

    The current implementation can't handle List<> as a return type in Hive UDF and
    throws meaningless Match Error.
    We assume an UDF below;
    public class UDFToListString extends UDF {
    public List<String> evaluate(Object o)
    { return Arrays.asList("xxx", "yyy", "zzz"); }
    }
    An exception of scala.MatchError is thrown as follows when the UDF used;
    scala.MatchError: interface java.util.List (of class java.lang.Class)
    at org.apache.spark.sql.hive.HiveInspectors$class.javaClassToDataType(HiveInspectors.scala:174)
    at org.apache.spark.sql.hive.HiveSimpleUdf.javaClassToDataType(hiveUdfs.scala:76)
    at org.apache.spark.sql.hive.HiveSimpleUdf.dataType$lzycompute(hiveUdfs.scala:106)
    at org.apache.spark.sql.hive.HiveSimpleUdf.dataType(hiveUdfs.scala:106)
    at org.apache.spark.sql.catalyst.expressions.Alias.toAttribute(namedExpressions.scala:131)
    at org.apache.spark.sql.catalyst.planning.PhysicalOperation$$anonfun$collectAliases$1.applyOrElse(patterns.scala:95)
    at org.apache.spark.sql.catalyst.planning.PhysicalOperation$$anonfun$collectAliases$1.applyOrElse(patterns.scala:94)
    at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33)
    at scala.collection.TraversableLike$$anonfun$collect$1.apply(TraversableLike.scala:278)
    ...
    To make udf developers more understood, we need to throw a more suitable exception.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/maropu/spark FixBugInHiveInspectors

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/7248.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #7248
    
----
commit e553f10c67f129e59c526d6149e60c77348f546a
Author: Takeshi YAMAMURO <linguin.m.s@gmail.com>
Date:   2015-04-07T16:41:17Z

    Support List as a return type in Hive UDF

commit e21ce7e33139b96581dbd37785f99854e60379fd
Author: Takeshi YAMAMURO <linguin.m.s@gmail.com>
Date:   2015-04-08T02:18:22Z

    Add a blank line at the end of UDFToListString

commit 94064169380d129a41574e7ee3ccc994532f71f0
Author: Takeshi YAMAMURO <linguin.m.s@gmail.com>
Date:   2015-04-14T16:46:44Z

    Apply review comments

commit f965c34bcea8efa6b58443ef31c4d857b5feb09e
Author: Takeshi YAMAMURO <linguin.m.s@gmail.com>
Date:   2015-04-14T18:04:31Z

    Fix code-style errors

commit 1c7b9d1aae14de43ca9bf6e3414a45f1104e5043
Author: Takeshi YAMAMURO <linguin.m.s@gmail.com>
Date:   2015-04-17T00:51:14Z

    Remove a new type

commit a4887127834fc73063851fd57f095ff212f9b483
Author: Takeshi YAMAMURO <linguin.m.s@gmail.com>
Date:   2015-05-15T04:36:50Z

    Add StringToUtf8 to comvert String into UTF8String

commit 21e8763ccba103187561c64a2fa6e38286ff13b1
Author: Takeshi YAMAMURO <linguin.m.s@gmail.com>
Date:   2015-05-15T07:18:40Z

    Add TODO comments in UDFToListString of HiveUdfSuite

commit 1e823167986113cc264be13bd22c631eeb1452e4
Author: Takeshi YAMAMURO <linguin.m.s@gmail.com>
Date:   2015-05-16T02:58:46Z

    Apply comments

commit ee232db94ffcb08b8cd33b94d03543bb4a663b60
Author: Takeshi YAMAMURO <linguin.m.s@gmail.com>
Date:   2015-04-07T16:41:17Z

    Support List as a return type in Hive UDF

commit 93e3d4e7e1fc93e95ce09e01318e3f6fb7639d8b
Author: Takeshi YAMAMURO <linguin.m.s@gmail.com>
Date:   2015-04-08T02:18:22Z

    Add a blank line at the end of UDFToListString

commit 6984bf4067ef8d98c21097cd2b2265aece3b8a87
Author: Takeshi YAMAMURO <linguin.m.s@gmail.com>
Date:   2015-04-14T16:46:44Z

    Apply review comments

commit 7f812fd8f6d07f6779011e2e98b631733264f2a9
Author: Takeshi YAMAMURO <linguin.m.s@gmail.com>
Date:   2015-04-14T18:04:31Z

    Fix code-style errors

commit af61f2ecc821cede51a0bb446ba67230d39b3f10
Author: Takeshi YAMAMURO <linguin.m.s@gmail.com>
Date:   2015-04-17T00:51:14Z

    Remove a new type

commit fdb2ae4a01b6cea154311632078f8fb3026ce945
Author: Takeshi YAMAMURO <linguin.m.s@gmail.com>
Date:   2015-05-15T04:36:50Z

    Add StringToUtf8 to comvert String into UTF8String

commit 7114a4735a4ccbd1211b10fa757f0e27012ef96d
Author: Takeshi YAMAMURO <linguin.m.s@gmail.com>
Date:   2015-05-15T07:18:40Z

    Add TODO comments in UDFToListString of HiveUdfSuite

commit 2844a8e6c4cbfafafadff7e072a971c6656ae113
Author: Takeshi YAMAMURO <linguin.m.s@gmail.com>
Date:   2015-05-16T02:58:46Z

    Apply comments

commit 92ed7a6710ced76823365f493f753bc6ca947b30
Author: Takeshi YAMAMURO <linguin.m.s@gmail.com>
Date:   2015-07-06T23:11:33Z

    Throw an exception when java list type used

commit 56305deddff7b7db1d800d72a2a43b09cbf127cc
Author: Takeshi YAMAMURO <linguin.m.s@gmail.com>
Date:   2015-07-06T23:43:01Z

    Fix conflicts

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message