spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hyukjin Kwon (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SPARK-21255) NPE when creating encoder for enum
Date Fri, 30 Jun 2017 09:12:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-21255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16069754#comment-16069754
] 

Hyukjin Kwon edited comment on SPARK-21255 at 6/30/17 9:11 AM:
---------------------------------------------------------------

Thanks for cc'ing me. It looks this was introduced long time ago in the first place.
I double checked {{declaringClass}} and https://docs.oracle.com/javase/8/docs/api/java/lang/Enum.html
and it works if we filter out. It sounds good to me too.

The reproducer I used is as below:

{code}
  public enum A {
    B("www.google.com");

    private String url;

    private A(String url) {
      this.url = url;
    }

    public String getUrl() {
      return url;
    }

    public void setUrl(String url) {
      this.url = url;
    }
  }

  @Test
  public void testEnum() {
    List<A> data = Arrays.asList(A.B);
    spark.createDataFrame(data, A.class).show();
  }
{code}

After the change:

{code}
+--------------+
|           url|
+--------------+
|www.google.com|
+--------------+
{code}



was (Author: hyukjin.kwon):
Thanks for cc'ing me. It looks this was introduced long time ago in the first place.
I double checked {{declaringClass}} and https://docs.oracle.com/javase/8/docs/api/java/lang/Enum.html
and it works if we filter out. It sounds good to me too.

The reproducer I used is as below:

{code}

public enum A {
  B("www.google.com");

  private String url;

  private A(String url) {
    this.url = url;
  }

  public String getUrl() {
    return url;
  }

  public void setUrl(String url) {
    return url;
  }
}

@Test
public void testEnum() {
  List<A> data = Arrays.asList(A.B);
  spark.createDataFrame(data, A.class).show();
}
{code}


> NPE when creating encoder for enum
> ----------------------------------
>
>                 Key: SPARK-21255
>                 URL: https://issues.apache.org/jira/browse/SPARK-21255
>             Project: Spark
>          Issue Type: Bug
>          Components: Java API
>    Affects Versions: 2.1.0
>         Environment: org.apache.spark:spark-core_2.10:2.1.0
> org.apache.spark:spark-sql_2.10:2.1.0
>            Reporter: Mike
>
> When you try to create an encoder for Enum type (or bean with enum property) via Encoders.bean(...),
it fails with NullPointerException at TypeToken:495.
> I did a little research and it turns out, that in JavaTypeInference:126 following code

> {code:java}
> val beanInfo = Introspector.getBeanInfo(typeToken.getRawType)
> val properties = beanInfo.getPropertyDescriptors.filterNot(_.getName == "class")
> val fields = properties.map { property =>
>           val returnType = typeToken.method(property.getReadMethod).getReturnType
>           val (dataType, nullable) = inferDataType(returnType)
>           new StructField(property.getName, dataType, nullable)
>         }
> (new StructType(fields), true)
> {code}
> filters out properties named "class", because we wouldn't want to serialize that. But
enum types have another property of type Class named "declaringClass", which we are trying
to inspect recursively. Eventually we try to inspect ClassLoader class, which has property
"defaultAssertionStatus" with no read method, which leads to NPE at TypeToken:495.
> I think adding property name "declaringClass" to filtering will resolve this.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message