Mailing-List: contact dev-help@phoenix.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@phoenix.apache.org
Date: Thu, 23 Mar 2017 17:27:41 +0000 (UTC)
From: "Nan Xu (JIRA)" <jira@apache.org>
To: dev@phoenix.apache.org
Message-ID: <JIRA.13058675.1490289962000.104023.1490290061820@Atlassian.JIRA>
In-Reply-To: <JIRA.13058675.1490289962000@Atlassian.JIRA>
References: <JIRA.13058675.1490289962000@Atlassian.JIRA> <JIRA.13058675.1490289962767@jira-lw-us.apache.org>
Subject: [jira] [Updated] (PHOENIX-3751) spark 2.1 with Phoenix 4.10 load
 data as dataframe fail, NullPointerException
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Thu, 23 Mar 2017 17:27:48 -0000


     [ https://issues.apache.org/jira/browse/PHOENIX-3751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nan Xu updated PHOENIX-3751:
----------------------------
    Summary: spark 2.1 with Phoenix 4.10 load data as dataframe fail, NullPointerException  (was: spark 2.1 with Phoenix 4.10 load data as dataframe fail)

> spark 2.1 with Phoenix 4.10 load data as dataframe fail, NullPointerException
> -----------------------------------------------------------------------------
>
>                 Key: PHOENIX-3751
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-3751
>             Project: Phoenix
>          Issue Type: Bug
>    Affects Versions: 4.10.0
>         Environment: HBase 1.14
> spark 2.10
> phoenix: 4.10
>            Reporter: Nan Xu
>
> create phoenix table: 
> create table phoenix.quote (  
> sym Varchar not null, 
> src varchar,
> kdbPublishTime time not null,
> location varchar,
> bid DOUBLE,
> ask DOUBLE,
> bsize unsigned_int,
> asize unsigned_int, 
> srcTime time,
> layer varchar,
> expiryTime time,
> quoteId varchar,
> recvTime time, 
> distTime  time,
> "TIME" time
> CONSTRAINT quote_pk PRIMARY KEY (sym, src, kdbPublishTime)) COMPRESSION='SNAPPY', DATA_BLOCK_ENCODING='FAST_DIFF', VERSIONS=1000
> insert data:
> SYM	SRC	KDBPUBLISHTIME	LOCATION	BID	ASK	BSIZE	ASIZE	SRCTIME	           LAYER	EXPIRYTIME	QUOTEID	      RECVTIME	   DISTTIME	TIME
> 6AH7	cme1	03:42:59	N	       0.7471	0.7506	20	25	03:42:59	   (null)	    (null)	(null)	       03:42:59	    (null)	03:42:59
> 6AH7	cme1	03:42:59	N	       0.7474	0.7506	25	25	03:42:59	   (null)	    (null)	(null)	       03:42:59	    (null)	03:42:59
> val spark = SparkSession
>     .builder()
>     .appName("load_avro")
>     .master("local[1]")
>     .config("spark.sql.warehouse.dir", "file:/tmp/spark-warehouse")
>     .getOrCreate()
>  val df = spark.sqlContext.phoenixTableAsDataFrame("PHOENIX.QUOTE", Seq("SYM","SRC", "EXPIRYTIME"), zkUrl = Some("a1.cluster:2181"))
>   df.show(100)
> problem is in PhoenixRDD:140
>  val rowSeq = columns.map { case (name, sqlType) =>
>         val res = pr.resultMap(name)
>           // Special handling for data types
>           if (dateAsTimestamp && (sqlType == 91 || sqlType == 19)) { // 91 is the defined type for Date and 19 for UNSIGNED_DATE
>             new java.sql.Timestamp(res.asInstanceOf[java.sql.Date].getTime)
>           } else if (sqlType == 92 || sqlType == 18) { // 92 is the defined type for Time and 18 for UNSIGNED_TIME
>             new java.sql.Timestamp(res.asInstanceOf[java.sql.Time].getTime)
>           } else {
>             res
>           }
>       }
> res.asInstanceOf[java.sql.Time].getTime could be null and get NPE.


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)