phoenix-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [phoenix-connectors] ChinmaySKulkarni commented on issue #2: PHOENIX-5197: Use extraOptions to set the configuration for Spark Workers
Date Wed, 03 Apr 2019 23:07:06 GMT
ChinmaySKulkarni commented on issue #2: PHOENIX-5197: Use extraOptions to set the configuration
for Spark Workers
URL: https://github.com/apache/phoenix-connectors/pull/2#issuecomment-479690145
 
 
   A couple of problems that we will run into:
   1. [`DataSourceOptions`](https://github.com/apache/spark/blob/branch-2.4/sql/core/src/main/java/org/apache/spark/sql/sources/v2/DataSourceOptions.java#L88-L93)
converts all keys in the options map to lowercase. When creating a reader or writer, we create
a `DataSourceOptions` object which is passed in all options. See [createReader](https://github.com/apache/spark/blob/branch-2.4/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala#L150)
and [createWriter](https://github.com/apache/spark/blob/branch-2.4/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala#L162).
This is a problem for passing in a lot of Phoenix configurations like `phoenix.query.timeoutMs`
which are case-sensitive.
   -> We can potentially get past this issue by serializing all configs against a single
key in the options map and passing that to the workers. 
   
   2. I can look into setting some small values for upsert batch size or query timeouts from
the driver to see if the workers honor these values.
   FYI @twdsilva 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message