spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Artemis User <>
Subject Client APIs for Accessing Spark Data Frames Directly
Date Wed, 21 Oct 2020 18:23:50 GMT
Is there anyway to access the Data Frames content directly/interactively 
via some client access APIs?  Some background info:

 1. We have a Java client application that uses spark launcher to submit
    a spark job to a spark master.
 2. The default spark launcher API has only a handle API that provides
    job status information
 3. It is possible to use the Hive JDBC to access data via the built-in
    thrifty server, but the data have to be saved into hive tables, not
    sure if these tables have to be persisted on disk and require huge
    overhead for data I/O.  Since DF is in memory, it would be more
    convenient and faster to access the data directly in memory if such
    APIs are available.

Thanks!  -- ND

View raw message