arrow-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andy Grove <andygrov...@gmail.com>
Subject Re: Example for Apache Arrow Flight
Date Wed, 13 May 2020 00:58:21 GMT
Hi Damien,

Here is a brief answer that hopefully at least points you in the right
direction.

You need to use the VectorSchemaRoot class to build batches of data. There
is some documentation on how to do that [1]. Then, in your FlightProducer
implementation, you need to pass the batches to the
FlightProducer.ServerStreamListener using the "start" and "next" methods
when batches are ready to be sent. There is sample code in the Arrrow repo
[2] and there is a Kotlin example that I wrote here [3].

Andy.

[1] https://arrow.apache.org/docs/java/ipc.html
[2]
https://github.com/apache/arrow/blob/master/java/flight/flight-core/src/main/java/org/apache/arrow/flight/example/ExampleFlightServer.java
[3]
https://github.com/ballista-compute/ballista/blob/master/jvm/executor/src/main/kotlin/BallistaFlightProducer.kt

On Tue, May 12, 2020 at 6:43 PM Damien Chaillou <damien.chaillou@gmail.com>
wrote:

> Hi!
>
> I'm currently playing with Apache Arrow Flight in java and cant get my
> head around how to implement something.
> In the *doGet* method, for example, I'm doing simple JDBC calls that I
> would like to stream over.
> If I understand correctly, the *FlightData*'s body should be a
> ArrowMessage serialised as a ByteString (?) I built from a ResultSet from
> my JDBC call. I though using JdbcToArrow
> <https://github.com/apache/arrow/blob/master/java/adapter/jdbc/src/main/java/org/apache/arrow/adapter/jdbc/JdbcToArrow.java#L149>
helper
> class to help, but I can't find any example of how to do such thing.
> I came with few questions :
>
>    - What is the data we must set in dataBody ?
>    - If those are ArrowMessages, how can I map a ResultSet to this type?
>    - How do we serialise to ByteString objects typed like ArrowMessage,
>    Schema ... ?
>
>
> Could anyone point a piece of code/blog post/anything to me please ?
>
> My toy project would be a generic proxy server in front of any database
> with available JDBC drivers that could stream queries over Arrow Flight
> (gRPC).
>
> Cheers, thanks!
>
>
> Damien
>

Mime
View raw message