arrow-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From swizz one <soswi...@gmail.com>
Subject Re: Sending arrow via socket
Date Sun, 03 May 2020 20:59:31 GMT
Is there any code in javascript to access the buffer?
Thank you for tour help

On Sun, May 3, 2020, 02:13 Sutou Kouhei <kou@clear-code.com> wrote:

> Hi,
>
> >> Is it necessary to create a buffer to send data via socket, is, for
> >> example I create the schemas, do I have to create a buffer to store the
> >> data and then send he that data?,
>
> If your C program and JavaScript program run in the same
> process, you don't need to do it. You can just pass memory
> address.
>
> If your C program and JavaScript program run in the
> difference processes (I think so), you need to do it.
>
> >> Also is this method more effective compared to writing data to a arrow
> >> file and reading from that arrow file in the javascript program?
>
> If you use memory file system such as tmpfs in Linux for
> writing and reading the Apache Arrow file, it may be faster
> than sending data via socket.
>
> >> Finally, what are the option that are available for passing data from
> one
> >> program language to another using arrow?
>
> 1. Sending data via socket
>    (It works with different processes and different hosts)
> 2. Writing data to a file and passing the file
>    (It works with different processes)
> 3. Sending memory address (the same process)
>    (It works with the same process)
>
> > And also, instead of sending the pointer, can you send the buffer
> directly?
>
> The example I introduced sends the buffer data not the
> pointer of the buffer data.
>
>
> Thanks,
> --
> kou
>
> In <CACKQ_V0T77ciwBbmKDFNEaKeB+kbyjnjoPawbdJ_P3LeJQZ64Q@mail.gmail.com>
>   "Re: Sending arrow via socket" on Sat, 2 May 2020 07:46:55 +0200,
>   swizz one <soswizzy@gmail.com> wrote:
>
> > And also, instead of sending the pointer, can you send the buffer
> directly?
> >
> > El sáb., 2 may. 2020 a las 7:07, swizz one (<soswizzy@gmail.com>)
> escribió:
> >
> >> Thank you very much, Your answer has been really helpful. Can I ask a
> >> question?
> >> Is it necessary to create a buffer to send data via socket, is, for
> >> example I create the schemas, do I have to create a buffer to store the
> >> data and then send he that data?,
> >> Also is this method more effective compared to writing data to a arrow
> >> file and reading from that arrow file in the javascript program?
> >> Finally, what are the option that are available for passing data from
> one
> >> program language to another using arrow?
> >>
> >> Sorry, for the questions, I am a bit new to programming.
> >>
> >> El vie., 1 may. 2020 a las 23:30, Sutou Kouhei (<kou@clear-code.com>)
> >> escribió:
> >>
> >>> Hi,
> >>>
> >>> Do you want to use Apache Arrow C GLib instead of Apache
> >>> Arrow C++, right?
> >>>
> >>> We don't provide a memory pool in Apache Arrow C GLib API.
> >>>
> >>> You can create a resizable buffer and output serialize data
> >>> to it:
> >>>
> >>> ---
> >>> GError *error = NULL;
> >>> GArrowResizableBuffer *buffer = garrow_resizable_buffer_new(1024,
> &error);
> >>> if (!buffer) {
> >>>   g_print("error: %s\n", error->message);
> >>>   g_error_free(error);
> >>>   return;
> >>> }
> >>>
> >>> GArrowBufferOutputStream *output =
> >>> garrow_buffer_output_stream_new(buffer);
> >>>
> >>> GArrowRecordBatchStreamWriter *writer =
> >>>   garrow_record_batch_stream_writer_new(GARROW_OUTPUT_STREAM(output),
> >>>                                         schema, /* You need to create
> >>> this */
> >>>                                         &error);
> >>> if (!writer) {
> >>>   g_print("error: %s\n", error->message);
> >>>   g_error_free(error);
> >>>   g_object_unref(output);
> >>>   g_object_unref(buffer);
> >>>   return;
> >>> }
> >>>
> >>> if (!garrow_record_batch_writer_write_record_batch(
> >>>        GARROW_RECORD_BATCH_WRITER(writer),
> >>>        record_batch, /* You need to create this */
> >>>        &error)) {
> >>>   g_print("error: %s\n", error->message);
> >>>   g_error_free(error);
> >>>   g_object_unref(writer);
> >>>   g_object_unref(output);
> >>>   g_object_unref(buffer);
> >>>   return;
> >>> }
> >>>
> >>> if (!garrow_record_batch_writer_close(
> >>>        GARROW_RECORD_BATCH_WRITER(writer),
> >>>        &error)) {
> >>>   g_print("error: %s\n", error->message);
> >>>   g_error_free(error);
> >>>   g_object_unref(writer);
> >>>   g_object_unref(output);
> >>>   g_object_unref(buffer);
> >>>   return;
> >>> }
> >>>
> >>> GBytes *data = garrow_buffer_get_data(GARROW_BUFFER(buffer));
> >>> gsize data_size;
> >>> gconstpointer data_raw = g_bytes_get_data(data, &data_size);
> >>> write(websocket_fd, data_raw, data_size);
> >>> g_bytes_unref(data);
> >>>
> >>> g_object_unref(writer);
> >>> g_object_unref(output);
> >>> g_object_unref(buffer);
> >>> ---
> >>>
> >>>
> >>> Thanks,
> >>> --
> >>> kou
> >>>
> >>> In <CACKQ_V3_yeuNn5TzPiF1RS=6WmwjL5jR4Vgx39GpQ3FvMgEqzg@mail.gmail.com
> >
> >>>   "Sending arrow via socket" on Fri, 1 May 2020 18:12:34 +0200,
> >>>   swizz one <soswizzy@gmail.com> wrote:
> >>>
> >>> > Please, I am currently working on a project that require sending data
> >>> from
> >>> > an c program to perpespective(javascript) via socket from c. Since
> >>> > perpespective works with arrow, it was a perfect choice. Is it
> possible
> >>> to
> >>> > send table at x interval from c to javascript via websocket without
> >>> > creating a an arrow binary format file?
> >>> > How to you create a memory pool primotive with c, without having to
> >>> size to
> >>> > have a fixed sized on the array like the memory pool in c++.
> >>> >
> >>> > Your's Faithfully,
> >>> > Thank you
> >>>
> >>
>

Mime
View raw message