arrow-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sutou Kouhei <...@clear-code.com>
Subject Re: Sending arrow via socket
Date Sun, 03 May 2020 00:13:13 GMT
Hi,

>> Is it necessary to create a buffer to send data via socket, is, for
>> example I create the schemas, do I have to create a buffer to store the
>> data and then send he that data?,

If your C program and JavaScript program run in the same
process, you don't need to do it. You can just pass memory
address.

If your C program and JavaScript program run in the
difference processes (I think so), you need to do it.

>> Also is this method more effective compared to writing data to a arrow
>> file and reading from that arrow file in the javascript program?

If you use memory file system such as tmpfs in Linux for
writing and reading the Apache Arrow file, it may be faster
than sending data via socket.

>> Finally, what are the option that are available for passing data from one
>> program language to another using arrow?

1. Sending data via socket
   (It works with different processes and different hosts)
2. Writing data to a file and passing the file
   (It works with different processes)
3. Sending memory address (the same process)
   (It works with the same process)

> And also, instead of sending the pointer, can you send the buffer directly?

The example I introduced sends the buffer data not the
pointer of the buffer data.


Thanks,
--
kou

In <CACKQ_V0T77ciwBbmKDFNEaKeB+kbyjnjoPawbdJ_P3LeJQZ64Q@mail.gmail.com>
  "Re: Sending arrow via socket" on Sat, 2 May 2020 07:46:55 +0200,
  swizz one <soswizzy@gmail.com> wrote:

> And also, instead of sending the pointer, can you send the buffer directly?
> 
> El sáb., 2 may. 2020 a las 7:07, swizz one (<soswizzy@gmail.com>) escribió:
> 
>> Thank you very much, Your answer has been really helpful. Can I ask a
>> question?
>> Is it necessary to create a buffer to send data via socket, is, for
>> example I create the schemas, do I have to create a buffer to store the
>> data and then send he that data?,
>> Also is this method more effective compared to writing data to a arrow
>> file and reading from that arrow file in the javascript program?
>> Finally, what are the option that are available for passing data from one
>> program language to another using arrow?
>>
>> Sorry, for the questions, I am a bit new to programming.
>>
>> El vie., 1 may. 2020 a las 23:30, Sutou Kouhei (<kou@clear-code.com>)
>> escribió:
>>
>>> Hi,
>>>
>>> Do you want to use Apache Arrow C GLib instead of Apache
>>> Arrow C++, right?
>>>
>>> We don't provide a memory pool in Apache Arrow C GLib API.
>>>
>>> You can create a resizable buffer and output serialize data
>>> to it:
>>>
>>> ---
>>> GError *error = NULL;
>>> GArrowResizableBuffer *buffer = garrow_resizable_buffer_new(1024, &error);
>>> if (!buffer) {
>>>   g_print("error: %s\n", error->message);
>>>   g_error_free(error);
>>>   return;
>>> }
>>>
>>> GArrowBufferOutputStream *output =
>>> garrow_buffer_output_stream_new(buffer);
>>>
>>> GArrowRecordBatchStreamWriter *writer =
>>>   garrow_record_batch_stream_writer_new(GARROW_OUTPUT_STREAM(output),
>>>                                         schema, /* You need to create
>>> this */
>>>                                         &error);
>>> if (!writer) {
>>>   g_print("error: %s\n", error->message);
>>>   g_error_free(error);
>>>   g_object_unref(output);
>>>   g_object_unref(buffer);
>>>   return;
>>> }
>>>
>>> if (!garrow_record_batch_writer_write_record_batch(
>>>        GARROW_RECORD_BATCH_WRITER(writer),
>>>        record_batch, /* You need to create this */
>>>        &error)) {
>>>   g_print("error: %s\n", error->message);
>>>   g_error_free(error);
>>>   g_object_unref(writer);
>>>   g_object_unref(output);
>>>   g_object_unref(buffer);
>>>   return;
>>> }
>>>
>>> if (!garrow_record_batch_writer_close(
>>>        GARROW_RECORD_BATCH_WRITER(writer),
>>>        &error)) {
>>>   g_print("error: %s\n", error->message);
>>>   g_error_free(error);
>>>   g_object_unref(writer);
>>>   g_object_unref(output);
>>>   g_object_unref(buffer);
>>>   return;
>>> }
>>>
>>> GBytes *data = garrow_buffer_get_data(GARROW_BUFFER(buffer));
>>> gsize data_size;
>>> gconstpointer data_raw = g_bytes_get_data(data, &data_size);
>>> write(websocket_fd, data_raw, data_size);
>>> g_bytes_unref(data);
>>>
>>> g_object_unref(writer);
>>> g_object_unref(output);
>>> g_object_unref(buffer);
>>> ---
>>>
>>>
>>> Thanks,
>>> --
>>> kou
>>>
>>> In <CACKQ_V3_yeuNn5TzPiF1RS=6WmwjL5jR4Vgx39GpQ3FvMgEqzg@mail.gmail.com>
>>>   "Sending arrow via socket" on Fri, 1 May 2020 18:12:34 +0200,
>>>   swizz one <soswizzy@gmail.com> wrote:
>>>
>>> > Please, I am currently working on a project that require sending data
>>> from
>>> > an c program to perpespective(javascript) via socket from c. Since
>>> > perpespective works with arrow, it was a perfect choice. Is it possible
>>> to
>>> > send table at x interval from c to javascript via websocket without
>>> > creating a an arrow binary format file?
>>> > How to you create a memory pool primotive with c, without having to
>>> size to
>>> > have a fixed sized on the array like the memory pool in c++.
>>> >
>>> > Your's Faithfully,
>>> > Thank you
>>>
>>
Mime
View raw message