incubator-etch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nicolae Mihalache <xproma...@gmail.com>
Subject Re: Big data transfer using etch
Date Tue, 14 Dec 2010 14:03:07 GMT
Hello and thanks for the answer.

My problem is not of chunking the data, that I can easily do. The
problem is how to efficiently send data at the maximum throughput
available. If I use normal methods (called actions as I learned in the
meanwhile), I will be very limited by the latency. For example the
Europe-US round-trip is >100ms, that will limit the speed to 10
messages/sec.

If I use oneway methods (called events as I also learned today), there
is no guaranteed order of delivery.
That's what I thought based on my previous CORBA experience (in CORBA
one has to setup a special single threaded POA to guarantee ordered
delivery and even that has problems). Reading more through the etch
mailing lists I found out that oneway methods are actually delivered
in order. And even better if there is a problem with the delivery
there is a notification mechanism. So it actually seems to work the
way I want (but I didn't tested yet).

It's a pity that the documentation isn't better when it comes to
threads and stuff.
Reading the mailing lists helps a lot, so I started reading them all.
I reached now the thread from September "Future of etch"...

I'll come with more questions after I do some tests.

nicolae

On Tue, Dec 14, 2010 at 2:21 PM, scott comer <wert1y@mac.com> wrote:
> hi Nicolae!
>
> if the data is easily chunked by you, such as a very large byte array, then
> the standard
> methods work just fine. for example, to transfer an image or sound, video,
> etc. you could
> easily implement an OutputStream to buffer up data and transmit in chunks
> via etch,
> reassemble on the other side, etc.
>
> the problem comes when you have objects with complicated structure, such as
> rows
> from a db table. can you write 100 rows? 1000? depends upon the data in each
> row.
>
> the big message problem has no easy solution, but one that works might be
> this:
>
> create a virtual stream of data by using the etch binary encoding to code
> your large
> data structure, then chop the stream up and transmit the chunks, and
> reassemble on
> the other side. you have to do the work yourself, but it isn't hard work and
> could serve
> as a basis for a real solution for the big message problem.
>
> can you say some more about your application?
>
> scott out
>
> On 12/14/2010 3:57 AM, Nicolae Mihalache wrote:
>>
>> Hello,
>>
>> I'm considering the possibilities to replace CORBA in some application
>> and I found the etch project.
>> It looks nice and seem to satisfy all my needs except the Big Message
>> Problem as described here:
>> http://incubator.apache.org/etch/big-message-problem.html
>>
>> What would be nice is the ability to stream messages, a bit like
>> @oneway but with guaranteed order and possibility to receive
>> acknowledge if a message has generated an exception.
>> The functionality would be somehow similar with standard TCP sockets:
>> one pushes the data as fast as possible and the write is blocked if
>> the the network or the reader cannot sustain the throughput.
>>  From the API point of view it will be like:
>> start transfer
>> while(not finished):
>>   id=push_data(new_chunk)
>> end transfer
>> -->  at this point all data is guaranteed to have been delivered
>> If an exception is caught, the transfer is interrupted and one can get
>> the id of the message that generated the exception.
>>
>>
>> Strangely enough this functionality useful for file or big data
>> transfer is missing from all the RPC frameworks I've checked so far.
>>
>> It can be emulated somehow with asynchronous calls but one has to
>> manually tune the number of the back buffers depending on the network
>> throughput and latency.
>>
>> Do you plan to implement such a thing in etch? Or to accept such feature?
>>
>>
>> nicolae
>
>

Mime
View raw message