Mailing-List: contact etch-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: etch-dev@incubator.apache.org
Received-SPF: pass (athena.apache.org: domain of wert1y@mac.com designates
 17.148.16.98 as permitted sender)
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; charset=ISO-8859-1; format=flowed
Message-id: <4D076F57.9060000@mac.com>
Date: Tue, 14 Dec 2010 07:21:27 -0600
From: scott comer <wert1y@mac.com>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.13)
 Gecko/20101207 Lightning/1.0b2 Thunderbird/3.1.7
To: etch-dev@incubator.apache.org
Subject: Re: Big data transfer using etch
References: <AANLkTik3ObZovmSDGzeHTA874_TewoPU409EuZwUJGnz@mail.gmail.com>
In-reply-to: <AANLkTik3ObZovmSDGzeHTA874_TewoPU409EuZwUJGnz@mail.gmail.com>

hi Nicolae!

if the data is easily chunked by you, such as a very large byte array, 
then the standard
methods work just fine. for example, to transfer an image or sound, 
video, etc. you could
easily implement an OutputStream to buffer up data and transmit in 
chunks via etch,
reassemble on the other side, etc.

the problem comes when you have objects with complicated structure, such 
as rows
from a db table. can you write 100 rows? 1000? depends upon the data in 
each row.

the big message problem has no easy solution, but one that works might 
be this:

create a virtual stream of data by using the etch binary encoding to 
code your large
data structure, then chop the stream up and transmit the chunks, and 
reassemble on
the other side. you have to do the work yourself, but it isn't hard work 
and could serve
as a basis for a real solution for the big message problem.

can you say some more about your application?

scott out

On 12/14/2010 3:57 AM, Nicolae Mihalache wrote:
> Hello,
>
> I'm considering the possibilities to replace CORBA in some application
> and I found the etch project.
> It looks nice and seem to satisfy all my needs except the Big Message
> Problem as described here:
> http://incubator.apache.org/etch/big-message-problem.html
>
> What would be nice is the ability to stream messages, a bit like
> @oneway but with guaranteed order and possibility to receive
> acknowledge if a message has generated an exception.
> The functionality would be somehow similar with standard TCP sockets:
> one pushes the data as fast as possible and the write is blocked if
> the the network or the reader cannot sustain the throughput.
>  From the API point of view it will be like:
> start transfer
> while(not finished):
>    id=push_data(new_chunk)
> end transfer
> -->  at this point all data is guaranteed to have been delivered
> If an exception is caught, the transfer is interrupted and one can get
> the id of the message that generated the exception.
>
>
> Strangely enough this functionality useful for file or big data
> transfer is missing from all the RPC frameworks I've checked so far.
>
> It can be emulated somehow with asynchronous calls but one has to
> manually tune the number of the back buffers depending on the network
> throughput and latency.
>
> Do you plan to implement such a thing in etch? Or to accept such feature?
>
>
> nicolae