qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lahiru Gunathilake <glah...@gmail.com>
Subject Re: How to handle large messages in Qpid Java client
Date Thu, 02 Dec 2010 09:58:43 GMT
FYI, I try to monitor all the reads from the socket InputStream from the
client and I can see that Client is reading a curupted message, so
I guarantee that there's no issue in Client side IoSendere and IoReceiver
classes.

I think the issue is with the broker IoReceiver or Sending, we found that
this is directly an issue with Java broker because things are working
perfect with Qpid C++ broker 0.8 RC2.

Thanks
Lahiru

On Mon, Nov 29, 2010 at 3:21 PM, Lahiru Gunathilake <glahiru@gmail.com>wrote:

> FYI, during receiving a message, I can see following
> malformed characters in my JMSTextMessage object,
>
> <OUTMOD>4</OUTMOD>^D^C��^@^A^@^@^@^@^@^@
>
> This cause the exception i have sent at the very first email, after
> increasing the frame-size sometimes it works sometimes it doesn't.
>
> Lahiru
>
>
> On Mon, Nov 29, 2010 at 8:24 AM, Lahiru Gunathilake <glahiru@gmail.com>wrote:
>
>> Hi Marnie,
>>
>> In IoSender and IoReceiver classes we are using a ByteBuffer to read the
>> write messages, to increase the size of this buffer, I simply changed a
>> constant defined in IoTransport.java file, if you look in to those three
>> classes, you will get it.
>>
>> In IoTransport.java I changed the following line of code,
>>
>> private static int DEFAULT_READ_WRITE_BUFFER_SIZE = 64 * 1024;
>>
>> but I saw there are some other constants defined with the name of
>> MAX_FRAME_SIZE but after going through the code i saw they are not
>> affecting.
>>
>> I feel there's an issue when we send a large message in to a Queue and
>> something get currupted, when do framing and send. because when I keep the
>> same small frame size for read buffer and increase the size for write
>> buffer, it worked for 500K messages, but couldn't get it work for more than
>> 1MB large messages.
>>
>> Lahiru
>>
>> Lahiru
>>
>>
>> On Mon, Nov 29, 2010 at 2:09 AM, Marnie McCormack <
>> marnie.mccormack@googlemail.com> wrote:
>>
>>> Hi Lahiru,
>>>
>>> What exactly have you done/set for the frame size change ?
>>>
>>> Thanks,
>>> Marnie
>>>
>>> On Sun, Nov 28, 2010 at 9:07 AM, Lahiru Gunathilake <glahiru@gmail.com
>>> >wrote:
>>>
>>> > Hi all,
>>> >
>>> > I did a mistake when increase the framesize without increasing to
>>> required
>>> > size, when I increase the size I was able to read some of the messages
>>> > which
>>> > are in the size of 700K, but sometimes it fails.
>>> >
>>> >
>>> > I kept on increasing the frame-size since in my setup there are
>>> messages
>>> > with the size of 1.5M but I couldn't read a any of them though i
>>> increase
>>> > the frame-size more.
>>> >
>>> > other than changing in IoTransport.java, do I have to change any other
>>> > constant values to sync the maximum-frame-size ?
>>> >
>>> > Regards
>>> > Lahiru
>>> >
>>> > On Sat, Nov 27, 2010 at 10:34 PM, Lahiru Gunathilake <
>>> glahiru@gmail.com
>>> > >wrote:
>>> >
>>> > > Hi Andrew,
>>> > >
>>> > > Thanks Andrew for the response.
>>> > > I have doubled  the default frame-size which was (64 * 1024) in
>>> > > IoTransport.java, but didn't worked for me. FYI, activeMQ works fine
>>> in
>>> > this
>>> > > scenario.
>>> > >
>>> > > On Sat, Nov 27, 2010 at 10:22 PM, Andrew Kennedy <
>>> > > andrewinternational@gmail.com> wrote:
>>> > >
>>> > >> Hi,
>>> > >>
>>> > >> The out of order frames only occur in 0-10 and with transports
that
>>> have
>>> > >> no
>>> > >> concept of sequencing, i.e. in-vm or udp, not tcp. Having said
that,
>>> a
>>> > >> frame
>>> > >> body being interpreted as a header would give those symptoms.
>>> > >
>>> > > I will debug and see.
>>> > >
>>> > > Regards
>>> > > Lahiru
>>> > >
>>> > >> Test code that
>>> > >> exercises the issue would be illuminating.
>>> > >>
>>> > >> Unsure if this helps?
>>> > >>
>>> > >> Andrew.
>>> > >>
>>> > >> On 26 Nov 2010 16:56, "Marnie McCormack" <
>>> > marnie.mccormack@googlemail.com
>>> > >> >
>>> > >> wrote:
>>> > >>
>>> > >> Hi Lahiru,
>>> > >>
>>> > >> Robbie mentioned earlier when I asked him about this that he
>>> *thinks*
>>> > it's
>>> > >> a
>>> > >> bug Andrew might have come across and possibly fixed to do with
OoO
>>> > >> frames.
>>> > >> Certainly the exception showing that the buffer length is 1 might
>>> > support
>>> > >> that.
>>> > >>
>>> > >> I haven't had a chance to look about for this - Andrew is not about
>>> > today.
>>> > >>
>>> > >> I'll check with him when he's back if you don't get a better reply
>>> here
>>> > >> first.
>>> > >>
>>> > >> Rgds,
>>> > >> Marnie
>>> > >>
>>> > >> On Fri, Nov 26, 2010 at 3:27 PM, Lahiru Gunathilake <
>>> glahiru@gmail.com
>>> > >> >wrote:
>>> > >>
>>> > >>
>>> > >> > Hi Marnie,
>>> > >> >
>>> > >> > Thanks for your prompt response. yes i tried to wait before
>>> consuming,
>>> > >> and
>>> > >> > it wo...
>>> > >>
>>> > >
>>> > >
>>> >
>>>
>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message