activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Francesco PADOVANI <Francesco.PADOV...@bticino.it>
Subject Re: MQTT retained messages with weird characters
Date Mon, 06 Feb 2017 16:41:11 GMT
Hi Martyn,

great to know it!

Thanks a lot for your work.


Francesco


________________________________
From: Martyn Taylor <mtaylor@redhat.com>
Sent: Monday, February 6, 2017 4:55:37 PM
To: users@activemq.apache.org
Subject: Re: MQTT retained messages with weird characters

Hi Francesco,

It appears that there is a bug in the MessageImpl code that is not properly
setting the endOfBodyPosition when the server restarts (reads from journal)
and then there's a subsequent message copy (i.e. retain).  I've enabled
correct pointer setting and added the fix to MQTT.

I'm running the full suite atm to ensure this doesn't affect other areas of
the code.  Should have this fix done shortly.

Thanks for all the info,
Martyn

On Thu, Jan 12, 2017 at 10:13 AM, Francesco PADOVANI <
Francesco.PADOVANI@bticino.it> wrote:

> Hi Martyn,
>
> I've just opened a JIRA related to this issue: it's the tkt ARTEMIS-917 (
> https://issues.apache.org/jira/browse/ARTEMIS-917). Of course it's
> currently unassigned: let me know if I have to assign it to someone in
> particular.
>
>
> A Thing That may help you (even if I understand you've already figured out
> where the problem lies): a developer of ours did put the Artemis code under
> debug and he has noticed that the message is yes, written correctly on the
> file system, but then it is read in a wrong way. It seems a matter of
> pointer position, which is kept to the end of metadata instead of the end
> of message body while reading it. The involved classes should be the
> followings:
>
>
> AbstractJournalStorageManager.java
>
> --> this is the class that probably doesn't correctly values the pointer.
>
>
> MessageImpl.java
>
> --> this (if I understand well) is the class containing the "decode"
> method used to read the message. A question: do you think the last bold
> line, below, could be a solution?
>
>
> public void decode(final ActiveMQBuffer buff) {
>
>       int start = buff.readerIndex();
>
>       endOfBodyPosition = buff.readInt();
>
>       endOfMessagePosition = buff.getInt(endOfBodyPosition -
> BUFFER_HEADER_SPACE + start);
>
>       int length = endOfMessagePosition - BUFFER_HEADER_SPACE;
>
>       buffer.setIndex(0, BUFFER_HEADER_SPACE);
>
>       buffer.writeBytes(buff, start, length);
>
>       decode();
>
>       buff.readerIndex(start + length);
>
>       buff.writerIndex(start +  endOfBodyPosition);
>
>    }
>
>
>
> Thanks a lot  for your help and your patience.
>
>
> Francesco
>
>
>
> ________________________________
> From: Martyn Taylor <mtaylor@redhat.com>
> Sent: Wednesday, January 11, 2017 3:53:00 PM
> To: users@activemq.apache.org
> Subject: Re: MQTT retained messages with weird characters
>
> It looks to me like the full message buffer (not just the body) is getting
> returned to the client.  Clebert I think you're right in that we do some
> hackery to handle retained messages.  I suspect this is where the error is.
>
> Francesco could you create a JIRA with some reproducer steps and I'll aim
> to take a look.
>
> Thanks
>
> On Wed, Jan 11, 2017 at 2:03 PM, Clebert Suconic <
> clebert.suconic@gmail.com>
> wrote:
>
> > the MQTT ProtocolManager does some playing with retaining, perhaps
> > there's something wrong with reloading from journal on retaining. I
> > was wondering if you could send us a test showing the issue with MQTT
> > so we could see if there's anything specific to your test while using
> > MQTT.
> >
> > On Wed, Jan 11, 2017 at 4:22 AM, Francesco PADOVANI
> > <Francesco.PADOVANI@bticino.it> wrote:
> > > Unfortunately no (not yet).
> > >
> > > Anyway, it's a clean Artemis installation of version 1.5.1, made by
> > > following the user manual instructions at
> > > https://activemq.apache.org/artemis/docs/1.5.1/using-server.html.
> > >
> > > The installation was done by using a dedicated user and also the
> artemis
> > > java process runs with this dedicated user. Attached you can find my
> > > broker.xml configuration file: it's pretty much the same default
> created
> > > during the installation procedure, but for the acceptors (which I've
> > > customized for my MQTT purpose) and the addition of parameter
> > > "<last-value-queue>true</last-value-queue>" inside the address-setting
> > > section (but I tried with and without it and my issue persists).
> > >
> > > The only other configuration I changed is the heap size dedicated to
> the
> > > process, by setting "-Xms4096M -Xmx4096M" among the JAVA_ARGS in the
> > > artemis.profile file.
> > >
> > >
> > > The platform where Artemis broker is installed is:
> > >
> > >
> > > - Machine:
> > >
> > > It's an EC2 instance on AWS cloud of type m4.large: 2 vCPU, 8G RAM and
> > SSD
> > > storage.
> > >
> > >
> > > - OS:
> > >
> > > CentOS Linux release 7.3.1611 (Core) - x86_64
> > >
> > >
> > > - JVM:
> > >
> > > java version "1.8.0_111"
> > > Java(TM) SE Runtime Environment (build 1.8.0_111-b14)
> > > Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)
> > >
> > >
> > > - Distro package version of libaio:
> > >
> > > Arch        : x86_64
> > > Version     : 0.3.109
> > > Release     : 13.el7
> > >
> > >
> > > It seems like the problem relates the Persistence (File Journal?).
> Until
> > the
> > > broker keeps the retained messages in ram it works. When it needs to
> > recover
> > > the retained messages  from the file system, clients receive them with
> > weird
> > > chars.
> > > And I suspect the same issue appears when RAM is full and messages
> start
> > to
> > > be paged... but this case is a bit more difficult to test in a
> systematic
> > > way.
> > >
> > >
> > > Francesco
> > >
> > >
> > >
> > > ________________________________
> > > From: Clebert Suconic <clebert.suconic@gmail.com>
> > > Sent: Tuesday, January 10, 2017 7:43 PM
> > > To: users@activemq.apache.org
> > > Subject: Re: MQTT retained messages with weird characters
> > >
> > > Do you have a working test you can share?
> > >
> > > On Tue, Jan 10, 2017 at 12:07 PM, Francesco PADOVANI
> > > <Francesco.PADOVANI@bticino.it> wrote:
> > >> Hello,
> > >>
> > >> I'm trying the Apache Artemis Broker (ver. 1.5.1) for MQTT protocol.
> > >>
> > >> Exactly I'm testing the retained feature for messages of MQTT
> protocol.
> > >>
> > >> While the broker is up it seems to work well:
> > >>
> > >> a) a client publishes a retained message to a specific topic
> > >>
> > >> b) any client which then subscribes that topic receives the retained
> > >> message (any time it re-subscribes the topic it receives the last
> > retained
> > >> message ...perfect!)
> > >>
> > >> But it happens a strange thing when I restart the broker. And I mean I
> > >> restart the broker without any change on configuration: simply I make
> a
> > >> "artemis-service stop" and "artemis-service start". After The broker
> is
> > up
> > >> and running, if a client subscribes the previous topic it still
> > receives the
> > >> retained message, but with weird characters appended to it. For new
> > retained
> > >> messages published it work well again... but the previous ones (before
> > the
> > >> broker restart) are sent all with these weird characters. E.g., the
> > >> following is a retained message published before the broker restart:
> > >>
> > >> "test 25 for a retained message 20170110"
> > >>
> > >> And this is exactly how clients get it before the broker restart.
> > >>
> > >> Instead, after broker restart, the same retained message is received
> by
> > >> clients in the following way:
> > >>
> > >> test 25 for a retained message 20170110?
> > >>                                         ?
> 6$sys.mqtt.retain..cro.test5 ?
> > >> &mqtt.message.retain ? mqtt.qos.level
> > >>
> > >>
> > >> Is this my bad configuration (but I don't know ehere)? Or anything
> else?
> > >> Or what?
> > >>
> > >> Someone can help me?
> > >>
> > >>
> > >> Thanks in advance.
> > >>
> > >>
> > >> Francesco
> > >>
> > >> ________________________________
> > >>
> > >> Ce message, ainsi que tous les fichiers joints à ce message, peuvent
> > >> contenir des informations sensibles et/ ou confidentielles ne devant
> pas
> > >> être divulguées. Si vous n'êtes pas le destinataire de ce message (ou
> > que
> > >> vous recevez ce message par erreur), nous vous remercions de le
> notifier
> > >> immédiatement à son expéditeur, et de détruire ce message. Toute
> copie,
> > >> divulgation, modification, utilisation ou diffusion, non autorisée,
> > directe
> > >> ou indirecte, de tout ou partie de ce message, est strictement
> > interdite.
> > >>
> > >>
> > >> This e-mail, and any document attached hereby, may contain
> confidential
> > >> and/or privileged information. If you are not the intended recipient
> (or
> > >> have received this e-mail in error) please notify the sender
> > immediately and
> > >> destroy this e-mail. Any unauthorized, direct or indirect, copying,
> > >> disclosure, distribution or other use of the material or parts thereof
> > is
> > >> strictly forbidden.
> > >
> > >
> > >
> > > --
> > > Clebert Suconic
> > >
> > > ________________________________
> > >
> > > Ce message, ainsi que tous les fichiers joints à ce message, peuvent
> > > contenir des informations sensibles et/ ou confidentielles ne devant
> pas
> > > être divulguées. Si vous n'êtes pas le destinataire de ce message (ou
> que
> > > vous recevez ce message par erreur), nous vous remercions de le
> notifier
> > > immédiatement à son expéditeur, et de détruire ce message. Toute copie,
> > > divulgation, modification, utilisation ou diffusion, non autorisée,
> > directe
> > > ou indirecte, de tout ou partie de ce message, est strictement
> interdite.
> > >
> > >
> > > This e-mail, and any document attached hereby, may contain confidential
> > > and/or privileged information. If you are not the intended recipient
> (or
> > > have received this e-mail in error) please notify the sender
> immediately
> > and
> > > destroy this e-mail. Any unauthorized, direct or indirect, copying,
> > > disclosure, distribution or other use of the material or parts thereof
> is
> > > strictly forbidden.
> >
> >
> >
> > --
> > Clebert Suconic
> >
>
> ________________________________
>
> Ce message, ainsi que tous les fichiers joints à ce message, peuvent
> contenir des informations sensibles et/ ou confidentielles ne devant pas
> être divulguées. Si vous n'êtes pas le destinataire de ce message (ou que
> vous recevez ce message par erreur), nous vous remercions de le notifier
> immédiatement à son expéditeur, et de détruire ce message. Toute copie,
> divulgation, modification, utilisation ou diffusion, non autorisée, directe
> ou indirecte, de tout ou partie de ce message, est strictement interdite.
>
>
> This e-mail, and any document attached hereby, may contain confidential
> and/or privileged information. If you are not the intended recipient (or
> have received this e-mail in error) please notify the sender immediately
> and destroy this e-mail. Any unauthorized, direct or indirect, copying,
> disclosure, distribution or other use of the material or parts thereof is
> strictly forbidden.
>

________________________________

Ce message, ainsi que tous les fichiers joints à ce message, peuvent contenir des informations
sensibles et/ ou confidentielles ne devant pas être divulguées. Si vous n'êtes pas le destinataire
de ce message (ou que vous recevez ce message par erreur), nous vous remercions de le notifier
immédiatement à son expéditeur, et de détruire ce message. Toute copie, divulgation, modification,
utilisation ou diffusion, non autorisée, directe ou indirecte, de tout ou partie de ce message,
est strictement interdite.


This e-mail, and any document attached hereby, may contain confidential and/or privileged
information. If you are not the intended recipient (or have received this e-mail in error)
please notify the sender immediately and destroy this e-mail. Any unauthorized, direct or
indirect, copying, disclosure, distribution or other use of the material or parts thereof
is strictly forbidden.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message