activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Clebert Suconic <clebert.suco...@gmail.com>
Subject Re: MQTT retained messages with weird characters
Date Mon, 06 Feb 2017 16:51:56 GMT
Perhaps we could add this to the next 1.5.3, if you cherry-pick the fix in 1.x

On Mon, Feb 6, 2017 at 10:55 AM, Martyn Taylor <mtaylor@redhat.com> wrote:
> Hi Francesco,
>
> It appears that there is a bug in the MessageImpl code that is not properly
> setting the endOfBodyPosition when the server restarts (reads from journal)
> and then there's a subsequent message copy (i.e. retain).  I've enabled
> correct pointer setting and added the fix to MQTT.
>
> I'm running the full suite atm to ensure this doesn't affect other areas of
> the code.  Should have this fix done shortly.
>
> Thanks for all the info,
> Martyn
>
> On Thu, Jan 12, 2017 at 10:13 AM, Francesco PADOVANI <
> Francesco.PADOVANI@bticino.it> wrote:
>
>> Hi Martyn,
>>
>> I've just opened a JIRA related to this issue: it's the tkt ARTEMIS-917 (
>> https://issues.apache.org/jira/browse/ARTEMIS-917). Of course it's
>> currently unassigned: let me know if I have to assign it to someone in
>> particular.
>>
>>
>> A Thing That may help you (even if I understand you've already figured out
>> where the problem lies): a developer of ours did put the Artemis code under
>> debug and he has noticed that the message is yes, written correctly on the
>> file system, but then it is read in a wrong way. It seems a matter of
>> pointer position, which is kept to the end of metadata instead of the end
>> of message body while reading it. The involved classes should be the
>> followings:
>>
>>
>> AbstractJournalStorageManager.java
>>
>> --> this is the class that probably doesn't correctly values the pointer.
>>
>>
>> MessageImpl.java
>>
>> --> this (if I understand well) is the class containing the "decode"
>> method used to read the message. A question: do you think the last bold
>> line, below, could be a solution?
>>
>>
>> public void decode(final ActiveMQBuffer buff) {
>>
>>       int start = buff.readerIndex();
>>
>>       endOfBodyPosition = buff.readInt();
>>
>>       endOfMessagePosition = buff.getInt(endOfBodyPosition -
>> BUFFER_HEADER_SPACE + start);
>>
>>       int length = endOfMessagePosition - BUFFER_HEADER_SPACE;
>>
>>       buffer.setIndex(0, BUFFER_HEADER_SPACE);
>>
>>       buffer.writeBytes(buff, start, length);
>>
>>       decode();
>>
>>       buff.readerIndex(start + length);
>>
>>       buff.writerIndex(start +  endOfBodyPosition);
>>
>>    }
>>
>>
>>
>> Thanks a lot  for your help and your patience.
>>
>>
>> Francesco
>>
>>
>>
>> ________________________________
>> From: Martyn Taylor <mtaylor@redhat.com>
>> Sent: Wednesday, January 11, 2017 3:53:00 PM
>> To: users@activemq.apache.org
>> Subject: Re: MQTT retained messages with weird characters
>>
>> It looks to me like the full message buffer (not just the body) is getting
>> returned to the client.  Clebert I think you're right in that we do some
>> hackery to handle retained messages.  I suspect this is where the error is.
>>
>> Francesco could you create a JIRA with some reproducer steps and I'll aim
>> to take a look.
>>
>> Thanks
>>
>> On Wed, Jan 11, 2017 at 2:03 PM, Clebert Suconic <
>> clebert.suconic@gmail.com>
>> wrote:
>>
>> > the MQTT ProtocolManager does some playing with retaining, perhaps
>> > there's something wrong with reloading from journal on retaining. I
>> > was wondering if you could send us a test showing the issue with MQTT
>> > so we could see if there's anything specific to your test while using
>> > MQTT.
>> >
>> > On Wed, Jan 11, 2017 at 4:22 AM, Francesco PADOVANI
>> > <Francesco.PADOVANI@bticino.it> wrote:
>> > > Unfortunately no (not yet).
>> > >
>> > > Anyway, it's a clean Artemis installation of version 1.5.1, made by
>> > > following the user manual instructions at
>> > > https://activemq.apache.org/artemis/docs/1.5.1/using-server.html.
>> > >
>> > > The installation was done by using a dedicated user and also the
>> artemis
>> > > java process runs with this dedicated user. Attached you can find my
>> > > broker.xml configuration file: it's pretty much the same default
>> created
>> > > during the installation procedure, but for the acceptors (which I've
>> > > customized for my MQTT purpose) and the addition of parameter
>> > > "<last-value-queue>true</last-value-queue>" inside the address-setting
>> > > section (but I tried with and without it and my issue persists).
>> > >
>> > > The only other configuration I changed is the heap size dedicated to
>> the
>> > > process, by setting "-Xms4096M -Xmx4096M" among the JAVA_ARGS in the
>> > > artemis.profile file.
>> > >
>> > >
>> > > The platform where Artemis broker is installed is:
>> > >
>> > >
>> > > - Machine:
>> > >
>> > > It's an EC2 instance on AWS cloud of type m4.large: 2 vCPU, 8G RAM and
>> > SSD
>> > > storage.
>> > >
>> > >
>> > > - OS:
>> > >
>> > > CentOS Linux release 7.3.1611 (Core) - x86_64
>> > >
>> > >
>> > > - JVM:
>> > >
>> > > java version "1.8.0_111"
>> > > Java(TM) SE Runtime Environment (build 1.8.0_111-b14)
>> > > Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)
>> > >
>> > >
>> > > - Distro package version of libaio:
>> > >
>> > > Arch        : x86_64
>> > > Version     : 0.3.109
>> > > Release     : 13.el7
>> > >
>> > >
>> > > It seems like the problem relates the Persistence (File Journal?).
>> Until
>> > the
>> > > broker keeps the retained messages in ram it works. When it needs to
>> > recover
>> > > the retained messages  from the file system, clients receive them with
>> > weird
>> > > chars.
>> > > And I suspect the same issue appears when RAM is full and messages
>> start
>> > to
>> > > be paged... but this case is a bit more difficult to test in a
>> systematic
>> > > way.
>> > >
>> > >
>> > > Francesco
>> > >
>> > >
>> > >
>> > > ________________________________
>> > > From: Clebert Suconic <clebert.suconic@gmail.com>
>> > > Sent: Tuesday, January 10, 2017 7:43 PM
>> > > To: users@activemq.apache.org
>> > > Subject: Re: MQTT retained messages with weird characters
>> > >
>> > > Do you have a working test you can share?
>> > >
>> > > On Tue, Jan 10, 2017 at 12:07 PM, Francesco PADOVANI
>> > > <Francesco.PADOVANI@bticino.it> wrote:
>> > >> Hello,
>> > >>
>> > >> I'm trying the Apache Artemis Broker (ver. 1.5.1) for MQTT protocol.
>> > >>
>> > >> Exactly I'm testing the retained feature for messages of MQTT
>> protocol.
>> > >>
>> > >> While the broker is up it seems to work well:
>> > >>
>> > >> a) a client publishes a retained message to a specific topic
>> > >>
>> > >> b) any client which then subscribes that topic receives the retained
>> > >> message (any time it re-subscribes the topic it receives the last
>> > retained
>> > >> message ...perfect!)
>> > >>
>> > >> But it happens a strange thing when I restart the broker. And I mean
I
>> > >> restart the broker without any change on configuration: simply I make
>> a
>> > >> "artemis-service stop" and "artemis-service start". After The broker
>> is
>> > up
>> > >> and running, if a client subscribes the previous topic it still
>> > receives the
>> > >> retained message, but with weird characters appended to it. For new
>> > retained
>> > >> messages published it work well again... but the previous ones (before
>> > the
>> > >> broker restart) are sent all with these weird characters. E.g., the
>> > >> following is a retained message published before the broker restart:
>> > >>
>> > >> "test 25 for a retained message 20170110"
>> > >>
>> > >> And this is exactly how clients get it before the broker restart.
>> > >>
>> > >> Instead, after broker restart, the same retained message is received
>> by
>> > >> clients in the following way:
>> > >>
>> > >> test 25 for a retained message 20170110?
>> > >>                                         ?
>> 6$sys.mqtt.retain..cro.test5 ?
>> > >> &mqtt.message.retain ? mqtt.qos.level
>> > >>
>> > >>
>> > >> Is this my bad configuration (but I don't know ehere)? Or anything
>> else?
>> > >> Or what?
>> > >>
>> > >> Someone can help me?
>> > >>
>> > >>
>> > >> Thanks in advance.
>> > >>
>> > >>
>> > >> Francesco
>> > >>
>> > >> ________________________________
>> > >>
>> > >> Ce message, ainsi que tous les fichiers joints à ce message, peuvent
>> > >> contenir des informations sensibles et/ ou confidentielles ne devant
>> pas
>> > >> être divulguées. Si vous n'êtes pas le destinataire de ce message
(ou
>> > que
>> > >> vous recevez ce message par erreur), nous vous remercions de le
>> notifier
>> > >> immédiatement à son expéditeur, et de détruire ce message. Toute
>> copie,
>> > >> divulgation, modification, utilisation ou diffusion, non autorisée,
>> > directe
>> > >> ou indirecte, de tout ou partie de ce message, est strictement
>> > interdite.
>> > >>
>> > >>
>> > >> This e-mail, and any document attached hereby, may contain
>> confidential
>> > >> and/or privileged information. If you are not the intended recipient
>> (or
>> > >> have received this e-mail in error) please notify the sender
>> > immediately and
>> > >> destroy this e-mail. Any unauthorized, direct or indirect, copying,
>> > >> disclosure, distribution or other use of the material or parts thereof
>> > is
>> > >> strictly forbidden.
>> > >
>> > >
>> > >
>> > > --
>> > > Clebert Suconic
>> > >
>> > > ________________________________
>> > >
>> > > Ce message, ainsi que tous les fichiers joints à ce message, peuvent
>> > > contenir des informations sensibles et/ ou confidentielles ne devant
>> pas
>> > > être divulguées. Si vous n'êtes pas le destinataire de ce message (ou
>> que
>> > > vous recevez ce message par erreur), nous vous remercions de le
>> notifier
>> > > immédiatement à son expéditeur, et de détruire ce message. Toute copie,
>> > > divulgation, modification, utilisation ou diffusion, non autorisée,
>> > directe
>> > > ou indirecte, de tout ou partie de ce message, est strictement
>> interdite.
>> > >
>> > >
>> > > This e-mail, and any document attached hereby, may contain confidential
>> > > and/or privileged information. If you are not the intended recipient
>> (or
>> > > have received this e-mail in error) please notify the sender
>> immediately
>> > and
>> > > destroy this e-mail. Any unauthorized, direct or indirect, copying,
>> > > disclosure, distribution or other use of the material or parts thereof
>> is
>> > > strictly forbidden.
>> >
>> >
>> >
>> > --
>> > Clebert Suconic
>> >
>>
>> ________________________________
>>
>> Ce message, ainsi que tous les fichiers joints à ce message, peuvent
>> contenir des informations sensibles et/ ou confidentielles ne devant pas
>> être divulguées. Si vous n'êtes pas le destinataire de ce message (ou que
>> vous recevez ce message par erreur), nous vous remercions de le notifier
>> immédiatement à son expéditeur, et de détruire ce message. Toute copie,
>> divulgation, modification, utilisation ou diffusion, non autorisée, directe
>> ou indirecte, de tout ou partie de ce message, est strictement interdite.
>>
>>
>> This e-mail, and any document attached hereby, may contain confidential
>> and/or privileged information. If you are not the intended recipient (or
>> have received this e-mail in error) please notify the sender immediately
>> and destroy this e-mail. Any unauthorized, direct or indirect, copying,
>> disclosure, distribution or other use of the material or parts thereof is
>> strictly forbidden.
>>



-- 
Clebert Suconic

Mime
View raw message