plc4x-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christofer Dutz <christofer.d...@c-ware.de>
Subject Re: Antwort: Re: Antwort: Re: Antwort: Re: Antwort: Re: Anyone got experience with "protocol buffers" or DFDL (for generation of the message (de)serialization code)?
Date Thu, 10 Jan 2019 09:41:27 GMT
Hi Markus,

that is not quite true.

In some cases definitely yes, but I'm currently looking into the option of using DFDL (Apache
Daffodil) for this.
The way I would like to do it, would be to use DFDL to describe the format and then generate
code from that with a general purpose generator for which we can provide the code-generation
templates.
This way we would have to write these templates once and exactly for PLC4X and then have all
protocols generated to perfectly fit. 

Right now I am not doing anything different ... I'm trying to refactor things in a way that
is as generic as possible without any performance drawbacks and then to adjust one protocol
after the other to match that.

I would like to automate this. 

So as soon as a new language should be supported, someone would setup the general skeleton
driver, write a prototype and then convert that into templates and "presto" we get all the
layers generated.

At least that's what I'm currently thinking of.

Chris



Am 10.01.19, 10:33 schrieb "Markus Sommer" <sommer@isb-fn.de>:

    Hello everyone,
    
    If we go the way of automatic object generation for messages, then we will have to live
with performance degradation.
    
    Best regards
    
    Markus
    
    Mit freundlichen Grüßen
    
    Markus Sommer
    Geschäftsführer
    
    isb innovative software businesses GmbH
    Otto-Lilienthal-Straße 2
    D 88046 Friedrichshafen
    
    Tel          +49 (0) 7541 3834-14
    Fax         +49 (0) 7541 3834-20
    Mobil    +49 (0) 171 537 8437
    
    Mail to  sommer@isb-fn.de
    http://www.isb-fn.de
    
    Geschäftsführer: Markus Sommer, Thomas Zeler
    Registergericht: Amtsgericht Ulm HRB-Nr. 631624
    Important Note: This e-mail and any attachments are confidential, may contain trade secrets
and may well also be legally privileged or otherwise protected from disclosure. If you have
received it in error, you are on notice of its status. Please notify us immediately by reply
e-mail and then delete this e-mail and any attachment from your system. If you are not the
intended recipient please understand that you must not copy this e-mail or any attachments
or disclose the contents to any other person. 
    Thank you.
    
    
    
    -----Ursprüngliche Nachricht-----
    Von: Christofer Dutz <christofer.dutz@c-ware.de> 
    Gesendet: Mittwoch, 9. Januar 2019 18:59
    An: dev@plc4x.apache.org
    Betreff: Re: Antwort: Re: Antwort: Re: Antwort: Re: Antwort: Re: Anyone got experience
with "protocol buffers" or DFDL (for generation of the message (de)serialization code)?
    
    Well I thought that went could use what's already there as I was expecting us but to be
the first. But I haven't found an option that works.
    
    Of course whipping up a coffee generator is quite simple (done it loads of times) but
you have to come up with the serialisation for all supported languages ourself. I wanted to
avoid that, cause it's a lot of work.
    
    I think we should do a little more evaluating. If we don't find anything well go down
that path.
    
    But having plugins used in a build that are also part of the build itself introduces more
issues. So perhaps becoming involved in the daffodil project and adding a code generator there
is the better option. (Code generator in Daffodil and generation templates here)
    
    Chris
    
    Outlook for Android<https://aka.ms/ghei36> herunterladen
    
    ________________________________
    From: Julian Feinauer <j.feinauer@pragmaticminds.de>
    Sent: Wednesday, January 9, 2019 6:46:14 PM
    To: dev@plc4x.apache.org
    Subject: Re: Antwort: Re: Antwort: Re: Antwort: Re: Antwort: Re: Anyone got experience
with "protocol buffers" or DFDL (for generation of the message (de)serialization code)?
    
    Okay, I see...
    When I learned something, that tooling is really important, and it makes me a bit of sorrows
that DFDL is not there yet.
    Do I take things too simple if we would just starting using something like json (or xml,
as I think chris likes xml better) to define these messages and then implement a (very simple)
converter to the class format?
    Shouldn’t this be doable?
    
    Of course there's still the lack of a Maven Plugin but I think this could be made easily
(????) if its just invocing a method (or we do something like ant task voodoo to invoke it
via CMD).
    But at least we would have nice syntax highlighting and such stuff and could rely on Jackson
to get the files to a Java Input (and in case of xml we could also do validation directly
and generate nice HTML documentatsions via xlst).
    
    Am I missing something or take things too simple?
    
    Best
    Julian
    
    Am 09.01.19, 18:26 schrieb "Christofer Dutz" <christofer.dutz@c-ware.de>:
    
        Hi Julian,
    
        Yes exactly ... Unfortunately the messages are composed of different types of structures.
Header, parameters of different type and length, payloads of different type. But in general,
yes
    
        Chris
    
        Outlook for Android<https://aka.ms/ghei36> herunterladen
    
        ________________________________
        From: Julian Feinauer <j.feinauer@pragmaticminds.de>
        Sent: Wednesday, January 9, 2019 6:16:01 PM
        To: dev@plc4x.apache.org
        Subject: Re: Antwort: Re: Antwort: Re: Antwort: Re: Antwort: Re: Anyone got experience
with "protocol buffers" or DFDL (for generation of the message (de)serialization code)?
    
        Hi Chris,
    
        I'm not sure if I got it right, but what we would need is a way to generate classes
in multiple languages based on a message definition, or?
        So we say something like
        Bit, byte, byte, bit
        And it generates a class (or struct?) which deserializes / serializes to that, or?
    
        Sorry for that (probably) dump question but I got a bit confused by this protobuf
/ thrift approach.
    
        Julian
    
        Am 09.01.19, 17:49 schrieb "Christofer Dutz" <christofer.dutz@c-ware.de>:
    
            Hi Sebastian,
    
            Well in general the difference between a read and a write in S7 is one byte value
;-)
            The overall structure is somewhat identical.
    
            Chris
    
    
    
            Von: Sebastian Wiendl <SWiendl@bhs-corrugated.de>
            Antworten an: "dev@plc4x.apache.org" <dev@plc4x.apache.org>
            Datum: Mittwoch, 9. Januar 2019 um 17:21
            An: "dev@plc4x.apache.org" <dev@plc4x.apache.org>
            Betreff: Antwort: Re: Antwort: Re: Antwort: Re: Antwort: Re: Anyone got experience
with "protocol buffers" or DFDL (for generation of the message (de)serialization code)?
    
            The developer's struggle: Which foot to shoot yourself in. ;-)
    
            I have no detailed knowledge about the protocols you want to serialize/deserialize
- maybe they are simple enough that the mentioned early serialization features of Kaitai are
sufficient. If not, I wouldn't recommend using it.
    
            Another aspect might be the anticipated distribution of read and write workloads
of a typical PLC4X application - if its mostly reads it might still be worth a try, because
you save a lot of (implementation) effort on the deserialization which frees up resources
for serialization or other things in general.
    
            Mit freundlichen Grüßen
            Kind regards
    
            Sebastian Wiendl
            DSE / Digital Solutions Software Engineer
            Phone: +49 9605 919 - 9341
            E-Mail: SWiendl@bhs-corrugated.de
            Internet: http://www.bhs-world.com
            ________________________________
             [cid:_1_13C7BE3013C7BA340059D4ACC125837D]
    
            BHS Corrugated Maschinen- und Anlagenbau GmbH
            Paul-Engel-Straße 1
            92729 WEIHERHAMMER
            GERMANY
    
            Management: Christian Engel, Lars Engel, Norbert Städele
            Registered at Amtsgericht Weiden, HR B 1320
    
            [cid:_2_13C7CB4413C7C7340059D4ACC125837D]
    
    
            ________________________________
    
    
            Diese Nachricht ist nur für den Empfänger bestimmt, da sie persönliche und/oder
rechtlich geschützte Informationen und vertrauliche Informationen enthält.
            Der Inhalt darf an keinen anderen außer an den Empfänger weitergeleitet werden.
Unbefugte Empfänger werden gebeten, diese Vertraulichkeit zu wahren und den Absender sofort
von diesem Übertragungsfehler in Kenntnis zu setzen.
            Vernichten Sie diese Nachricht. Das unerlaubte Kopieren sowie die unbefugte Weitergabe
dieser Nachricht ist nicht gestattet.
    
            This message is intended for the addressee only as it contains private and/or
privileged and confidential information.
            The contents are not to be disclosed to anyone else than the addressee. Unauthorized
recipients are requested to comply with the above and to inform the sender immediately of
any errors in transmission and destroy this e-mail.
            Any unauthorized copying, disclosure or distribution of the material in this e-mail
is strictly prohibited.
    
    
    
    
            Von:        "Christofer Dutz" <christofer.dutz@c-ware.de>
            An:        "dev@plc4x.apache.org" <dev@plc4x.apache.org>
            Datum:        09.01.2019 17:09
            Betreff:        Re: Antwort: Re: Antwort: Re: Antwort: Re: Anyone got experience
with "protocol buffers" or DFDL (for generation of the message (de)serialization code)?
            ________________________________
    
    
    
            Hi Sebastian,
    
            yeah … I asked in the Incubator list and they told me it would be safe to use,
if the output was not GPLed …
    
            so I contacted the maintainer of the project and simply asked him.
            He told me that the output would match the input license. So if our definitions
would be Apache 2.0, so would the output … so it seems we would be safe and able to use
that.
            However he also told me this:
    
            “That's great news! Please note, however, that serialization is in its
            early stages in Kaitai Struct, so it might be no exactly up to your
            expectations.“
    
            So I don’t know if we should go down that path … right now … :-/
    
            Chris
    
    
    
            Von: Sebastian Wiendl <SWiendl@bhs-corrugated.de>
            Antworten an: "dev@plc4x.apache.org" <dev@plc4x.apache.org>
            Datum: Mittwoch, 9. Januar 2019 um 16:46
            An: "dev@plc4x.apache.org" <dev@plc4x.apache.org>
            Betreff: Antwort: Re: Antwort: Re: Antwort: Re: Anyone got experience with "protocol
buffers" or DFDL (for generation of the message (de)serialization code)?
    
            Another thing to consider: Last time I checked Kaitai only supported deserialization
(https://github.com/kaitai-io/kaitai_struct/issues/27). Might be inconvenient to introduce
another framework for serialization...
    
            Mit freundlichen Grüßen
            Kind regards
    
            Sebastian Wiendl
            DSE / Digital Solutions Software Engineer
            Phone: +49 9605 919 - 9341
            E-Mail: SWiendl@bhs-corrugated.de
            Internet: http://www.bhs-world.com<http://www.bhs-world.com/>
            ________________________________
            [cid:_1_147DDC68147DD8840056A390C125837D]
    
            BHS Corrugated Maschinen- und Anlagenbau GmbH
            Paul-Engel-Straße 1
            92729 WEIHERHAMMER
            GERMANY
    
            Management: Christian Engel, Lars Engel, Norbert Städele
            Registered at Amtsgericht Weiden, HR B 1320
    
            [cid:_2_147DE97C147DE56C0056A390C125837D]
    
    
            ________________________________
    
    
            Diese Nachricht ist nur für den Empfänger bestimmt, da sie persönliche und/oder
rechtlich geschützte Informationen und vertrauliche Informationen enthält.
            Der Inhalt darf an keinen anderen außer an den Empfänger weitergeleitet werden.
Unbefugte Empfänger werden gebeten, diese Vertraulichkeit zu wahren und den Absender sofort
von diesem Übertragungsfehler in Kenntnis zu setzen.
            Vernichten Sie diese Nachricht. Das unerlaubte Kopieren sowie die unbefugte Weitergabe
dieser Nachricht ist nicht gestattet.
    
            This message is intended for the addressee only as it contains private and/or
privileged and confidential information.
            The contents are not to be disclosed to anyone else than the addressee. Unauthorized
recipients are requested to comply with the above and to inform the sender immediately of
any errors in transmission and destroy this e-mail.
            Any unauthorized copying, disclosure or distribution of the material in this e-mail
is strictly prohibited.
    
    
    
    
            Von:        "Christofer Dutz" <christofer.dutz@c-ware.de>
            An:        "dev@plc4x.apache.org" <dev@plc4x.apache.org>
            Datum:        09.01.2019 15:57
            Betreff:        Re: Antwort: Re: Antwort: Re: Anyone got experience with "protocol
buffers" or DFDL (for generation of the message (de)serialization code)?
            ________________________________
    
    
    
            The thing is that we would actually not be bundling that … the compiler is only
used during the build and not at runtime.
            I am currently double-checking this with the others in the incubator mailing-list.
    
            So it’s not completely impossible … let’s see what the others have to say
about it.
    
            But I agree … it does look as if it would suit our needs.
    
            Chris
    
    
            Von: Sebastian Wiendl <SWiendl@bhs-corrugated.de>
            Antworten an: "dev@plc4x.apache.org" <dev@plc4x.apache.org>
            Datum: Mittwoch, 9. Januar 2019 um 15:42
            An: "dev@plc4x.apache.org" <dev@plc4x.apache.org>
            Betreff: Antwort: Re: Antwort: Re: Anyone got experience with "protocol buffers"
or DFDL (for generation of the message (de)serialization code)?
    
            Yeah, i just checked... bummer the compiler is GPLed...
    
            The project itself has some really good ideas how to handle binary parsing. I
used it prototypically in decoding a proprietary UDP message format and it worked great.
    
            Mit freundlichen Grüßen
            Kind regards
    
            Sebastian Wiendl
            DSE / Digital Solutions Software Engineer
            Phone: +49 9605 919 - 9341
            E-Mail: SWiendl@bhs-corrugated.de
            Internet: http://www.bhs-world.com<http://www.bhs-world.com/><http://www.bhs-world.com/>
            ________________________________
            [cid:_1_147D3C84147D38880050BA3BC125837D]
    
            BHS Corrugated Maschinen- und Anlagenbau GmbH
            Paul-Engel-Straße 1
            92729 WEIHERHAMMER
            GERMANY
    
            Management: Christian Engel, Lars Engel, Norbert Städele
            Registered at Amtsgericht Weiden, HR B 1320
    
            [cid:_2_147D4998147D45880050BA3BC125837D]
    
    
            ________________________________
    
    
            Diese Nachricht ist nur für den Empfänger bestimmt, da sie persönliche und/oder
rechtlich geschützte Informationen und vertrauliche Informationen enthält.
            Der Inhalt darf an keinen anderen außer an den Empfänger weitergeleitet werden.
Unbefugte Empfänger werden gebeten, diese Vertraulichkeit zu wahren und den Absender sofort
von diesem Übertragungsfehler in Kenntnis zu setzen.
            Vernichten Sie diese Nachricht. Das unerlaubte Kopieren sowie die unbefugte Weitergabe
dieser Nachricht ist nicht gestattet.
    
            This message is intended for the addressee only as it contains private and/or
privileged and confidential information.
            The contents are not to be disclosed to anyone else than the addressee. Unauthorized
recipients are requested to comply with the above and to inform the sender immediately of
any errors in transmission and destroy this e-mail.
            Any unauthorized copying, disclosure or distribution of the material in this e-mail
is strictly prohibited.
    
    
    
    
            Von:        "Christofer Dutz" <christofer.dutz@c-ware.de>
            An:        "dev@plc4x.apache.org" <dev@plc4x.apache.org>
            Datum:        09.01.2019 15:37
            Betreff:        Re: Antwort: Re: Anyone got experience with "protocol buffers"
or DFDL (for generation of the message (de)serialization code)?
            ________________________________
    
    
    
            Hi Sebasitan,
    
            thanks for that pointer … unfortunately I think we might be unable to use this
option as the compiler is GPLv3 licensed … that is a category X license.
            Even if the compiler is not bundled with our software or used at runtime (will
probably only need it at compile-time), I doubt we would be allowed to use it.
    
            But still I’ll look into it …
    
            Chris
    
            Von: Sebastian Wiendl <SWiendl@bhs-corrugated.de>
            Antworten an: "dev@plc4x.apache.org" <dev@plc4x.apache.org>
            Datum: Mittwoch, 9. Januar 2019 um 15:18
            An: "dev@plc4x.apache.org" <dev@plc4x.apache.org>
            Betreff: Antwort: Re: Anyone got experience with "protocol buffers" or DFDL (for
generation of the message (de)serialization code)?
    
            Hi,
    
            maybe the Kaitai Struct project (https://kaitai.io/) can help you achieve this?
    
            Mit freundlichen Grüßen
            Kind regards
    
            Sebastian Wiendl
            DSE / Digital Solutions Software Engineer
            Phone: +49 9605 919 - 9341
            E-Mail: SWiendl@bhs-corrugated.de
            Internet: http://www.bhs-world.com<http://www.bhs-world.com/><http://www.bhs-world.com/><http://www.bhs-world.com/>
            ________________________________
            [cid:_1_0DB11BC80DB117CC004E9C08C125837D]
    
            BHS Corrugated Maschinen- und Anlagenbau GmbH
            Paul-Engel-Straße 1
            92729 WEIHERHAMMER
            GERMANY
    
            Management: Christian Engel, Lars Engel, Norbert Städele
            Registered at Amtsgericht Weiden, HR B 1320
    
            [cid:_2_0DB128DC0DB124CC004E9C08C125837D]
    
    
            ________________________________
    
    
            Diese Nachricht ist nur für den Empfänger bestimmt, da sie persönliche und/oder
rechtlich geschützte Informationen und vertrauliche Informationen enthält.
            Der Inhalt darf an keinen anderen außer an den Empfänger weitergeleitet werden.
Unbefugte Empfänger werden gebeten, diese Vertraulichkeit zu wahren und den Absender sofort
von diesem Übertragungsfehler in Kenntnis zu setzen.
            Vernichten Sie diese Nachricht. Das unerlaubte Kopieren sowie die unbefugte Weitergabe
dieser Nachricht ist nicht gestattet.
    
            This message is intended for the addressee only as it contains private and/or
privileged and confidential information.
            The contents are not to be disclosed to anyone else than the addressee. Unauthorized
recipients are requested to comply with the above and to inform the sender immediately of
any errors in transmission and destroy this e-mail.
            Any unauthorized copying, disclosure or distribution of the material in this e-mail
is strictly prohibited.
    
    
    
    
            Von:        "Christofer Dutz" <christofer.dutz@c-ware.de>
            An:        "dev@plc4x.apache.org" <dev@plc4x.apache.org>
            Datum:        09.01.2019 15:06
            Betreff:        Re: Anyone got experience with "protocol buffers" or DFDL (for
generation of the message (de)serialization code)?
            ________________________________
    
    
    
            Hi all,
    
            Ok ... so protobuf seems to be semi-ideal ...
    
            It seems that you can use it to model the structure of data. Protobuf is good
for generating model classes, parsers and serializers for a given model ... the binary data-format
is a result of this.
    
            We want the opposite: We want to generate a model from a known output data-format.
In general this could be somehow achieved with protobuf, however it is very difficult to produce
the definition in a way that it is able to parse a given data format.
            For example simply outputting one byte seems to be problematic. I was able to
somehow hack an enum and provide some extension to allow providing code values, but we don't
have the level of control we would need to and the result is not very readable.
            I was able to quite easily setup the maven build to generate java code for parsing
and serializing a model ... so that was good.
    
            DFDL looks as if it's ideal for describing the data format, however I couldn't
find tooling to generate model, parser and serializer from a DFDL definition. I subscribed
to our brother incubating project Daffodil and asked on their list ... perhaps I have to get
my hands dirty and implement the maven plugin and code generators as part of that project
... I am hoping not having to do that.
    
            I'll check out Thrift in parallel  ;-)
    
    
            Chris
    
    
    
            Am 09.01.19, 11:19 schrieb "Christofer Dutz" <christofer.dutz@c-ware.de>:
    
            From my first look at thrift some time ago, that's more about API and not about
the actual payload, is it?
    
            How about I try to do a protobuf version of the "s7-protocol" and you give thrift
a try? Another option would be the DFDL option.
    
            Chris
    
            Am 09.01.19, 11:13 schrieb "Julian Feinauer" <j.feinauer@pragmaticminds.de>:
    
                Hi Chris,
    
                we worked (and work) with Thrift [1] at several places.
                Thrift is a strong contender to protobuf and both have their specific advantages
and disadvantages.
                Perhaps I would prefer Thrift as it comes from the Apache Ecosystm (and supports
more langauges) but generally, Tim can say more about working with Thrift.
    
                Best
                Julian
    
                [1] https://thrift.apache.org/
    
                Am 09.01.19, 10:45 schrieb "Christofer Dutz" <christofer.dutz@c-ware.de>:
    
                    Hi all,
    
                    while I’m currently working on refactoring the S7 driver to a simpler
structure so we can convert it to other languages more easily. A colleague of mine pointed
me to protobuf/protocol buffers from google [1]
                    From a quick look at it, it does seem as if it could suit our needs quite
nicely. I would like to try out if it’s possible to model the S7 data structures in this
way. If it works we could eventually quickly create something that serializes/deserializes
given data in any language …
    
                    It seems to be a lot simpler than the DFDL [2] I was thinking of, so guess
we have to find out if it has all the capabilities we need.
    
                    Any thoughts?
    
                    Chris
    
    
    
    
                    [1] https://developers.google.com/protocol-buffers/docs/javatutorial
                    [2] https://en.wikipedia.org/wiki/Data_Format_Description_Language
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    

Mime
View raw message