plc4x-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otto Fowler <ottobackwa...@gmail.com>
Subject Re: Antwort: Re: Antwort: Re: Antwort: Re: Antwort: Re: Anyone got experience with "protocol buffers" or DFDL (for generation of the message (de)serialization code)?
Date Tue, 15 Jan 2019 12:12:49 GMT
Maybe Chris can post the Jira Issues he has opened in Daffodil?


On January 15, 2019 at 04:33:20, Julian Feinauer (
j.feinauer@pragmaticminds.de) wrote:

Hi Chris,

thank you so much for your effort.
I think this ist he right way to go forward and get more languages
on-board.
Hopefully, we can gather some interest in the Daffodil community that it
doesn't all last on your shoulders (as I'm not capable of helping out
currently).

But perhaps also someone else from the list has an interest to jump in? : )

Best
Julian

Am 15.01.19, 10:24 schrieb "Christofer Dutz" <christofer.dutz@c-ware.de>:

Hi all,

just some feedback on my efforts towards formal definition of protocol
message types:

It seems that DFDL is a perfect match for defining the data formats.
Right now I'm working hard on writing a DFDL schema for the S7 Protocol as
this has quite a number of "specialities" that provide a challenge when
writing such a definition.
As soon as that's finished I guess we know if this is the way to go.

Unfortunately this is where the real work begins. As Daffodil (The Apache
implementation of a DFDL framework ... the other one is from IBM but not so
feature-rich) parses any data using DFDL schemas into a JSON or XML
presentation.
What it currently doesn't do is parse into a generic model and/or generate
code.

This is where I will probably have to invest some time in the Apache
Daffodil Incubator podling and help them implement the missing parts.
I know this is quite some work, but it will be a HUGE benefit for our
project and it will also help our brother project greatly.

What will be possible with Daffodil, is that we can replace the "pcapng"
binary captures with XML (or JSON) versions which Daffodil can already
serialize to binary data for tests ... So we get human readable test-case
input.
We can dump incoming packets into a human readable form and eventually
(automatically) generate test-case input for inspecting IO problems.
Also could we implement a low-performance version of new protocols, by
using the existing Daffodil parsers/serializers to process a protocol
specification to and from XML/JSON ... this should be a huge benefit for
new-protocol-implementors.
As soon as a protocol is roughly implemented, we can switch to generated
model, (de)serializers.

So far the update and a little excuse for my silence ;-)

Chris


Am 10.01.19, 10:41 schrieb "Christofer Dutz" <christofer.dutz@c-ware.de>:

Hi Markus,

that is not quite true.

In some cases definitely yes, but I'm currently looking into the option of
using DFDL (Apache Daffodil) for this.
The way I would like to do it, would be to use DFDL to describe the format
and then generate code from that with a general purpose generator for which
we can provide the code-generation templates.
This way we would have to write these templates once and exactly for PLC4X
and then have all protocols generated to perfectly fit.

Right now I am not doing anything different ... I'm trying to refactor
things in a way that is as generic as possible without any performance
drawbacks and then to adjust one protocol after the other to match that.

I would like to automate this.

So as soon as a new language should be supported, someone would setup the
general skeleton driver, write a prototype and then convert that into
templates and "presto" we get all the layers generated.

At least that's what I'm currently thinking of.

Chris



Am 10.01.19, 10:33 schrieb "Markus Sommer" <sommer@isb-fn.de>:

Hello everyone,

If we go the way of automatic object generation for messages, then we will
have to live with performance degradation.

Best regards

Markus

Mit freundlichen Grüßen

Markus Sommer
Geschäftsführer

isb innovative software businesses GmbH
Otto-Lilienthal-Straße 2
D 88046 Friedrichshafen

Tel +49 (0) 7541 3834-14
Fax +49 (0) 7541 3834-20
Mobil +49 (0) 171 537 8437

Mail to sommer@isb-fn.de
http://www.isb-fn.de

Geschäftsführer: Markus Sommer, Thomas Zeler
Registergericht: Amtsgericht Ulm HRB-Nr. 631624
Important Note: This e-mail and any attachments are confidential, may
contain trade secrets and may well also be legally privileged or otherwise
protected from disclosure. If you have received it in error, you are on
notice of its status. Please notify us immediately by reply e-mail and then
delete this e-mail and any attachment from your system. If you are not the
intended recipient please understand that you must not copy this e-mail or
any attachments or disclose the contents to any other person.
Thank you.



-----Ursprüngliche Nachricht-----
Von: Christofer Dutz <christofer.dutz@c-ware.de>
Gesendet: Mittwoch, 9. Januar 2019 18:59
An: dev@plc4x.apache.org
Betreff: Re: Antwort: Re: Antwort: Re: Antwort: Re: Antwort: Re: Anyone got
experience with "protocol buffers" or DFDL (for generation of the message
(de)serialization code)?

Well I thought that went could use what's already there as I was expecting
us but to be the first. But I haven't found an option that works.

Of course whipping up a coffee generator is quite simple (done it loads of
times) but you have to come up with the serialisation for all supported
languages ourself. I wanted to avoid that, cause it's a lot of work.

I think we should do a little more evaluating. If we don't find anything
well go down that path.

But having plugins used in a build that are also part of the build itself
introduces more issues. So perhaps becoming involved in the daffodil
project and adding a code generator there is the better option. (Code
generator in Daffodil and generation templates here)

Chris

Outlook for Android<https://aka.ms/ghei36> herunterladen

________________________________
From: Julian Feinauer <j.feinauer@pragmaticminds.de>
Sent: Wednesday, January 9, 2019 6:46:14 PM
To: dev@plc4x.apache.org
Subject: Re: Antwort: Re: Antwort: Re: Antwort: Re: Antwort: Re: Anyone got
experience with "protocol buffers" or DFDL (for generation of the message
(de)serialization code)?

Okay, I see...
When I learned something, that tooling is really important, and it makes me
a bit of sorrows that DFDL is not there yet.
Do I take things too simple if we would just starting using something like
json (or xml, as I think chris likes xml better) to define these messages
and then implement a (very simple) converter to the class format?
Shouldn’t this be doable?

Of course there's still the lack of a Maven Plugin but I think this could
be made easily (????) if its just invocing a method (or we do something
like ant task voodoo to invoke it via CMD).
But at least we would have nice syntax highlighting and such stuff and
could rely on Jackson to get the files to a Java Input (and in case of xml
we could also do validation directly and generate nice HTML documentatsions
via xlst).

Am I missing something or take things too simple?

Best
Julian

Am 09.01.19, 18:26 schrieb "Christofer Dutz" <christofer.dutz@c-ware.de>:

Hi Julian,

Yes exactly ... Unfortunately the messages are composed of different types
of structures. Header, parameters of different type and length, payloads of
different type. But in general, yes

Chris

Outlook for Android<https://aka.ms/ghei36> herunterladen

________________________________
From: Julian Feinauer <j.feinauer@pragmaticminds.de>
Sent: Wednesday, January 9, 2019 6:16:01 PM
To: dev@plc4x.apache.org
Subject: Re: Antwort: Re: Antwort: Re: Antwort: Re: Antwort: Re: Anyone got
experience with "protocol buffers" or DFDL (for generation of the message
(de)serialization code)?

Hi Chris,

I'm not sure if I got it right, but what we would need is a way to generate
classes in multiple languages based on a message definition, or?
So we say something like
Bit, byte, byte, bit
And it generates a class (or struct?) which deserializes / serializes to
that, or?

Sorry for that (probably) dump question but I got a bit confused by this
protobuf / thrift approach.

Julian

Am 09.01.19, 17:49 schrieb "Christofer Dutz" <christofer.dutz@c-ware.de>:

Hi Sebastian,

Well in general the difference between a read and a write in S7 is one byte
value ;-)
The overall structure is somewhat identical.

Chris



Von: Sebastian Wiendl <SWiendl@bhs-corrugated.de>
Antworten an: "dev@plc4x.apache.org" <dev@plc4x.apache.org>
Datum: Mittwoch, 9. Januar 2019 um 17:21
An: "dev@plc4x.apache.org" <dev@plc4x.apache.org>
Betreff: Antwort: Re: Antwort: Re: Antwort: Re: Antwort: Re: Anyone got
experience with "protocol buffers" or DFDL (for generation of the message
(de)serialization code)?

The developer's struggle: Which foot to shoot yourself in. ;-)

I have no detailed knowledge about the protocols you want to
serialize/deserialize - maybe they are simple enough that the mentioned
early serialization features of Kaitai are sufficient. If not, I wouldn't
recommend using it.

Another aspect might be the anticipated distribution of read and write
workloads of a typical PLC4X application - if its mostly reads it might
still be worth a try, because you save a lot of (implementation) effort on
the deserialization which frees up resources for serialization or other
things in general.

Mit freundlichen Grüßen
Kind regards

Sebastian Wiendl
DSE / Digital Solutions Software Engineer
Phone: +49 9605 919 - 9341
E-Mail: SWiendl@bhs-corrugated.de
Internet: http://www.bhs-world.com
________________________________
[cid:_1_13C7BE3013C7BA340059D4ACC125837D]

BHS Corrugated Maschinen- und Anlagenbau GmbH
Paul-Engel-Straße 1
92729 WEIHERHAMMER
GERMANY

Management: Christian Engel, Lars Engel, Norbert Städele
Registered at Amtsgericht Weiden, HR B 1320

[cid:_2_13C7CB4413C7C7340059D4ACC125837D]


________________________________


Diese Nachricht ist nur für den Empfänger bestimmt, da sie persönliche
und/oder rechtlich geschützte Informationen und vertrauliche Informationen
enthält.
Der Inhalt darf an keinen anderen außer an den Empfänger weitergeleitet
werden. Unbefugte Empfänger werden gebeten, diese Vertraulichkeit zu wahren
und den Absender sofort von diesem Übertragungsfehler in Kenntnis zu
setzen.
Vernichten Sie diese Nachricht. Das unerlaubte Kopieren sowie die unbefugte
Weitergabe dieser Nachricht ist nicht gestattet.

This message is intended for the addressee only as it contains private
and/or privileged and confidential information.
The contents are not to be disclosed to anyone else than the addressee.
Unauthorized recipients are requested to comply with the above and to
inform the sender immediately of any errors in transmission and destroy
this e-mail.
Any unauthorized copying, disclosure or distribution of the material in
this e-mail is strictly prohibited.




Von: "Christofer Dutz" <christofer.dutz@c-ware.de>
An: "dev@plc4x.apache.org" <dev@plc4x.apache.org>
Datum: 09.01.2019 17:09
Betreff: Re: Antwort: Re: Antwort: Re: Antwort: Re: Anyone got experience
with "protocol buffers" or DFDL (for generation of the message
(de)serialization code)?
________________________________



Hi Sebastian,

yeah … I asked in the Incubator list and they told me it would be safe to
use, if the output was not GPLed …

so I contacted the maintainer of the project and simply asked him.
He told me that the output would match the input license. So if our
definitions would be Apache 2.0, so would the output … so it seems we would
be safe and able to use that.
However he also told me this:

“That's great news! Please note, however, that serialization is in its
early stages in Kaitai Struct, so it might be no exactly up to your
expectations.“

So I don’t know if we should go down that path … right now … :-/

Chris



Von: Sebastian Wiendl <SWiendl@bhs-corrugated.de>
Antworten an: "dev@plc4x.apache.org" <dev@plc4x.apache.org>
Datum: Mittwoch, 9. Januar 2019 um 16:46
An: "dev@plc4x.apache.org" <dev@plc4x.apache.org>
Betreff: Antwort: Re: Antwort: Re: Antwort: Re: Anyone got experience with
"protocol buffers" or DFDL (for generation of the message (de)serialization
code)?

Another thing to consider: Last time I checked Kaitai only supported
deserialization (https://github.com/kaitai-io/kaitai_struct/issues/27).
Might be inconvenient to introduce another framework for serialization...

Mit freundlichen Grüßen
Kind regards

Sebastian Wiendl
DSE / Digital Solutions Software Engineer
Phone: +49 9605 919 - 9341
E-Mail: SWiendl@bhs-corrugated.de
Internet: http://www.bhs-world.com<http://www.bhs-world.com/>
________________________________
[cid:_1_147DDC68147DD8840056A390C125837D]

BHS Corrugated Maschinen- und Anlagenbau GmbH
Paul-Engel-Straße 1
92729 WEIHERHAMMER
GERMANY

Management: Christian Engel, Lars Engel, Norbert Städele
Registered at Amtsgericht Weiden, HR B 1320

[cid:_2_147DE97C147DE56C0056A390C125837D]


________________________________


Diese Nachricht ist nur für den Empfänger bestimmt, da sie persönliche
und/oder rechtlich geschützte Informationen und vertrauliche Informationen
enthält.
Der Inhalt darf an keinen anderen außer an den Empfänger weitergeleitet
werden. Unbefugte Empfänger werden gebeten, diese Vertraulichkeit zu wahren
und den Absender sofort von diesem Übertragungsfehler in Kenntnis zu
setzen.
Vernichten Sie diese Nachricht. Das unerlaubte Kopieren sowie die unbefugte
Weitergabe dieser Nachricht ist nicht gestattet.

This message is intended for the addressee only as it contains private
and/or privileged and confidential information.
The contents are not to be disclosed to anyone else than the addressee.
Unauthorized recipients are requested to comply with the above and to
inform the sender immediately of any errors in transmission and destroy
this e-mail.
Any unauthorized copying, disclosure or distribution of the material in
this e-mail is strictly prohibited.




Von: "Christofer Dutz" <christofer.dutz@c-ware.de>
An: "dev@plc4x.apache.org" <dev@plc4x.apache.org>
Datum: 09.01.2019 15:57
Betreff: Re: Antwort: Re: Antwort: Re: Anyone got experience with "protocol
buffers" or DFDL (for generation of the message (de)serialization code)?
________________________________



The thing is that we would actually not be bundling that … the compiler is
only used during the build and not at runtime.
I am currently double-checking this with the others in the incubator
mailing-list.

So it’s not completely impossible … let’s see what the others have to say
about it.

But I agree … it does look as if it would suit our needs.

Chris


Von: Sebastian Wiendl <SWiendl@bhs-corrugated.de>
Antworten an: "dev@plc4x.apache.org" <dev@plc4x.apache.org>
Datum: Mittwoch, 9. Januar 2019 um 15:42
An: "dev@plc4x.apache.org" <dev@plc4x.apache.org>
Betreff: Antwort: Re: Antwort: Re: Anyone got experience with "protocol
buffers" or DFDL (for generation of the message (de)serialization code)?

Yeah, i just checked... bummer the compiler is GPLed...

The project itself has some really good ideas how to handle binary parsing.
I used it prototypically in decoding a proprietary UDP message format and
it worked great.

Mit freundlichen Grüßen
Kind regards

Sebastian Wiendl
DSE / Digital Solutions Software Engineer
Phone: +49 9605 919 - 9341
E-Mail: SWiendl@bhs-corrugated.de
Internet: http://www.bhs-world.com<http://www.bhs-world.com/><
http://www.bhs-world.com/>
________________________________
[cid:_1_147D3C84147D38880050BA3BC125837D]

BHS Corrugated Maschinen- und Anlagenbau GmbH
Paul-Engel-Straße 1
92729 WEIHERHAMMER
GERMANY

Management: Christian Engel, Lars Engel, Norbert Städele
Registered at Amtsgericht Weiden, HR B 1320

[cid:_2_147D4998147D45880050BA3BC125837D]


________________________________


Diese Nachricht ist nur für den Empfänger bestimmt, da sie persönliche
und/oder rechtlich geschützte Informationen und vertrauliche Informationen
enthält.
Der Inhalt darf an keinen anderen außer an den Empfänger weitergeleitet
werden. Unbefugte Empfänger werden gebeten, diese Vertraulichkeit zu wahren
und den Absender sofort von diesem Übertragungsfehler in Kenntnis zu
setzen.
Vernichten Sie diese Nachricht. Das unerlaubte Kopieren sowie die unbefugte
Weitergabe dieser Nachricht ist nicht gestattet.

This message is intended for the addressee only as it contains private
and/or privileged and confidential information.
The contents are not to be disclosed to anyone else than the addressee.
Unauthorized recipients are requested to comply with the above and to
inform the sender immediately of any errors in transmission and destroy
this e-mail.
Any unauthorized copying, disclosure or distribution of the material in
this e-mail is strictly prohibited.




Von: "Christofer Dutz" <christofer.dutz@c-ware.de>
An: "dev@plc4x.apache.org" <dev@plc4x.apache.org>
Datum: 09.01.2019 15:37
Betreff: Re: Antwort: Re: Anyone got experience with "protocol buffers" or
DFDL (for generation of the message (de)serialization code)?
________________________________



Hi Sebasitan,

thanks for that pointer … unfortunately I think we might be unable to use
this option as the compiler is GPLv3 licensed … that is a category X
license.
Even if the compiler is not bundled with our software or used at runtime
(will probably only need it at compile-time), I doubt we would be allowed
to use it.

But still I’ll look into it …

Chris

Von: Sebastian Wiendl <SWiendl@bhs-corrugated.de>
Antworten an: "dev@plc4x.apache.org" <dev@plc4x.apache.org>
Datum: Mittwoch, 9. Januar 2019 um 15:18
An: "dev@plc4x.apache.org" <dev@plc4x.apache.org>
Betreff: Antwort: Re: Anyone got experience with "protocol buffers" or DFDL
(for generation of the message (de)serialization code)?

Hi,

maybe the Kaitai Struct project (https://kaitai.io/) can help you achieve
this?

Mit freundlichen Grüßen
Kind regards

Sebastian Wiendl
DSE / Digital Solutions Software Engineer
Phone: +49 9605 919 - 9341
E-Mail: SWiendl@bhs-corrugated.de
Internet: http://www.bhs-world.com<http://www.bhs-world.com/><
http://www.bhs-world.com/><http://www.bhs-world.com/>
________________________________
[cid:_1_0DB11BC80DB117CC004E9C08C125837D]

BHS Corrugated Maschinen- und Anlagenbau GmbH
Paul-Engel-Straße 1
92729 WEIHERHAMMER
GERMANY

Management: Christian Engel, Lars Engel, Norbert Städele
Registered at Amtsgericht Weiden, HR B 1320

[cid:_2_0DB128DC0DB124CC004E9C08C125837D]


________________________________


Diese Nachricht ist nur für den Empfänger bestimmt, da sie persönliche
und/oder rechtlich geschützte Informationen und vertrauliche Informationen
enthält.
Der Inhalt darf an keinen anderen außer an den Empfänger weitergeleitet
werden. Unbefugte Empfänger werden gebeten, diese Vertraulichkeit zu wahren
und den Absender sofort von diesem Übertragungsfehler in Kenntnis zu
setzen.
Vernichten Sie diese Nachricht. Das unerlaubte Kopieren sowie die unbefugte
Weitergabe dieser Nachricht ist nicht gestattet.

This message is intended for the addressee only as it contains private
and/or privileged and confidential information.
The contents are not to be disclosed to anyone else than the addressee.
Unauthorized recipients are requested to comply with the above and to
inform the sender immediately of any errors in transmission and destroy
this e-mail.
Any unauthorized copying, disclosure or distribution of the material in
this e-mail is strictly prohibited.




Von: "Christofer Dutz" <christofer.dutz@c-ware.de>
An: "dev@plc4x.apache.org" <dev@plc4x.apache.org>
Datum: 09.01.2019 15:06
Betreff: Re: Anyone got experience with "protocol buffers" or DFDL (for
generation of the message (de)serialization code)?
________________________________



Hi all,

Ok ... so protobuf seems to be semi-ideal ...

It seems that you can use it to model the structure of data. Protobuf is
good for generating model classes, parsers and serializers for a given
model ... the binary data-format is a result of this.

We want the opposite: We want to generate a model from a known output
data-format. In general this could be somehow achieved with protobuf,
however it is very difficult to produce the definition in a way that it is
able to parse a given data format.
For example simply outputting one byte seems to be problematic. I was able
to somehow hack an enum and provide some extension to allow providing code
values, but we don't have the level of control we would need to and the
result is not very readable.
I was able to quite easily setup the maven build to generate java code for
parsing and serializing a model ... so that was good.

DFDL looks as if it's ideal for describing the data format, however I
couldn't find tooling to generate model, parser and serializer from a DFDL
definition. I subscribed to our brother incubating project Daffodil and
asked on their list ... perhaps I have to get my hands dirty and implement
the maven plugin and code generators as part of that project ... I am
hoping not having to do that.

I'll check out Thrift in parallel ;-)


Chris



Am 09.01.19, 11:19 schrieb "Christofer Dutz" <christofer.dutz@c-ware.de>:

>From my first look at thrift some time ago, that's more about API and not
about the actual payload, is it?

How about I try to do a protobuf version of the "s7-protocol" and you give
thrift a try? Another option would be the DFDL option.

Chris

Am 09.01.19, 11:13 schrieb "Julian Feinauer" <j.feinauer@pragmaticminds.de>:


Hi Chris,

we worked (and work) with Thrift [1] at several places.
Thrift is a strong contender to protobuf and both have their specific
advantages and disadvantages.
Perhaps I would prefer Thrift as it comes from the Apache Ecosystm (and
supports more langauges) but generally, Tim can say more about working with
Thrift.

Best
Julian

[1] https://thrift.apache.org/

Am 09.01.19, 10:45 schrieb "Christofer Dutz" <christofer.dutz@c-ware.de>:

Hi all,

while I’m currently working on refactoring the S7 driver to a simpler
structure so we can convert it to other languages more easily. A colleague
of mine pointed me to protobuf/protocol buffers from google [1]
>From a quick look at it, it does seem as if it could suit our needs quite
nicely. I would like to try out if it’s possible to model the S7 data
structures in this way. If it works we could eventually quickly create
something that serializes/deserializes given data in any language …

It seems to be a lot simpler than the DFDL [2] I was thinking of, so guess
we have to find out if it has all the capabilities we need.

Any thoughts?

Chris




[1] https://developers.google.com/protocol-buffers/docs/javatutorial
[2] https://en.wikipedia.org/wiki/Data_Format_Description_Language

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message