From dev-return-2193-archive-asf-public=cust-asf.ponee.io@plc4x.apache.org Thu May 9 17:48:38 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 5CFA4180649 for ; Thu, 9 May 2019 19:48:38 +0200 (CEST) Received: (qmail 54051 invoked by uid 500); 9 May 2019 17:48:37 -0000 Mailing-List: contact dev-help@plc4x.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@plc4x.apache.org Delivered-To: mailing list dev@plc4x.apache.org Received: (qmail 54007 invoked by uid 99); 9 May 2019 17:48:37 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 May 2019 17:48:37 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id F0589CAD38 for ; Thu, 9 May 2019 17:48:36 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.002 X-Spam-Level: X-Spam-Status: No, score=0.002 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, KAM_SHORT=0.001, RCVD_IN_DNSWL_NONE=-0.0001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=code-house-org.20150623.gappssmtp.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id L_GmpgfwHwgh for ; Thu, 9 May 2019 17:48:32 +0000 (UTC) Received: from mail-wm1-f66.google.com (mail-wm1-f66.google.com [209.85.128.66]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 337EE612D6 for ; Thu, 9 May 2019 17:48:32 +0000 (UTC) Received: by mail-wm1-f66.google.com with SMTP id q15so251579wmj.0 for ; Thu, 09 May 2019 10:48:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=code-house-org.20150623.gappssmtp.com; s=20150623; h=subject:to:references:from:openpgp:autocrypt:cc:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=rJqZI9p9TU9Nq8b/uJY6+ViGSQZKkDCC4eyrOkY806Y=; b=MW47xHPl/pdEFi5I3vCjAAv2+/H5zcT4mgsh6bHMBKSlxOBsiBgD9kG+xVyVqvoov6 /Ru+q0ptDRj5dz4tWfkGZAqcgdGsHTAl9mpg3KrPTM7TGlUA9vO61h5m70pRMbrMT3dm +jBPXm1PxPM8emI5MQI6r//YNNsDi98t7wZMjeYioZRgq/myGGtc8RQly9eA6kV3rMv5 ueeX4imxGaByP3EShc5wuf5edRflLpumzMTB6hJXA7O6TsCBPYEdcEcgBpJjrbG78MYb dU70ZwOgGlnkJhG6OMdM1di5JRMm2HyIL9ZldPB9FZvn+tbjDo7y7jluMTXgZWQ6BMo7 M9Ow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:openpgp:autocrypt:cc :message-id:date:user-agent:mime-version:in-reply-to :content-language:content-transfer-encoding; bh=rJqZI9p9TU9Nq8b/uJY6+ViGSQZKkDCC4eyrOkY806Y=; b=c4NNV8eZuRUXRuyPx7MM13kZcubJYHxxPLyjoUYjsozA2XLxgyHmoeub08XSazEURj C6YdzRIbJ6kWZegJXEmnCG3ealsXP5UJnOSLnisiCIiTvWuFD5mU06B16tnF3YPP1ZRW n+iTXp4RkYnUlQCfnnDe32Jmu6UFLVnms8jPrw8EkrdPQMe0LFOIJqrVZ5Hl0JlepNUl yqtiTrwMKVfaFxHlEI61m4RJ6OlG0RzaFb8rk06VJ7vlMoTghySlzaOKYsPVVOwUrsd2 izh7FZH0ztsGS1Yt94HzGfPSE9k74xGN/5omzvOnrxL0jqeYTC9FIWTwN211MW00sJQL 3HyA== X-Gm-Message-State: APjAAAWUs2lVPumrGJ9NqaOI6ciISoIYUabZY3QJpFIELycqJXgVM3Gg axSjt2ZroxQVAfJbKmML2ovLQvUungZRXw== X-Google-Smtp-Source: APXvYqzROhY7XOIxiGotJpzAMrHYe6tuBZ6UK4lRHV3GRKneZR/Sn5SxBToi0idoaGuYbzzo5Ymopg== X-Received: by 2002:a7b:c5c4:: with SMTP id n4mr3914610wmk.151.1557424110891; Thu, 09 May 2019 10:48:30 -0700 (PDT) Received: from [192.168.2.106] (18.ip-217-182-76.eu. [217.182.76.18]) by smtp.gmail.com with ESMTPSA id s11sm8119436wrb.71.2019.05.09.10.48.28 (version=TLS1_3 cipher=AEAD-AES128-GCM-SHA256 bits=128/128); Thu, 09 May 2019 10:48:30 -0700 (PDT) Subject: Re: [DriverGen] Possible solution for type inheritance To: dev@plc4x.apache.org References: <332161C3-0BE5-45BA-B0AC-EED135192E5E@c-ware.de> <27B52551-A0A2-4829-BE0B-C8D5276E71A7@c-ware.de> From: =?UTF-8?Q?=c5=81ukasz_Dywicki?= Openpgp: preference=signencrypt Autocrypt: addr=luke@code-house.org; prefer-encrypt=mutual; keydata= mQINBFbvMOABEADeAG5uLcccuUO7RfEdu4sEyhZgb0ZdrlLKkJv+QgwHpjgvKYg16YTLi0/0 ZO2OvBKUAEOf5/Z6NhEVtHiqUUeQ0Qvw/iv7VsxW944xC4rpiyL81u8FFkphgQfPM8HQjf6D 6zG7Ds90M27OblKCXA4VGmqIh3fhcyoyWZ/GMZHTtHcGRZtTMt9vkcmCxMgDJnKDmaL87ko/ ib7GR8HJa9UZ940qW7ecAG40Z008fGQTzOQ8DbGDUKS0+a5egndUk7l9ypcI3A/PulHf9i3z Bgqo6uRZzkwgUxgmR3n1qe1z7iRGgPL810Yvn/5leFimFYGciNlj1zIdGg5Len5lyRetSd8w U/ARcPnSFnzTnZTrYlYGwhQNvcuC+2m/iKOgXy/tPZOxoeJLfeoFKA8EoSFit6AfYK0CZ1Sf W2soN7FkPYJxxCW74l2KIioZrI/Q/zoMNLk1eapqcsgIHUERDOSqSEJFyd9SR8iWEIAenC0g 5WNoFd19grgZ6FUuXzb79/cxrwZbGA6NxzJU497wUPOxphUKqIwZQw89Xgq2qiQaO6tKHd1Q 0jpRnkGAteYfNcy9LnRIo/2/aMQEltLgHU+Z//gzkBlz6/XChdA7Vcss9AIOK0pFb+BY1QAA wDBwzFjSEhTZsyzOk4er2us8of0jdlVTzSXuh2R+uVgyXIfnfwARAQABtDTFgXVrYXN6IER5 d2lja2kgKHBncC1tYWlsLWtleSkgPGx1a2VAY29kZS1ob3VzZS5vcmc+iQI/BBMBCAApBQJW 7zDgAhsDBQkHhh+ABwsJCAcDAgEGFQgCCQoLBBYCAwECHgECF4AACgkQAdrcj4PeJmUuTxAA jnxIUMqQaAJWL8f4ZksxdrHj8f54H39M/P02Tmf0875LyT28R4lc0dvTX0VlfhDay0cGSg4w AwqCNozzTu4f0JnrUH2/HTTLgWHMxtK/X65Y9xyxpmfnNPm8YZ7aLm7zDIkEDPAEwaMtRzxM Yy9IBupg+l/N9EvX7tEQjhm94MFj4NmP+sv0IYMWfNjtKYUpeOhSxre5MLOAeDqFG25TT/+d Uf58BekEGK7OeA6aXa+m77AYNq6s+bjVnGTYxUwLc9GtjKraaDMO/fgr6EvlUD0EPhAmn+7E IPj+IwNTcbdBAt/se75aHNeIcn0OLL2WTHHRIqiNCnjNwAGgIn9V3swV5I0o0dJOiEdP/f7/ v5wkmqMxhk/dQW3UCLwLOIeZEXpl5Y/CKm8Toe0sAr+voAWmBUl3fqIywtOf9NLI7M4bTxr/ SM5VfyEuBhlbJbDZjXhGoQAa96hQApKqdrKQs62JwMBlg7JwUb1xReHUsTSQtZW+Uzb0Ou0r Hh4rWGHZYOUDZrvnJPjO5ip1kWg5x/ceJHeeZzZ36UkcMlqqNpOfy3cW4I3/IL1PWSP29zcw +nDVGwkcJdRKl3OVjaP0ADTDla6zo59swjHFUogE61j06zkFdrbFdMB7LT8Q5G9YoFx482WD /L2hQVPBtuA4s1KcZlsJIY8EExAa+FHe/K65Ag0EVu8w4AEQAKbLgubA/Deby6e/GvsGETvQ UfHRpyPUbylhXWKKWVgQj76UdHCmPEEw3OQtvSwsvF0HY3VvmA4EsWd+9wGqa+WW0soJAgrs 4Iv4n4XragL2qdye9tgDxQkJZjejsYg9XRywq1r54xAu6xCKqfsIzIis2BJ85jXdpsnXIiIB HaM0oGrCoXksDgeaQpSGqGr8vwqwtKq3xN9Mfj+AkoAsg3IPkkebxTeQAWofIRERzkTRtSHZ EBu5Zey36zr3ef9V65OsxDNPkO13NG6TME//WCp0Lv6VKqPWL/2wvPapz5TGLaVahrWRo9zF Hn6XydLlWes6s87zswQ0L9sKPk4OY9l8LS5lqmSpKRynv8pU3HvUsilPnOjlxF1y5vAqfqck Uze+CJsnO2E4s7q4dX5996ipOGkbALkArWqfv+IHQMU2+xUSfXzm7DjRIHYL6wxfQYTwdGgk AuGduWq8Lw0qRQqTbIs7/OdDFMfa9ks/v/VI8gFPae+v0fW7q941XqlhPDHFcY00PYZV/h+g KQ1lsyrm6/Sjs/Wzs+r4SuA2rpz2yrnN0JoKMuT6T/BkE4YNhieC8QOOBbO+tGbrFGVWFKS1 v0xi+aaWfYljJudPcEWCQWFWM1YcPta9dABkAuf3k9ZAbn9+i5ph2ulg23Lm8C6cqVh4Rs++ /sFYwe2UbbGzABEBAAGJAiUEGAEIAA8FAlbvMOACGwwFCQeGH4AACgkQAdrcj4PeJmUY7w/8 CKOOzaolESY8kacIliL919OpVSGJq3AfoxdOP3YqhMC4RtMwqKdF2ygrzr9YjwtAJDNMciIm 7EFYhbIWgl17WR9Dg41Kee2GrA3B89qyHpyL1Ke1vvoNgCKSeuuT/NPSLF/v8+rGAyjTD3y9 sfQ2gcVbTqOIlV67pBIJ6RQa6OXGlKMoAHtiXlcoOaSb41L1vQXRvdmMvX3OUJUXZmbhv5UH fqRE9hUI8pruA9EfCoftQz+3nXVyOSxyCc1jMxLwY6Aokbo/eESz9AWOoPl2wSz6Y/nnILSw kO0XTGzE6YRJ+EECoBxH4kapiQvqN2a5Mrp6qc5BD2bGfNs3hZSsbZjojqL9fd4qVZoKHNas LbWGWu0sin/0qeL1me0BFhF++W75wRK8PelvJMdTktWy04IxFYlRczUX+AEnRyGZITZTk3Fx ya3rvlWFXvCAS168vRRxx2WkkdD6TSnfp6+v+qQO41LV8DffyUoIOKJXoc/FzLRSa7A2HVW0 wWv052YgcvxSiCdDqP/oxAhu8czJzddpQy4DEF3ogGhh+VGldtzdGGHUzXdtOp88iqLZ8a4t fGA7wTpIDuQ5I7DIlJowf/s0eOCH/qarWxYGvAjn3Jd1YeLhwHhdXJpZgGcKmreuKSMoJZSD qwvPGScZ+o90/cZnGejc4RA7NRAfrQkk+ts= Cc: Christofer Dutz , Julian Feinauer Message-ID: <2342a31b-75e5-820d-33f3-70a395ad6d5a@code-house.org> Date: Thu, 9 May 2019 19:48:26 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US-large Content-Transfer-Encoding: 8bit Christian you might need to check what are the XSD substitution groups. I've seen a great use of these to leave "open doors" for document extension with limited use of new name spaces. Beside above I need to support Julian a bit since no one really knows how to handle it. Part of blame goes toward us as we didn't read DFDL spec yet. XML Schema is great tool and works quite reliably for situations where data is structured in certain way. This means that in most of cases XML Schema was introduced to create a contract for new services and its caller, in some other it was added as verification mechanism for existing XML payloads. All above are a bit in contrast to what PLC communication is - all of them already have communication contract in form of native protocols. These are distinct from XML and use byte payloads instead of text formats (mostly). I see a clear match between XSD/DFDL and these protcols - these are primitive and complex types plus payload definitions, but do we have any chance to use a validation capabilities of these XSDs at any point? Only if we will generate a parser. Keeping DFDL makes a clear point because it will be very helpful for all "enterpriseish" scenarios which require a more "XML" a like type definitions. I think that if we would have any traditional ESB programmers from big corps we would have their claps already. We know that PLC frames are mostly sequential, usually have length field somewhere in beginning, some might have CRC embedded at beginning or end. More advanced formats might have complex types, some in the end might be encrypted. I come over very few binary protocols during my career and PLCs are not first ones. I had a touch with EBus (do not confuse with EEBus), BACnet and WM-Bus. These represent completely different ways to communicate with devices where first one is typical serial interface, second is radio with rich application layer and possible AES encryption and third have rich, abstract and extensible structure. I can't talk much about S7, ADS or KNX/IP simply because I don't know these protocols yet. However based on my BACnet knowledge I can already tell you that reflecting its will be difficult without leaving a plug at parser and application layer. Doing that at XML Schema will be supper difficult as we would have a "main" protocol and later "something" to plug vendor specific extensions (if any). Extensibility of BACnet starts at frame types and ends at enumerations used to describe elements. On other hand WM-Bus with its simplicity bring few points. Frame is structured with control byte, header, crc, payload and crc again. Payload can be turned into series of variable length elements where first bytes and their bits determine kind of data, multiplier, unit and so on. I can't see myself doing bit operations in plain Java and I can't see this readable in XML Schema/DFDL. To add more context of WM-Bus, as it is radio based, payload can be encoded with AES thus there is extra lookup from parser/receiver to obtain device key which will allow to decode data. BACnet supports encryption too, however according to my knowledge it is very rarely used in field due to complicated nature of this part of standard. One of unique abilities of BACnet is support for segmentation of payloads, meaning that longer answers will go with several frames (AFAIR up to 250 or 255). This represents yet another problem which needs to be addressed at parser/generator level to do not build application layer structures until data is fully collected. To make things even more complicated BACnet have several layers: - virtual link - network layer - application layer 2016 version of BACnet ASN.1 with comments, but not all enumerations is 4600 lines long. I'm writing all this to add additional edge cases which we will need to cover - with DFDL or any other tool. It won't be easy neither way! Coming to the point, I am not sure if DFDL is the best way of describing payload and application layer formats. As you already committed a lot of time and will in it you know it best and feel natural. However do we want each and everyone who would like to contribute to PLC4X to go over 244 pages long [1] PDF and check some of more than 30 external references, just to write their protocol description? For me this represents a quite step to go over especially that it seems to be a mixture of XML Schema, XPath, XQuery and/or schematron. None or very few folks who works with low level stuff will be able to get over it. I know most of above, and I would say that I am fluent with XML Schema as I did a lot of gymnastics with WSDL and SOAP, I've written plenty of transformations with XSLT. It always worked well because these fit perfectly into XML universe. I'm not saying that DFDL does not fit into what PLC4X is trying to do, but PLCs don't do any XML processing. Commonalities I mentioned above (types, payloads) do not speak loud enough to invest into such thing. I wouldn't mind to have a simplistic way to describe frames with all possible edge cases we know and generate DFDL out of it. With this in mind we could retain work which has been done already to generate parsers (reuse what Apache Daffodil gives us) but also limit amount of work for new protocols and reduce verbosity of first step. Once someone will feel comfortable enough to get into DFDL - we still have this option. After all big and fast adoption of protobufs, thrift and other formats started from very simplistic descriptor format. Cheers, Łukasz -- http://connectorio.com [1] https://www.ogf.org/documents/GFD.207.pdf On 08.05.2019 15:20, Julian Feinauer wrote: > Hi Chris, > > sorry, for just responding now. > I guess you are the only one that deeply involved in the DFDL Schema and stuff. > I only scratched at the surface and guess it is incredibly powerful. > But as everybody knows... with great power complex great complexity. > > So, on one hand, your suggestion sounds reasonable, to have some kind of "base" definition. > But on the other hand I mislike the "central" nature of that. > I meant, there are some standards which will be used by all drivers like units, ints, C-Strings and so with LE/BE but other than that I'm pretty sure (but only a feeling, nothing I could prove) that every protocol we touch brings something we did not think about yet. > > So I personally would prefer if we do these types directly in the protocol, and perhaps simply with a clear description how to deserialize or so (byte shifts, bit shifts, ...). > Otherwise we have some kind of coupling behind all these drivers through the backdoor and in the worst case we would make one driver fall silently, because we fix some kind of bug with the types in another one or something. > > Don’t we have some possibilities to do something like that? > > Julian > > Am 08.05.19, 15:12 schrieb "Christofer Dutz" : > > Hi all, > > I also just had another idea ... > > No matter how we define the schemas we'll always have one problem in the end ... how to map some type like an "unsigned-16-bit-integer" into something the language can understand. > So we were thinking of some Language adapters ... now this could handle the mapping to code, but we don't have control over how these types are defined in the protocol specifications. > Each protocol spec currently defines all the types it needs locally. > > Now I had an idea that might help solve both problems: > - I create a "plc4x-dfdl" schema which contains definitions for all of the base types > - We use and import this schema into dfdl protocol specs to have the same base-line in all plc4x protocol specs > - When we write new language packs, we do so by providing implementations for all of the types in the plc4x-dfdl schema > > Guess this should be a pretty clean definition of what plc4x provides, what protocol engineers need to define in their drivers and what language engineers need to provide in their language templates. > > Chris > > > > Am 08.05.19, 11:29 schrieb "Christofer Dutz" : > > Hi, > > I think while refactoring the DFDL schemas a little more, I came up with an idea on how we can support inheritance with DFDL: > > > * In all cases with inheritance, we have a “choice” element in the schema > * Some sort of “type” element is parsed before the choice element itself > > Now the idea is that if a type contains a choice, that the name of the base class of all sub-types is based on the name of the element that contains the choice. > > Example: > > > > > > > > > dfdl:lengthKind="explicit" dfdl:lengthUnits="bytes" dfdl:length="{../parametersLength}" > dfdl:occursCountKind="expression" > dfdl:occursCount="{if(../parametersLength gt 0) then 1 else 0}"> > > > > > > > > type="s7:S7GeneralParameterSetupCommunication"/> > type="s7:S7RequestParameterReadVar"/> > type="s7:S7RequestParameterWriteVar"/> > > > > > > > > > In this case we would have an S7RequestMessage type which contains a property “parameters” of type “List”. > Parameter (containing a choice) would be an abstract class with an abstract “getDenominator” method. > S7GeneralParameterSetupCommunication would extend Parameter. > > You think that’s a path to go? … Had to add some artificial elements in order to set the boundaries of the types. > > Chris > > > >