ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Igor Sapego <isap...@apache.org>
Subject Re: Thin clients all in one
Date Tue, 18 Dec 2018 09:59:11 GMT
I've created documentation ticket for UUID type - [1].

Now about Timestamp type. Binary protocol specification only specifies
how the data type should be transmitted by the network. The purpose and
main requirements for the protocol is to be compact and fast, it is not
designed to be used by user in raw.

Thus binary protocol does not define nor mandates any type representation
for the specific thin client or its API as it is implied that different
languages
may have different practicies and paradigms.

So binary protocol is not something we should refer to when we are talking
about user API. Protocol should be compact and fast, user API should be
convenient.

[1] - https://issues.apache.org/jira/browse/IGNITE-10721.

Best Regards,
Igor


On Mon, Dec 17, 2018 at 6:18 PM Alexey Kosenchuk <
alexey.kosenchuk@nobitlost.com> wrote:

> Hi Stepan,
>
> 1) About UUID...
>
> UUID binary encoding must be defined in the Binary Protocol
> specification
> (
> https://apacheignite.readme.io/docs/binary-client-protocol-data-format#section-uuid-guid-
> )
> So, it's the issue against the spec first.
>
> Regarding the clients..
>
> Python client uses Python UUID class
> (https://docs.python.org/2/library/uuid.html) which, if I understand
> correctly, can operate with different encodings. Dmitry M. can correct me.
>
> NodeJs and PHP do not have standard UUID classes.
> So, the clients return/accept UUID as just a 16-byte array with binary
> encoding inside, exactly as it is read / will be written via the binary
> protocol.
> An application may convert this array to/from a required representation.
>
> 2) About Timestamp...
>
> Python, NodeJs and PHP clients behave strictly according to the Binary
> Protocol specification
> (
> https://apacheignite.readme.io/docs/binary-client-protocol-data-format#section-timestamp)
>
> which defines that the second field is "a nanoseconds fraction of a last
> millisecond, which value could be in a range from 0 to 999999."
>
> Regards,
> -Alexey
>
> 14.12.2018 17:41, Stepan Pilschikov пишет:
> > Hello again
> >
> > Starting to check compatibility between thin clients in java/c++ and
> > py/php/nodejs and met some problems
> >
> > 1) Found that UUID mixed in a strange way if we taken it from Java/C++
> > to PY/PHP/JS client and backwards
> > Have issue about it https://issues.apache.org/jira/browse/IGNITE-10691,
> > please look at it
> >
> > 2) Is more like conversation about seamless experience between all thin
> > clients
> > PY/PHP/JS/C++/Java have Timestamp data type which is contain
> > tuple(millis, nanos)
> > Main concern about nanos
> >
> > In PHP/JS/Py Nanos count like "nanoseconds of the last millisecond" but
> > in Java and C++ its like we have millis and same millis but with nanos
> > and its strange for PHP/PY/JS if we take Java how is Base thin client
> >
> > For example in Java we have new Timestamp(111111) and it
> > (fastTime=111000, nanos=111000000)
> > In C++ if we getting this cache we take (millis=111111, nanos=111000000)
> > which is kind a right and same as in java
> > But in py/php/js its look like (millis=111111, nanos=0), its kind a
> > understandable what logic is, but sill different behavior and experience
> > What you thinking about it?
> > For now i can't understand why its done how its done, looks like it
> > should be same but something going wrong
> >
> > пт, 30 нояб. 2018 г. в 01:18, Alexey Kosenchuk
> > <alexey.kosenchuk@nobitlost.com <mailto:alexey.kosenchuk@nobitlost.com
> >>:
> >
> >     Hi Stepan,
> >
> >     pls check the Ignite cfg you use - see the comments in the jira.
> >
> >     Also, the examples executors (including AuthTlsExample) are included
> >     into NodeJS test suite in TeamCity which run periodically and
> >     successfully passed. Eg. the latest one:
> >
> https://ci.ignite.apache.org/viewLog.html?buildId=2426645&tab=buildResultsDiv&buildTypeId=IgniteTests24Java8_ThinClientNodeJs
> >
> >     Regards,
> >     -Alexey
> >
> >     28.11.2018 17:08, Stepan Pilschikov пишет:
> >      > Hello again
> >      >
> >      > If NodeJS sources found that example AuthTlsExample.js throwing
> >     exception
> >      > during execution
> >      > Output and grid configuration in
> >      > https://issues.apache.org/jira/browse/IGNITE-10447
> >      >
> >      > Can someone have a look at it?
> >      >
> >      > вс, 25 нояб. 2018 г. в 19:11, Stepan Pilschikov
> >     <pilshchikov.s.n@gmail.com <mailto:pilshchikov.s.n@gmail.com>>:
> >      >
> >      >> My bad,
> >      >> You right
> >      >>
> >      >> вс, 25 нояб. 2018 г. в 05:37, Dmitry Melnichuk <
> >      >> dmitry.melnichuk@nobitlost.com
> >     <mailto:dmitry.melnichuk@nobitlost.com>>:
> >      >>
> >      >>> Stepan,
> >      >>>
> >      >>> AFAIK Map type did always behave correctly on client side, as
> >     it does
> >      >>> now. This is a corresponding piece of my test suite:
> >      >>>
> >      >>> ```
> >      >>> def test_put_get_map(client):
> >      >>>
> >      >>>       cache = client.get_or_create_cache('test_map_cache')
> >      >>>
> >      >>>       cache.put(
> >      >>>           'test_map',
> >      >>>           (
> >      >>>               MapObject.HASH_MAP,
> >      >>>               {
> >      >>>                   (123, IntObject): 'test_data',
> >      >>>                   456: ((1, [456, 'inner_test_string', 789]),
> >      >>> CollectionObject),
> >      >>>                   'test_key': 32.4,
> >      >>>               }
> >      >>>           ),
> >      >>>           value_hint=MapObject
> >      >>>       )
> >      >>>       value = cache.get('test_map')
> >      >>>       assert value == (MapObject.HASH_MAP, {
> >      >>>           123: 'test_data',
> >      >>>           456: (1, [456, 'inner_test_string', 789]),
> >      >>>           'test_key': 32.4,
> >      >>>       })
> >      >>>
> >      >>> ```
> >      >>>
> >      >>> Or is there another, more specific problem with maps?
> >      >>>
> >      >>> Dmitry
> >      >>>
> >      >>> On 11/25/18 3:56 AM, Stepan Pilschikov wrote:
> >      >>>> Dmitry,
> >      >>>>
> >      >>>> Great, checked, now all things woks well
> >      >>>> Hope that Igor made review for this PR
> >      >>>>
> >      >>>> But what about Maps? Looks like different ticket? or it can
be
> >     done in
> >      >>> same
> >      >>>> ticket scope?
> >      >>>>
> >      >>>> пт, 23 нояб. 2018 г. в 23:58, Dmitry Melnichuk <
> >      >>>> dmitry.melnichuk@nobitlost.com
> >     <mailto:dmitry.melnichuk@nobitlost.com>>:
> >      >>>>
> >      >>>>> Stepan,
> >      >>>>>
> >      >>>>> Sorry, I forgot to update from upstream prior to start
> >     working on this
> >      >>>>> issue, and thus brought a regression. My bad. Just merged
> >     with the
> >      >>>>> latest master. Please, check it out again.
> >      >>>>>
> >      >>>>> Dmitry
> >      >>>>>
> >      >>>>> On 11/24/18 1:37 AM, Stepan Pilschikov wrote:
> >      >>>>>> Dmitry,
> >      >>>>>>
> >      >>>>>> Iv checked and its actually work
> >      >>>>>> But a specially in this branch i found another bug
> >      >>>>>> Please look at my last comment:
> >      >>>>>>
> >      >>>>>
> >      >>>
> >
> https://issues.apache.org/jira/browse/IGNITE-10358?focusedCommentId=16697285&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16697285
> >      >>>>>>
> >      >>>>>> пт, 23 нояб. 2018 г. в 01:21, Dmitry Melnichuk
<
> >      >>>>>> dmitry.melnichuk@nobitlost.com
> >     <mailto:dmitry.melnichuk@nobitlost.com>>:
> >      >>>>>>
> >      >>>>>>> Stepan,
> >      >>>>>>>
> >      >>>>>>> Thank you for your great job in evaluating Python
thin
> >     client, as
> >      >>> well
> >      >>>>>>> as other thin clients.
> >      >>>>>>>
> >      >>>>>>> There was indeed a bug in Python client regarding
the
> >     handling of
> >      >>> type
> >      >>>>>>> hints in Collection type. I created a fix and
did a PR under
> >      >>>>>>> IGNITE-10358 task, but the same PR is also fixes
the
> problem in
> >      >>>>>>> IGNITE-10230 task.
> >      >>>>>>>
> >      >>>>>>> As of handling the type mapping in gists you provided,
I
> left
> >      >>> comments
> >      >>>>>>> on both tasks.
> >      >>>>>>>
> >      >>>>>>> Dmitry
> >      >>>>>>>
> >      >>>>>>> On 11/21/18 6:37 PM, Stepan Pilschikov wrote:
> >      >>>>>>>> Dmitry, Alexey
> >      >>>>>>>>
> >      >>>>>>>> Thank you for help, this answers help me a
lot with
> >     understanding
> >      >>> how
> >      >>>>>>>> clients are work
> >      >>>>>>>>
> >      >>>>>>>> Not so long time ago i met problem which is
have expected
> >     behavior,
> >      >>> but
> >      >>>>>>> its
> >      >>>>>>>> may broke some workflows in future for some
users
> >      >>>>>>>>
> >      >>>>>>>> Its all about not specified data types in
collections and
> >     map's
> >      >>>>>>>> All description and examples in
> >      >>>>>>>> https://issues.apache.org/jira/browse/IGNITE-10358
> >      >>>>>>>>
> >      >>>>>>>> Dmitry, can you have a quick look at it and
maybe in
> >     future somehow
> >      >>> fix
> >      >>>>>>> it?
> >      >>>>>>>>
> >      >>>>>>>> пт, 26 окт. 2018 г. в 19:05, Dmitry
Melnichuk <
> >      >>>>>>>> dmitry.melnichuk@nobitlost.com
> >     <mailto:dmitry.melnichuk@nobitlost.com>>:
> >      >>>>>>>>
> >      >>>>>>>>> Stepan!
> >      >>>>>>>>>
> >      >>>>>>>>> TL/DR: what you got with Python client
in your gist is an
> >     intended
> >      >>>>>>>>> behavior.
> >      >>>>>>>>>
> >      >>>>>>>>> Explanation: As per docs, Object array
contains of type
> >     ID (which
> >      >>> is
> >      >>>>>>>>> defaults to -1) and an array of objects.
> >      >>>>>>>>>
> >      >>>>>>>>>
> >      >>>>>>>>>
> >      >>>>>>>
> >      >>>>>
> >      >>>
> >
> https://apacheignite.readme.io/docs/binary-client-protocol-data-format#section-object-array
> >      >>>>>>>>>
> >      >>>>>>>>> Your gist might be fixed accordingly:
> >      >>>>>>>>>
> >      >>>>>>>>> ```
> >      >>>>>>>>> from pyignite import Client
> >      >>>>>>>>> from pyignite.datatypes import *
> >      >>>>>>>>>
> >      >>>>>>>>> OBJECT_ARRAY_TYPE_ID = -1
> >      >>>>>>>>> OBJECT_ARRAY_CONTENTS = [1, 2]
> >      >>>>>>>>>
> >      >>>>>>>>> client = Client()
> >      >>>>>>>>> client.connect('127.0.0.1', 10800)
> >      >>>>>>>>> cache = client.get_or_create_cache("PY_OBJECT_ARRAY")
> >      >>>>>>>>> cache.put(
> >      >>>>>>>>>          1,
> >      >>>>>>>>>          (OBJECT_ARRAY_TYPE_ID, OBJECT_ARRAY_CONTENTS),
> >      >>>>>>>>>          key_hint=IntObject,
> >      >>>>>>>>>          value_hint=ObjectArrayObject,
> >      >>>>>>>>> )
> >      >>>>>>>>>
> >      >>>>>>>>> # Python  output: print(cache.get(1))
> >      >>>>>>>>> # (-1, [1, 2])
> >      >>>>>>>>> ```
> >      >>>>>>>>>
> >      >>>>>>>>> The situation is similar with Map and
Collection, they
> >     have types.
> >      >>>>> Types
> >      >>>>>>>>> and type IDs are mostly useless in Python,
but I left
> >     them for
> >      >>>>>>>>> interoperability reasons. If you think
I should kick them
> >     out, just
> >      >>>>> let
> >      >>>>>>>>> me know.
> >      >>>>>>>>>
> >      >>>>>>>>> The usage of these 3 data types is documented
and tested.
> >     You can
> >      >>>>> refer
> >      >>>>>>>>> to the table in “Data types” section:
> >      >>>>>>>>>
> >      >>>>>>>>>
> >      >>>>>>>>>
> >      >>>>>>>
> >      >>>>>
> >      >>>
> >
> https://apache-ignite-binary-protocol-client.readthedocs.io/en/latest/datatypes/parsers.html
> >      >>>>>>>>>
> >      >>>>>>>>> The tests are here:
> >      >>>>>>>>>
> >      >>>>>>>>>
> >      >>>>>>>>>
> >      >>>>>>>
> >      >>>>>
> >      >>>
> >
> https://github.com/apache/ignite/blob/master/modules/platforms/python/tests/test_datatypes.py#L116-L124
> >      >>>>>>>>>
> >      >>>>>>>>> On 10/26/18 11:57 PM, Stepan Pilschikov
wrote:
> >      >>>>>>>>>> Hi, everyone
> >      >>>>>>>>>>
> >      >>>>>>>>>> Create new thread to centralize cross
compatibility and
> >     others
> >      >>> common
> >      >>>>>>>>>> problems between thin clients
> >      >>>>>>>>>>
> >      >>>>>>>>>> Tying to use Object array to exchange
different data
> >     between JS,
> >      >>> PHP
> >      >>>>>>> and
> >      >>>>>>>>>> Python thin clients
> >      >>>>>>>>>>
> >      >>>>>>>>>> JS and PHP simply can't put any type
of arrays
> >      >>>>>>>>>> Python can put data, but if you take
it, data would be
> >     completely
> >      >>>>>>>>>> different, maybe during this put process
data is changed
> >      >>>>>>>>>>
> >      >>>>>>>>>> Code and output:
> >      >>>>>>>>>> PHP -
> >      >>>>>>>>>
> >      >>>
> >     https://gist.github.com/pilshchikov/e887d31d4f2f36923470fead14c7801a
> >      >>>>>>>>>> JS -
> >      >>>>>>>>>
> >      >>>
> >     https://gist.github.com/pilshchikov/ba49067fd8924ebdf4414ec63838b86d
> >      >>>>>>>>>> Python -
> >      >>>>>>>>>>
> >      >>>
> >     https://gist.github.com/pilshchikov/f4bbf76e31547e2dca7d4cc5d55bd24f
> >      >>>>>>>>>>
> >      >>>>>>>>>> I'm tried different data types (string,
double, float,
> >     complex
> >      >>>>> objects,
> >      >>>>>>>>>> just random objects, char, byte, Date),
still
> >      >>>>>>>>>>
> >      >>>>>>>>>> How, from my perspective, it should
works:
> >      >>>>>>>>>> put array of any type and then get
this array.
> >      >>>>>>>>>> Example: put [1,2,3] -> get [1,2,3]
or put [new Date(),
> new
> >      >>> Date()]
> >      >>>>> ->
> >      >>>>>>>>>> get [[Date object], [Date object]]
(like in java thin
> >     client)
> >      >>>>>>>>>>
> >      >>>>>>>>>> Looks like bug in all clients, what
you think?
> >      >>>>>>>>>>
> >      >>>>>>>>>> I wanted to try Collections, but i
think this type have
> same
> >      >>> problem
> >      >>>>>>>>>
> >      >>>>>>>>
> >      >>>>>>>
> >      >>>>>>>
> >      >>>>>>
> >      >>>>>
> >      >>>>>
> >      >>>>
> >      >>>
> >      >>>
> >      >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message