From user-return-1534-archive-asf-public=cust-asf.ponee.io@kudu.apache.org Sat Nov 17 22:47:03 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 1F3FF180658 for ; Sat, 17 Nov 2018 22:47:01 +0100 (CET) Received: (qmail 6745 invoked by uid 500); 17 Nov 2018 21:47:01 -0000 Mailing-List: contact user-help@kudu.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@kudu.apache.org Delivered-To: mailing list user@kudu.apache.org Received: (qmail 6735 invoked by uid 99); 17 Nov 2018 21:47:00 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 17 Nov 2018 21:47:00 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 0960FC030E for ; Sat, 17 Nov 2018 21:47:00 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.099 X-Spam-Level: * X-Spam-Status: No, score=1.099 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (1024-bit key) header.d=boristyukin.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id c4XNMPdtZ_Bm for ; Sat, 17 Nov 2018 21:46:57 +0000 (UTC) Received: from mx26-out30.antispamcloud.com (mx26-out30.antispamcloud.com [148.251.71.30]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id F1CDC5F60D for ; Sat, 17 Nov 2018 21:46:56 +0000 (UTC) Received: from s2.fcomet.com ([99.198.101.250]) by mx105.antispamcloud.com with esmtps (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.89) (envelope-from ) id 1gO8QQ-0009qc-8m for user@kudu.apache.org; Sat, 17 Nov 2018 22:46:48 +0100 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=boristyukin.com; s=default; h=Content-Type:To:Subject:Message-ID:Date:From: In-Reply-To:References:MIME-Version:Sender:Reply-To:Cc: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=GZDYs0zJJU71lxT3bb9OP4RR/uzqsKhQ1A/FvbLnm3o=; b=qzohumzq9bztrX6VtupUNV2Db 8Uqv+Iwzz0FDZ1aleBFM+F4H1jiNuyMikxeuWW4qTdZ54X0FeXZSVPOUDO1t//6X5WIu9YjohE+Qc qFALrjKWNArqNgCfyopzdAiqPSwjmbmuE6tMYGvmJH4VPvrWq/swSxZRsPbQ7Ga8s9Rv4=; Received: from mail-it1-f181.google.com ([209.85.166.181]:54043) by s2.fcomet.com with esmtpsa (TLSv1.2:ECDHE-RSA-AES128-GCM-SHA256:128) (Exim 4.91) (envelope-from ) id 1gO8Pj-0054K3-9x for user@kudu.apache.org; Sat, 17 Nov 2018 15:46:03 -0600 Received: by mail-it1-f181.google.com with SMTP id g85so1443299ita.3 for ; Sat, 17 Nov 2018 13:46:04 -0800 (PST) X-Gm-Message-State: AGRZ1gJyjyiv9DVDlwA76XzOC4dI9/SwYAKmx7AjYEKHZPkz+bb2AysS DMz+yTHour4kbTDPVvRTxQIKXEx3vh0uoGH5b6g= X-Google-Smtp-Source: AFSGD/UI5R5k708W3gSdRIVoqeTrEnoHd/Q+6aTsuKHLJfjGk0JXPPNqRVzIRjE2XlFI24aiUvAkBzufzcR4pof0hjo= X-Received: by 2002:a05:660c:1d1:: with SMTP id s17mr3328856itk.79.1542491163785; Sat, 17 Nov 2018 13:46:03 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Boris Tyukin Date: Sat, 17 Nov 2018 16:45:52 -0500 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: strange behavior of getPendingErrors To: user@kudu.apache.org Content-Type: multipart/alternative; boundary="000000000000e30400057ae33557" X-AuthUser: boris@boristyukin.com X-Originating-IP: 99.198.101.250 X-Spampanel-Domain: s2.fcomet.com X-Spampanel-Username: 99.198.101.250 Authentication-Results: antispamcloud.com; auth=pass smtp.auth=99.198.101.250@s2.fcomet.com X-Spampanel-Outgoing-Class: unsure X-Spampanel-Outgoing-Evidence: Combined (0.18) X-Recommended-Action: accept X-Filter-ID: EX5BVjFpneJeBchSMxfU5j1hWQ7V/d4eoeX/L8dbNPh602E9L7XzfQH6nu9C/Fh9KJzpNe6xgvOx q3u0UDjvO2HFEYQDlNqPthodLGs7Ym7sT+bYMn6GaM87VISxooGybqoTJxc+StOEliYmzGDryFpr jQPFk8m4tSTfORUp3ynEm+h0A2koB3qKN5bbUQlC/r1JcqkdPiI1TwYjLceCFZFVgpT1b21uZVck Gp0ccOYNj3IsPxUoOvqBoVWc32LibjGdSRNeAsqwADJIlioxB+Ri1Gwjhmdwj/RC7BTJQIETmEFB zGJ4I3iI+cUBLpxHZqMsFXHkY4b0tMjYHlbEsQ6yCIj+j+sW7DnHSTh9wIB9qzbFFetd0V4Svjqc FPolF8VpymDAm51vSeaku+X2qYfWdHzoB2yW92yX6zvRr8FsMyewd83qhRHUN79OaH7QZM3NuHht 4ah3jKpVe9Q5dQwPf1ekvTsJu+noz87G5EthgB37cSRdX+S8eSGMVvuOehIqUczFWeS6sE8e1b5/ UuG84yBZdFDsZzjqLz0+lM04T8v64x/gEzewkDaAqz+5vO8Dp7WB9A+XqMtIj9D5XfMKBu8BGJTw bLkE/2AqTOXVHYdPf6H9bB9SMTbXeTyi/UrFmK5wYa/eXfUwTQnr3wRidxm0V+KLdcupr3oUEcbj EWTqls501mLJw8/+/dTbuP0iZyxhueMNLK8wwrVwuCuOyNB4uEc7/ajN+tIWCqoFalpSEh/nnzWV N0QODlNBIQii8UmQ22Qds9ISSFyQQMESgE2ZGg51fM4+j3QAvRErOkZQ0gdEYNW3+3t4PP++647l NwN4qOsSZg+fYhVZG8V5YtbNPlagNs3+OO4fHCXmcNMYrgEyPifyrwrvAF3mj8dzHQPLO1WC2ToG 4+imXZFVgpT1b21uZVckGp0ccOZ0u747HEjxBT3YXFq5jdQhizABe0DPoZtQcTLoONusdG+U8T3K fj7tHuHDvXqVSlU= X-Report-Abuse-To: spam@quarantine1.antispamcloud.com --000000000000e30400057ae33557 Content-Type: text/plain; charset="UTF-8" Great, thanks for the explanation and for your help. I think, for now, I am going to attempt to reinsert rows with errors second time but this time in auto_flush mode to identify actual bad rows. Hope that performance won't suffer too much. I enjoy performance of auto background sync (10-20 times faster in my simple test) but I cannot afford losing good data and it makes troubleshooting really challenging. Boris On Sat, Nov 17, 2018, 10:13 Alexey Serbin Hey Todd, > > Yes, that behavior is a bit strange especially given the fact that the > behavior differs when it comes to duplicate rows and other errors that > happen at later stages of 'applying' write operations at the server side. > > I'll open a JIRA item about this issue. If anyone disagrees, we can > resolve the JIRA item as needed. > > > Thanks, > > Alexey > > On Sat, Nov 17, 2018 at 12:01 AM Todd Lipcon wrote: > >> Hey Alexey, >> >> I think your explanation makes sense from an implementation perspective. >> But, I think we should treat this behavior as a bug. From the user >> perspective, such an error is a per-row data issue and should only affect >> the row with the problem, not some arbitrary subset of rows in the batch >> which happened to share a partition. >> >> Does anyone disagree? >> >> Todd >> >> On Fri, Nov 16, 2018, 9:28 PM Alexey Serbin > >>> Hi Boris, >>> >>> Kudu clients (both Java and C++ ones) send write operations to >>> corresponding tablet servers in batches when using the AUTO_FLUSH_BACKGROUND >>> and MANUAL_FLUSH modes. When a tablet server receives a Write RPC >>> (WriteRequestPB is the corresponding type of the parameter), it decodes the >>> operations from the batch: >>> https://github.com/apache/kudu/blob/master/src/kudu/tablet/local_tablet_writer.h#L97 >>> >>> While decoding operations from a batch, various constraints are being >>> checked. One of those is checking for nulls in non-nullable columns. If >>> there is a row in the batch that violates the non-nullable constraint, the >>> whole batch is rejected. >>> >>> That's exactly what happened in your example: a batch to one tablet >>> consisted of 3 rows one of which had a row with violation of the >>> non-nullable constraint for the dt_tm column, so the whole batch of 3 >>> operations was rejected. You can play with different partition schemes: >>> e.g., in case of 10 hashed partitions it might happen that only 2 >>> operations would be rejected, in case of 30 partitions -- just the single >>> key==2 row could be rejected. >>> >>> BTW, that might also happen if using the MANUAL_FLUSH mode. However, >>> with the AUTO_FLUSH_SYNC mode, the client sends operations in batches of >>> size 1. >>> >>> >>> Kind regards, >>> >>> Alexey >>> >>> On Fri, Nov 16, 2018 at 7:24 PM Boris Tyukin >>> wrote: >>> >>>> Hi Todd, >>>> >>>> We are on Kudu 1.5 still and I used Kudu client 1.7 >>>> >>>> Thanks, >>>> Boris >>>> >>>> On Fri, Nov 16, 2018, 17:07 Todd Lipcon >>> >>>>> Hi Boris, >>>>> >>>>> This is interesting. Just so we're looking at the same code, what >>>>> version of the kudu-client dependency have you specified, and what version >>>>> of the server? >>>>> >>>>> -Todd >>>>> >>>>> On Fri, Nov 16, 2018 at 1:12 PM Boris Tyukin >>>>> wrote: >>>>> >>>>>> Hey guys, >>>>>> >>>>>> I am playing with Kudu Java client (wow it is fast), using mostly >>>>>> code from Kudu Java example. >>>>>> >>>>>> While learning about exceptions during rows inserts, I stumbled upon >>>>>> something I could not explain. >>>>>> >>>>>> If I insert 10 rows into a brand new Kudu table >>>>>> (AUTO_FLUSH_BACKGROUND mode) and I make one row to be "bad" intentionally >>>>>> (one column cannot be NULL), I actually get 3 rows that cannot be inserted >>>>>> into Kudu, not 1 as I was expected. >>>>>> >>>>>> But if I do session.flush() after every single insert, I get only one >>>>>> error row (but this ruins the purpose of AUTO_FLUSH_BACKGROUND mode). >>>>>> >>>>>> Any ideas one? We cannot afford losing data and need to track all >>>>>> rows which cannot be inserted. >>>>>> >>>>>> AUTO_FLUSH mode works much better and I do not have an issue like >>>>>> above, but then it is way slower than AUTO_FLUSH_BACKGROUND. >>>>>> >>>>>> My code is below. It is in Groovy, but I think you will get an idea :) >>>>>> https://gist.github.com/boristyukin/8703d2c6ec55d6787843aa133920bf01 >>>>>> >>>>>> Here is output from my test code that hopefully illustrates my >>>>>> confusion - out of 10 rows inserted, 9 should be good and 1 bad, but it >>>>>> turns out Kudu flagged 3 as bad: >>>>>> >>>>>> Created table kudu_groovy_example >>>>>> Inserting 10 rows in AUTO_FLUSH_BACKGROUND flush mode ... >>>>>> (int32 key=1, string value="value 1", unixtime_micros >>>>>> dt_tm=2018-11-16T20:57:03.469000Z) >>>>>> (int32 key=2, string value=NULL) BAD ROW >>>>>> (int32 key=3, string value="value 3", unixtime_micros >>>>>> dt_tm=2018-11-16T20:57:03.595000Z) >>>>>> (int32 key=4, string value=NULL, unixtime_micros >>>>>> dt_tm=2018-11-16T20:57:03.596000Z) >>>>>> (int32 key=5, string value="value 5", unixtime_micros >>>>>> dt_tm=2018-11-16T20:57:03.597000Z) >>>>>> (int32 key=6, string value=NULL, unixtime_micros >>>>>> dt_tm=2018-11-16T20:57:03.597000Z) >>>>>> (int32 key=7, string value="value 7", unixtime_micros >>>>>> dt_tm=2018-11-16T20:57:03.598000Z) >>>>>> (int32 key=8, string value=NULL, unixtime_micros >>>>>> dt_tm=2018-11-16T20:57:03.602000Z) >>>>>> (int32 key=9, string value="value 9", unixtime_micros >>>>>> dt_tm=2018-11-16T20:57:03.603000Z) >>>>>> (int32 key=10, string value=NULL, unixtime_micros >>>>>> dt_tm=2018-11-16T20:57:03.603000Z) >>>>>> 3 errors inserting rows - why 3???? only 1 expected to be bad... >>>>>> there were errors inserting rows to Kudu >>>>>> the first few errors follow: >>>>>> ??? key 1 and 6 supposed to be fine! >>>>>> Row error for primary key=[-128, 0, 0, 1], tablet=null, server=null, >>>>>> status=Invalid argument: No value provided for required column: >>>>>> dt_tm[unixtime_micros NOT NULL] (error 0) >>>>>> Row error for primary key=[-128, 0, 0, 2], tablet=null, server=null, >>>>>> status=Invalid argument: No value provided for required column: >>>>>> dt_tm[unixtime_micros NOT NULL] (error 0) >>>>>> Row error for primary key=[-128, 0, 0, 6], tablet=null, server=null, >>>>>> status=Invalid argument: No value provided for required column: >>>>>> dt_tm[unixtime_micros NOT NULL] (error 0) >>>>>> Rows counted in 485 ms >>>>>> Table has 7 rows - ??? supposed to be 9! >>>>>> INT32 key=4, STRING value=NULL, UNIXTIME_MICROS >>>>>> dt_tm=2018-11-16T20:57:03.596000Z >>>>>> INT32 key=8, STRING value=NULL, UNIXTIME_MICROS >>>>>> dt_tm=2018-11-16T20:57:03.602000Z >>>>>> INT32 key=9, STRING value=value 9, UNIXTIME_MICROS >>>>>> dt_tm=2018-11-16T20:57:03.603000Z >>>>>> INT32 key=3, STRING value=value 3, UNIXTIME_MICROS >>>>>> dt_tm=2018-11-16T20:57:03.595000Z >>>>>> INT32 key=10, STRING value=NULL, UNIXTIME_MICROS >>>>>> dt_tm=2018-11-16T20:57:03.603000Z >>>>>> INT32 key=5, STRING value=value 5, UNIXTIME_MICROS >>>>>> dt_tm=2018-11-16T20:57:03.597000Z >>>>>> INT32 key=7, STRING value=value 7, UNIXTIME_MICROS >>>>>> dt_tm=2018-11-16T20:57:03.598000Z >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> -- >>>>> Todd Lipcon >>>>> Software Engineer, Cloudera >>>>> >>>> --000000000000e30400057ae33557 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Great, thanks for the explanation and for your help. I th= ink, for now, I am going to attempt to reinsert rows with errors second tim= e but this time in auto_flush mode to identify actual bad rows. Hope that p= erformance won't suffer too much. I enjoy performance of auto backgroun= d sync (10-20 times faster in my simple test) but I cannot afford losing go= od data and it makes troubleshooting really challenging.
<= br>
Boris



On Sat, Nov 17, 2018, 10:13 Alexey Serbin <aserbin@cloudera.com = wrote:
Hey Todd,

Yes= , that behavior is a bit strange especially given the fact that the behavio= r differs when it comes to duplicate rows and other errors that happen at l= ater stages of 'applying' write operations at the server side.

I'll open a JIRA item = about this issue.=C2=A0 If anyone disagrees, we can resolve the JIRA item a= s needed.


Thanks,

Alexey

On Sat= , Nov 17, 2018 at 12:01 AM Todd Lipcon <todd@cloudera.com&= gt; wrote:
Hey Al= exey,

I think your explanation= makes sense from an implementation perspective. But, I think we should tre= at this behavior as a bug. From the user perspective, such an error is a pe= r-row data issue and should only affect the row with the problem, not some = arbitrary subset of rows in the batch which happened to share a partition.<= /div>

Does anyone disagree?

Todd

On Fri, Nov 16, 2018, 9:28 PM Alexey = Serbin <aserbin@cloudera.com wrote:
H= i Boris,

Kudu clients (both Java and C++ on= es) send write operations to corresponding tablet servers in batches when u= sing the=C2=A0AUTO_FLUSH_BACKGROUND and MANUAL_FLUSH modes.=C2=A0 When a= tablet server receives a Write RPC (WriteRequestPB is the corresponding ty= pe of the parameter), it decodes the operations from the batch:=C2=A0https://github.com/apache/kudu/blob/master/src/kudu/tablet/local_tab= let_writer.h#L97

While decoding operations from a batch, various constrai= nts are being checked.=C2=A0 One of those is checking for nulls in non-null= able columns.=C2=A0 If there is a row in the batch that violates the non-nu= llable constraint, the whole batch is rejected.

That's exactly what happe= ned in your example: a batch to one tablet consisted of 3 rows one of which= had a row with violation of the non-nullable constraint for the dt_tm colu= mn, so the whole batch of 3 operations was rejected.=C2=A0 You can play wit= h different partition schemes: e.g., in case of 10 hashed partitions it mig= ht happen that only 2 operations would be rejected, in case of 30 partition= s -- just the single key=3D=3D2 row could be rejected.

BTW, that might also h= appen if using the MANUAL_FLUSH mode.=C2=A0 However, with the AUTO_FLUSH_SY= NC mode, the client sends operations in batches of size 1.


Kind regards,
Alexey

On Fri, Nov 16, 2018 at 7:24 PM Boris Tyu= kin <boris@boristyukin.com> wrote:
Hi Todd,

We are on Kudu 1.5 still and I used Kudu cl= ient 1.7

Thanks,
Boris

On Fri, Nov 16, 2018, 17:07 Todd Lipcon <todd= @cloudera.com wrote:
Hi Boris,

This is interesting. Just so we'r= e looking at the same code, what version of the kudu-client dependency have= you specified, and what version of the server?

-T= odd

On Fri, Nov = 16, 2018 at 1:12 PM Boris Tyukin <bo= ris@boristyukin.com> wrote:
=
Hey guys,

I am playing with Kudu Java client (= wow it is fast), using mostly code from Kudu Java example.=C2=A0
=
While learning about exceptions during rows inserts, I stumb= led upon something I could not explain.

If I inser= t 10 rows into a brand new Kudu table (AUTO_FLUSH_BACKGROUND mode) and I ma= ke one row to be "bad" intentionally (one column cannot be NULL),= I actually get 3 rows that cannot be inserted into Kudu, not 1 as I was ex= pected.=C2=A0

But if I do session.flush() after ev= ery single insert, I get only one error row (but this ruins the purpose of= =C2=A0AUTO_FLUSH_BACKGROUND mode).

Any ideas one? = We cannot afford losing data and need to track all rows which cannot be ins= erted.

AUTO_FLUSH mode works much better and I do = not have an issue like above, but then it is way slower than AUTO_FLUSH_BAC= KGROUND.

My code is below. It is in Groovy, but I = think you will get an idea :)

Here is= output from my test code that hopefully illustrates my confusion - out of = 10 rows inserted, 9 should be good and 1 bad, but it turns out Kudu flagged= 3 as bad:

Created table kudu_groovy_example<= /div>
Inserting 10 rows= in AUTO_FLUSH_BACKGROUND flush mode ...
(int32 key=3D1, s= tring value=3D"value 1", unixtime_micros dt_tm=3D2018-11-16T20:57= :03.469000Z)
(i= nt32 key=3D2, string value=3DNULL)=C2=A0 BAD ROW
(int32 ke= y=3D3, string value=3D"value 3", unixtime_micros dt_tm=3D2018-11-= 16T20:57:03.595000Z)
(int32 key=3D4, string value=3DNULL, unixtim= e_micros dt_tm=3D2018-11-16T20:57:03.596000Z)
(int32 key=3D5, str= ing value=3D"value 5", unixtime_micros dt_tm=3D2018-11-16T20:57:0= 3.597000Z)
(int32 key=3D6, string value=3DNULL, unixtime_micros d= t_tm=3D2018-11-16T20:57:03.597000Z)
(int32 key=3D7, string value= =3D"value 7", unixtime_micros dt_tm=3D2018-11-16T20:57:03.598000Z= )
(int32 key=3D8, string value=3DNULL, unixtime_micros dt_tm=3D20= 18-11-16T20:57:03.602000Z)
(int32 key=3D9, string value=3D"v= alue 9", unixtime_micros dt_tm=3D2018-11-16T20:57:03.603000Z)
(int32 key=3D10, string value=3DNULL, unixtime_micros dt_tm=3D2018-11-16T= 20:57:03.603000Z)
3 errors inserting rows - why 3???? only 1 expected to be bad...
there were errors inserting rows to Kudu
the first few er= rors follow:
??? = key 1 and 6 supposed to be fine!
Row error for primary key=3D[-128, 0, 0, 1], tab= let=3Dnull, server=3Dnull, status=3DInvalid argument: No value provided for= required column: dt_tm[unixtime_micros NOT NULL] (error 0)
Row error for primary key= =3D[-128, 0, 0, 2], tablet=3Dnull, server=3Dnull, status=3DInvalid argument= : No value provided for required column: dt_tm[unixtime_micros NOT NULL] (e= rror 0)
Ro= w error for primary key=3D[-128, 0, 0, 6], tablet=3Dnull, server=3Dnull, st= atus=3DInvalid argument: No value provided for required column: dt_tm[unixt= ime_micros NOT NULL] (error 0)
Rows counted in 485 ms
Table has 7 rows - ??= ? supposed to be 9!
INT32 key=3D4, STRING value=3DNULL, UN= IXTIME_MICROS dt_tm=3D2018-11-16T20:57:03.596000Z
INT32 key=3D8, = STRING value=3DNULL, UNIXTIME_MICROS dt_tm=3D2018-11-16T20:57:03.602000Z
INT32 key=3D9, STRING value=3Dvalue 9, UNIXTIME_MICROS dt_tm=3D2018= -11-16T20:57:03.603000Z
INT32 key=3D3, STRING value=3Dvalue 3, UN= IXTIME_MICROS dt_tm=3D2018-11-16T20:57:03.595000Z
INT32 key=3D10,= STRING value=3DNULL, UNIXTIME_MICROS dt_tm=3D2018-11-16T20:57:03.603000Z
INT32 key=3D5, STRING value=3Dvalue 5, UNIXTIME_MICROS dt_tm=3D201= 8-11-16T20:57:03.597000Z
INT32 key=3D7, STRING value=3Dvalue 7, U= NIXTIME_MICROS dt_tm=3D2018-11-16T20:57:03.598000Z






--
Todd Lipcon
Software Engineer, C= loudera
--000000000000e30400057ae33557--