Return-Path: X-Original-To: apmail-hive-user-archive@www.apache.org Delivered-To: apmail-hive-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 6B36E17A88 for ; Thu, 3 Sep 2015 01:58:08 +0000 (UTC) Received: (qmail 72465 invoked by uid 500); 3 Sep 2015 01:58:06 -0000 Delivered-To: apmail-hive-user-archive@hive.apache.org Received: (qmail 72394 invoked by uid 500); 3 Sep 2015 01:58:06 -0000 Mailing-List: contact user-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hive.apache.org Delivered-To: mailing list user@hive.apache.org Received: (qmail 72384 invoked by uid 99); 3 Sep 2015 01:58:06 -0000 Received: from Unknown (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Sep 2015 01:58:06 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 42D611AB4EE for ; Thu, 3 Sep 2015 01:58:06 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.898 X-Spam-Level: ** X-Spam-Status: No, score=2.898 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=3, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-eu-west.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id zzNPcfDuUKo3 for ; Thu, 3 Sep 2015 01:58:04 +0000 (UTC) Received: from mail-ig0-f171.google.com (mail-ig0-f171.google.com [209.85.213.171]) by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with ESMTPS id 4A85C20DD8 for ; Thu, 3 Sep 2015 01:58:03 +0000 (UTC) Received: by igbni9 with SMTP id ni9so28847453igb.0 for ; Wed, 02 Sep 2015 18:58:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=BuF9v+M0pbvvxqAorXRLyBuc0aP4nyaa9vzZVy3vuSU=; b=VpSRsRWPbFf/HsoHttK3t1q28UxA4n1Cs4+3cyrELFUNnhtnq/H8IwsDgQGaHlUkQD crRoAt288fbz32e2yv3hzs4bYJ2EeE9w6Hpf69/wKEjw65P1HO0PKnU8ZCguDYq7V4Sh mAKWGZAAKQrzgfJK0zKfGdYuUa40PbF8agV3jFSTtWc4QvG8JJWy7sSIb4LsBpqS4hEU VKlbVWnWa81wh/rPuYDvD+J1lRpCUapNoT6OTSYfI7e1xtyy3TAzNpb3WmIYLgySvXLB e6ZgN0ibmEoHx3iIdMgNKhg1ef7BYuVT6omyYiF0HwsvXfFnykDD2kKBdMNw0hPnHPZD YCqw== MIME-Version: 1.0 X-Received: by 10.50.6.50 with SMTP id x18mr6906077igx.12.1441245482127; Wed, 02 Sep 2015 18:58:02 -0700 (PDT) Received: by 10.107.148.70 with HTTP; Wed, 2 Sep 2015 18:58:01 -0700 (PDT) Received: by 10.107.148.70 with HTTP; Wed, 2 Sep 2015 18:58:01 -0700 (PDT) In-Reply-To: References: Date: Wed, 2 Sep 2015 18:58:01 -0700 Message-ID: Subject: Re: ORC NPE while writing stats From: David Capwell To: user@hive.apache.org Content-Type: multipart/alternative; boundary=047d7bdc1678ff19c2051ece1c65 --047d7bdc1678ff19c2051ece1c65 Content-Type: text/plain; charset=UTF-8 I'll try that out and see if it goes away (not seen this in the past 24 hours, no code change). Doing this now means that I can't share the memory, so will prob go with a thread local and allocate fixed sizes to the pool per thread (50% heap / 50 threads). Will most likely be awhile before I can report back (unless it fails fast in testing) On Sep 2, 2015 2:11 PM, "Owen O'Malley" wrote: > (Dropping dev) > > Well, that explains the non-determinism, because the MemoryManager will be > shared across threads and thus the stripes will get flushed at effectively > random times. > > Can you try giving each writer a unique MemoryManager? You'll need to put > a class into the org.apache.hadoop.hive.ql.io.orc package to get access to > the necessary class (MemoryManager) and method > (OrcFile.WriterOptions.memory). We may be missing a synchronization on the > MemoryManager somewhere and thus be getting a race condition. > > Thanks, > Owen > > On Wed, Sep 2, 2015 at 12:57 PM, David Capwell wrote: > >> We have multiple threads writing, but each thread works on one file, so >> orc writer is only touched by one thread (never cross threads) >> On Sep 2, 2015 11:18 AM, "Owen O'Malley" wrote: >> >>> I don't see how it would get there. That implies that minimum was null, >>> but the count was non-zero. >>> >>> The ColumnStatisticsImpl$StringStatisticsImpl.serialize looks like: >>> >>> @Override >>> OrcProto.ColumnStatistics.Builder serialize() { >>> OrcProto.ColumnStatistics.Builder result = super.serialize(); >>> OrcProto.StringStatistics.Builder str = >>> OrcProto.StringStatistics.newBuilder(); >>> if (getNumberOfValues() != 0) { >>> str.setMinimum(getMinimum()); >>> str.setMaximum(getMaximum()); >>> str.setSum(sum); >>> } >>> result.setStringStatistics(str); >>> return result; >>> } >>> >>> and thus shouldn't call down to setMinimum unless it had at least some non-null values in the column. >>> >>> Do you have multiple threads working? There isn't anything that should be introducing non-determinism so for the same input it would fail at the same point. >>> >>> .. Owen >>> >>> >>> >>> >>> On Tue, Sep 1, 2015 at 10:51 PM, David Capwell >>> wrote: >>> >>>> We are writing ORC files in our application for hive to consume. >>>> Given enough time, we have noticed that writing causes a NPE when >>>> working with a string column's stats. Not sure whats causing it on >>>> our side yet since replaying the same data is just fine, it seems more >>>> like this just happens over time (different data sources will hit this >>>> around the same time in the same JVM). >>>> >>>> Here is the code in question, and below is the exception: >>>> >>>> final Writer writer = OrcFile.createWriter(path, >>>> OrcFile.writerOptions(conf).inspector(oi)); >>>> try { >>>> for (Data row : rows) { >>>> List struct = Orc.struct(row, inspector); >>>> writer.addRow(struct); >>>> } >>>> } finally { >>>> writer.close(); >>>> } >>>> >>>> >>>> Here is the exception: >>>> >>>> java.lang.NullPointerException: null >>>> at >>>> org.apache.hadoop.hive.ql.io.orc.OrcProto$StringStatistics$Builder.setMinimum(OrcProto.java:1803) >>>> ~[hive-exec-0.14.0.jar:0.14.0] >>>> at >>>> org.apache.hadoop.hive.ql.io.orc.ColumnStatisticsImpl$StringStatisticsImpl.serialize(ColumnStatisticsImpl.java:411) >>>> ~[hive-exec-0.14.0.jar:0.14.0] >>>> at >>>> org.apache.hadoop.hive.ql.io.orc.WriterImpl$StringTreeWriter.createRowIndexEntry(WriterImpl.java:1255) >>>> ~[hive-exec-0.14.0.jar:0.14.0] >>>> at >>>> org.apache.hadoop.hive.ql.io.orc.WriterImpl$TreeWriter.createRowIndexEntry(WriterImpl.java:775) >>>> ~[hive-exec-0.14.0.jar:0.14.0] >>>> at >>>> org.apache.hadoop.hive.ql.io.orc.WriterImpl$TreeWriter.createRowIndexEntry(WriterImpl.java:775) >>>> ~[hive-exec-0.14.0.jar:0.14.0] >>>> at >>>> org.apache.hadoop.hive.ql.io.orc.WriterImpl.createRowIndexEntry(WriterImpl.java:1978) >>>> ~[hive-exec-0.14.0.jar:0.14.0] >>>> at >>>> org.apache.hadoop.hive.ql.io.orc.WriterImpl.flushStripe(WriterImpl.java:1985) >>>> ~[hive-exec-0.14.0.jar:0.14.0] >>>> at >>>> org.apache.hadoop.hive.ql.io.orc.WriterImpl.checkMemory(WriterImpl.java:322) >>>> ~[hive-exec-0.14.0.jar:0.14.0] >>>> at >>>> org.apache.hadoop.hive.ql.io.orc.MemoryManager.notifyWriters(MemoryManager.java:168) >>>> ~[hive-exec-0.14.0.jar:0.14.0] >>>> at >>>> org.apache.hadoop.hive.ql.io.orc.MemoryManager.addedRow(MemoryManager.java:157) >>>> ~[hive-exec-0.14.0.jar:0.14.0] >>>> at >>>> org.apache.hadoop.hive.ql.io.orc.WriterImpl.addRow(WriterImpl.java:2276) >>>> ~[hive-exec-0.14.0.jar: >>>> >>>> >>>> Versions: >>>> >>>> Hadoop: apache 2.2.0 >>>> Hive Apache: 0.14.0 >>>> Java 1.7 >>>> >>>> >>>> Thanks for your time reading this email. >>>> >>> >>> > --047d7bdc1678ff19c2051ece1c65 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable

I'll try that out and see if it goes away (not seen this= in the past 24 hours, no code change).

Doing this now means that I can't share the memory, so w= ill prob go with a thread local and allocate fixed sizes to the pool per th= read (50% heap / 50 threads).=C2=A0 Will most likely be awhile before I can= report back (unless it fails fast in testing)

On Sep 2, 2015 2:11 PM, "Owen O'Malley&= quot; <omalley@apache.org> = wrote:
(Dropping dev)

Well, that explains the non-determini= sm, because the MemoryManager will be shared across threads and thus the st= ripes will get flushed at effectively random times.

Can you try giving each writer a unique MemoryManager? You'll need to= put a class into the org.apache.hadoop.hive.ql.io.orc package to get acces= s to the necessary class (MemoryManager) and method (OrcFile.WriterOptions.= memory). We may be missing a synchronization on the MemoryManager somewhere= and thus be getting a race condition.

Thanks,
=C2=A0 =C2=A0Owen

On Wed, Sep 2, 2015 at 12:57 PM, David Capwell <dcapwe= ll@gmail.com> wrote:

We have multiple threads writing, but each thread works on one fil= e, so orc writer is only touched by one thread (never cross threads)

On Sep 2, 2015 11:18 AM, "Owen O'Malley= " <omalley@= apache.org> wrote:
I don't see how it would get there. That implies= that minimum was null, but the count was non-zero.=C2=A0

The ColumnStatisticsImpl$StringStatisticsImpl.serialize looks like:
= 

@Override
Orc=
Proto.ColumnStatistics.Builder serialize() {
  OrcProto.ColumnStatistics=
.Builder result =3D sup=
er.serialize();
  OrcProto.StringStatistics.Builder str =3D
  =
  OrcProto.StringStatistics.newBuilder();
  if =
(getNumberOfValues() !=3D 0) {    str.setMinimum(getMinimum());
    str.setMaximum(getMaximum());
=
    str.setSum(sum);
  }
  result.setStringStatistics(str);
  return result;
}

= 
and t=
hus shouldn't call down to setMinimum unless it had at least some non-n=
ull values in the column.
Do you have multiple threads working? There isn&#=
39;t anything that should be introducing non-determinism so for the same in=
put it would fail at the same point.
.. Owen



= On Tue, Sep 1, 2015 at 10:51 PM, David Capwell <dcapwell@gmail.com>= ; wrote:
We are writing ORC files = in our application for hive to consume.
Given enough time, we have noticed that writing causes a NPE when
working with a string column's stats.=C2=A0 Not sure whats causing it o= n
our side yet since replaying the same data is just fine, it seems more
like this just happens over time (different data sources will hit this
around the same time in the same JVM).

Here is the code in question, and below is the exception:

final Writer writer =3D OrcFile.createWriter(path,
OrcFile.writerOptions(conf).inspector(oi));
try {
for (Data row : rows) {
=C2=A0 =C2=A0List<Object> struct =3D Orc.struct(row, inspector);
=C2=A0 =C2=A0writer.addRow(struct);
}
} finally {
=C2=A0 =C2=A0writer.close();
}


Here is the exception:

java.lang.NullPointerException: null
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.hadoop.hive.ql.io.orc.OrcProto$St= ringStatistics$Builder.setMinimum(OrcProto.java:1803)
~[hive-exec-0.14.0.jar:0.14.0]
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.hadoop.hive.ql.io.orc.ColumnStati= sticsImpl$StringStatisticsImpl.serialize(ColumnStatisticsImpl.java:411)
~[hive-exec-0.14.0.jar:0.14.0]
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.hadoop.hive.ql.io.orc.WriterImpl$= StringTreeWriter.createRowIndexEntry(WriterImpl.java:1255)
~[hive-exec-0.14.0.jar:0.14.0]
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.hadoop.hive.ql.io.orc.WriterImpl$= TreeWriter.createRowIndexEntry(WriterImpl.java:775)
~[hive-exec-0.14.0.jar:0.14.0]
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.hadoop.hive.ql.io.orc.WriterImpl$= TreeWriter.createRowIndexEntry(WriterImpl.java:775)
~[hive-exec-0.14.0.jar:0.14.0]
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.hadoop.hive.ql.io.orc.WriterImpl.= createRowIndexEntry(WriterImpl.java:1978)
~[hive-exec-0.14.0.jar:0.14.0]
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.hadoop.hive.ql.io.orc.WriterImpl.= flushStripe(WriterImpl.java:1985)
~[hive-exec-0.14.0.jar:0.14.0]
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.hadoop.hive.ql.io.orc.WriterImpl.= checkMemory(WriterImpl.java:322)
~[hive-exec-0.14.0.jar:0.14.0]
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.hadoop.hive.ql.io.orc.MemoryManag= er.notifyWriters(MemoryManager.java:168)
~[hive-exec-0.14.0.jar:0.14.0]
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.hadoop.hive.ql.io.orc.MemoryManag= er.addedRow(MemoryManager.java:157)
~[hive-exec-0.14.0.jar:0.14.0]
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.hadoop.hive.ql.io.orc.WriterImpl.= addRow(WriterImpl.java:2276)
~[hive-exec-0.14.0.jar:


Versions:

Hadoop: apache 2.2.0
Hive Apache: 0.14.0
Java 1.7


Thanks for your time reading this email.


--047d7bdc1678ff19c2051ece1c65--