Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 361C6200C33 for ; Sat, 11 Mar 2017 13:13:55 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 34E42160B7B; Sat, 11 Mar 2017 12:13:55 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 302B3160B5D for ; Sat, 11 Mar 2017 13:13:54 +0100 (CET) Received: (qmail 43761 invoked by uid 500); 11 Mar 2017 12:13:53 -0000 Mailing-List: contact user-help@ignite.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@ignite.apache.org Delivered-To: mailing list user@ignite.apache.org Received: (qmail 43751 invoked by uid 99); 11 Mar 2017 12:13:53 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Mar 2017 12:13:53 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id D1DF5180B10 for ; Sat, 11 Mar 2017 12:13:52 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.88 X-Spam-Level: ** X-Spam-Status: No, score=2.88 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, KAM_NUMSUBJECT=0.5, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id QNl_Ho8uV95c for ; Sat, 11 Mar 2017 12:13:48 +0000 (UTC) Received: from mail-ua0-f178.google.com (mail-ua0-f178.google.com [209.85.217.178]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 1A6F15F5CC for ; Sat, 11 Mar 2017 12:13:48 +0000 (UTC) Received: by mail-ua0-f178.google.com with SMTP id u30so135087098uau.0 for ; Sat, 11 Mar 2017 04:13:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=K12zwpTI1L61juukh8K/3JVM2/IRCs6mq57Fbe7X0FA=; b=lXebQEERPc5ofku3PGoqqKTDZ7dzeMQ8u1KVpG5ODedAwCeKINrWaJ384wYnc9bQ4n 8zRXOH69Oej+Yxsn0Mmo0OIO0P2AypriAeKlybkTCBcrZoiRpRyBuxJMEbVNid0MHI7d X9Zp1ZZ6gC8u6D1u6mDF+TKE2uD/TXxefH0ZbxaK+TFDSFrTn5MXBb2qyRicsdsUoW5u 4lFnwhfpnwlUfIemxQuuWD4HptG/X2RdVdE+/q6jm5gxQ6Nn3TDj7O7t4/dx4R+9u91h UkcQJCZJ/jrtOMUWNUP+2gvOeRDm7ypKJRVhs0yE4Z8TBR1deBvsxOaJvxfgIY2bWoSO TrvQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=K12zwpTI1L61juukh8K/3JVM2/IRCs6mq57Fbe7X0FA=; b=gVq7vBD3Xg+JVYpUWLivZ2Hzp4Edfg+0FYehM2m4XxUX5KwR95no/UTClzSCCOYYAX XaphW378pQRRT12ntFgPEpmjhoGqgOnizd/mFDh8j+IztrETpbNpA2GoWS5hiN6b487m ChZDMGfzfR5tT2aE6kl4xmEglzfnuh1TUoRgYERgfszqZsAYqA5qRT6Ycb2uuwAzFzev 4LqXYxUQF+TpstjqNY/EiJnCi2F1bvBT9C4spWceqw80103q6fV+0OPOGPOZv9DP1cLu tr0wvh96awaUgKgTLnEwpfAENlmiSiv4eA+89rItUJaliqY/MANuPb2bzSYT/sFryxir IEWg== X-Gm-Message-State: AMke39mCnrJCF0r3Z4OjbmKA/cTRBZtuAyQFAeMl4LaEcj61riAK3Mxm1q9b96kr8ZR5OSRPPUFUCDn+q5eDaQ== X-Received: by 10.159.38.78 with SMTP id 72mr11888767uag.109.1489234072861; Sat, 11 Mar 2017 04:07:52 -0800 (PST) MIME-Version: 1.0 Received: by 10.176.1.210 with HTTP; Sat, 11 Mar 2017 04:07:52 -0800 (PST) Received: by 10.176.1.210 with HTTP; Sat, 11 Mar 2017 04:07:52 -0800 (PST) In-Reply-To: References: From: Andrey Mashenkov Date: Sat, 11 Mar 2017 15:07:52 +0300 Message-ID: Subject: Re: IGNITE-4106 To: user@ignite.apache.org Content-Type: multipart/alternative; boundary=001a113ab816e68499054a7353f9 archived-at: Sat, 11 Mar 2017 12:13:55 -0000 --001a113ab816e68499054a7353f9 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Thanks Anil. I'll take a look. 11 =D0=BC=D0=B0=D1=80=D1=82=D0=B0 2017 =D0=B3. 9:58 =D0=BF=D0=BE=D0=BB=D1= =8C=D0=B7=D0=BE=D0=B2=D0=B0=D1=82=D0=B5=D0=BB=D1=8C "Anil" =D0=BD=D0=B0=D0=BF=D0=B8=D1=81=D0=B0=D0=BB: Hi Andrey, i am able to reproduce the issue with following reproducer. https://github.com/adasari/test-ignite-parallelism.git Please let me know if you see any issue with reproducer. Thanks On 11 March 2017 at 11:46, Anil wrote: > Hi Andrey, > > i have created test project which pushes the entries to drastreamer using > for loop and it is working without any issue. > > But my application which does the same thing but loads the data from hbas= e > and it is failing with *"java.lang.IllegalStateException: Data streamer > has been closed." * > > > > *java.lang.IllegalStateException: Data streamer has been closed. a= t > org.apache.ignite.internal.pro > cessors.datastreamer.DataStreamerI= mpl.enterBusy(DataStreamerImpl.java:406) > at org.apache.ignite.internal.pro > cessors.datastreamer.DataStreamerI= mpl.addDataInternal(DataStreamerImpl.java:613) > at org.apache.ignite.internal.pro > cessors.datastreamer.DataStreamerI= mpl.addData(DataStreamerImpl.java:667)* > > i set the system and public thread pool as 32 for an 8 core machine and > using 4 node cluster. > > and i see that loading data into cache for which queryParallelism > configured is failing and other cache loads working fine. > > Thanks > > > On 10 March 2017 at 05:05, Andrey Mashenkov > wrote: > >> Hi Anil, >> >> I don't think it is a thead starvation due to it work fine for 1 thread. >> Whould you please share a repro? >> >> On Thu, Mar 9, 2017 at 9:09 PM, Anil wrote: >> >>> >>> Hi Andrey, >>> >>> i tried to set the parallelism to 2, 4 on 4 node cluster (8 core >>> machines) and initiated the data load (using compute job). Data streame= rs >>> are getting closed and data load is failing. >>> >>> When I set the parallelism to 1, data load working as expected. the >>> issue could be threads starvation/ non availability of resources and no= t >>> ignite issue. Do you think any other issue ? >>> >>> I will try on 16 core machines cluster tomorrow. >>> >>> Thanks. >>> >>> On 8 March 2017 at 18:04, Andrey Mashenkov >>> wrote: >>> >>>> Hi Anil, >>>> >>>> SQL queries are run in system pool. >>>> >>>> On Wed, Mar 8, 2017 at 3:04 PM, Anil wrote: >>>> >>>>> Hi, >>>>> >>>>> Does queryparallelism uses system pool or separate thread pool other >>>>> than system and public thread pool ? please clarify. >>>>> >>>>> 1. https://apacheignite.readme.io/docs/sql-performance-and-d >>>>> ebugging#sql-performance-and-usability-considerations >>>>> >>>>> Thanks. >>>>> >>>>> >>>> >>>> >>>> >>>> -- >>>> Best regards, >>>> Andrey V. Mashenkov >>>> >>> >>> >> >> >> -- >> Best regards, >> Andrey V. Mashenkov >> > > --001a113ab816e68499054a7353f9 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Thanks Anil.

I'll take a look.

11 =D0=BC=D0=B0=D1=80=D1=82=D0=B0 2017 =D0= =B3. 9:58 =D0=BF=D0=BE=D0=BB=D1=8C=D0=B7=D0=BE=D0=B2=D0=B0=D1=82=D0=B5=D0= =BB=D1=8C "Anil" <anilkl= ce@gmail.com> =D0=BD=D0=B0=D0=BF=D0=B8=D1=81=D0=B0=D0=BB:
Hi Andrey,

i am able to reproduce the issue with following repro= ducer.


Please let me k= now if you see any issue with reproducer.

Thanks

On 11 March 2017 at 11:46, Anil <<= a href=3D"mailto:anilklce@gmail.com" target=3D"_blank">anilklce@gmail.com> wrote:
Hi = Andrey,

i have created test project which pushes the ent= ries to drastreamer using for loop and it is working without any issue.

But my application which does the same thing but load= s the data from hbase and it is failing with "java.lang.IllegalStat= eException: Data streamer has been closed."=C2=A0
<= b>
jav= a.lang.IllegalStateException: Data streamer has been closed.
=
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.ignite.internal.processors.datastr= eamer.DataStreamerImpl.enterBusy(DataStreamerImpl.java:406)
=
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.ignite.internal.processors.datastreamer.Data= StreamerImpl.addDataInternal(DataStreamerImpl.java:613)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.ignite.internal.processors.datastreamer.DataStre= amerImpl.addData(DataStreamerImpl.java:667)

<= /b>

i set the system and public thread pool= as 32 for an 8 core machine and using 4 node cluster.

=
and i see that loading data into cache for which queryParallelism conf= igured is failing and other cache loads working fine.

<= div>Thanks


On 10 March 2017 at 05:05, Andrey Mashenkov <andrey.mashenkov@gmail.com> wrote:
Hi Anil,

I don't think it= is a thead starvation due to it work fine for 1 thread.
Whould y= ou please share a repro?

On Thu, Mar 9, 2017 at 9:09 PM, Anil &= lt;anilklce@gmail.c= om> wrote:

Hi Andrey,

i tried to set the parallelis= m to 2, 4 on 4 node cluster (8 core machines) and initiated the data load (= using compute job). Data streamers are getting closed and data load is fail= ing.=C2=A0

When I set the parallelism to 1, data l= oad working as expected. the issue could be threads starvation/ non availab= ility of resources and not ignite issue. Do you think any other issue ?

I will try on 16 core machines cluster tomorrow.=C2= =A0

Thanks.

On 8 March 2017 at 18:04, Andrey Mashenkov <andrey.mashenkov@gmail.com> wrote:
Hi Anil,

SQL queries are = run in system pool.=C2=A0

On Wed, Mar 8, 2017 at 3:04 PM, Anil = <anilklce@gmail.= com> wrote:
Hi,

Does queryparallelism uses system pool or separate= thread pool other than system and public thread pool ? please clarify.


Thanks.
=C2=A0



--
B= est regards,
Andrey V. Mashenkov




--
Best regards,
Andrey V. Mashenkov



--001a113ab816e68499054a7353f9--