From user-return-34180-archive-asf-public=cust-asf.ponee.io@flink.apache.org Thu Apr 16 08:38:35 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id A60EA180638 for ; Thu, 16 Apr 2020 10:38:35 +0200 (CEST) Received: (qmail 69669 invoked by uid 500); 16 Apr 2020 08:38:34 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@flink.apache.org Received: (qmail 69643 invoked by uid 99); 16 Apr 2020 08:38:34 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Apr 2020 08:38:34 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 8D395C1D07 for ; Thu, 16 Apr 2020 08:38:33 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.821 X-Spam-Level: X-Spam-Status: No, score=-0.821 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.821, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-ec2-va.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id JIU5nMKUqkPM for ; Thu, 16 Apr 2020 08:38:32 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.166.41; helo=mail-io1-f41.google.com; envelope-from=elkhan.dadashov@gmail.com; receiver= Received: from mail-io1-f41.google.com (mail-io1-f41.google.com [209.85.166.41]) by mx1-ec2-va.apache.org (ASF Mail Server at mx1-ec2-va.apache.org) with ESMTPS id 47258BB980 for ; Thu, 16 Apr 2020 08:38:32 +0000 (UTC) Received: by mail-io1-f41.google.com with SMTP id y17so20122716iow.9 for ; Thu, 16 Apr 2020 01:38:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=D8fZiPtDCKD5oyjYjFpaXWLWterFG0up+2goWW7fwPg=; b=I4HB1XSs3m6U1+UthU5igbbddo0sJ9MfGptQx+25YYg2L0ZdtseHGs+RL4sN+vhkhk qyLbUIM2yR96BOmhs/9ElhinEp95GYKbwGMP7ZBLOtT5D8/gJgbMh5ozZJqS9KsF2vG0 2MbJpBIHqfqti5eMF42+ryVOQbC0cLoCraB4NK+e+T6M/TXwlha3cqlpPRsUlmFbh6xX zoqb+0jINXwK/aqxNmYjh3eirmQ4VCXYG+hW+FnOzm2W+Z3Y0Hjb33HVkxa3N19iqGRS pp5xqSC20COHHCJuKuSXhFtDaoSRWnAlWKGfEGBAsEeaxjHrocicRl9fYpXp4c/+aUYX howg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=D8fZiPtDCKD5oyjYjFpaXWLWterFG0up+2goWW7fwPg=; b=C1DGRaAUhLKkF4X5dlLfPas4wmd+JlhEV7tziSMB5fTvmRsVPEDGxWVj/EAdZGpzY3 C8R80hAHJupvD6Dqiw+atp8gvBfYMsE6ACuC4xzcN2InNDLShtDtzHD2nPu9xNhsajS7 vOwdmClDQWtT2qhutmcHftQO7s3de+Y5JUyJo63xXXavPI5NTnArG3BVSQbPopjMjQEl TqnjW1DxHcnQxwwCMolYrxG41/lUBYHb80usaRnvAznOEDMSEjks0bFLZSo1yeeMiky2 TqRGPqNeEXrDMToM8N94HqmxyAX97HRHEHRR3Hv3zDbBJzCSYsmbyG2XEaMUWRRGamfy YMng== X-Gm-Message-State: AGi0PuYfrlfT13CQgziSZc3GQVaeRZhCkIowxGjcrXVjqREE51SVY7EN RHVbrf3smeXdYgTzneyc1/6hkZD7w+TAKj3aOg0eIAGxRY/m2Q== X-Google-Smtp-Source: APiQypJS0KDVhIAwhw/l0UgSq8am9OmHzwXzBo0W6rYBSezRCI4KTse2M1AxoPoS2Bh82YLCJYqSoWJH8ZWDm/akf50= X-Received: by 2002:a02:6d02:: with SMTP id m2mr29817785jac.54.1587026311516; Thu, 16 Apr 2020 01:38:31 -0700 (PDT) MIME-Version: 1.0 From: Elkhan Dadashov Date: Thu, 16 Apr 2020 01:37:55 -0700 Message-ID: Subject: How to scale a streaming Flink pipeline without abusing parallelism for long computation tasks? To: user Content-Type: multipart/alternative; boundary="0000000000008c164e05a3645bec" --0000000000008c164e05a3645bec Content-Type: text/plain; charset="UTF-8" Hi Flink users, I have a basic Flnk pipeline, doing flatmap. inside flatmap, I get the input, path it to the client library to compute some result. That library execution takes around 30 seconds to 2 minutes (depending on the input ) for producing the output from the given input ( it is time-series based long-running computation). As it takes the library long time to compute, the input payloads keep buffered, and if not given enough parallelism, the job will crash/restart. (java.lang.RuntimeException: Buffer pool is destroyed.) Wanted to check what are other options for scaling Flink streaming pipeline without abusing parallelism for long-running computations in Flink operator? Is multi-threading inside the operator recommended? ( even though the single input computation takes a long time, but I can definitely run 4-8 of them in parallel threads, instead of one by one, inside the same FlatMap operator. 1 core for each yarn slot ( which will hold 1 flatmap operator) seems too expensive. If we could launch more link operators with only 1 core, it could have been easier. If anyone faced a similar issue please share your experience. I'm using Flink 1..6.3 version. Thanks. --0000000000008c164e05a3645bec Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Flink users,

I have a basic Flnk=C2= =A0pipeline, doing flatmap.

inside flatmap, I get = the input, path it to the client library to compute some result.
=
That library execution takes around 30 seconds to 2 minutes = (depending on the input ) for producing=C2=A0the output from the given inpu= t ( it is time-series=C2=A0based long-running computation).=C2=A0

As it takes the library long time to compute, the input pay= loads keep buffered, and if not given enough parallelism, the job will cras= h/restart. (java.lang.RuntimeException: Buffer pool is destroyed.)

Wanted to check what are other options for scaling Flink s= treaming pipeline without abusing parallelism for long-running computations= in Flink operator?

Is multi-threading=C2=A0inside= the operator recommended? ( even though the single input computation takes= a long time, but I can definitely run 4-8 of them in parallel threads, ins= tead of one by one, inside the same FlatMap operator.

<= div>1 core for each yarn slot ( which will hold 1 flatmap operator) seems t= oo expensive. If we could launch more link operators with only 1 core, it c= ould have been easier.

If anyone faced a similar i= ssue please share your experience. I'm using Flink 1..6.3 version.

Thanks.
--0000000000008c164e05a3645bec--