Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id B04B8200C43 for ; Sun, 26 Mar 2017 15:21:12 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id AEDC6160B87; Sun, 26 Mar 2017 13:21:12 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id F1F10160B6B for ; Sun, 26 Mar 2017 15:21:11 +0200 (CEST) Received: (qmail 80370 invoked by uid 500); 26 Mar 2017 13:21:11 -0000 Mailing-List: contact user-help@ignite.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@ignite.apache.org Delivered-To: mailing list user@ignite.apache.org Received: (qmail 80360 invoked by uid 99); 26 Mar 2017 13:21:11 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Mar 2017 13:21:11 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 863D3C0C04 for ; Sun, 26 Mar 2017 13:21:10 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.629 X-Spam-Level: ** X-Spam-Status: No, score=2.629 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id 60oWjCwkGieE for ; Sun, 26 Mar 2017 13:21:09 +0000 (UTC) Received: from mail-yw0-f180.google.com (mail-yw0-f180.google.com [209.85.161.180]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 0013A5F610 for ; Sun, 26 Mar 2017 13:21:08 +0000 (UTC) Received: by mail-yw0-f180.google.com with SMTP id d191so16635151ywe.2 for ; Sun, 26 Mar 2017 06:21:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=Spa0kA/XZFjo4en+Z812ZiJGF/lN+LJHcaj+W1EAWqY=; b=JgX6nvvucU/uMoGtn3shYgVO7BonkvlXKoDacyL2muLNlpe+nl9b+kgC52BoOtoHs3 dMIAlQ3nzawD9RyouvGdX774LYvzQ/LqDIO3FuiOzNy4EdBqX3Wwm3bwZUpyvEAYhi2W +VAHBcZLYmnncLrOJGVpHF+OVUxwRxl5Mj4mrXk8+E8EsdBgShcSWE1BaANoFSW0swqE CuXPVE6ldeYcDcK0+FZpuyOI6ItY1Yr3h+qLhQMX47MHPpyc69vAZdHeIu6+s1KBulkJ w5TdlHaZNBk2tGh3RsXkgTipHN/V/hQVd6ab6oCIpVsbqZcIe3F/mQwz5ONbZvo71KlY ASFw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=Spa0kA/XZFjo4en+Z812ZiJGF/lN+LJHcaj+W1EAWqY=; b=QTctqB42KIHc9Nu7CTgRVFTe2NYGZdrQ61kWl8XKnW+a83Yw0wDfSc9oCWUuuqi0yd 0frwKEeSdlt7f6fBA3Dqjfmn/IrULhrGaVX+3FzC+dTiWUL+34zSu+m/ipNqcp4gPa+y 5+w8Y0qAQJJmqgX1PRYKcJ9yVQXXV+h3i5rpqwqQGUnF5grrDnGRPoUPmYMmDR1jHYaO YwYNA2AN80OBPxApvq0oAYUbBuR7ScE2q9rPh7i7gbh0s4Dl9VmFdzQVuwGc+VEd7Jeb jxno3qY81SVo5ICm2P71EPYxzFUjZ3t1U9ZMVaZDfKVR+NKo5YMKCKFqzYpMm+C0c2ef QTlQ== X-Gm-Message-State: AFeK/H05P0FHV8j+xhvipeFviC3GhkS+oeu/yrQ32mvzXbUz/3MfTF1zGu4UYvoJ5cDTfNEaQKC1PJOS/C4m/w== X-Received: by 10.13.234.204 with SMTP id t195mr14582165ywe.58.1490534060535; Sun, 26 Mar 2017 06:14:20 -0700 (PDT) MIME-Version: 1.0 Received: by 10.13.198.66 with HTTP; Sun, 26 Mar 2017 06:13:40 -0700 (PDT) From: Raul Date: Sun, 26 Mar 2017 14:13:40 +0100 Message-ID: Subject: What is the best approach for large number of concurrent mappings? To: user@ignite.apache.org Content-Type: multipart/alternative; boundary=94eb2c08791e343b83054ba20142 archived-at: Sun, 26 Mar 2017 13:21:12 -0000 --94eb2c08791e343b83054ba20142 Content-Type: text/plain; charset=UTF-8 > > Hi, > > I am trying to work out what the best approach would be for the following > problem? > > Conditions: > > 1. You have a river of data flowing at a varying rate into a cache, > the cache holds the last N minuets of data. > 2. The river of data has to be analyzed as fast as possible > 3. You have over 10,000 different map reduce functions that has to > aggregate into at least 1,000,000 buckets > > Question: > Is it a good idea to spin up 10k continuous mappers or is there a better > way > > Thank you for assistance > > > --94eb2c08791e343b83054ba20142 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable

Hi,=

I am= trying to work out what the best approach would be for the following probl= em?

Condi= tions:

  1. You have a river of data flowing at a varying rate into a cache= , the cache holds the last N minuets of data.
  2. The river of data has to be analyzed as fast as poss= ible
  3. You have over 10,0= 00 different map reduce functions that has to aggregate into at least 1,000= ,000 buckets

Question:
Is it a good idea to= spin up 10k continuous mappers or is there a better way

Thank you for assistance


--94eb2c08791e343b83054ba20142--