Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 84B6F200C85 for ; Tue, 30 May 2017 16:36:06 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 836BA160BC9; Tue, 30 May 2017 14:36:06 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id CB321160BC1 for ; Tue, 30 May 2017 16:36:05 +0200 (CEST) Received: (qmail 82548 invoked by uid 500); 30 May 2017 14:36:05 -0000 Mailing-List: contact users-help@apex.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@apex.apache.org Delivered-To: mailing list users@apex.apache.org Received: (qmail 82536 invoked by uid 99); 30 May 2017 14:36:03 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 May 2017 14:36:03 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 04197C2084 for ; Tue, 30 May 2017 14:36:03 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -2.797 X-Spam-Level: X-Spam-Status: No, score=-2.797 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-2.796, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=datatorrent-com.20150623.gappssmtp.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id hGmEruIl65bb for ; Tue, 30 May 2017 14:35:59 +0000 (UTC) Received: from mail-pf0-f175.google.com (mail-pf0-f175.google.com [209.85.192.175]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 311785F3FE for ; Tue, 30 May 2017 14:35:58 +0000 (UTC) Received: by mail-pf0-f175.google.com with SMTP id e193so74238258pfh.0 for ; Tue, 30 May 2017 07:35:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=datatorrent-com.20150623.gappssmtp.com; s=20150623; h=subject:to:references:from:organization:message-id:date:user-agent :mime-version:in-reply-to:content-transfer-encoding:content-language; bh=ReV/Mb/SvBKKAFnTMm0XMbIyuddbTYi+4u0kcKSybR4=; b=RjdiRiNyiFRY0badPEStmzwdGSOoii8PqW1kSPOnNnUxg1Qj4RGjpnU62E+t9tduUI zRFYu9zg9gf0OejAVBGNYSt874af/uVIoXiCWFV4IyBq/3VydXMxlllaZCji1JwNksaj Nlm+D1MGMtKOfNEtwjOLw49fkq1JWVcEyHVTAC+eX6W06LfR6cTrjbf6EJ33Za8dtjcu uM9+JsJK6NVF8168KlbFucSoH8+QKR56qEHSxufbz9G4WxLEpSbRKFCfwyi5JgX9OPcr H6VkLrwwBK/kWAEiH/uDye30Br0KChG9NFw6W20P7DAd7DtqxzJAiswkj8SHejpHBRFk uARg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:organization :message-id:date:user-agent:mime-version:in-reply-to :content-transfer-encoding:content-language; bh=ReV/Mb/SvBKKAFnTMm0XMbIyuddbTYi+4u0kcKSybR4=; b=nE0t13yxnepTx5xIeADzvlgZlruwZcaqiOvplqtPh0/YOrZsNQ6JDXZLhjIgE8LFpf 1hDyiadOuYdjYEbr8LO23CuoV3IPQdd93wiYyajb7gNCIKGDGowBQulThYT2D9Gk0wJt xWVfZ0XsIHVHoJFS3yIfh/U6RK679VVjTs8Wb+gafsOwY1PQStrwak1se4b3hQGDwEIx XIKn7JxtwbkWQwPmqhSgqqNeZi4rw6Ho4JzaoVlTCYwnvdG5vII3/OjebDJ3810fJY3A 1P8wSNPUZ2cDSF1nWgRbgaSOoi+gcx+unt2y/7lZ3qZDsJs8i320qh3Itm7xB3Abp/xG sFMA== X-Gm-Message-State: AODbwcAUUy4QLjf8FTZOUf7cy1A7aSPb014JzBdzcdK4cL+sgs0si0UM OKRwdPfGhYBTyONFJ7o= X-Received: by 10.84.233.200 with SMTP id m8mr81733828pln.118.1496154956423; Tue, 30 May 2017 07:35:56 -0700 (PDT) Received: from vrozov.local (c-71-204-190-196.hsd1.ca.comcast.net. [71.204.190.196]) by smtp.googlemail.com with ESMTPSA id e124sm20203868pgc.17.2017.05.30.07.35.55 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 30 May 2017 07:35:55 -0700 (PDT) Subject: Re: Exceeded allowed memory block for AvroFileInputOperator To: users@apex.apache.org References: <1495808224234-1651.post@n6.nabble.com> From: Vlad Rozov Organization: DataTorrent Message-ID: Date: Tue, 30 May 2017 07:34:39 -0700 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:52.0) Gecko/20100101 Thunderbird/52.1.1 MIME-Version: 1.0 In-Reply-To: <1495808224234-1651.post@n6.nabble.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US archived-at: Tue, 30 May 2017 14:36:06 -0000 The warning means that the downstream operator(s) is not capable to keep up with the upstream operator and tuples are accumulated in the upstream buffer server leading to the increased latency. If the latency is within SLA, you may ignore the warning, otherwise try to partition downstream or decrease emit rate. Thank you, Vlad On 5/26/17 07:17, Vivek Bhide wrote: > I have a AvroFileInputOperator which has large no of files to be read when > the application starts. While the fire reading goes well at the start, I > start getting below warning in log after reading around 50+ files > > Is there something to be worried about and is there a way to mitigate it? > > 2017-05-26 09:10:36,233 INFO fs.AbstractFileInputOperator > (AbstractFileInputOperator.java:openFile(796)) - opening file > hdfs://///partition_date=2017-05-05/000000_0_copy_28 > 2017-05-26 09:11:32,145 WARN internal.DataList (DataList.java:run(743)) - > Exceeded allowed memory block allocation by 1 > 2017-05-26 09:11:58,595 WARN internal.DataList (DataList.java:run(743)) - > Exceeded allowed memory block allocation by 1 > > Regards > Vivek > > > > -- > View this message in context: http://apache-apex-users-list.78494.x6.nabble.com/Exceeded-allowed-memory-block-for-AvroFileInputOperator-tp1651.html > Sent from the Apache Apex Users list mailing list archive at Nabble.com.