Return-Path: X-Original-To: apmail-storm-dev-archive@minotaur.apache.org Delivered-To: apmail-storm-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 06C3210DC7 for ; Sat, 15 Feb 2014 19:44:43 +0000 (UTC) Received: (qmail 48590 invoked by uid 500); 15 Feb 2014 19:44:41 -0000 Delivered-To: apmail-storm-dev-archive@storm.apache.org Received: (qmail 48516 invoked by uid 500); 15 Feb 2014 19:44:40 -0000 Mailing-List: contact dev-help@storm.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@storm.incubator.apache.org Delivered-To: mailing list dev@storm.incubator.apache.org Received: (qmail 48507 invoked by uid 99); 15 Feb 2014 19:44:40 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 15 Feb 2014 19:44:40 +0000 X-ASF-Spam-Status: No, hits=-2000.6 required=5.0 tests=ALL_TRUSTED,RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [140.211.11.3] (HELO mail.apache.org) (140.211.11.3) by apache.org (qpsmtpd/0.29) with SMTP; Sat, 15 Feb 2014 19:44:39 +0000 Received: (qmail 48472 invoked by uid 99); 15 Feb 2014 19:44:19 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 15 Feb 2014 19:44:19 +0000 Date: Sat, 15 Feb 2014 19:44:19 +0000 (UTC) From: "P. Taylor Goetz (JIRA)" To: dev@storm.incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (STORM-241) "storm jar" underperforms compared to scp MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/STORM-241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13902500#comment-13902500 ] P. Taylor Goetz commented on STORM-241: --------------------------------------- I agree that this would be good to have. I think the best approach would be to add it as a configuration value to Conf.java with sane defaults defined in defaults.yaml. This is a pretty simple change that would be a good one for prospective contributors to tackle, and one which I would vote in favor of. > "storm jar" underperforms compared to scp > ----------------------------------------- > > Key: STORM-241 > URL: https://issues.apache.org/jira/browse/STORM-241 > Project: Apache Storm (Incubating) > Issue Type: Improvement > Environment: Internet connected dev machine deploying to storm cluster on AWS. Storm 0.9.0.1 > Reporter: Adam Lewis > Labels: configuration, newbie > > "storm jar" is very slow at sending the jar file, achieving roughly 10% of the throughput compared to SCP for the same network configuration. This is especially problematic when deploying jars from over the Internet. > This appears to be related to the chunk size used to transmit the JAR file over the thrift protocol. Anecdotally, a 300KB chunk size (compared to the currently used default of 15KB), achieved performance in line with scp. > Default can be overridden in the BufferFileInputStream ctor: https://github.com/apache/incubator-storm/blob/master/storm-core/src/jvm/backtype/storm/StormSubmitter.java?source=cc#L154 -- This message was sent by Atlassian JIRA (v6.1.5#6160)