storm-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Adam Lewis (JIRA)" <>
Subject [jira] [Commented] (STORM-241) "storm jar" underperforms compared to scp
Date Sun, 16 Feb 2014 13:26:19 GMT


Adam Lewis commented on STORM-241:

I took a crack at it, see PR #32

Are the configuration names I choose agreeable?

Is there a convention for also adding helper methods to deal with Config values (seems inconsistent
now, wasn't sure where to put this if at all)

Are there other places than StormSubmitter that should respect this config value?

> "storm jar" underperforms compared to scp
> -----------------------------------------
>                 Key: STORM-241
>                 URL:
>             Project: Apache Storm (Incubating)
>          Issue Type: Improvement
>         Environment: Internet connected dev machine deploying to storm cluster on AWS.
>            Reporter: Adam Lewis
>              Labels: configuration, newbie
> "storm jar" is very slow at sending the jar file, achieving roughly 10% of the throughput
compared to SCP for the same network configuration.  This is especially problematic when deploying
jars from over the Internet.
> This appears to be related to the chunk size used to transmit the JAR file over the thrift
protocol.  Anecdotally, a 300KB chunk size (compared to the currently used default of 15KB),
achieved performance in line with scp.
> Default can be overridden in the BufferFileInputStream ctor:

This message was sent by Atlassian JIRA

View raw message