Mailing-List: contact commits-help@beam.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@beam.apache.org
Date: Thu, 31 Aug 2017 04:09:00 +0000 (UTC)
From: "Kenneth Knowles (JIRA)" <jira@apache.org>
To: commits@beam.apache.org
Message-ID: <JIRA.13082590.1498499317000.166493.1504152540156@Atlassian.JIRA>
In-Reply-To: <JIRA.13082590.1498499317000@Atlassian.JIRA>
References: <JIRA.13082590.1498499317000@Atlassian.JIRA> <JIRA.13082590.1498499317723@jira-lw-us.apache.org>
Subject: [jira] [Commented] (BEAM-2516) User reports 4 minutes to process 1
 million line CSV in DirectRunner
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Thu, 31 Aug 2017 04:09:08 -0000


    [ https://issues.apache.org/jira/browse/BEAM-2516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16148418#comment-16148418 ] 

Kenneth Knowles commented on BEAM-2516:
---------------------------------------

I think for 2.2.0 it is best to remove the translation to/from a proto by hiding it behind PipelineOptions.

There's a lot of overhead right now because of the impedance mismatch between the parts that are still Java-specific and the parts which are SDK-agnostic. In the full story for the portability framework, the DoFns and other UDFs can't even be deserialized, but shipped to the SDK harness. The harness will own the caching, so it probably doesn't make sense to add it to the DirectRunner unless there's one silly repeated deserialization we can eliminate. Based on the profiling results, perhaps there is, but no need to block anything on it.

> User reports 4 minutes to process 1 million line CSV in DirectRunner
> --------------------------------------------------------------------
>
>                 Key: BEAM-2516
>                 URL: https://issues.apache.org/jira/browse/BEAM-2516
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-direct
>            Reporter: Kenneth Knowles
>            Priority: Minor
>             Fix For: 2.2.0
>
>
> https://stackoverflow.com/questions/44736414/simple-apache-beam-manipulations-work-very-slow
> I don't know what the expectation are here, so I wasn't ready to say this is WAI. Low priority since it isn't what the runner is for anyhow, but this seems like the scale of data that should be snappy. Worth investigating, or maybe you can quickly indicate why it is expected?


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)