Mailing-List: contact issues-help@spark.apache.org; run by ezmlm
Precedence: bulk
Date: Thu, 14 Sep 2017 12:05:01 +0000 (UTC)
From: =?utf-8?Q?Maciej_Bry=C5=84ski_=28JIRA=29?= <jira@apache.org>
To: issues@spark.apache.org
Message-ID: <JIRA.13102207.1505385886000.111752.1505390701490@Atlassian.JIRA>
In-Reply-To: <JIRA.13102207.1505385886000@Atlassian.JIRA>
References: <JIRA.13102207.1505385886000@Atlassian.JIRA> <JIRA.13102207.1505385886859@jira-lw-us.apache.org>
Subject: [jira] [Commented] (SPARK-22010) Slow fromInternal conversion for
 TimestampType
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
archived-at: Thu, 14 Sep 2017 12:05:07 -0000


    [ https://issues.apache.org/jira/browse/SPARK-22010?page=3Dcom.atlassia=
n.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D161=
66136#comment-16166136 ]=20

Maciej Bry=C5=84ski commented on SPARK-22010:
----------------------------------------

The reason of this Jira is this profiling (attachment).
As you can see about 80% of pyspark time is spent in Spark internals.


> Slow fromInternal conversion for TimestampType
> ----------------------------------------------
>
>                 Key: SPARK-22010
>                 URL: https://issues.apache.org/jira/browse/SPARK-22010
>             Project: Spark
>          Issue Type: Bug
>          Components: PySpark
>    Affects Versions: 2.2.0
>            Reporter: Maciej Bry=C5=84ski
>
> To convert timestamp type to python we are using=20
> `datetime.datetime.fromtimestamp(ts // 1000000).replace(microsecond=3Dts =
% 1000000)`
> code.
> {code}
> In [34]: %%timeit
>     ...: datetime.datetime.fromtimestamp(1505383647).replace(microsecond=
=3D12344)
>     ...:
> 4.2 =C2=B5s =C2=B1 558 ns per loop (mean =C2=B1 std. dev. of 7 runs, 1000=
00 loops each)
> {code}
> It's slow, because:
> # we're trying to get TZ on every conversion
> # we're using replace method
> Proposed solution: custom datetime conversion and move calculation of TZ =
to module


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org