spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Owen <so...@cloudera.com>
Subject Re: Result obtained before the completion of Stages
Date Tue, 26 Dec 2017 22:27:49 GMT
My guess is that either they haven't actually finished before the result
and something about timestamps you're comparing is misleading, or else,
you're looking at stages executing that are part of a later part of the
program.

On Tue, Dec 26, 2017 at 3:49 PM ckhari4u <ckhari4u@gmail.com> wrote:

> I found this interesting behavior while running some adhoc analysis query.
> I
> have a Spark SQL query where I am joining 2 tables and then performing a
> count operation. In the Spark Web UI, I see there are 4 Stages getting
> launched.
>
> The interesting behavior I see here is that I see the result before all
> stages are executed. The Stage 2 which performs the Sort merge join is
> running but I see the result in the Spark Shell before the completion of
> Stage 2. However, the application still continues to run?
>
> Any thoughts on this behavior?
>
>
>
>
>
> --
> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>

Mime
View raw message