spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacek Laskowski <ja...@japila.pl>
Subject Re: Environment tab meaning
Date Tue, 07 Jun 2016 20:16:01 GMT
Hi,

I'm not surprised to see Hadoop jars on the driver (yet I couldn't
explain exactly why they need to be there). I can't find a way now to
display the classpath for executors.

Pozdrawiam,
Jacek Laskowski
----
https://medium.com/@jaceklaskowski/
Mastering Apache Spark http://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski


On Tue, Jun 7, 2016 at 10:05 PM, satish saley <satishsaleyos@gmail.com> wrote:
> Thank you Jacek.
> In case of YARN, I see that hadoop jars are present in system classpath for
> Driver. Will it be the same for all executors?
>
> On Tue, Jun 7, 2016 at 11:22 AM, Jacek Laskowski <jacek@japila.pl> wrote:
>>
>> Ouch, I made a mistake :( Sorry.
>>
>> You've asked about spark **history** server. It's pretty much the same.
>>
>> HistoryServer is a web interface for completed and running (aka
>> incomplete) Spark applications. It uses EventLoggingListener to
>> collect events as JSON using org.apache.spark.util.JsonProtocol
>> object.
>>
>> Pozdrawiam,
>> Jacek Laskowski
>> ----
>> https://medium.com/@jaceklaskowski/
>> Mastering Apache Spark http://bit.ly/mastering-apache-spark
>> Follow me at https://twitter.com/jaceklaskowski
>>
>>
>> On Tue, Jun 7, 2016 at 8:18 PM, Jacek Laskowski <jacek@japila.pl> wrote:
>> > Hi,
>> >
>> > It is the driver - see the port. Is this 4040 or similar? It's started
>> > when SparkContext starts and is controlled by spark.ui.enabled.
>> >
>> > spark.ui.enabled (default: true) = controls whether the web UI is
>> > started or not.
>> >
>> > It's through JobProgressListener which is the SparkListener for web UI
>> > that the console knows what happens under the covers (and can
>> > calculate the stats).
>> >
>> > BTW, spark.ui.port (default: 4040) controls the port Web UI binds to.
>> >
>> > Pozdrawiam,
>> > Jacek Laskowski
>> > ----
>> > https://medium.com/@jaceklaskowski/
>> > Mastering Apache Spark http://bit.ly/mastering-apache-spark
>> > Follow me at https://twitter.com/jaceklaskowski
>> >
>> >
>> > On Tue, Jun 7, 2016 at 8:11 PM, satish saley <satishsaleyos@gmail.com>
>> > wrote:
>> >> Hi,
>> >> In spark history server, we see environment tab. Is it show environment
>> >> of
>> >> Driver or Executor or both?
>> >>
>> >> Jobs
>> >> Stages
>> >> Storage
>> >> Environment
>> >> Executors
>> >>
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message