spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sumedh Wale <sw...@snappydata.io>
Subject Re: Add Jars to Master/Worker classpath
Date Wed, 02 Mar 2016 17:35:18 GMT
<html>
  <head>
    <meta content="text/html; charset=utf-8" http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    <div class="moz-cite-prefix">On Wednesday 02 March 2016 09:39 PM,
      Matthias Niehoff wrote:<br>
    </div>
    <blockquote
cite="mid:CADjSX1z5fq29SZxMb_9FCcrL+YQ4GNkN92BoFHu9uFAN9JZf-g@mail.gmail.com"
      type="cite">
      <div dir="ltr">no, not to driver and executor but to the master
        and worker instances of the spark standalone cluster</div>
      <div class="gmail_extra"><br>
      </div>
    </blockquote>
    <br>
    Why exactly does adding jars to driver/executor extraClassPath not
    work?<br>
    <br>
    Classpath of master/worker is setup by AbstractCommandBuilder that
    explicitly adds the following:<br>
    <br>
    jars named "datanucleus-*", environment variables: _SPARK_ASSEMBLY
    (for assembly jar), SPARK_DIST_CLASSPATH, HADOOP_CONF_DIR,
    YARN_CONF_DIR<br>
    <br>
    So you can set SPARK_DIST_CLASSPATH in conf/spark-env.sh to add the
    required jars (separated by platform's File.pathSeparator).<br>
    <br>
    <br>
    thanks<br>
    <pre class="moz-signature" cols="72">-- 
Sumedh Wale
SnappyData (<a class="moz-txt-link-freetext" href="http://www.snappydata.io">http://www.snappydata.io</a>)</pre>
    <br>
    <blockquote
cite="mid:CADjSX1z5fq29SZxMb_9FCcrL+YQ4GNkN92BoFHu9uFAN9JZf-g@mail.gmail.com"
      type="cite">
      <div class="gmail_extra">
        <div class="gmail_quote">Am 2. März 2016 um 17:05 schrieb Igor
          Berman <span dir="ltr">&lt;<a moz-do-not-send="true"
              href="mailto:igor.berman@gmail.com" target="_blank">igor.berman@gmail.com</a>&gt;</span>:<br>
          <blockquote class="gmail_quote" style="margin:0 0 0
            .8ex;border-left:1px #ccc solid;padding-left:1ex">
            <div dir="ltr"><span
                style="color:rgb(68,68,68);font-family:Menlo,'Lucida
                Console',monospace;font-size:12px;line-height:20px">spark.driver.extraClassPath</span><br>
              <div><span
                  style="color:rgb(68,68,68);font-family:Menlo,'Lucida
                  Console',monospace;font-size:12px;line-height:20px">spark.executor.extraClassPath</span><span
                  style="color:rgb(68,68,68);font-family:Menlo,'Lucida
                  Console',monospace;font-size:12px;line-height:20px"><br>
                </span></div>
            </div>
            <div class="HOEnZb">
              <div class="h5">
                <div class="gmail_extra"><br>
                  <div class="gmail_quote">2016-03-02 18:01 GMT+02:00
                    Matthias Niehoff <span dir="ltr">&lt;<a
                        moz-do-not-send="true"
                        href="mailto:matthias.niehoff@codecentric.de"
                        target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:matthias.niehoff@codecentric.de">matthias.niehoff@codecentric.de</a></a>&gt;</span>:<br>
                    <blockquote class="gmail_quote" style="margin:0 0 0
                      .8ex;border-left:1px #ccc solid;padding-left:1ex">
                      <div dir="ltr">Hi,
                        <div><br>
                        </div>
                        <div>we want to add jars to the Master and
                          Worker class path mainly for logging reason
                          (we have a redis appender to send logs to
                          redis -&gt; logstash -&gt; elasticsearch). </div>
                        <div><br>
                        </div>
                        <div>
                          <div>While it is worki<font color="#000000">ng
                              with setting SPARK_CLASSPATH, this
                              solution is afaik deprecated and should
                              not be used. Furthermore we are also using<font
                                face="arial, helvetica, sans-serif"> <span
                                  style="line-height:20px">—driver-java-options
                                  and </span><span
                                  style="line-height:20px">spark.executor.extraClassPath
                                  which leads to exceptions when running
                                  our apps in standalone cluster mode. </span></font></font></div>
                          <div><br>
                          </div>
                          <div><font face="arial, helvetica, sans-serif"
                              color="#000000">So what is the best way to
                              add jars to the master and
                              worker classpath?</font></div>
                          <div><font face="arial, helvetica, sans-serif"
                              color="#000000"><br>
                            </font></div>
                          <div><font face="arial, helvetica, sans-serif"
                              color="#000000">Thank you</font></div>
                          <span><font color="#888888">
                              <div><br>
                              </div>
                              -- <br>
                              <div>
                                <div dir="ltr">
                                  <div>
                                    <div dir="ltr"><span
                                        style="color:rgb(136,136,136);font-size:12.8000001907349px">Matthias
                                        Niehoff | IT-Consultant | Agile
                                        Software Factory  | Consulting</span><br
style="color:rgb(136,136,136);font-size:12.8000001907349px">
                                      <span
                                        style="color:rgb(136,136,136);font-size:12.8000001907349px">codecentric
                                        AG | Zeppelinstr 2 | 76185
                                        Karlsruhe | Deutschland</span><br
style="color:rgb(136,136,136);font-size:12.8000001907349px">
                                      <span
                                        style="color:rgb(136,136,136);font-size:12.8000001907349px">tel: </span><a
                                        moz-do-not-send="true"
                                        href="tel:%2B49%20%280%29%20721.9595-681"
                                        value="+497219595683"
                                        style="color:rgb(17,85,204);font-size:12.8000001907349px"
                                        target="_blank">+49 (0)
                                        721.9595-681</a><span
                                        style="color:rgb(136,136,136);font-size:12.8000001907349px"> |
                                        fax: </span><a
                                        moz-do-not-send="true"
                                        href="tel:%2B49%20%280%29%20721.9595-666"
                                        value="+497219595666"
                                        style="color:rgb(17,85,204);font-size:12.8000001907349px"
                                        target="_blank">+49 (0)
                                        721.9595-666</a><span
                                        style="color:rgb(136,136,136);font-size:12.8000001907349px"> |
                                        mobil: </span><a
                                        moz-do-not-send="true"
                                        href="tel:%2B49%20%280%29%20172.1702676"
                                        value="+4915201385470"
                                        style="color:rgb(17,85,204);font-size:12.8000001907349px"
                                        target="_blank">+49 (0)
                                        172.1702676</a><br
                                        style="color:rgb(136,136,136);font-size:12.8000001907349px">
                                      <font face="arial, helvetica,
                                        sans-serif"><span
                                          style="font-size:11pt;color:rgb(31,73,125)"><a
                                            moz-do-not-send="true"
                                            href="http://www.codecentric.de/"
                                            style="color:rgb(17,85,204)"
                                            target="_blank"><span
                                              style="font-size:10pt;color:black"><a
class="moz-txt-link-abbreviated" href="http://www.codecentric.de">www.codecentric.de</a></span></a></span><span
                                          style="font-size:10pt"> | </span><span
style="font-size:11pt;color:rgb(31,73,125)"><a moz-do-not-send="true"
                                            href="http://blog.codecentric.de/"
                                            style="color:rgb(17,85,204)"
                                            target="_blank"><span
                                              style="font-size:10pt;color:black">blog.codecentric.de</span></a></span><span
                                          style="font-size:10pt"> | </span><span
style="font-size:11pt;color:rgb(31,73,125)"><a moz-do-not-send="true"
                                            href="http://www.meettheexperts.de/"
                                            style="color:rgb(17,85,204)"
                                            target="_blank"><span
                                              style="font-size:10pt;color:black">www.meettheexperts.de</span></a></span><span
                                          style="font-size:10pt"> | </span><span
style="font-size:11pt;color:rgb(31,73,125)"><a moz-do-not-send="true"
                                            href="http://www.more4fi.de/"
                                            style="color:rgb(17,85,204)"
                                            target="_blank"><span
                                              style="font-size:10pt;color:black">www.more4fi.de</span></a></span><span
                                          style="font-size:10pt"> </span></font></div>
                                    <div dir="ltr"><font face="Verdana,
                                        sans-serif"><span
                                          style="font-size:13.3333330154419px"><br>
                                        </span></font><span
                                        style="color:rgb(136,136,136);font-size:12.8000001907349px">Sitz
                                        der Gesellschaft: Solingen | HRB
                                        25917| Amtsgericht Wuppertal</span>
                                      <div
                                        style="color:rgb(136,136,136);font-size:12.8000001907349px">Vorstand:
                                        Michael Hochgürtel . Mirko
                                        Novakovic . Rainer Vehns<br>
                                        Aufsichtsrat: Patric Fedlmeier
                                        (Vorsitzender) . Klaus Jäger .
                                        Jürgen Schütz</div>
                                      <div
                                        style="color:rgb(136,136,136);font-size:12.8000001907349px"><br>
                                      </div>
                                      <span
                                        style="color:rgb(136,136,136);font-size:12.8000001907349px">Diese
                                        E-Mail einschließlich evtl.
                                        beigefügter Dateien enthält
                                        vertrauliche und/oder rechtlich
                                        geschützte Informationen. Wenn
                                        Sie nicht der richtige Adressat
                                        sind oder diese E-Mail
                                        irrtümlich erhalten haben,
                                        informieren Sie bitte sofort den
                                        Absender und löschen Sie diese
                                        E-Mail und evtl. beigefügter
                                        Dateien umgehend. Das unerlaubte
                                        Kopieren, Nutzen oder Öffnen
                                        evtl. beigefügter Dateien sowie
                                        die unbefugte Weitergabe dieser
                                        E-Mail ist nicht gestattet</span><br>
                                    </div>
                                  </div>
                                </div>
                              </div>
                            </font></span></div>
                      </div>
                    </blockquote>
                  </div>
                  <br>
                </div>
              </div>
            </div>
          </blockquote>
        </div>
        <br>
        <br clear="all">
        <div><br>
        </div>
        -- <br>
        <div class="gmail_signature">
          <div dir="ltr">
            <div>
              <div dir="ltr"><span
                  style="color:rgb(136,136,136);font-size:12.8000001907349px">Matthias
                  Niehoff | IT-Consultant | Agile Software Factory  |
                  Consulting</span><br
                  style="color:rgb(136,136,136);font-size:12.8000001907349px">
                <span
                  style="color:rgb(136,136,136);font-size:12.8000001907349px">codecentric
                  AG | Zeppelinstr 2 | 76185 Karlsruhe | Deutschland</span><br
style="color:rgb(136,136,136);font-size:12.8000001907349px">
                <span
                  style="color:rgb(136,136,136);font-size:12.8000001907349px">tel: </span><a
                  moz-do-not-send="true"
                  href="tel:%2B49%20%280%29%20721.9595-681"
                  value="+497219595683"
                  style="color:rgb(17,85,204);font-size:12.8000001907349px"
                  target="_blank">+49 (0) 721.9595-681</a><span
                  style="color:rgb(136,136,136);font-size:12.8000001907349px"> |
                  fax: </span><a moz-do-not-send="true"
                  href="tel:%2B49%20%280%29%20721.9595-666"
                  value="+497219595666"
                  style="color:rgb(17,85,204);font-size:12.8000001907349px"
                  target="_blank">+49 (0) 721.9595-666</a><span
                  style="color:rgb(136,136,136);font-size:12.8000001907349px"> |
                  mobil: </span><a moz-do-not-send="true"
                  href="tel:%2B49%20%280%29%20172.1702676"
                  value="+4915201385470"
                  style="color:rgb(17,85,204);font-size:12.8000001907349px"
                  target="_blank">+49 (0) 172.1702676</a><br
                  style="color:rgb(136,136,136);font-size:12.8000001907349px">
                <font face="arial, helvetica, sans-serif"><span
                    style="font-size:11pt;color:rgb(31,73,125)"><a
                      moz-do-not-send="true"
                      href="http://www.codecentric.de/"
                      style="color:rgb(17,85,204)" target="_blank"><span
                        style="font-size:10pt;color:black"><a class="moz-txt-link-abbreviated"
href="http://www.codecentric.de">www.codecentric.de</a></span></a></span><span
                    style="font-size:10pt"> | </span><span
                    style="font-size:11pt;color:rgb(31,73,125)"><a
                      moz-do-not-send="true"
                      href="http://blog.codecentric.de/"
                      style="color:rgb(17,85,204)" target="_blank"><span
                        style="font-size:10pt;color:black">blog.codecentric.de</span></a></span><span
                    style="font-size:10pt"> | </span><span
                    style="font-size:11pt;color:rgb(31,73,125)"><a
                      moz-do-not-send="true"
                      href="http://www.meettheexperts.de/"
                      style="color:rgb(17,85,204)" target="_blank"><span
                        style="font-size:10pt;color:black"><a class="moz-txt-link-abbreviated"
href="http://www.meettheexperts.de">www.meettheexperts.de</a></span></a></span><span
                    style="font-size:10pt"> | </span><span
                    style="font-size:11pt;color:rgb(31,73,125)"><a
                      moz-do-not-send="true"
                      href="http://www.more4fi.de/"
                      style="color:rgb(17,85,204)" target="_blank"><span
                        style="font-size:10pt;color:black"><a class="moz-txt-link-abbreviated"
href="http://www.more4fi.de">www.more4fi.de</a></span></a></span><span
                    style="font-size:10pt"> </span></font></div>
              <div dir="ltr"><font face="Verdana, sans-serif"><span
                    style="font-size:13.3333330154419px"><br>
                  </span></font><span
                  style="color:rgb(136,136,136);font-size:12.8000001907349px">Sitz
                  der Gesellschaft: Solingen | HRB 25917| Amtsgericht
                  Wuppertal</span>
                <div
                  style="color:rgb(136,136,136);font-size:12.8000001907349px">Vorstand:
                  Michael Hochgürtel . Mirko Novakovic . Rainer Vehns<br>
                  Aufsichtsrat: Patric Fedlmeier (Vorsitzender) . Klaus
                  Jäger . Jürgen Schütz</div>
                <div
                  style="color:rgb(136,136,136);font-size:12.8000001907349px"><br>
                </div>
                <span
                  style="color:rgb(136,136,136);font-size:12.8000001907349px">Diese
                  E-Mail einschließlich evtl. beigefügter Dateien
                  enthält vertrauliche und/oder rechtlich geschützte
                  Informationen. Wenn Sie nicht der richtige Adressat
                  sind oder diese E-Mail irrtümlich erhalten haben,
                  informieren Sie bitte sofort den Absender und löschen
                  Sie diese E-Mail und evtl. beigefügter Dateien
                  umgehend. Das unerlaubte Kopieren, Nutzen oder Öffnen
                  evtl. beigefügter Dateien sowie die unbefugte
                  Weitergabe dieser E-Mail ist nicht gestattet</span><br>
              </div>
            </div>
          </div>
        </div>
      </div>
    </blockquote>
    <br>
  </body>
</html>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message