From dev-return-23742-archive-asf-public=cust-asf.ponee.io@spark.apache.org Wed Jan 10 04:24:57 2018 Return-Path: X-Original-To: archive-asf-public@eu.ponee.io Delivered-To: archive-asf-public@eu.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by mx-eu-01.ponee.io (Postfix) with ESMTP id C21A4180718 for ; Wed, 10 Jan 2018 04:24:57 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id B1973160C3F; Wed, 10 Jan 2018 03:24:57 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 314CC160C17 for ; Wed, 10 Jan 2018 04:24:56 +0100 (CET) Received: (qmail 11809 invoked by uid 500); 10 Jan 2018 03:24:54 -0000 Mailing-List: contact dev-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list dev@spark.apache.org Received: (qmail 11798 invoked by uid 99); 10 Jan 2018 03:24:53 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 10 Jan 2018 03:24:53 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 726741807A9 for ; Wed, 10 Jan 2018 03:24:53 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.3 X-Spam-Level: * X-Spam-Status: No, score=1.3 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=2, MIME_QP_LONG_LINE=0.001, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (1024-bit key) header.d=palantirtech.onmicrosoft.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id 2G2PiETf5FAA for ; Wed, 10 Jan 2018 03:24:50 +0000 (UTC) Received: from mx0a-00153501.pphosted.com (mx0a-00153501.pphosted.com [67.231.148.48]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 7458C5F250 for ; Wed, 10 Jan 2018 03:24:49 +0000 (UTC) Received: from pps.filterd (m0131697.ppops.net [127.0.0.1]) by mx0a-00153501.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id w0A3OFsF030334; Tue, 9 Jan 2018 19:24:44 -0800 Authentication-Results: palantir.com; spf=pass smtp.mailfrom=mcheah@palantir.com Received: from mxw1.palantir.com (mxw1.palantir.com [66.70.54.21]) by mx0a-00153501.pphosted.com with ESMTP id 2fda8wg1bw-1 (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NOT); Tue, 09 Jan 2018 19:24:44 -0800 Received: from EXDR01-WEST.YOJOE.local (10.160.10.135) by ex02-west.YOJOE.local (10.160.10.131) with Microsoft SMTP Server (TLS) id 14.3.319.2; Tue, 9 Jan 2018 19:24:43 -0800 Received: from NAM02-SN1-obe.outbound.protection.outlook.com (216.32.180.24) by mxw3.palantir.com (10.160.10.135) with Microsoft SMTP Server (TLS) id 14.3.319.2; Tue, 9 Jan 2018 19:24:43 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=palantirtech.onmicrosoft.com; s=selector1-palantir-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=3BpThl4nCJynkvKPpcx3JgnGgJXVsuBUbndB0JEB3tw=; b=ppKL3KP51LjTanF0/mdLhJFajBJKldgHA/5El3beLqygtHIPsHU5T/XiyEbsaVBKrmrodUUY0D+9JZne74zWnTR/ExcFTQ/OLDWcY12Ug15af3akwbcDPVjazp85B+KE7PwBdU8BOSKixrxpVoAZS8sG8xccQLvioeIawMmKbJw= Received: from MWHPR18MB1503.namprd18.prod.outlook.com (10.173.243.21) by MWHPR18MB1502.namprd18.prod.outlook.com (10.173.243.20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.386.5; Wed, 10 Jan 2018 03:24:40 +0000 Received: from MWHPR18MB1503.namprd18.prod.outlook.com ([10.173.243.21]) by MWHPR18MB1503.namprd18.prod.outlook.com ([10.173.243.21]) with mapi id 15.20.0386.008; Wed, 10 Jan 2018 03:24:40 +0000 From: Matt Cheah To: Yinan Li , Nicholas Chammas CC: Anirudh Ramanathan , Marcelo Vanzin , Kimoon Kim , dev Subject: Re: Kubernetes: why use init containers? Thread-Topic: Kubernetes: why use init containers? Thread-Index: AQHTibflijP5rL00M0eMTN92xlJL3KNsYSmAgAAN4YD//3yCgA== Date: Wed, 10 Jan 2018 03:24:40 +0000 Message-ID: <66A44EC8-B80A-4756-BA5F-FA1D3731BCCF@palantir.com> References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: x-originating-ip: [206.188.26.36] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1;MWHPR18MB1502;7:TWL/r6hZztQza2iqg1TxWp4IBMM4gGHuCzOkGOY0fLvd7k7TWqtode7hT1wTW6NiWc996MGEEYcdFT+gf8OKO26l2GPvdWbhzukWm8e3F5PGaxAL2BAyG4n4eMS636h4iqwY7bQkjG85aGnHKAxUkpwztqlElAB8Bkc8ERep/lw3s90w/Z8JA3wIk2Rz5EFruIc3H1xlJ0CX5eVxS/xYyfgG3OnvuV6e00oELFmA83fguk+D824ZmtiBORjGdy/G x-ms-exchange-antispam-srfa-diagnostics: SSOS; x-ms-office365-filtering-correlation-id: 74fd4209-32f8-4350-64dc-08d557d9af72 x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(4534020)(4602075)(4627115)(201703031133081)(201702281549075)(5600026)(4604075)(3008032)(2017052603307)(7153060)(49563074)(7193020);SRVR:MWHPR18MB1502; x-ms-traffictypediagnostic: MWHPR18MB1502: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(28532068793085)(158342451672863)(278428928389397)(166708455590820)(85827821059158)(211936372134217)(21748063052155)(17755550239193)(10436049006162); x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(102415395)(6040470)(2401047)(5005006)(8121501046)(93006095)(93001095)(10201501046)(3002001)(3231023)(944501075)(6041268)(20161123560045)(20161123558120)(20161123562045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123564045)(6072148)(201708071742011);SRVR:MWHPR18MB1502;BCL:0;PCL:0;RULEID:(100000803101)(100110400095);SRVR:MWHPR18MB1502; x-forefront-prvs: 0548586081 x-forefront-antispam-report: SFV:NSPM;SFS:(10009020)(376002)(366004)(396003)(39380400002)(39850400004)(346002)(252514010)(24454002)(199004)(189003)(316002)(93886005)(606006)(77096006)(966005)(99936001)(102836004)(6512007)(53546011)(6486002)(6506007)(6306002)(6436002)(86362001)(54896002)(59450400001)(3846002)(236005)(478600001)(229853002)(54906003)(82746002)(110136005)(14454004)(36756003)(6246003)(2906002)(6116002)(83716003)(53936002)(97736004)(39060400002)(4326008)(25786009)(66066001)(8936002)(105586002)(8676002)(33656002)(81166006)(106356001)(3660700001)(5660300001)(99286004)(76176011)(3280700002)(2950100002)(2900100001)(7736002)(68736007)(81156014);DIR:OUT;SFP:1101;SCL:1;SRVR:MWHPR18MB1502;H:MWHPR18MB1503.namprd18.prod.outlook.com;FPR:;SPF:None;PTR:InfoNoRecords;A:1;MX:1;LANG:en; received-spf: None (protection.outlook.com: palantir.com does not designate permitted sender hosts) x-microsoft-antispam-message-info: pjnG39S+zYok4CebniUUYW2Soe0iqvIwc+/mfJTFx3Qs47vm/fI9Yosx0CqF//mTkUKVhSrUYcLv2uEi3y3xFQ== spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha1; boundary="B_3598370680_2000185397" MIME-Version: 1.0 X-MS-Exchange-CrossTenant-Network-Message-Id: 74fd4209-32f8-4350-64dc-08d557d9af72 X-MS-Exchange-CrossTenant-originalarrivaltime: 10 Jan 2018 03:24:40.4572 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 76463010-5dd7-40c7-b509-7ce28ba39430 X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR18MB1502 X-OriginatorOrg: palantir.com X-Proofpoint-SPF-Result: pass X-Proofpoint-SPF-Record: v=spf1 include:spf-00153501.pphosted.com include:fspf0.palantir.com ~all X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2018-01-10_02:,, signatures=0 --B_3598370680_2000185397 Content-type: multipart/alternative; boundary="B_3598370679_2002387124" --B_3598370679_2002387124 Content-type: text/plain; charset="UTF-8" Content-transfer-encoding: quoted-printable A few reasons to prefer init-containers come to mind: =20 Firstly, if we used spark-submit from within the driver container, the exec= utors wouldn=E2=80=99t receive the jars on their class loader until after the exec= utor starts because the executor has to launch first before localizing resou= rces. It is certainly possible to make the class loader work with the user=E2=80= =99s jars here, as is the case with all the client mode implementations, but, = it seems cleaner to have the classpath include the user=E2=80=99s jars at executor= launch time instead of needing to reason about the classloading order. =20 We can also consider the idiomatic approach from the perspective of Kuberne= tes. Yinan touched on this already, but init-containers are traditionally me= ant to prepare the environment for the application that is to be run, which = is exactly what we do here. This also makes it such that the localization pr= ocess can be completely decoupled from the execution of the application itse= lf. We can then for example detect the errors that happen on the resource lo= calization layer, say when an HDFS cluster is down, before the application i= tself launches. The failure at the init-container stage is explicitly noted = via the Kubernetes pod status API. =20 Finally, running spark-submit from the container would make the SparkSubmit= code inadvertently allow running client mode Kubernetes applications as wel= l. We=E2=80=99re not quite ready to support that. Even if we were, it=E2=80=99s not enti= rely intuitive for the cluster mode code path to depend on the client mode c= ode path. This isn=E2=80=99t entirely without precedent though, as Mesos has a sim= ilar dependency. =20 Essentially the semantics seem neater and the contract is very explicit whe= n using an init-container, even though the code does end up being more compl= ex. =20 From: Yinan Li Date: Tuesday, January 9, 2018 at 7:16 PM To: Nicholas Chammas Cc: Anirudh Ramanathan , Marcelo Vanzin , Matt Cheah , Kimoon Kim , dev Subject: Re: Kubernetes: why use init containers? =20 The init-container is required for use with the resource staging server (ht= tps://github.com/apache-spark-on-k8s/userdocs/blob/master/src/jekyll/running= -on-kubernetes.md#resource-staging-server[github.com]). The resource staging= server (RSS) is a spark-on-k8s component running in a Kubernetes cluster fo= r staging submission client local dependencies to Spark pods. The init-conta= iner is responsible for downloading the dependencies from the RSS. We haven'= t upstream the RSS code yet, but this is a value add component for Spark on = K8s as a way for users to use submission local dependencies without resortin= g to other mechanisms that are not immediately available on most Kubernetes = clusters, e.g., HDFS. We do plan to upstream it in the 2.4 timeframe. Additi= onally, the init-container is a Kubernetes native way of making sure that th= e dependencies are localized before the main driver/executor containers are = started. IMO, this guarantee is positive to have and it helps achieve separa= tion of concerns. So IMO, I think the init-container is a valuable component= and should be kept. =20 On Tue, Jan 9, 2018 at 6:25 PM, Nicholas Chammas wrote: I=E2=80=99d like to point out the output of =E2=80=9Cgit show =E2=80=94stat=E2=80=9D for that diff: 29 files changed, 130 insertions(+), 1560 deletions(-) +1 for that and generally for the idea of leveraging spark-submit. You can argue that executors downloading from external servers would be faster than downloading from the driver, but I=E2=80=99m not sure I=E2=80=99d agree - it can go both ways. On a tangentially related note, one of the main reasons spark-ec2[github.co= m] is so slow to launch clusters is that it distributes files like the Spark= binaries to all the workers via the master. Because of that, the launch tim= e scaled with the number of workers requested[issues.apache.org]. When I wrote Flintrock[github.com], I got a large improvement in launch tim= e over spark-ec2 simply by having all the workers download the installation = files in parallel from an external host (typically S3 or an Apache mirror). = And launch time became largely independent of the cluster size. That may or may not say anything about the driver distributing application = files vs. having init containers do it in parallel, but I=E2=80=99d be curious to = hear more. Nick =E2=80=8B =20 On Tue, Jan 9, 2018 at 9:08 PM Anirudh Ramanathan wrote: We were running a change in our fork which was similar to this at one point= early on. My biggest concerns off the top of my head with this change would= be localization performance with large numbers of executors, and what we lo= se in terms of separation of concerns. Init containers are a standard constr= uct in k8s for resource localization. Also how this approach affects the HDF= S work would be interesting. =20 =20 +matt +kimoon Still thinking about the potential trade offs here. Adding Matt and Kimoon = who would remember more about our reasoning at the time.=20 =20 =20 On Jan 9, 2018 5:22 PM, "Marcelo Vanzin" wrote: Hello, Me again. I was playing some more with the kubernetes backend and the whole init container thing seemed unnecessary to me. Currently it's used to download remote jars and files, mount the volume into the driver / executor, and place those jars in the classpath / move the files to the working directory. This is all stuff that spark-submit already does without needing extra help. So I spent some time hacking stuff and removing the init container code, and launching the driver inside kubernetes using spark-submit (similar to how standalone and mesos cluster mode works): https://github.com/vanzin/spark/commit/k8s-no-init[github.com] I'd like to point out the output of "git show --stat" for that diff: 29 files changed, 130 insertions(+), 1560 deletions(-) You get massive code reuse by simply using spark-submit. The remote dependencies are downloaded in the driver, and the driver does the job of service them to executors. So I guess my question is: is there any advantage in using an init containe= r? The current init container code can download stuff in parallel, but that's an easy improvement to make in spark-submit and that would benefit everybody. You can argue that executors downloading from external servers would be faster than downloading from the driver, but I'm not sure I'd agree - it can go both ways. Also the same idea could probably be applied to starting executors; Mesos starts executors using "spark-class" already, so doing that would both improve code sharing and potentially simplify some code in the k8s backend. -- Marcelo --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscribe@spark.apache.org =20 --B_3598370679_2002387124 Content-type: text/html; charset="UTF-8" Content-transfer-encoding: quoted-printable

A few reasons to prefer init-contain= ers come to mind:

 

Firstly, if we used spark-submit from within the driver conta= iner, the executors wouldn=E2=80=99t receive the jars on their class loader until = after the executor starts because the executor has to launch first before lo= calizing resources. It is certainly possible to make the class loader work w= ith the user=E2=80=99s jars here, as is the case with all the client mode implemen= tations, but, it seems cleaner to have the classpath include the user=E2=80=99s ja= rs at executor launch time instead of needing to reason about the classloadi= ng order.

 

We can also consider the idiomatic approach from the perspective of K= ubernetes. Yinan touched on this already, but init-containers are traditiona= lly meant to prepare the environment for the application that is to be run, = which is exactly what we do here. This also makes it such that the localizat= ion process can be completely decoupled from the execution of the applicatio= n itself. We can then for example detect the errors that happen on the resou= rce localization layer, say when an HDFS cluster is down, before the applica= tion itself launches. The failure at the init-container stage is explicitly = noted via the Kubernetes pod status API.

 

Finally, running spark-submit from the= container would make the SparkSubmit code inadvertently allow running clien= t mode Kubernetes applications as well. We=E2=80=99re not quite ready to support t= hat. Even if we were, it=E2=80=99s not entirely intuitive for the cluster mode cod= e path to depend on the client mode code path. This isn=E2=80=99t entirely without= precedent though, as Mesos has a similar dependency.

 

Essentially the semantics= seem neater and the contract is very explicit when using an init-container,= even though the code does end up being more complex.

 

From: Yinan Li <liyinan926@gmail.com>
Date: Tuesday, Ja= nuary 9, 2018 at 7:16 PM
To: Nicholas Chammas <nicholas.chammas= @gmail.com>
Cc: Anirudh Ramanathan <ramanathana@google.com.i= nvalid>, Marcelo Vanzin <vanzin@cloudera.com>, Matt Cheah <mchea= h@palantir.com>, Kimoon Kim <kimoon@pepperdata.com>, dev <dev@sp= ark.apache.org>
Subject: Re: Kubernetes: why use init container= s?

 

<= /div>

The init-container is required for use with the= resource staging server (https://github.com/= apache-spark-on-k8s/userdocs/blob/master/src/jekyll/running-on-kubernetes.md= #resource-staging-server[github.com]). The resource staging server (RSS)= is a spark-on-k8s component running in a Kubernetes cluster for staging sub= mission client local dependencies to Spark pods. The init-container is respo= nsible for downloading the dependencies from the RSS. We haven't upstream th= e RSS code yet, but this is a value add component for Spark on K8s as a way = for users to use submission local dependencies without resorting to other me= chanisms that are not immediately available on most Kubernetes clusters, e.g= ., HDFS. We do plan to upstream it in the 2.4 timeframe. Additionally, the i= nit-container is a Kubernetes native way of making sure that the dependencie= s are localized before the main driver/executor containers are started. IMO,= this guarantee is positive to have and it helps achieve separation of conce= rns. So IMO, I think the init-container is a valuable component and should b= e kept.

 

On Tue, Jan 9, 2018 at 6:25 PM, Nicholas Chammas <nicholas.chammas@g= mail.com> wrote:

I=E2=80=99d like to point out the= output of =E2=80=9Cgit show =E2=80=94stat=E2=80=9D for that diff:
29 files changed, 130 in= sertions(+), 1560 deletions(-)

+1 for that and generally for the idea of leveraging = spark-submit.

You can argue that executors downloading fr= om
external servers would be faster than downloading from the driver, but=
I=E2=80=99m not sure I=E2=80=99d agree - it can go both ways.

<= /blockquote>

On a tangentially related note= , one of the main reasons spark-ec2[github.com]= is so slow to launch clusters is that it distributes files like the Spark b= inaries to all the workers via the master. Because of that, the launch time = scaled with the number of workers requ= ested[issues.apache.org].

When I wrote Flintrock[github.com], I got a l= arge improvement in launch time over spark-ec2 simply by having all the work= ers download the installation files in parallel from an external host (typic= ally S3 or an Apache mirror). And launch time became largely independent of = the cluster size.

That may o= r may not say anything about the driver distributing application files vs. h= aving init containers do it in parallel, but I=E2=80=99d be curious to hear more.<= o:p>

Nick

=E2=80=8B

=

 

On Tue, Jan 9, 2018 at 9:08 PM Anirudh Ramanathan <ramanathana@google.com.invalid> wrot= e:

We were running a change in our fork which was simi= lar to this at one point early on. My biggest concerns off the top of my hea= d with this change would be localization performance with large numbers of e= xecutors, and what we lose in terms of separation of concerns. Init containe= rs are a standard construct in k8s for resource localization. Also how this = approach affects the HDFS work would be interesting.  <= /p>

 

+matt +kimoon

Still thin= king about the potential trade offs here. Adding Matt and Kimoon who would r= emember more about our reasoning at the time. 

 

 

On Jan 9, 2018 5:22 PM, "= Marcelo Vanzin" <vanzin@cloudera.com> wrote:

Hell= o,

Me again. I was playing some more with the kubernetes backend and = the
whole init container thing seemed unnecessary to me.

Currently= it's used to download remote jars and files, mount the
volume into the d= river / executor, and place those jars in the
classpath / move the files = to the working directory. This is all stuff
that spark-submit already doe= s without needing extra help.

So I spent some time hacking stuff and = removing the init container
code, and launching the driver inside kuberne= tes using spark-submit
(similar to how standalone and mesos cluster mode = works):

https://github.com/= vanzin/spark/commit/k8s-no-init[github.com]

I'd like to point out= the output of "git show --stat" for that diff:
 29 files = changed, 130 insertions(+), 1560 deletions(-)

You get massive code re= use by simply using spark-submit. The remote
dependencies are downloaded = in the driver, and the driver does the job
of service them to executors.<= br>
So I guess my question is: is there any advantage in using an init co= ntainer?

The current init container code can download stuff in parall= el, but
that's an easy improvement to make in spark-submit and that would=
benefit everybody. You can argue that executors downloading from
exte= rnal servers would be faster than downloading from the driver, but
I'm no= t sure I'd agree - it can go both ways.

Also the same idea could prob= ably be applied to starting executors;
Mesos starts executors using "= ;spark-class" already, so doing that
would both improve code sharing= and potentially simplify some code in
the k8s backend.

--
Marc= elo

-----------------------------------------------------------------= ----
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org

 

--B_3598370679_2002387124-- --B_3598370680_2000185397 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" MIIRlQYJKoZIhvcNAQcCoIIRhjCCEYICAQExCzAJBgUrDgMCGgUAMAsGCSqGSIb3DQEHAaCC D14wggVNMIIENaADAgECAhACF2X8ZyyKwIiphAHAVrY6MA0GCSqGSIb3DQEBCwUAMGUxCzAJ BgNVBAYTAlVTMRUwEwYDVQQKEwxEaWdpQ2VydCBJbmMxGTAXBgNVBAsTEHd3dy5kaWdpY2Vy dC5jb20xJDAiBgNVBAMTG0RpZ2lDZXJ0IFNIQTIgQXNzdXJlZCBJRCBDQTAeFw0xNzA3MjUw MDAwMDBaFw0yMDA3MjQxMjAwMDBaMHAxCzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpDYWxpZm9y bmlhMRIwEAYDVQQHEwlQYWxvIEFsdG8xIzAhBgNVBAoTGlBhbGFudGlyIFRlY2hub2xvZ2ll cyBJbmMuMRMwEQYDVQQDEwpNYXR0IENoZWFoMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB CgKCAQEAvg0TwcqjJxYL1AKm1f5LkxUSMu6K9YlZ4P524pT3fS43/7VS3Rn8SOiA7iJ+5oWz t+g2VtykWZ1mUsapWaKSY0LkARVWCP30VsEq5UobkRrWlLRSo4W4tXLy0qBZJ+M/svvDzHZS j42ftFKvOaEr4dNz7rqhiH++Nqj9oq2qNJ0bX+LvZneGTP6meBrVTp1B8FHClOZG83ZaeQMf ShlfAA+sHT2gr3NuTgeuqbbOLnXg4ItQ/fQWt9ga3s/U5dGYwU98LFhVXMDbqwesne2qdutU RBcLToSlZsqBPRtJa1goiY44eWWCEdJEJtr6Ypr7zZ6eiuaE9N+HLecVA0dHyQIDAQABo4IB 7DCCAegwHwYDVR0jBBgwFoAU5wIjgABP2Ne8lAvZP3Q5STI8inkwHQYDVR0OBBYEFEVcOLwl 2swb2Y1x1N4Mi8LVmmHnMAwGA1UdEwEB/wQCMAAwHgYDVR0RBBcwFYETbWNoZWFoQHBhbGFu dGlyLmNvbTAOBgNVHQ8BAf8EBAMCBaAwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwME MEMGA1UdIAQ8MDowOAYKYIZIAYb9bAQBAjAqMCgGCCsGAQUFBwIBFhxodHRwczovL3d3dy5k aWdpY2VydC5jb20vQ1BTMIGIBgNVHR8EgYAwfjA9oDugOYY3aHR0cDovL2NybDMuZGlnaWNl cnQuY29tL0RpZ2lDZXJ0U0hBMkFzc3VyZWRJRENBLWcyLmNybDA9oDugOYY3aHR0cDovL2Ny bDQuZGlnaWNlcnQuY29tL0RpZ2lDZXJ0U0hBMkFzc3VyZWRJRENBLWcyLmNybDB5BggrBgEF BQcBAQRtMGswJAYIKwYBBQUHMAGGGGh0dHA6Ly9vY3NwLmRpZ2ljZXJ0LmNvbTBDBggrBgEF BQcwAoY3aHR0cDovL2NhY2VydHMuZGlnaWNlcnQuY29tL0RpZ2lDZXJ0U0hBMkFzc3VyZWRJ RENBLmNydDANBgkqhkiG9w0BAQsFAAOCAQEAC1ZgZ7Llh8sFV1Nrsla695/ZnYuOgvBf/z3Q QBEwLkUny08fSDTseubp70MuB1xROFIPgrwPMGJ8OIiqDV4Y9nHeEvm+afBydwMuXwuWSxPG YQ4/oIXe3jWX1SlNqLX7Q+CquGp1ZVG/AtOOLX+32xQN2fNPwbYJdu5Wd0F4j9axCOFWa5KC 2DGN0sza4EQI8XlFszWIWLgVdyzUMfJnHaL5RoNX8/+GJZXg/+JySt14XTQDufYGY4NwNIKY 9XpxZlyVsUegoNfXOZ/kE5Qs6e+LMpSNNhCTDoHU2AB1SJODpprmMMWf6v6P28s0d8+Y4G7G +OUK8+ABEI87l9eI0DCCBk4wggU2oAMCAQICEASueWBmZpAaucV/pmxb3M0wDQYJKoZIhvcN AQELBQAwZTELMAkGA1UEBhMCVVMxFTATBgNVBAoTDERpZ2lDZXJ0IEluYzEZMBcGA1UECxMQ d3d3LmRpZ2ljZXJ0LmNvbTEkMCIGA1UEAxMbRGlnaUNlcnQgQXNzdXJlZCBJRCBSb290IENB MB4XDTEzMTEwNTEyMDAwMFoXDTI4MTEwNTEyMDAwMFowZTELMAkGA1UEBhMCVVMxFTATBgNV BAoTDERpZ2lDZXJ0IEluYzEZMBcGA1UECxMQd3d3LmRpZ2ljZXJ0LmNvbTEkMCIGA1UEAxMb RGlnaUNlcnQgU0hBMiBBc3N1cmVkIElEIENBMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB CgKCAQEA3PgRIz9qte/AJ3kbLQWHohBDMd8O1BUbT3ekIs4+jHDwvgeO3ScqvAEdtiwKyt1p WB9B7WoFH9pjeFkeIiwr+Lp+yTU7VvEffEJ+JbAjGcZFONc9RPkgfGCuHLBaGAS+jzv3qfCU mqYMY0m2QRdTQDK9T+ZQelAfJUXo8Ymvzf9e/1Dz8BcR/73FifW9YrnY+45FBIVtmc3FSE39 JqsCNkXqNtdfauIagkEK3OnZ9ZEXjsYhrTg8E+Yef2ac1U3ZRtr2z1KnfTskw7TBUTXGm+vU 737kewPhRL16CzfgT8uCig1xGOSm4IksG/OyczzBsJKeGH29q33FfQihLMKfcwIDAQABo4IC +DCCAvQwEgYDVR0TAQH/BAgwBgEB/wIBADAOBgNVHQ8BAf8EBAMCAYYwNAYIKwYBBQUHAQEE KDAmMCQGCCsGAQUFBzABhhhodHRwOi8vb2NzcC5kaWdpY2VydC5jb20wgYEGA1UdHwR6MHgw OqA4oDaGNGh0dHA6Ly9jcmw0LmRpZ2ljZXJ0LmNvbS9EaWdpQ2VydEFzc3VyZWRJRFJvb3RD QS5jcmwwOqA4oDaGNGh0dHA6Ly9jcmwzLmRpZ2ljZXJ0LmNvbS9EaWdpQ2VydEFzc3VyZWRJ RFJvb3RDQS5jcmwwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMEMIIBswYDVR0gBIIB qjCCAaYwggGiBgpghkgBhv1sAAIEMIIBkjAoBggrBgEFBQcCARYcaHR0cHM6Ly93d3cuZGln aWNlcnQuY29tL0NQUzCCAWQGCCsGAQUFBwICMIIBVh6CAVIAQQBuAHkAIAB1AHMAZQAgAG8A ZgAgAHQAaABpAHMAIABDAGUAcgB0AGkAZgBpAGMAYQB0AGUAIABjAG8AbgBzAHQAaQB0AHUA dABlAHMAIABhAGMAYwBlAHAAdABhAG4AYwBlACAAbwBmACAAdABoAGUAIABEAGkAZwBpAEMA ZQByAHQAIABDAFAALwBDAFAAUwAgAGEAbgBkACAAdABoAGUAIABSAGUAbAB5AGkAbgBnACAA UABhAHIAdAB5ACAAQQBnAHIAZQBlAG0AZQBuAHQAIAB3AGgAaQBjAGgAIABsAGkAbQBpAHQA IABsAGkAYQBiAGkAbABpAHQAeQAgAGEAbgBkACAAYQByAGUAIABpAG4AYwBvAHIAcABvAHIA YQB0AGUAZAAgAGgAZQByAGUAaQBuACAAYgB5ACAAcgBlAGYAZQByAGUAbgBjAGUALjAdBgNV HQ4EFgQU5wIjgABP2Ne8lAvZP3Q5STI8inkwHwYDVR0jBBgwFoAUReuir/SSy4IxLVGLp6ch nfNtyA8wDQYJKoZIhvcNAQELBQADggEBAE7UiSe5/R2Hd34PKAWQ8QovyTs+vZOckMav+pFR hzJUa+jKwXFRXJmOtfrgYhmZpgeafBMn2+UCooQS2RX2CkRXxDSPbXMfOtagAT3e44LkRWuy 6yX9gF4dOZC+W0L2zpFg4/mgVgxIEM4zaHvNk6vwastPWA+5e10bBIGepyLiV0kn7pKTCL5p CFMCOi5dyBn0UIBOAtmwXZG0k4f5lpaBVUCOZu2C2LsoX+1MYe0GWCgZUxFEvEcgKbIEbNiJ VJk7ddtneCweknjGVT1YEhEybr1DDE0023vGQtvsvqubYUwGkuOO3yEqUFcEwGCiNdUknmY3 CUnP1fhls+DibsIwggO3MIICn6ADAgECAhAM5+DlF9hG/o/lYPwb8DA5MA0GCSqGSIb3DQEB BQUAMGUxCzAJBgNVBAYTAlVTMRUwEwYDVQQKEwxEaWdpQ2VydCBJbmMxGTAXBgNVBAsTEHd3 dy5kaWdpY2VydC5jb20xJDAiBgNVBAMTG0RpZ2lDZXJ0IEFzc3VyZWQgSUQgUm9vdCBDQTAe Fw0wNjExMTAwMDAwMDBaFw0zMTExMTAwMDAwMDBaMGUxCzAJBgNVBAYTAlVTMRUwEwYDVQQK EwxEaWdpQ2VydCBJbmMxGTAXBgNVBAsTEHd3dy5kaWdpY2VydC5jb20xJDAiBgNVBAMTG0Rp Z2lDZXJ0IEFzc3VyZWQgSUQgUm9vdCBDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC ggEBAK0OFc7kQ4BcsYfzt2D5cRKlrtwmlIiq9M71IDkoWGAM+IDaqRWVMmE8tbEohIqK3J8K DIMXeo+QrIrneVNcMYQq9g+YMjZ2zN7dPKii72r7IfJSYd+fINcf4rHZ/hhk0hJbX/lYGDW8 R82hNvlrf9SwOD7BG8OMM9nYLxj+KA+zp4PWw25EwGE1lhb+WZyLdm3X8aJLDSv/C3LanmDQ jpA1xnhVhyChz+VtCshJfDGYM2wi6YfQMlqiuhOCEe05F52ZOnKh5vqk2dUXMXWuhX0irj8B Rob2KHnIsdrkVxfEfhwOsLSSplazvbKX7aqn8LfFqD+VFtD/oZbrCF8Yd08CAwEAAaNjMGEw DgYDVR0PAQH/BAQDAgGGMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFEXroq/0ksuCMS1R i6enIZ3zbcgPMB8GA1UdIwQYMBaAFEXroq/0ksuCMS1Ri6enIZ3zbcgPMA0GCSqGSIb3DQEB BQUAA4IBAQCiDrzf4u3w43JzemSUv/dyZtgy5EJ1Yq6H6/LV2d5Ws5/MzhQouQ2XYFwSTFjk 0z2DSUVYlzVpGqhH6lbGeasS2GeBhN9/CTyU5rgmLCC9PbMoifdf/yLil4Qf6WXvh+DfwWdJ s13rsgkq6ybteL59PyvztyY1bV+JAbZJW58BBZurPSXBzLZ/wvFvhsb6ZGjrgS2U60K3+owe 3WLxvlBnt2y98/Efaww2BxZ/N3ypW2168RJGYIPXJwS+S86XvsNnKmgR34DnDDNmvxMNFG7z fx9jEB76jRslbWyPpbdhAbHSoyahEHGdreLD+cOZUbcrBwjOLuZQsqf6CkUvovDyMYIB/zCC AfsCAQEweTBlMQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQL ExB3d3cuZGlnaWNlcnQuY29tMSQwIgYDVQQDExtEaWdpQ2VydCBTSEEyIEFzc3VyZWQgSUQg Q0ECEAIXZfxnLIrAiKmEAcBWtjowCQYFKw4DAhoFAKBdMCMGCSqGSIb3DQEJBDEWBBRRfGtp mulp2u7xUQap0mweFS3EmzAYBgkqhkiG9w0BCQMxCwYJKoZIhvcNAQcBMBwGCSqGSIb3DQEJ BTEPFw0xODAxMTAwMzI0NDBaMA0GCSqGSIb3DQEBAQUABIIBAJpJ9r7xjrs35xbjvG82J5rU 58rusC9+yktVZNOxKulgFdX0vqnwsfGhLMpOw/4yLja2ggLF6xdguHUNwsbDfvYotL4v1WYS C5Fwzx891HyyjAB1m35e8Pr8UCHfBjX2tq0JLKl1gYe75FMYvYp5UF4d2y0HCzAaKCIDv9ar VB/YEk1/i3bMoGbe401tfo7rbqvpfwrt7LqqNDF/zcsezwlT2AEtnxT7IVoaaKoyt2UjTSzD HgqCGKgMKkQSfp1ltd+85QIkEvs656jR0TS9OJwN9vb1sIWEkJQyMLtiVLKso82WD2vux0VS TGE1+nz602vv9XSIz+d4M1gCk4upmWU= --B_3598370680_2000185397--