spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shivaram Venkataraman <shiva...@eecs.berkeley.edu>
Subject Re: What is d3kbcqa49mib13.cloudfront.net ?
Date Wed, 13 Sep 2017 19:05:09 GMT
Mark, I agree with your point on the risks of using Cloudfront while
building Spark. I was only trying to provide background on when we
started using Cloudfront.

Personally, I don't have enough about context about the test case in
question (e.g. Why are we downloading Spark in a test case ?).

Thanks
Shivaram

On Wed, Sep 13, 2017 at 11:50 AM, Mark Hamstra <mark@clearstorydata.com> wrote:
> Yeah, but that discussion and use case is a bit different -- providing a
> different route to download the final released and approved artifacts that
> were built using only acceptable artifacts and sources vs. building and
> checking prior to release using something that is not from an Apache mirror.
> This new use case puts us in the position of approving spark artifacts that
> weren't built entirely from canonical resources located in presumably secure
> and monitored repositories. Incorporating something that is not completely
> trusted or approved into the process of building something that we are then
> going to approve as trusted is different from the prior use of cloudfront.
>
> On Wed, Sep 13, 2017 at 10:26 AM, Shivaram Venkataraman
> <shivaram@eecs.berkeley.edu> wrote:
>>
>> The bucket comes from Cloudfront, a CDN thats part of AWS. There was a
>> bunch of discussion about this back in 2013
>>
>> https://lists.apache.org/thread.html/9a72ff7ce913dd85a6b112b1b2de536dcda74b28b050f70646aba0ac@1380147885@%3Cdev.spark.apache.org%3E
>>
>> Shivaram
>>
>> On Wed, Sep 13, 2017 at 9:30 AM, Sean Owen <sowen@cloudera.com> wrote:
>> > Not a big deal, but Mark noticed that this test now downloads Spark
>> > artifacts from the same 'direct download' link available on the
>> > downloads
>> > page:
>> >
>> >
>> > https://github.com/apache/spark/blob/master/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala#L53
>> >
>> > https://d3kbcqa49mib13.cloudfront.net/spark-$version-bin-hadoop2.7.tgz
>> >
>> > I don't know of any particular problem with this, which is a parallel
>> > download option in addition to the Apache mirrors. It's also the
>> > default.
>> >
>> > Does anyone know what this bucket is and if there's a strong reason we
>> > can't
>> > just use mirrors?
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org


Mime
View raw message