spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nick Pentreath <nick.pentre...@gmail.com>
Subject Re: [VOTE] Apache Spark 2.2.0 (RC2)
Date Fri, 19 May 2017 18:53:02 GMT
All the outstanding ML QA doc and user guide items are done for 2.2 so from
that side we should be good to cut another RC :)

On Thu, 18 May 2017 at 00:18 Russell Spitzer <russell.spitzer@gmail.com>
wrote:

> Seeing an issue with the DataScanExec and some of our integration tests
> for the SCC. Running dataframe read and writes from the shell seems fine
> but the Redaction code seems to get a "None" when doing
> SparkSession.getActiveSession.get in our integration tests. I'm not sure
> why but i'll dig into this later if I get a chance.
>
> Example Failed Test
>
> https://github.com/datastax/spark-cassandra-connector/blob/v2.0.1/spark-cassandra-connector/src/it/scala/com/datastax/spark/connector/sql/CassandraSQLSpec.scala#L311
>
> ```[info]   org.apache.spark.SparkException: Job aborted due to stage
> failure: Task serialization failed: java.util.NoSuchElementException:
> None.get
> [info] java.util.NoSuchElementException: None.get
> [info] at scala.None$.get(Option.scala:347)
> [info] at scala.None$.get(Option.scala:345)
> [info] at org.apache.spark.sql.execution.DataSourceScanExec$class.org
> $apache$spark$sql$execution$DataSourceScanExec$$redact(DataSourceScanExec.scala:70)
> [info] at
> org.apache.spark.sql.execution.DataSourceScanExec$$anonfun$4.apply(DataSourceScanExec.scala:54)
> [info] at
> org.apache.spark.sql.execution.DataSourceScanExec$$anonfun$4.apply(DataSourceScanExec.scala:52)
> ```
>
> Again this only seems to repo in our IT suite so i'm not sure if this is a
> real issue.
>
>
> On Tue, May 16, 2017 at 1:40 PM Joseph Bradley <joseph@databricks.com>
> wrote:
>
>> All of the ML/Graph/SparkR QA blocker JIRAs have been resolved.  Thanks
>> everyone who helped out on those!
>>
>> We still have open ML/Graph/SparkR JIRAs targeted at 2.2, but they are
>> essentially all for documentation.
>>
>> Joseph
>>
>> On Thu, May 11, 2017 at 3:08 PM, Marcelo Vanzin <vanzin@cloudera.com>
>> wrote:
>>
>>> Since you'll be creating a new RC, I'd wait until SPARK-20666 is
>>> fixed, since the change that caused it is in branch-2.2. Probably a
>>> good idea to raise it to blocker at this point.
>>>
>>> On Thu, May 11, 2017 at 2:59 PM, Michael Armbrust
>>> <michael@databricks.com> wrote:
>>> > I'm going to -1 given the outstanding issues and lack of +1s.  I'll
>>> create
>>> > another RC once ML has had time to take care of the more critical
>>> problems.
>>> > In the meantime please keep testing this release!
>>> >
>>> > On Tue, May 9, 2017 at 2:00 AM, Kazuaki Ishizaki <ISHIZAKI@jp.ibm.com>
>>> > wrote:
>>> >>
>>> >> +1 (non-binding)
>>> >>
>>> >> I tested it on Ubuntu 16.04 and OpenJDK8 on ppc64le. All of the tests
>>> for
>>> >> core have passed.
>>> >>
>>> >> $ java -version
>>> >> openjdk version "1.8.0_111"
>>> >> OpenJDK Runtime Environment (build
>>> >> 1.8.0_111-8u111-b14-2ubuntu0.16.04.2-b14)
>>> >> OpenJDK 64-Bit Server VM (build 25.111-b14, mixed mode)
>>> >> $ build/mvn -DskipTests -Phive -Phive-thriftserver -Pyarn -Phadoop-2.7
>>> >> package install
>>> >> $ build/mvn -Phive -Phive-thriftserver -Pyarn -Phadoop-2.7 test -pl
>>> core
>>> >> ...
>>> >> Run completed in 15 minutes, 12 seconds.
>>> >> Total number of tests run: 1940
>>> >> Suites: completed 206, aborted 0
>>> >> Tests: succeeded 1940, failed 0, canceled 4, ignored 8, pending 0
>>> >> All tests passed.
>>> >> [INFO]
>>> >>
>>> ------------------------------------------------------------------------
>>> >> [INFO] BUILD SUCCESS
>>> >> [INFO]
>>> >>
>>> ------------------------------------------------------------------------
>>> >> [INFO] Total time: 16:51 min
>>> >> [INFO] Finished at: 2017-05-09T17:51:04+09:00
>>> >> [INFO] Final Memory: 53M/514M
>>> >> [INFO]
>>> >>
>>> ------------------------------------------------------------------------
>>> >> [WARNING] The requested profile "hive" could not be activated because
>>> it
>>> >> does not exist.
>>> >>
>>> >>
>>> >> Kazuaki Ishizaki,
>>> >>
>>> >>
>>> >>
>>> >> From:        Michael Armbrust <michael@databricks.com>
>>> >> To:        "dev@spark.apache.org" <dev@spark.apache.org>
>>> >> Date:        2017/05/05 02:08
>>> >> Subject:        [VOTE] Apache Spark 2.2.0 (RC2)
>>> >> ________________________________
>>> >>
>>> >>
>>> >>
>>> >> Please vote on releasing the following candidate as Apache Spark
>>> version
>>> >> 2.2.0. The vote is open until Tues, May 9th, 2017 at 12:00 PST and
>>> passes if
>>> >> a majority of at least 3 +1 PMC votes are cast.
>>> >>
>>> >> [ ] +1 Release this package as Apache Spark 2.2.0
>>> >> [ ] -1 Do not release this package because ...
>>> >>
>>> >>
>>> >> To learn more about Apache Spark, please see http://spark.apache.org/
>>> >>
>>> >> The tag to be voted on is v2.2.0-rc2
>>> >> (1d4017b44d5e6ad156abeaae6371747f111dd1f9)
>>> >>
>>> >> List of JIRA tickets resolved can be found with this filter.
>>> >>
>>> >> The release files, including signatures, digests, etc. can be found
>>> at:
>>> >> http://home.apache.org/~pwendell/spark-releases/spark-2.2.0-rc2-bin/
>>> >>
>>> >> Release artifacts are signed with the following key:
>>> >> https://people.apache.org/keys/committer/pwendell.asc
>>> >>
>>> >> The staging repository for this release can be found at:
>>> >>
>>> https://repository.apache.org/content/repositories/orgapachespark-1236/
>>> >>
>>> >> The documentation corresponding to this release can be found at:
>>> >>
>>> http://people.apache.org/~pwendell/spark-releases/spark-2.2.0-rc2-docs/
>>> >>
>>> >>
>>> >> FAQ
>>> >>
>>> >> How can I help test this release?
>>> >>
>>> >> If you are a Spark user, you can help us test this release by taking
>>> an
>>> >> existing Spark workload and running on this release candidate, then
>>> >> reporting any regressions.
>>> >>
>>> >> What should happen to JIRA tickets still targeting 2.2.0?
>>> >>
>>> >> Committers should look at those and triage. Extremely important bug
>>> fixes,
>>> >> documentation, and API tweaks that impact compatibility should be
>>> worked on
>>> >> immediately. Everything else please retarget to 2.3.0 or 2.2.1.
>>> >>
>>> >> But my bug isn't fixed!??!
>>> >>
>>> >> In order to make timely releases, we will typically not hold the
>>> release
>>> >> unless the bug in question is a regression from 2.1.1.
>>> >>
>>> >
>>>
>>>
>>>
>>> --
>>> Marcelo
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>
>>>
>>
>>
>> --
>>
>> Joseph Bradley
>>
>> Software Engineer - Machine Learning
>>
>> Databricks, Inc.
>>
>> [image: http://databricks.com] <http://databricks.com/>
>>
>

Mime
View raw message