Return-Path: X-Original-To: apmail-spark-commits-archive@minotaur.apache.org Delivered-To: apmail-spark-commits-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 7C94D109B9 for ; Sat, 11 Jan 2014 00:26:22 +0000 (UTC) Received: (qmail 75369 invoked by uid 500); 11 Jan 2014 00:26:22 -0000 Delivered-To: apmail-spark-commits-archive@spark.apache.org Received: (qmail 75345 invoked by uid 500); 11 Jan 2014 00:26:22 -0000 Mailing-List: contact commits-help@spark.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@spark.incubator.apache.org Delivered-To: mailing list commits@spark.incubator.apache.org Received: (qmail 75338 invoked by uid 99); 11 Jan 2014 00:26:22 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jan 2014 00:26:22 +0000 X-ASF-Spam-Status: No, hits=-2000.1 required=5.0 tests=ALL_TRUSTED,RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [140.211.11.3] (HELO mail.apache.org) (140.211.11.3) by apache.org (qpsmtpd/0.29) with SMTP; Sat, 11 Jan 2014 00:26:17 +0000 Received: (qmail 73024 invoked by uid 99); 11 Jan 2014 00:25:56 -0000 Received: from tyr.zones.apache.org (HELO tyr.zones.apache.org) (140.211.11.114) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jan 2014 00:25:56 +0000 Received: by tyr.zones.apache.org (Postfix, from userid 65534) id C723C32AF2A; Sat, 11 Jan 2014 00:25:55 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: pwendell@apache.org To: commits@spark.incubator.apache.org Date: Sat, 11 Jan 2014 00:26:05 -0000 Message-Id: <6ed65c7807d743a7a600f07fa6c9d33b@git.apache.org> In-Reply-To: References: X-Mailer: ASF-Git Admin Mailer Subject: [11/20] git commit: Merge remote-tracking branch 'apache/master' into driver-test X-Virus-Checked: Checked by ClamAV on apache.org Merge remote-tracking branch 'apache/master' into driver-test Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/38d75e18 Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/38d75e18 Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/38d75e18 Branch: refs/heads/master Commit: 38d75e18fa48bd230e2ff9478e87274fb0cceb9f Parents: 4a5558c 4b074fa Author: Tathagata Das Authored: Thu Jan 9 19:31:36 2014 -0800 Committer: Tathagata Das Committed: Thu Jan 9 19:31:36 2014 -0800 ---------------------------------------------------------------------- conf/log4j.properties.template | 1 + .../org/apache/spark/log4j-defaults.properties | 1 + .../scala/org/apache/spark/Partitioner.scala | 7 ++- .../scala/org/apache/spark/SparkContext.scala | 48 ++++++++++------ .../main/scala/org/apache/spark/SparkEnv.scala | 2 +- .../org/apache/spark/SparkHadoopWriter.scala | 4 -- .../apache/spark/api/java/JavaDoubleRDD.scala | 12 +++- .../org/apache/spark/api/java/JavaPairRDD.scala | 6 ++ .../org/apache/spark/api/java/JavaRDD.scala | 6 ++ .../org/apache/spark/api/java/JavaRDDLike.scala | 6 ++ .../spark/api/java/JavaSparkContext.scala | 54 +++++++++++++++++- .../org/apache/spark/api/python/PythonRDD.scala | 4 +- .../apache/spark/broadcast/HttpBroadcast.scala | 4 +- .../spark/broadcast/TorrentBroadcast.scala | 2 +- .../org/apache/spark/deploy/master/Master.scala | 6 +- .../org/apache/spark/deploy/worker/Worker.scala | 2 +- .../org/apache/spark/io/CompressionCodec.scala | 2 +- .../spark/network/ConnectionManager.scala | 18 +++--- .../spark/network/netty/ShuffleCopier.scala | 2 +- .../org/apache/spark/rdd/CheckpointRDD.scala | 4 +- .../org/apache/spark/rdd/NewHadoopRDD.scala | 2 +- .../org/apache/spark/rdd/PairRDDFunctions.scala | 59 ++++++++++---------- .../main/scala/org/apache/spark/rdd/RDD.scala | 32 ++++++----- .../spark/scheduler/TaskResultGetter.scala | 2 +- .../spark/scheduler/TaskSchedulerImpl.scala | 8 +-- .../apache/spark/scheduler/TaskSetManager.scala | 8 +-- .../cluster/CoarseGrainedSchedulerBackend.scala | 5 +- .../cluster/SimrSchedulerBackend.scala | 2 +- .../mesos/CoarseMesosSchedulerBackend.scala | 2 +- .../cluster/mesos/MesosSchedulerBackend.scala | 2 +- .../spark/serializer/KryoSerializer.scala | 4 +- .../spark/storage/BlockFetcherIterator.scala | 2 +- .../org/apache/spark/storage/BlockManager.scala | 22 ++++---- .../spark/storage/BlockManagerMaster.scala | 4 +- .../spark/storage/BlockManagerMasterActor.scala | 7 ++- .../apache/spark/storage/DiskBlockManager.scala | 2 +- .../spark/storage/ShuffleBlockManager.scala | 4 +- .../spark/ui/jobs/JobProgressListener.scala | 2 +- .../scala/org/apache/spark/util/AkkaUtils.scala | 20 +++---- .../org/apache/spark/util/MetadataCleaner.scala | 2 +- .../org/apache/spark/util/XORShiftRandom.scala | 2 +- .../spark/scheduler/TaskSetManagerSuite.scala | 2 +- .../apache/spark/util/XORShiftRandomSuite.scala | 2 +- docs/configuration.md | 2 +- docs/running-on-yarn.md | 2 + make-distribution.sh | 2 +- pom.xml | 5 ++ sbt/sbt | 51 +++++++---------- .../streaming/dstream/NetworkInputDStream.scala | 4 +- .../apache/spark/streaming/scheduler/Job.scala | 2 +- .../streaming/scheduler/JobGenerator.scala | 2 +- .../streaming/scheduler/JobScheduler.scala | 2 +- .../org/apache/spark/deploy/yarn/Client.scala | 2 +- .../cluster/YarnClientSchedulerBackend.scala | 4 +- .../org/apache/spark/deploy/yarn/Client.scala | 2 +- 55 files changed, 282 insertions(+), 186 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/38d75e18/core/src/main/scala/org/apache/spark/SparkContext.scala ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/38d75e18/streaming/src/main/scala/org/apache/spark/streaming/scheduler/JobGenerator.scala ----------------------------------------------------------------------