Mailing-List: contact user-help@spark.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@spark.apache.org
Received-SPF: pass (nike.apache.org: local policy includes SPF record at
 spf.trusted-forwarder.org)
Message-ID: <535AA18B.7020208@media.mit.edu>
Date: Fri, 25 Apr 2014 13:55:23 -0400
From: Yadid Ayzenberg <yadid@media.mit.edu>
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6;
 rv:24.0) Gecko/20100101 Thunderbird/24.4.0
MIME-Version: 1.0
To: user@spark.apache.org
Subject: Strange lookup behavior. Possible bug?
References: <1398448013435-4837.post@n3.nabble.com>
In-Reply-To: <1398448013435-4837.post@n3.nabble.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Hi All,

Im running a lookup on a JavaPairRDD<String, Tuple2>.
When running on local machine - the lookup is successfull. However, when 
running a standalone cluster with the exact same dataset - one of the 
tasks never ends (constantly in RUNNING status).
When viewing the worker log, it seems that the task has finished 
successfully:

14/04/25 13:40:38 INFO BlockManager: Found block rdd_2_0 locally
14/04/25 13:40:38 INFO Executor: Serialized size of result for 2 is 10896794
14/04/25 13:40:38 INFO Executor: Sending result for 2 directly to driver
14/04/25 13:40:38 INFO Executor: Finished task ID 2

But it seems the driver is not aware of this, and hangs indefinitely.

If I execute a count priot to the lookup - I get the correct number 
which suggests that the cluster is operating as expected.

The exact same scenario works with a different type of key (Tuple2): 
JavaPairRDD<Tuple2, Tuple2>.

Any ideas on how to debug this problem ?

Thanks,

Yadid