spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tathagata Das <>
Subject Re: Spark streaming on YARN?
Date Wed, 22 Jan 2014 00:09:41 GMT
Actually, I just realised why it wasnt working. I just made some changes in
the examples and last night and they are in the 0.9-rc4 that is being voted
now. With those changes the streaming examples should work.

The issue was that the thread running the main() of the examples was not
waiting for the streaming context to continue processing. So the processing
was being cancelled as soon as the streaming context was started. This has
been fixed with the examples, and running the twitter examples should be
the same as running the SparkPi example (though you have to set the
twitter4j.oauth.* properties for the twitter example to work).


On Tue, Jan 21, 2014 at 12:13 PM, Mike Percy <> wrote:

> Just FYI I got it working in standalone mode with no probs, so thanks for
> the help Tathagata. I was not able to get it working on YARN, I gave up.
> Thanks,
> Mike
> On Fri, Jan 10, 2014 at 6:17 PM, Tathagata Das <
>> wrote:
>> Let me know if you have any trouble getting it running on the standalone
>> mode. That will be easier for us to help out with.
>> TD
>> On Fri, Jan 10, 2014 at 5:47 PM, Mike Percy <> wrote:
>>> On Fri, Jan 10, 2014 at 6:35 AM, Neal Sidhwaney <>wrote:
>>>> Are you setting the SPARK_JAVA_OPTS environment variable? Under 0.8.1,
>>>> we had to set that to -Dlog4j.configuration=<path to logging
>>>> properties file>  to get STDOUT for workers.  This was in standalone
>>>> mode, though not with YARN, so YMMV.
>>>>  Neal
>>> Thanks for the idea Neal. The docs say that just putting it into conf/
>>> should do the trick, but apparently that's not the case. I tried setting
>>> that and it just stopped printing on the shell and had no effect on the
>>> deployed container.
>>> Too much struggling with this, I'm going to work on getting this running
>>> in standalone mode.
>>> Mike

View raw message