incubator-s4-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matthieu Morel <mmo...@apache.org>
Subject Re: Top_n_hashtags were not appearing. Correct way to stop cluster/adapter?
Date Mon, 12 Dec 2011 09:36:53 GMT
Hi,

I'm answering on the s4-user@incubator.apache.org mailing list, since 
this is the mailing list we are now using.

On 12/9/11 9:05 PM, scotavoosh wrote:
> While running the sample twittertopiccount app, I ran into the problem
> that the top_n_hashtags file was not appearing. First issue was
> configuring the app to pass through a proxy. That can be done by
> adding these lines of code to the connectAndRead method in
> TwitterFeedListener before opening the URLConnection:
>
> System.setProperty("https.proxyHost","your.proxy.com");
> System.setProperty("https.proxyPort","8080");

I assume this was specific to your network configuration. But that will 
be useful to others with similar network configs, thanks!

>
> The problem still persisted. The log files showed that events were
> being dispatched but were not being consumed. Also in the log file was
> a line that said:
>
> 2011-12-08 15:07:29,544 io.s4.comm.file.StaticTaskManager INFO
> (StaticTaskManager.java:185) Process taken up by another process
> lockFile:/Users/user/Development/s4/build/s4-image/scripts/../s4-core/
> lock/s4s4node-0
> 2011-12-08 15:07:29,544 io.s4.comm.file.StaticTaskManager INFO
> (StaticTaskManager.java:102) processAvailable:false
>
> I found the solution was to remove the lock files in /Users/user/
> Development/s4/build/s4-image/scripts/../s4-core/lock/s4s4node-0
>
> These need to be removed after killing the running processes for the
> cluster, client-adapter, and feed listener.
>
> What is the correct way to stop the cluster and adapter so that these
> lock files are released?

There are currently 2 ways to control assignation of logical tasks to S4 
nodes:
- through the file system, by creating "lock" files
- through Zookeeper, by using ephemeral nodes

S4 nodes delete owned lock files upon exit using the File.deleteOnExit 
method ( 
http://docs.oracle.com/javase/6/docs/api/java/io/File.html#deleteOnExit() ). 
Unfortunately, this won't work well if you kill the S4 node process: 
lock files will remain.

Therefore you should either:
- clean-up lock files at start-up through a script or an initialization 
routine
- use the Zookeeper task assignation mechanism ( 
http://docs.s4.io/tutorials/zookeeper.html )

By the way, future versions of S4 (from 0.5) will only use Zookeeper for 
task assignation.


Hope this helps,

Matthieu


Mime
View raw message