incubator-s4-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matthieu Morel <>
Subject Re: Top_n_hashtags were not appearing. Correct way to stop cluster/adapter?
Date Mon, 12 Dec 2011 09:36:53 GMT

I'm answering on the mailing list, since 
this is the mailing list we are now using.

On 12/9/11 9:05 PM, scotavoosh wrote:
> While running the sample twittertopiccount app, I ran into the problem
> that the top_n_hashtags file was not appearing. First issue was
> configuring the app to pass through a proxy. That can be done by
> adding these lines of code to the connectAndRead method in
> TwitterFeedListener before opening the URLConnection:
> System.setProperty("https.proxyHost","");
> System.setProperty("https.proxyPort","8080");

I assume this was specific to your network configuration. But that will 
be useful to others with similar network configs, thanks!

> The problem still persisted. The log files showed that events were
> being dispatched but were not being consumed. Also in the log file was
> a line that said:
> 2011-12-08 15:07:29,544 io.s4.comm.file.StaticTaskManager INFO
> ( Process taken up by another process
> lockFile:/Users/user/Development/s4/build/s4-image/scripts/../s4-core/
> lock/s4s4node-0
> 2011-12-08 15:07:29,544 io.s4.comm.file.StaticTaskManager INFO
> ( processAvailable:false
> I found the solution was to remove the lock files in /Users/user/
> Development/s4/build/s4-image/scripts/../s4-core/lock/s4s4node-0
> These need to be removed after killing the running processes for the
> cluster, client-adapter, and feed listener.
> What is the correct way to stop the cluster and adapter so that these
> lock files are released?

There are currently 2 ways to control assignation of logical tasks to S4 
- through the file system, by creating "lock" files
- through Zookeeper, by using ephemeral nodes

S4 nodes delete owned lock files upon exit using the File.deleteOnExit 
method ( ). 
Unfortunately, this won't work well if you kill the S4 node process: 
lock files will remain.

Therefore you should either:
- clean-up lock files at start-up through a script or an initialization 
- use the Zookeeper task assignation mechanism ( )

By the way, future versions of S4 (from 0.5) will only use Zookeeper for 
task assignation.

Hope this helps,


View raw message