incubator-s4-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gowtham S <seldomgowt...@gmail.com>
Subject Re: Problem in running a same PE in multiple nodes
Date Mon, 25 Feb 2013 07:24:34 GMT
Hi matthieu,

that was a sample design of the system. but i m trying to build a dynamic
design of system during the time of execution.. for eg if i give
the initial number of ProcessingPE as 10 means it has to map dynamically
from 10 -> 8 , 8 ->4 , 4 ->2 , 2 ->1. what ever may be the initial number
of Processing PE given the code will manipulate and send to corresponding
PE. but whether s4 supports this kind of dynamic mapping during the time of
execution. the sorting example which i gave was a sample example with 4 ->
2, 2 ->1 .so finally i m trying to achieve log base2 (n) levels of a same
PE(Processing PE)  given n at the Details Receiver PE stage. will s4 map
this kind of system design.




On Mon, Feb 25, 2013 at 2:53 AM, Matthieu Morel <mmorel@apache.org> wrote:

> Hi,
>
> I had a quick look and I wonder why you are not using different PE
> prototypes for the different layers in your system? That looks like a more
> adequate design, since I don't think you need to keep the state from layer
> 1 in layer 2 or layer 3 because instead you send aggregated info through
> messages.
>
> Regards,
>
> Matthieu
>
> On Feb 24, 2013, at 12:14 , Gowtham S wrote:
>
>
> this is the input file for the above application..
>
>
> On Sun, Feb 24, 2013 at 12:25 AM, Gowtham S <seldomgowtham@gmail.com>wrote:
>
>> hi matthieu,
>>
>> i have no problem in app. and also 3 nodes are processing the whole data
>> while one has stopped after some time..
>>
>> i have attached the sample project of our application. most of the
>> description regarding the application is given in the attachment. if
>> possible try checking it.. i ran it in a single machine with 4 nodes as
>> said in previous message.
>>
>>
>> i will describe what is actually happening ...
>>
>> i am having a record in which there are 18237 lines in which each line
>> has 4 numbers. i m just trying to sort four numbers in each line based on
>> the design of PE's as given in the document.
>>
>> same result one node stopped printing in the terminal while the other
>> three continues it work.
>>
>> see this after reading the document..
>>
>> after seeing the document u will see the PE graph... in that Processing
>> PE with Key ID "1" and Processing PE with Key ID "10" are running in the
>> same node..
>>
>>  ProcessingPE with Key Id "10" should receive events from ProcessingPE of
>> ID(0 and 1) but it mainly receives events from only PE with Key id "1" and
>> not from Key Id  "0".
>>
>> so i cant get alternate events from both PE's. but i should receive
>> events from both PE's one after the other as i  should sort 4 numbers in a
>> single line. so PE in one node is not sending events that much quick to the
>> PE in other nodes than to the PE in same node..
>>
>> so Processing PE with id "10" receives more events from PE with key ID
>> "0" which is in the same node. so what i have to do in order to get events
>> from both PE's with id "0" and "1" one after the other rather than majorly
>> from "1" and then finally from "1"(i.e for e.g if "10" should receive 100
>> from both "0" and "1".. first 100 is received from "1" which is at the same
>> node as "10" at that time only 2 or 3 received from "0". then the remaining
>> from "0" is received. but i should receive 1 event from "0 " and 1 from "1"
>> and so on ).
>>
>> "0" ,"1" - are Processing PE with Key ID "0" and "1".
>>
>>
>>
>>
>>
>>
>>
>> On Fri, Feb 22, 2013 at 7:03 PM, Matthieu Morel <mmorel@apache.org>wrote:
>>
>>> Thanks for the info,
>>>
>>> with the cluster status that you show - which look ok - is there any
>>> problem with the app?
>>>
>>> Also, I would again recommend to check whether all data is processed as
>>> expected, regardless of the location. Can you check that?
>>>
>>> Thanks,
>>>
>>> Matthieu
>>>
>>>
>>> On Feb 22, 2013, at 14:02 , Gowtham S wrote:
>>>
>>> sorry for the wrong thread and gimp format..
>>>
>>> here i have attached the .png format of the screenshot which shows that
>>> all the 4 nodes are active .
>>>
>>> and also i didnt modify the hash function. when ever i run it in 4
>>> nodes, it always map the 4 different keyed PE to four different nodes. i
>>> didnt manually  change the hash function ..
>>>
>>>
>>>
>>> On Thu, Feb 21, 2013 at 1:30 AM, Matthieu Morel <mmorel@apache.org>wrote:
>>>
>>>> Hi Jihyoun,
>>>>
>>>> PE graphs with cycles can certainly get a bit tricky sometimes, but
>>>> mainly because they can lead to deadlocks.
>>>>
>>>> I don't see how you'd lose events in the case you present if you don't
>>>> lose or reconfigure nodes. In the very specific case of sending an event
>>>> from PE instance a1 to the same PE instance a1, the event is simply put in
>>>> an in-memory blocking queue.
>>>>
>>>> It would be great to understand in which conditions you can see this
>>>> issue. Could you be more specific? (I can't reproduce that with simple
>>>> tests, but I don't have your own settings/code/environment).
>>>>
>>>> Thanks,
>>>>
>>>> Matthieu
>>>>
>>>> On Feb 20, 2013, at 03:04 , JiHyoun Park wrote:
>>>>
>>>> Dear Matthieu
>>>>
>>>> I also experienced the same problem with Gowtham.
>>>> Unlike him, I created a downstream to itself with only one key. (It was
>>>> like a flag to be turned on/off.)
>>>> I tested it and found sometimes one PE instance does not receive the
>>>> event.
>>>> I remember that it didn't depend on a specific node.
>>>>
>>>> At the time, I also wondered why.
>>>> Since it's a very simple logic, you can test it by yourself without
>>>> much efforts.
>>>> It would be really appreciated if you can tell us the solution or at
>>>> least the reason.
>>>>
>>>> Best Regards
>>>> Jihyoun
>>>>
>>>>
>>>>
>>>> On Tue, Feb 19, 2013 at 11:38 PM, Matthieu Morel <mmorel@apache.org>wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Can you provide more detail about a node not running correctly? what
>>>>> does that mean to you, that it does not receive messages? Maybe that's
>>>>> normal because of the distribution of keys. Or maybe you only have 3
active
>>>>> nodes in the cluster. (you can check that with the s4 status tool)
>>>>>
>>>>> The distribution of keys across nodes is a simple hash+mod , which you
>>>>> can override if needed.
>>>>>
>>>>> I would recommend to check whether all data is processed as expected,
>>>>> regardless of the location.
>>>>>
>>>>> Let us know,
>>>>>
>>>>> thanks,
>>>>>
>>>>> Matthieu
>>>>>
>>>>>
>>>>> On Feb 19, 2013, at 16:27 , Gowtham S wrote:
>>>>>
>>>>> Hi S4 community,
>>>>>
>>>>> I have been working with s4 piper for 3 months. now i am just trying
>>>>> to make a downstream from a PE to the same PE with different Key Id.
>>>>>
>>>>> i just have a scenario in which i would have an unique PE whose
>>>>> downstream is also to that PE  but each time the Key Id of the targeted
PE
>>>>> may vary. this continues until it satisfies a condition where it just
stops.
>>>>>
>>>>> i started implementing this in a single machine with 4 nodes, as i had
>>>>> 4 different keys for that unique PE(say 0,1,2,3). so now every node will
>>>>> have that PE clone with one individual key Id from the available Key
Ids.
>>>>> one PE may send some events to PE in other  nodes , until certain condition
>>>>> is reached where the process is stopped.
>>>>>
>>>>> now three nodes are running correctly except one node.. i wonder why
>>>>> .. because all the nodes have same computational part to execute.
>>>>>
>>>>> please help me as how s4 deals while assigning a single PE to
>>>>> different nodes with different key Id ..
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Feb 19, 2013 at 8:22 PM, Gowtham S <seldomgowtham@gmail.com>wrote:
>>>>>
>>>>>> Hi S4 community,
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>> <s4image.png>
>>>
>>>
>>>
>>
> <textfile.txt>
>
>
>

Mime
View raw message