axis-c-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Samisa Abeysinghe <sam...@wso2.com>
Subject Re: Solution to Core Dump Issue Raised in User List [was: Re: Core dumps in Axis libraries]
Date Mon, 10 Mar 2008 12:01:40 GMT
Were you able to re-create the problem without passing -1 to accept?

Samisa...

Senaka Fernando wrote:
>> Senaka Fernando wrote:
>>     
>>> Hi Samisa,
>>>
>>> I fixed two bugs regarding stream handling inside
>>> simple_http_svr_conn.c.
>>> This fixes an issue of this nature.
>>>
>>> Say, we have two server threads that try to serve requests offered to
>>> the
>>> axis2 server. Now, according to what is intended, one must serve and the
>>> other must wait until it can serve. The previous implementation caused
>>> this to fail with a segfault. But, with the fix, it will rather not
>>> crash
>>> but, loop until it gets a chance to serve the request.
>>>
>>>       
>> Did you test before the fix and did it seg fault for you? Can you please
>> explain the scenario under which it seg faulted? If it is a problem, I
>> too should be able to re-create it.
>>
>> And how did you solve the problem? By adding a sleep?
>>     
>
> I made the accept(2) call fail. By passing a -1 to it. Once that failed,
> there were two places that segfaulted which I already fixed, few minutes
> ago. No, I didn't add a sleep(). I simply prevented the segfault by adding
> necessary is_null checks before calling methods. Then, the server just
> kept on spawning threads and freeing them as it can't serve the request.
> This is the expected behaviour in http_svr_thread.c I believe.
>
> In a real world scenario, a situation similar to a -1 passed to accept(2)
> will be temporal. So a respawned thread down the line should server the
> request without any problem.
>
> Passing -1 to accept was done by,
>
> Index: util/src/network_handler.c
> ===================================================================
> --- util/src/network_handler.c  (revision 634578)
> +++ util/src/network_handler.c  (working copy)
> @@ -221,7 +221,7 @@
>      AXIS2_ENV_CHECK(env, AXIS2_CRITICAL_FAILURE);
>
>      cli_len = sizeof(cli_addr);
> -    cli_socket = accept(svr_socket, (struct sockaddr *) &cli_addr,
> &cli_len);
> +    cli_socket = accept(-1, (struct sockaddr *) &cli_addr, &cli_len);
>      if (cli_socket < 0)
>          AXIS2_LOG_ERROR(env->log, AXIS2_LOG_SI,
>                          "[Axis2][network_handler] Socket accept \
> ---
>
> I'm not sure whether this is what really lead to the core dump, or was it
> something else.
>
> This was all that I could try on based on the less informative
> valgrind.log that was sent. :(...
>
> Also i did test 1000 back-to-back requests and they all were served
> without any issues.
>
> Regards,
> Senaka
>
>   
>> Samisa...
>>
>>     
>>> Could this solve the core dump issue?
>>>
>>> Regards,
>>> Senaka
>>>
>>>
>>>       
>>>> Raghavendra SM wrote:
>>>>
>>>>         
>>>>> As expected, when I put a delay of about 50 milliseconds between the
>>>>> requests there is no dump.
>>>>>
>>>>> But we would still like to know the following,
>>>>> + Isn't Axis capable of handling such huge bombardment of requests (of
>>>>> the order of 1000's) one after another without any delay between the
>>>>> requests? Has anyone tried such client before?
>>>>>
>>>>>
>>>>>           
>>>> Yes it is supposed to. We have done performance testing that proved
>>>> that
>>>> Axis2/C can serve more than 300 rps.
>>>>
>>>>
>>>>         
>>>>> + Can we make out anything out of the valgrind report or the axis logs
>>>>> that were attached?
>>>>>
>>>>>
>>>>>           
>>>> Unfortunately not. When valgrind runs, it makes the system slow, that
>>>> prevents the system crashing in situations like that you are
>>>> experiencing. To solve this kind of a problem, it is a must that you
>>>> provide us with some sample code that can be used to reproduce the
>>>> problem. Without having a way to re-produce the situation, there is no
>>>> way to fix this problem - it could well be a problem in Axis2/C as well
>>>> as it could also be in the code that you use.
>>>>
>>>> Samisa...
>>>>
>>>>
>>>>
>>>>         
>>>>> Regards,
>>>>> ~raghav
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: Senaka Fernando [mailto:senaka@wso2.com]
>>>>> Sent: Thursday, March 06, 2008 4:22 PM
>>>>> To: Apache AXIS C User List
>>>>> Subject: RE: Core dumps in Axis libraries
>>>>>
>>>>> Hi Raghav,
>>>>>
>>>>> Can you please try this with --enable-guthilla=yes, and send the
>>>>> dump.log,
>>>>> and also the appropriate log_files generated by the server/client.
>>>>>
>>>>> There also can be issues with your "condition variable" being used in
>>>>> multiple threads.
>>>>>
>>>>> It also seems that if you pause between your requests you might get
>>>>> this
>>>>> solved. According to what you say, when you increase the log_level,
>>>>> obviously you waste some time there which might be the solution
>>>>> rather.
>>>>> You can use the AXIS2_SLEEP() and AXIS2_USLEEP() inside Axis2/C to
>>>>> pause
>>>>> between requests. You can get an idea on how to use these functions by
>>>>> taking a look at the Axis2/C user_guide/client samples.
>>>>>
>>>>> Regards,
>>>>> Senaka
>>>>>
>>>>>
>>>>>
>>>>>           
>>>>>> Thanks,
>>>>>>
>>>>>> As suggested, I tried using axis2c-1.3.0 both with and without
>>>>>> --enable-guthilla=yes. But I still see different dumps consistently.
>>>>>> Please find the attached file with back-traces.
>>>>>>
>>>>>> Are you using the same service client instance from multiple threads?
>>>>>>
>>>>>>
>>>>>>             
>>>>>>>>> We have a axis thread pool with default number of threads
and a
>>>>>>>>> set
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                   
>>>>>> of our own threads(about 10, lets call them "database threads").
>>>>>> + On receiving a HTTP soap request, axis thread invokes the
>>>>>>
>>>>>>
>>>>>>             
>>>>> appropriate
>>>>>
>>>>>
>>>>>           
>>>>>> function that is bound to our logic.
>>>>>> + Now, the axis thread requests a "database thread" to perform the
>>>>>> appropriate query and waits on a condition variable.
>>>>>> + our "db thread" performs the necessary query and signals on that
>>>>>> condition variable so that axis thread shall take the control over.
>>>>>> + axis thread will create the om output and dispatch as usual.
>>>>>>
>>>>>> My doubts:
>>>>>> + does this design has any apparent drawback when there is a flurry
>>>>>> of
>>>>>> back-to-back soap requests?
>>>>>> + Is there a need to increase the axis thread pool size? If yes,
how
>>>>>>
>>>>>>
>>>>>>             
>>>>> to
>>>>>
>>>>>
>>>>>           
>>>>>> do it?
>>>>>> + As told before, the dump is seen during a particular scenario only.
>>>>>> i.e: while trying to add a list of already existing users to my
>>>>>> database. For each user there is a SOAP request generated and the
>>>>>> requests are sequential, i.e the second request is sent only after
>>>>>> the
>>>>>> response for previous request is received.
>>>>>> + if I increase the log level of my own module that uses axis2c,
>>>>>> there
>>>>>> is no dump. On increasing the log level, my module writes quite a
bit
>>>>>>
>>>>>>
>>>>>>             
>>>>> of
>>>>>
>>>>>
>>>>>           
>>>>>> information to my log file.
>>>>>>
>>>>>> Your help is appreciated.
>>>>>>
>>>>>> Thanks & Regards,
>>>>>> ~raghav
>>>>>>
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Samisa Abeysinghe [mailto:samisa@wso2.com]
>>>>>> Sent: Wednesday, March 05, 2008 7:25 PM
>>>>>> To: Apache AXIS C User List
>>>>>> Subject: Re: Core dumps in Axis libraries
>>>>>>
>>>>>> Raghavendra SM wrote:
>>>>>>
>>>>>>
>>>>>>             
>>>>>>> Thanks Manjula,
>>>>>>>
>>>>>>> But thats not a viable option for us. We would still want to
>>>>>>> continue
>>>>>>> using axis2C-1.0 for some more time.
>>>>>>>
>>>>>>> Are these dumps well known?
>>>>>>>
>>>>>>>
>>>>>>>               
>>>>>> No.
>>>>>>
>>>>>> Are you using the same service client instance from multiple threads?
>>>>>>
>>>>>> Samisa...
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>>
>>>>>>
>>>>>>             
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>           
>>>> --
>>>> Samisa Abeysinghe
>>>> Software Architect; WSO2 Inc.
>>>>
>>>> http://www.wso2.com/ - "Oxygenating the Web Service Platform."
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>
>>>>
>>>>
>>>>         
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
>>> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>>>
>>>
>>>
>>>
>>>       
>> --
>> Samisa Abeysinghe
>> Software Architect; WSO2 Inc.
>>
>> http://www.wso2.com/ - "Oxygenating the Web Service Platform."
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
>> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>>
>>
>>     
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>
>
>
>   


-- 
Samisa Abeysinghe 
Software Architect; WSO2 Inc.

http://www.wso2.com/ - "Oxygenating the Web Service Platform."


---------------------------------------------------------------------
To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-c-dev-help@ws.apache.org


Mime
View raw message