axis-c-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Senaka Fernando" <sen...@wso2.com>
Subject Re: Solution to Core Dump Issue Raised in User List [was: Re: Core dumps in Axis libraries]
Date Mon, 10 Mar 2008 11:39:17 GMT
> Senaka Fernando wrote:
>> Hi Samisa,
>>
>> I fixed two bugs regarding stream handling inside
>> simple_http_svr_conn.c.
>> This fixes an issue of this nature.
>>
>> Say, we have two server threads that try to serve requests offered to
>> the
>> axis2 server. Now, according to what is intended, one must serve and the
>> other must wait until it can serve. The previous implementation caused
>> this to fail with a segfault. But, with the fix, it will rather not
>> crash
>> but, loop until it gets a chance to serve the request.
>>
>
> Did you test before the fix and did it seg fault for you? Can you please
> explain the scenario under which it seg faulted? If it is a problem, I
> too should be able to re-create it.
>
> And how did you solve the problem? By adding a sleep?

I made the accept(2) call fail. By passing a -1 to it. Once that failed,
there were two places that segfaulted which I already fixed, few minutes
ago. No, I didn't add a sleep(). I simply prevented the segfault by adding
necessary is_null checks before calling methods. Then, the server just
kept on spawning threads and freeing them as it can't serve the request.
This is the expected behaviour in http_svr_thread.c I believe.

In a real world scenario, a situation similar to a -1 passed to accept(2)
will be temporal. So a respawned thread down the line should server the
request without any problem.

Passing -1 to accept was done by,

Index: util/src/network_handler.c
===================================================================
--- util/src/network_handler.c  (revision 634578)
+++ util/src/network_handler.c  (working copy)
@@ -221,7 +221,7 @@
     AXIS2_ENV_CHECK(env, AXIS2_CRITICAL_FAILURE);

     cli_len = sizeof(cli_addr);
-    cli_socket = accept(svr_socket, (struct sockaddr *) &cli_addr,
&cli_len);
+    cli_socket = accept(-1, (struct sockaddr *) &cli_addr, &cli_len);
     if (cli_socket < 0)
         AXIS2_LOG_ERROR(env->log, AXIS2_LOG_SI,
                         "[Axis2][network_handler] Socket accept \
---

I'm not sure whether this is what really lead to the core dump, or was it
something else.

This was all that I could try on based on the less informative
valgrind.log that was sent. :(...

Also i did test 1000 back-to-back requests and they all were served
without any issues.

Regards,
Senaka

>
> Samisa...
>
>> Could this solve the core dump issue?
>>
>> Regards,
>> Senaka
>>
>>
>>> Raghavendra SM wrote:
>>>
>>>> As expected, when I put a delay of about 50 milliseconds between the
>>>> requests there is no dump.
>>>>
>>>> But we would still like to know the following,
>>>> + Isn't Axis capable of handling such huge bombardment of requests (of
>>>> the order of 1000's) one after another without any delay between the
>>>> requests? Has anyone tried such client before?
>>>>
>>>>
>>> Yes it is supposed to. We have done performance testing that proved
>>> that
>>> Axis2/C can serve more than 300 rps.
>>>
>>>
>>>> + Can we make out anything out of the valgrind report or the axis logs
>>>> that were attached?
>>>>
>>>>
>>> Unfortunately not. When valgrind runs, it makes the system slow, that
>>> prevents the system crashing in situations like that you are
>>> experiencing. To solve this kind of a problem, it is a must that you
>>> provide us with some sample code that can be used to reproduce the
>>> problem. Without having a way to re-produce the situation, there is no
>>> way to fix this problem - it could well be a problem in Axis2/C as well
>>> as it could also be in the code that you use.
>>>
>>> Samisa...
>>>
>>>
>>>
>>>> Regards,
>>>> ~raghav
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Senaka Fernando [mailto:senaka@wso2.com]
>>>> Sent: Thursday, March 06, 2008 4:22 PM
>>>> To: Apache AXIS C User List
>>>> Subject: RE: Core dumps in Axis libraries
>>>>
>>>> Hi Raghav,
>>>>
>>>> Can you please try this with --enable-guthilla=yes, and send the
>>>> dump.log,
>>>> and also the appropriate log_files generated by the server/client.
>>>>
>>>> There also can be issues with your "condition variable" being used in
>>>> multiple threads.
>>>>
>>>> It also seems that if you pause between your requests you might get
>>>> this
>>>> solved. According to what you say, when you increase the log_level,
>>>> obviously you waste some time there which might be the solution
>>>> rather.
>>>> You can use the AXIS2_SLEEP() and AXIS2_USLEEP() inside Axis2/C to
>>>> pause
>>>> between requests. You can get an idea on how to use these functions by
>>>> taking a look at the Axis2/C user_guide/client samples.
>>>>
>>>> Regards,
>>>> Senaka
>>>>
>>>>
>>>>
>>>>> Thanks,
>>>>>
>>>>> As suggested, I tried using axis2c-1.3.0 both with and without
>>>>> --enable-guthilla=yes. But I still see different dumps consistently.
>>>>> Please find the attached file with back-traces.
>>>>>
>>>>> Are you using the same service client instance from multiple threads?
>>>>>
>>>>>
>>>>>>>> We have a axis thread pool with default number of threads
and a
>>>>>>>> set
>>>>>>>>
>>>>>>>>
>>>>> of our own threads(about 10, lets call them "database threads").
>>>>> + On receiving a HTTP soap request, axis thread invokes the
>>>>>
>>>>>
>>>> appropriate
>>>>
>>>>
>>>>> function that is bound to our logic.
>>>>> + Now, the axis thread requests a "database thread" to perform the
>>>>> appropriate query and waits on a condition variable.
>>>>> + our "db thread" performs the necessary query and signals on that
>>>>> condition variable so that axis thread shall take the control over.
>>>>> + axis thread will create the om output and dispatch as usual.
>>>>>
>>>>> My doubts:
>>>>> + does this design has any apparent drawback when there is a flurry
>>>>> of
>>>>> back-to-back soap requests?
>>>>> + Is there a need to increase the axis thread pool size? If yes, how
>>>>>
>>>>>
>>>> to
>>>>
>>>>
>>>>> do it?
>>>>> + As told before, the dump is seen during a particular scenario only.
>>>>> i.e: while trying to add a list of already existing users to my
>>>>> database. For each user there is a SOAP request generated and the
>>>>> requests are sequential, i.e the second request is sent only after
>>>>> the
>>>>> response for previous request is received.
>>>>> + if I increase the log level of my own module that uses axis2c,
>>>>> there
>>>>> is no dump. On increasing the log level, my module writes quite a bit
>>>>>
>>>>>
>>>> of
>>>>
>>>>
>>>>> information to my log file.
>>>>>
>>>>> Your help is appreciated.
>>>>>
>>>>> Thanks & Regards,
>>>>> ~raghav
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: Samisa Abeysinghe [mailto:samisa@wso2.com]
>>>>> Sent: Wednesday, March 05, 2008 7:25 PM
>>>>> To: Apache AXIS C User List
>>>>> Subject: Re: Core dumps in Axis libraries
>>>>>
>>>>> Raghavendra SM wrote:
>>>>>
>>>>>
>>>>>> Thanks Manjula,
>>>>>>
>>>>>> But thats not a viable option for us. We would still want to
>>>>>> continue
>>>>>> using axis2C-1.0 for some more time.
>>>>>>
>>>>>> Are these dumps well known?
>>>>>>
>>>>>>
>>>>> No.
>>>>>
>>>>> Are you using the same service client instance from multiple threads?
>>>>>
>>>>> Samisa...
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>
>>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>
>>>>
>>>>
>>>>
>>>>
>>> --
>>> Samisa Abeysinghe
>>> Software Architect; WSO2 Inc.
>>>
>>> http://www.wso2.com/ - "Oxygenating the Web Service Platform."
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>
>>>
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
>> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>>
>>
>>
>>
>
>
> --
> Samisa Abeysinghe
> Software Architect; WSO2 Inc.
>
> http://www.wso2.com/ - "Oxygenating the Web Service Platform."
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-c-dev-help@ws.apache.org


Mime
View raw message