Mailing-List: contact axis-c-dev-help@ws.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Apache AXIS C Developers List" <axis-c-dev@ws.apache.org>
Received-SPF: neutral (nike.apache.org: local policy)
Message-ID: <47D54126.7060902@wso2.com>
Date: Mon, 10 Mar 2008 19:39:42 +0530
From: Samisa Abeysinghe <samisa@wso2.com>
User-Agent: Thunderbird 2.0.0.12 (Windows/20080213)
MIME-Version: 1.0
To: Apache AXIS C Developers List <axis-c-dev@ws.apache.org>
Subject: Re: Solution to Core Dump Issue Raised in User List [was: Re: Core
                     dumps in Axis libraries]
References: <H00000a00022aa7f.1205140213.dakghar.aylus.com@MHS>
             <47D5076D.8050602@wso2.com>
          <48499.124.43.212.189.1205146734.squirrel@webmail4.pair.com>
          <47D51751.90307@wso2.com>
       <42005.124.43.212.189.1205149157.squirrel@webmail4.pair.com>
       <47D52324.7060408@wso2.com>
    <34686.124.43.215.89.1205152663.squirrel@webmail4.pair.com>
 <39707.124.43.215.89.1205155148.squirrel@webmail4.pair.com>
In-Reply-To: <39707.124.43.215.89.1205155148.squirrel@webmail4.pair.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

There are two sides to this problem. One is that, if it is failing for 
some scenario, we should be able to re-produce it. So without some code 
to re-produce the fault, we cannot come to a conclusion that it fails.
If accepts fails, and it crashes, then there needs to be someuser 
scenario that makes this happen. If we cannot find a user scenario that 
makes accept to fail, then even though the bug is there, it will never 
be exposed.

Anyway we can ask the user to try with your fixes for accept failure 
case and test to see if the problem persists.

Samisa...

Senaka Fernando wrote:
> Hi again Samisa,
>
> And, also, I don't have any similar database accessing logic etc.
> Therefore, it is hard to conclude without seeing the actual code. Also,
> the user seems to be using a modified version of "http_server_main.c", it
> is actually a cpp file if you go through that portion which I highlighted
> in his log. May be some issues due to not handling concurrency properly
> could also lead him into trouble.
>
> However, what I can say is that there was a bug if accept(2) was to fail,
> according to the tests I did. And, according to literature available
> online that can fail. Plus, we do have a mechanism of retrying, if it does
> fail which did not work due to the segfault.
>
> Regards,
> Senaka
>
>   
>>> Were you able to re-create the problem without passing -1 to accept?
>>>       
>> No I was not able to make accept fail without passing -1. Added a printf
>> there to print any failure in accept and it didn't. However, ran into the
>> segfault issue when accept actually failed, which I had to do manually.
>>
>> I did test with 1000 back-2-back mtom requests in 5 terminals, and also
>> 1000 repeated calls to an echo client that sends 1000 back-2-back
>> requests, and in 4 terminals.
>>
>> None of these cases failed. There were occasional hangs but still it
>> didn't result in any failure on either client or server side.
>>
>> However, I didn't have any way to test multiple requests made to a single
>> port from multiple machines. That may be a different scenario.
>>
>> Regards,
>> Senaka
>>
>>     
>>> Samisa...
>>>
>>> Senaka Fernando wrote:
>>>       
>>>>> Senaka Fernando wrote:
>>>>>
>>>>>           
>>>>>> Hi Samisa,
>>>>>>
>>>>>> I fixed two bugs regarding stream handling inside
>>>>>> simple_http_svr_conn.c.
>>>>>> This fixes an issue of this nature.
>>>>>>
>>>>>> Say, we have two server threads that try to serve requests offered to
>>>>>> the
>>>>>> axis2 server. Now, according to what is intended, one must serve and
>>>>>> the
>>>>>> other must wait until it can serve. The previous implementation
>>>>>> caused
>>>>>> this to fail with a segfault. But, with the fix, it will rather not
>>>>>> crash
>>>>>> but, loop until it gets a chance to serve the request.
>>>>>>
>>>>>>
>>>>>>             
>>>>> Did you test before the fix and did it seg fault for you? Can you
>>>>> please
>>>>> explain the scenario under which it seg faulted? If it is a problem, I
>>>>> too should be able to re-create it.
>>>>>
>>>>> And how did you solve the problem? By adding a sleep?
>>>>>
>>>>>           
>>>> I made the accept(2) call fail. By passing a -1 to it. Once that
>>>> failed,
>>>> there were two places that segfaulted which I already fixed, few
>>>> minutes
>>>> ago. No, I didn't add a sleep(). I simply prevented the segfault by
>>>> adding
>>>> necessary is_null checks before calling methods. Then, the server just
>>>> kept on spawning threads and freeing them as it can't serve the
>>>> request.
>>>> This is the expected behaviour in http_svr_thread.c I believe.
>>>>
>>>> In a real world scenario, a situation similar to a -1 passed to
>>>> accept(2)
>>>> will be temporal. So a respawned thread down the line should server the
>>>> request without any problem.
>>>>
>>>> Passing -1 to accept was done by,
>>>>
>>>> Index: util/src/network_handler.c
>>>> ===================================================================
>>>> --- util/src/network_handler.c  (revision 634578)
>>>> +++ util/src/network_handler.c  (working copy)
>>>> @@ -221,7 +221,7 @@
>>>>      AXIS2_ENV_CHECK(env, AXIS2_CRITICAL_FAILURE);
>>>>
>>>>      cli_len = sizeof(cli_addr);
>>>> -    cli_socket = accept(svr_socket, (struct sockaddr *) &cli_addr,
>>>> &cli_len);
>>>> +    cli_socket = accept(-1, (struct sockaddr *) &cli_addr, &cli_len);
>>>>      if (cli_socket < 0)
>>>>          AXIS2_LOG_ERROR(env->log, AXIS2_LOG_SI,
>>>>                          "[Axis2][network_handler] Socket accept \
>>>> ---
>>>>
>>>> I'm not sure whether this is what really lead to the core dump, or was
>>>> it
>>>> something else.
>>>>
>>>> This was all that I could try on based on the less informative
>>>> valgrind.log that was sent. :(...
>>>>
>>>> Also i did test 1000 back-to-back requests and they all were served
>>>> without any issues.
>>>>
>>>> Regards,
>>>> Senaka
>>>>
>>>>
>>>>         
>>>>> Samisa...
>>>>>
>>>>>
>>>>>           
>>>>>> Could this solve the core dump issue?
>>>>>>
>>>>>> Regards,
>>>>>> Senaka
>>>>>>
>>>>>>
>>>>>>
>>>>>>             
>>>>>>> Raghavendra SM wrote:
>>>>>>>
>>>>>>>
>>>>>>>               
>>>>>>>> As expected, when I put a delay of about 50 milliseconds between
>>>>>>>> the
>>>>>>>> requests there is no dump.
>>>>>>>>
>>>>>>>> But we would still like to know the following,
>>>>>>>> + Isn't Axis capable of handling such huge bombardment of requests
>>>>>>>> (of
>>>>>>>> the order of 1000's) one after another without any delay between
>>>>>>>> the
>>>>>>>> requests? Has anyone tried such client before?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                 
>>>>>>> Yes it is supposed to. We have done performance testing that proved
>>>>>>> that
>>>>>>> Axis2/C can serve more than 300 rps.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>               
>>>>>>>> + Can we make out anything out of the valgrind report or the axis
>>>>>>>> logs
>>>>>>>> that were attached?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                 
>>>>>>> Unfortunately not. When valgrind runs, it makes the system slow,
>>>>>>> that
>>>>>>> prevents the system crashing in situations like that you are
>>>>>>> experiencing. To solve this kind of a problem, it is a must that you
>>>>>>> provide us with some sample code that can be used to reproduce the
>>>>>>> problem. Without having a way to re-produce the situation, there is
>>>>>>> no
>>>>>>> way to fix this problem - it could well be a problem in Axis2/C as
>>>>>>> well
>>>>>>> as it could also be in the code that you use.
>>>>>>>
>>>>>>> Samisa...
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>               
>>>>>>>> Regards,
>>>>>>>> ~raghav
>>>>>>>>
>>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> From: Senaka Fernando [mailto:senaka@wso2.com]
>>>>>>>> Sent: Thursday, March 06, 2008 4:22 PM
>>>>>>>> To: Apache AXIS C User List
>>>>>>>> Subject: RE: Core dumps in Axis libraries
>>>>>>>>
>>>>>>>> Hi Raghav,
>>>>>>>>
>>>>>>>> Can you please try this with --enable-guthilla=yes, and send the
>>>>>>>> dump.log,
>>>>>>>> and also the appropriate log_files generated by the server/client.
>>>>>>>>
>>>>>>>> There also can be issues with your "condition variable" being used
>>>>>>>> in
>>>>>>>> multiple threads.
>>>>>>>>
>>>>>>>> It also seems that if you pause between your requests you might get
>>>>>>>> this
>>>>>>>> solved. According to what you say, when you increase the log_level,
>>>>>>>> obviously you waste some time there which might be the solution
>>>>>>>> rather.
>>>>>>>> You can use the AXIS2_SLEEP() and AXIS2_USLEEP() inside Axis2/C to
>>>>>>>> pause
>>>>>>>> between requests. You can get an idea on how to use these functions
>>>>>>>> by
>>>>>>>> taking a look at the Axis2/C user_guide/client samples.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Senaka
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                 
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> As suggested, I tried using axis2c-1.3.0 both with and without
>>>>>>>>> --enable-guthilla=yes. But I still see different dumps
>>>>>>>>> consistently.
>>>>>>>>> Please find the attached file with back-traces.
>>>>>>>>>
>>>>>>>>> Are you using the same service client instance from multiple
>>>>>>>>> threads?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                   
>>>>>>>>>>>> We have a axis thread pool with default number of threads and a
>>>>>>>>>>>> set
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                         
>>>>>>>>> of our own threads(about 10, lets call them "database threads").
>>>>>>>>> + On receiving a HTTP soap request, axis thread invokes the
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                   
>>>>>>>> appropriate
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                 
>>>>>>>>> function that is bound to our logic.
>>>>>>>>> + Now, the axis thread requests a "database thread" to perform the
>>>>>>>>> appropriate query and waits on a condition variable.
>>>>>>>>> + our "db thread" performs the necessary query and signals on that
>>>>>>>>> condition variable so that axis thread shall take the control
>>>>>>>>> over.
>>>>>>>>> + axis thread will create the om output and dispatch as usual.
>>>>>>>>>
>>>>>>>>> My doubts:
>>>>>>>>> + does this design has any apparent drawback when there is a
>>>>>>>>> flurry
>>>>>>>>> of
>>>>>>>>> back-to-back soap requests?
>>>>>>>>> + Is there a need to increase the axis thread pool size? If yes,
>>>>>>>>> how
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                   
>>>>>>>> to
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                 
>>>>>>>>> do it?
>>>>>>>>> + As told before, the dump is seen during a particular scenario
>>>>>>>>> only.
>>>>>>>>> i.e: while trying to add a list of already existing users to my
>>>>>>>>> database. For each user there is a SOAP request generated and the
>>>>>>>>> requests are sequential, i.e the second request is sent only after
>>>>>>>>> the
>>>>>>>>> response for previous request is received.
>>>>>>>>> + if I increase the log level of my own module that uses axis2c,
>>>>>>>>> there
>>>>>>>>> is no dump. On increasing the log level, my module writes quite a
>>>>>>>>> bit
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                   
>>>>>>>> of
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                 
>>>>>>>>> information to my log file.
>>>>>>>>>
>>>>>>>>> Your help is appreciated.
>>>>>>>>>
>>>>>>>>> Thanks & Regards,
>>>>>>>>> ~raghav
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: Samisa Abeysinghe [mailto:samisa@wso2.com]
>>>>>>>>> Sent: Wednesday, March 05, 2008 7:25 PM
>>>>>>>>> To: Apache AXIS C User List
>>>>>>>>> Subject: Re: Core dumps in Axis libraries
>>>>>>>>>
>>>>>>>>> Raghavendra SM wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                   
>>>>>>>>>> Thanks Manjula,
>>>>>>>>>>
>>>>>>>>>> But thats not a viable option for us. We would still want to
>>>>>>>>>> continue
>>>>>>>>>> using axis2C-1.0 for some more time.
>>>>>>>>>>
>>>>>>>>>> Are these dumps well known?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                     
>>>>>>>>> No.
>>>>>>>>>
>>>>>>>>> Are you using the same service client instance from multiple
>>>>>>>>> threads?
>>>>>>>>>
>>>>>>>>> Samisa...
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                   
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                 
>>>>>>> --
>>>>>>> Samisa Abeysinghe
>>>>>>> Software Architect; WSO2 Inc.
>>>>>>>
>>>>>>> http://www.wso2.com/ - "Oxygenating the Web Service Platform."
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: axis-c-user-unsubscribe@ws.apache.org
>>>>>>> For additional commands, e-mail: axis-c-user-help@ws.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>               
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
>>>>>> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>             
>>>>> --
>>>>> Samisa Abeysinghe
>>>>> Software Architect; WSO2 Inc.
>>>>>
>>>>> http://www.wso2.com/ - "Oxygenating the Web Service Platform."
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
>>>>> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>>>>>
>>>>>
>>>>>
>>>>>           
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
>>>> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>>>>
>>>>
>>>>
>>>>
>>>>         
>>> --
>>> Samisa Abeysinghe
>>> Software Architect; WSO2 Inc.
>>>
>>> http://www.wso2.com/ - "Oxygenating the Web Service Platform."
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
>>> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>>>
>>>
>>>       
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
>> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>>
>>
>>     
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>
>
>
>   


-- 
Samisa Abeysinghe 
Software Architect; WSO2 Inc.

http://www.wso2.com/ - "Oxygenating the Web Service Platform."


---------------------------------------------------------------------
To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-c-dev-help@ws.apache.org