axis-java-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andreas Veithen <andreas.veit...@gmail.com>
Subject Re: [GSoC Project Proposal] Distributed TCP Monitor
Date Tue, 30 Mar 2010 22:17:25 GMT
Probably we are already going too far into implementation details. I
think that the project would have 3 parts:

1. Establishing the general (implementation independent) framework:

- Define protocols: SOAP headers, but also mappings to HTTP headers
(for REST) and JMS properties.
- Define the structure of the correlation identifiers so that the
entire sequence can be reconstructed in a meaningful way.
- Look for strategies to detect and/or avoid duplicate logging
(because of redundant interceptors or simply because a message is
logged at the transmitter and receiver side).
- Find solutions for cases where the message is not XML.
- Think about what happens if one of the services doesn't understand
the protocol.
- Think about security implications.
- ...

2. Build a generic Java framework that supports this style of logging
(with components for remote logging, classes that represent the
context, tools to browse the intercepted messages, etc.).

3. Build a set of interceptors that make use of this framework (and
that log the messages and propagate the context). This can be Axis2
modules, JAX-WS handlers or AspectJ aspects.

On Tue, Mar 30, 2010 at 23:28, nilupa bandara <nilupas@gmail.com> wrote:
> Hello Everyone,
>
> First let me thank all of you for your valuable inputs.
>
> Now let me ask few questions to make sure that I understood the
> suggested approach correctly.
>
> - What are the components that you meant by 'Components who call
> RemoteLogger.log()'  ? A Custom Message Receiver which logs the
> messages that are going in (and out) to a Service Impl perhaps ?
>
> - Does Service Impl needs check the availability of RemoteLogger (via
> a thread local variable) and log the message it sends ( and receive)
> using Service Clients to other applications ? I mean is it the case
> where service author needs to insert 'RemoteLogger.Log()' in
> appropriate places to ensure complete traceability ?
>
> Using AOP - I just read few documentation on AspectJ and something
> like the following could be done (But I'm not absolutely sure though
> !!)
>
> We could use some AOP code (package in a jar) which will, just before
> each Service Client invocation insert the appropriate debug info
> headers to outgoing payload , plus logs the response it gets.
>
> This AOP could be instrumented (or weaved) into all the service
> archive files in a given repository.
>
> We could provide a script (and AOP jar) to automate this. For instance
> ./our_script AOP.jar <path-to-repository> will make all the services
> that are available in the repository traceable.
>
> Any suggestions ?
>
> Nilupa
>
>
> On Tue, Mar 30, 2010 at 3:11 PM, Glen Daniels <glen@thoughtcraft.com> wrote:
>> I'm a really terribly inconsistent blogger.  However, my last actual entry,
>> from 2009, was about exactly this:
>>
>> http://glendaniels.blogspot.com/2009/03/debugging-multi-hop-soa-interactions.html
>>
>> I like Andreas' suggestion below, but I think it might be a bit heavyweight
>> for a GSoC project(?).  Maybe we could lower the bar to something like this:
>>
>> * Build a distributed logging web service which can accept log messages and
>> correlationIDs, logging to a file per correlationID.
>>
>> * Build a Module that processes incoming <debug:logInfo> headers and makes
an
>> appropriate RemoteLogger (or whatever) available.  This could, at first, go
>> in one of Axis2's contexts (OperationContext?) as a demonstration.
>> Components who call RemoteLogger.log() send messages (hopefully on another
>> thread) to the central log service.
>>
>> * The Module could automatically put the right metadata on outgoing messages,
>> but only the responses, so you still have the problem Andreas points out
>> about needing the app to transfer the logging info over to new
>> ServiceClients.  I was imagining that could be achieved by also setting a
>> ThreadLocal - so if your clients run on the same thread you're all set, and
>> if there are new threads it's up to the framework to copy that ThreadLocal over.
>>
>> * Regardless of the discovery mechanism, you'd want something static like
>> RemoteLogger.getLogger() to hide it.
>>
>> This approach is definitely a bit more intrusive, as it relies on the
>> components themselves to be looking for a RemoteLogger, but it seems like a
>> reasonable starting point which could be evolved.  However, maybe the AOP
>> stuff isn't as challenging as I'm imagining....
>>
>> Thoughts?
>>
>> --Glen
>>
>> On 3/30/2010 4:57 AM, Andreas Veithen wrote:
>>> I was actually referring to scenarios where a service may call other
>>> services, which in turn may call again other services. In these
>>> scenarios, it is not sufficient to simply collect received/sent
>>> messages on different hosts, unless the system is isolated
>>> (development environment) or the request rate is sufficiently low so
>>> that messages can be correlated based on time. Here is a concrete
>>> scenario from a project in my company:
>>>
>>> - Request comes in on an ESB that does security and validation.
>>> - Request is processed by an application server which persists the
>>> received information and publishes a JMS message (pub/sub events).
>>> - The event is consumed by one or more components that may in turn
>>> interact with other services.
>>>
>>> If you want track this flow, it is not sufficient to intercept the
>>> messages on the different hosts: in addition, you need to instrument
>>> the services so that the outgoing requests are correlated with the
>>> incoming requests (by adding SOAP and/or transport headers). What I
>>> would like to be able to do in the above scenario is to enable
>>> end-to-end monitoring on a per-message basis: I add a special SOAP or
>>> HTTP header to the initial request to enable logging and this
>>> information is propagated along the chain. Every service/component
>>> then sends a copy of the incoming/outgoing requests/responses to a
>>> central place where the sequence of events is reconstructed.
>>>
>>> One way this could be achieved is with a tool that postprocesses the
>>> artifacts from the build process. For each artifact, the tool would
>>> disassemble the artifact, instrument the code using a set of AspectJ
>>> aspects and reassemble the artifact for deployment. The responsibility
>>> of the aspects would be to intercept messages, log them and modify
>>> them to ensure proper correlation. Of course this only works if the
>>> artifacts are J2EE deployables (WARs or EARs). Note that we already
>>> use AspectJ for a similar use case (although at a much smaller scale)
>>> in the transport test suites [1]. Interestingly, this approach would
>>> allow to cover interactions that are not SOAP based, e.g. one could
>>> even intercept the database queries executed during the flow.
>>>
>>> Andreas
>>>
>>> [1] https://svn.apache.org/repos/asf/axis/axis2/java/transports/trunk/modules/jms/src/test/java/org/apache/axis2/transport/jms/LogAspect.java
>>>
>>> On Tue, Mar 30, 2010 at 09:11, Srinath Perera <hemapani@gmail.com> wrote:
>>>> Hi Nilupa;
>>>>
>>>> When we collect messages from a one location by installing proper
>>>> handler that will intercept and send messages to to that one location
>>>> (this one location can be a single server, pub/sub channel etc). There
>>>> is many ways to make sense of those collected messages. What Andreas
>>>> mentioned (following complete transaction) is a one possibility.
>>>>
>>>> I think you should come up with few scenarios on how you would make
>>>> sense of the message.
>>>>
>>>> Thanks
>>>> Srinath
>>>>
>>>> On Sun, Mar 28, 2010 at 9:46 PM, nilupa bandara <nilupas@gmail.com>
wrote:
>>>>> Hi,
>>>>>
>>>>> First let me thank you for commenting.
>>>>>
>>>>> As far as I understood, what you would like to see from the proposed
>>>>> tool is to view set of messages that are exchanged in reponse to a
>>>>> particular input message. With the understanding that I am having at
>>>>> the momnet, one way to do it is to filter out the central repository
>>>>> of messages based on 'To' , 'From' headers and try to contruct the
>>>>> message chain from it. We can allow the client GUI wich connects to
>>>>> the central repository to provide the paramenters (For instance the
>>>>> value of 'To' header) from which an intelligent filtering can be done
>>>>> for the set of messages avialable at the central repository.
>>>>>
>>>>> Perhaps someone has an idea of a better way of doing it and is willing
>>>>> to share it with us. It would be really nice to hear from them.
>>>>>
>>>>> Thanks in advance ..!!
>>>>>
>>>>> Best Regards,
>>>>> Nilupa
>>>>>
>>>>> On Sat, Mar 27, 2010 at 6:40 PM, Andreas Veithen
>>>>> <andreas.veithen@gmail.com> wrote:
>>>>>> Personally, I think that the added value of extending the SOAP monitor
>>>>>> module to collect messages in a central place is too limited to
>>>>>> attract enough interest from the user community, so that it will
be
>>>>>> difficult to ensure the evolution of such a project in the future.
>>>>>> However, what many people are looking for is a tool that allows to
>>>>>> track the flow of a message through a distributed system, or more
>>>>>> generally to track the sequence of events triggered by a given input
>>>>>> message (sort of end-to-end transaction monitoring).
>>>>>>
>>>>>> On Wed, Mar 24, 2010 at 23:03, nilupa bandara <nilupas@gmail.com>
wrote:
>>>>>>> Hello,
>>>>>>>
>>>>>>> I am graduate student at Politecnica de Madrid and I am thinking
of
>>>>>>> proposing a GSoC project this summer. I would like to take project
>>>>>>> "Distribute TCP Monitor" descried[1] which is very interesting
and
>>>>>>> also should be helpful to any Java developer using Apache Axis2
Web
>>>>>>> services middleware. I will submit more detailed proposal later.
I
>>>>>>> would really like to hear any feedback from you. Any suggestions
or
>>>>>>> features that you would like to see in a "Distributed TCP Monitor"
are
>>>>>>> most welcome.
>>>>>>>
>>>>>>> Best Regards,
>>>>>>> Nilupa Bandara
>>>>>>>
>>>>>>> [1]
>>>>>>>
>>>>>>> Distributed TCP monitor
>>>>>>>
>>>>>>> To use TCP monitor we have to tweak the endpoints, which is bit
hard
>>>>>>> sometimes (e.g. two channels Async case for Axis2). We can solve
the
>>>>>>> problem by writing a Axis2 module (Handler) that intercept messages
>>>>>>> comes in and goes out of a Axis2 server and send to a UI (via
a
>>>>>>> pub/sub channel or a message Box e.g. ) or record them so user
can
>>>>>>> pull those messages. Also, we can do something to turn the module
on
>>>>>>> and off remotely, so users can have his module deployed, but
turn it
>>>>>>> on only when they want to debug the system.
>>>>>>>
>>>>>>> Then we can take this to next level by adding this module to
all
>>>>>>> services in a system , configuring modules to send all collected
>>>>>>> messages to a pub/sub channel, and subscribing to those messages
via a
>>>>>>> UI, and depicting those messages through a UI. Then users have
a
>>>>>>> TCPMon for a whole system.
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
>>>>>>> For additional commands, e-mail: java-dev-help@axis.apache.org
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
>>>>>> For additional commands, e-mail: java-dev-help@axis.apache.org
>>>>>>
>>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
>>>>> For additional commands, e-mail: java-dev-help@axis.apache.org
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> ============================
>>>> Srinath Perera, Ph.D.
>>>>   WSO2 Inc. http://wso2.com
>>>>   Blog: http://srinathsview.blogspot.com/
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
>>>> For additional commands, e-mail: java-dev-help@axis.apache.org
>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
>>> For additional commands, e-mail: java-dev-help@axis.apache.org
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
>> For additional commands, e-mail: java-dev-help@axis.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
> For additional commands, e-mail: java-dev-help@axis.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
For additional commands, e-mail: java-dev-help@axis.apache.org


Mime
View raw message