cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Saurav Lahiri <>
Subject Re: [DISCUSS](CLOUDSTACK-3471)Provide an API to extract the log statements of a given jobid
Date Fri, 13 Dec 2013 05:21:58 GMT
Hi All,
I hope some of us have been able to read through the proposal. Awaiting
thoughts/suggestion/feedback.  I tried to upload more details on this
proposal to the wiki but unable to add pages.

Can somebody please add permission so that i could add a design doc to
wiki. wiki User - (


On Wed, Dec 4, 2013 at 7:56 PM, Saurav Lahiri <>wrote:

> All,
> This is to initiate discussion towards implementing an api to extract the
> log messages
> specific to a jobid.
> As has been described in the bug descrition as of cloudstack 4.3 there is
> no api that can
> aggregate log messages by the job id. An api to extract logs by the jobid
> would make it
> easier to identify the sequence of steps that could have caused any issue
> at hand. Manually
> sifting through the huge log files is always an inconvenience.
> To resolve this issue I am looking to implement the api which will extract
> log messages by
> job id. This will be available for use by any required consumers like UI,
> cloudmonkey etc.
> The api would in almost all cases be used to identify the sequence of
> steps which have
> occurred in relation to a job. In cases of failure this could potentially
> point out the steps
> which caused the issue. In other scenarios it could to an extent point out
> how much a job has
> progressed. Though progress indicator is not the purpose of the api.
> Being an api, it will follow the standard api invocation procedures. It
> can be invoked programatically
> using the desired programming languages.
> Possible inputs to the api could be  filter parameters like jobid, start
> timestap, end timestamp,
> severity of the message. It will produce a response which will comprise of
> all the lines from the
> logfiles which correspond to this jobid and  satisfy the specified
> filtering criteria. The api would be
> available for root admin, domain admin and users. The security
> consideration should be that users
> should only be able to query log messages for jobs that have been
> triggerred by them.Since the
> user is expecting an output it would be a good idea to implement it as a a
> blocking/synchronous
> API.
> [QUESTION] Does anybody other than root admin need this API?
> To filter and search the log messages instead of implementing the
>  filtering code in cloudstack the
> bulk of the job can be done through a opensource tool like
> logstash+elasticsearch or fluentd. From
> preliminary investigation it appears that logstash+elasticsearch could be
> a feasible option.
> Inputs on any other tool which can better accomplish the task is also
> welcome. A point of consideration
> with  logstash+elasticsearch is that if the log files grow very large then
> to function optimally it could
> require a dedicated hardware since these tools tend to cache large amount
> of data in memory.
> Currently I do not have any data to point to what large means but I shall
> be running further tests during
> evaluation of this combination.
> Above is a brief description on what is being proposed, do let me know if
> above sounds workable.
> Suggestions/Objections are welcome.
> Thanks
> Saurav

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message