cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Saurav Lahiri <>
Subject [DISCUSS](CLOUDSTACK-3471)Provide an API to extract the log statements of a given jobid
Date Wed, 04 Dec 2013 14:26:37 GMT
This is to initiate discussion towards implementing an api to extract the
log messages
specific to a jobid.

As has been described in the bug descrition as of cloudstack 4.3 there is
no api that can
aggregate log messages by the job id. An api to extract logs by the jobid
would make it
easier to identify the sequence of steps that could have caused any issue
at hand. Manually
sifting through the huge log files is always an inconvenience.

To resolve this issue I am looking to implement the api which will extract
log messages by
job id. This will be available for use by any required consumers like UI,
cloudmonkey etc.
The api would in almost all cases be used to identify the sequence of steps
which have
occurred in relation to a job. In cases of failure this could potentially
point out the steps
which caused the issue. In other scenarios it could to an extent point out
how much a job has
progressed. Though progress indicator is not the purpose of the api.

Being an api, it will follow the standard api invocation procedures. It can
be invoked programatically
using the desired programming languages.

Possible inputs to the api could be  filter parameters like jobid, start
timestap, end timestamp,
severity of the message. It will produce a response which will comprise of
all the lines from the
logfiles which correspond to this jobid and  satisfy the specified
filtering criteria. The api would be
available for root admin, domain admin and users. The security
consideration should be that users
should only be able to query log messages for jobs that have been
triggerred by them.Since the
user is expecting an output it would be a good idea to implement it as a a

[QUESTION] Does anybody other than root admin need this API?

To filter and search the log messages instead of implementing the
 filtering code in cloudstack the
bulk of the job can be done through a opensource tool like
logstash+elasticsearch or fluentd. From
preliminary investigation it appears that logstash+elasticsearch could be a
feasible option.
Inputs on any other tool which can better accomplish the task is also
welcome. A point of consideration
with  logstash+elasticsearch is that if the log files grow very large then
to function optimally it could
require a dedicated hardware since these tools tend to cache large amount
of data in memory.
Currently I do not have any data to point to what large means but I shall
be running further tests during
evaluation of this combination.

Above is a brief description on what is being proposed, do let me know if
above sounds workable.
Suggestions/Objections are welcome.


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message