airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Suresh Marru <>
Subject Re: Experiment log messages to different log files
Date Thu, 28 Aug 2014 17:08:03 GMT
I think this is subjective. We may want logs to be separated by experiment, workflows, applications,
resources, users, type of resources (all ec2 logs), and so forth. 

The trend seems to be log verbosely (with proper tags as Terri listed and may be more fields),
rotate aggressively, and let a downstream component on a non-critical path stream all these
into HDFS like stores and allow processing on them. 

An example would be funnel all airavata log in last hour, and the quires can be: search for
a particular experiment id, search for all experiment which ran on trestles, search for all
experiments which were executed by a particular gfac instance, search by user, inputs and
so on. 


On Aug 28, 2014, at 10:53 AM, Marlon Pierce <> wrote:

> I'm wondering if it would be a good idea to create a separate log file for each experiment.
 We current direct everything to airavata.log and airavata-server.out. As a consequence, log
entries for different submissions get interleaved, which makes tracing a particular experiment's
life cycle difficult.
> Note there is a 32,000 file limit per directory in Linux by default (unless my knowledge
is obsolete), so we'd need to take this into account.
> Alternatively, we could do a better job of labeling the entries in airavata.log so that
it was clear which experiment is associated with the entry.
> Other suggestions?
> Marlon

View raw message