airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Schwartz, Terri" <te...@sdsc.edu>
Subject RE: Experiment log messages to different log files
Date Thu, 28 Aug 2014 15:13:19 GMT
For cipres logging we use a single file, rolled over daily, and include thread-id, and when
relevant, task-id in each entry using log4j's nested diagnostic context.   It's usually easy
enough to grep for just the task I'm interested in.

Terri
________________________________________
From: Miller, Mark [mmiller@sdsc.edu]
Sent: Thursday, August 28, 2014 8:02 AM
To: dev@airavata.apache.org
Subject: RE: Experiment log messages to different log files

Re max files, here is what I found:

It depends on the file system. ext3 suppport ~32000 subdirectories (not files!) in a given
directory, with ext4 it's 64000 by default. xfs has no limit to my knowledge.  You should
consider not putting too many files in a single directory. Most software doesn't handle that
well (e.g. mc will be slow, many gui tools will be unusable). It's better to create a hierarchy
of nested folders and distribute the files in them using some algorithm (hash of file name
or content or any other method which will distribute the files equally). That's what many
mature programs (e.g. squid) do.


-----Original Message-----
From: Marlon Pierce [mailto:marpierc@iu.edu]
Sent: Thursday, August 28, 2014 7:54 AM
To: Airavata Dev
Subject: Experiment log messages to different log files

I'm wondering if it would be a good idea to create a separate log file for each experiment.
 We current direct everything to airavata.log and airavata-server.out. As a consequence, log
entries for different submissions get interleaved, which makes tracing a particular experiment's
life cycle difficult.

Note there is a 32,000 file limit per directory in Linux by default (unless my knowledge is
obsolete), so we'd need to take this into account.

Alternatively, we could do a better job of labeling the entries in airavata.log so that it
was clear which experiment is associated with the entry.

Other suggestions?

Marlon


Mime
View raw message