Return-Path: Delivered-To: apmail-incubator-stdcxx-dev-archive@www.apache.org Received: (qmail 94508 invoked from network); 11 Oct 2006 22:22:01 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (209.237.227.199) by minotaur.apache.org with SMTP; 11 Oct 2006 22:22:01 -0000 Received: (qmail 8979 invoked by uid 500); 11 Oct 2006 22:22:01 -0000 Delivered-To: apmail-incubator-stdcxx-dev-archive@incubator.apache.org Received: (qmail 8964 invoked by uid 500); 11 Oct 2006 22:22:01 -0000 Mailing-List: contact stdcxx-dev-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: stdcxx-dev@incubator.apache.org Delivered-To: mailing list stdcxx-dev@incubator.apache.org Received: (qmail 8953 invoked by uid 99); 11 Oct 2006 22:22:01 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (140.211.166.49) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Oct 2006 15:22:01 -0700 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received-SPF: pass (asf.osuosl.org: local policy) Received: from [208.30.140.160] (HELO moroha.quovadx.com) (208.30.140.160) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Oct 2006 15:21:59 -0700 Received: from [10.70.3.48] ([10.70.3.48]) by moroha.quovadx.com (8.13.6/8.13.6) with ESMTP id k9BMKcAZ031495 for ; Wed, 11 Oct 2006 22:20:38 GMT Message-ID: <452D6E71.9020207@roguewave.com> Date: Wed, 11 Oct 2006 16:21:37 -0600 From: Andrew Black User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.7) Gecko/20060910 SeaMonkey/1.0.5 MIME-Version: 1.0 To: stdcxx-dev@incubator.apache.org Subject: Re: [PATCH] stdcxx target warning counting References: <452BB8CB.1050201@roguewave.com> <452D302C.2070303@roguewave.com> In-Reply-To: <452D302C.2070303@roguewave.com> Content-Type: multipart/mixed; boundary="------------020907020800080407050202" X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N --------------020907020800080407050202 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Revised patch, attached. I believe the same change log can be used. Martin Sebor wrote: > Andrew Black wrote: >> Greetings all. >> [...] >> + " icc Intel icc\n" > > Does this handle the Windows version of the compiler or just > the Linux one? If both, we should change the text to Intel C++ > to make it clear (or add an alias for icl). This handles just the Linux version of the compiler. I added 'for Linux' to the end of the name in the revised patch in an effort to provide clarity [...] > > You could also add aliases if you like (e.g., for IBM VisualAge, > we could have "vacpp IBM VisualAge C++" in addition to XLC++). I had been trying to stick with 3-4 letter alias names for a measure of consistency, but clarity in naming convention is more important. I did add the vacpp entry in addition to the xlc entry, but I didn't note that they were equivalent, outside of a comment in the translation table. [...] > We could probably just compare the names without regard to case, > don't you think? Or did you have a specific reason for recognizing > only these three forms? One version of the patch did do a case insensitive comparison, moving the rw_strcasecmp (and rw_charcasecmp) functions from exec.cpp to util.cpp. The reason I chose to do things this way was so that different warning parsing modes could be specified for the compile and link stages. An example where this could be useful would be when building with GCC on Solaris, but linking with Sun's linker. If we chose to restrict things to a single tool chain (ie: GCC or sunpro, but not mixing them), I would agree that it would make sense to match in a case-insensitive manner. Looking at the config files seems to indicate to me that we are using a single tool chain normally, but I wonder if this can be assumed to always be true. My suspicion is that we shouldn't. > > [...] >> Index: util/output.cpp >> =================================================================== >> --- util/output.cpp (revision 454401) >> +++ util/output.cpp (working copy) >> @@ -109,7 +109,13 @@ >> return; >> } >> } >> - /* else if (1 < r_lvl) warning*/ >> + else if (1 < r_lvl) { >> + status->t_warn += r_active; >> + if (status->t_warn < r_active) { >> + status->status = ST_OVERFLOW; > > Is this arithmetic overflow? (How long would it take for the counter > to reach 4 billion and how big would the input file have to be in > order for it to happen? I suspect we can drop this check and keep > it simple.) > This is a math overflow check. If we assume ~5 assertions take 1 k of disk space in a result log (each assertion ~200 characters (bytes) long), the resulting file would be ~800,000,000 kb, or ~800 Gb. A file this long is theoretically within the capabilities of a modern OS, but you are likely to run out of disk space prior to writing the entire log. This check was added for consistency with the assertion counting code, and it would be outside the scope of this patch to remove that check (and the ST_OVERFLOW item). > [...] > >> Index: util/display.cpp >> =================================================================== >> --- util/display.cpp (revision 454401) >> +++ util/display.cpp (working copy) >> @@ -40,8 +40,8 @@ >> */ >> static void print_header_plain () >> { >> - puts ("NAME STATUS ASSERTS FAILED PERCNT >> USER " >> - "SYS WALL"); >> + puts ("NAME STATUS WARN ASSERTS FAILED >> PERCNT " >> + "USER SYS WALL"); > > I would suggest to rename WALL to REAL as that's what the times > command reports (and it's probably more a well known term than > WALL alone). > I would agree with renaming the column header, but I feel doing so would also be outside the scope of this patch. If it were to be mixed in with another commit, the patch that cleans up the timing methods would be a logical candidate, but would introduce a conflict between this patch and that one. [...] > [...] >> @@ -212,10 +282,41 @@ >> { >> struct stat file_info; >> int exists = 1; >> + size_t path_len; >> + char* tmp_name; >> >> assert (0 != target); >> assert (0 != status); >> >> + path_len = strlen (target); >> + >> +#if !defined (_WIN32) && !defined (_WIN64) > > As I mentioned before, these conditionals can be simplified to > just #ifndef _WIN32 since _WIN32 is guaranteed to be defined in > 64-bit mode. This is part relocation artifact, part (misguided) consistency with other parts of code that use the same construct. I'm still not certain about the guarantee, but have been unsuccessful in compiling on 64 bit windows. > >> + /* Otherwise, check for the .o file on non-windows systems */ >> + tmp_name = (char*)RW_MALLOC (path_len + 3); > > Is there a call to free() corresponding to this malloc()? (I > don't see it in the patch but it might be in the rest of the > code). There was a free in one part of the code path, but the other (more common) paths lacked the call to free. (This was a result of moving the code block from within a condition to an earlier, unconditional, part of the code flow.) > > Martin --Andrew Black --------------020907020800080407050202 Content-Type: text/x-patch; name="warnparse2.diff" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="warnparse2.diff" Index: etc/config/makefile.common =================================================================== --- etc/config/makefile.common (revision 454401) +++ etc/config/makefile.common (working copy) @@ -143,8 +143,10 @@ # if LOGFILE is being created, tee command output into it # IMPORTANT: $(TEEOPTS) must be last on the command line -ifneq ($(LOGFILE),/dev/null) - TEEOPTS = 2>&1 | tee -a $(LOGFILE) +ifeq ($(LOGFILE),/dev/null) + TEEOPTS = 2>&1 | tee $@.log +else + TEEOPTS = 2>&1 | tee $@.log | tee -a $(LOGFILE) endif # determine the name of the results file against which to compute regressions Index: etc/config/makefile.rules =================================================================== --- etc/config/makefile.rules (revision 454401) +++ etc/config/makefile.rules (working copy) @@ -60,16 +60,16 @@ ifneq ($(AS_EXT),".") %.o: %$(AS_EXT) - $(CXX) -c $(CPPFLAGS) $(CXXFLAGS) $< + $(CXX) -c $(CPPFLAGS) $(CXXFLAGS) $< $(TEEOPTS) endif # ifneq ($(AS_EXT),".") endif # ifneq ($(AS_EXT),) %.o: %.cpp - $(CXX) -c $(CPPFLAGS) $(CXXFLAGS) $< + $(CXX) -c $(CPPFLAGS) $(CXXFLAGS) $< $(TEEOPTS) %: %.o - $(LD) $< -o $@ $(LDFLAGS) $(LDLIBS) + $(LD) $< -o $@ $(LDFLAGS) $(LDLIBS) $(TEEOPTS) # disable compilation and linking in the same step # %: %.cpp @@ -78,7 +78,7 @@ # compile and link in one step to eliminate the space overhead of .o files %: %.cpp - $(CXX) $< -o $@ $(CPPFLAGS) $(CXXFLAGS) $(LDFLAGS) $(LDLIBS) + $(CXX) $< -o $@ $(CPPFLAGS) $(CXXFLAGS) $(LDFLAGS) $(LDLIBS) $(TEEOPTS) endif # eq ($(NO_DOT_O),) Index: util/cmdopt.cpp =================================================================== --- util/cmdopt.cpp (revision 454401) +++ util/cmdopt.cpp (working copy) @@ -89,6 +89,7 @@ " --signal=sig Send itself the specified signal.\n" " --ignore=sig Ignore the specified signal.\n" " --ulimit=lim Set child process usage limits (see below).\n" + " --warn=opts Set compiler log warning pattern (see below).\n" "\n" " All short (single dash) options must be specified seperately.\n" " If a short option takes a value, it may either be provided like\n" @@ -121,6 +122,26 @@ " Note: Some operating systems lack support for some or all of the\n" " ulimit modes. If a system is unable to limit a given property, a\n" " warning message will be produced.\n" + "\n" + " --warn set the string used to parse compile and link logs. Rather\n" + " than specifying a search string, an alias code is provided,\n" + " coresponding to the output of a compiler/linker. Compiler patterns\n" + " are specified with lowercase letters. Linker patterns are specified\n" + " with uppercase letters. To set both patterns, title case can be used.\n" + " Only one compiler and one linker pattern can be active at any given\n" + " time. If multiple patterns are to be specified, they can be specified\n" + " in the same call, seperated by commas.\n" + "\n" + " --warn modes:\n" + " acc HP aCC\n" + " cxx Compaq C++\n" + " eccp EDG eccp\n" + " gcc GNU gcc\n" + " icc Intel icc for Linux\n" + " mipspro SGI MIPSpro\n" + " sunpro Sun C++\n" + " vacpp IBM VisualAge C++\n" + " xlc IBM XLC++\n" }; #if !defined (_WIN32) && !defined (_WIN64) @@ -365,7 +386,77 @@ return 0; } +/** + Helper function to parse a warning value string + @param opts ulimit value string to pares + @see child_limits +*/ +static bool +parse_warn_opts (const char* opts, struct target_opts* defaults) +{ + static const struct { + const char* name; + const char* caps; + const char* mixd; + size_t len; + const char* pat; + } warn_set [] = { + { "acc", "ACC", "Acc", 3, "Warning " }, +/* + { "cds", "CDS", "Cds", 3, "UNKNOWN"}, + { "como", "COMO", "Como", 4, "UNKNOWN"}, +*/ + { "cxx", "CXX", "Cxx", 3, "Warning:"}, + { "eccp", "ECCP", "Eccp", 4, "warning:"}, + { "gcc", "GCC", "Gcc", 3, "warning:"}, + { "icc", "ICC", "Icc", 3, "warning #"}, + { "mipspro", "MIPSPRO", "Mipspro", 7, "CC: WARNING"}, + { "sunpro", "SUNPRO", "Sunpro", 6, "Warning:"}, + { "vacpp", "VACPP", "Vacpp", 5, ": (W) "}, + { "xlc", "XLC", "Xlc", 3, ": (W) "}, /* xlc and vacpp are synonyms. */ + { 0, 0, 0, 0, 0 } + }; + + const char* arg = opts; + + assert (0 != opts); + + while (arg && *arg) { + + const size_t arglen = strlen (arg); + + for (size_t i = 0; warn_set [i].name; ++i) { + if ( warn_set [i].len <= arglen + && ( 0 == memcmp (warn_set [i].name, arg, warn_set [i].len) + || 0 == memcmp (warn_set [i].caps, arg, warn_set [i].len) + || 0 == memcmp (warn_set [i].mixd, arg, warn_set [i].len))) { + + arg += warn_set [i].len + 1; + + if ('\0' != *arg && ',' != *arg) + break; + + if (islower (arg [1])) /* Set for compile if lowercase. */ + defaults->c_warn = warn_set [i].pat; + + if (isupper (arg [0])) /* Set for link if uppercase. */ + defaults->l_warn = warn_set [i].pat; + + break; + } + } + + if (',' == *arg) { + ++arg; + } + else if ('\0' != *arg) { + return 1; + } + } + return 0; +} + /** Helper function to produce 'Bad argument' error message. @@ -423,11 +514,41 @@ const char opt_signal[] = "--signal"; const char opt_sleep[] = "--sleep"; const char opt_ulimit[] = "--ulimit"; + const char opt_warn[] = "--warn"; int i; assert (0 != argv); + /* The chain of preprocesor logic below initializes the defaults->c_warn + and defaults->l_warn values. + */ +#ifdef __GNUG__ + parse_warn_opts ("Gcc", defaults); +#elif defined (__HP_aCC) + parse_warn_opts ("Acc", defaults); +#elif defined (__IBMCPP__) + parse_warn_opts ("Xlc", defaults); +#elif defined (__SUNPRO_CC) + parse_warn_opts ("Sunpro", defaults); +#elif defined (SNI) + parse_warn_opts ("Cds", defaults); +#elif defined (__APOGEE__) /* EDG variant that doesn't define __EDG__. */ + parse_warn_opts ("Como", defaults); + +/* The following are EDG variants, that define __EDG__ */ +#elif defined (__DECCXX) + parse_warn_opts ("Cxx", defaults); +#elif defined (_SGI_COMPILER_VERSION) + parse_warn_opts ("Mipspro", defaults); +#elif defined (__INTEL_COMPILER) + parse_warn_opts ("Icc", defaults); + +/* So we need to check for __EDG__ after we check for them. */ +#elif defined (__EDG__) + parse_warn_opts ("Eccp", defaults); +#endif + if (1 == argc || '-' != argv [1][0]) return 1; @@ -563,6 +684,16 @@ } } } + else if ( sizeof opt_warn - 1 <= arglen + && !memcmp (opt_warn, argv [i], sizeof opt_warn - 1)) { + optname = opt_warn; + optarg = get_long_val (argv, &i, sizeof opt_warn - 1); + if (optarg && *optarg) { + if (!parse_warn_opts (optarg, defaults)) { + break; + } + } + } /* fall through */ } default: Index: util/output.cpp =================================================================== --- util/output.cpp (revision 454401) +++ util/output.cpp (working copy) @@ -109,7 +109,13 @@ return; } } - /* else if (1 < r_lvl) warning*/ + else if (1 < r_lvl) { + status->t_warn += r_active; + if (status->t_warn < r_active) { + status->status = ST_OVERFLOW; + return; + } + } fmt_ok = 1; } } @@ -138,7 +144,7 @@ assert (0 != data); assert (0 != status); - fseek (data, -64, SEEK_END); /* Seek near the end of the file */ + fseek (data, -80, SEEK_END); /* Seek near the end of the file */ for (tok = fgetc (data); fsm < 4 && !feof (data); tok = fgetc (data)) { switch (tok) { @@ -160,13 +166,13 @@ fsm = 0; } } - - if (!feof (data)) { /* leading "## A" eaten above */ - read = fscanf (data, "ssertions = %u\n## FailedAssertions = %u", - &status->assert, &status->failed); + if (!feof (data)) { /* leading "## W" eaten above */ + read = fscanf (data, "arnings = %u\n## Assertions = %u\n" + "## FailedAssertions = %u", + &status->t_warn, &status->assert, &status->failed); } - if (2 != read) { + if (3 != read) { status->status = ST_FORMAT; } } Index: util/display.cpp =================================================================== --- util/display.cpp (revision 454401) +++ util/display.cpp (working copy) @@ -40,8 +40,8 @@ */ static void print_header_plain () { - puts ("NAME STATUS ASSERTS FAILED PERCNT USER " - "SYS WALL"); + puts ("NAME STATUS WARN ASSERTS FAILED PERCNT " + "USER SYS WALL"); } /** @@ -76,6 +76,8 @@ else printf (" 0"); + printf (" %4u", status->c_warn + status->l_warn + status->t_warn); + /* Print assetions, if any registered */ if (status->assert) { printf (" %7u %6u %5d%%", status->assert, status->failed, Index: util/target.h =================================================================== --- util/target.h (revision 454401) +++ util/target.h (working copy) @@ -98,6 +98,8 @@ char** argv; /**< Target argv array. */ unsigned timeout; /**< Allowed time for process execution. */ const char* data_dir; /**< Root dir for reference input/output files. */ + const char* c_warn; /**< Warning flag string when compiling. */ + const char* l_warn; /**< Warning flag string when linking. */ int compat; /**< Test suite compatability mode switch. */ rw_rlimit* core; rw_rlimit* cpu; @@ -141,7 +143,9 @@ const rw_timeval* user; /**< Elapsed user time spent in execution. */ const rw_timeval* sys; /**< Elapsed system time spent in execution. */ const rw_timeval* wall; /**< Wall clock time spent in execution. */ - unsigned warn; /**< Number of (test) warnings. */ + unsigned c_warn; /**< Number of compile warnings. */ + unsigned l_warn; /**< Number of link warnings. */ + unsigned t_warn; /**< Number of (test) warnings. */ unsigned assert; /**< Number of (test) assertions. */ unsigned failed; /**< Number of failed (test) assertions. */ }; Index: util/runall.cpp =================================================================== --- util/runall.cpp (revision 454401) +++ util/runall.cpp (working copy) @@ -28,6 +28,7 @@ #include /* for errno */ #include /* for exit(), free() */ #include /* for memcpy(), ... */ +#include /* for FILE, fopen(), ... */ #include /* for isspace */ #include @@ -193,11 +194,82 @@ } /** + Arbitrary constant controling static read buffer size. + + @see count_warnings +*/ +#define READ_BUF_LEN 64 + +/** + Compiler/linker output parser. + + This method tries to open the compiler/linker log file, based on the name + of a the target file that was generated by the compiler/linker. + Upon opening the file, it tries to count the number of warnings present. + + @param target name of target to parse the log for + @param counter pointer to the counter to count warnings in. + @param match string to match when detecting a warning. +*/ +static +void count_warnings (const char* const target, unsigned* counter, + const char* const match) +{ + size_t path_len; + char* tmp_name; + FILE* data; + + assert (0 != target); + assert (0 != counter); + + if (0 == match) + return; /* Don't have a match string, so don't tally warnings */ + + path_len = strlen (target); + tmp_name = (char*)RW_MALLOC (path_len + 5); + memcpy (tmp_name, target, path_len + 1); + strcat (tmp_name,".log"); + + data = fopen (tmp_name, "r"); + if (data) { + size_t pos = 0, limit = strlen (match); + while (!feof (data)) { + char buf [READ_BUF_LEN]; + size_t nread; /* bytes read */ + + /* First, read a block from the file into the buffer */ + nread = fread (buf, 1, sizeof buf, data); + if (ferror (data)) { + warn ("Error reading %s: %s\n", tmp_name, strerror (errno)); + break; + } + + /* Then, look for the search string within it. */ + for (size_t i = 0; i < nread; ++i) { + if (buf [i] != match [pos]) + pos = 0; + else if (++pos == limit) { + pos = 0; + ++*counter; + } + } + } + + fclose (data); + } + else if (ENOENT != errno) + warn ("Error opening %s: %s\n", tmp_name, strerror (errno)); + + free (tmp_name); +} + +/** Preflight check to ensure that target is something that should be run. This method checks to see if target exists and is theoretically executable. If a problem is detected, the condition is recorded in the status structure - and 0 is returned. + and 0 is returned. This method also attempts to determine the number of + compile and link warnings that occurred, using count_warnings. A special case is the situation where the target name ends in .o, indicating that the target only needed to compile, and doesn't need to @@ -212,31 +284,63 @@ { struct stat file_info; int exists = 1; + size_t path_len; + char* tmp_name; assert (0 != target); assert (0 != status); + path_len = strlen (target); + +#if !defined (_WIN32) && !defined (_WIN64) + /* Otherwise, check for the .o file on non-windows systems */ + tmp_name = (char*)RW_MALLOC (path_len + 3); + memcpy (tmp_name, target, path_len + 1); + strcat (tmp_name,".o"); +#else + /* Or the target\target.obj file on windows systems*/ + { + const char* const target_name = get_target (); + size_t target_len = strlen (target_name); + size_t tmp_len = path_len + target_len - 2; + /* - 2 comes from removing 4 characters (extra .exe) and adding 2 + characters (\ directory seperator and trailing null) */ + tmp_name = (char*)RW_MALLOC (tmp_len); + memcpy (tmp_name, target, path_len - 4); + tmp_name [path_len - 4] = default_path_sep; + memcpy (tmp_name + path_len - 3, target_name, target_len); + tmp_name [tmp_len - 4] = 'o'; + tmp_name [tmp_len - 3] = 'b'; + tmp_name [tmp_len - 2] = 'j'; + tmp_name [tmp_len - 2] = '\0'; + } +#endif /* _WIN{32,64} */ + + count_warnings (target, &status->l_warn, "warning:"); + count_warnings (tmp_name, &status->c_warn, "warning:"); + if (0 > stat (target, &file_info)) { if (ENOENT != errno) { warn ("Error stating %s: %s\n", target, strerror (errno)); status->status = ST_SYSTEM_ERROR; + free (tmp_name); return 0; } file_info.st_mode = 0; /* force mode on non-existant file to 0 */ exists = 0; /* note that it doesn't exist */ } + if (0 == (file_info.st_mode & (S_IXUSR | S_IXGRP | S_IXOTH))) { /* This is roughly equivlent to the -x bash operator. It checks if the file can be run, /not/ if we can run it */ - const size_t path_len = strlen (target); - char* tmp_name; #if 0 /* Disable .o target check as unused */ /* If target is a .o file, check if it exists */ if ('.' == target [path_len-1] && 'o' == target [path_len]) { if (!exists) status->status = ST_COMPILE; + free (tmp_name); return 0; } #endif @@ -244,34 +348,10 @@ /* If the target exists, it doesn't have valid permissions */ if (exists) { status->status = ST_EXECUTE_FLAG; + free (tmp_name); return 0; } -#if !defined (_WIN32) && !defined (_WIN64) - /* Otherwise, check for the .o file on non-windows systems */ - tmp_name = (char*)RW_MALLOC (path_len + 3); - memcpy (tmp_name, target, path_len + 1); - strcat (tmp_name,".o"); -#else - /* Or the target\target.obj file on windows systems*/ - { - const char* const target_name = get_target (); - size_t target_len = strlen (target_name); - size_t tmp_len = path_len + target_len - 2; - /* - 2 comes from removing 4 characters (extra .exe) and - adding 2 characters (\ directory seperator and trailing - null) */ - tmp_name = (char*)RW_MALLOC (tmp_len); - memcpy (tmp_name, target, path_len - 4); - tmp_name [path_len - 4] = default_path_sep; - memcpy (tmp_name + path_len - 3, target_name, target_len); - tmp_name [tmp_len - 4] = 'o'; - tmp_name [tmp_len - 3] = 'b'; - tmp_name [tmp_len - 2] = 'j'; - tmp_name [tmp_len - 2] = '\0'; - } -#endif /* _WIN{32,64} */ - if (0 > stat (tmp_name, &file_info)) { if (ENOENT != errno) { warn ("Error stating %s: %s\n", tmp_name, strerror (errno)); @@ -287,6 +367,7 @@ free (tmp_name); return 0; } + free (tmp_name); return 1; } --------------020907020800080407050202--