hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-12036) Consolidate all of the cmake extensions in one directory
Date Sat, 30 May 2015 02:03:18 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-12036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14565779#comment-14565779

Colin Patrick McCabe commented on HADOOP-12036:

bq. What about the test_bulk_crc32 app for example, is that multithreaded? If it's OK to use
-pthread on that then I think -pthread can safely be added to the global flags, as you suggest.

Some test programs aren't multithreaded, but there is no harm in passing {{\-pthread}} in
all cases.  The overhead is extremely minimal, and it brings the test programs closer to what
they're supposed to test (which is real hadoop performance / correctness in a multi-threaded

The main thing to keep in mind is that we don't want JNI linked into programs unless they
actually use JNI.  But that shouldn't be an issue based on my reading of the patch.

bq. I've put in a modification that allows you to preset a variable, GCC_SHARED_FLAGS. If
it's not set it currently defaults to "-g -O2 -Wall", if it is OK to do so I'll change that
to "-g -O2 -Wall -pthread -D_LARGEFILE_SOURCE", then the hadoop-yarn-project CMakeLists.txt
can simply set it appropriately before including HadoopCommon.cmake

Doesn't that mean that if you change {{-O2}} to {{-O3}} in the common code, yarn will be unaffected?
 I would prefer to override only the thing that needs to be different, aka {{_LARGEFILE_SOURCE}}.

bq. Sorry to sound like a stuck record but you still haven't given me what I consider to be
a satisfactory reason why _GNU_SOURCE should be set as a compiler command-line flag, even
on Linux. Just to be completely clear, I'm not intending to just remove it and have stuff
blow up on Linux, I'm intending to go through all the source and add the necessary #define/#undef
blocks so that all the Linux-specific code continues to work. Stopping this practice will
help keep the source portable, as I've explained at length in HADOOP-11997.

Everyone agrees that not setting _GNU_SOURCE (or setting it in a more limited way) *won't*
enforce portability.  For example, you still get the non-POSIX definition of {{strerror_r}}.
 You still have all the actually non-portable things in the places they are needed (i.e. cgroups,

On the other hand, everyone agrees that having a nightly build on Solaris / BSD / whatever
*will* enforce portability.

>From experience, trying to conditionally set {{_GNU_SOURCE}} is a huge pain in the rear
for everyone, leads to mysterious compile errors, and makes life difficult for new contributors.
 There's just no point.

If you put the energy and effort that is going into worrying about {{_GNU_SOURCE}} into creating
a Solaris or BSD nightly build, you would have a guarantee that portability would not regress.
 It is a waste of time to worry about {{_GNU_SOURCE}}, although if you want to avoid setting
it on Solaris (just to keep things "tidy") that is fine.

> Consolidate all of the cmake extensions in one directory
> --------------------------------------------------------
>                 Key: HADOOP-12036
>                 URL: https://issues.apache.org/jira/browse/HADOOP-12036
>             Project: Hadoop Common
>          Issue Type: Sub-task
>            Reporter: Allen Wittenauer
>            Assignee: Alan Burlison
>         Attachments: prototype01.txt
> Rather than have a half-dozen redefinitions, custom extensions, etc, we should move them
all to one location so that the cmake environment is consistent between the various native

This message was sent by Atlassian JIRA

View raw message