hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From malcolm <malcolm.kaval...@oracle.com>
Subject Re: Solaris Port
Date Wed, 10 Dec 2014 10:31:09 GMT
Hi Colin,

Thanks for the hints around JIRAs.

You are correct errno still exists, however sys_errlist does not.

Hadoop uses a function terror (defined in exception.c) which indexes 
sys_errlist by errno to return the error message from the array. This 
function is called 26 times in various places (in 2.2)

Originally, I thought to replace all calls to terror with strerror, but 
there can be issues with multi-threading (it returns a buffer which can 
be overwritten), so it seemed simpler just to recreate the sys_errlist 
message array.

There is also a multi-threaded version strerror_r where you pass the 
buffer as a parameter, but this would necessitate changing every call to 
terror with mutiple lines of code.

Sorry, I wasn't clear.

Also, I have been requested to ensure my port is available on 2.4, 
perceived as a more stable release. If I make changes to this branch are 
they automatically available for 2.6, or will I need multiple JIRAs ?

Thanks,
Malcolm

On 12/10/2014 10:45 AM, Colin McCabe wrote:
> Hi Malcolm,
>
> In general we file JIRAs for particular issues.  So if one issue is
> handling errlist on Solaris, that might be one JIRA.  Another issue
> might be handling socket write timeouts on Solaris.  And so on.  Most
> of these should probably be HADOOP tickets since they sound like they
> are mostly in the generic hadoop-common code.
>
> "solaris does not have errno" seems like a bold statement.  errno is
> part of POSIX, and Solaris is a POSIX os, right?  Am I way off base on
> this?
> I googled around and one of the first results I found talked about
> errno values on Solaris.
> http://www.pixelstech.net/article/1413273556-A-trick-of-building-multithreaded-application-on-Solaris
>   Perhaps I misunderstood what you meant by this statement.
>
> Anyway, please file JIRAs for any portability improvements you can think of!
>
> best,
> Colin
>
> On Mon, Dec 8, 2014 at 9:09 PM, malcolm <malcolm.kavalsky@oracle.com> wrote:
>> Hi Colin,
>>
>> A short summary of my changes are as follows:
>>
>> - Native C source files: added 5,  modified 6, requiring also changes to
>> CMakeLists.txt. Of course, all changes are "ifdeffed" for Solaris
>> appropriately and new files, are prefixed with solaris_ as well.
>>
>> For example, Solaris does not have errno, or errlist any more which are used
>> quite a lot in hadoop native code. I could have replaced all calls to use
>> strerror instead which would be compatible with Linux, however in the
>> interests of making minimal changes, I recreated and added these files from
>> a running Solaris machine instead.
>>
>> Another issue is that Solaris doesn't have the timeout option for sockets,
>> so I had to write my own solaris_read routine with timeout and added it to
>> DomainSocket.c . A few issues with lz4 on Sparc needed modification, and
>> some other OS specific issues: getgrouplist, container-executer (from yarn).
>>
>> - Some very minor changes were made to some Java source files (mainly tests
>> to get them to pass on Solaris)
>>
>> The above changes were made to 2.2, I will recheck everything against the
>> latest trunk, maybe some fixes aren't needed any more.
>>
>> I have generated a single patch file with all changes. Perhaps it would be
>> better to file multiple JIRAs for each change, perhaps grouped, one per
>> issue ? Or should I file a JIRA for each modified source file ?
>>
>> Thank you,
>> Malcolm
>>
>>
>> On 12/08/2014 09:53 PM, Colin McCabe wrote:
>>> Hi Malcolm,
>>>
>>> It's great that you are going to contribute!  Please make your patches
>>> against trunk.
>>>
>>> 2.2 is fairly old at this point.  It hasn't been the focus of
>>> development in more than a year.
>>>
>>> We don't use github or pull requests.
>>>
>>> Check the section on http://wiki.apache.org/hadoop/HowToContribute
>>> that talks about "Contributing your work".  Excerpt:
>>> "Finally, patches should be attached to an issue report in Jira via
>>> the Attach File link on the issue's Jira. Please add a comment that
>>> asks for a code review following our code review checklist. Please
>>> note that the attachment should be granted license to ASF for
>>> inclusion in ASF works (as per the Apache License ยง5)."
>>>
>>> As this says, you attach the patch file to a JIRA that you have
>>> created, and then hit "submit patch."
>>>
>>> I don't think a branch is required for this work since it is just
>>> build fixes, right?
>>>
>>> best,
>>> Colin
>>>
>>>
>>> On Mon, Dec 8, 2014 at 3:30 AM, malcolm <malcolm.kavalsky@oracle.com>
>>> wrote:
>>>> I have ported Hadoop  native libraries to Solaris 11 (both Sparc and
>>>> Intel )
>>>> Oracle have agreed to release my changes to the community so that Solaris
>>>> platforms can benefit.
>>>> Reading the HowToContribute and GitandHadoop documents, I am not 100%
>>>> clear
>>>> on how to get my changes into the main tree. I am also a Git(hub) newbie,
>>>> and was using svn previously.
>>>>
>>>> Please let me know if I am going the correct path:
>>>>
>>>> 1. I forked Hadoop on Github and downloaded a clone to my development
>>>> machine.
>>>>
>>>> 2. The changes I made were to 2.2.0, can I still add changes to this
>>>> branch,
>>>> and hopefully get them accepted or must I migrate my changes to 2.6 ? (On
>>>> the main Hadoop download page, 2.2 is still listed as the GA version )
>>>>
>>>> 3. I understand that I should create a new branch for my changes, and
>>>> then
>>>> generate pull requests after uploading them to Github.
>>>>
>>>> 4. I also registered  at Jira in the understanding that I need to
>>>> generate a
>>>> Jira number for my changes, and to name my branch accordingly ?
>>>>
>>>> Does all this make sense ?
>>>>
>>>> Thanks,
>>>> Malcolm
>>>>
>>>>


Mime
View raw message