hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "T Meyarivan (JIRA)" <j...@apache.org>
Subject [jira] Created: (HADOOP-7160) Configurable initial buffersize for getGroupDetails()
Date Thu, 03 Mar 2011 16:56:37 GMT
Configurable initial buffersize for getGroupDetails()
-----------------------------------------------------

                 Key: HADOOP-7160
                 URL: https://issues.apache.org/jira/browse/HADOOP-7160
             Project: Hadoop Common
          Issue Type: Improvement
          Components: native, security
    Affects Versions: 0.22.0
            Reporter: T Meyarivan


trunk/src/native/src/org/apache/hadoop/security/getGroup.c

"""
int getGroupDetails(gid_t group, char **grpBuf) {
  struct group * grp = NULL;
  size_t currBufferSize = sysconf(_SC_GETGR_R_SIZE_MAX);
  if (currBufferSize < 1024) {
    currBufferSize = 1024;
  }
  *grpBuf = NULL; 
  char *buf = (char*)malloc(sizeof(char) * currBufferSize);

  if (!buf) {
    return ENOMEM;
  }
  int error;
  for (;;) {
    error = getgrgid_r(group, (struct group*)buf,
                       buf + sizeof(struct group),
                       currBufferSize - sizeof(struct group), &grp);
    if(error != ERANGE) {
       break;
    }
    free(buf);
    currBufferSize *= 2;
    buf = malloc(sizeof(char) * currBufferSize);
    if(!buf) {
      return ENOMEM;
    }
...
"""

For large groups, this implies at least 2 queries for the group (number of queries = math.ceil(math.log(response_size/1024,
2)))

In the case of a large cluster with central user/group databases (exposed via LDAP etc), this
leads to unnecessary load on the central services. This can be alleviated to a large extent
by changing the initial buffer size to a configurable parameter

--

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message