hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-10640) Implement Namenode RPCs in HDFS native client
Date Tue, 10 Jun 2014 18:53:02 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-10640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14026841#comment-14026841
] 

Colin Patrick McCabe commented on HADOOP-10640:
-----------------------------------------------

bq. Do we need to call free in hadoop_err_prepend.c for asprintf error cases? Docs say no
memory is allocated in this case.

The string being freed here was allocated further up in the function.  It's not allocated
by the (failed) asprintf.

bq. The hash table implementation has an unbounded while loop. Though, it will probably never
happen since we guarantee there will always be an open spot, would we add a terminal case
to it?

The hash table is never more than half full.  Check this code in {{htable_put}}:
{code}
+    // Re-hash if we have used more than half of the hash table
+    nused = htable->used + 1;
+    if (nused >= (htable->capacity / 2)) {
+        ret = htable_realloc(htable, htable->capacity * 2);
+        if (ret)
+            return ret;
+    }
+    htable_insert_internal(htable->elem, htable->capacity,
+                                htable->hash_fun, key, val);
{code}

I will add a comment to {{htable_insert_internal}} making this invariant clear.

bq. Should the above hash table be modified to allow custom hash functions in the future?
Modifications would include ensuring the hash function was within bounds, providing an interface,
etc.

Already done :)

{code}
+struct htable *htable_alloc(uint32_t size,
+                htable_hash_fn_t hash_fun, htable_eq_fn_t eq_fun)
{code}

You can supply your own hash function as the {{hash_fun}} argument.

bq. The config object seems to be using the builder pattern. Wouldn't it make sense to just
create a configuration object and provide 'set' and 'get' functions? Unless the configuration
object is immutable?

The configuration object is immutable once created.  I wanted to avoid multithreading problems
with get and set... we've had a lot of those with {{Configuration}} in Hadoop.  This also
simplifies the C code, since we can simply use strings from inside the {{hconf}} object without
worrying about whether someone is going to {{free}} them while we're using them.  This means,
for example, that we don't need to copy them inside {{hconf_get}}.  All the strings get freed
at the end, when the {{hconf}} is freed.  I'll add a comment that hconf is immutable.

> Implement Namenode RPCs in HDFS native client
> ---------------------------------------------
>
>                 Key: HADOOP-10640
>                 URL: https://issues.apache.org/jira/browse/HADOOP-10640
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: native
>    Affects Versions: HADOOP-10388
>            Reporter: Colin Patrick McCabe
>            Assignee: Colin Patrick McCabe
>         Attachments: HADOOP-10640-pnative.001.patch, HADOOP-10640-pnative.002.patch,
HADOOP-10640-pnative.003.patch
>
>
> Implement the parts of libhdfs that just involve making RPCs to the Namenode, such as
mkdir, rename, etc.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message