hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jothi Padmanabhan (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3638) Cache the iFile index files in memory to reduce seeks during map output serving
Date Wed, 17 Sep 2008 11:55:44 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12631744#action_12631744

Jothi Padmanabhan commented on HADOOP-3638:

During the offline discussions, Chris expressed his reservations on the dataStructure used
in IndexCache.java for synchronizing access to the IndexCache in the case where several threads
try to insert the information to the cache at the same time. The current logic uses a separate
HashMap <MapID, Lock>  to synchronize the access. Any suggestions on how to do this
better? The current code is 

    IndexRecord getIndexInformation () {

     indexInfo = cache.get(mapId);
    if (indexInfo == null) {
      indexInfo = readIndexFileToCache(fileName,mapId);
   // do some book keeping for lru
   return IndexInfo;

  IndexInformation readIndexFileToCache {
    synchronized (this) {
      lock = lockMap.get(mapId);
      if (lock == null) {
        lock = new Object();
        lockMap.put (mapId, lock);

    synchronized (lock) {

      info = cache.get(mapId);
      if (info != null) return info;
      //Code to Read Index Information from file
        cache.put(info.mapId, info);
    return info;

> Cache the iFile index files in memory to reduce seeks during map output serving
> -------------------------------------------------------------------------------
>                 Key: HADOOP-3638
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3638
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>    Affects Versions: 0.17.0
>            Reporter: Devaraj Das
>            Assignee: Jothi Padmanabhan
>             Fix For: 0.19.0
>         Attachments: hadoop-3638-v1.patch, hadoop-3638-v2.patch, hadoop-3638-v3.patch,
hadoop-3638-v4.patch, hadoop-3638-v5.patch
> The iFile index files can be cached in memory to reduce seeks during map output serving.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message