hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xudong Cao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDDS-1785) OOM error in Freon due to the concurrency handling
Date Wed, 17 Jul 2019 08:26:00 GMT

    [ https://issues.apache.org/jira/browse/HDDS-1785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16886777#comment-16886777
] 

Xudong Cao commented on HDDS-1785:
----------------------------------

sizeof VolumeProcessor: 2621008

> OOM error in Freon due to the concurrency handling
> --------------------------------------------------
>
>                 Key: HDDS-1785
>                 URL: https://issues.apache.org/jira/browse/HDDS-1785
>             Project: Hadoop Distributed Data Store
>          Issue Type: Bug
>            Reporter: Elek, Marton
>            Assignee: Doroszlai, Attila
>            Priority: Blocker
>              Labels: pull-request-available
>          Time Spent: 50m
>  Remaining Estimate: 0h
>
> HDDS-1532 modified the concurrent framework usage of Freon (RandomKeyGenerator).
> The new approach uses separated tasks (Runnable) to create the volumes/buckets/keys.
> Unfortunately it doesn't work very well in some cases.
>  # When Freon starts it creates an executor with fixed number of threads (10)
>  # The first loop submits numOfVolumes (10) VolumeProcessor tasks to the executor
>  # The 10 threads starts to execute the 10 VolumeProcessor tasks
>  # Each VolumeProcessor tasks creates numOfBuckets (1000) BucketProcessor tasks. All
together 10000 tasks are submitted to the executor.
>  # The 10 threads starts to execute the first 10 BucketProcessor tasks, they starts to
create the KeyProcessor tasks: 500 000 * 10 tasks are submitted.
>  # At this point of the time no keys are generated, but the next 10 BucketProcessor tasks
are started to execute..
>  # To execute the first key creation we should process all the BucketProcessor tasks
which means that all the Key creation tasks (10 * 1000 * 500 000) are created and added to
the executor
>  # Which requires a huge amount of time and memory



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message