hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chen Liang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-12504) Ozone: Improve SQLCLI performance
Date Wed, 11 Oct 2017 22:48:02 GMT

    [ https://issues.apache.org/jira/browse/HDFS-12504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16201118#comment-16201118
] 

Chen Liang commented on HDFS-12504:
-----------------------------------

Thanks [~yuanbo] for working on this! v001 patch looks pretty good to me. Just some minor
comments:
1. {{void accept(T item) throws IOException;}}, rename accept to something like batchConsume?
2. "This class is used to batch operate kv"  ==>  "This class is used to batch kv operations"
3. Change the log "Insert to sql container db, for container" to something like "Insert to
sql batch for container", and add some log to {{batchIterateStore}} such that we can see the
progress from log.

Also it would be ideal if we can have some simple benchmark results to see the performance
improvement, I will be looking into this too.


> Ozone: Improve SQLCLI performance
> ---------------------------------
>
>                 Key: HDFS-12504
>                 URL: https://issues.apache.org/jira/browse/HDFS-12504
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: ozone
>            Reporter: Weiwei Yang
>            Assignee: Yuanbo Liu
>              Labels: performance
>         Attachments: HDFS-12504-HDFS-7240.001.patch
>
>
> In my test, my {{ksm.db}} has *3017660* entries with total size of *128mb*, SQLCLI tool
runs over *2 hours* but still not finish exporting the DB. This is because it iterates each
entry and inserts that to another sqllite DB file, which is not efficient. We need to improve
this to be running more efficiently on large DB files.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message