hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Weiwei Yang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-12504) Ozone: Improve SQLCLI performance
Date Fri, 13 Oct 2017 05:47:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-12504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16203089#comment-16203089
] 

Weiwei Yang commented on HDFS-12504:
------------------------------------

bq.  if we can have some simple benchmark results to see the performance improvement,

Agree with this idea. Actually I suggest to add some log to record the time consumed on critical
paths, e.g insert a record to target DB, insert a batch of records to a target DB. So that
we can estimate the performance improvement given by this patch. [~yuanbo], does that make
sense to you?

> Ozone: Improve SQLCLI performance
> ---------------------------------
>
>                 Key: HDFS-12504
>                 URL: https://issues.apache.org/jira/browse/HDFS-12504
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: ozone
>            Reporter: Weiwei Yang
>            Assignee: Yuanbo Liu
>              Labels: performance
>         Attachments: HDFS-12504-HDFS-7240.001.patch
>
>
> In my test, my {{ksm.db}} has *3017660* entries with total size of *128mb*, SQLCLI tool
runs over *2 hours* but still not finish exporting the DB. This is because it iterates each
entry and inserts that to another sqllite DB file, which is not efficient. We need to improve
this to be running more efficiently on large DB files.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message