drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hongze Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Created] (DRILL-5028) Opening profiles page from web ui gets very slow when a lot of history files have been stored in HDFS or Local FS.
Date Thu, 10 Nov 2016 06:45:58 GMT
Hongze Zhang created DRILL-5028:
-----------------------------------

             Summary: Opening profiles page from web ui gets very slow when a lot of history
files have been stored in HDFS or Local FS.
                 Key: DRILL-5028
                 URL: https://issues.apache.org/jira/browse/DRILL-5028
             Project: Apache Drill
          Issue Type: Improvement
          Components: Functions - Drill
    Affects Versions: 1.8.0
            Reporter: Hongze Zhang
             Fix For: Future


We have a Drill cluster with 20+ Nodes and we store all history profiles in hdfs. Without
doing periodically cleans for hdfs, the profiles page gets slower while serving more queries.

Code from LocalPersistentStore.java uses fs.list(false, basePath) for fetching the latest
100 history profiles by default, I guess this operation blocks that page (Millions small files
can be stored in the basePath), maybe we can try some other ways to reach the same goal.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message