phoenix-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "James Taylor (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PHOENIX-2995) Write performance severely degrades with large number of views
Date Sat, 13 Aug 2016 05:02:20 GMT

    [ https://issues.apache.org/jira/browse/PHOENIX-2995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15419808#comment-15419808
] 

James Taylor commented on PHOENIX-2995:
---------------------------------------

Thanks, [~tdsilva]. Looks very good. Couple of questions/comments:
- Would it be possible to move the {{new PhoenixConnection()}} outside of the synchronize
block?
{code}
     @Override
     public PhoenixConnection connect(String url, Properties info) throws SQLException {
-        checkClosed();
-        PMetaData metadata = latestMetaData;
-        if (metadata == null) {
-            throwConnectionClosedException();
+        synchronized (latestMetaDataLock) {
+            checkClosed();
+            PMetaData metadata = latestMetaData;
+            if (metadata == null) {
+                throwConnectionClosedException();
+            }
+    
+            return new PhoenixConnection(this, url, info, metadata);
         }
-
-        return new PhoenixConnection(this, url, info, metadata);
     }
{code}
- Just curious on the updateCache changes to load the physical table - was that being done
differently before?

Would you mind reviewing the new synchronization, [~samarthjain]?

> Write performance severely degrades with large number of views 
> ---------------------------------------------------------------
>
>                 Key: PHOENIX-2995
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-2995
>             Project: Phoenix
>          Issue Type: Bug
>            Reporter: Mujtaba Chohan
>            Assignee: Thomas D'Silva
>              Labels: Argus
>             Fix For: 4.8.1
>
>         Attachments: PHOENIX-2995-v2.patch, PHOENIX-2995-v3.patch, PHOENIX-2995.patch,
create_view_and_upsert.png, image.png, image2.png, image3.png, upsert_rate.png
>
>
> Write performance for each 1K batch degrades significantly when there are *10K* views
being written in random with default {{phoenix.client.maxMetaDataCacheSize}}. With all views
created, upsert rate remains around 25 seconds per 1K batch i.e. ~2K rows/min upsert rate.

> When {{phoenix.client.maxMetaDataCacheSize}} is increased to 100MB+ then view does not
need to get re-resolved and upsert rate gets back to normal ~60K rows/min.
> With *100K* views and {{phoenix.client.maxMetaDataCacheSize}} set to 1GB, I wasn't able
create all 100K views as upsert time for each 1K batch keeps on steadily increasing. 
> Following graph shows 1K batch upsert rate over time with variation of number of views.
Rows are upserted to random views {{CREATE VIEW IF NOT EXISTS ... APPEND_ONLY_SCHEMA = true,
UPDATE_CACHE_FREQUENCY=900000}} is executed before upsert statement.
> !upsert_rate.png!
> Base table is also created with {{APPEND_ONLY_SCHEMA = true, UPDATE_CACHE_FREQUENCY =
900000, AUTO_PARTITION_SEQ}}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message