From issues-return-184745-archive-asf-public=cust-asf.ponee.io@spark.apache.org Mon Feb 12 14:37:04 2018 Return-Path: X-Original-To: archive-asf-public@eu.ponee.io Delivered-To: archive-asf-public@eu.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by mx-eu-01.ponee.io (Postfix) with ESMTP id 1EF5618067B for ; Mon, 12 Feb 2018 14:37:04 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 0E719160C31; Mon, 12 Feb 2018 13:37:04 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 5C04C160C3F for ; Mon, 12 Feb 2018 14:37:03 +0100 (CET) Received: (qmail 43263 invoked by uid 500); 12 Feb 2018 13:37:02 -0000 Mailing-List: contact issues-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list issues@spark.apache.org Received: (qmail 43206 invoked by uid 99); 12 Feb 2018 13:37:02 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Feb 2018 13:37:02 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id DFBE8C20FE for ; Mon, 12 Feb 2018 13:37:01 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -109.511 X-Spam-Level: X-Spam-Status: No, score=-109.511 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, KAM_ASCII_DIVIDERS=0.8, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id VcP9h6la9f37 for ; Mon, 12 Feb 2018 13:37:01 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id 0151C5F640 for ; Mon, 12 Feb 2018 13:37:01 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 8EEB1E01A9 for ; Mon, 12 Feb 2018 13:37:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 4B2E421E84 for ; Mon, 12 Feb 2018 13:37:00 +0000 (UTC) Date: Mon, 12 Feb 2018 13:37:00 +0000 (UTC) From: "Marcelo Vanzin (JIRA)" To: issues@spark.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (SPARK-23394) Storage info's Cached Partitions doesn't consider the replications (but sc.getRDDStorageInfo does) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/SPARK-23394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16360757#comment-16360757 ] Marcelo Vanzin commented on SPARK-23394: ---------------------------------------- I talked to Attila offline, and to me it seems like the new UI is more correct. There are only 10 cached partitions, each one replicated to 2 executors; the table also reflects that (instead of the old UI, where the same block showed up twice). The only potential adjustment here would be to show the executor addresses instead of the executor IDs. In the context of what lead us here (SPARK-20659 / https://github.com/apache/spark/pull/20546#discussion_r167070392), I think that we should fix the tests that rely on the old code returning the total count including replication, so that they work with the new code that returns more accurate information. > Storage info's Cached Partitions doesn't consider the replications (but sc.getRDDStorageInfo does) > -------------------------------------------------------------------------------------------------- > > Key: SPARK-23394 > URL: https://issues.apache.org/jira/browse/SPARK-23394 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 2.3.0 > Reporter: Attila Zsolt Piros > Priority: Major > Attachments: Spark_2.2.1.png, Spark_2.4.0-SNAPSHOT.png, Storage_Tab.png > > > Start spark as: > {code:bash} > $ bin/spark-shell --master local-cluster[2,1,1024] > {code} > {code:scala} > scala> import org.apache.spark.storage.StorageLevel._ > import org.apache.spark.storage.StorageLevel._ > scala> sc.parallelize((1 to 100), 10).persist(MEMORY_AND_DISK_2).count > res0: Long = 100 > scala> sc.getRDDStorageInfo(0).numCachedPartitions > res1: Int = 20 > {code} > h2. Cached Partitions > On the UI at the Storage tab Cached Partitions is 10: > !Storage_Tab.png! . > h2. Full tab > Moreover the replicated partitions was also listed on the old 2.2.1 like: > !Spark_2.2.1.png! > But now it is like: > !Spark_2.4.0-SNAPSHOT.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org For additional commands, e-mail: issues-help@spark.apache.org