spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Rosen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-15698) Ability to remove old metadata for structure streaming MetadataLog
Date Mon, 19 Sep 2016 19:29:20 GMT

    [ https://issues.apache.org/jira/browse/SPARK-15698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15504410#comment-15504410
] 

Josh Rosen commented on SPARK-15698:
------------------------------------

[~jerryshao] [~zsxwing], should 2.0.1 really be a target version for this or can I remove
it? Trying to do a bit of JIRA cleanup to make sure nothing important slips the 2.0.1 RC.

> Ability to remove old metadata for structure streaming MetadataLog
> ------------------------------------------------------------------
>
>                 Key: SPARK-15698
>                 URL: https://issues.apache.org/jira/browse/SPARK-15698
>             Project: Spark
>          Issue Type: Sub-task
>          Components: SQL, Streaming
>            Reporter: Saisai Shao
>            Priority: Minor
>
> Current MetadataLog lacks the ability to remove old checkpoint file, we'd better add
this functionality to the MetadataLog and honor it in the place where MetadataLog is used,
that will reduce unnecessary small files in the long running scenarios.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message