Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 3E09F200CAD for ; Tue, 6 Jun 2017 01:31:31 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 3CB86160BE1; Mon, 5 Jun 2017 23:31:31 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 812AB160BD4 for ; Tue, 6 Jun 2017 01:31:30 +0200 (CEST) Received: (qmail 84688 invoked by uid 500); 5 Jun 2017 23:31:29 -0000 Mailing-List: contact commits-help@ponymail.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@ponymail.incubator.apache.org Delivered-To: mailing list commits@ponymail.incubator.apache.org Received: (qmail 84678 invoked by uid 99); 5 Jun 2017 23:31:29 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Jun 2017 23:31:29 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 51A4A1857B1 for ; Mon, 5 Jun 2017 23:31:29 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -4.222 X-Spam-Level: X-Spam-Status: No, score=-4.222 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, RCVD_IN_DNSWL_HI=-5, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id UG8jKHES03Te for ; Mon, 5 Jun 2017 23:31:28 +0000 (UTC) Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with SMTP id 6C60F5F24C for ; Mon, 5 Jun 2017 23:31:27 +0000 (UTC) Received: (qmail 84675 invoked by uid 99); 5 Jun 2017 23:31:26 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Jun 2017 23:31:26 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 2E6A7DFF16; Mon, 5 Jun 2017 23:31:26 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: sebb@apache.org To: commits@ponymail.incubator.apache.org Message-Id: <9fcccd4e7e064a7c9e287478248dc3c2@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: incubator-ponymail git commit: Add some documentation Date: Mon, 5 Jun 2017 23:31:26 +0000 (UTC) archived-at: Mon, 05 Jun 2017 23:31:31 -0000 Repository: incubator-ponymail Updated Branches: refs/heads/master 15c355442 -> 0a8e8b8ff Add some documentation Project: http://git-wip-us.apache.org/repos/asf/incubator-ponymail/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-ponymail/commit/0a8e8b8f Tree: http://git-wip-us.apache.org/repos/asf/incubator-ponymail/tree/0a8e8b8f Diff: http://git-wip-us.apache.org/repos/asf/incubator-ponymail/diff/0a8e8b8f Branch: refs/heads/master Commit: 0a8e8b8ffba08ff262579a99c16dbe08c1bde2d3 Parents: 15c3554 Author: Sebb Authored: Tue Jun 6 00:31:00 2017 +0100 Committer: Sebb Committed: Tue Jun 6 00:31:00 2017 +0100 ---------------------------------------------------------------------- tools/generators.py | 45 ++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 42 insertions(+), 3 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-ponymail/blob/0a8e8b8f/tools/generators.py ---------------------------------------------------------------------- diff --git a/tools/generators.py b/tools/generators.py index b1d8e28..2ac2ca8 100644 --- a/tools/generators.py +++ b/tools/generators.py @@ -26,12 +26,34 @@ import time # Full generator: uses the entire email (including server-dependent data) # This is the recommended generator for single-node setups. def full(msg, body, lid, attachments): + """ + Full generator: uses the entire email + (including server-dependent data) + The id is almost certainly unique, + but different copies of the message are likely to have different headers, thus ids + + Parameters: + msg - the parsed message + body - the parsed text content (not used) + lid - list id + attachments - list of attachments (not used) + """ mid = "%s@%s" % (hashlib.sha224(msg.as_bytes()).hexdigest(), lid) return mid # Medium: Standard 0.9 generator - Not recommended for future installations. # See 'full' or 'redundant' generators instead. def medium(msg, body, lid, attachments): + """ + Standard 0.9 generator - Not recommended for future installations. + (does not generate sufficiently unique ids) + + Parameters: + msg - the parsed message (used to get the date) + body - the parsed text content + lid - list id + attachments - list of attachments (not used) + """ # Use text body xbody = body if type(body) is bytes else body.encode('ascii', 'ignore') # Use List ID @@ -61,6 +83,16 @@ def medium(msg, body, lid, attachments): # Unlike 'medium', this only makes use of the Date: header and not the archived-at, # as the archived-at may change from node to node (and will change if not in the raw mbox file) def redundant(msg, body, lid, attachments): + """ + Use data that is guaranteed to be the same across redundant setups + (does not guarantee to create unique ids) + + Parameters: + msg - the parsed message (used to get the date) + body - the parsed text content + lid - list id + attachments - list of attachments (uses the hashes) + """ # Use text body xbody = body if type(body) is bytes else body.encode('ascii', 'ignore') # Use List ID @@ -92,6 +124,16 @@ def redundant(msg, body, lid, attachments): # Old school way of making IDs def legacy(msg, body, lid, attachments): + """ + Original generator - DO NOT USE + (does not generate unique ids) + + Parameters: + msg - the parsed message (used to get the date) + body - the parsed text content + lid - list id + attachments - list of attachments (not used) + """ uid_mdate = 0 # Default if no date found try: mdate = email.utils.parsedate_tz(msg.get('date')) @@ -100,6 +142,3 @@ def legacy(msg, body, lid, attachments): pass mid = "%s@%s@%s" % (hashlib.sha224(body if type(body) is bytes else body.encode('ascii', 'ignore')).hexdigest(), uid_mdate, lid) return mid - - -