How to process Apple Mail Flags with imapfilter

I am looking into using imapfilter for spam filtering. imapfilter is a super powerful mail processing utility. It talks IMAP towards the mail server and binds IMAP functionality to an embedded Lua scripting environment. That allows manipulating a mailbox with Lua code.

One of the things I would like to do is work with mail flags in Apple Mail and interact with flagged mails with imapfilter. It is not straight forward how Apple Mail’s mail flags show up in IMAP and in imapfilter. Therefore I wrote some scratch code to understand how this mapping works. That allowed me to write imapfilter Lua code for handling Apple Mail flags. This post is about what I did to figure out the mapping and the results.

Test Case

Apple Mail allows attaching one of seven differently colored flags to a mail. Don't ask me why only seven and not eight. The way Apple Mail stores its flags would perfectly support another flag color.

In order to have a test case, I selected seven spam mails from my mailbox and tagged each of these mails with a flag, one color each. That allows to experiment with Lua code to have imapfilter processing these mails.

This is a screenshot from Apple Mail on my iPhone, showing the seven mails I have used to test the access via imapfilter.

List Flags

The first code snippet lists all IMAP flags for all flagged mails. is_newer(3) limits the number of emails to be processed to the mails I have prepared for my test case.

ls_flags.lua

options.info = false

account = IMAP {
    server = "imap.domain.tld",
    username = "name@domain.tld",
    password = "password",
    ssl = "ssl3"
}

results = account.INBOX:is_newer(3) * account.INBOX:is_flagged()

for _, message in ipairs(results) do
    mailbox, uid = table.unpack(message)

    from = mailbox[uid]:fetch_field("From")
    subject = mailbox[uid]:fetch_field("Subject")
    print(from)
    print(subject)

    flags = mailbox[uid]:fetch_flags()
    for _, flag in ipairs(flags) do
        print("  Flag:", flag)
    end
end

Running the code produces this output:

# imapfilter -c ./ls_flags.lua
From: Gesundheit-ZuHause <send05@ddhs.trappine.tips.gr>
Subject: =?UTF-8?B?U2llIHdlcmRlbiBzaWNoIHZvciBGcmV1bmRlbiB1bmQgRmFtaWxpZSBuaWNodCBtZWhyIGbDvHIgSWhyZW4gS8O2cnBlciBzY2jDpG1lbiBtw7xzc2Vu?=
Fetched the flags of bernd.viefhues@web.de@imap.web.de/INBOX[1646194378].
  Flag: \Flagged
  Flag: \Seen
  Flag: $MailFlagBit0
From: Oral B iO Series 9 Abteilung<info@peritary.com>
Subject: =?UTF-8?B?V2lyIGhhYmVuIGVpbmUgw5xiZXJyYXNjaHVuZyBmw7xyIEt1bmRlbiB2b24gT3JhbCBCIGlPIFNlcmllcyA5?=
Fetched the flags of bernd.viefhues@web.de@imap.web.de/INBOX[1646194370].
  Flag: \Flagged
  Flag: \Seen
  Flag: $MailFlagBit0
  Flag: $MailFlagBit2
From: Paypal <obu1ycm54d9@veratrate.com.com>
Subject: Erhalten Sie ein 1000&#8364; Prepaid Guthaben f&#252;r PAYPAL
Fetched the flags of bernd.viefhues@web.de@imap.web.de/INBOX[1646194366].
  Flag: \Flagged
  Flag: \Seen
  Flag: $MailFlagBit1
From: =?UTF-8?B?SW50ZWxsaWdlbnRlIMOcYmVyd2FjaHVuZ3Nsw7ZzdW5nZW4=?= <MzqTCZg@futaemabuta.org>
Subject:  Keine Installation, keine monatlichen Kosten!
Fetched the flags of bernd.viefhues@web.de@imap.web.de/INBOX[1646194362].
  Flag: \Flagged
  Flag: \Seen
  Flag: $MailFlagBit1
  Flag: $MailFlagBit2
From: Handgefertigte-Messer <send5043@pelene.trappine.tips.gr>
Subject: Invention:Sichern Sie sich JETZT Ihre japanischen Messer
Fetched the flags of bernd.viefhues@web.de@imap.web.de/INBOX[1646194373].
  Flag: \Flagged
  Flag: \Seen
From: KETOVIAX <g2ntyidml7j7a@setts.sligosaurs.com.com>
Subject: Achtung: Hochgradig süchtig machende KetoViax-Gummis im Inneren! Vorsicht bei der Anwendung!
Fetched the flags of bernd.viefhues@web.de@imap.web.de/INBOX[1646194369].
  Flag: \Flagged
  Flag: \Seen
  Flag: $MailFlagBit2
From: t-online <47rf7bgbg4w3rdu@veratrate.com.com>
Subject: Ihre Kontoinformationen gefahrdet
Fetched the flags of bernd.viefhues@web.de@imap.web.de/INBOX[1646194365].
  Flag: \Flagged
  Flag: \Seen
  Flag: $MailFlagBit0
  Flag: $MailFlagBit1

Yay, something works. Two observations: 1) the code selects the test case mails 2) seems that Apple Mail's flags are encoded through a set of IMAP flags - $MailFlagBit0, $MailFlagBit1 and $MailFlagBit2. It is quite easy to map the output of imapfilter to the mails screenshotted above.

That very much looks like three bits to encode eight states. In table form that looks like this:

A Filter

Lets build an imapfilter composite filter which returns all mails which have a green flag set in Apple Mail:

filter_green.lua

options.info = false

account = IMAP {
    server = "imap.domain.tld",
    username = "name@domain.tld",
    password = "password",
    ssl = "ssl3"
}

is_flag_green = function(mailbox, age)
    return mailbox:is_newer(age) * 
        mailbox:is_flagged() *
        mailbox:has_keyword("$MailFlagBit0") * 
        mailbox:has_keyword("$MailFlagBit1") * 
        mailbox:has_unkeyword("$MailFlagBit2")
end

function print_results(results)
    for _, message in ipairs(results) do
        mailbox, uid = table.unpack(message)

        from = mailbox[uid]:fetch_field("From")
        subject = mailbox[uid]:fetch_field("Subject")
        print(from)
        print(subject)
    end
end

print_results(is_flag_green(account.INBOX, 3))

Running the code produces this output:

# imapfilter -c ./filter_green.lua
From: t-online <47rf7bgbg4w3rdu@veratrate.com.com>
Subject: Ihre Kontoinformationen gefahrdet

Again, this works. The code dumps just the one mail tagged with a green flag.

Filter Functions

Let's expand this to all flag colors. First we need a data structure for mapping flag colors to IMAP keywords.

-- IMAP keywords for flags
local bit0 = "$MailFlagBit0"
local bit1 = "$MailFlagBit1"
local bit2 = "$MailFlagBit2"

-- Mapping table for color to IMAP mail flags/keywords to be set/not set.
local applemailflags = {
    ["red"]    = {[bit0] = false, [bit1] = false, [bit2] = false},
    ["blue"]   = {[bit0] = false, [bit1] = false, [bit2] = true },
    ["yellow"] = {[bit0] = false, [bit1] = true , [bit2] = false},
    ["gray"]   = {[bit0] = false, [bit1] = true , [bit2] = true },
    ["orange"] = {[bit0] = true,  [bit1] = false, [bit2] = false},
    ["purple"] = {[bit0] = true,  [bit1] = false, [bit2] = true },
    ["green"]  = {[bit0] = true,  [bit1] = true , [bit2] = false},
}

Now we can write a composite filter to tell imapfilter how to filter for a certain flag color:

--- Composite filter to search a mailbox for Apple Mail flags.
-- Color can be either "red", "blue", "yellow", "gray", "orange", "purple" or "green".
-- @param mailbox The mailbox to search.
-- @param color The Apple mail flag color string to search for.
-- @return table of messages in imapfilter results format.
is_applemailflag = function(mailbox, color)
    local flags = applemailflags[color]
    local results = mailbox:is_flagged()
    for _, bit in pairs({bit0, bit1, bit2}) do
        local value = applemailflags[color][bit]
        assert(value ~= nil, "Apple Mail flag bit lookup failed")
        if value then
            results = results * mailbox:has_keyword(bit)
        else
            results = results * mailbox:has_unkeyword(bit)
        end
    end
    return results
end

Example code for using this function in an imapfilter script:

results = is_applemailflag(account.INBOX, "green") * account.INBOX:is_newer(3)

This example returns all green-flagged mails which are younger than three days.

Telling ima-filter how to mark mails with a certain flag color works in a similar way. We use Lua's flexibility to extend the ima-filter-builtin Set object with a method for marking mails:

--- Processing method to mark a set of messages with an Apple Mail flag.
-- Color can be either "red", "blue", "yellow", "gray", "orange", "purple" or "green".
-- @param self Set of messages to apply flag to.
-- @param color Apple Mail flag color to appyl.
-- @return self (Set of messages) if OK, empty set if error.
function Set:mark_applemailflag(color)
    local flags = applemailflags[color]
    local add = {}
    local remove = {}
    for _, bit in pairs({bit0, bit1, bit2}) do
        local value = applemailflags[color][bit]
        assert(value ~= nil, "Apple Mail flag bit lookup failed")
        if value then
            table.insert(add, bit)
        else
            table.insert(remove, bit)
        end
    end
    r1 = self:mark_flagged()
    r2 = self:add_flags(add)
    r3 = self:remove_flags(remove)
    if r1 and r2 and r3 then return self else return Set {} end
end

Example code for setting a mail flag:

results = account.INBOX:is_newer(3)
results:mark_applemailflag("green")

This example marks all mails which are younger than three days with a green flag.

These functions do what I need for my own litte spamfilter tool. Mission accomplished.