Discovering and fixing CVE-2021-33038 in Mailman3
By Kunal Mehta, Software Engineer
Note: No private messages were actually exposed to non-NDA individuals.
Wikimedia recently finished its migration from the old and outdated Mailman version 2 software to the newer and more modern Mailman version 3 software for our public and private mailing lists hosted at lists.wikimedia.org. We’ll have more to write about this migration, but for now, I want to discuss and explain a security issue we discovered in the migration process: CVE-2021-33038: private list archives would be public for the duration of the import, which could take anywhere from seconds to hours.
Discovery
We have around 800 mailing lists and figured out how to migrate them individually from Mailman2 to Mailman3. We asked for some early adopters with list owners who were willing to be guinea pigs while we ironed out some of the initial bugs. The wikimediacz-l mailing list volunteered to be one of our early migrants and was the first non-test private mailing list that kept archives we moved over. We try to do all of our server actions in a transparent manner, so Amir announced in our public #wikimedia-operations IRC channel (then on Freenode, now on Libera Chat) that he was moving some mailing lists over, so people opened up the web interface to take a look. Majavah, a trusted volunteer, noticed that the wikimediacz-l archives were public for a few minutes but private after the import concluded. Once we confirmed this was an issue, we opened a private security bug report in our Phabricator installation: T281402 (now public). We quickly verified using server access logs that no one else had accessed the archives while they were temporarily public. At this point, we paused the migration of private lists with archives until we could fix the issue.
One question worth asking is, “Why didn’t we notice this in our earlier testing?” We had set up a temporary test Mailman3 install called “lists-next.wikimedia.org”, and practiced the migration process there, including with a private test list. But there are two primary reasons we didn’t identify this as an issue earlier. First, both Amir and I, who did most of the testing, have “superuser” accounts in the Mailman web interface, which means we bypass all permission restrictions, so we wouldn’t see any permission denied error if the list was correctly set as private. Second, the test list we imported had only about 10 emails, which took about a second to import – not enough time for us to check that it stayed private for the duration of the import.
Identifying the cause
We are not running the latest versions of the Mailman3 software (we use Debian packages), so I wanted to be sure that this was still an issue in the latest version before we reported it as a security issue to the Mailman developers. I created a dummy mbox archive with thousands of blank messages and then ran through the migration process on our Mailman3 test server (hosted in Cloud VPS) carefully watching the database state throughout the process.
For background, whether a list is private or not is controlled by the Mailman3 Core daemon, which stores the archive policy as a numerical enum in the mailinglist database table. In the archive_policy column on that table, 0 means never archive, 1 is a private archive and 2 is a public archive. “Hyperkitty” is a Django application that provides the web interface for mailing list archives. Hyperkitty talks to Mailman3 Core over an internal REST API and stores whether a list is public or private in its own hyperkitty_mailinglist table, which has an archive_policy column using the same numerical format. At various points, hyperkitty will sync the archive policy and other settings from Mailman3 Core into its own database.
In my testing, I confirmed that the archive_policy was correctly set to private on the Mailman3 Core side, but while the import was in progress, hyperkitty would have it set to public, and only at the end of the import would it set it to private. After reading through the code it came down to two different code issues combining to create this security issue.
First, typically hyperkitty will sync with Mailman3 Core after each message it receives. This ensures that when the first message in a list is received by hyperkitty it’ll have the correct visibility (and list name, description, etc.). However, when doing a bulk import, it would be incredibly slow to sync after every message, when it’s unlikely that anything will have changed. So during the import, HYPERKITTY_BATCH_MODE is enabled, which skips the sync step.
Second, since the mailing list does not yet exist in hyperkitty’s database and the sync was skipped, it is created and saved with the default settings. The default archive_policy for a list is unfortunately public.
Fixing either issue would neutralize the bug, so I suggested doing both, first, explicitly syncing the mailing list with Mailman3 Core before importing any messages and changing the archive_policy default to private.
Remediation and disclosure
I reported the issue to the Mailman developers and we discussed a few other ways to address the issue. I worked on patch to change the archive_policy default since it would have been a one-line fix, however it caused a significant amount of test failures that were relying on the default. I switched to the other approach and added an explicit sync before beginning to import messages. After verifying it worked in our test instance, we tried it on lists.wikimedia.org and the archives stayed private for the duration of the import (and afterwards). This patch was merged into hyperkitty and will be included in the next release.
Discovery credit: Amir Sarabadani, Taavi Väänänen, Kunal Mehta.
Thanks to Abhilash Raj, Stephen J. Turnbull and Mark Sapiro for their assistance on the Mailman side and to Martin Urbanec for volunteering the wikimediacz-l list.
Timeline:
- April 28: Issue discovered
- May 4: Privately reported to Mailman developers
- May 6: Patched lists.wikimedia.org
- May 17: CVE-2021-33038 assigned
- May 17: Patch merged upstream and issue made public
- May 29: Debian Security Advisory 4922 issued
About this post
Featured image credit: Mailboxes in Greeley Hill, California, Ryanx7, CC BY-SA 4.0
