GSoC 2024 - New Email Service with Internationalization

This post is also available on the MetaBrainz blog

Personal Introduction

Hello! I’m Jade Ellis, AKA JadedBlueEyes. I’m an undergraduate Computer Science student at the University of Kent in England. I was thrilled to be selected as a participant in the Google Summer of Code (GSoC) 2024 program. My project focused on developing an email service with internationalisation support, MJML-based markup and text fallback to produce emails that are visually appealing and accessible to as many people as possible.

Reason for the Project

The MusicBrainz project sends thousands of emails every day to its diverse contributors. However, the existing email system was limited, having organically grown over the project’s lifetime. It could only send emails in English, and only in plain text. The experience that new contributors got could only be described as unfriendly. The overarching goal of my proposal was to improve this.

Aims of the Project

The primary goals of the project were to:

  1. Develop an email service capable of handling multiple languages.
  2. Send both HTML-formatted and plain-text multipart emails.
  3. Implement MJML-based formatting for responsive and accessible email design.
  4. Integrate the service with the existing MusicBrainz infrastructure.
  5. Significantly improve performance.

Proposed Solution

To achieve these goals, I proposed the development of a modular email service that:

  • Was written in Rust.
  • Used MJML (Mailjet Markup Language) via the mrml crate to create responsive email templates that render consistently across various email clients.
  • Could convert HTML into plain text using the html2text crate.
  • Supported internationalisation, allowing emails to be sent in different languages based on user preferences.
  • Integrated with MusicBrainz using a robust API, making it easy to send emails triggered by events within the MusicBrainz ecosystem.
  • Integrated with existing MetaBrainz translation infrastructure.
  • Included documentation and testing to ensure long-term maintainability.

Sequence diagram describing the communication in sending a single email

Stats

  • ~370 commits across 3 repositories
  • 16736 additions and 4062 deletions across 76 files
  • ~5 pull requests to other repositories
  • Technologies: Rust, SMTP, Docker, Prometheus, GitHub Actions

Results

Some of the templates I created

The mail service

https://github.com/metabrainz/mb-mail-service

The central part of the project is the mail service, which is written in Rust. It is responsible for rendering the email templates, converting them to HTML and plain text, and sending them via SMTP. By utilising the Axum web framework, it offers a REST API for sending emails, both individually and in bulk. It also includes other endpoints, encompassing various options for previewing emails and checking available locales.

The repository contains the email templates, which are written in MRMX. These templates are compiled into HTML using the mrml crate and then converted to plain text using the html2text crate. Every string can be translated using the mf1 crate, which allows dynamically switching between languages based on user preferences.

I worked with Simon Hartman on the email designs, and he was a massive help in getting the designs right. I’m super happy with how this turned out and excited to see the project deployed.

MRMX

https://github.com/JadedBlueEyes/mrmx

MRMX is a Rust crate for writing templates for the mrml crate using a syntax similar to JSX. It provides a macro for easily creating templates, allowing easy interpolation of XML-like syntax and Rust code.

To improve the utility of MRMX, I created a patch for the mrml crate to allow passing around groups of elements, which allows components in the templates, and added the features in mrmx to support that.

This was my first time writing Rust procedural macros, and I’m glad I did it. I learned a lot about how procedural macros work and how to write them.

Messageformat (mf1)

https://github.com/JadedBlueEyes/messageformat

mf1 is a Rust crate for formatting messages using the ICU MessageFormat syntax. ICU MessageFormat is a standard syntax for translation strings which allows interpolation, pluralisation and other functionality. The crate converts translation strings, including interpolation and switching based on a key, into Rust code and provides a macro for easily using those strings in Rust code.

I learned a lot about parsers with this project. Converting the ICU MessageFormat syntax into Rust code was a challenge which pushed my Rust skills to the next level and expanded my knowledge of procedural macros even more. This is a project that I’m proud of, and I know I will come back to in the future.

CI Pipelines & Automation

Making sure the project was high-quality and maintainable was a top priority, so one of the first things I did was set up a CI pipeline to automate testing and linting using GitHub Actions. This ran standard rust tooling, including clippy, rustfmt, and cargo-deny, as other checks with pre-commit and gitlint. It also ran the project’s test suite and generated code coverage reports. This allowed me to catch any issues early on and ensure that the code was always in a good state.

I also set up Renovate to automatically update dependencies, which helped keep the project up-to-date and prevent any issues caused by outdated dependencies. I configured Renovate to create a pull request with the updated dependencies, where CI checks would run and pass before merging the PR.

In addition, I set up various other CI pipelines, including one for merging translations into the project’s translation files and another for keeping the Docker Hub description up-to-date.

Docker and Deployment

I created a Dockerfile for the project, which allowed me to quickly build and deploy the service to a Docker container. This helped make the service easy to test and deploy on different environments, such as a local development environment or a production server. To help automate this, I set up a Github Actions workflow to automatically build and publish the Docker image to Docker Hub whenever a new tag is pushed to the repository.

I’ve also set up the builds to include an SBOM - a list of all the dependencies included in the build - which helps ensure the security of the project. The GitHub Actions workflow will also automatically sign an attestation to ensure the build is genuine and untampered with.

The docker image also includes a health check, which can be used to monitor the service’s availability and automatically restart the container if it fails.

Monitoring and integrations

I’ve set up a metrics endpoint for the project, allowing monitoring and alerting the service’s performance with Prometheus and Grafana. I’ve also included a Sentry integration to automatically capture and report any errors that occur in the service.

The project includes an example Docker Compose file, with Prometheus and Grafana preconfigured to monitor the service.

Translations

I set up a Weblate project to manage the translations for the project. This allows others to easily add new translations and contribute to the project’s localisation efforts. I ensured Weblate’s integrations were working correctly. I set up a Github Actions workflow to automatically merge translations into the project’s files whenever a new translation was added to Weblate.

Documentation

I also wrote documentation for the project, including a contribution guide and a deployment guide.

What’s Next?

While the project met all its expected outcomes, there are always more things I’d like to do - not least the extension suggestions from my proposal!

  • Reading editor subscriptions directly from the database to improve performance.
  • Handling errors more effectively, including dead email addresses.
  • Integrating with other MetaBrainz projects.
  • Adding more automation to the project to make it easier to maintain.
  • Adding pluralisation and interpolation support to the mf1 crate.

I feel like I’ve accomplished a lot in this project, yet there’s an endless list of improvements I would love to make.

Conclusion

Participating in GSoC 2024 was an incredible privilege. It’s an amazing experience that’s allowed me to contribute meaningfully to the MusicBrainz project and the Open Source community whilst expanding my skills. The email service I developed not only met the community’s immediate needs but also laid the groundwork for future enhancements. I’m excited to see how the service evolves and hope to continue contributing to open-source projects in the future. Thank you to my mentor, Michael Wiencek, and the MusicBrainz community for their support throughout this journey! It was a pleasure to work with you all.