Description

The goal is to augment the Mail Sender API to be able to send large number of emails.

• Use Cases
• Proposed API
  • Java API
  • Scripting API
  • Alternative API for errors and results
• Implementation Ideas
• Old API Choices
  • Solution 1
  • Solution 1 Bis
  • Solution 2
  • Solution 3
  • Solution 4

Use Cases

• UC1: Be able to send mails asynchronously to a group (or list of users)
• UC2: Have an Admin UI to see the status of all sent mails (success/failure)
• UC3: Be able to resend a mail if it failed to be sent (from Admin UI)
• UC4: Email Throttling

Example of usages:

• Use the mail sending API for the "share page by email" feature to send to a group (or list of users)
• Use the mail sending API to send meeting invitations for the Meeting Application

Proposed API

Java API

@Role
public interface MailSender
{
    UUID send(Iterator<MimeMessage>, Session) throws MessagingException;
    UUID sendAsynchronously(Iterator<MimeMessage>, Session, MailListener listener);
}

Implementation Note: We pass an Iterator so that it scales and the MimeMessage are created one by one and sent. Thus we can support an infinite number of mails to be sent. We will also put the Iterator on the sending queue instead of putting the messages themselves. This allows to scale.

We'll have 3 iterator implementations by default:

• One for generating messages from a group reference:

  public class GroupMimeMessageIterator implements Iterator<MimeMessage>
  {
      public GroupMimeMessageIterator(DocumentReference groupReference, MimeMessageFactory factory, Map<String, Object> parameters)
      {
          ...
      }
  ... 
  }

• One for generating messages from a list of user references:

  public class UsersMimeMessageIterator implements Iterator<MimeMessage>
  {
      public UsersMimeMessageIterator(List<DocumentReference> userReferences, MimeMessageFactory factory, Map<String, Object> parameters)
      {
          ...
      }
  ... 
  }

• One for generating messages from serialized MimeMessages on the FS:

  public class SerializedFilesMimeMessageIterator implements Iterator<MimeMessage>
  {
      public SerializedFilesMimeMessageIterator(List<File> files)
      {
          ...
      }
  ... 
  }

Usage example to send a single email:

@Inject MailSenderConfiguration configuration;
@Inject @Named("text/html") MimeBodyPartFactory htmlPartFactory;
@Inject MailSender mailSender;

// Step 1: Create a JavaMail Session
Session session = Session.getInstance(configuration.getAllProperties(), new XWikiAuthenticator(configuration));

// Step 2: Create the Message to send
MimeMessage message = new MimeMessage(session);
message.setSubject("subject");
message.setRecipient(MimeMessage.RecipientType.TO, new InternetAddress("[email protected]"));

// Step 3: Add the Message Body
Multipart multipart = new MimeMultipart("mixed");
// Add HTML in the body, with a text alternative and attachments
Map<String, Object> parameters = new HashMap<>();
parameters.put("alternative", "text");
parameters.put("attachments", attachments);
multipart.addBodyPart(htmlPartFactory.create("some html here", parameters));
message.setContent(multipart);

// Step 4: Send the mail (asynchronously in this example) and use a memory listener to collect the results.
MemoryMailListener memoryListener = new MemoryMailListener();
UUID id = mailSender.sendAsynchronously(Arrays.asList(message), session, memoryListener);

// Step 5: Check for errors
Iterator<MailStatus> errorIterator = memoryListener.getErrors();
while (errorIterator.hasNext()) {
    MailStatus status = errorIterator.next();
    ... status.getException() ...
}

Note: In the scripting API we use a component for the Memory MailListener so we might need to have 2 implementations: a simple POJO for the Java API and a component for the Scripting API or use the same one but in this case, it must not be injected any component itself... Another possibility is to add result methods in the MailListener API (like getErrors(), getSuccesses(), etc) but it's not very nice... This part still needs tuning!

Usage example to send a template email from a list of users:

@Inject MailSenderConfiguration configuration;
@Inject MailSender mailSender;
@Inject @Named("template") MimeMessageFactory<DocumentReference> templateFactory;
@Inject UsersMimeMessageIteratorFactory usersIteratorFactory;
@Inject @Named("database") MailListener databaseListener;

... Create user list...

// Step 1: Create a JavaMail Session
Session session = Session.getInstance(configuration.getAllProperties(), new XWikiAuthenticator(configuration));

// Step 2: Create the message iterator to generate messages to send
Map<String, Object> parameters = new HashMap<>();
parameters.put("source", new DocumentReference("wiki", "space", "MailTemplatePage"));
Iterator<MimeMessage> messageIterator = usersIteratorFactory.create(users, templateFactory, parameters);

// Step 3: Send the mail (asynchronously in this example) and use a database listener to collect the results.
UUID id = mailSender.sendAsynchronously(messageIterator, session, databaseListener);

// Step 4: Check for errors
... use the Query Manager to do a database query using the UUID...

Generic use case when no MimeMessageFactory exist:

@Inject MailSender mailSender;
... Create Session...
... Create user list...

Iterator<MimeMessage> messageIterator = new Iterator<MimeMessage>() {
    ...
    public MimeMessage next()
    {
        ... use various MimeMessageBodyFactory here...
        return message;
    }
});
...

Scripting API

#set ($msg = $services.mailsender.createMessage(...))
...
#set ($id = $services.mailsender.send([$msg], "database")) 
#set ($id = $services.mailsender.send([$msg], "memory")) 
#set ($id = $services.mailsender.send([$msg])) <-- memory (default)
 
$services.mailsender.getErrors($id) (for "memory". When using "database" use a query with $batchid)

Alternative API for errors and results

#set ($msg = $services.mailsender.createMessage(...))
...
#set ($results = $services.mailsender.send([$msg, ...]))
...
$results.getStatusResults().getAll() <-- all statuses (ready to send, sent successfully, sent with error), returns Iterator<MailStatus>
$results.getStatusResults().getByStatus("error") <--- only mails not sent successfully, returns Iterator<MailStatus>
$results.isSent() <--- true when all mails have been sent (can be false when sending asynchronously)
$results.waitTillSent(timeout) <--- wait till all mails have been sent
$results.getBatchId() <--- returns the batch id

public interface MailResult
{
    /**
     * Wait till all messages on the sending queue have been sent (for this batch) before returning.
     *
     * @param timeout the maximum amount of time to wait in milliseconds
     */
    void waitTillSent(long timeout);

    /**
     * @return true if all the mails from this batch have been sent (successfully or not) or false otherwise
     */
    boolean isSent();

    /**
     * @return the batch id for this session of mail sending
     */
    UUID getBatchId();
}

public class ScriptMailResult implements MailResult
{
    private MailResult wrappedMailResult;

    private MailStatusResult mailStatusResults;

    public ScriptMailResult(MailResult wrappedMailResult, MailStatusResult mailStatusResults)
    {
        this.wrappedMailResult = wrappedMailResult;
        this.mailStatusResults = mailStatusResults;
    }

    public MailStatusResult getStatusResults()
    {
        return this.mailStatusResults;
    }

    @Override
    public void waitTillSent(long timeout)
    {
        this.wrappedMailResult.waitTillSent(timeout);
    }

    @Override
    public boolean isSent()
    {
        return this.wrappedMailResult.isSent();
    }

    @Override
    public UUID getBatchId()
    {
        return this.wrappedMailResult.getBatchId();
    }
}

For the Java API, we add the MailStatusResult getMailStatusResult() method to MailListener.

Consequences:

• MailSender.send() method won't return a UUID anymore but a MailResult object
• MailSender.waitTillSent() is removed
• MailSenderScriptService.getErrors() is removed

Implementation Ideas

• Modify the existing Mail modules instead of adding a new batch module
  • Rename MailResultListener into MailListener and add a onPrepare() method. Note: Initially we wanted to use Event and EventListener but it's less flexible than passing a MailListener to the send() method and it's possible to send events inside an implementation of MailListener if wanted.
  • Modify API to be able to send several messages and to return an id for the batch
  • Modify the mail sending queue implementation to store an Iterator<MimeMessage> instead of MimeMessage to scale (imagine a mail with a 10MB attachment sent to 1000 persons; if we were storing the MimeMessage we would need a lot of memory; whereas the iterator will construct the message on the go when they are ready to be sent and will be discarded just when they are sent.
• Have a storage module (xwiki-platform-mail-send-storage) to store mails and their statuses in the DB
  • The DB is needed to perform querying (for example in the Admin UI livetable, see below)
  • Implement a DatabaseMailListener to store and update mail statuses in the DB
  • Store actual emails in the permanent directory (since they can be large) with a reference to them in the DB. This is needed for resending failed mail later on. Use MimeMessage.writeTo to save the message and new MimeMessage(InputStream) to read it
• New xwiki-platform-mail-ui module to have an Admin UI view using a Livetable to display all email activity (this is why storing the mail statuses in the DB is important)
  • 4 columns in Livetable of email admin UI: Date of event, Id (batch id), To email address, Status
  • Add a Resend action in the Livetable to resend individual emails
  • Use Livetable mass action (when available) for resending a list of emails (new batch created for that)
• Throttling: 2 parameters batch size + wait time between batches

Old API Choices

There are various possibilities for the API. The main issue is that XWiki's CM doesn't support constructor injection which would be much nicer when constructing non singleton components...

Solution 1

    @Inject BatchMailSender batchSender;
    @Inject @Named("template") MimeMessageFactory<DocumentReference> templateFactory;
--> @Inject UsersMimeMessageIteratorFactory usersIteratorFactory;
    ... Create Session...
    ... Create user list...

    Map<String, Object> parameters = new HashMap<>();
    parameters.put("source", new DocumentReference("wiki", "space", "MailTemplatePage"));
--> Iterator<MimeMessage> messageIterator = usersIteratorFactory.create(users, templateFactory, parameters);
    batchSender.send(messageIterator, session);

Solution 1 Bis

    @Inject BatchMailSender batchSender;
    @Inject @Named("template") MimeMessageFactory<DocumentReference> templateFactory;
--> @Inject @Named("users") MimeMessageIteratorFactory usersIteratorFactory;
    ... Create Session...
    ... Create user list...

    Map<String, Object> parameters = new HashMap<>();
    parameters.put("source", new DocumentReference("wiki", "space", "MailTemplatePage"));
--> Iterator<MimeMessage> messageIterator = usersIteratorFactory.create(users, templateFactory, parameters);
    batchSender.send(messageIterator, session);

Note: This means using a vararg of Object in the MimeMessageIteratorFactory interface:

Iterator<MimeMessage> create(Object... parameters);

Solution 2

    @Inject BatchMailSender batchSender;
    @Inject @Named("template") MimeMessageFactory<DocumentReference> templateFactory;
--> @Inject @Named("users") Provider<MimeMessageIterator> usersIteratorProvider;
    ... Create Session...
    ... Create user list...

    Map<String, Object> parameters = new HashMap<>();
    parameters.put("source", new DocumentReference("wiki", "space", "MailTemplatePage"));
--> MimeMessageIterator messageIterator = usersIteratorProvider.get();
--> messageIterator.initialize(users, templateFactory, parameters);
    batchSender.send(messageIterator, session);

Note: This means using a vararg of Object in the MimeMessageIterator interface:

void initialize(Object... parameters);

Solution 3

    @Inject BatchMailSender batchSender;
    @Inject @Named("template") MimeMessageFactory<DocumentReference> templateFactory;
--> @Inject ComponentManager componentManager;
    ... Create Session...
    ... Create user list...

    Map<String, Object> parameters = new HashMap<>();
    parameters.put("source", new DocumentReference("wiki", "space", "MailTemplatePage"));
--> Iterator<MimeMessage> messageIterator = new UsersMimeMessageIterator(users, templateFactory, parameters, componentManager); 
    batchSender.send(messageIterator, session);

Solution 4

--> @Inject @Name("users") BatchMailSender usersBatchSender;
    @Inject @Named("template") MimeMessageFactory<DocumentReference> templateFactory;
    ... Create Session...
    ... Create user list...

    Map<String, Object> parameters = new HashMap<>();
    parameters.put("source", new DocumentReference("wiki", "space", "MailTemplatePage"));
--> usersBatchSender.send(users, templateFactory, parameters, session);

Note: This means using a vararg of Objects in the BatchMailSender interface:

void send(Object... parameters, Session session);