Chapter 8: Emails and SMS
Emails and SMS
Setting up email
Web2py provides the gluon.tools.Mail class to make it easy to send emails using web2py. One can define a mailer with
from gluon.tools import Mail
mail = Mail()
mail.settings.server = 'smtp.example.com:25'
mail.settings.sender = '[email protected]'
mail.settings.login = 'username:password'
Note, if your application uses Auth (discussed in the next chapter), the auth object will include its own mailer in auth.settings.mailer, so you can use that instead as follows:
mail = auth.settings.mailer
mail.settings.server = 'smtp.example.com:25'
mail.settings.sender = '[email protected]'
mail.settings.login = 'username:password'
You need to replace the mail.settings with the proper parameters for your SMTP server. Set mail.settings.login = None if the SMTP server does not require authentication. If you don't want to use TLS, set mail.settings.tls = False
For debugging purposes you can set
mail.settings.server = 'logging'and emails will not be sent but logged to the console instead.
Configuring email for Google App Engine
For sending emails from Google App Engine account:
mail.settings.server = 'gae'
At the time of writing web2py does not support attachments and encrypted emails on Google App Engine. Notice cron and scheduler do not work on GAE.
x509 and PGP Encryption
It is possible to send x509 (SMIME) encrypted emails using the following settings:
mail.settings.cipher_type = 'x509'
mail.settings.sign = True
mail.settings.sign_passphrase = 'your passphrase'
mail.settings.encrypt = True
mail.settings.x509_sign_keyfile = 'filename.key'
mail.settings.x509_sign_certfile = 'filename.cert'
mail.settings.x509_crypt_certfiles = 'filename.cert'
It is possible to send PGP encrypted emails. First of all you need to install the python-pyme package. Then you can use GnuPG (GPG) to create the key-files for the sender (take the email-address from mail.settings.sender) and put the files pubring.gpg and secring.gpg in a directory (e.g. "/home/www-data/.gnupg").
Use the following settings:
mail.settings.gpg_home = '/home/www-data/.gnupg/'
mail.settings.cipher_type = 'gpg'
mail.settings.sign = True
mail.settings.sign_passphrase = 'your passphrase'
mail.settings.encrypt = True
Sending emails
Once mail is defined, it can be used to send email via:
mail.send(to=['[email protected]'],
subject='hello',
# If reply_to is omitted, then mail.settings.sender is used
reply_to='[email protected]',
message='hi there')
Mail returns True if it succeeds in sending the email and False otherwise. A complete argument list for mail.send() is as follows:
send(self, to, subject='None', message='None', attachments=[],
cc=[], bcc=[], reply_to=[], sender=None, encoding='utf-8',
raw=True, headers={})
Note, to, cc, and bcc each take a list of email addresses.
sender defaults to None and in this case the sender will be set to mail.settings.sender.
headers is dictionary of headers to refine the headers just before sending the email. For example:
headers = {'Return-Path' : '[email protected]'}
Following are some additional examples demonstrating the use of mail.send().
Simple text email
mail.send('[email protected]',
'Message subject',
'Plain text body of the message')
HTML emails
mail.send('[email protected]',
'Message subject',
'<html>html body</html>')
If the email body starts with <html> and ends with </html>, it will be sent as a HTML email.
Combining text and HTML emails
The email message can be a tuple (text, html):
mail.send('[email protected]',
'Message subject',
('Plain text body', '<html>html body</html>'))
cc and bcc emails
mail.send('[email protected]',
'Message subject',
'Plain text body',
cc=['[email protected]', '[email protected]'],
bcc=['[email protected]', '[email protected]'])
Attachments
mail.send('[email protected]',
'Message subject',
'<html><img src="cid:photo" /></html>',
attachments = mail.Attachment('/path/to/photo.jpg', content_id='photo'))
Multiple attachments
mail.send('[email protected]',
'Message subject',
'Message body',
attachments = [mail.Attachment('/path/to/fist.file'),
mail.Attachment('/path/to/second.file')])
Sending SMS messages
Sending SMS messages from a web2py application requires a third party service that can relay the messages to the receiver. Usually this is not a free service, but it differs from country to country. We have tried a few of these services with little success. Phone companies block emails originating from these services since they are eventually used as a source of spam.
A better way is to use the phone companies themselves to relay the SMS. Each phone company has an email address uniquely associated with every cell-phone number, so SMS messages can be sent as emails to the phone number.
web2py comes with a module to help in this process:
from gluon.contrib.sms_utils import SMSCODES, sms_email
email = sms_email('1 (111) 111-1111','T-Mobile USA (tmail)')
mail.send(to=email, subject='test', message='test')
SMSCODES is a dictionary that maps names of major phone companies to the email address postfix. The sms_email function takes a phone number (as a string) and the name of a phone company and returns the email address of the phone.
Using the template system to generate messages
It is possible to use the template system to generate emails. For example, consider the database table
db.define_table('person', Field('name'))
where you want to send to every person in the database the following message, stored in a view file "message.html":
Dear {{=person.name}},
You have won the second prize, a set of steak knives.
You can achieve this in the following way
for person in db(db.person).select():
context = dict(person=person)
message = response.render('message.html', context)
mail.send(to=['[email protected]'],
subject='None',
message=message)
Most of the work is done in the statement
response.render('message.html', context)
It renders the view "message.html" with the variables defined in the dictionary "context", and it returns a string with the rendered email text. The context is a dictionary that contains variables that will be visible to the template file.
If the message starts with <html> and ends with </html>, the email will be an HTML email.
Note, if you want to include a link back to your website in an HTML email, you can use the URL function. However, by default, the URL function generates a relative URL, which will not work from an email. To generate an absolute URL, you need to specify the scheme and host arguments to the URL function. For example:
<a href="{{=URL(..., scheme=True, host=True)}}">Click here</a>
or
<a href="{{=URL(..., scheme='http', host='www.site.com')}}">Click here</a>
The same mechanism that is used to generate email text can also be used to generate SMS messages or any other type of message based on a template.
Sending messages using a background task
The operation of sending an email message can take up to several seconds because of the need to log into and communicate with a potentially remote SMTP server. To keep the user from having to wait for the send operation to complete, it is sometimes desirable to queue the email to be sent at a later time via a background task. As described in Chapter 4, this can be done by setting up a homemade task queue or using the web2py scheduler. Here we provide an example using a homemade task queue.
First, in a model file within our application, we set up a database model to store our email queue:
db.define_table('queue',
Field('status'),
Field('email'),
Field('subject'),
Field('message'))
From a controller, we can then enqueue messages to be sent by:
db.queue.insert(status='pending',
email='[email protected]',
subject='test',
message='test')
Next, we need a background processing script that reads the queue and sends the emails:
## in file /app/private/mail_queue.py
import time
while True:
rows = db(db.queue.status=='pending').select()
for row in rows:
if mail.send(to=row.email,
subject=row.subject,
message=row.message):
row.update_record(status='sent')
else:
row.update_record(status='failed')
db.commit()
time.sleep(60) # check every minute
Finally, as described in Chapter 4, we need to run the mail_queue.py script as if it were inside a controller in our app:
python web2py.py -S app -M -R applications/app/private/mail_queue.py
where -S app tells web2py to run "mail_queue.py" as "app", -M tells web2py to execute models.
Here we assume that the mail object referenced in "mail_queue.py" is defined in a model file in our app and is therefore available in the "mail_queue.py" script because of the -M option. Also notice that it is important to commit any change as soon as possible in order not to lock the database to other concurrent processes.
As noted in Chapter 4, this type of background process should not be executed via cron (except perhaps for cron @reboot) because you need to be sure that no more than one instance is running at the same time.
Note, one drawback to sending email via a background process is that it makes it difficult to provide feedback to the user in case the email fails. If email is sent directly from the controller action, you can catch any errors and immediately return an error message to the user. With a background process, however, the email is sent asynchronously, after the controller action has already returned its response, so it becomes more complex to notify the user of a failure.
Reading and managing email boxes (Experimental)
The IMAP adapter is intended as an interface with email IMAP servers to perform simple queries in the web2py DAL query syntax, so email read, search and other related IMAP mail services (as those implemented by brands like Google(r), and Yahoo(r) can be managed from web2py applications.
It creates its table and field names "statically", meaning that the developer should leave the table and field definitions to the DAL instance by calling the adapter .define_tables() method. The tables are defined with the IMAP server mailbox list information.
Connection
For a single mail account, this is the code recommended to start IMAP support at the app's model
# Replace user, password, server and port in the connection string
# Set port as 993 for SSL support
imapdb = DAL("imap://user:password@server:port", pool_size=1)
imapdb.define_tables()
Note that <imapdb>.define_tables() returns a dictionary of strings mapping DAL tablenames to the server mailbox names with the structure {<tablename>: <server mailbox name>, ...}, so you can get the actual mailbox name in the IMAP server.
If you want to set you own tablename/mailbox configuration and skip the automatic name configuration, you can pass a custom dictionary to the adapter in this way:
imapdb.define_tables({"inbox":"MAILBOX", "trash":"SPAM"})
To handle the different native mailbox names for the user interface, the following attributes give access to the adapter auto mailbox mapped names (which native mailbox has what table name and vice versa):
| Attribute | Type | Format |
| imapdb.mailboxes | dict | {<tablename>: <server native name>, ...} |
| imapdb.<table>.mailbox | string | "server native name" |
The first can be useful to retrieve IMAP query sets by the native email service mailbox
# mailbox is a string containing the actual mailbox name
tablenames = dict([(v, k) for k, v in imapdb.mailboxes.items()])
myset = imapdb(imapdb[tablenames[mailbox]])
Fetching mail and updating flags
Here's a list of IMAP commands you could use in the controller. For the examples, it's assumed that your IMAP service has a mailbox named INBOX, which is the case for Gmail(r) accounts.
To count today's unseen messages smaller than 6000 octets from the inbox mailbox do
q = imapdb.INBOX.seen == False
q &= imapdb.INBOX.created == request.now.date()
q &= imapdb.INBOX.size < 6000
unread = imapdb(q).count()
You can fetch the previous query messages with
rows = imapdb(q).select()
Usual query operators are implemented, including belongs
messages = imapdb(imapdb.INBOX.uid.belongs(<uid sequence>)).select()
Note: It's strongly advised that you keep the query results below a given data size threshold to avoid jamming the server with large select commands.
To perform faster email queries, it is recommended to pass a filtered set of fields:
fields = ["INBOX.uid", "INBOX.sender", "INBOX.subject", "INBOX.created"]
rows = imapdb(q).select(*fields)
The adapter knows when to retrieve partial message payloads (fields like content, size and attachments require retrieving the complete message data)
It is possible to filter query select results with limitby and sequences of mailbox fields
# Replace the arguments with actual values
myset.select(<fields sequence>, limitby=(<int>, <int>))
Say you want to have an app action show a mailbox message. First we retrieve the message (If your IMAP service supports it, fetch messages by uid field to avoid using old sequence references).
mymessage = imapdb(imapdb.INBOX.uid == <uid>).select().first()
Otherwise, you can use the message's id.
mymessage = imapdb.INBOX[<id>]
Note that using the message's id as reference is not recommended, because sequence numbers can change with mailbox maintenance operations as message deletions. If you still want to record references to messages (i.e. in another database's record field), the solution is to use the uid field as reference whenever supported, and retrieve each message with the recorded value.
Finally, add something like the following to show the message content in a view
{{=P(T("Message from"), " ", mymessage.sender)}}
{{=P(T("Received on"), " ", mymessage.created)}}
{{=H5(mymessage.subject)}}
{{for text in mymessage.content:}}
{{=DIV(text)}}
{{=TR()}}
{{pass}}
As expected, we can take advantage of the SQLTABLE helper to build message lists in views
{{=SQLTABLE(myset.select(), linkto=URL(...))}}
And of course, it's possible to feed a form helper with the appropriate sequence id value
{{=SQLFORM(imapdb.INBOX, <message id>, fields=[...])}}
The current adapter supported fields available are the following:
| Field | Type | Description |
| uid | string | |
| answered | boolean | Flag |
| created | date | |
| content | list:string | A list of text or html parts |
| to | string | |
| cc | string | |
| bcc | string | |
| size | integer | the amount of octets of the message* |
| deleted | boolean | Flag |
| draft | boolean | Flag |
| flagged | boolean | Flag |
| sender | string | |
| recent | boolean | Flag |
| seen | boolean | Flag |
| subject | string | |
| mime | string | The mime header declaration |
| string | The complete RFC822 message** | |
| attachments | list | Each non text decoded part as dictionary |
| encoding | string | The message's main detected charset |
*At the application side it is measured as the length of the RFC822 message string
WARNING: As row id's are mapped to email sequence numbers, make sure your IMAP client web2py app does not delete messages during select or update actions, to prevent updating or deleting different messages.
Standard CRUD database operations are not supported. There's no way of defining custom fields or tables and make inserts with different data types because updating mailboxes with IMAP services is usually reduced to posting flag updates to the server. Still, it's possible to access those flag commands through the DAL IMAP interface
To mark last query messages as seen
seen = imapdb(q).update(seen=True)
Here we delete messages in the IMAP database that have mails from mr. Gumby
deleted = 0
for tablename in imapdb.tables():
deleted += imapdb(imapdb[tablename].sender.contains("gumby")).delete()
It is possible also to mark messages for deletion instead of erasing them directly with
myset.update(deleted=True)