Stop Forwarding and Start POSTing
Feb. 25, 2022
Background
Most of our data contribution and parsing clients send emails to a dropbox (e.g [email protected]) using a rule-based email forwarding tool. But there is a better way.
Stop Forwarding and Start POSTing
Below are links to example scripts that utilize a new API endpoint /submit-email/ to programmatically send emails to us instead of forwarding to a role-based email address. The benefits of this methodology are manifold. For example, forwarding an email incurs a doubling of storage and bandwidth costs. By default, each forwarded email is saved in your Sent folder (i.e. stored on disk). Email providers have limits on the number of emails sent and received and on overall bandwidth use. Published limits for a few of them are linked below. By transmitting via a script you deploy, you'll have greater precision and control over which items to transmit and how to log such transmissions. Our API endpoint also logs the receipt of such transmissions for fail-safe record keeping.
Example Scripts
Data Contribution via IMAP
gist.github.com/empirasign/6f3feab4730c94be747723f0b56f6502
Data Contribution via Outlook
gist.github.com/empirasign/5aeb506978b8689f053ed21e4417c30d
Save the Fibers
The /submit-email/ endpoint now accepts gzip-encoded payloads. If using the requests library, it's a small change (see below) with potentially large bandwidth savings. In our experience, gzipping email reduces payload size by about 50%.
Code Snippet
# uncompressed API call
resp = requests.post(URL, json=post_data, proxies=_proxies_dict(PROXY_SERVER))
# https://gist.github.com/empirasign/6f3feab4730c94be747723f0b56f6502#file-imap_daq-py-L185
# compressed API call
resp = requests.post(URL,
data=zlib.compress(json.dumps(post_data).encode("utf-8"))
proxies=_proxies_dict(PROXY_SERVER)
headers={"Content-Encoding": "gzip"})
Link to full docs: www.empirasign.com/api-docs/
Coming Soon
The above Outlook script interacts directly with an Outlook client. We'll soon be releasing example scripts that interact directly with Exchange Server. Support for Exchange Web Services and the new Microsoft Graph API will be provided. Interacting directly with Exchange Server allows for easier handling of multiple contributor inboxes as well as code portability. Your data contribution scripts can run on any box and OS you chose. They don't need to run on the client machine or the Exchange Server machine. This is already the case if your back-end email services are IMAP-based.
Developer Resources
We have posted a Developer Resources page. This page is the best place to start when looking for help on anything technical or API-related.
www.empirasign.com/developer-resources/
Email Service Provider Bandwidth Limits
Office 365 Exchange Bandwidth Limits
docs.microsoft.com/en-us/office365/servicedescriptions/exchange-online-service-description/exchange-online-limits
Google Workspace Limits
2.5GB per day for IMAP. That's around 100K messages per day for corps (maybe 25K for Structured Products)
support.google.com/a/answer/1071518?hl=en
Fastmail Professional Limits
80K messages per day
www.fastmail.help/hc/en-us/articles/1500000277382-Account-limits#sending/receiving