Mozilla Services Documentation

Release

Tarek Ziade

Sep 11, 2017

Contents

1 How To... 3

2 Services 21

3 Server-side development guide 121

4 Client Development 151

5 Miscellaneous 185

Mozilla Services Documentation, Release

Welcome to the Mozilla Services Documentation front page.

This site contains technical information about the various services and products provided by the Mozilla Services

Team. The documentation on this site is meant to be more declarative and static in nature.

Other information surrounding the products and services detailed on this site can be found at:

• Mozilla Services Team Wiki

• Mozilla Developer Network (MDN)

The wiki contains team info and updates. MDN contains content speciﬁc for helping developers.

To contribute to this site, see About this Website.

Contents 1

Mozilla Services Documentation, Release

2 Contents

CHAPTER 1

How To...

This section contains simple how-tos. If you want to see a new one, let us know in the mailing list.

Run your own Sync-1.1 Server

The Firefox Sync Server is deployed on our systems using RPM packaging, and we don’t provide any other packaging

or publish ofﬁcial RPMs yet.

The easiest way to install a Sync Server is to checkout our repository and run a build in-place. Once this is done, Sync

can be run behind any Web Server that supports the WSGI protocol.

Note: These instructions are for the sync server protocol used by Firefox 28 and earlier. Firefox 29 and later include

a new sync service that is incompatible with this server. For a server compatible with Firefox 29 and later, see Run

your own Sync-1.5 Server.

Prerequisites

The various parts are using Python 2.6 and Virtualenv. Make sure your system has them. Or install them:

• Python 2.6 downloads: http://python.org/download/releases/2.6.6

• Virtualenv: http://pypi.python.org/pypi/virtualenv

To run the server, you will also need to have these packages installed:

• python-dev

• make

• mercurial

• sqlite3

• openssl-dev

Mozilla Services Documentation, Release

For example, under a fresh Ubuntu, you can run this command to meet all requirements:

$ sudo apt-get install python-dev mercurial sqlite3 python-virtualenv libssl-dev

Building the server

Get the latest version at https://hg.mozilla.org/services/server-full and run the build command:

$ hg clone https://hg.mozilla.org/services/server-full

$ cd server-full

$ make build

This command will create an isolated Python environment and pull all the required dependencies in it. A bin directory

is created and contains a paster command that can be used to run the server, using the built-in web server.

Note: Occasionally the build may fail due to network issues that make PyPI inaccessible. If you receive an error

about “Could not ﬁnd suitable distribution”, try waiting a little while and then running the build again.

If you like, you can run the testsuite to make sure everything is working properly:

$ make test

If this gives you an error about “pysqlite2”, you may need to install the “pysqlite” package like so:

$ ./bin/pip install pysqlite

Basic Conﬁguration

The server is conﬁgured using an ini-like ﬁle to specify various runtime settings. The ﬁle “etc/sync.conf” will provide

a useful starting point”.

There is one setting that you must specify before running the server: the client-visible URL for the storage service

node. To ensure that the Registration and Node-Assignment ﬂow works correctly, this should be set to the URL at

which you will be running the server.

Open “etc/sync.conf”, locate and uncomment the following lines:

[nodes]

fallback_node = http://localhost:5000/

By default the server is conﬁgured to use a SQLite database for the storage and the user APIs, with the database ﬁle

stored at “/tmp/test.db”. You will almost certainly want to change this to a more permanent location:

[storage]

sqluri = sqlite:////path/to/database/file.db

[auth]

sqluri = sqlite:////path/to/database/file.db

Alternatively, consider using a different database backend as described in Using MYSQL or LDAP or ....

4 Chapter 1. How To...

Mozilla Services Documentation, Release

Running the Server

Now you can run the server using paster and the provided “development.ini” ﬁle:

$ bin/paster serve development.ini

Starting server in PID 29951.

serving on 0.0.0.0:5000 view at http://127.0.0.1:5000

Once the server is launched, you can run the Firefox Sync Wizard and choose http://localhost:5000 as your Firefox

Custom Sync Server.

You should then see a lot of output in the stdout, which are the calls made by the browser for the initial sync.

Note: As of October 2015, the Mozilla-hosted sync-1.1 service has been decommissioned, so you must perform some

additional conﬁguration to let Firefox use a self-hosted key-exchange server; see below.

Conﬁguring Firefox

While the sync setup dialog allows you to specify a custom storage server, it does not provide any UI for specifying a

custom key-exchange server. In order to pair with a second device, you will need to go to “about:conﬁg”, search for

“jpake.serverURL” and change its value to the URL of your server with a “/jpake” sufﬁx:

• services.sync.jpake.serverURL: http://localhost:5000/jpake

Updating the server

You should periodically update your code to make sure you’ve got the latest ﬁxes. The following commands will

update server-full in place:

$ cd /path/to/server-full

$ hg pull

$ hg update

$ make build

By default, the build command will checkout the latest released tags for each server product. If you need access to a

ﬁx that has not yet been released (or if you just want to live on the bleeding edge) then you can build the development

channel like so:

$ make build CHANNEL=dev

Note: Due to a change in how authentication is handled, users upgrading from a build made prior to January 2012

may need to migrate user accounts into a new database table. To do so:

1. Check that the [auth] section in your conﬁg ﬁle is using the “services.user.sql.SQLUser” backend.

2. Check if your database contains a “users” table.

3. If so, use the following migration script to move data into the “user” table:

deps/server-core/migrations/auth.sql_to_user.sql_migration.txt

1.1. Run your own Sync-1.1 Server 5

Mozilla Services Documentation, Release

Security Notes

File Permissions

The default conﬁguration of the server uses a ﬁle-based sqlite database, so you should carefully check that the permis-

sions on this ﬁle are appropriate for your setup. The ﬁle and its containing directory should be writable by the user

under which the server is running, and inaccessible to other users on the system.

You may like to set the umask of the server process to ensure that any ﬁles it creates are readable only by the appropriate

user. For example:

$ umask 007

$ bin/paster serve development.ini

Disabling New Users

The default conﬁguration of the server allows new users to create an account through Firefox’s builtin setup screen.

This is useful during initial setup, but it means that anybody could sync against your server if they know its URL.

You can disable creation of new accounts by setting auth.allow_new_users to false in the conﬁg ﬁle:

[auth]

allow_new_users = false

Using MYSQL or LDAP or ...

Instead of SQLite, you can use alternative backends:

• Open-LDAP to store the users

• A SQLAlchemy-compatible database, to store the sync data and/or the users

Sync has been tested on MySQL and Postgres.

In order to use a speciﬁc Database, you need to install the required headers, and the required Python library in the

local Python environment.

See http://www.sqlalchemy.org/docs/core/engines.html#supported-dbapis

For example, to run everything in MySQL:

1. install libmysqlclient-dev and mysql-server

2. install Mysql-Python by running bin/easy_install Mysql-Python

3. change the conﬁguration ﬁle located at etc/sync.conf

For #3, see Conﬁguring the application.

For SQL databases, the code will create three tables:

• user: contains the user accounts, mapping email to numeric id.

• collections: contains collection names for each user, by numeric id.

• wbo: contains individual sync records for each user, by numeric id.

6 Chapter 1. How To...

Mozilla Services Documentation, Release

Running behind a Web Server

The built-in server should not be used in production, as it does not really support a lot of load.

If you want to set up a production server, you can use different web servers that are compatible with the WSGI protocol.

For example:

• Apache combined with mod_wsgi

• NGinx with Gunicorn or uWSGI

• lighttpd with ﬂup, using the fcgi or scgi protocol

Note: Remember, you must set the nodes.fallback_node option to the client-visible URL of your sync server.

For example, if your server will be located at http://example.com/ff-sync/, the fallback node should be set to this value

in your conﬁg ﬁle:

[nodes]

fallback_node = http://example.com/ff-sync/

Apache + mod_wsgi

Here’s an example of an Apache 2.2 setup that uses mod_wsgi:

Order deny,allow

Allow from all

</Directory>

<VirtualHost \

:80>

ServerName example.com

DocumentRoot /path/to/sync

WSGIProcessGroup sync

WSGIDaemonProcess sync user=sync group=sync processes=2 threads=25

WSGIPassAuthorization On

WSGIScriptAlias / /path/to/sync/sync.wsgi

CustomLog /var/log/apache2/example.com-access.log combined

ErrorLog /var/log/apache2/example.com-error.log

</VirtualHost>

Here’s the equivalent setup for Apache 2.4, which uses a different syntax for acess control:

Require all granted

</Directory>

<VirtualHost \

:80>

ServerName example.com

DocumentRoot /path/to/sync

WSGIProcessGroup sync

WSGIDaemonProcess sync user=sync group=sync processes=2 threads=25

WSGIPassAuthorization On

WSGIScriptAlias / /path/to/sync/sync.wsgi

CustomLog /var/log/apache2/example.com-access.log combined

ErrorLog /var/log/apache2/example.com-error.log

</VirtualHost>

1.1. Run your own Sync-1.1 Server 7

Mozilla Services Documentation, Release

We provide a sync.wsgi ﬁle for your convenience in the repository. Before running Apache, edit the ﬁle and check

that it loads the the right .ini ﬁle with its full path.

Nginx + Gunicorn

Tested with debian stable/squeeze

1. First install gunicorn in the server-full python version:

$ cd /usr/src/server-full

$ bin/easy_install gunicorn

2. Then enable gunicorn in the developement.ini:

[server:main]

use = egg:gunicorn

host = 127.0.0.1

port = 5000

workers = 2

timeout = 60

3. Edit etc/sync.conf:

[nodes]

fallback_node = https://www.yourserver.net/some/path/

4. Finally edit your nginx vhost ﬁle:

server {

listen 443 ssl;

server_name sync.example.com;

ssl_certificate /path/to/your.crt;

ssl_certificate_key /path/to/your.key;

location / {

proxy_pass_header Server;

proxy_set_header Host $http_host;

proxy_redirect off;

proxy_set_header X-Real-IP $remote_addr;

proxy_set_header X-Scheme $scheme;

proxy_connect_timeout 10;

proxy_read_timeout 120;

proxy_pass http://localhost:5000/;

}

5. After restarting your nginx and server-full you should be able to use the sync server behind your nginx installa-

tion

lighttpd + ﬂup + fcgi

Tested under Gentoo.

1. Make sure you have the following packages installed:

• virtualenv

8 Chapter 1. How To...

Mozilla Services Documentation, Release

• mercurial

With Gentoo use:

emerge -avuDN virtualenv mercurial

1. Install ﬂup in the server-full python version:

$ cd /usr/src/server-full

$ bin/easy_install flup

4. I had to edit the Makeﬁle to take out the memcache dependency. YMMV.

5. Edit development.ini:

[server:main]

use = egg:Flup#fcgi_thread

host = 0.0.0.0

port = 5000

Be sure to remove the “use_threadpool” and “threadpool_workers” options from this section, since fcgi does not

support them.

6. Edit etc/sync.conf:

[storage]

backend = syncstorage.storage.sql.SQLStorage

sqluri = sqlite:////usr/src/server-full/weave_storage

create_tables = true

[auth]

backend = services.user.sql.SQLUser

sqluri = sqlite:////usr/src/server-full/weave_user

create_tables = true

[nodes]

fallback_node = https://www.yourserver.net/some/path/

7. Edit your lighttpd.conf:

server.modules += ( "mod_fastcgi" )

fastcgi.server = ( "/some/path" => ((

"host" => "127.0.0.1",

"port" => 5000,

"idle-imeout" => 32,

"check-local" => "disable",

"disable-time" => 1,

"fix-root-scriptname" => "enable"

))

)

Be sure to not add a trailing slash after “/some/path”, otherwise you will get a 404 error.

8. Start the Python server:

/usr/src/server-full/paster serve /usr/src/server-full/development.ini --daemon

9. Restart your lighttpd:

1.1. Run your own Sync-1.1 Server 9

Mozilla Services Documentation, Release

/etc/init.d/lighttpd restart

Troubleshooting

Most issues with the server are caused by bad conﬁguration. If your server does not work properly, the ﬁrst thing to

do is to visit about:sync-log in Firefox to see if there’s any error.

You will see a lot of logs and if the sync failed probably an error.

Misconﬁgured storage node

If the last successful call is ﬁnishing like this:

2011-02-24 11:17:57 Net.Resource DEBUG GET success 200 http://server/user/1.

˓→0/.../node/weave

But is not followed by:

2011-02-24 11:17:57 Service.Main DEBUG cluster value = http://server/

2011-02-24 11:17:57 Service.Main DEBUG Caching URLs under storage user base:

˓→http://server/.../

2011-02-24 11:17:57 Net.Resource DEBUG GET success 200 http://server/.../

˓→info/collections

It probably means that your server fallback_node option is not properly conﬁgured. See the previous section.

Getting a lot of 404

Check your server logs and make sure your VirtualHost is properly conﬁgured. Looking at the server log might help.

Getting some 500 errors

Check your server logs and look for some tracebacks. Also, make sure your server-full code is up-to-date by running

make build

Some common errors:

• KeyError: “Unknown fully qualiﬁed name for the backend: ‘sql”’

This error means that your backend conﬁguration is outdated. Use the fully qualiﬁed names described in the

previous sections.

• Various datatype-related errors

This could indicate that your webserver’s own authentication system is interacting badly with the sync server’s

own system. You may need to e.g. disable apache’s basic auth system.

Firefox says the server URL is invalid

Check that you have entered the full URL, including a leading “http://” or “https://” component.

Check that you’re not running your server on a port number that is commonly used for other services, such as port 22

(used by ssh) or port 6000 (used by X11). Firefox may prevent outgoing HTTP connections to these ports for security

reasons.

10 Chapter 1. How To...

Mozilla Services Documentation, Release

The current list of blocked ports can be viewed at http://dxr.mozilla.org/mozilla-central/netwerk/base/src/nsIOService.

cpp.html#l70.

Can’t get it to work

Ask for help:

• on IRC (irc.mozilla.org) in the #sync channel

• in our Mailing List: https://mail.mozilla.org/listinfo/sync-dev

Run your own Sync-1.5 Server

The Firefox Sync Server is deployed on our systems using RPM packaging, and we don’t provide any other packaging

or publish ofﬁcial RPMs yet.

The easiest way to install a Sync Server is to checkout our repository and run a build in-place. Once this is done, Sync

can be run behind any Web Server that supports the WSGI protocol.

Important Notes

These instructions are for the sync-1.5 server protocol used by the the new sync service in Firefox 29 and later. For a

server compatible with earlier versions of Firefox, see Run your own Sync-1.1 Server.

The new sync service uses Firefox Accounts for user authentication, which is a separate service and is not covered by

this guide.

Note: By default, a server set up using this guide will defer authentication to the Mozilla-hosted accounts server at

https://accounts.ﬁrefox.com.

You can safely use the Mozilla-hosted Firefox Accounts server in combination with a self-hosted sync storage server.

The authentication and encryption protocols are designed so that the account server does not know the user’s plaintext

password, and therefore cannot access their stored sync data.

Alternatively, you can also Run your own Firefox Accounts Server to control all aspects of the system. The process for

doing so is currently very experimental and not well documented.

Prerequisites

The various parts are using Python 2.7 and Virtualenv. Make sure your system has them, or install them:

• Python 2.7 downloads: http://python.org/download/releases/2.7.6

• Virtualenv: http://pypi.python.org/pypi/virtualenv

To build and run the server, you will also need to have these packages installed:

• python-dev

• make

• git

• c and c++ compiler

1.2. Run your own Sync-1.5 Server 11

Mozilla Services Documentation, Release

For example, under a fresh Ubuntu, you can run this command to meet all requirements:

$ sudo apt-get install python-dev git-core python-virtualenv g++

Building the server

Get the latest version at https://github.com/mozilla-services/syncserver and run the build command:

$ git clone https://github.com/mozilla-services/syncserver

$ cd syncserver

$ make build

This command will create an isolated Python environment and pull all the required dependencies in it. A local/bin

directory is created and contains a gunicorn command that can be used to run the server.

If you like, you can run the testsuite to make sure everything is working properly:

$ make test

Basic Conﬁguration

The server is conﬁgured using an ini-like ﬁle to specify various runtime settings. The ﬁle “syncserver.ini” will provide

a useful starting point.

There is one setting that you must specify before running the server: the client-visible URL for the service. Open

”./syncserver.ini” and locate the following lines:

[syncserver]

public_url = http://localhost:5000/

The default value of “public_url” will work for testing purposes on your local machine. For ﬁnal deployment, change

it to the external, publicly-visible URL of your server.

By default the server will use an in-memory database for storage, meaning that any sync data will be lost on server

restart. You will almost certainly want to conﬁgure a more permanent database, which can be done with the “sqluri”

setting:

[syncserver]

sqluri = sqlite:////path/to/database/file.db

This setting will accept any SQLAlchemy database URI; for example the following would connect to a mysql server:

[syncserver]

sqluri = pymysql://username:password@db.example.com/sync

Running the Server

Now you can run the server using gunicorn and the provided “syncserver.ini” ﬁle. The simplest way is to use the

Makeﬁle like this:

$ make serve

Or if you’d like to pass additional arguments to gunicorn, like this:

12 Chapter 1. How To...

Mozilla Services Documentation, Release

$ local/bin/gunicorn --threads 4 --paste syncserver.ini

Once the server is launched, you need to tell Firefox about its location.

To conﬁgure desktop Firefox to talk to your new Sync server, go to “about:conﬁg”, search for “iden-

tity.sync.tokenserver.uri” and change its value to the URL of your server with a path of “token/1.0/sync/1.5”:

• identity.sync.tokenserver.uri: http://sync.example.com/token/1.0/sync/1.5

Alternatively, if you’re running your own Firefox Accounts server, and running Firefox 52 or later, see the documen-

tation on how to Run your own Firefox Accounts Server for how to conﬁgure your client for both Sync and Firefox

Accounts with a single preference.

Since Firefox 33, Firefox for Android has supported custom sync servers. To conﬁgure Android Firefox 44 and later to

talk to your new Sync server, just set the “identity.sync.tokenserver.uri” exactly as above before signing in to Firefox

Accounts and Sync on your Android device.

Important: after creating the Android account, changes to “identity.sync.tokenserver.uri” will be ignored. (If you

need to change the URI, delete the Android account using the Settings > Sync > Disconnect... menu item, update

the pref, and sign in again.) Non-default TokenServer URLs are displayed in the Settings > Sync panel in Firefox for

Android, so you should be able to verify your URL there.

Prior to Firefox 44, a custom add-on was needed to conﬁgure Firefox for Android. For Firefox 43 and earlier, see the

blog post How to connect Firefox for Android to self-hosted Firefox Account and Firefox Sync servers.

(Prior to Firefox 42, the TokenServer preference name for Firefox Desktop was “services.sync.tokenServerURI”.

While the old preference name will work in Firefox 42 and later, the new preference is recommended as the old

preference name will be reset when the user signs out from Sync causing potential confusion.)

Further Conﬁguration

Once the server is running and Firefox is syncing successfully, there are further conﬁguration options you can tweak

in the “syncserver.ini” ﬁle.

The “secret” setting is used by the server to generate cryptographically-signed authentication tokens. It is blank by

default, which means the server will randomly generate a new secret at startup. For long-lived server installations this

should be set to a persistent value, generated from a good source of randomness. An easy way to generate such a value

on posix-style systems is to do:

$ head -c 20 /dev/urandom | sha1sum db8a203aed5fe3e4594d4b75990acb76242efd35 -

Then copy-paste the value into the conﬁg ﬁle like so:

[syncserver]

...other settings...

secret = db8a203aed5fe3e4594d4b75990acb76242efd35

The “allow_new_users” setting controls whether the server will accept requests from previously-unseen users. It is

allowed by default, but once you have conﬁgured Firefox and successfully synced with your user account, additional

users can be disabled by setting:

[syncserver]

...other settings...

allow_new_users = false

1.2. Run your own Sync-1.5 Server 13

Mozilla Services Documentation, Release

Updating the server

You should periodically update your code to make sure you’ve got the latest ﬁxes. The following commands will

update syncserver in place:

$ cd /path/to/syncserver

$ git stash # to save any local changes to the config file

$ git pull # to fetch latest updates from github

$ git stash pop # to re-apply any local changes to the config file

$ make build # to pull in any updated dependencies

Running behind a Web Server

The built-in server should not be used in production, as it does not really support a lot of load.

If you want to set up a production server, you can use different web servers that are compatible with the WSGI protocol.

For example:

• Apache combined with mod_wsgi

• NGinx with Gunicorn or uWSGI

Note: Remember, you must set the syncserver.public_url option to the client-visible URL of your server.

For example, if your server will be located at http://example.com/ff-sync/, the public_url should be set to this value in

your conﬁg ﬁle:

[syncserver]

public_url = http://example.com/ff-sync/

Apache + mod_wsgi

Here’s an example of an Apache 2.2 setup that uses mod_wsgi:

Order deny,allow

Allow from all

</Directory>

<VirtualHost

:80>

ServerName example.com

DocumentRoot /path/to/syncserver

WSGIProcessGroup sync

WSGIDaemonProcess sync user=sync group=sync processes=2 threads=25 python-path=/

˓→path/to/syncserver/local/lib/python2.7/site-packages/

WSGIPassAuthorization On

WSGIScriptAlias / /path/to/syncserver/syncserver.wsgi

CustomLog /var/log/apache2/example.com-access.log combined

ErrorLog /var/log/apache2/example.com-error.log

</VirtualHost>

Here’s the equivalent setup for Apache 2.4, which uses a different syntax for access control:

14 Chapter 1. How To...

Mozilla Services Documentation, Release

Require all granted

</Directory>

<VirtualHost

:80>

ServerName example.com

DocumentRoot /path/to/syncserver

WSGIProcessGroup sync

WSGIDaemonProcess sync user=sync group=sync processes=2 threads=25 python-path=/

˓→path/to/syncserver/local/lib/python2.7/site-packages/

WSGIPassAuthorization On

WSGIScriptAlias / /path/to/syncserver/syncserver.wsgi

CustomLog /var/log/apache2/example.com-access.log combined

ErrorLog /var/log/apache2/example.com-error.log

</VirtualHost>

We provide a syncserver.wsgi ﬁle for your convenience in the repository. Before running Apache, edit the ﬁle and

check that it loads the the right .ini ﬁle with its full path.

Nginx + Gunicorn

Tested with debian stable/squeeze

1. First install gunicorn in the syncserver python environment:

$ cd /usr/src/syncserver

$ local/bin/easy_install gunicorn

2. Then enable gunicorn in the syncserver.ini ﬁle:

[server:main]

use = egg:gunicorn

host = 127.0.0.1

port = 5000

workers = 2

timeout = 60

3. Finally edit your nginx vhost ﬁle:

server {

listen 443 ssl;

server_name sync.example.com;

ssl_certificate /path/to/your.crt;

ssl_certificate_key /path/to/your.key;

location / {

proxy_set_header Host $http_host;

proxy_set_header X-Forwarded-Proto $scheme;

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

proxy_set_header X-Real-IP $remote_addr;

proxy_redirect off;

proxy_read_timeout 120;

proxy_connect_timeout 10;

proxy_pass http://127.0.0.1:5000/;

}

1.2. Run your own Sync-1.5 Server 15

Mozilla Services Documentation, Release

5. After restarting your nginx and syncserver you should be able to use the sync server behind your nginx installa-

tion

Note: If you see errors about a mismatch between public_url and application_url, you may need to tell gunicorn that

it should trust the X-Forwarded-Proto header being sent by nginx. Add the following to the gunicorn conﬁguration

in syncserver.ini:

forwarded_allow_ips =

Note: If you see errors about “client sent too long header line” in your nginx logs, you may need to conﬁgure nginx

to allow large client header buffers by adding this to the nginx conﬁg:

large_client_header_buffers 4 8k;

Nginx + uWSGI

1. Install uWSGI and its Python 2 plugin

2. Start it with the following options:

uwsgi --plugins python27 --manage-script-name \

--mount /<location>=/path/to/syncserver/syncserver.wsgi \

--socket /path/to/uwsgi.sock

3. Use the following nginx conﬁguration:

location /<location>/ {

include uwsgi_params;

uwsgi_pass unix:/path/to/uwsgi.sock;

}

Things that still need to be Documented

• how to restrict new-user signups

• how to interoperate with a self-hosted accounts server

• periodic pruning of expired sync data

Asking for help

Don’t hesitate to jump online and ask us for help:

• on IRC (irc.mozilla.org) in the #sync channel

• in our Mailing List: https://mail.mozilla.org/listinfo/sync-dev

16 Chapter 1. How To...

Mozilla Services Documentation, Release

Run your own Firefox Accounts Server

The Firefox Accounts server is deployed on our systems using RPM packaging, and we don’t provide any other

packaging or publish ofﬁcial builds yet.

Note: This guide is preliminary and vastly incomplete. If you have any questions or ﬁnd any bugs, please don’t

hesitate to drop by the IRC channel or mailing list and let us know.

Note: You might also be interested in this Docker-based self-hosting guide (use at your own risk).

The Firefox Accounts server is hosted in git and requires nodejs. Make sure your system has these, or install them:

• git: http://git-scm.com/downloads

• nodejs: http://nodejs.org/download

A self-hosted Firefox Accounts server requires two components: an auth-server that manages the accounts database,

and a content-server that hosts a web-based user interface.

Clone the fxa-auth-server repository and follow the README to deploy your own auth-server:

• https://github.com/mozilla/fxa-auth-server/

Clone the fxa-content-server repository and follow the README to deploy your own content-server:

• https://github.com/mozilla/fxa-content-server/

Now direct Firefox to use your servers rather than the default, Mozilla-hosted ones. The procedure varies a little

between desktop and mobile Firefox, and may not work on older versions of the browser.

For desktop Firefox version 52 or later:

• Enter “about:conﬁg” in the URL bar

• Right click anywhere on the “about:conﬁg” page and choose New > String

• Enter “identity.fxaccounts.autoconﬁg.uri” for the name, and your content-server URL for the value.

• Restart Firefox for the change to take effect.

Note that this must be set prior to loading the sign-up or sign-in page in order to take effect, and its effects are reset on

sign-out.

For Firefox for iOS version 9.0 or later:

• Go to Settings.

• Tap on the Version number 5 times.

• Tap on “Advance Account Settings”

• Enter your content-server URL

• Toggle “Use Custom Account Service” to on.

For Firefox for Android version 44 or later:

• Enter “about:conﬁg” in the URL bar,

• Search for items containing “fxaccounts”, and edit them to use your self-hosted URLs:

– use your auth-server URL to replace “api.accounts.ﬁrefox.com” in the following settings: - iden-

tity.fxaccounts.auth.uri

1.3. Run your own Firefox Accounts Server 17

Mozilla Services Documentation, Release

– use your content-server URL to replace “accounts.ﬁrefox.com” in the following settings: - iden-

tity.fxaccounts.remote.webchannel.uri - webchannel.allowObject.urlWhitelist

– optionally, use your oauth- and proﬁle-server URLs to replace “{oauth,proﬁle}.accounts.ﬁrefox.com”

in - identity.fxaccounts.remote.proﬁle.uri - identity.fxaccounts.remote.oauth.uri

Important: after creating the Android account, changes to “identity.fxaccounts” prefs will be ignored. (If you need

to change the prefs, delete the Android account using the Settings > Sync > Disconnect... menu item, update the

pref(s), and sign in again.) Non-default Firefox Account URLs are displayed in the Settings > Sync panel in Firefox

for Android, so you should be able to verify your URL there.

Since the Mozilla-hosted sync servers will not trust assertions issued by third-party accounts servers, you will also

need to run your own sync-1.5 server.

Please note that the fxa-content-server repository includes graphics and other assets that make use of Mozilla trade-

marks. If you are doing anything other than running unmodiﬁed copies of the software for personal use, please review

the Mozilla Trademark Policy and Mozilla Branding Guidelines:

• https://www.mozilla.org/en-US/foundation/trademarks/policy/

• http://www.mozilla.org/en-US/styleguide/identity/mozilla/branding/

You can ask for help:

• on IRC (irc.mozilla.org) in the #fxa channel

• in our Mailing List: https://mail.mozilla.org/listinfo/dev-fxacct

Conﬁgure your Sync server for TLS

Firefox for Android versions 39 and up request the following protocols and cipher suites, depending on the Android

OS version.

The use of AES128 in preference to AES256 is driven by power and CPU concerns.

Cipher Suites

Android 20+:

• TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256

• TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256

• TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256

• TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA

• TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384

• TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384

• TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA

• TLS_DHE_RSA_WITH_AES_128_CBC_SHA

• TLS_RSA_WITH_AES_128_CBC_SHA

18 Chapter 1. How To...

Mozilla Services Documentation, Release

Android 11+:

• TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA

• TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA

• TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA

• TLS_DHE_RSA_WITH_AES_128_CBC_SHA

• TLS_RSA_WITH_AES_128_CBC_SHA

Android 9+ (Gingerbread):

• TLS_DHE_RSA_WITH_AES_128_CBC_SHA

• TLS_DHE_DSS_WITH_AES_128_CBC_SHA

• TLS_DHE_RSA_WITH_AES_128_CBC_SHA

• TLS_RSA_WITH_AES_128_CBC_SHA

Protocols

API 9 through 15 support only TLSv1.0. Modern versions of Android support all versions of TLS, v1.0 through v1.2.

We intend to eliminate TLSv1.0 on suitable Android versions as soon as possible.

No version of Firefox for Android beyond 38 supports SSLv3 for Sync.

1.4. Conﬁgure your Sync server for TLS 19

Mozilla Services Documentation, Release

20 Chapter 1. How To...

CHAPTER 2

Services

This section contains documentation about the services we operate, like their API speciﬁcations

If you want to see the big picture, check Overview of the services. Or jump to the detailed documentation for each

service:

Firefox Accounts Server

The Firefox Accounts server provides a centralized database of all user accounts for accessing Mozilla-hosted services.

It replaces the sync-speciﬁc registration and secure-registration services.

Firefox Accounts support is included in Firefox version 29 and later.

By default, Firefox will use Mozilla’s hosted accounts server at https://accounts.ﬁrefox.com. This conﬁguration will

work well for most use-cases, including for those who want to self-host a storage server. User who want to minimize

their dependency on Mozilla-hosted services may also self-host an accounts server, but this setup is incompatible with

other Mozilla-hosted services.

• Protocol documentation: https://github.com/mozilla/fxa-auth-server/blob/master/docs/api.md

• API server code: https://github.com/mozilla/fxa-auth-server/

• Web interface code: https://github.com/mozilla/fxa-content-server/

Storage Service

Goal of the Service

The storage server provides web services that can be used to store and retrieve any kind of data. It’s used to power

Firefox Sync.

If you are using our hosted services with your own client code, please read the Term of Services.

Mozilla Services Documentation, Release

Documentation content

Storage API v1.0

This document describes the Sync Server Storage API, version 1.0. It has been replaced by Storage API v1.1.

Weave Basic Object (WBO)

A Weave Basic Object is the generic wrapper around all items passed into and out of the Weave server. The Weave

Basic Object has the following ﬁelds:

Parameter Default Max Description

id required 64 An identifying string. For

a user, the id must be

unique for a WBO within

a collection, though ob-

jects in different collec-

tions may have the same

ID. Ids should be ASCII

and not contain commas.

parentid none 64 The id of a parent ob-

ject in the same collection.

This allows for the cre-

ation of hierarchical struc-

tures (such as folders).

predecessorid none 64 The id of a predecessor

in the same collection.

This allows for the cre-

ation of linked-list-esque

structures.

modiﬁed time submitted ﬂoat

2 decimal places

The last-modiﬁed date, in

seconds since 1970-01-01

(UNIX epoch time see

Set by the server.

sortindex none 256K A string containing a

JSON structure encap-

sulating the data of the

record. This structure

is deﬁned separately for

each WBO type. Parts

of the structure may

be encrypted, in which

case the structure should

also specify a record for

decryption.

payload none 256K

Weave Basic Objects and all data passed into the Weave Server should be utf-8 encoded.

Sample:

{

"id": "B1549145-55CB-4A6B-9526-70D370821BB5",

http://www.ecma-international.org/publications/standards/Ecma-262.htm ecma-262

22 Chapter 2. Services

Mozilla Services Documentation, Release

"parentid": "88C3865F-05A6-4E5C-8867-0FAC9AE264FC",

"modified": "2454725.98",

"payload": "{\"encryption\":\"http://server/prefix/version/user/crypto-meta/

˓→B1549145-55CB-4A6B-9526-70D370821BB5\", \"data\": \"a89sdmawo58aqlva.

˓→8vj2w9fmq2af8vamva98fgqamff...\"}"

}

Collections

Each WBO is assigned to a collection with other related WBOs. Collection names may only contain alphanumeric

characters, period, underscore and hyphen. Collections supported at this time are:

• bookmarks

• history

• forms

• prefs

• tabs

• passwords

Additionally, the following collections are supported for internal Weave client use:

• clients

• crypto

• keys

• meta

URL Semantics

Weave URLs follow, for the most part, REST semantics. Request and response bodies are all JSON-encoded.

The URL for Weave Storage requests is structured as follows:

https://<server name>/<api pathname>/<version>/<username>/<further instruction>

Component Mozilla Default Description

server name (deﬁned by user account

node)

the hostname of the server

pathname (none) the preﬁx associated with the service on the box

version 1.0 The API version. May be integer or decimal

username (none) The name of the object (user) to be manipulated

further

instruction

(none) The additional function information as deﬁned in the paths

below

Weave uses HTTP basic auth (over SSL, so as to maintain password security). If the auth username does not match

the username in the path, the server will issue an Error Response.

The Weave API has a set of Weave Response Codes to cover errors in the request or on the server side. The format of

a successful response is deﬁned in the appropriate request method section.

2.2. Storage Service 23

Mozilla Services Documentation, Release

GET

https://*server*/*pathname*/*version*/*username*/info/collections

Returns a hash of collections associated with the account, along with the last modiﬁed timestamp for each

collection.

https://*server*/*pathname*/*version*/*username*/info/collection_counts

Returns a hash of collections associated with the account, along with the total number of items for each

collection.

https://*server*/*pathname*/*version*/*username*/info/quota

Returns a tuple containing the user’s current usage (in K) and quota.

https://*server*/*pathname*/*version*/*username*/storage/*collection*

Returns a list of the WBO ids contained in a collection. This request has additional optional parameters:

• ids: Returns the ids for objects in the collection that are in the provided comma-separated list.

• predecessorid: Returns the ids for objects in the collection that are directly preceded by the id given.

Usually only returns one result.

• parentid: Returns the ids for objects in the collection that are the children of the parent id given.

• older: Returns only ids for objects in the collection that have been last modiﬁed before the date

given.

• newer: Returns only ids for objects in the collection that have been last modiﬁed since the date

given.

• full: If deﬁned, returns the full WBO, rather than just the id.

• index_above: If deﬁned, only returns items with a higher sortindex than the value speciﬁed.

• index_below: If deﬁned, only returns items with a lower sortindex than the value speciﬁed.

• limit: Sets the maximum number of ids that will be returned.

• offset: Skips the ﬁrst n ids. For use with the limit parameter (required) to paginate through a result

set.

• sort:: sorts before getting - ‘oldest’ - Orders by modiﬁcation date (oldest ﬁrst) - ‘newest’ - Orders

by modiﬁcation date (newest ﬁrst) - ‘index’ - Orders by the sortindex descending (highest weight

ﬁrst)

https://*server*/*pathname*/*version*/*username*/storage/*collection*/*id*

Returns the WBO in the collection corresponding to the requested id

Alternate Output Formats

Two alternate output formats are available for multiple record GET requests. They are triggered by the presence of the

appropriate format in the Accept header (with application/whoisi taking precedence)

• application/whoisi: each record consists of a 32-bit integer, deﬁning the length of the record, followed by the

json record for a wbo

• application/newlines: each record is a separate json object on its own line. Newlines in the body of the json

object are replaced by ‘u000a’

24 Chapter 2. Services

Mozilla Services Documentation, Release

APIs

PUT

https://*server*/*pathname*/*version*/*username*/storage/*collection*/*id*

Adds the WBO deﬁned in the request body to the collection. If the WBO does not contain a payload, it will only

update the provided metadata ﬁelds on an already deﬁned object.

The server will return the timestamp associated with the modiﬁcation.

POST

https://*server*/*pathname*/*version*/*username*/storage/*collection*

Takes an array of WBOs in the request body and iterates over them, effectively doing a series of atomic PUTs with the

same timestamp.

Returns a hash of successful and unsuccessful saves, including guidance as to possible errors:

{“modiﬁed”:1233702554.25,”success”:[”{GXS58IDC}12”,”{GXS58IDC}13”,”{GXS58IDC}15”,”{GXS58IDC}16”,”{GXS58IDC}18”,”{GXS58IDC}19”],”failed”:{“{GXS58IDC}11”:[”invalid

parentid”],”{GXS58IDC}14”:[”invalid parentid”],”{GXS58IDC}17”:[”invalid paren-

tid”],”{GXS58IDC}20”:[”invalid parentid”]}}

DELETE

https://*server*/*pathname*/*version*/*username*/storage/*collection*

Deletes the collection and all contents. Additional request parameters may modify the selection of which items to

delete:

• ids: Deletes the ids for objects in the collection that are in the provided comma-separated list.

• parentid: Only deletes objects in the collection that are the children of the parent id given.

• older: Only deletes objects in the collection that have been last modiﬁed before the date given.

• newer: Only deletes objects in the collection that have been last modiﬁed since the date given.

• limit: Sets the maximum number of objects that will be deleted.

• offset: Skips the ﬁrst n objects in the deﬁned set. Must be used with the limit parameter. [This function is not

currently operational in the mysql implementation]

• sort: Sorts items before deletion - ‘oldest’ - Orders by modiﬁcation date (oldest ﬁrst) - ‘newest’ - Orders by

modiﬁcation date (newest ﬁrst) - ‘index’ - Orders by the sortindex (ordered lists)

https://*server*/*pathname*/*version*/*username*/storage/*collection*/*id*

Deletes the WBO at the location given

All delete requests return the timestamp of the action.

https://*server*/*pathname*/*version*/*username*/storage*

Deletes all records for the user. Will return a precondition error unless an X-Conﬁrm-Delete header is

included.

All delete requests return the timestamp of the action.

2.2. Storage Service 25

Mozilla Services Documentation, Release

General Weave Headers

X-Weave-Backoff

Indicates that the server is under heavy load or has suffered a failure and the client should not try again

for the speciﬁed number of seconds (usually 1800)

X-If-Unmodiﬁed-Since

On any write transaction (PUT, POST, DELETE), this header may be added to the request, set to a times-

tamp. If the collection to be acted on has been modiﬁed since the timestamp given, the request will

fail.

X-Weave-Alert

This header may be sent back from any transaction, and contains potential warning messages, information,

or other alerts. The contents are intended to be human-readable.

X-Weave-Timestamp

This header will be sent back with all requests, indicating the current timestamp on the server. If the

request was a PUT or POST, this will also be the modiﬁcation date of any WBOs submitted or modiﬁed.

X-Weave-Records If supported by the db, this header will return the number of records total in the request body of

any multiple-record GET request.

Storage API v1.1

The Storage server provides web services that can be used to store and retrieve Weave Basic Objects or WBOs

organized into collections.

Weave Basic Object

A Weave Basic Object (WBO) is the generic JSON wrapper around all items passed into and out of the storage server.

Like all JSON, Weave Basic Objects need to be UTF-8 encoded. WBOs have the following ﬁelds:

26 Chapter 2. Services

Mozilla Services Documentation, Release

Parameter Default Type/Max Description

id required string 64 An identifying string. For

a user, the id must be

unique for a WBO within

a collection, though ob-

jects in different collec-

tions may have the same

ID.

This should be exactly

12 characters from the

base64url alphabet. While

this isn’t enforced by the

server, the Firefox client

expects this in most cases.

modiﬁed time submitted ﬂoat 2 decimal places The last-modiﬁed

date, in seconds since

01/01/1970

sortindex none integer An integer indicating the

relative importance of this

item in the collection.

payload none string 256k A string containing a

JSON structure encap-

sulating the data of the

record. This structure

is deﬁned separately for

each WBO type. Parts

of the structure may

be encrypted, in which

case the structure should

also specify a record for

decryption.

ttl none integer The number of seconds to

keep this record. After

that time, this item will

not be returned.

parentid

none string 64 The id of a parent ob-

ject in the same collection.

This allows for the cre-

ation of hierarchical struc-

tures (such as folders).

predecessorid

none string 64 The id of a predecessor

in the same collection.

This allows for the cre-

ation of linked-list-esque

structures.

Sample:

{

"id":"-F_Szdjg3GzY",

"modified":1278109839.96,

"sortindex":140,

See ecma-262: http://www.ecma-international.org/publications/standards/Ecma-262.htm

Set automatically by the server

This is deprecated and likely going away in future versions

2.2. Storage Service 27

Mozilla Services Documentation, Release

"payload":"{\"ciphertext\":\"e2zLWJYX\/

˓→iTw3WXQqffo00kuuut0Sk3G7erqXD8c65S5QfB85rqolFAU0r72GbbLkS7ZBpcpmAvX6LckEBBhQPyMt7lJzfwCUxIN\/

˓→uCTpwlf9MvioGX0d4uk3G8h1YZvrEs45hWngKKf7dTqOxaJ6kGp507A6AvCUVuT7jzG70fvTCIFyemV+Rn80rgzHHDlVy4FYti6tDkmhx8t6OMnH9o\/

˓→ax\/3B2cM+6J2Frj6Q83OEW\/QBC8Q6\/XHgtJJlFi6fKWrG+XtFxS2\/

˓→AazbkAMWgPfhZvIGVwkM2HeZtiuRLM=\",\"IV\":\"GluQHjEH65G0gPk\/d\/OGmg==\",\"hmac\":\

˓→"c550f20a784cab566f8b2223e546c3abbd52e2709e74e4e9902faad8611aa289\"}"

}

Collections

Each WBO is assigned to a collection with other related WBOs. Collection names may only contain alphanumeric

characters, period, underscore and hyphen. Default Mozilla collections are:

• bookmarks

• history

• forms

• prefs

• tabs

• passwords

Additionally, the following collections are supported for internal storage client use:

• clients

• crypto

• keys

• meta

URL semantics

Storage URLs follow, for the most part, REST semantics. Request and response bodies are all JSON-encoded.

The URL for Weave Storage requests is structured as follows:

https://<server name>/<api pathname>/<version>/<username>/<further instruction>

Component Mozilla Default Description

server name deﬁned by user account the hostname of the server

pathname (none) the preﬁx associated with the service on the box

version 1.1 The API version.

username (none) the name of the object (user) to be manipulated

further instruction (none) The additional function information as deﬁned in the paths below

Certain functions use HTTP basic auth (over SSL, so as to maintain password security). If the auth username does not

match the username in the path, the server will issue an Error Response.

The User API has a set of Response codes to cover errors in the request or on the server side. The format of a successful

response is deﬁned in the appropriate request method section.

28 Chapter 2. Services

Mozilla Services Documentation, Release

APIs

GET https://server/pathname/version/username/info/collections

Returns a hash of collections associated with the account, along with the last modiﬁed timestamp for each

collection.

GET https://server/pathname/version/username/info/collection_usage

Returns a hash of collections associated with the account, along with the data volume used for each (in

KB).

GET https://server/pathname/version/username/info/collection_counts

Returns a hash of collections associated with the account, along with the total number of items in each

collection.

GET https://server/pathname/version/username/info/quota

Returns a list containing the user’s current usage and quota (in KB). The second value will be null if no

quota is deﬁned.

GET https://server/pathname/version/username/storage/collection

Returns a list of the WBO ids contained in a collection. This request has additional optional parameters:

• ids: returns the ids for objects in the collection that are in the provided comma-separated list.

• predecessorid: returns the ids for objects in the collection that are directly preceded by the id given.

Usually only returns one result.

• parentid: returns the ids for objects in the collection that are the children of the parent id given.

• older: returns only ids for objects in the collection that have been last modiﬁed before the date

given.

• newer: returns only ids for objects in the collection that have been last modiﬁed since the date given.

• full: if deﬁned, returns the full WBO, rather than just the id.

• index_above: if deﬁned, only returns items with a higher sortindex than the value speciﬁed.

• index_below: if deﬁned, only returns items with a lower sortindex than the value speciﬁed.

• limit: sets the maximum number of ids that will be returned.

• offset: skips the ﬁrst n ids. For use with the limit parameter (required) to paginate through a result

set.

• sort: sorts the output.

• ‘oldest’ - Orders by modiﬁcation date (oldest ﬁrst)

• ‘newest’ - Orders by modiﬁcation date (newest ﬁrst)

• ‘index’ - Orders by the sortindex descending (highest weight ﬁrst)

Two alternate output formats are available for multiple record GET requests. They are triggered by the

presence of the appropriate format in the Accept header (with application/whoisi taking precedence):

• application/whoisi: each record consists of a 32-bit integer, deﬁning the length of the record, fol-

lowed by the json record for a WBO

• application/newlines: each record is a separate json object on its own line. Newlines in the body of

the json object are replaced by ‘u000a’

Deprecated in 1.1

2.2. Storage Service 29

Mozilla Services Documentation, Release

GET https://server/pathname/version/username/storage/collection/id

Returns the WBO in the collection corresponding to the requested id

PUT https://server/pathname/version/username/storage/collection/id

Adds the WBO deﬁned in the request body to the collection. If the WBO does not contain a payload, it

will only update the provided metadata ﬁelds on an already deﬁned object.

The server will return the timestamp associated with the modiﬁcation.

POST https://server/pathname/version/username/storage/collection

Takes an array of WBOs in the request body and iterates over them, effectively doing a series of atomic

PUTs with the same timestamp.

Returns a hash of successful and unsuccessful saves, including guidance as to possible errors:

{"modified": 1233702554.25,

"success": ["{GXS58IDC}12", "{GXS58IDC}13", "{GXS58IDC}15",

"{GXS58IDC}16", "{GXS58IDC}18", "{GXS58IDC}19"],

"failed": {"{GXS58IDC}11": ["invalid parentid"],

"{GXS58IDC}14": ["invalid parentid"],

"{GXS58IDC}17": ["invalid parentid"],

"{GXS58IDC}20": ["invalid parentid"]}}

DELETE https://server/pathname/version/username/storage/collection

Deletes the collection and all contents. Additional request parameters may modify the selection of which

items to delete:

• ids: deletes the ids for objects in the collection that are in the provided comma-separated list.

• parentid: only deletes objects in the collection that are the children of the parent id given.

• older: only deletes objects in the collection that have been last modiﬁed before the date given.

• newer: only deletes objects in the collection that have been last modiﬁed since the date given.

• limit: sets the maximum number of objects that will be deleted.

• offset: skips the ﬁrst n objects in the deﬁned set. Must be used with the limit parameter.

• sort: sorts before deleting

• ‘oldest’ - Orders by modiﬁcation date (oldest ﬁrst)

• ‘newest’ - Orders by modiﬁcation date (newest ﬁrst)

• ‘index’ - Orders by the sortindex (ordered lists)

DELETE https://server/pathname/version/username/storage/collection/id

Deletes the WBO at the location given

DELETE https://server/pathname/version/username/storage

Deletes all records for the user. Will return a precondition error unless an X-Conﬁrm-Delete header is

included.

All delete requests return the timestamp of the action.

This function is not currently operational in the mysql implementation

30 Chapter 2. Services

Mozilla Services Documentation, Release

Headers

Retry-After

When sent together with an HTTP 503 status code, it signiﬁes that the server is undergoing maintenance.

The client should not attempt another sync for the number of seconds speciﬁed in the header value.

X-Weave-Backoff

Indicates that the server is under heavy load and the client should not trigger another sync for the number

of seconds speciﬁed in the header value (usually 1800).

X-If-Unmodiﬁed-Since

On any write transaction (PUT, POST, DELETE), this header may be added to the request, set to a times-

tamp in decimal seconds. If the collection to be acted on has been modiﬁed since the provided timestamp,

the request will fail with an HTTP 412 Precondition Failed status.

X-Weave-Alert

This header may be sent back from any transaction, and contains potential warning messages, information,

or other alerts. The contents are intended to be human-readable.

X-Weave-Timestamp

This header will be sent back with all requests, indicating the current timestamp on the server. If the

request was a PUT or POST, this will also be the modiﬁcation date of any WBOs submitted or modiﬁed.

X-Weave-Records

If supported by the DB, this header will return the number of records total in the request body of any

multiple-record GET request.

HTTP status codes

200

The request was processed successfully.

400

The request itself or the data supplied along with the request is invalid. The response contains a numeric

code indicating the reason for why the request was rejected. See Response codes for a list of valid response

codes.

401

The username and password are invalid on this node. This may either be caused by a node reassignment or

by a password change. The client should check with the auth server whether the user’s node has changed.

If it has changed, the current sync is to be aborted and should be retried against the new node. If the node

hasn’t changed, the user’s password was changed.

404

The requested resource could not be found. This may be returned for GET and DELETE requests, for

non-existent records and empty collections.

503

Indicates, in conjunction with the Retry-After header, that the server is undergoing maintenance. The

client should not attempt another sync for the number of seconds speciﬁed in the header value. The

response body may contain a JSON string describing the server’s status or error.

2.2. Storage Service 31

Mozilla Services Documentation, Release

SyncStorage API v1.5

The SyncStorage API deﬁnes a HTTP web service used to store and retrieve simple objects called Basic Storage

Objects (BSOs), which are organized into named collections.

Concepts

Basic Storage Object

A Basic Storage Object (BSO) is the generic JSON wrapper around all items passed into and out of the SyncStorage

server. Like all JSON documents, BSOs are composed of unicode character data rather than raw bytes and must be

encoded for transmission over the network. The SyncStorage service always encodes BSOs in UTF8.

Basic Storage Objects have the following ﬁelds:

32 Chapter 2. Services

Mozilla Services Documentation, Release

Parameter Default Type/Max Description

id required string, 64 An identifying string. For

a user, the id must be

unique for a BSO within

a collection, though ob-

jects in different collec-

tions may have the same

ID.

BSO ids must only con-

tain printable ASCII char-

acters. They should be

exactly 12 base64-urlsafe

characters; while this isn’t

enforced by the server, the

Firefox client expects it in

most cases.

modiﬁed none ﬂoat, 2 decimal places The timestamp at which

this object was last mod-

iﬁed, in seconds since

UNIX epoch (1970-01-01

00:00:00 UTC). This is

set automatically by the

server according to its

own clock; any client-

supplied value for this

ﬁeld is ignored.

sortindex none integer, 9 digits An integer indicating the

relative importance of this

item in the collection.

payload empty string string, at least 256KiB A string containing the

data of the record. The

structure of this string

is deﬁned separately for

each BSO type. This

spec makes no require-

ments for its format. In

practice, JSONObjects are

common.

Servers must support pay-

loads up to 256KiB in

size. They may ac-

cept larger payloads, and

advertise their maximum

payload size via dynamic

conﬁguration.

ttl none integer, positive, 9 digits The number of seconds to

keep this record. After

that time this item will no

longer be returned in re-

sponse to any request, and

it may be pruned from the

database. If not speciﬁed

or null, the record will not

expire.

This ﬁeld may be set on

write, but is not returned

by the server.

2.2. Storage Service 33

Mozilla Services Documentation, Release

Example:

{

"id": "-F_Szdjg3GzX",

"modified": 1388635807.41,

"sortindex": 140,

"payload": "{ \"this is\": \"an example\" }"

}

Collections

Each BSO is assigned to a collection with other related BSOs. Collection names may be up to 32 characters long, and

must contain only characters from the urlsafe-base64 alphaebet (i.e. alphanumeric characters, underscore and hyphen)

and the period.

Collections are created implicitly when a BSO is stored in them for the ﬁrst time. They continue to exist until they are

explicitly deleted, even if they no longer contain any BSOs.

The default collections used by Firefox to store sync data are:

• bookmarks

• history

• forms

• prefs

• tabs

• passwords

The following additional collections are used for internal management purposes by the storage client:

• clients

• crypto

• keys

• meta

Timestamps

In order to allow multiple clients to coordinate their changes, the SyncStorage server associates a last-modiﬁed time

with the data stored for each user. This is a server-assigned decimal value, precise to two decimal places, that is

updated from the server’s clock with every modiﬁcation made to the user’s data.

The last-modiﬁed time is tracked at three levels of nesting:

• The store as a whole has a last-modiﬁed time that is updated whenever any change is made to the user’s data.

• Each collection has a last-modiﬁed time that is updated whenever an item in that collection is modiﬁed or

deleted. It will always be less than or equal to the overall last-modiﬁed time.

• Each BSO has a last-modiﬁed time that is updated whenever that speciﬁc item is modiﬁed. It will always be

less than or equal to the last-modiﬁed time of the containing collection.

The last-modiﬁed time is guaranteed to be monotonically increasing and can be used for coordination and conﬂict

management as described in Concurrency and Conﬂict Management.

34 Chapter 2. Services

Mozilla Services Documentation, Release

Note that the last-modiﬁed time of a collection may be larger than that of any item within in. For example, if all items

are deleted from the collection, its last-modiﬁed time will be the timestamp of the last deletion.

API Instructions

The SyncStorage data for a given user may be accessed via authenticated HTTP requests to their SyncStorage API

endpoint. Request and response bodies are all UTF8-encoded JSON unless otherwise speciﬁed. All requests are to

URLs of the form:

https://<endpoint-url>/<api-instruction>

The user’s SyncStorage endpoint URL can be obtained via the Token Server authentication ﬂow. All requests must be

signed using HAWK Authentication credentials obtained from the tokenserver.

Error responses generated by the SyncStorage server will, wherever possible, conform to the Response codes deﬁned

for the User API. The format of a successful response is deﬁned in the appropriate section of the API Instructions

documentation.

General Info

APIs in this section provide a mechanism for high-level interactions with the user’s data store as a whole.

GET https://<endpoint-url>/info/collections

Returns an object mapping collection names associated with the account to the last-modiﬁed time for each

collection.

The server may allow requests to this endpoint to be authenticated with an expired token, so that clients

can check for server-side changes before fetching an updated token from the Token Server.

GET https://<endpoint-url>/info/quota

Returns a two-item list giving the user’s current usage and quota (in KB). The second item will be null if

the server does not enforce quotas.

Note that usage numbers may be approximate.

GET https://<endpoint-url>/info/collection_usage

Returns an object mapping collection names associated with the account to the data volume used for each

collection (in KB).

Note that this request may be very expensive as it calculates more detailed and accurate usage information

than the request to /info/quota.

GET https://<endpoint-url>/info/collection_counts

Returns an object mapping collection names associated with the account to the total number of items in

each collection.

GET https://<endpoint-url>/info/conﬁguration

Provides information about the conﬁguration of this storage server with respect to various protocol and

size limits. Returns an object mapping conﬁguration item names to their values as enforced by this server.

The following conﬁguration items may be present:

• max_request_bytes: the maximum size in bytes of the overall HTTP request body that will be

accepted by the server.

2.2. Storage Service 35

Mozilla Services Documentation, Release

• max_post_records: the maximum number of records that can be uploaded to a collection in a single

POST request.

• max_post_bytes: the maximum combined size in bytes of the record payloads that can be uploaded

to a collection in a single POST request.

• max_total_records: the maximum total number of records that can be uploaded to a collection as

part of a batched upload.

• max_total_bytes: the maximum total combined size in bytes of the record payloads that can be

uploaded to a collection as part of a batched upload.

• max_record_payload_bytes: the maximum size of an individual BSO payload, in bytes.

DELETE https://<endpoint-url>/storage

Deletes all records for the user. This is URL is provided for backwards- compatibility with the previous

version of the syncstorage API; new clients should use DELETE https://<endpoint-url>.

DELETE https://<endpoint-url>

Deletes all records for the user.

Individual Collection Interaction

APIs in this section provide a mechanism for interacting with a single collection.

GET https://<endpoint-url>/storage/<collection>

Returns a list of the BSOs contained in a collection. For example:

["GXS58IDC_12", "GXS58IDC_13", "GXS58IDC_15"]

By default only the BSO ids are returned, but full objects can be requested using the full parameter. If the

collection does not exist, an empty list is returned.

This request has additional optional query parameters:

• ids: a comma-separated list of ids. Only objects whose id is in this list will be returned. A maximum

of 100 ids may be provided.

• newer: a timestamp. Only objects whose last-modiﬁed time is strictly greater than this value will

be returned.

• older: a timestamp. Only objects whose last-modiﬁed time is strictly smaller than this value will be

returned.

• full: any value. If provided then the response will be a list of full BSO objects rather than a list of

ids.

• limit: a positive integer. At most that many objects will be returned. If more than that many objects

matched the query, an X-Weave-Next-Offset header will be returned.

• offset: a string, as returned in the X-Weave-Next-Offset header of a previous request using the limit

parameter.

• sort: sorts the output:

– ‘newest’ - orders by last-modiﬁed time, largest ﬁrst

– ‘oldest’ - orders by last-modiﬁed time, smallest ﬁrst

– ‘index’ - orders by the sortindex, highest weight ﬁrst

36 Chapter 2. Services

Mozilla Services Documentation, Release

The response may include an X-Weave-Records header indicating the total number of records to expect in

the body, if the server can efﬁciently provide this information.

If the request included a limit parameter and there were more than that many items matching the query,

the response will include an X-Weave-Next-Offset header. This value can be passed back to the server in

the offset parameter to efﬁciently skip over the items that have already been read. See Example: paging

through a large set of items for an example.

Two output formats are available for multiple-record GET requests. They are triggered by the presence of

the appropriate format in the Accept request header and are prioritized in the order listed below:

• application/json: the output is a JSON list of the request records, as either string ids or full JSON

objects.

• application/newlines: the output contains each individual record followed by a newline, as either a

string id or a full JSON object.

Potential HTTP error responses include:

• 400 Bad Request: too many ids were included in the query parameter.

GET https://<endpoint-url>/storage/<collection>/<id>

Returns the BSO in the collection corresponding to the requested id

PUT https://<endpoint-url>/storage/<collection>/<id>

Creates or updates a speciﬁc BSO within a collection. The request body must be a JSON object containing

new data for the BSO.

If the target BSO already exists then it will be updated with the data from the request body. Fields that

are not provided in the request body will not be overwritten, so it is possible to e.g. update the ttl ﬁeld of

a BSO without re-submitting its payload. Fields that are explicitly set to null in the request body will be

set to their default value by the server.

If the target BSO does not exist, then ﬁelds that are not provided in the request body will be set to their

default value by the server.

This request may include the X-If-Unmodiﬁed-Since header to avoid overwriting the data if it has been

changed since the client fetched it.

Successful responses will return the new last-modiﬁed time for the collection.

Note that the server may impose a limit on the amount of data submitted for storage in a single BSO.

Potential HTTP error responses include:

• 400 Bad Request: the user has exceeded their storage quota.

• 413 Request Entity Too Large: the object is larger than the server is willing to store.

POST https://<endpoint-url>/storage/<collection>

Takes a list of BSOs in the request body and iterates over them, effectively doing a series of individual

PUTs with the same timestamp.

Each BSO record in the request body must include an “id” ﬁeld, and the corresponding BSO will be

created or updated according to the semantics of a PUT request targeting that speciﬁc record. In particular,

this means that ﬁelds not provided in the request body will not be overwritten on BSOs that already exist.

Two input formats are available for multiple-record POST requests, selected by the Content-Type header

of the request:

• application/json: the input is a JSON list of objects, one for for each BSO in the request.

• application/newlines: each BSO is sent as a separate JSON object followed by a newline.

2.2. Storage Service 37

Mozilla Services Documentation, Release

For backwards-compatibility with existing clients, the server will also treat text/plain input as JSON.

Note that the server may impose a limit on the total amount of data included in the request, and/or may

decline to process more than a certain number of BSOs in a single request. The default limit on the

number of BSOs per request is 100.

Successful responses will contain a JSON object with details of success or failure for each BSO. It will

have the following keys:

• modiﬁed: the new last-modiﬁed time for the updated items.

• success: a (possibly empty) list of ids of BSOs that were successfully stored.

• failed: a (possibly empty) object whose keys are the ids of BSOs that were not stored successfully,

and whose values are strings describing the reason for the failure.

For example:

{

"modified": 1233702554.25,

"success": ["GXS58IDC_12", "GXS58IDC_13", "GXS58IDC_15",

"GXS58IDC_16", "GXS58IDC_18", "GXS58IDC_19"],

"failed": {"GXS58IDC_11": "invalid ttl"],

"GXS58IDC_14": "invalid sortindex"}

}

Posted BSOs whose ids do not appear in either “success” or “failed” should be treated as having failed for

an unspeciﬁed reason.

To allow upload of large numbers of items while ensuring that other clients do not sync down inconsistent

data, servers may support combining several POST requests into a single “batch” so that all modiﬁed

BSOs appear to have been submitted at the same time. Batching behaviour is controlled by the following

query parameters:

• batch: indicates that uploads should be batched together into a single conceptual update. To begin a

new batch pass the string ‘true’. To add more items to an existing batch pass a previously-obtained

batch identiﬁer. This parameter is ignored by servers that do not support batching.

• commit: indicates that the batch should be committed, and all items uploaded to that batch made

visible to other clients. If present, it must be the string ‘true’ and the batch query parameter must

also be speciﬁed.

When submitting items for inclusion in a multi-request batch upload, successful responses will have

a “202 Accepted” status code, and will contain a JSON object giving the batch identiﬁer rather than

modiﬁcation time, alongside individual success or failure status for each item that was sent.

For example:

{

"batch": "OPAQUEBATCHID",

"success": ["GXS58IDC_12", "GXS58IDC_13", "GXS58IDC_15",

"GXS58IDC_16", "GXS58IDC_18", "GXS58IDC_19"],

"failed": {"GXS58IDC_11": "invalid ttl"],

"GXS58IDC_14": "invalid sortindex"}

}

The returned value of “batch” can be passed in the “batch” query parameter to add more items to the

batch. Items that appear in the “success” list are guaranteed to become available to other clients if and

when the batch is successfully committed.

38 Chapter 2. Services

Mozilla Services Documentation, Release

Note that the value of “batch” may not be safe to include directly in a URL, and that you need to be

sure to encode it ﬁrst (via JavaScript’s encodeURIComponent, Python’s urllib.quote, or your

language’s equivalent).

If the server does not support batching, it will ignore the batch parameter and return a “200 OK” response

without a batch identiﬁer.

The response when committing a batch is identical to that generated by a non-batched request. Note

that the semantics of a request with batch=true&commit=true (i.e. starting a batch and immediately

committing it) are therefore identical to those of a non-batched request.

Note that the server may impose a limit on the total amount of payload data included in a batch, and/or may

decline to process more than a certain number of BSOs as part of a single batch. If the uploaded items

exceed this limit, the server will produce a 400 Bad Request response with response code 17. Where

possible, clients should use the X-Weave-Total-Records* and X-Weave-Total-Bytes headers to signal the

expected total size of the uploads, so that oversize batches can be rejected before the items are uploaded.

Potential HTTP error responses include:

• 400 Bad Request, response code 14: the user has exceeded their storage quota.

• 400 Bad Request, response code 17: server size or item-count limit exceeded.

• 413 Request Entity Too Large: the request contains more data than the server is willing to process

in a single batch.

DELETE https://<endpoint-url>/storage/<collection>

Deletes an entire collection.

After executing this request, the collection will not appear in the output of GET /info/collections and

calls to GET /storage/<collection> will return an empty list.

DELETE https://<endpoint-url>/storage/<collection>?ids=<ids>

Deletes multiple BSOs from a collection with a single request.

This request takes a parameter to select which items to delete:

• ids: deletes BSOs from the collection whose ids that are in the provided comma-separated list. A

maximum of 100 ids may be provided.

The collection itself will still exist on the server after executing this request. Even if all the BSOs in

the collection are deleted, it will receive an updated last-modiﬁed time, appear in the output of GET

/info/collections, and be readable via GET /storage/<collection>

Successful responses will have a JSON object body with ﬁeld “modiﬁed” giving the new last-modiﬁed

time for the collection.

Potential HTTP error responses include:

• 400 Bad Request: too many ids were included in the query parameter.

DELETE https://<endpoint-url>/storage/<collection>/<id>

Deletes the BSO at the given location.

Request Headers

X-If-Modiﬁed-Since

2.2. Storage Service 39

Mozilla Services Documentation, Release

This header may be added to any GET request, set to a decimal timestamp. If the last-modiﬁed time of

the target resource is less than or equal to the time given in this header, then a 304 Not Modiﬁed response

will be returned and re-transmission of the unchanged data will be avoided.

It is similar to the standard HTTP If-Modiﬁed-Since header, but the value is a decimal timestamp rather

than a HTTP-format date.

If the value of this header is not a valid positive decimal value, or if the X-If-Unmodiﬁed-Since header is

also present, then a 400 Bad Request response will be returned.

X-If-Unmodiﬁed-Since

This header may be added to any request to a collection or item, set to a decimal timestamp. If the

last-modiﬁed time of the target resource is greater than the time given, the request will fail with a 412

Precondition Failed response.

It is similar to the standard HTTP If-Unmodiﬁed-Since header, but the value is a decimal timestamp

rather than a HTTP-format date.

If the value of this header is not a valid positive decimal value, or if the X-If-Modiﬁed-Since header is

also present, then a 400 Bad Request response will be returned.

X-Weave-Records

This header may be sent with multi-record uploads, to indicate the total number of records included in the

request. If the server would not accept an upload containing that many records, then a 400 Bad Request

response will be returned with response code 17.

X-Weave-Bytes

This header may be sent with multi-record uploads, to indicate the combined size of payloads in the

upload, in bytes. If the server would not accept an upload containing that many bytes, then a 400 Bad

Request response will be returned with response code 17.

X-Weave-Total-Records

This header may be included with a POST request using the batch query parameter, to indicate the total

number of records in the batch. If the server would not accept a batch containing that many records, then

a 400 Bad Request response will be returned with response code 17.

If the value of this header is not a valid positive integer value, or if the request is not operating on a batch,

then a 400 Bad Request response will be returned with response code 1.

X-Weave-Total-Bytes

This header may be included with a POST request using the batch query parameter, to indicate the total

combined size of payloads in the batch, in bytes. If the server would not accept a batch containing that

many bytes, then a 400 Bad Request response will be returned with response code 17.

If the value of this header is not a valid positive integer value, or if the request is not operating on a batch,

then a 400 Bad Request response will be returned with response code 1.

Response Headers

Retry-After

When sent together with an HTTP 503 status code, this header signiﬁes that the server is undergoing

maintenance. The client should not attempt any further requests to the server for the number of seconds

speciﬁed in the header value.

When sent together with a HTTP 409 status code, this header gives the time after which the conﬂicting

edits are expected to complete. Clients should wait until at least this time before retrying the request.

40 Chapter 2. Services

Mozilla Services Documentation, Release

X-Weave-Backoff

This header may be sent to indicate that the server is under heavy load but is still capable of servicing

requests. Unlike the Retry-After header, X-Weave-Backoff may be included with any type of response,

including a 200 OK.

Clients should perform the minimum number of additional requests required to maintain consistency of

their stored data, then not attempt any further requests for the number of seconds speciﬁed in the header

value.

X-Last-Modiﬁed

This header gives the last-modiﬁed time of the target resource as seen during processing of the request,

and will be included in all success responses (200, 201, 204). When given in response to a write request,

this will be equal to the server’s current time and to the new last-modiﬁed time of any BSOs created or

changed by the request.

It is similar to the standard HTTP Last-Modiﬁed header, but the value is a decimal timestamp rather than

a HTTP-format date.

X-Weave-Timestamp

This header will be sent back with all responses, indicating the current timestamp on the server. When

given in response to a write request, this will be equal to the new timestamp value of any BSOs created or

changed by that request.

It is similar to the standard HTTP Date header, but the value is a decimal timestamp rather than a HTTP-

format date.

X-Weave-Records

This header may be sent back with multi-record responses, to indicate the total number of records included

in the response.

X-Weave-Next-Offset

This header may be sent back with multi-record responses where the request included a limit parameter.

Its presence indicates that the number of available records exceeded the given limit. The value from this

header can be passed back in the offset parameter to retrieve additional records.

The value of this header will always be a string of characters from the urlsafe-base64 alphabet. The

speciﬁc contents of the string are an implementation detail of the server, so clients should treat it as an

opaque token.

X-Weave-Quota-Remaining

This header may be returned in response to write requests, indicating the amount of storage space remain-

ing for the user (in KB). It will not be returned if quotas are not enabled on the server.

X-Weave-Alert

This header may be returned in response to any request, and contains potential warning messages, infor-

mation, or other alerts.

If the ﬁrst character of the header is not “{” then it is intended to be a human-readable string that may be

included in logs.

If the ﬁrst character of the header is “{” then it is a JSON object signalling impending shutdown of the

service. It will contain the following ﬁelds:

• code: one of the strings “soft-eol” or “hard-eol”.

• message: a human-readable message that may be included in logs.

• url: a URL at which more information is available.

2.2. Storage Service 41

Mozilla Services Documentation, Release

HTTP status codes

Since the syncstorage protocol is implemented on top of HTTP, clients should be prepared to deal gracefully with any

valid HTTP response. This section serves to highlight the response codes that explicitly form part of the syncstorage

protocol.

200 OK

The request was processed successfully, and the server is returning useful information in the response

body.

304 Not Modiﬁed

For requests that include the X-If-Modiﬁed-Since header, this response code indicates that the resource

has not been modiﬁed. The client should continue to use its local copy of the data.

400 Bad Request

The request itself or the data supplied along with the request is invalid and could not be processed by the

server. For example, this response will be returned if a header value is incorrectly formatted or if a JSON

request body cannot be parsed.

If the response has a Content-Type of application/json then the body will be an integer response code as

documented in Response codes. The respcodes with particular meaning in this protocol include:

• 6: JSON parse failure, likely due to badly-formed POST data.

• 8: invalid BSO, likely due to badly-formed POST data.

• 13: invalid collection, likely invalid chars incollection name.

• 14: user has exceeded their storage quota.

• 16: client is known to be incompatible with the server.

• 17: server limit exceeded, likely due to too many items or too large a payload in a POST request.

401 Unauthorized

The authentication credentials are invalid on this node. This may be caused by a node reassignment or by

an expired/invalid auth token. The client should check with the tokenserver whether the user’s endpoint

URL has changed. If it has changed, the current sync is to be aborted and should be retried against the

new endpoint URL.

404 Not Found

The requested resource could not be found. This may be returned for GET and DELETE requests on

non-existent items. Non-existent collections do not trigger a 404 Not Found for backwards-compatibility

reasons.

405 Method Not Allowed

The request URL does not support the speciﬁc request method. For example, attempting a PUT request

to /info/quota would produce a 405 response.

409 Conﬂict

The write request (PUT, POST, DELETE) has been rejected due conﬂicting changes made by another

client, either to the target resource itself or to a related resource. The server cannot currently complete the

request without violating its consistency guarantees.

The client should retry the request after accounting for any changes introduced by other clients.

This response may include a Retry-After header indicating the time after which the conﬂicting edits are

expected to complete. If present, clients should wait at least this many seconds before retrying the request.

42 Chapter 2. Services

Mozilla Services Documentation, Release

412 Precondition Failed

For requests that included the X-If-Unmodiﬁed-Since header, this response code indicates that the resource

has in fact been modiﬁed more recently than the given time. The requested write operation will not have

been performed.

413 Request Entity Too Large

The body submitted with a write request (PUT, POST) was larger than the server is willing to accept. For

multi-record POST requests, the client should retry by sending the records in smaller batches.

415 Unsupported Media Type

The Content-Type header submitted with a write request (PUT, POST) speciﬁed a data format that is not

supported by the server.

503 Service Unavailable

Indicates that the server is undergoing maintenance. Such a response will include a Retry-After header,

and the client should not attempt another sync for the number of seconds speciﬁed in the header value.

The response body may contain a JSON string describing the server’s status or error.

513 Service Decommissioned

Indicates that the service has been decommissioned, and presumably replaced with a new and better

service using some as-yet-undesigned protocol. This response will include an X-Weave-Alert header

whose value is a JSON object with the following ﬁelds:

• code: the string “hard-eol”.

• message: a human-readable message that may be included in logs.

• url: a URL at which more information is available.

The client should display an appropriate message to the user and cease any further attempts to use the

service.

Concurrency and Conﬂict Management

The SyncStorage service allows multiple clients to synchronize data via a shared server without requiring inter-client

coordination or blocking. To achieve proper synchronization without skipping or overwriting data, clients are expected

to use timestamp-driven coordination features such as X-Last-Modiﬁed and X-If-Unmodiﬁed-Since.

The server guarantees a strictly consistent and monotonically-increasing timestamp across the user’s stored data. Any

request that alters the contents of a collection will cause the last-modiﬁed time to increase. Any BSOs added or

modiﬁed by such a request will have their “modiﬁed” ﬁeld set to the updated timestamp.

Conceptually, each write request will perform the following operations as an atomic unit:

• Read the current time T, and check that it’s greater than the overall last-modiﬁed time for the user’s data. If not

then return a 409 Conﬂict.

• Create any new BSOs as speciﬁed by the request, setting their “modiﬁed” ﬁeld to T.

• Modify any existing BSOs as speciﬁed by the request, setting their “modiﬁed” ﬁeld to T.

• Delete any BSOs as speciﬁed by the request.

• Set the last-modiﬁed time for the collection to T.

• Set the overall last-modiﬁed time for the user’s data to T.

• Generate a 200 OK response with the X-Last-Modiﬁed and X-Weave-Timestamp headers set to T.

2.2. Storage Service 43

Mozilla Services Documentation, Release

While write requests from different clients may be processed concurrently by the server, they will appear to the clients

to have occurred sequentially, instantaneously and atomically according to the above sequence.

To avoid having the server transmit data that has not changed since the last request, clients should set the X-If-

Modiﬁed-Since header and/or the newer parameter to the last known value of X-Last-Modiﬁed on the target resource.

To avoid overwriting changes made by others, clients should set the X-If-Unmodiﬁed-Since header to the last known

value of X-Last-Modiﬁed on the target resource.

Examples

Example: polling for changes to a BSO

To efﬁciently check for changes to an individual BSO, use GET /storage/<collection>/<id> with the X-If-Modiﬁed-

Since header set to the last known value of X-Last-Modiﬁed for that item. This will return the updated item if it has

been changed since the last request, and give a 304 Not Modiﬁed response if it has not:

last_modified = 0

while True:

headers = {"X-If-Modified-Since": last_modified}

r = server.get("/collection/id", headers)

if r.status != 304:

print " MODIFIED ITEM: ", r.json_body

last_modified = r.headers["X-Last-Modified"]

Example: polling for changes to a collection

To efﬁciently poll the server for changes within a collection, use GET /storage/<collection> with the newer parameter

set to the last known value of X-Last-Modiﬁed for that collection. This will return only the BSOs that have been added

or changed since the last request:

last_modified = 0

while True:

r = server.get("/collection?newer=" + last_modified)

for item in r.json_body["items"]:

print "MODIFIED ITEM: ", item

last_modified = r.headers["X-Last-Modified"]

Example: safely updating items in a collection

To update items in a collection without overwriting any changes made by other clients, use POST /stor-

age/<collection> with the X-If-Unmodiﬁed-Since header set to the last known value of X-Last-Modiﬁed for that

collection. If other clients have made changes to the collection since the last request, the write will fail with a 412

Precondition Failed response:

r = server.get("/collection")

last_modified = r.headers["X-Last-Modified"]

bsos = generate_changes_to_the_collection()

headers = {"X-If-Unmodified-Since": last_modified}

r = server.post("/collection", bsos, headers)

44 Chapter 2. Services

Mozilla Services Documentation, Release

if r.status == 412:

print "WRITE FAILED DUE TO CONCURRENT EDITS"

The client may choose to abort the write, or to merge the changes from the server and re-try with an updated value of

X-Last-Modiﬁed.

A similar technique can be used to safely update a single BSO using PUT /storage/<collection>/<id>.

Example: creating a BSO only if it does not exist

To specify that a BSO should be created only if it does not already exist, use the X-If-Unmodiﬁed-Since header with

the special value of 0:

headers = {"X-If-Unmodified-Since": "0"}

r = server.put("/collection/item", data, headers)

if r.status == 412:

print "ITEM ALREADY EXISTS"

Example: paging through a large set of items

The syncstorage server allows efﬁcient paging through a large set of items by using the limit and offset parameters.

Clients should begin by issuing a GET /storage/<collection>?limit=<LIMIT> request, which will return up to

<LIMIT> items. If there were additional items matching the query, the response will include an X-Weave-Next-Offset

header to let subsequent requests skip over the items that were just returned.

To fetch additional items, repeat the request using the value from X-Weave-Next-Offset as the offset parameter. If the

response includes a new X-Weave-Next-Offset value, then there are yet more items to be fetched and the process should

be repeated; if it does not then all available items have been returned.

When repeating requests and specifying an offset parameter, it is important to maintain any parameters which may

change underlying data or its ordering. Other than the offset, one may only change the limit parameter.

To guard against other clients making concurrent changes to the collection, this technique should always be combined

with the X-If-Unmodiﬁed-Since header as shown below:

r = server.get("/collection?limit=100")

print "GOT ITEMS: ", r.json_body["items"]

last_modified = r.headers["X-Last-Modified"]

next_offset = r.headers.get("X-Weave-Next-Offset")

while next_offset:

headers = {"X-If-Unmodified-Since": last_modified}

r = server.get("/collection?limit=100&offset=" + next_offset, headers)

if r.status == 412:

print "COLLECTION WAS MODIFIED WHILE READING ITEMS"

break

print "GOT ITEMS: ", r.json_body["items"]

next_offset = r.headers.get("X-Weave-Next-Offset")

2.2. Storage Service 45

Mozilla Services Documentation, Release

Example: uploading a large batch of items

The syncstorage server allows several upload requests to be combined into a single “batch” so that they all become

visible to other clients as a single atomic unit. This is achieved by using the batch and commit parameters on the

upload request.

Clients should begin by issuing a POST /storage/<collection>?batch=true request, which will accept items for up-

load and issue a new batch id in the response body.

To add more items to the batch, make additional POST requests to the collection using the encoded value of batch

from the response body as the batch query parameter.

When the ﬁnal items have been uploaded, pass the commit query parameter to the POST request. This will ﬁnalize

the batch and make the uploaded items visible to other clients. The last-modiﬁed time of the collection, as well as of

all items included as part of the batch, will be incremented to the timestamp of this ﬁnal commit request.

To guard against other clients concurrently committing changes to the collection, this technique should always be

combined with the X-If-Unmodiﬁed-Since header as shown below:

# Make an initial request to start a batch upload.

# It's possible to send some items here, but not required.

r = server.post("/collection?batch=true", [])

# Note that the batch id is opaque and cannot be safely put in a URL directly

batch_id = urllib.quote(r.json_body["batch"])

# Always use X-If-Unmodified-Since to detect conflicts.

last_modified = r.headers["X-Last-Modified"]

headers = {"X-If-Unmodified-Since": last_modified}

for items in split_items_into_smaller_batches():

# Send the items in several smaller batches.

r = server.post("/collection?batch=" + batch_id, items, headers)

if r.status == 412:

raise Exception("COLLECTION WAS MODIFIED WHILE UPLOADING ITEMS")

# The collection will not be modified yet.

assert r.headers['X-Last-Modified'] == last_modified

# Commit the batch once all items are uploaded.

# Again, it's possible to send some final items here, but not required.

r = server.post("/collection?commit=true&batch=" + batch_id, [], headers)

if r.status == 412:

raise Exception("COLLECTION WAS MODIFIED WHILE COMMITTING ITEMS")

# At this point all the uploaded items become visible,

# and the collection appears modified to other clients.

assert r.headers['X-Last-Modified'] > last_modified

Changes from v1.1

The following is a summary of protocol changes from Storage API v1.1 along with a justiﬁcation for each change:

46 Chapter 2. Services

Mozilla Services Documentation, Release

What Changed Why

Authentication is now performed using a

BrowserID-based tokenserver ﬂow and

HAWK Access Authentication.

This supports authentication via Firefox Accounts and allows us to

iterate the details of that ﬂow without changing the sync protocol.

The structure of the endpoint URL is no

longer speciﬁed, and should be considered

an implementation detail.

This was unnecessary coupling and clients do not need to

change/conﬁgure components of the endpoint URL. URL handling

must change already to support TokenServer-based authentication.

The datatypes and defaults of BSO ﬁelds are

more precisely speciﬁed.

This reﬂects current server behavior, and seems prudent to specify

more explicitly.

The BSO ﬁelds “parentid” and

“predecessorid” have been removed along

with any related query parameters.

These were deprecated in version 1.1 and are not in active use in

current versions of Firefox.

The ‘application/whoisi’ output format has

been removed.

This is not used in any current versions of Firefox.

The previously-undocumented

X-Weave-Quota-Remaining header has been

documented.

This actually is used so we better document it.

The X-Conﬁrm-Delete header has been

removed.

This is sent unconditionally by current client code, and is

therefore useless. Existing client code can safely continue to send

it, and it will be ignored by the server.

The X-Weave-Alert header has grown

additional semantics related to service

end-of-life announcements.

This is already implemented in current Firefox so we better

document it.

GET /storage/collection no longer accepts

‘index_above’ or ‘index_below’

These are not in active use in current versions of Firefox, and

impose additional requirements on the server that may limit

operational ﬂexibility.

DELETE /storage/collection no longer

accepts query parameters other than ‘ids’

These are not in active use in current versions of Firefox, are not

all implemented correctly in the current server, and impose

additional requirements on the server that may limit operational

ﬂexibility.

POST /storage/collection now accepts

‘application/newlines’ input in addition to

‘application/json’.

This matches nicely with ‘application/newlines’ as supported

already in response bodies, and may enable more efﬁcient request

streaming in future. Existing client code doesn’t need to change.

The offset parameter is now an opaque

server-generated value, and clients must not

create their own values for it.

The parameter is not in active use in current versions of Firefox,

and its existing semantics are difﬁcult to implement efﬁciently on

the server. This change allows for more efﬁcient pagination of

results in future client code.

The X-Last-Modiﬁed header has been added. This has slightly different semantics to the X-Weave-Timestamp

header and may be used by future clients for better conﬂict

management. Existing client code doesn’t need to change.

The X-If-Modiﬁed-Since header has been

added and can be used on all GET requests.

Existing client code doesn’t need to change, but will allow future

client code to avoid transmission of redundant data.

The X-If-Unmodiﬁed-Since header can be

used on some GET request.

Existing client code doesn’t need to change, but will allow future

client code to detect changes during paginated fetching of results.

The server may reject concurrent write

attempts with a 409 Conﬂict.

This will be visible to existing client code, but can be handled like

a 503 error. It lets the server provide much stronger consistency

guarantees that will improve overall robustness of the service.

Batch uploads are supported that cross

several POST requests.

This is a backwards-compatible API extension that allows clients

to ensure consistency of their uploaded items.

Various server-speciﬁc size limits can be

read from a new /info/conﬁguration

endpoint.

This is a backwards-compatible API extension that allows clients

to ensure interoperability with conﬁgurable server behaviour.

2.2. Storage Service 47

Mozilla Services Documentation, Release

History

XXX

Resources

• Server (v1.1): https://hg.mozilla.org/services/server-storage

• Server (v1.5): https://github.com/mozilla-services/server-syncstorage

• Client: Firefox 4, Firefox Home

Registration

Goal of the Service

The reg server provides web services on the top of the authentication back-end that can be used to:

• create or delete a new user

• change a user password or e-mail

• assign/get a storage node for a user

• return an HTML reCaptcha challenge

It’s currently used by all Sync clients applications.

If you are using our hosted services with your own client code, please read the Term of Services.

Documentation content

User API v1.0

General description

The URL for a Reg request is structured as follows:

https://<server name>/<api path set>/<version>/<username>/<further instruction>

Component Mozilla Default Description

server name auth.services.mozilla.com the hostname of the server

pathname user the preﬁx associated with the service on the box

version 1.0 The API version.

username (none) the name of the object (user) to be manipulated

further

instruction

(none) The additional function information as deﬁned in the paths

below

Certain functions use HTTP basic auth (over SSL, so as to maintain password security). If the auth username does not

match the username in the path, the server will issue an Error Response.

The Sync User API has a set of Response codes to cover errors in the request or on the server side. The format of a

successful response is deﬁned in the appropriate request method section.

48 Chapter 2. Services

Mozilla Services Documentation, Release

APIs

GET https://server/pathname/version/username

Returns 1 if the username is in use, 0 if it is available. The answer is in plain text.

Possible errors:

• 503: there was an error getting the information

GET https://server/pathname/version/username/node/weave

Returns the Weave (aka Sync) Node that the client is located on. Sync-speciﬁc calls should be directed to

that node.

Return value: the node URL, an unadorned (not JSON) string.

node may be ‘null’ if no node can be assigned at this time, probably due to sign up throttling.

Possible errors:

• 503: there was an error getting a node | empty body

• 404: user not found | empty body

GET https://server/pathname/version/username/password_reset

Requests a password reset email be mailed to the email address on ﬁle. Returns ‘success’ if an email was

successfully sent.

If captchas are enabled for the site, requires captcha-challenge and captcha-response parameters.

Return value:

• “success”

Possible errors:

• 503: problems with looking up the user or sending the email

• 400: 12 (No email address on ﬁle)

• 400: 3 (Incorrect or missing username)

• 400: 2 (Incorrect or missing captcha)

PUT https://server/pathname/version/username

Requests that an account be created for username.

The body is a JSON mapping and should include:

• password: the password to be associated with the account.

• e-mail: Email address associated with the account.

• captcha-challenge: The challenge string from the captcha.

• captcha-response: The response to the captcha.

An X-Weave-Secret can be provided containing a secret string known by the server. When provided, it

will override the captcha. This is useful for testing and automation.

The server will return the lowercase username on success.

Possible errors:

• 503: there was an error creating the reset code

• 400: 4 (user already exists)

2.3. Registration 49

Mozilla Services Documentation, Release

• 400: 6 (Json parse failure)

• 400: 12 (No email address on ﬁle)

• 400: 7 (Missing password ﬁeld)

• 400: 9 (Requested password not strong enough)

• 400: 2 (Incorrect or missing captcha)

POST https://server/pathname/version/username/password

Changes the password associated with the account to the value speciﬁed in the POST body.

NOTE: Requires basic authentication with the username and (current) password associated with the ac-

count. The auth username must match the username in the path.

Alternately, a valid X-Weave-Password-Reset header can be used, if it contains a code previously obtained

from the server.

Return values: “success” on success.

Possible errors:

• 400: 7 (Missing password ﬁeld)

• 400: 10 (Invalid or missing password reset code)

• 400: 9 (Requested password not strong enough)

• 404: the user does not exists in the database

• 503: there was an error updating the password

• 401: authentication failed

POST https://server/pathname/version/username/email

Changes the email associated with the account to the value speciﬁed in the POST body.

NOTE: Requires basic authentication with the username and password associated with the account. The

auth username must match the username in the path.

Alternately, a valid X-Weave-Password-Reset header can be used, if it contains a code previously obtained

from the server.

Return values: The user email on success.

Possible errors:

• 400: 12 (No email address on ﬁle)

• 404: the user does not exists in the database

• 503: there was an error updating the email

• 401: authentication failed

DELETE https://server/pathname/version/username

Deletes the user account.

NOTE: Requires simple authentication with the username and password associated with the account. The

auth username must match the username in the path.

Return value:

• 0 on success

Possible errors:

50 Chapter 2. Services

Mozilla Services Documentation, Release

• 503: there was an error removing the user

• 404: the user does not exist in the database

• 401: authentication failed

GET https://server/misc/1.0/captcha_html

Returns an html body string containing a reCaptcha challenge captcha. The PUT API to create a user will

expect the challenge and response from this captcha.

Note: this function outputs html, not json.

X-Weave-Alert

This header may be sent back from any transaction, and contains potential warning messages, information, or other

alerts. The contents are intended to be human-readable.

History

XXX

Resources

• Original wiki page: https://wiki.mozilla.org/Services/Sync/Server/API/User/1.0

• Continuous Integration server: XXX

• Server: https://hg.mozilla.org/services/server-reg

• Client: Firefox 4+

Easy Setup

Goal of the Service

Setting up a new mobile device should only involve entering a short code on the desktop device.

Secondary request, not a hard requirement, is that if the user has a mobile device, and is setting up a desktop device,

that the ﬂow is similar and still involves entering the key on the desktop.

If you are using our hosted services with your own client code, please read the Term of Services.

Documentation content

User Flow

Desired User Flow

There are two possible scenarios when pairing two devices. In the simplest case, one of the devices is already connected

to Sync. This is the only scenario considered in v1/v2. In other other scenario, none of the devices are connected to

Sync and the user must connect to or create a Sync account on one of them after the pairing. This scenario is new in

v3.

2.4. Easy Setup 51

Mozilla Services Documentation, Release

For the sake of this document, “Mobile” refers to a mobile device or a desktop computer, and “Desktop” refers to a

desktop computer.

1. User chooses “Pair Device” on Mobile.

2. Mobile displays a setup key that contains both the initial secret and a channel ID.

3. On Desktop, user chooses “Pair Device” according to the instructions displayed on the device.

4. Mobile and Desktop exchange messages to build the secure channel.

5. Once the channel has been established and Mobile and Desktop are “paired”, the user creates a Sync account on

Desktop and Desktop uploads its data to the server during the ﬁrst sync. This step is not present in v1/v2 of this

ﬂow. It can be omitted if the user already has an account.

6. Desktop transmits the Sync account credentials (username, password, Sync Key) to Mobile via the channel.

7. Mobile completes setup and starts syncing.

Overview

• Mobile and Desktop complete the two roundtrips of J-PAKE messages to agree upon a strong secret K.

• A 256-bit key is derived from K using HMAC-SHA256 using a ﬁxed extraction key.

• The encryption and HMAC keys are derived from that 256-bit key using HMAC-SHA256.

• To establish the pairing, Mobile encrypts the known message “0123456789ABCDEF” with the AES key and

uploads it. Desktop veriﬁes that it has the same key by encrypting the known message with the key that Desktop

derived.

• To exchange credentials after a successful pairing and possibly account creation on Desktop:

– Desktop encrypts the credentials with the encryption key and uploads the encrypted credentials in turn,

adding a HMAC-SHA256 hash of the ciphertext (using the HMAC key).

– Mobile veriﬁes the ciphertext against the HMAC-SHA256 hash. If successful, Mobile decrypts ciphertext

and applies credentials

– At this point, Mobile can begin synchronizing.

Mobile Server Desktop

===================================================================

retrieve channel <---------------|

generate random secret |

show PIN = secret + channel | ask user for PIN

upload Mobile's message 1 ------>|

|----> retrieve Mobile's message 1

|<----- upload Desktop's message 1

retrieve Desktop's message 1 <---|

upload Mobile's message 2 ------>|

|----> retrieve Mobile's message 2

| compute key

|<----- upload Desktop's message 2

retrieve Desktop's message 2 <---|

compute key |

encrypt known value ------------>|

|-------> retrieve encrypted value

| verify against local known value

At this point Desktop knows whether the PIN was entered correctly.

52 Chapter 2. Services

Mozilla Services Documentation, Release

If it wasn't, Desktop deletes the session. If it was, the account

setup can proceed. If Desktop doesn't have an account set up yet,

it will keep the channel open and let the user connect to or

create an account.

| encrypt credentials

|<------------- upload credentials

retrieve credentials <-----------|

verify HMAC |

decrypt credentials |

delete session ----------------->|

start syncing |

Detailed Flow

1. Mobile asks server for new channel ID (4 characters a-z0-9)

C: GET /new_channel HTTP/1.1

S: "a7id"

2. Mobile generates PIN from random weak secret (4 characters a-z0-9) and the channel ID, computes and uploads

J-PAKE msg 1.

New in v2: To prevent double uploads in case of retries, the If-None-Match:

header may be speciﬁed.

This ensures that the message is only uploaded if the channel is empty. If it is not then the request will fail with

a 412 Precondition Failed which should be considered the same as 200 OK. The 412 will also contain the Etag

of the data was the client just uploaded.

C: PUT /a7id HTTP/1.1

C: If-None-Match:

C: {

C: 'type': 'receiver1',

C: 'version': 3, // new in v3

C: 'payload': {

C: 'gx1': '45...9b',

C: 'zkp_x1': {

C: 'b': '09e22607ead737150b1a6e528d0c589cb6faa54a',

C: 'gr': '58...7a'

C: 'id': 'receiver',

C: }

C: 'gx2': 'be...93',

C: 'zkp_x2': {

C: 'b': '222069aabbc777dc988abcc56547cd944f056b4c',

C: 'gr': '5c...23'

C: 'id': 'receiver',

C: }

Success response:

S: HTTP/1.1 200 OK

S: ETag: "etag-of-receiver1-message"

New in v2: Response that will be returned on retries if the Desktop already replaced the message:

2.4. Easy Setup 53

Mozilla Services Documentation, Release

S: HTTP/1.1 412 Precondition Failed

S: ETag: "etag-of-receiver1-message"

3. Desktop asks user for the PIN, extracts channel ID and weak secret, fetches Mobile’s msg 1:

C: GET /a7id HTTP/1.1

Success response:

S: HTTP/1.1 200 OK

S: ETag: "etag-of-receiver1-message"

New in v3: Prior to v3, clients would only allow a 10 second timeout for messages after the ﬁrst. This

means that if Desktop does not yet have credentials, a Mobile client that implements v2 or lower will not

wait for the account setup to ﬁnish. Desktop should therefore detect Mobile’s API version at this point

and abort the pairing right away if there are no credentials present on Desktop.

4. Desktop computes and uploads msg 1.

New in v2: The If-Match header may be set so that we only upload this message if the other side’s previous

message is still in the channel. This is to prevent double PUTs during retries. If a 412 is received then it means

that our ﬁrst PUT was actually correctly received by the server and that the other side has already uploaded its

next message. So just consider the 412 to be a 200.

C: PUT /a7id HTTP/1.1

C: If-Match: "etag-of-receiver1-message"

C: {

C: 'type': 'sender1',

C: 'version': 3, // new in v3

C: 'payload': {

C: 'gx1': '45...9b',

C: 'zkp_x1': {

C: 'b': '09e22607ead737150b1a6e528d0c589cb6faa54a',

C: 'gr': '58...7a'

C: 'id': 'sender',

C: }

C: 'gx2': 'be...93',

C: 'zkp_x2': {

C: 'b': '222069aabbc777dc988abcc56547cd944f056b4c',

C: 'gr': '5c...23'

C: 'id': 'sender',

C: }

Success response:

S: HTTP/1.1 200 OK

S: Etag: "etag-of-sender1-message"

New in v2: Response that will be returned on retries if Mobile already replaced the message:

S: HTTP/1.1 412 Precondition Failed

S: Etag: "etag-of-sender1-message"

5. Mobile polls for Desktop’s msg 1 once per second for at least 300 seconds:

54 Chapter 2. Services

Mozilla Services Documentation, Release

C: GET /a7id HTTP/1.1

C: If-None-Match: "etag-of-receiver1-message"

S: HTTP/1.1 304 Not Modified

Mobile tries again after 1 second:

C: GET /a7id HTTP/1.1

S: HTTP/1.1 200 OK

S: Etag: "etag-of-sender1-message"

Mobile computes and uploads msg 2.

New in v2: The If-Match header may be set so that we only upload this message if the other side’s previous

message is still in the channel. This is to prevent double PUTs during retries. If a 412 is received then it means

that our ﬁrst PUT was actually correctly received by the server and that the other side has already uploaded its

next message. In this instance, the client can effectively consider the 412 to be a 200.:

C: PUT /a7id HTTP/1.1

C: If-Match: "etag-of-sender1-message"

C: {

C: 'type': 'receiver2',

C: 'version': 3, // new in v3

C: 'payload': {

C: 'A': '87...82',

C: 'zkp_A': {

C: 'b': '6f...08',

C: 'id': 'receiver',

C: 'gr': 'f8...49'

C: }

S: HTTP/1.1 200 OK

S: ETag: "etag-of-receiver2-message"

New in v2: Response that will be returned on retries if Desktop already replaced the message:

S: HTTP/1.1 412 Precondition Failed

S: ETag: "etag-of-receiver2-message"

6. Desktop polls for Mobile’s msg 2 once per second for at least 10 seconds:

C: GET /a7id HTTP/1.1

C: If-None-Match: "etag-of-sender1-message"

S: HTTP/1.1 304 Not Modified

and eventually retrieves it:

S: HTTP/1.1 200 OK

S: Etag: "etag-of-receiver2-message"

Desktop computes key, computes and uploads msg 2.

New in v2: The If-Match header may be set so that we only upload this message if the other side’s previous

2.4. Easy Setup 55

Mozilla Services Documentation, Release

message is still in the channel. This is to prevent double PUTs during retries. If a 412 is received then it means

that our ﬁrst PUT was actually correctly received by the server and that the other side has already uploaded its

next message. In this instance, the client can effectively consider the 412 to be a 200.

C: PUT /a7id HTTP/1.1

C: If-Match: "etag-of-receiver2-message"

C: {

C: 'type': 'sender2',

C: 'version': 3, // new in v3

C: 'payload': {

C: 'A': '87...82',

C: 'zkp_A': {

C: 'b': '6f...08',

C: 'id': 'sender',

C: 'gr': 'f8...49'

C: }

S: HTTP/1.1 200 OK

S: ETag: "etag-of-sender2-message"

New in v2: Response that will be returned on retries if Mobile already replaced the message:

S: HTTP/1.1 412 Precondition Failed

S: ETag: "etag-of-sender2-message"

7. Mobile polls for Desktop’s msg 2 once per second for at least 10 seconds and eventually retrieves it:

C: GET /a7id HTTP/1.1

C: If-No-Match: "etag-of-receiver2-message"

S: HTTP/1.1 200 OK

S: Etag: "etag-of-sender2-message"

{ 'type': 'sender2', ... }

S: HTTP/1.1 304 Not Modified

Mobile computes key, uploads encrypted known message “0123456789ABCDEF” to prove its knowledge (msg

3).

New in v2: The If-Match header may be set so that we only upload this message if the other side’s previous

message is still in the channel. This is to prevent double PUTs during retries. If a 412 is received then it means

that our ﬁrst PUT was actually correctly received by the server and that the other side has already uploaded its

next message. In this instance, the client can effectively consider the 412 to be a 200.

C: PUT /a7id HTTP/1.1

C: If-Match: "etag-of-sender2-message"

C: {

C: 'type': 'receiver3',

C: 'version': 3, // new in v3

C: 'payload': {

C: 'ciphertext': "base64encoded=",

C: 'IV': "base64encoded=",

C: }

56 Chapter 2. Services

Mozilla Services Documentation, Release

S: HTTP/1.1 200 OK

S: Etag: "etag-of-receiver3-message"

New in v2: Response that will be returned on retries if Desktop already replaced the message:

S: HTTP/1.1 412 Precondition failed

S: Etag: "etag-of-receiver3-message"

8. Desktop retrieves Mobile’s msg 3 to conﬁrm the key. It polls once per second for at least 10 seconds:

C: GET /a7id HTTP/1.1

C: If-No-Match: "etag-of-sender2-message"

S: HTTP/1.1 200 OK

C: ETag: "etag-of-receiver3-message"

...

Desktop veriﬁes it against its own version. If the values don’t match, the pairing is aborted and the session

should be deleted.

Once credentials are available, and if the channel is still available, Desktop encrypts the credentials and uploads

them.

New in v2: The If-Match header may be set so that we only upload this message if the other side’s previous

message is still in the channel. This is to prevent double PUTs during retries. If a 412 is received then it means

that our ﬁrst PUT was actually correctly received by the server and that the other side has already uploaded its

next message. In this instance, the client can effectively consider the 412 to be a 200.

New in v3: Desktop must include the If-Match header to ensure the session hasn’t been deleted yet (e.g., due to

a timeout) or tampered with in the meantime.

C: PUT /a7id HTTP/1.1

C: If-Match: "etag-of-receiver3-message"

C: {

C: 'type': 'sender3',

C: 'version': 3, // new in v3

C: 'payload': {

C: 'ciphertext': "base64encoded=",

C: 'IV': "base64encoded=",

C: 'hmac': "base64encoded=",

C: }

S: HTTP/1.1 200 OK

S: Etag: "etag-of-sender3-message"

New in v2: Response that will be returned on retries if Mobile already replaced the message:

S: HTTP/1.1 412 Precondition failed

S: Etag: "etag-of-sender3-message"

If the hash does not match, the Desktop deletes the session:

C: DELETE /a7id HTTP/1.1

2.4. Easy Setup 57

Mozilla Services Documentation, Release

S: HTTP/1.1 200 OK

...

This means that Mobile will receive a 404 when it tries to retrieve the encrypted credentials.

9. Mobile polls for the encrypted credentials once per second for at least 300 seconds to allow for the account

process (the increased timeout is new in v3):

C: GET /a7id HTTP/1.1

C: If-None-Match: "etag-of-receiver3-message"

S: HTTP/1.1 200 OK

...

Decrypts Sync credentials and veriﬁes HMAC.

10. Mobile deletes the session [OPTIONAL]

C: DELETE /a7id HTTP/1.1

S: HTTP/1.1 200 OK

...

11. Mobile starts syncing.

Server API v1.0

General Description

The only valid HTTP response codes are 200 and 304 since those are part of the protocol and expected to happen.

Anything else, like 400, 403, 404 or 503 must result in a complete termination of the password exchange. The client

can retry the exchange then at a later time, starting all over with clean state.

Every call must be done with a ‘’X-KeyExchange-Id” header, containing a half-session identiﬁer for the channel. This

client ID must be a string of 256 chars. The server will keep track of the two ﬁrst ids used for a given channel, from its

creation to its deletion and will close the channel and issue a 400 if any request is made with an unknown id or with

no id at all.

Last, if a given IP attempts to ﬂood the server with a lot of calls in a short time, it will be blacklisted for 10 minutes

return 403s in the interim for any requests made from the same IP. When receiving this error code, legitimate clients

can fall back to a manual transaction. A client that generates a lot of bad requests will also be blacklisted, but for an

hour.

APIS

GET https://server/new_channel

Returns in the response body a JSON-encoded random channel id of N chars from [a-z0-9].

When the API is called, the id returned is guaranteed to be unique. The created channel will have a limited

TTL (currently conﬁgured to 5 minutes).

Return codes:

• 200: channel created successfully

• 503: the server was unable to create a new channel.

58 Chapter 2. Services

Mozilla Services Documentation, Release

• 400: Bad or no client ID. The channel is deleted.

• 403: the IP is blacklisted.

GET https://server/channel_id

Returns in the response body the content of the channel of id ‘’channel_id’‘. Returns an ‘’ETag” response

header containing a unique hash.

The request can contain a ‘’If-None-Match” header containing a hash. If the hash is similar to the current

hash of the channel, the server will return a 304 and an empty body.

The number of GET calls for a given channel are limited to 6. The channel will be deleted by the server

after 6 successful GETs.

Return codes:

• 200: data retrieved successfully

• 404: the channel does not exists. It was not created by a call to ‘’new_channel” or timed out.

• 304: the data was not changed.

• 400: Bad or no client ID. The channel is deleted.

• 403: the IP is blacklisted.

PUT https://server/channel_id

Put in the channel of id ‘’channel_id” the content of the request body. Returns an ‘’ETag” response header

containing a unique hash. If an If-Match Header is provided, it must be the value of the etag before the

update to the channel is applied, or *. If different, a precondition failed code is returned.

If a If-None-Match header is provided and equals to *, and if the channel is not empty (e.g. some data has

been already put in the channel), a precondition failed code is returned.

Return codes:

• 200: data set successfully

• 404: the channel does not exists. It was not created by a call to ‘’new_channel” or timed out.

• 400: Bad or no client ID. The channel is deleted.

• 403: the IP is blacklisted.

• 412: a precondition failed.

POST https://server/report

Reports a log to the server, and optionally asks for a channel deletion.

The log is the body of the request. If the request contains a ‘’X-KeyExchange-Log” header, its value is

prepended to the log provided in the body. In other words, the header can be used for small logs, and the

body for more info. The body size is limited to 2000 chars. If both body and headers are empty, nothing

is logged.

The current errors reported by the client are described in the next section, but the log is a free-form string.

Warning: Under a normal exchange the server is able to count the number of calls and close the channel

at the end, so this API is not to be used to close the channel. Some value should be reported to generate a

security log. The client is therefore encouraged to always provide a report value when calling.

Optionally, if the request contains the ‘’X-KeyExchange-Id” header and a ‘’X-KeyExchange-Cid” header

containing the channel id, the channel will be deleted by the server.

Return codes:

2.4. Easy Setup 59

Mozilla Services Documentation, Release

• 200: logged successfully

• 403: the IP is blacklisted.

• 400: bad request (missing log or bad ids)

Error messages

• jpake.error.timeout (Timeout) : Reported when the exchange is aborted due to timeouts.

• jpake.error.invalid (Invalid message): Reported when a malformed message is received. A malformed message

is one that doesn’t correctly parse as JSON.

• jpake.error.wrongmessage (Wrong message): Reported when the wrong message is received, as identiﬁed by

the type property in the JSON blob.

• jpake.error.internal (Internal J-PAKE failure): Reported when a J-PAKE computation step or encryp-

tion/decryption step fails.

• jpake.error.keymismatch (Key mismatch): Reported when the SHA256 or HMAC veriﬁcation fails, in other

words when the PIN wasn’t entered correctly and both sides ended up with different keys.

• jpake.error.server (Unexpected Server Response): Reported when unexpected HTTP response from the J-

PAKE server is received.

• jpake.error.userabort (User Abort) : Reported when a client aborts the J-PAKE transaction; for example,

when canceling a setup wizard.

Security

DOS Defense

• Least Recently Used (LRU) queue approach for monitoring IP addresses issuing frequent requests

• Conﬁgurable threshold for adding IP address to Blacklist/Penalty Box

• Conﬁgurable time-out for IP addresses added to Blacklist/Penalty Box

• A single shared blacklist will exist within memcache

• LRU queues will be unique to each server and will penalize an IP to the shared blacklist on memcache

• All thresholds will be controlled via the conﬁguration page

TearDown DOS Defense

• Tear down requires valid channel and valid x-keyexchange-id value

• Statistically unlikely. Channel is 4 characters and keyexchange-id is 255 characters

• Brute force attempts will generate lots of noise and will be limited per DOS defense

Logging Points

CEF Logging

• Bad action taken against a valid channel id (denoted by 400 error code)

60 Chapter 2. Services

Mozilla Services Documentation, Release

• Examples: non-existent x-keyexchange-id, bad x-keyexchange-id

• Action taken against an invalid channel id

• Examples: request for properly formed, but not existing, channel id

• IP address sent to black list due to DOS prevention controls

• Examples: Flood of requests from a single IP

• Client fallback to original sync method

• Examples: Client unable to complete J-PAKE sync for any number of reasons and falls back to original sync

approach

• Reported by client to server via reporting API

Application Logging

• Full application logging will be created to enable incident response review

• Logged to application server and not via CEF

• Logs will include:

– Timestamp

– IP address

– Full URL

– x-keyexchange-id

– Event

– Other non-essential headers will be discarded

Admin Web Page

• A small web administrator page will be created which will allow an admin to view all IP addresses that are

currently blacklisted.

• The admin will be able to unblock any of the IP addresses through this page

• Otherwise the IP address will be removed from the black list after the time has elapsed that is deﬁned within the

conﬁguration ﬁle

• Access to the web page will be password-protected with a simple .htaccess ﬁle and IP ﬁltering access (10.*.*.*)

History

v3 - 2012-01-26

• Added a version attribute to the JSON payloads.

• Requiring longer timeout for the ﬁnal message (credentials exchange) to allow for the account creation ﬂow on

Desktop after pairing devices.

2.4. Easy Setup 61

Mozilla Services Documentation, Release

v2 - 2011-04-26

• Added support for If-None-Match and If-Match on PUT requests and the corresponding 412 Precondition

Failed response code to improve reliability on ﬂaky networks.

• Initial implementation

Resources

• Original wiki design page/meeting notes: https://wiki.mozilla.org/Services/Sync/SyncKey/J-PAKE

• Continuous Integration server: XXX

• Server: https://hg.mozilla.org/services/server-key-exchange

• Clients

– Firefox Home: https://hg.mozilla.org/services/fx-home

– Firefox: XXX deﬁne a link to a search to display all related ﬁles in mozilla-central

– Firefox Mobile: XXX deﬁne a link to a search to display all related ﬁles in mobile-browser

Secure-Registration (Mozilla speciﬁc)

Warning: The Sreg service is used internally in the Mozilla infrastructure for security reasons and it is most likely

that you will never need it outside Mozilla.

Goal of the Service

The Sreg server provides web services on the top of the authentication back-end that can be used to:

• create a new user

• delete a user

• change a user password

• e-mail password reset codes

• delete password reset codes

It’s used by the mozilla auth backend in server-core in order to separate system writes that can be done with the users

credentials from those that require ldap admin credentials. This provides an additional level of security by keeping the

auth credentials only on a box with limited access. The mozilla backend is used by the Authentication server and the

Account manager server.

Note that some write operations (2) are not delegated, like changing the DN & the e-mail address of the user, so the

Account manager will still bind for writes as the user.

Sreg does not require any authentication, as it is not intended to be used without ﬁrst going through a primary gate-

way that performs any necessary authentication before proxying the requests. The server must remain private to our

infrastructure with as little outside access as possible

62 Chapter 2. Services

Mozilla Services Documentation, Release

Documentation content

Server API v1.0

General description

The URL for a Sreg request is structured as follows:

http://<server name>/<username>/<further instruction>

The Sync User API has a set of Response codes to cover errors in the request or on the server side. The format of a

successful response is deﬁned in the appropriate request method section.

APIs

GET https://server/username/node/weave

Gets the user a weave node. In general, if the Sreg version is called, it is expected that a new node will

be assigned (client should already have failed to get the node out of ldap), but this is not necessarily the

case, and it can return a currently assigned node.

The plan is to abstract this call into a separate api that handles node assignment, but it is noted here to

make sure we remain backwards compatible until that work is complete.

Return value: <node>

2.5. Secure-Registration (Mozilla speciﬁc) 63

Mozilla Services Documentation, Release

node may be ‘null’ if no node can be assigned at this time. Note that the returned value is a json-encoded

value, so strings are quoted.

Possible errors:

• 503: there was an error getting a node | empty body

• 404: user not found | empty body

GET https://server/username/password_reset_code

Generates a reset code and mails it to the email address associated with <username>. For security reasons,

it does not return the code to the registration server.

Sreg uses an internal constant to determine the URL to send users to in the email. It isn’t clear whether

this will pose a problem in the future.

Return value:

• 0 on success

Possible errors:

• 503: problems with looking up the user or sending the email | empty body

• 400: no email address on ﬁle for user | WEAVE_NO_EMAIL_ADRESS

• 404: user not found | empty body

PUT https://server/username

Requests that an account be created for username

The body is a JSON mapping and should include:

• password: the password to be associated with the account.

• e-mail: Email address associated with the account.

The server will return the username in a json-encoded value.

Possible errors:

• 503: there was an error creating the reset code

• 400: 4 (user already exists)

• 400: 6 (Json parse failure)

PUT https://server/username/password

Changes the password associated with the account to the value speciﬁed in the PUT body.

The PUT body is a JSON mapping containing:

• reset_code: the reset code

• password: the new password

Note that the server does not check if the password is valid. This should be done by the client.

Return values:

• 0: The operation was successful

• 400: 7 (Missing password ﬁeld)

• 400: 10 (Invalid or missing password reset code)

• 400: 6 (Json parse failure)

64 Chapter 2. Services

Mozilla Services Documentation, Release

• 404: the user does not exists in the database

• 503: there was an error updating the password (including a potentially bad reset code)

DELETE https://server/username/password_reset_code

Removes a password reset code, if any exists.

Return value:

• 0 on success

Possible errors:

• 503: there was an error removing the password reset code

• 404: the user does not exist in the database

DELETE https://server/username

Delete the user’s account.

The body is a JSON mapping and should include:

• password: The user’s password for conﬁrmation.

Return Value:

• 0 on success

• 400: 7 (Missing password ﬁeld)

• 400: 6 (Json parse failure)

• 404: the user does not exist in the database

• 503: there was an error removing the user (including a potential bad password)

History

XXX

Resources

• Original wiki design page/meeting notes: https://wiki.mozilla.org/Services/Sync/SRegAPI

• Continuous Integration server: XXX

• Server: https://hg.mozilla.org/services/server-sreg

• Client: https://hg.mozilla.org/services/server-core/ﬁle/tip/services/auth/mozilla_sreg.py

Token Server

Goal of the Service

So here’s the challenge we face. Current login for sync looks like this:

1. provide username and password

2. we log into ldap with that username and password and grab your sync node

2.6. Token Server 65

Mozilla Services Documentation, Release

3. we check the sync node against the url you’ve accessed, and use that to conﬁgure where your data is stored.

This solution works great for centralized login. It’s fast, has a minimum number of steps, and caches the data centrally.

The system that does node-assignment is lightweight, since the client and server both cache the result, and has support

for multiple applications with the /node/<app> API protocol.

However, this breaks horribly when we don’t have centralized login. And adding support for browserid to the Sync-

Storage protocol means that we’re now there. We’re going to get valid requests from users who don’t have an account

in LDAP. We won’t even know, when they make a ﬁrst request, if the node-assignment server has ever heard of them.

So, we have a bunch of requirements for the system. Not all of them are must-haves, but they’re all things we need to

think about trading off in whatever system gets designed:

• need to support multiple services (not necessarily centrally)

• need to be able to assign users to different machines as a service scales out, or somehow distribute them

• need to consistently send a user back to the same server once they’ve been assigned

• need to give operations some level of control over how users are allocated

• need to provide some recourse if a particular node dies

• need to handle exhaustion attacks. For example, I could set up an primary that just auto-approved any username,

then loop through users until all nodes were full.

• need support for future developments like bucketed assignment

• needs to be a system that scales inﬁnitely.

Assumptions

• A Login Server detains the secret for all the Service Nodes for a given Service.

• Any given webhead in a cluster can receive calls to all service nodes in the cluster.

• The Login Server will support only BrowserID at ﬁrst, but could support any authentication protocol in the

future, as long as it can be done with a single call

• All servers are time-synced

• The expires value for a token is a ﬁxed value per application. For example it could be 30 minutes for Sync and

2 hours for bipostal.

• The Login Server keeps a white list of domains for BID veriﬁcations

Documentation content

Token Server API v1.0

Note: Unless stated otherwise, all APIs are using application/json for the requests and responses content types.

GET /1.0/<app_name>/<app_version>

Asks for new token given some credentials in the Authorization header.

By default, the authentication scheme is BrowserID but other schemes can potentially be used if supported

by the login server.

• app_name is the name of the application to access, like sync.

66 Chapter 2. Services

Mozilla Services Documentation, Release

• app_version is the speciﬁc version number of the api that you want to access.

The ﬁrst /1.0/ in the URL deﬁnes the version of the authentication token itself.

Example for BrowserID:

GET /1.0/sync/1.5

Host: token.services.mozilla.com

Authorization: BrowserID <assertion>

This API returns several values in a json mapping:

• id – a signed authorization token, containing the user’s id for the application and the node.

• key – a secret derived from the shared secret

• uid – the user id for this service

• api_endpoint – the root URL for the user for the service.

• duration – the validity duration of the issued token, in seconds.

Example:

HTTP/1.1 200 OK

Content-Type: application/json

{'id': <token>,

'key': <derived-secret>,

'uid': 12345,

'api_endpoint': 'https://db42.sync.services.mozilla.com/1.5/12345',

'duration': 300,

}

If the X-Client-State header is included in the request, the server will compare the submitted value to

any previously-seen value. If it has changed then a new uid and api_endpoint are generated, in effect

“resetting” the node allocation for this user.

Request Headers

X-Client-State

An optional string that can be sent to identify a unique conﬁguration of client-side state. It may be up to

32 characters long, and must contain only characters from the urlsafe-base64 alphabet (i.e. alphanumeric

characters, underscore and hyphen) and the period.

A change in the value of this header will cause the user’s node allocation to be reset. Clients should include

any client-side state that is necessary for accessing the selected app. For example, clients accessing

SyncStorage API v1.5 would include a hex-encoded hash of the encryption key in this header, since a

change in the encryption key will make any existing data unreadable.

Updated values of the X-Client-State will be rejected with an error status of “invalid-client-state” if:

• The proposed new value is in the server’s list of previously-seen client-state values for that user.

• The client-state is missing or empty, but the server has previously seen a non-empty client-state for

that user.

• The user’s IdP provides generation numbers in their identity certiﬁcates, and the changed client-state

value does not correspond to an increase in generation number.

2.6. Token Server 67

Mozilla Services Documentation, Release

Response Headers

Retry-After

When sent together with an HTTP 503 status code, this header signiﬁes that the server is undergoing

maintenance. The client should not attempt any further requests to the server for the number of seconds

speciﬁed in the header value.

X-Backoff

This header may be sent to indicate that the server is under heavy load but is still capable of servicing

requests. Unlike the Retry-After header, X-Backoff may be included with any type of response, including

a 200 OK.

Clients should avoid unnecessary requests to the server for the number of seconds speciﬁed in the header

value. For example, clients may avoid pre-emptively refreshing token if an X-Backoff header was recently

seen.

X-Timestamp

This header will be included with all “200” and “401” responses, giving the current POSIX timestamp

as seen by the server, in seconds. It may be useful for client to adjust their local clock when generating

BrowserID assertions.

Error Responses

All errors are also returned, wherever possible, as json responses following the structure described in Cornice.

In cases where generating such a response is not possible (e.g. when a request if so malformed as to be unparsable)

then the resulting error response will have a Content-Type that is not application/json.

The top-level JSON object in the response will always contain a key named status, which maps to a string identifying

the cause of the error. Unexpected errors will have a status string of “error”; errors expected as part of the protocol

ﬂow will have a speciﬁc status string as detailed below.

Error status codes and their corresponding output are:

• 404 : unknown URL, or unsupported application.

• 400 : malformed request. Possible causes include a missing option, bad values or malformed json.

• 401 : authentication failed or protocol not supported. The response in that case will contain WWW-Authenticate

headers (one per supported scheme) and may report the following status strings:

– “invalid-credentials”: authentication failed due to invalid credentials e.g. a bad signature on the

BrowserID assertion.

– “invalid-timestamp”: authentication failed because the included timestamp differed too greatly from the

server’s current time.

– “invalid-generation”: authentication failed because the server has seen credentials with a more recent

generation number.

– “invalid-client-state”: authentication failed because the server has seen an updated value of the X-Client-

State header.

– “new-users-disabled”: authentication failed because the user has not been seen previously on this server,

and new user accounts have been disabled in the application conﬁg.

• 405 : unsupported method

• 406 : unacceptable - the client asked for an Accept we don’t support

68 Chapter 2. Services

Mozilla Services Documentation, Release

• 503 : service unavailable (ldap or snode backends may be down)

User Flow

Here’s the proposed two-step ﬂow (with BrowserID/FxA assertions):

1. the client trades a BrowserID assertion for an Auth token and corresponding secret

2. the client uses the auth token to sign subsequent requests using Hawk Auth.

Getting an Auth token:

seqdiag-8f2c0f2a4d333dd9d7424ad643ddc4670b0ee5f6.png

Calling the Service:

seqdiag-3610210bb1156abc60717e459e3fe6908a1a13c1.png

Detailed steps:

• the client requests a token, giving its browser id assertion [1]:

GET /1.0/sync/request_token HTTP/1.1

Host: token.services.mozilla.com

Authorization: Browser-ID <assertion>

• The Login Server checks the BrowserID assertion [2] this step will be done locally without calling an external

browserid server – but this could potentially happen (we can use PyBrowserID + use the BID.org certiﬁcate)

The user’s email address is extracted, along with any Generation Number associated with the BrowserID cer-

tiﬁcate.

• The Login Server asks the Users DB for an existing record matching the users’ email address.

If so, the allocated Node and previously-seen Generation Number are returned.

• If the submitted Generation Number is smaller than the recorded one, the Login Server returns an error as the

client’s BrowserID credentials are out of date.

If the submitted Generation Number is larger than the recorded one, the Login Server updates the Users DB

with the new value.

• If the user is not allocated to a Node, the Login Server asks for a new one from the Node Assignment Server [4]

• The Login Server creates a response with an Auth Token and corresponding Token Secret [5] and sends it back

to the user.

The Auth Token contains the user id and a timestamp, and is signed using the Signing Secret. The Token Secret

is derived from the Master Secret and Auth Token using HKDF.

It also adds the Node url in the response under api_endpoint [6]

2.6. Token Server 69

Mozilla Services Documentation, Release

HTTP/1.1 200 OK

Content-Type: application/json

{'id': <token>,

'secret': <derived-secret>,

'uid': 12345,

'api_endpoint': 'https://example.com/app/1.0/users/12345',

}

• The client saves the node location and hawkauth parameters to use in subsequent requests. [6]

• For each subsequent request to the Service, the client calculates a special Authorization header using Hawk Auth

[7] and sends the request to the allocated node location [8]:

POST /request HTTP/1.1

Host: some.node.services.mozilla.com

Authorization: Hawk id=<auth-token>

ts="137131201", (client timestamp)

nonce="7d8f3e4a",

mac="bYT5CMsGcbgUdFHObYMEfcx6bsw="

• The node uses the Signing Secret to validate the Auth Token [9]. If invalid or expired then the node returns a 401

• The node calculates the Token Secret from its Master Secret and the Auth Token, and checks whether the signa-

ture in the Authorization header is valid [10]. If it is invalid then the node returns a 401

• The node processes the request as deﬁned by the Service [11]

History

v1 - 2012-03-28

• Initial implementation

Resources

• Server: https://github.com/mozilla-services/tokenserver

Heka

Goal

Heka is a high-volume logging infrastructure designed to simplify data collection and analysis across multiple input

sources and output formats. It is lightweight, but can be expanded through the addition of plugins written in Go or

Lua.

It can be used for:

• Application performance metrics

• Server load, memory consumption, and other machine metrics

• Database, Cache server, and other daemon metrics

• Log-ﬁle parsing and shipping

70 Chapter 2. Services

Mozilla Services Documentation, Release

• Statsd-like time series data

It is currently being used at Mozilla in the Marketplace and Sync infrastructures.

Resources

• Heka documentation: https://hekad.readthedocs.io

• Heka binaries: https://github.com/mozilla-services/heka/releases

• Heka source: https://github.com/mozilla-services/heka

Loop Server

Goal of the Service

Loop server allows ﬁrefox users to call each others via WebRTC. It is a rendezvous API built on top of an external

service provider for NAT traversal and supplementary services.

Assumptions

• The Loop Server supports BrowserID authentication using FxA certiﬁcates and MSISDN certiﬁcates. It uses it

on the /register endpoint to create an Hawk session.

• All servers are time-synced

• The Loop server keeps a white list of origins that can do Cross-Origin resource sharing.

Documentation content

Loop Server API v1.0

This document is based on the current status of the server. All the examples had been done with real calls. It doesn’t

reﬂect any future implementation and tries to stick with the currently deployed version.

This document describes the HTTP APIs and the Websockets APIs.

Contents

• Loop Server API v1.0

– HTTP APIs

Overview

Versioning

Authentication

· Derive hawk credentials from the hawk session token

Conﬁguration

· GET /

2.8. Loop Server 71

Mozilla Services Documentation, Release

· GET /push-server-conﬁg

Healthcheck

· GET /__healthcheck__

Registration

· POST /registration

· DELETE /registration

Call URLs

· GET /call-url

· POST /call-url

· PUT /call-url/{token}

· DELETE /call-url/{token}

Calls

· GET /calls/{token}

· POST /calls/{token}

· POST /calls

· GET /calls?version=<version>

Rooms

· Basic Auth Authorization

· POST /rooms

· PATCH /rooms

· PATCH /rooms/:token

· DELETE /rooms/:token

· POST /rooms/:token

· Joining the room

· Refreshing membership in a room

· Leaving the room

· Update Status

· Shared Domain Logs

· POST /events

· GET /rooms/:token

· GET /rooms

· Participant information

Channel information

Account and Session

· DELETE /account

72 Chapter 2. Services

Mozilla Services Documentation, Release

· DELETE /session

Integration with Firefox Accounts using OAuth

· POST /fxa-oauth/params

· GET /fxa-oauth/token

· POST /fxa-oauth/token

Error Responses

– Websockets APIs

Call Setup States

Call Progress Protocol

· Initial Connection (hello)

· Call Progress State Change (progress)

· Client Action (action)

Termination Reasons

Timer Supervision

· Server Timers

· Supervisory Timer

· Ringing Timer

· Connection Timer

· Client Timers

· Response Timer

· Media Setup Timer

· Alerting Timer

HTTP APIs

Overview

Note: Unless stated otherwise, all APIs are using application/json for the requests and responses content types.

Parameters for the GET requests are form encoded (?key=value&key2=value2)

To ease testing, you can use httpie in order to make requests. Examples of use with httpie are provided when possible.

In order to authenticate with hawk, you’ll need to install the requests-hawk module

Versioning

The current API is versioned, using only a major version. All the endpoints for version 1 are preﬁxed by /v1/. In case

you don’t specify the preﬁx, your requests will be redirected automatically with an http 307 status.

2.8. Loop Server 73

Mozilla Services Documentation, Release

Authentication

To deal with authentication, the Loop server uses Hawk sessions. When you register, you can do so with different

authentications schemes, but you are always given an hawk session back, that you should use when requesting the

endpoints which need authentication.

When authenticating using the /register endpoint, you will be given an hawk session token in the Hawk-Session-Token

header. You will need to derive it, as explained at Derive hawk credentials from the hawk session token.

Derive hawk credentials from the hawk session token

In order to get the hawk credentials to use on the client you will need to:

1. Do an HKDF derivation on the given session token. You’ll need to use the following parameters:

key_material = HKDF(hawk_session, "", 'identity.mozilla.com/picl/v1/sessionToken',

˓→ 32

2. The key material you’ll get out of the HKDF need to be separated into two parts, the ﬁrst 32 hex caracters are

the hawk id, and the next 32 ones are the hawk key.

Credentials:

credentials = {

'id': keyMaterial[0:32]

'key': keyMaterial[32:64]

'algorithm': 'sha256'

}

If you are writting a client, you might ﬁnd these resources useful:

• With javascript: https://mxr.mozilla.org/mozilla-central/source/services/fxaccounts/FxAccountsClient.jsm#309

& https://github.com/mozilla/gecko-projects/blob/elm/browser/components/loop/content/shared/libs/token.js#

L55-L77

• With python: https://github.com/mozilla-services/loop-server/blob/master/loadtests/loadtest.py#L99-L122

Conﬁguration

GET /

Displays version information, for instance:

http GET localhost:5000/v1 --verbose

GET /v1/ HTTP/1.1

Accept:

Accept-Encoding: gzip, deflate

Host: localhost:5000

User-Agent: HTTPie/0.8.0

HTTP/1.1 200 OK

Connection: keep-alive

Content-Length: 247

74 Chapter 2. Services

Mozilla Services Documentation, Release

Content-Type: application/json; charset=utf-8

Date: Wed, 16 Jul 2014 12:57:13 GMT

ETag: W/"f7-762153207"

Timestamp: 1405515433

{

"description": "The Mozilla Loop (WebRTC App) server",

"endpoint": "http://localhost:5000",

"fakeTokBox": false,

"homepage": "https://github.com/mozilla-services/loop-server/",

"name": "mozilla-loop-server",

"version": "0.9.0"

}

GET /push-server-conﬁg

Retrieves the conﬁguration of the push server. Speciﬁcally, returns the websocket endpoint that should be

used to reach simple push.

The response should contain a pushServerURI parameter with this information.

http localhost:5000/push-server-config

HTTP/1.1 200 OK

Connection: keep-alive

Content-Length: 57

Content-Type: application/json; charset=utf-8

Date: Tue, 19 Aug 2014 14:26:42 GMT

ETag: W/"39-351294056"

Timestamp: 1408458402

{

"pushServerURI": "wss://push.services.mozilla.com/"

}

Server should acknowledge your request and answer with a status code of 200 OK.

Healthcheck

GET /__healthcheck__

• Returns 200 in case of success

• Returns 503 with the backend error message in case backends are broken

http localhost:5000/__heartbeat__

HTTP/1.1 200 OK

Connection: keep-alive

Content-Length: 32

Content-Type: application/json; charset=utf-8

Date: Fri, 07 Nov 2014 13:02:45 GMT

ETag: W/"20-e938360a"

Timestamp: 1415365365

2.8. Loop Server 75

Mozilla Services Documentation, Release

{

"provider": true,

"storage": true

}

Registration

POST /registration

Associates a Simple Push Endpoint (URL) with a user. Always return an hawk session token in the

Hawk-Session-Token header.

May require authentication

You don’t need to be authenticated to register. In case you don’t register with a Firefox Accounts assertion

or a valid hawk session, you’ll be given an hawk session token and be connected as an anonymous user.

This hawk session token should be derived by the client and used for subsequent requests.

You can currently authenticate by sending a valid Firefox Accounts assertion or a valid Hawk session.

Body parameters:

• simplePushURL, the simple push endpoint url as deﬁned in https://wiki.mozilla.org/WebAPI/

SimplePush#Deﬁnitions

Example (when not authenticated):

http POST localhost:5000/v1/registration --verbose \

simplePushURL=https://push.services.mozilla.com/update/MGlYke2SrEmYE8ceyu

POST /v1/registration HTTP/1.1

Accept: application/json

Accept-Encoding: gzip, deflate

Content-Length: 35

Content-Type: application/json; charset=utf-8

Host: localhost:5000

User-Agent: HTTPie/0.8.0

{

"simplePushURL": "https://test"

}

HTTP/1.1 200 OK

Access-Control-Expose-Headers: Hawk-Session-Token

Connection: keep-alive

Content-Length: 4

Content-Type: application/json; charset=utf-8

Date: Wed, 16 Jul 2014 12:58:56 GMT

Hawk-Session-Token:

˓→c7ee533a75a4f3b8a2a44b0b417eec15295ad43ff2b402776078ec87abb31cd9

Timestamp: 1405515536

"ok"

Server should acknowledge your request and answer with a status code of 200 OK.

Potential HTTP error responses include:

76 Chapter 2. Services

Mozilla Services Documentation, Release

• 400 Bad Request: You forgot to pass the simple_push_url, or it’s not a valid URL.

• 401 Unauthorized: The credentials you passed aren’t valid.

DELETE /registration

Requires authentication

Unregister the given session’s SimplePushURLs. The server will not be able to notify the client for this

session.

Example:

http DELETE localhost:5000/v1/registration --verbose \

--auth-type=hawk --auth=

˓→'c0d8cd2ec579a3599bef60f060412f01f5dc46f90465f42b5c47467481315f51:'

DELETE /v1/registration HTTP/1.1

Accept: application/json

Accept-Encoding: gzip, deflate

Authorization: <Stripped>

Host: localhost:5000

Content: 0

User-Agent: HTTPie/0.8.0

HTTP/1.1 204 No Content

Connection: keep-alive

Date: Wed, 16 Jul 2014 13:03:39 GMT

Server-Authorization: <stripped>

Server should acknowledge your request and answer with a status code of 204 No Content.

Potential HTTP error responses include:

• 400 Bad Request: You forgot to pass the simplePushURL, or it’s not a valid URL.

• 401 Unauthorized: The credentials you passed aren’t valid.

Call URLs

GET /call-url

Requires authentication

List all user valid call-urls.

Response from the server:

The server should answer this with a 200 status code and a list of JSON objects with the following

properties:

• callerId The name of the person to whom the call-url has been issued ;

• expires The date when the url will expire (the unix epoch, in seconds).

• timestamp The date when the url has been created (the unix epoch, in seconds).

Example:

2.8. Loop Server 77

Mozilla Services Documentation, Release

http GET localhost:5000/v1/call-url --verbose \

--auth-type=hawk --auth=

˓→'c0d8cd2ec579a3599bef60f060412f01f5dc46f90465f42b5c47467481315f51:'

GET /v1/call-url HTTP/1.1

Accept: application/json

Accept-Encoding: gzip, deflate

Authorization: <stripped>

Host: localhost:5000

User-Agent: HTTPie/0.8.0

HTTP/1.1 200 OK

Connection: keep-alive

Content-Length: 186

Content-Type: application/json; charset=utf-8

Date: Thu, 13 Nov 2014 16:19:59 GMT

Server-Authorization: <stripped>

Timestamp: 1415895599

[

{

"callerId": "Natim",

"expires": 1416499576,

"timestamp": 1415894776

}

]

Potential HTTP error responses include:

• 401 Unauthorized: You need to authenticate to call this URL.

• 503 Service Unavailable: Something is wrong on the server side.

POST /call-url

Requires authentication

Generates a call url for the given callerId. This is an URL the caller can click on in order to call the caller.

Body parameters:

• callerId, the caller (the person you will give the link to) identiﬁer.

• expiresIn, the number of hours the call-url will be valid for.

• issuer, The friendly name of the issuer of the token.

Optional parameters:

• subject, The subject of the conversation.

Response from the server:

The server should answer this with a 200 status code and a JSON object with the following properties:

• callUrl The call url;

• callToken The call token;

• expiresAt The date when the url will expire (the unix epoch, in seconds).

78 Chapter 2. Services

Mozilla Services Documentation, Release

Example:

http POST localhost:5000/v1/call-url --verbose \

callerId=Remy expiresIn=5 issuer=Alexis \

--auth-type=hawk --auth=

˓→'c0d8cd2ec579a3599bef60f060412f01f5dc46f90465f42b5c47467481315f51:'

POST /v1/call-url HTTP/1.1

Accept: application/json

Accept-Encoding: gzip, deflate

Authorization: <stripped>

Content-Length: 40

Content-Type: application/json; charset=utf-8

Host: localhost:5000

User-Agent: HTTPie/0.8.0

{

"callerId": "Remy",

"expiresIn": "5",

"issuer": "Alexis",

"subject": "MySubject"

}

HTTP/1.1 200 OK

Connection: keep-alive

Content-Length: 186

Content-Type: application/json; charset=utf-8

Date: Wed, 16 Jul 2014 13:09:40 GMT

Server-Authorization: <stripped>

Timestamp: 1405516180

{

"callToken": "_nxD4V4FflQ",

"callUrl": "http://localhost:3000/static/#call/_nxD4V4FflQ",

"expiresAt": 1405534180

}

Potential HTTP error responses include:

• 400 Bad Request: You forgot to pass the callerId, or it’s not valid;

• 401 Unauthorized: You need to authenticate to call this URL.

PUT /call-url/{token}

Requires authentication

Updates data associated with an already created call-url.

Body parameters:

• callerId, the caller (the person you will give the link to) identiﬁer. The callerId is supposed to be a

valid email address.

• expiresIn, the number of hours the call-url will be valid for.

• issuer, The friendly name of the issuer of the token.

Optional parameters:

2.8. Loop Server 79

Mozilla Services Documentation, Release

• subject, The subject of the conversation.

Response from the server:

The server should answer this with a 200 status code and a JSON object with the following properties:

• expiresAt The date when the url will expire (the unix epoch, in seconds).

Example:

http PUT localhost:5000/v1/call-url/B65nvlGh8iM --verbose \

issuer=Adam --auth-type=hawk --auth=

˓→'c0d8cd2ec579a3599bef60f060412f01f5dc46f90465f42b5c47467481315f51:'

PUT /v1/call-url/B65nvlGh8iM HTTP/1.1

Accept: application/json

Accept-Encoding: gzip, deflate

Authorization: <stripped>

Content-Length: 18

Content-Type: application/json; charset=utf-8

Host: localhost:5000

User-Agent: HTTPie/0.8.0

{

"issuer": "Adam",

"subject": "MySubject2"

}

HTTP/1.1 200 OK

Connection: keep-alive

Content-Length: 29

Content-Type: application/json; charset=utf-8

Date: Wed, 16 Jul 2014 14:16:54 GMT

Server-Authorization: <stripped>

Timestamp: 1405520214

{

"expiresAt": 1408112214

}

DELETE /call-url/{token}

Requires authentication

Delete a previously created call url. You need to be the user who generated this link in order to delete it.

Example:

http DELETE localhost:5000/v1/call-url/_nxD4V4FflQ --verbose \

--auth-type=hawk --auth=

˓→'c0d8cd2ec579a3599bef60f060412f01f5dc46f90465f42b5c47467481315f51:'

DELETE /v1/call-url/_nxD4V4FflQ HTTP/1.1

Accept:

Accept-Encoding: gzip, deflate

Authorization: <stripped>

Content-Length: 0

Host: localhost:5000

80 Chapter 2. Services

Mozilla Services Documentation, Release

User-Agent: HTTPie/0.8.0

HTTP/1.1 204 No Content

Connection: keep-alive

Date: Wed, 16 Jul 2014 13:12:46 GMT

Server-Authorization: <stripped>

Potential HTTP error responses include:

• 400 Bad Request: The token you passed is not valid or expired.

• 404 Not Found: The token you passed doesn’t exist.

Calls

GET /calls/{token}

Returns information about the token.

• token is the token returned by the POST on /call-url.

Response from the server:

The server should answer this with a 200 status code and a JSON object with the following properties:

• calleeFriendlyName the friendly name the creator of the call-url gave.

• urlCreationDate, the unix timestamp when the url was created.

Optional:

• subject, the subject of the conversation.

Example:

http GET localhost:5000/v1/calls/3jKS_Els9IU --verbose

GET /v1/calls/3jKS_Els9IU HTTP/1.1

Accept:

Accept-Encoding: gzip, deflate

Host: localhost:5000

User-Agent: HTTPie/0.8.0

HTTP/1.1 200 OK

Connection: keep-alive

Content-Length: 30

Content-Type: application/json; charset=utf-8

Date: Wed, 16 Jul 2014 13:23:04 GMT

ETag: W/"1e-2896316483"

Timestamp: 1405516984

{

"calleeFriendlyName": "Alexis",

"urlCreationDate": 1405517546,

"subject": "MySubject"

}

Potential HTTP error responses include:

2.8. Loop Server 81

Mozilla Services Documentation, Release

• 400 Bad Request: The token you passed is not valid or expired.

POST /calls/{token}

Creates a new incoming call for the given token. Gets tokens and session from the provider and does a

simple push notiﬁcation, then returns caller tokens.

Body parameters:

• callType, Speciﬁes the type of media the remote party intends to send. Valid values are “audio” or

“audio-video”.

Optional parameters:

• subject, the subject of the conversation

• channel, the TokBox channel to use for the call. More information about the channel parameter.

Server should answer with a status of 200 and the following information in its body (json encoded):

• apiKey, the provider public api Key.

• callId, an unique identiﬁer for the call;

• progressURL, the location to reach for websockets;

• sessionId, the provider session identiﬁer;

• sessionToken, the provider session token (for the caller);

• websocketToken, the token to use when authenticating to the websocket.

Example:

http POST localhost:5000/v1/calls/QzBbvGmIZWU callType="audio-video" --

˓→verbose

POST /v1/calls/QzBbvGmIZWU HTTP/1.1

Accept: application/json

Accept-Encoding: gzip, deflate

Content-Length: 27

Content-Type: application/json; charset=utf-8

Host: localhost:5000

User-Agent: HTTPie/0.8.0

{

"callType": "audio-video",

"channel": "nightly",

"subject": "MySubject"

}

HTTP/1.1 200 OK

Connection: keep-alive

Content-Length: 614

Content-Type: application/json; charset=utf-8

Date: Wed, 16 Jul 2014 13:37:39 GMT

Timestamp: 1405517859

{

"apiKey": "44669102",

"callId": "35e7c3a511f424d3b1d6fba442b3a9a5",

82 Chapter 2. Services

Mozilla Services Documentation, Release

"progressURL": "ws://localhost:5000/websocket",

"sessionId": "1_MX40NDY2OTEwMn5-V2VkIEp1bCAxNiAwNjo",

"sessionToken": "T1==cGFydG5lcl9pZD00NDY2OTEwMiZzaW",

"websocketToken": "44ee04b9694ae121c03a1db685cfad6d"

}

(note that return values have been truncated for readability purposes.)

Potential HTTP error responses include:

• 400 Bad Request: The token you passed is not valid.

• 410 Gone: The token expired.

POST /calls

Requires authentication

Similar to POST /calls/{token}, it creates a new incoming call to a known identity. Gets tokens and session

from the provider and does a simple push notiﬁcation, then returns caller tokens.

Body parameters:

• calleeId, array of strings containing the identities of the receiver(s) of the call. These identities

should be one of the valid Loop identities (Firefox Accounts email or MSISDN) and can belong to

none, an unique or multiple Loop users. It can also be an object with two properties:

– phoneNumber The phone number on a local form

– mcc The current SIM card Mobile Country Code

In that case, the server will try to convert the phoneNumber as an MSISDN identity

• callType, Speciﬁes the type of media the remote party intends to send. Valid values are “audio” or

“audio-video”.

Optional parameters:

• subject, the subject of the conversation

• channel, the client channel to use for the call. More information about the channel parameter.

Server should answer with a status of 200 and the following information in its body (json encoded):

• apiKey, the provider public api Key.

• callId, an unique identiﬁer for the call;

• progressURL, the location to reach for websockets;

• sessionId, the provider session identiﬁer;

• sessionToken, the provider session token (for the caller);

• websocketToken, the token to use when authenticating to the websocket.

Example:

http POST localhost:5000/v1/calls --verbose \

calleeId=alexis callType="audio-video" \

--auth-type=hawk --auth=

˓→'c0d8cd2ec579a3599bef60f060412f01f5dc46f90465f42b5c47467481315f51:'

2.8. Loop Server 83

Mozilla Services Documentation, Release

POST /v1/calls HTTP/1.1

Accept: application/json

Accept-Encoding: gzip, deflate

Authorization: <stripped>

Content-Length: 27

Content-Type: application/json; charset=utf-8

Host: localhost:5000

User-Agent: HTTPie/0.8.0

{

"callType": "audio-video"

"calleeId": ["[email protected]", "+34123456789"],

"channel": "nightly",

"subject": "MySubject"

}

HTTP/1.1 200 OK

Connection: keep-alive

Content-Length: 614

Content-Type: application/json; charset=utf-8

Date: Wed, 16 Jul 2014 13:37:39 GMT

Server-Authorization: <stripped>

Timestamp: 1405517859

{

"apiKey": "44669102",

"callId": "35e7c3a511f424d3b1d6fba442b3a9a5",

"progressURL": "ws://localhost:5000/websocket",

"sessionId": "1_MX40NDY2OTEwMn5-V2VkIEp1bCAxNiAwNjo",

"sessionToken": "T1==cGFydG5lcl9pZD00NDY2OTEwMiZzaW",

"websocketToken": "44ee04b9694ae121c03a1db685cfad6d"

}

(note that return values have been truncated for readability purposes.)

Potential HTTP error responses include:

• 400 Bad Request: You forgot to pass calleeId or is not valid.

• 401 Unauthorized: You need to authenticate to call this URL.

GET /calls?version=<version>

Requires authentication

List incoming calls for the authenticated user since the given version.

Querystring parameters:

• version, the version simple push gave to the client when waking it up. Only calls that happened

since this version will be returned.

Server should answer with a status of 200 and a list of calls in its body. Each call has the following

attributes:

• apiKey, the provider public api Key.

• callId, an unique identiﬁer for the call.

• callType, the call type (“audio” or “audio-video”).

84 Chapter 2. Services

Mozilla Services Documentation, Release

• progressURL, the location to reach for websockets.

• sessionId, the provider session identiﬁer.

• sessionToken, the provider session token (for the caller).

• websocketToken, the token to use when authenticating to the websocket.

Optional:

• subject, the subject of the call

In case of call initiated from an URL you will also have:

• callToken, the call-url token used for this call.

• callUrl, the call-url used for this call.

• urlCreationDate, the unix timestamp when the used call-url was created.

GET /v1/calls?version=0 HTTP/1.1

Accept:

Accept-Encoding: gzip, deflate

Authorization: <stripped>

Host: localhost:5000

User-Agent: HTTPie/0.8.0

HTTP/1.1 200 OK

Connection: keep-alive

Content-Length: 1785

Content-Type: application/json; charset=utf-8

Date: Wed, 16 Jul 2014 14:10:38 GMT

ETag: W/"6f9-2990115590"

Server-Authorization: <stripped>

Timestamp: 1405519838

{

"calls": [

{

"apiKey": "44669102",

"callId": "6744b8919d7d74e8c0b39590aa183565",

"callToken": "QzBbvGmIZWU",

"callUrl": "http://localhost:3000/static/#call/QzBbvGmIZWU",

"call_url": "http://localhost:3000/static/#call/QzBbvGmIZWU",

"callerId": "alexis",

"progressURL": "ws://localhost:5000/websocket",

"sessionId": "2_MX40NDY2OTEwMn5-

˓→V2VkIEp1bCAxNiAwNzoxMDoyMCBQRFQgMjAxNH4wLj",

"sessionToken":

˓→"T1==cGFydG5lcl9pZD00NDY2OTEwMiZzaWc9NzMyMGVmZjY1YWU0ZmFkZTY1NmU0",

"urlCreationDate": 1405517546,

"websocketToken": "a2fc1ee029169b62b08a4ba87c328d71",

"subject": "MySubject"

}

]

}

Potential HTTP error responses include:

• 400 Bad Request: The version you passed is not valid.

2.8. Loop Server 85

Mozilla Services Documentation, Release

Rooms

Some endpoints requires owner authentication, it is the account used to create the room on the POST /rooms.

On these endpoints only the owner can perform the action on the room.

Some endpoints requires participants authentiﬁcation, it is either the Hawk Session used to join the room using the

Hawk Authorization scheme or the sessionToken the user has got when joining anonymously using the Basic Auth

Authorization scheme.

Basic Auth Authorization

In that case, just use the room participant sessionToken as a Basic Auth username with no password.

http POST localhost:5000/rooms/:token –auth “_sessionToken_:”

Authorization: Basic X3Nlc3Npb25Ub2tlbl86

POST /rooms

Requires owner authentication

Creates a new room

Request body parameters:

• roomOwner, The room owner name.

• maxSize, The maximum number of people the room can handle.

At least one of the following two parameters must be supplied:

• context, An encrypted room context string.

• roomName, The name of the room, this is now obsolete, but remains to support older clients.

Optional parameter:

• expiresIn, the number of hours for which the room will exist.

• channel, the client channel to use for the room. More information about the channel parameter.

Response body parameters:

• roomToken, The token used to identify the created room.

• roomUrl, A URL that can be given to other users to allow them to join the room.

• expiresAt, The date after which the room will no longer be valid (in seconds since the Unix epoch).

Potential HTTP error responses include:

• 400 Bad Request: Missing or invalid body parameters

Example:

http POST localhost:5000/v1/rooms --verbose \

roomName="My Room" roomOwner="Natim" maxSize=5 \

--auth-type=hawk --auth=

˓→'c0d8cd2ec579a3599bef60f060412f01f5dc46f90465f42b5c47467481315f51:'

86 Chapter 2. Services

Mozilla Services Documentation, Release

POST /rooms HTTP/1.1

Accept: application/json

Accept-Encoding: gzip, deflate

Authorization: <stripped>

Content-Length: 61

Content-Type: application/json; charset=utf-8

Host: localhost:5000

User-Agent: HTTPie/0.8.0

{

"maxSize": "5",

"roomName": "My Room",

"roomOwner": "Natim"

}

HTTP/1.1 201 Created

Connection: keep-alive

Content-Length: 109

Content-Type: application/json; charset=utf-8

Date: Mon, 10 Nov 2014 14:29:41 GMT

Server: nginx/1.6.1

Server-Authorization: <stripped>

{

"expiresAt": 1418221780,

"roomToken": "pPVoaqiH89M",

"roomUrl": "http://localhost:3000/static/#rooms/pPVoaqiH89M"

}

PATCH /rooms

Requires owner authentication

Remove given rooms

Request body parameters:

• deleteRoomTokens, a list of rooms to delete.

Response body parameters:

• responses, a mapping of room’s tokens and the request’s status for each.

Potential HTTP error responses include:

• 207 Multi-Status: When tokens are processed, each token having it’s own status.

• 404 Not Found: If none of the given roomTokens where found for this user.

• 400 Bad Requests: If no room tokens where provided.

Example:

echo '{"deleteRoomTokens": ["pPVoaqiH89M"]}' | http PATCH localhost:5000/v1/

˓→rooms -v \

--auth-type=hawk --auth=

˓→'c0d8cd2ec579a3599bef60f060412f01f5dc46f90465f42b5c47467481315f51:'

2.8. Loop Server 87

Mozilla Services Documentation, Release

PATCH /rooms HTTP/1.1

Accept: application/json

Accept-Encoding: gzip, deflate

Authorization: <stripped>

Content-Length: 39

Content-Type: application/json; charset=utf-8

Host: localhost:5000

User-Agent: HTTPie/0.8.0

{

"deleteRoomTokens": ["pPVoaqiH89M"]

}

HTTP/1.1 207 Multi-Status

Connection: keep-alive

Content-Length: 40

Content-Type: application/json; charset=utf-8

Date: Tue, 30 Dec 2014 15:39:41 GMT

Server: nginx/1.6.1

Server-Authorization: <stripped>

{

"responses": {

"pPVoaqiH89M": {"code": 200},

"_nxD4V4FflQ": {"code": 404, "errno": "105", "message": "Room not

˓→found."}

}

PATCH /rooms/:token

Requires owner authentication

Updates an existing room

Optional request body parameters:

• context, An encrypted room context string.

• roomName, The name of the room.

• roomOwner, The room owner name.

• maxSize, The maximum number of people the room can handle.

• expiresIn, the number of hours for which the room will exist.

You only need set the body parameters you want to update.

Response body parameters:

• expiresAt, The date after which the room will no longer be valid (in seconds since the Unix epoch)

Potential HTTP error responses include:

• 400 Bad Request: Missing or invalid body parameters

Example:

http PATCH localhost:5000/v1/rooms/pPVoaqiH89M --verbose \

roomName="My Room" roomOwner="Natim" maxSize=5 \

--auth-type=hawk --auth=

˓→'c0d8cd2ec579a3599bef60f060412f01f5dc46f90465f42b5c47467481315f51:'

88 Chapter 2. Services

Mozilla Services Documentation, Release

PATCH /rooms/pPVoaqiH89M HTTP/1.1

Accept: application/json

Accept-Encoding: gzip, deflate

Authorization: <stripped>

Content-Length: 61

Content-Type: application/json; charset=utf-8

Host: localhost:5000

User-Agent: HTTPie/0.8.0

{

"maxSize": "5",

"roomName": "My Room",

"roomOwner": "Natim"

}

HTTP/1.1 200 OK

Connection: keep-alive

Content-Length: 24

Content-Type: application/json; charset=utf-8

Date: Mon, 10 Nov 2014 14:33:19 GMT

Server: nginx/1.6.1

Server-Authorization: <stripped>

Timestamp: 1415629999

{

"expiresAt": 1418221999

}

DELETE /rooms/:token

Requires owner authentication

Deletes an existing room.

Example:

http DELETE localhost:5000/v1/rooms/LURlwjMc8wI --verbose \

--auth-type=hawk --auth=

˓→'c0d8cd2ec579a3599bef60f060412f01f5dc46f90465f42b5c47467481315f51:'

DELETE /rooms/LURlwjMc8wI HTTP/1.1

Accept:

Accept-Encoding: gzip, deflate

Authorization: <stripped>

Content-Length: 0

Host: localhost:5000

User-Agent: HTTPie/0.8.0

HTTP/1.1 204 No Content

Connection: keep-alive

Date: Mon, 10 Nov 2014 14:35:37 GMT

Server: nginx/1.6.1

Server-Authorization: <stripped>

2.8. Loop Server 89

Mozilla Services Documentation, Release

POST /rooms/:token

This endpoint handles three kinds of actions:

• join, A new participant joins the room.

• refresh, A participant notiﬁes she is still in the room.

• leave, A participant notiﬁes she is leaving the room.

• status, A participant update his status in the room.

• logDomain, A participant send metrics about whitelisted shared domains.

Joining the room

Request body parameters:

• action, Should be “join” in that case.

• displayName, The participant friendly name for this room.

• clientMaxSize, Maximum number of room participants the user’s client is capable of supporting.

Response body parameters:

• apiKey, The TokBox public api key.

• sessionId, The TokBox session identiﬁer (identiﬁes the room).

• sessionToken, The TokBox session token (identiﬁes the room participant).

• expires, The number of seconds within which the client must send another POST to this endpoint

with the refresh action to remain a participant in this room.

Potential HTTP error responses include:

• 400 Bad Request: Missing or invalid body parameters

Example:

http POST localhost:5000/v1/rooms/pPVoaqiH89M --verbose \

action=join displayName=Natim clientMaxSize=5 \

--auth-type=hawk --auth=

˓→'c0d8cd2ec579a3599bef60f060412f01f5dc46f90465f42b5c47467481315f51:'

POST /rooms/pPVoaqiH89M HTTP/1.1

Accept: application/json

Accept-Encoding: gzip, deflate

Authorization: <stripped>

Content-Length: 64

Content-Type: application/json; charset=utf-8

Host: localhost:5000

User-Agent: HTTPie/0.8.0

{

"action": "join",

"clientMaxSize": "5",

"displayName": "Natim"

}

HTTP/1.1 200 OK

90 Chapter 2. Services

Mozilla Services Documentation, Release

Connection: keep-alive

Content-Length: 461

Content-Type: application/json; charset=utf-8

Date: Mon, 10 Nov 2014 14:39:12 GMT

Server: nginx/1.6.1

Server-Authorization: <stripped>

Timestamp: 1415630346

{

"apiKey": "44669102",

"expires": 300,

"sessionId": "1_XM40NYDO2TEwMI5-

˓→MTQxNTYyOTc4MTIzOH5PaGxlZlNRTXdqVi9XRGUIel8jZWh0KZz-VH4",

"sessionToken": "T1==cGFydG5lcl9pZD00NDY2OTEw...=="

}

Refreshing membership in a room

Requires participant authentication

Request body parameters:

• action, Should be “refresh” in that case.

Potential HTTP error responses include:

• 400 Bad Request: Missing or invalid body parameters

• 410 Participation has expired: The referesh did not occur within the speciﬁed time period.

Example:

http POST localhost:5000/v1/rooms/pPVoaqiH89M --verbose \

action=refresh \

--auth-type=hawk --auth=

˓→'c0d8cd2ec579a3599bef60f060412f01f5dc46f90465f42b5c47467481315f51:'

POST /rooms/pPVoaqiH89M HTTP/1.1

Accept: application/json

Accept-Encoding: gzip, deflate

Authorization: <stripped>

Content-Length: 21

Content-Type: application/json; charset=utf-8

Host: localhost:5000

User-Agent: HTTPie/0.8.0

{

"action": "refresh"

}

HTTP/1.1 200 OK

Connection: keep-alive

Content-Length: 461

Content-Type: application/json; charset=utf-8

Date: Mon, 10 Nov 2014 14:40:06 GMT

Server: nginx/1.6.1

Server-Authorization: <stripped>

Timestamp: 1415630346

2.8. Loop Server 91

Mozilla Services Documentation, Release

{

"expires": 300

}

Leaving the room

Request body parameters:

• action, Should be “leave” in that case.

The endpoint will return a 204 No Content response.

Potential HTTP error responses include:

• 400 Bad Request: Missing or invalid body parameters

Example:

http POST localhost:5000/v1/rooms/pPVoaqiH89M --verbose \

action=leave \

--auth-type=hawk --auth=

˓→'c0d8cd2ec579a3599bef60f060412f01f5dc46f90465f42b5c47467481315f51:'

POST /rooms/pPVoaqiH89M HTTP/1.1

Accept: application/json

Accept-Encoding: gzip, deflate

Authorization: <stripped>

Content-Length: 19

Content-Type: application/json; charset=utf-8

Host: localhost:5000

User-Agent: HTTPie/0.8.0

{

"action": "leave"

}

HTTP/1.1 204 No Content

Connection: keep-alive

Date: Mon, 10 Nov 2014 14:48:24 GMT

Server: nginx/1.6.1

Server-Authorization: <stripped>

Update Status

This endpoint is used to send some WebRTC metrics to be logged on server side.

Request body parameters:

• action, Should be “status” in that case.

The endpoint will return a 204 No Content response.

Potential HTTP error responses include:

• 400 Bad Request: Missing or invalid body parameters

Example:

92 Chapter 2. Services

Mozilla Services Documentation, Release

http POST localhost:5000/v1/rooms/pPVoaqiH89M --verbose \

action=status event=Session.connectionCreated state=sendrecv \

connections=2 sendStreams=1 recvStreams=1 \

--auth-type=hawk --auth=

˓→'c0d8cd2ec579a3599bef60f060412f01f5dc46f90465f42b5c47467481315f51:'

POST /rooms/pPVoaqiH89M HTTP/1.1

Accept: application/json

Accept-Encoding: gzip, deflate

Authorization: <stripped>

Content-Length: 19

Content-Type: application/json; charset=utf-8

Host: localhost:5000

User-Agent: HTTPie/0.8.0

{

"action": "status",

"event": "Session.connectionCreated",

"state": "sendrecv",

"connections": 2,

"sendStreams": 1,

"recvStreams": 1

}

HTTP/1.1 204 No Content

Connection: keep-alive

Date: Mon, 10 Nov 2014 14:48:24 GMT

Server: nginx/1.6.1

Server-Authorization: <stripped>

Shared Domain Logs

This endpoint is used to log domains that where shared with the Loop-client tab sharing feature.

Request body parameters:

• action, Should be “logDomain” in that case.

The endpoint will return a 204 No Content response.

Potential HTTP error responses include:

• 400 Bad Request: Missing or invalid body parameters

Example:

echo '{

"action": "logDomain",

"domains": [{"domain": "mozilla.org", "count": 1}, {"domain": "others":

˓→10}]

}' | \

http POST localhost:5000/v1/rooms/pPVoaqiH89M --verbose \

--auth-type=hawk --auth=

˓→'c0d8cd2ec579a3599bef60f060412f01f5dc46f90465f42b5c47467481315f51:'

POST /rooms/pPVoaqiH89M HTTP/1.1

Accept: application/json

2.8. Loop Server 93

Mozilla Services Documentation, Release

Accept-Encoding: gzip, deflate

Authorization: <stripped>

Content-Length: 19

Content-Type: application/json; charset=utf-8

Host: localhost:5000

User-Agent: HTTPie/0.8.0

{

"action": "logDomain",

"domains": [{

"domain": "mozilla.org",

"count": 1

}, {

"domain": "others",

"count": 10

}]

}

HTTP/1.1 204 No Content

Connection: keep-alive

Date: Mon, 10 Nov 2014 14:48:24 GMT

Server: nginx/1.6.1

Server-Authorization: <stripped>

POST /events

Requires authentication

Event logging to the Google Analytics panel.

• event, The event name

• action, The action

• label, The label

The endpoint will return a 204 No Content response.

Potential HTTP error responses include:

• 400 Bad Request: Missing or invalid body parameters

http POST http://localhost:5000/v1/event -v \

event="Addon" action="start" label="Clicked Start Browsing" \

--auth-type=hawk --auth=

˓→'ca13d91d1d4b67edf0b9523a2867b3d1b74eb63823732c441992f813f9da1f76:' --json

POST /v1/event HTTP/1.1

Accept: application/json

Accept-Encoding: gzip, deflate

Authorization: <stripped>

Content-Type: application/json; charset=utf-8

Host: localhost:5000

User-Agent: HTTPie/0.8.0

{

"action": "start",

"event": "Addon",

94 Chapter 2. Services

Mozilla Services Documentation, Release

"label": "Clicked Start Browsing."

}

HTTP/1.1 204 No Content

Connection: keep-alive

Date: Mon, 29 Feb 2016 14:39:41 GMT

Server: nginx/1.6.1

Server-Authorization: Hawk mac="jcDymnhIosT8NzEdd+JZ9mwv4ZXHKDyfK/1+04gm5bA="

Vary: Origin

GET /rooms/:token

Retrieves information about the room.

Response body parameters:

• roomToken, The token used to identify this room.

• context, An encrypted room context string.

• roomName, The name of the room.

• roomUrl, A URL that can be given to other users to allow them to join the room.

• roomOwner, The user-friendly display name indicating the name of the room’s owner.

If a participant authentication is provided, additional information is returned:

• maxSize, The maximum number of users allowed in the room at one time (as conﬁgured by the

room owner).

• clientMaxSize, The current maximum number of users allowed in the room, as constrained by

the clients currently participating in the session. If no client has a supported size smaller than

“maxSize”, then this will be equal to “maxSize”. Under no circumstances can “clientMaxSize” be

larger than “maxSize”.

• creationTime, The time (in seconds since the Unix epoch) at which the room was created.

• expiresAt, The time (in seconds since the Unix epoch) at which the room goes away.

• participants, An array containing a list of the current room participants. More information about

the participant properties.

• ctime, The time, in seconds since the Unix epoch, that any of the following happened to the room:

– The room was created

– The owner modiﬁed its attributes with “PATCH /rooms/{token}”

– A user joined the room

– A user left the room

Example:

http GET localhost:5000/v1/rooms/pPVoaqiH89M --verbose \

--auth-type=hawk --auth=

˓→'c0d8cd2ec579a3599bef60f060412f01f5dc46f90465f42b5c47467481315f51:'

2.8. Loop Server 95

Mozilla Services Documentation, Release

GET /rooms/pPVoaqiH89M HTTP/1.1

Accept:

Accept-Encoding: gzip, deflate

Authorization: <stripped>

Host: localhost:5000

User-Agent: HTTPie/0.8.0

HTTP/1.1 200 OK

Connection: keep-alive

Content-Length: 284

Content-Type: application/json; charset=utf-8

Date: Mon, 10 Nov 2014 14:52:20 GMT

ETag: W/"11c-d426a3d5"

Server: nginx/1.6.1

Server-Authorization: <stripped>

Timestamp: 1415631140

{

"clientMaxSize": 5,

"creationTime": 1415629780,

"ctime": 1415631010,

"expiresAt": 1418221999,

"maxSize": 5,

"participants": [

{

"displayName": "Natim",

"roomConnectionId": "0bc7fa46-3df0-4621-b904-afdd2390d9ef",

"owner": true,

"account": "[email protected]"

}

"roomName": "My Room",

"roomOwner": "Natim",

"roomUrl": "http://locahost:3000/#/rooms/pPVoaqiH89M"

}

GET /rooms

Requires owner authentication

Retrieves a list of rooms owned by the owner.

The response is a list of objects with this information:

• roomToken, The token used to identify this room.

• roomName, The name of the room.

• context, An encrypted room context string.

• roomUrl, A URL that can be given to other users to allow them to join the room.

• roomOwner, The user-friendly display name indicating the name of the room’s owner.

• maxSize, The maximum number of users allowed in the room at one time (as conﬁgured by the

room owner).

96 Chapter 2. Services

Mozilla Services Documentation, Release

• clientMaxSize, The current maximum number of users allowed in the room, as constrained by

the clients currently participating in the session. If no client has a supported size smaller than

“maxSize”, then this will be equal to “maxSize”. Under no circumstances can “clientMaxSize” be

larger than “maxSize”.

• creationTime, The time (in seconds since the Unix epoch) at which the room was created.

• expiresAt, The time (in seconds since the Unix epoch) at which the room goes away.

• participants, An array containing a list of the current room participants. More information about

the participant properties.

• ctime, The time, in seconds since the Unix epoch, that any of the following happened to the room:

– The room was created

– The owner modiﬁed its attributes with “PATCH /rooms/{token}”

– A user joined the room

– A user left the room

Example:

http GET localhost:5000/v1/rooms --verbose \

--auth-type=hawk --auth=

˓→'c0d8cd2ec579a3599bef60f060412f01f5dc46f90465f42b5c47467481315f51:'

GET /rooms/ HTTP/1.1

Accept:

Accept-Encoding: gzip, deflate

Authorization: <stripped>

Host: localhost:5000

User-Agent: HTTPie/0.8.0

HTTP/1.1 200 OK

Connection: keep-alive

Content-Length: 312

Content-Type: application/json; charset=utf-8

Date: Mon, 10 Nov 2014 14:50:12 GMT

ETag: W/"138-9bb2c1c"

Server: nginx/1.6.1

Server-Authorization: <stripped>

Timestamp: 1415631012

[

{

"clientMaxSize": 5,

"creationTime": 1415629780,

"ctime": 1415631010,

"expiresAt": 1418221999,

"maxSize": 5,

"participants": [

{

"displayName": "Natim",

"roomConnectionId": "0bc7fa46-3df0-4621-b904-afdd2390d9ef",

"owner": true,

"account": "[email protected]"

}

2.8. Loop Server 97

Mozilla Services Documentation, Release

"roomName": "My Room",

"roomOwner": "Natim",

"roomToken": "pPVoaqiH89M",

"roomUrl": "http://localhost:3000/static/#rooms/pPVoaqiH89M"

}

]

Participant information

When retrieving the room information you get a list of participants. It is a list of objects with these

properties:

• displayName, The user-friendly name that should be displayed for this participant.

• account, If the user is logged in, this is the FxA account name or MSISDN that was used to authen-

ticate the user for this session.

• owner, if the user is also the owner of the room, this property will be true, it will be false otherwise.

• roomConnectionId, An id, unique within the room for the lifetime of the room, used to identify a

partcipant for the duration of one instance of joining the room. If the user departs and re-joins, this

id will change.

Channel information

The client can send its channel in order to let the server use different API keys (of the underlying service provider)

depending on it.

Channel can be one of:

• release

• esr

• beta

• aurora

• nightly

• default

• mobile – used for the Firefox OS Mobile client

• standalone – used for the standalone / “link-clicker” client

Account and Session

DELETE /account

Requires authentication

Deletes the current account and all data associated to it.

Example:

http DELETE localhost:5000/v1/account --verbose \

--auth-type=hawk --auth=

˓→'c0d8cd2ec579a3599bef60f060412f01f5dc46f90465f42b5c47467481315f51:'

98 Chapter 2. Services

Mozilla Services Documentation, Release

DELETE /v1/account HTTP/1.1

Accept:

Accept-Encoding: gzip, deflate

Authorization: <stripped>

Content-Length: 0

Host: localhost:5000

User-Agent: HTTPie/0.8.0

HTTP/1.1 204 No Content

Connection: keep-alive

Date: Wed, 16 Jul 2014 13:03:39 GMT

Server-Authorization: <stripped>

DELETE /session

Requires authentication

Deletes the current session.

This should be used to clear the hawk session of a Firefox Account user. You should not attempt to call

this endpoint with a non-ﬁrefox account session, since it would mean as a client you could not attach a

session anymore.

In case you want to destroy a non-FxA session, please use the DELETE /account endpoint.

Example:

http DELETE localhost:5000/v1/session --verbose \

--auth-type=hawk --auth=

˓→'c0d8cd2ec579a3599bef60f060412f01f5dc46f90465f42b5c47467481315f51:'

DELETE /v1/session HTTP/1.1

Accept:

Accept-Encoding: gzip, deflate

Authorization: <stripped>

Content-Length: 0

Host: localhost:5000

User-Agent: HTTPie/0.8.0

HTTP/1.1 204 No Content

Connection: keep-alive

Date: Wed, 16 Jul 2014 13:03:39 GMT

Server-Authorization: <stripped>

Potential HTTP error responses include:

• 403 Forbidden: If you remove this session you will loose access to your loop-server data because

you will not be able to link them to a new session. Use DELETE /account instead.

Integration with Firefox Accounts using OAuth

A few endpoints are available for integration with Firefox Accounts. This is the prefered way to login with your

Firefox Accounts for loop. For more information on how to integrate with Firefox Accounts, have a look at the

Firefox Accounts documentation on MDN

2.8. Loop Server 99

Mozilla Services Documentation, Release

POST /fxa-oauth/params

Requires authentication

Provide the client with the parameters needed for the OAuth dance.

• client_id, the client id used by the server;

• content_uri, URI of the content server (to get account information);

• oauth_uri, URI of the OAuth server;

• redirect_uri, URI where the client should redirect once authenticated;

• scope, The scope of the token returned;

• state, A nonce used to check that the session matches.

http POST http://localhost:5000/v1/fxa-oauth/params --verbose \

--auth-type=hawk --auth=

˓→'ca13d91d1d4b67edf0b9523a2867b3d1b74eb63823732c441992f813f9da1f76:' --json

POST /v1/fxa-oauth/params HTTP/1.1

Accept: application/json

Accept-Encoding: gzip, deflate

Authorization: <stripped>

Content-Type: application/json; charset=utf-8

Host: localhost:5000

User-Agent: HTTPie/0.8.0

HTTP/1.1 200 OK

Connection: keep-alive

Server-Authorization: <stripped>

Timestamp: 1409052727

{

"client_id": "263ceaa5546dce83",

"content_uri": "https://accounts.firefox.com",

"oauth_uri": "https://oauth.accounts.firefox.com/v1",

"redirect_uri": "urn:ietf:wg:oauth:2.0:fx:webchannel",

"scope": "profile",

"state":

˓→"b56b3753c15efdcae80ea208134ecd6ae97f27027ce9bb11f7c333be6ea7029c"

}

GET /fxa-oauth/token

Requires authentication

Returns the current status of the hawk session (e.g. if it’s authenticated or not):

http GET http://localhost:5000/v1/fxa-oauth/token --verbose \

--auth-type=hawk --auth=

˓→'ca13d91d1d4b67edf0b9523a2867b3d1b74eb63823732c441992f813f9da1f76:' --json

If the current session is authenticated using OAuth, it returns it in the access_token attribute.

100 Chapter 2. Services

Mozilla Services Documentation, Release

GET /v1/fxa-oauth/token HTTP/1.1

Accept: application/json

Accept-Encoding: gzip, deflate

Authorization: <stripped>

Content-Type: application/json; charset=utf-8

Host: localhost:5000

User-Agent: HTTPie/0.8.0

HTTP/1.1 200 OK

Connection: keep-alive

Content-Type: application/json; charset=utf-8

Server-Authorization: <stripped>

Timestamp: 1409058431

POST /fxa-oauth/token

Requires authentication

Trades an OAuth code with an oauth bearer token:

http POST http://localhost:5000/v1/fxa-oauth/token --verbose \

state=b56b3753c15efdcae80ea208134ecd6ae97f27027ce9bb11f7c333be6ea7029c \

code=12345 \

--auth-type=hawk --auth=

˓→'ca13d91d1d4b67edf0b9523a2867b3d1b74eb63823732c441992f813f9da1f76:' --json

Checks the validity of the given code and state and exchange it with a bearer token with the OAuth servers.

The token is returned in the access_token attribute. A few additional parameters are returned:

• scope the scope of the token;

• token_type the type of the token returned (here, it will be “bearer”).

Error Responses

All errors are also returned, wherever possible, as json responses with a code, errno and error message.

Error status codes and codes and their corresponding outputs are:

• 404 : unknown URL, or unsupported application.

• 400 : malformed request. Possible causes include a missing option, bad values or malformed json.

• 401 : you need to be authenticated

• 403 [you are authenticated but don’t have access to the resource you are] requesting.

• 405 : unsupported method

• 406 : unacceptable - the client asked for an Accept we don’t support

• 503 : service unavailable (provider or database backends may be down)

Also the associated errno can be one of:

• 105 INVALID_TOKEN: This come with a 404 on a wrong call-url token;

• 106 BADJSON: This come with a 406 if the sent JSON is not parsable;

• 107 INVALID_PARAMETERS: This come with a 400 and describe invalid parameters with a reason;

2.8. Loop Server 101

Mozilla Services Documentation, Release

• 108 MISSING_PARAMETERS: This come with a 400 and list all missing parameters;

• 110 INVALID_AUTH_TOKEN: This come with a 401 and deﬁne a problem during Auth;

• 111 EXPIRED: This come with a 410 and deﬁne a EXPIRE ressource;

• 113 REQUEST_TOO_LARGE: This come with a 400 and deﬁne a too large request;

• 114 INVALID_OAUTH_STATE: This come with a 400 and tells the oauth state is invalid;

• 122 USER_UNAVAILABLE: This come with a 400 and tell the user could not be found in the database;

• 201 BACKEND: This come with a 503 when a third party is not available at the moment.

Websockets APIs

During the setup phase of a call, the websocket protocol is used to let clients broadcast their state to other clients and

to listen to changes.

The client will establish a WebSockets connection to the resource indicated in the “progressURL” when it receives it.

The client never closes this connection; that is the responsibility of the server. The times at which the server closes the

connection are detailed below. If the server sees the client close the connection, it assumes that the client has failed,

and informs the other party of such call failure.

For forward compatibility purposes:

• Unknown ﬁelds in messages are ignored

• Unknown message types received by the client (indicating an earlier release) result in the client sending an

“error” message ({“messageType”: “error”, “reason”: “unknown message”}). The call setup should continue.

• Unknown message types received by the server result in the server sending an “error” message (as above);

however, since this situation can only arise due to a misimplemented client or an out-of-date server, it results in

call setup failure. The server closes the connection.

Call Setup States

Call setup goes through the following states:

102 Chapter 2. Services

Mozilla Services Documentation, Release

Call Progress Protocol

Initial Connection (hello)

Upon connecting to the server, the client sends an immediate “hello” message, which serves two purposes: it identiﬁes

the call that the progress channel corresponds to (using the “callId”), as well as authenticating the connecting user, so

that they can be veriﬁed to be authorized to view/impact the call setup state.

Note that the callId with which this connection is to be associated is encoded as a component of the WSS URL.

UA -> Server:

2.8. Loop Server 103

Mozilla Services Documentation, Release

{

"messageType": "hello",

"auth": "''<authentication information>''"

}

• auth: Information to authenticate the user, so that they can be veriﬁed to be authorized to access call setup

information. This is the websocketToken returned by a POST to /calls/{token}, POST /calls and GET /calls.

If the hello is valid (the callId is known, the auth information is valid, and the authenticated user is a party to the call),

then the server responds with a “hello.” This “hello” includes the current call setup state.

Server -> UA:

{

"messageType": "hello",

"state": "alerting"

// may contain "reason" field for certain states.

}

• state: See states in “progress”, below.

If the hello is invalid for any reason, then the server sends an “error” message, as follows. It then closes the connection.

Server -> UA:

{

"messageType": "error",

"reason": "unknown callId"

}

reason: The reason the hello was rejected:

• unknown callId

• invalid authentication - The auth information was not valid

• unauthorized - The auth information was valid, but did not match the indicated callId

Call Progress State Change (progress)

The server informs users of the current state of call setup. The state sent to both parties ‘’is always the same state’‘.

So, for example, when a user rejects a call, he will receive a “progress” message with a state of “terminated” and a

reason of “rejected.”

Server -> UA:

{

"messageType": "progress",

"state": "alerting"

// may contain optional "reason" field for certain events.

}

Deﬁned states are:

• init: The call is starting, and the remote party is not yet being alerted.

• alerting: The called party is being alerted (triggered by remote party sending a “hello” message).

104 Chapter 2. Services

Mozilla Services Documentation, Release

• terminated: The call is no longer being set up. After sending a “terminated” message, the server closes the

WebSockets connection. This message will include a “reason” ﬁeld with one of the reason values described

below.

• connecting: The called party has indicated that he has answered the call, but the media is not yet conﬁrmed

• half-connected: One of the two parties has indicated successful media set up, but the other has not yet.

• connected: Both endpoints have reported successfully establishing media. After sending a “connected” message,

the server closes the WebSockets connection.

Client Action (action)

During call setup, clients send progress information about their own state so that it can be reﬂected in the call state.

UA -> Server:

{

"messageType": "action",

"event": "accept"

// May contain "reason" field for certain events

}

Deﬁned event types are:

• accept: Only sent by called party. The user has answered this call. This is sent before the called party attempts

to set up the media.

• media-up: Sent by both parties. Communications have been successfully established.

• terminate: Sent by both parties. Ends attempt to set up call. Includes a “reason” ﬁeld with one of values detailed

below.

Termination Reasons

The following reasons appear in “action”/”terminate” and “progress” / “terminated” messages. The “” columns indi-

cate whether the indicated element is permitted to generate the reason. When generated a “terminated” message as the

result of receiving a “terminate” action from either client, the server will copy the reason code from the “terminate”

action message into all resulting “terminated” progress messages, ‘’even if it does not recognize the reason code’‘.

To provide for forwards compatibility, clients must be prepared to process “terminated” progress messages with un-

known reason codes. The reaction to this situation should be the display of a generic “call setup failed” message.

If the server receives an action of “terminate” with a reason it does not recognize, it copies that reason into the resulting

“terminated” message.

Reason CallerCalleeServerNote

reject The called user has declined the call.

busy The user is logged in, but cannot answer the call due to some current state (e.g.,

DND, in another call).

timeout The call setup has timed out (The called party’s client has exceeded the amount

of time it is willing to alert the user, or one of the server’s timers expired)

cancel The calling party has cancelled a pending call.

media-fail The called user has declined the call.

closed The other user’s WSS connection closed unexpectedly.

answered-

elsewhere

This call has been answered on another device.

2.8. Loop Server 105

Mozilla Services Documentation, Release

Timer Supervision

Server Timers

The server uses three timers to ensure that the call created by a setup attempt is cleaned up in a timely fashion.

Supervisory Timer

After responding to a `POST /call/{token}` or `POST /call/user` message, the server starts a supervi-

sory timer of 10 seconds.

• If the calling user does not connect and send a “hello” in this time period, the server considers the call to be

failed. The called user, if connected, will receive a “progress”/”terminated” message with a reason of “timeout”.

• If the called user does not connect and send a “hello” in this time period, the server considers the call to be failed.

The calling user, if connected, will receive a “progress”/”terminated” message with a reason of “timeout”.

Ringing Timer

Upon receiving a “hello” from the called user, the server starts a ringing timer of 30 seconds. If the called user does

not send an “accept” message in this time period, then both parties will receive a “progress”/”terminated” message

with a reason of “timeout”.

Connection Timer

Upon receiving an “accept” from the called user, the server starts a connection timer of 10 seconds. If the call setup

state does not reach “connected” in this time period, then both parties will receive a “progress”/”terminated” message

with a reason of “timeout”.

Client Timers

Response Timer

Every client message triggers a response from the server: “hello” results in “hello” or “error”; and “action” will always

cause a corresponding “progress” message to be sent. When the client sends a message, it sets a timer for 5 seconds.

If the server does not respond in that time period, it disconnects from the server and considers the call failed.

Media Setup Timer

After sending a “media-up” action, the client sets a timer for 10 seconds. If the server does not indicate that the call

setup has entered the “connected” state before the timer expires, the client disconnects from the server and considers

the call failed.

Alerting Timer

We may wish to let users conﬁgure the maximum amount of time the call is allowed to ring (up to 30 seconds) before

it considers it unanswered. This timer would start as soon as user alerting begins. If it expires before the call is set up,

then the called party sends a “action”/”disconnect” message with a reason of “timeout.”

106 Chapter 2. Services

Mozilla Services Documentation, Release

Administration Tools

Around loop-server code we have created some little tools that helps administrate the server in production and trou-

bleshooting.

Redis Tools

This are tools built to administrate the Redis storage.

HawkSession expiration TTL

If you want know in how many seconds will an Hawk Session expire, you can use the test_ttl.js tool:

NODE_ENV=production node test_ttl.js <hawkId>

This will use the server environment conﬁguration to read the expiration in the REDIS database.

The output looks like:

NODE_ENV=loadtest node ttl_hawk.js

˓→634ecc3dc394170edbd8b2b6d3c4c4526a354b5eda8f9ed3abaeb3a89a0f83a8

redis-cli TTL hawk.4d8535d8dfbde737e611828a690d0881d8cfd2e3eddd0dd6cb6150990bd39b5b

expire in 2591943 seconds

Number of keys of each types

If you want to have general information about the memory usage and know how many of each keys are used, you can

use:

$ ./redis_usage.sh localhost

[...]

# Memory

used_memory:592032

used_memory_human:578.16K

used_memory_rss:6832128

used_memory_peak:3169928

used_memory_peak_human:3.02M

used_memory_lua:33792

mem_fragmentation_ratio:11.54

mem_allocator:jemalloc-3.4.1

[...]

Keys for spurl.

=====================

spurl.89272fca80167430382fae1a92e4e561e186db71ba0e37178a5f4cb8ce81fa6c.

˓→e7f1f6adf79ed64b6b48b64cf63d75b81ec66f9ac615884a5736b209266048c7

Total of keys: 5

[...]

2.8. Loop Server 107

Mozilla Services Documentation, Release

Number of callUrls per user

This script gives you the average number of callUrls per users:

$ pip install redis hiredis

$ python callurls_per_user.py

average is 6.5

Statsd

Loop-Server have got a number of Statsd counters and timers that can help monitor what’s going on in near real-time.

Name Type Description

loop.activated-users counter New Hawk Session created for a user.

loop.call-urls counter New call-url creation.

loop.simplepush.call counter Calls made to a SimplePush URL.

loop.simplepush.call.

(success|failures)

counter SimplePush calls success and failures.

loop.simplepush.call.{reason} counter Calls made to a SimplePush URL for a

given reason.

loop.simplepush.call.{reason}.

(success|failures)

counter Calls success and failures on SP urls for a

given reason.

loop.aws.write timer AWS S3 roomContext write calls.

loop.aws.read timer AWS S3 roomContext read calls.

loop.aws.remove timer AWS S3 roomContext deletion calls.

loop.filesystem.write timer Filesystem roomContext write calls.

loop.filesystem.read timer Filesystem roomContext read calls.

loop.filesystem.remove timer Filesystem roomContext deletion calls.

loop.tokbox.createSession timer TokBox createSession calls.

You can also ﬁnd the user ﬂow in the wiki, at https://wiki.mozilla.org/Loop/Architecture.

Resources

• Server: https://github.com/mozilla-services/loop-server

MSISDN Gateway

Goal of the Service

MSISDN Gateway server allows people to log-on BrowserID applications using a validated phone number.

MSISDN Gateway server propose multiple ways to validate the phone number with regards to the country, operator

and validation cost.

Assumptions

• The MSISDN supports multiple veriﬁcation ﬂows with regards to the MSISDN and MCC/MNC codes.

• The client can discover availables veriﬁcation methods using the /discover endpoint.

108 Chapter 2. Services

Mozilla Services Documentation, Release

• The client /register to get an Hawk session that will be valid until a /unregister call.

• The client can verify one and only one number per session.

• The client can use its Hawk session to generate as many BrowserID certiﬁcates (valid for maximum a day) as

needed until the /unregister call, there is not automatic expiration of the Hawk session.

• A mobile number can be validated by multiple session (one per device or per app. i.e: One for Loop and one for

the Marketplace.)

Documentation content

MSISDN Gateway API v1.0

This document is based on the current status of the server. All the examples had been done with real calls. It doesn’t

reﬂect any future implementation and tries to stick with the currently deployed version.

This document describes the HTTP API and the SMS API.

HTTP APIs

Note: Unless stated otherwise, all APIs are using application/json for the requests and responses content types.

Parameters for the GET requests are form encoded (?key=value&key2=value2)

To ease testing, you can use httpie in order to make requests. Examples of use with httpie are provided when possible.

In order to authenticate with hawk, you’ll need to install the requests-hawk module

Authentication

To deal with authentication, the MSISDN Gateway server uses Hawk sessions. When you register, you can do so

with different authentications schemes, but you are always given an hawk session back, that you should use when

requesting the endpoints which need authentication.

When authenticating using the /register endpoint, you will be given an hawk session token called msisdnSessionToken

in the body. You will need to derive as explained at Derive hawk credentials from the hawk session token.

APIs

GET /

Displays version information, for instance:

http GET localhost:5000 --verbose

GET / HTTP/1.1

Accept: application/json

Accept-Encoding: gzip, deflate

Content-Type: application/json; charset=utf-8

Host: localhost:5000

User-Agent: HTTPie/0.8.0

2.9. MSISDN Gateway 109

Mozilla Services Documentation, Release

HTTP/1.1 200 OK

Access-Control-Allow-Credentials: true

Connection: keep-alive

Content-Length: 219

Content-Type: application/json; charset=utf-8

Date: Mon, 28 Jul 2014 13:52:16 GMT

ETag: W/"db-3773650714"

Timestamp: 1406555536

{

"description": "The Mozilla MSISDN Gateway",

"endpoint": "http://localhost:5000",

"homepage": "https://github.com/mozilla-services/msisdn-gateway/",

"name": "mozilla-msisdn-gateway",

"version": "0.5.0"

}

POST /register

Creates a new msisndSessionToken associated with an Hawk session.

This is the ﬁrst step to start verifying a new number.

Example:

http POST localhost:5000/register --verbose

POST /register HTTP/1.1

Accept:

Accept-Encoding: gzip, deflate

Content-Length: 0

Host: localhost:5000

User-Agent: HTTPie/0.8.0

HTTP/1.1 200 OK

Access-Control-Allow-Credentials: true

Connection: keep-alive

Content-Length: 94

Content-Type: application/json; charset=utf-8

Date: Mon, 28 Jul 2014 13:55:20 GMT

Timestamp: 1406555720

{

"msisdnSessionToken":

˓→"8feb2f78227ff8f8d4addd8ba77c06d9ee7acb59d86bd78ae2fd94e242dfd1ee"

}

Server should acknowledge your request, return a msisdnSessionToken and answer with a status code of

200 OK.

Potential HTTP error responses include:

• 429 Too Many Requests: Client has sent too many requests (errno: 117)

• 503 Service Unavailable: Service temporarily unavailable due to high load or misconﬁguration of

the storage backend. (errno: 201)

110 Chapter 2. Services

Mozilla Services Documentation, Release

POST /unregister

Requires authentication

Unregister an Hawk session.

To revoke a device or once we don’t want to use this validation number anymore, we can unregister the

session token to prevent the user from continuing to generate BrowserID certiﬁcates with it.

Example:

http POST localhost:5000/unregister --verbose \

--auth-type=hawk \

--auth='8feb2f78227ff8f8d4addd8ba77c06d9ee7acb59d86bd78ae2fd94e242dfd1ee:

˓→'

POST /unregister HTTP/1.1

Accept:

Accept-Encoding: gzip, deflate

Authorization: Hawk mac="Tpny...a+A=", hash="B0we...z8=", id="bc...2f", ts=

˓→"1406556506", nonce="E_GRLT"

Content-Length: 0

Host: localhost:5000

User-Agent: HTTPie/0.8.0

HTTP/1.1 204 No Content

Access-Control-Allow-Credentials: true

Connection: keep-alive

Date: Mon, 28 Jul 2014 14:08:26 GMT

Server-Authorization: Hawk mac="lTGx...PNM=", hash="B0we...Uz8="

Server should acknowledge your request and answer with a status code of 204 No Content.

Potential HTTP error responses include:

• 401 Unauthorized: The credentials you passed aren’t valid. (errno: 109 or 110)

• 429 Too Many Requests: Client has sent too many requests (errno: 117)

• 503 Service Unavailable: Service temporarily unavailable due to high load or misconﬁguration of

the storage backend. (errno: 201)

POST /discover

Discover which validation methods are available for a given MSISDN number or a MCC/MNC network-

code.

Body parameters:

• mcc, the Mobile Country Code.

• mnc, the Mobile Network Code (optional).

• msisdn, the Mobile Station ISDN Number that is the user phone number to validate in its interna-

tional form i.e 33623456789 (optional).

Example (with MCC only):

2.9. MSISDN Gateway 111

Mozilla Services Documentation, Release

http POST localhost:5000/discover mcc=208 --verbose

POST /discover HTTP/1.1

Accept: application/json

Accept-Encoding: gzip, deflate

Content-Length: 14

Content-Type: application/json; charset=utf-8

Host: localhost:5000

User-Agent: HTTPie/0.8.0

{

"mcc": "214"

}

HTTP/1.1 200 OK

Access-Control-Allow-Credentials: true

Connection: keep-alive

Content-Length: 169

Content-Type: application/json; charset=utf-8

Date: Mon, 28 Jul 2014 14:18:05 GMT

Timestamp: 1406557085

{

"verificationDetails": {

"sms/momt": {

"moVerifier": "+34191600777",

"mtSender": "Mozilla@"

}

"verificationMethods": [

"sms/momt"

]

}

Example (with all parameters):

http POST localhost:5000/discover msisdn=+3412578946 mcc=208 mnc=07 --verbose

POST /discover HTTP/1.1

Accept: application/json

Accept-Encoding: gzip, deflate

Content-Length: 51

Content-Type: application/json; charset=utf-8

Host: localhost:5000

User-Agent: HTTPie/0.8.0

{

"mcc": "214",

"mnc": "07",

"msisdn": "3412578946"

}

HTTP/1.1 200 OK

Access-Control-Allow-Credentials: true

Connection: keep-alive

Content-Length: 286

Content-Type: application/json; charset=utf-8

112 Chapter 2. Services

Mozilla Services Documentation, Release

Date: Mon, 28 Jul 2014 14:20:07 GMT

Timestamp: 1406557207

{

"verificationDetails": {

"sms/momt": {

"moVerifier": "+34191600777",

"mtSender": "Mozilla@"

"sms/mt": {

"mtSender": "Mozilla@",

"url": "http://localhost:5000/sms/mt/verify"

}

"verificationMethods": [

"sms/mt",

"sms/momt"

]

}

Response from the server:

The server should answer this with a 200 OK status code and a JSON object with the following properties:

• veriﬁcationMethods, a list of veriﬁcation methods available for the given set of parameters, in order

of preferred use

• veriﬁcationDetails, an object whose keys are the elements of veriﬁcationMethods and whose values

are the details of each method

The methods listed in veriﬁcationMethods are sorted in the preferred order from the perspective of the

server, i.e., the method listed ﬁrst is the most preferred method.

Potential HTTP error responses include:

• 400 Bad Request: MCC, MNC or MSISDN missing (errno: 108) or invalids (errno: 107)

• 406 Bad JSON: The body is not valid JSON. (errno: 106)

• 411 Length Required: Content-Length header wasn’t provided. (errno: 112)

• 413 Request Too Large: Request Too Large. (errno: 113)

• 429 Too Many Requests: Client has sent too many requests (errno: 117)

• 503 Service Unavailable: Service temporarily unavailable due to high load or misconﬁguration of

the storage backend. (errno: 201)

POST /sms/mt/verify

Requires authentication

Starts the SMS MT ﬂow by sending a SMS to the MSISDN to register

Body parameters:

• msisdn, the Mobile Station ISDN Number that is the user phone number to validate in its interna-

tional form i.e +33623456789.

• mcc, the Mobile Country Code.

• mnc, the Mobile Network Code (optional).

2.9. MSISDN Gateway 113

Mozilla Services Documentation, Release

• shortVeriﬁcationCode, a parameter to ask a human transcribable 6 digits code if set to true. In that

case the server will also take care of the Accept-Language header to localize any text in the SMS

(optional)

Response from the server:

The server should answer this with a 204 No Content status code.

Example:

http POST localhost:5000/sms/mt/verify msisdn=+33123456789 mcc=208 \

--verbose \

--auth-type=hawk \

--auth='8feb2f78227ff8f8d4addd8ba77c06d9ee7acb59d86bd78ae2fd94e242dfd1ee:

˓→'

POST /sms/mt/verify HTTP/1.1

Accept: application/json

Accept-Encoding: gzip, deflate

Authorization: Hawk mac="THYl...=", hash="mw68...=", id="9ee4...1a81", ts=

˓→"1406557901", nonce="xoIdtg"

Content-Length: 53

Content-Type: application/json; charset=utf-8

Host: localhost:5000

User-Agent: HTTPie/0.8.0

{

"mcc": "208",

"msisdn": "+33123456789"

}

HTTP/1.1 204 No Content

Access-Control-Allow-Credentials: true

Connection: keep-alive

Date: Mon, 28 Jul 2014 14:31:41 GMT

Server-Authorization: Hawk mac="wx9m...=", hash="B0we...="

Potential HTTP error responses include:

• 400 Bad Request: MCC, MNC or MSISDN missing (errno: 108) or invalids (errno: 107)

• 406 Bad JSON: The body is not valid JSON. (errno: 106)

• 411 Length Required: Content-Length header wasn’t provided. (errno: 112)

• 413 Request Too Large: Request too large. (errno: 113)

• 429 Too Many Requests: Client has sent too many requests (errno: 117)

• 503 Service Unavailable: Service temporarily unavailable due to high load or misconﬁguration of

the storage backend. (errno: 201)

POST /sms/verify_code

Requires authentication

Validates the code received by SMS

Body parameters:

• code, the code received by SMS.

114 Chapter 2. Services

Mozilla Services Documentation, Release

Response from the server:

The server should answer this with a 200 OK status code and a JSON object with the following properties:

• msisdn The Mobile phone number that has been validated during the session.

Example:

http POST localhost:5000/sms/verify_code

˓→code=15d3b227b0e58f216ee49b8da41c05c8 \

--verbose \

--auth-type=hawk \

--auth='c0d8cd2ec579a3599bef60f060412f01f5dc46f90465f42b5c47467481315f51:

˓→'

POST /sms/verify_code HTTP/1.1

Accept: application/json

Accept-Encoding: gzip, deflate

Authorization: Hawk mac="8/Rg...=", hash="oAXy...=", id="9ee4...1a81", ts=

˓→"1406558280", nonce="WBjO3I"

Content-Length: 44

Content-Type: application/json; charset=utf-8

Host: localhost:5000

User-Agent: HTTPie/0.8.0

{

"code": "15d3b227b0e58f216ee49b8da41c05c8"

}

HTTP/1.1 200 OK

Access-Control-Allow-Credentials: true

Connection: keep-alive

Content-Length: 30

Content-Type: application/json; charset=utf-8

Date: Mon, 28 Jul 2014 14:38:00 GMT

Server-Authorization: Hawk mac="nFOD...=", hash="NVuB...="

Timestamp: 1406558280

{

"msisdn": "+33123456789"

}

Potential HTTP error responses include:

• 400 Bad Request: code missing (errno: 108) or invalid (errno: 105)

• 406 Bad JSON: the body is not valid JSON. (errno: 106)

• 410 Expired: the session MSISDN has expired. (errno: 111)

• 411 Length Required: Content-Length header wasn’t provided. (errno: 112)

• 413 Request Too Large: Request too large. (errno: 113)

• 429 Too Many Requests: Client has sent too many requests (errno: 117)

• 503 Service Unavailable: Service temporarily unavailable due to high load or misconﬁguration of

the storage backend. (errno: 201)

2.9. MSISDN Gateway 115

Mozilla Services Documentation, Release

POST /certiﬁcate/sign

Requires authentication

Generate a BrowserID certiﬁcate with the given public key for the validated number.

Example:

http POST localhost:5000/certificate/sign \

duration=3600 \

publicKey='{"algorithm":"DS","y":"e6...40","p":"d6...01","q":"b1...3b","g

˓→":"9a...ef"}' \

--verbose \

--auth-type=hawk \

--auth='8feb2f78227ff8f8d4addd8ba77c06d9ee7acb59d86bd78ae2fd94e242dfd1ee:

˓→'

POST /certificate/sign HTTP/1.1

Accept: application/json

Accept-Encoding: gzip, deflate

Authorization: Hawk mac="6vKD...=", hash="PKZT...=", id="9e...81", ts=

˓→"1406558679", nonce="IFjRIR"

Content-Length: 1702

Content-Type: application/json; charset=utf-8

Host: localhost:5000

User-Agent: HTTPie/0.8.0

{

"duration": "3600",

"publicKey": "{\"algorithm\":\"DS\",\"y\":\"e6...40\",\"p\":\"d6...01\",\

˓→"q\":\"b1...3b\",\"g\":\"9a...ef\"}"

}

HTTP/1.1 200 OK

Access-Control-Allow-Credentials: true

Connection: keep-alive

Content-Length: 2602

Content-Type: application/json; charset=utf-8

Date: Mon, 28 Jul 2014 14:44:39 GMT

Server-Authorization: Hawk mac="QMCs...=", hash="NVuB...="

Timestamp: 1406558679

{

"cert": "eyJh...1Rgg"

}

Potential HTTP error responses include:

• 400 Bad Request: duration or publicKey parameter missing (errno: 108) or invalid (errno: 107)

• 406 Bad JSON: The body is not valid JSON. (errno: 106)

• 411 Length Required: Content-Length header wasn’t provided. (errno: 112)

• 413 Request Too Large: Request Too Large. (errno: 113)

• 429 Too Many Requests: Client has sent too many requests (errno: 117)

• 503 Service Unavailable: Service temporarily unavailable due to high load or misconﬁguration of

the storage backend. (errno: 201)

116 Chapter 2. Services

Mozilla Services Documentation, Release

GET /.well-known/browserid

Returns information for the BrowserID veriﬁer.

Response from the server:

The server should answer this with a 200 status code and a JSON object with the following properties:

• public-key the server public-key used to validate the BrowserId certiﬁcate

• authentication, the link to the authentication page

• provisionning, the link to the provisionning page

Example:

http GET localhost:5000/.well-known/browserid --verbose

GET /.well-known/browserid HTTP/1.1

Accept:

Accept-Encoding: gzip, deflate

Host: localhost:5000

User-Agent: HTTPie/0.8.0

HTTP/1.1 200 OK

Access-Control-Allow-Credentials: true

Connection: keep-alive

Content-Length: 1815

Content-Type: application/json; charset=utf-8

Date: Mon, 28 Jul 2014 14:54:50 GMT

ETag: W/"717-781509924"

Timestamp: 1406559290

{

"authentication": "/.well-known/browserid/warning.html",

"provisioning": "/.well-known/browserid/warning.html",

"public-key": {

"algorithm": "DS",

"g": "9a...ef",

"p": "d6...01",

"q": "b1...3b",

"y": "7e...b9"

}

GET /api-specs

An endpoint that gives back the videur conﬁguration.

Server should answer with a status of 200 and the API routes as a JSON object.

Example:

http GET localhost:5000/api-specs --verbose

GET /api-specs HTTP/1.1

Accept:

2.9. MSISDN Gateway 117

Mozilla Services Documentation, Release

Accept-Encoding: gzip, deflate

Host: localhost:5000

User-Agent: HTTPie/0.8.0

HTTP/1.1 200 OK

Access-Control-Allow-Credentials: true

Connection: keep-alive

Content-Length: 1069

Content-Type: application/json; charset=utf-8

Date: Mon, 28 Jul 2014 14:57:54 GMT

ETag: W/"42d-3425668954"

Timestamp: 1406559474

{

"service": {

"location": "http://localhost:5000",

"resources": {

"/": {

"GET": {}

"/.well-known/browserid": {

"GET": {}

"/.well-known/browserid/warning.html": {

"GET": {}

"/__heartbeat__": {

"GET": {}

"/certificate/sign": {

"POST": {

"max_body_size": "10k"

}

"/discover": {

"POST": {

"max_body_size": "10k"

}

"/register": {

"POST": {

"max_body_size": "10k"

}

"/sms/momt/": {

"GET": {},

"POST": {

"max_body_size": "10k"

}

"/sms/mt/verify": {

"POST": {

"max_body_size": "10k"

}

"/sms/verify_code": {

"POST": {

118 Chapter 2. Services

Mozilla Services Documentation, Release

"max_body_size": "10k"

}

"/unregister": {

"POST": {

"max_body_size": "10k"

}

"version": "0.5.0",

"videur_version": "0.1"

}

SMS /sms/momt/verify

The SMS message to a moVeriﬁer number will start the MOMT ﬂow.

SMS should be of the form of:

/sms/momt/verify <hawkId>

i.e:

/sms/momt/verify

˓→9ee442e5c8575c4077db786a40f603a70ae8eee09f5b41e34c096410f6fc1a81

• path, future proof path, in order to be able to route SMS to the right endpoint.

• hawkId, the hawk session id extracted from the msisdnSessionToken using HKDF.

When the SMS Gateway receive the Inbound Message, it will make a call on the conﬁgured endpoint.

For Nexmo — GET or POST /sms/momt/?provider=nexmo

Querystring parameters:

• msisdn, the phone number from which the message is coming

• text, the content of the message

• network-code, the MCC/MNC unique identiﬁer

For BeepSend — GET or POST /sms/momt/?provider=beepsend

Querystring or Body parameters:

• from, the phone number from which the message is coming

• message, the content of the message

• mcc, The Mobile Country Code (in GET)

• mnc, The Mobile Network Code (in GET)

• mccmnc, {“mcc”: “<Mobile Country Code>”, “mnc”: “<Mobile Network Code>”} (in POST)

Error Responses

All errors are also returned, wherever possible, as json responses with a code, errno and error message.

2.9. MSISDN Gateway 119

Mozilla Services Documentation, Release

Error status codes and codes and their corresponding outputs are:

• 404 : unknown URL or unsupported application.

• 400 : malformed request. Possible causes include a missing option, bad values or malformed json.

• 401 : you need to be authenticated

• 403 [you are authenticated but don’t have access to the resource you are] requesting.

• 405 : unsupported method

• 406 : unacceptable - the client asked for an Accept we don’t support

• 503 : service unavailable (provider or database backends may be down)

Also the associated errno can be one of:

• 105 INVALID_CODE: This come with a 404 on a wrong validation code;

• 106 BADJSON: This come with a 406 if the sent JSON is not parsable;

• 107 INVALID_PARAMETERS: This come with a 400 and describe invalid parameters with a reason;

• 108 MISSING_PARAMETERS: This come with a 400 and list all missing parameters;

• 109 INVALID_REQUEST_SIG: This come with a 401 and deﬁne a problem with the Hawk hash;

• 110 INVALID_AUTH_TOKEN: This come with a 401 and deﬁne a problem during Auth;

• 111 EXPIRED: This come with a 410 and deﬁne a EXPIRE ressource;

• 112 LENGTH_MISSING: This come with a 411 and deﬁned a missing Content-Length header.

• 113 REQUEST_TOO_LARGE: This come with a 400 and deﬁne a too large request;

• 201 BACKEND: This come with a 503 when a third party is not available at the moment.

Resources

• Server: https://github.com/mozilla-services/msisdn-gateway

• Test webapp: https://github.com/ferjm/msisdn-veriﬁer-client

120 Chapter 2. Services

CHAPTER 3

Server-side development guide

Python Server Development Guide

This documentation is aimed at developers that want to work with an existing Mozilla Services server project, or want

to create a new one that follow the same conventions.

Warning: This document is currently under heavy development.

Contents of the documentation:

Overview of Services Python applications

Services Python applications are all WSGI applications based on the same stack of tools:

NGinx <= TCP => Gunicorn <= WSGI => SyncServerApp <=> WebOb

• NGinx : A high-speed HTTP Server/reverse proxy.

• GUnicorn: A Python WSGI Server.

• SyncServerApp: A base WSGI application for all Services apps. Located in the server-core repository.

• WebOb: a Request and a Response object with a simple interface.

Services provides two base libraries to build WSGI applications:

• cef: implements a CEF logger for ArcSight.

• server-core: provides helpers to build Services applications.

121

Mozilla Services Documentation, Release

cef

Most Services applications need to generate CEF logs. A CEF Log is a formatted log that can be used by ArcSight, a

central application used by the Infrasec team to manage application security.

The cef module provides a log_cef() function that can be used to emit CEF logs:

log_cef(message, severity, environ, conﬁg, [username, [signature]], **kw)

Creates a CEF record, and emits it in syslog or another ﬁle.

Args:

• message: message to log

• severity: integer from 0 to 10

• environ: the WSGI environ object

• conﬁg: conﬁguration dict

• username: user name, defaults to ‘none’

• signature: CEF signature code, defaults to ‘AuthFail’

• extra keywords: extra keys used in the CEF extension

Example:

>>> from cef import log_cef

>>> log_cef('SecurityAlert!', 5, environ, config,

... msg='Someone has stolen my chocolate')

With environ and conﬁg provided by the web environment.

Note that the CEF library is published at PyPI: http://pypi.python.org/pypi/cef

See CEF for more info on this.

server-core

The server-core library provides helpers to build Services applications.

In server-core‘s philosophy, a WSGI application is a SyncServerApp instance which will contain a conﬁg attribute

that’s a mapper containing all the conﬁguration needed by the code. This conﬁguration is loaded from a unique ini-like

ﬁle.

When a request comes in, Routes is used to dispatch it to a controller method.

Controllers are simple classes whose methods receive the request and need to return a response.

Conﬁguration ﬁles

The conﬁguration ﬁles we use in Services applications are based on an extended version of Ini ﬁles. You can ﬁnd a

description of the ﬁle at https://wiki.mozilla.org/Services/Sync/Server/GlobalConfFile.

server-core provides a simple reader:

>>> from services.config import Config

>>> cfg = Config('/etc/keyexchange/keyexchange.conf')

>>> cfg.sections()

['keyexchange', 'filtering', 'cef']

122 Chapter 3. Server-side development guide

Mozilla Services Documentation, Release

>>> cfg.items('keyexchange')

[('max_gets', 10), ('root_redirect', 'https://services.mozilla.com'),

('use_memory', False), ('ttl', 300),

('cache_servers', ['127.0.0.1:11211']), ('cid_len', 4)]

>>> cfg.get('keyexchange', 'cid_len')

Note that SyncServerApp will automatically create a Conﬁg instance over a central conﬁguration ﬁle when it’s

used to create a web application.

See Conﬁguration ﬁle for more info on this.

SyncServerApp

The SyncServerApp is a base class that can be used to get a few automation and some useful helpers when you

want to create an application for Services.

It provides:

• a central conﬁguration ﬁle

• a pluggable authentication backend with an LDAP and an SQL plugin provided.

• an overridable authentication process, defaulting to Basic Authentication.

• a basic URL dispatcher based on Routes.

• an error handler that ensures backend errors are logged and a 500 error is raised.

• a heartbeat page useful for monitoring the server

• a debug page to display useful information on the server

• a few middlewares integrated: a proﬁler, an error catcher and a console logger.

To create an application using SyncServerApp, see Complete layout.

Setting up a development environment

This section makes the assumption that you are under Mac OS X or Linux. A section for Windows might

be added later.

Prerequisites

Setting up a development environment for Services consists of installing those packages:

1. make (installed by default on most Linuxes)

2. the latest gcc

3. Python 2.6 (installed by default under Debuntu, python26 under CentOS)

4. Python 2.6 headers (python2.6-dev under Debuntu, python26-devel under CentOS)

5. python26-proﬁler under Ubuntu

6. Mercurial (mercurial in most distros).

3.1. Python Server Development Guide 123

Mozilla Services Documentation, Release

7. virtualenv

8. Distribute

9. Flake8

10. Paste

11. PasteDeploy

12. MoPyTools

One simple way to install all tools from 7. to 12. in your environment is to run the Distribute bootstrap script, then

install MoPyTools:

$ curl -O http://python-distribute.org/distribute_setup.py

$ python distribute_setup.py

$ easy_install MoPyTools

This will pull all other tools for you and install them.

Although, each project provides a Makeﬁle that bootstraps this step and installs tools 8. to 12. automatically. So if you

prefer, it should sufﬁce to have only virtualenv and distribute installed system-wide:

$ curl -O http://python-distribute.org/distribute_setup.py

$ python distribute_setup.py

$ easy_install virtualenv

Setting up the Project Environment

Once you have all the above tools installed, working on a project consists of creating an isolated Python environment

using Virtualenv and develop in it.

Each project provides a Makeﬁle that bootstraps this step, so you should not have to do it manually.

For example, to create an environment for the Sync project, you can run:

$ hg clone http://hg.mozilla.org/services/server-full

$ cd server-full

$ make build

The code is currently developed and tested using python2.6, and it most likely will not work with other versions of

python.

The project Makeﬁle uses the default python interpreter found on your $PATH. If your system default python version

is not 2.6, you will need to create a python2.6 virtualenv to run make build. For example:

$ virtualenv --python=python2.6 ~/venvs/py26

$ source ~/venvs/py26/bin/activate

$ make build

Once the environment has been created, you can do a sanity check by running all tests:

$ make test

Conﬁguring Flake8

Flake8 can be used from the command-line, by simply running it over one or several ﬁles:

124 Chapter 3. Server-side development guide

Mozilla Services Documentation, Release

$ flake8 syncreg/controllers/

syncreg/controllers/user.py:44: 'urlunparse' imported but unused

syncreg/controllers/user.py:44: 'urlparse' imported but unused

syncreg/controllers/user.py:49: 'Response' imported but unused

syncreg/controllers/user.py:55: 'get_url' imported but unused

syncreg/controllers/user.py:276: undefined name 'environ'

syncreg/controllers/user.py:276: undefined name 'config'

syncreg/controllers/user.py:180:8: E111 indentation is not a multiple of four

syncreg/controllers/user.py:240:1: 'UserController.change_password' is too complex

˓→(10)

syncreg/controllers/user.py:318:1: 'UserController.do_password_reset' is too complex

˓→(11)

A simpler way to use it without having to think about it, is to conﬁgure Mercurial to call it every time you commit a

change.

To use the Mercurial hook on any commit or qrefresh, change your .hgrc ﬁle like this:

[hooks]

commit = python:flake8.run.hg_hook

qrefresh = python:flake8.run.hg_hook

[flake8]

strict = 0

If the strict option is set to 1, any warning will block the commit. When strict is set to 0, warnings are just displayed

in the standard output.

Using a non-strict mode is good enough: it will show you the issues without blocking your commits, so you can decide

what should be done.

In some case, you might need to simply silent the warnings. You can do this with NOQA markers:

• all modules that starts with a # ﬂake8: noqa comment line are skipped.

• all lines that ends with a “# NOQA” comment are skipped as well.

Code layout

There are two code layouts:

• the minimal layout: simple Python application.

• the complete layout: Web application.

Minimal layout

The minimal layout is an empty application that includes all the boiler-plate code all our applications should have.

A minimal Services’ project usually contains:

• Makeﬁle and build.py: used to build the environment, run tests, etc.

• RPM spec: used to build the project’s RPM.

• a directory for the Python code.

• README: brief intro about the project

• setup.py: deﬁnes Distutils’ options.

3.1. Python Server Development Guide 125

Mozilla Services Documentation, Release

• MANIFEST.in: deﬁnes extra options for Distutils.

Paster Template

You can create a new application layout by using the services_base Paster template provided by MoPyTools, which

will ask you a few questions:

$ paster create -t services_base MyApp

Selected and implied templates:

MoPyTools#services_base A Mozilla Services application

Variables:

egg: MyApp

package: myapp

project: MyApp

Enter version (Version (like 0.1)) ['']: 0.1

Enter description (One-line description of the project) ['']: A cool app that does it

Enter author (Author name) ['']: Tarek

Enter author_email (Author email) ['']: [email protected]

Enter url (URL of homepage (or Repository root)) ['']: http://hg.mozilla.org/services/

˓→myapp

Creating template services_base

...

Generating Application...

Once the application is generated, a default layout is created:

$ find MyApp/

MyApp/

MyApp/Makefile

MyApp/setup.py

MyApp/README.txt

MyApp/pylintrc

MyApp/myapp

MyApp/myapp/tests

MyApp/myapp/tests/test_sample.py

MyApp/myapp/tests/__init__.py

MyApp/myapp/__init__.py

MyApp/MyApp.spec

MyApp/build.py

You can build and test that the project is ready, by going in the directory and running:

$ make build test

Makeﬁle

Every project should have a Makeﬁle with these targets:

• build: used to build the project in-place and all its dependencies.

• test: used to run all tests and produce tests coverage and lint reports.

• build_rpms: used to build all RPMs for the project.

126 Chapter 3. Server-side development guide

Mozilla Services Documentation, Release

The build target can take optional arguments to deﬁne for the project or any of its dependency that leaves in our

repositories, a Mercurial tag.

For example, to build the KeyExchange project with the “rpm-0.1-10” tag, and its ServerCore dependency with the

tag “rpm-0.1-15”, one may call:

$ make build SERVER_KEYEXCHANGE=rpm-0.1.10 SERVER_CORE=rpm-0.1-15

The option name is the repository named in upper case, with the dashes (“-”) replaced by underlines (“_”). So “server-

core” becomes “SERVER_CORE”.

For more info on building and releasing, see Releasing an application.

The test target runs the Nose test runner, and can be used to work on the code. It’s also used by Jenkins to continuously

test your project.

See 5. Build the app and all RPMS for more info on the building process.

RPM Spec ﬁle

The spec ﬁle that gets generated is used by “make build_rpm” to generate a RPM for your application. It contains all

the required dependencies for a stock Services application, but will require that you add any new dependency your

code needs.

setup.py and MANIFEST.in

XXX

Complete layout

The complete layout contains all the things the minimal layout has, plus everything needed to make it a Web applica-

tion:

• etc/ : all default conﬁg ﬁles.

• package/wsgiapp.py: the web application itself.

• package/controller.py: the web controller to start adding features.

• package/run.py: the bootstrap ﬁle used by Gunicorn to run the app.

• package/tests/functional/: a minimal functional test using WebTest.

XXX

All Projects repositories are located in http://hg.mozilla.org/services

Development cycle

The usual cycle to add a feature or ﬁx a bug is:

1. Make sure there’s a bug in Bugzilla.

2. Create a functional or unit-test, or both. Then change the code until the tests pass.

3. Ask for a review in the bug then push your changes.

4. Get ready to revert or ﬁx.

3.1. Python Server Development Guide 127

Mozilla Services Documentation, Release

Note that ramping up a new project is a bit speciﬁc since you can’t really follow a per-feature cycle until it has reached

a certain state.

1. A reference in Bugzilla

Every planned change to the code base of a project should start by adding a bug in Bugzilla. This is the central place

where all discussions related to the changes, code reviews will happen.

Here’s an example: https://bugzilla.mozilla.org/show_bug.cgi?id=631233

2. Writing the test and the change

Ideally, you should start to write a test that demonstrates the bug or the new feature. See Writing tests for more info

on how to write tests.

For bugs, it’s fairly easy: you need to write a test that reproduces the exact same problem, then ﬁx the code until the

code passes. For new features, a test that demonstrates how it works needs to be written.

Running tests is done with the test target:

$ make test

It’s important to run all tests to make sure your changes are not breaking the code base elsewhere. You won’t be able

to try out all possible execution environments of course, and that’s the job of the Jenkins CI server.

make test needs to include the tests from all the project dependencies. That’s the case when you start a fresh project

with a template.

External dependencies

In Jenkins the tests are running in an environment built from scratch so the tests should not:

1. depend on any ﬁle that is not in the checkout

2. use the LDAP backend - or if it does, mock anything that is trying to call ldap

The test can use MySQL but the database should be conﬁgured to use sqlite rather than MySQL, since there is no

MySQL server.

MemCached can be used and if you have a speciﬁc volatile backend, it can potentially be installed (Redis, etc.).

In any case, make sure you still have a fallback for those so anyone that checkouts a project can run the tests without

having to install a third-party server.

Note that Mozilla production backends are speciﬁcally tested via the functional tests that call the dev and stage clusters.

3. Ask for a review

XXX

4. Revert your change

The next steps are taken care of by Jenkins, who launch a test cycle to make sure that your change has not broken

anything under every environment your code is used in. If it happens, an email is sent at services-builds.

128 Chapter 3. Server-side development guide

Mozilla Services Documentation, Release

In that case you need to ﬁx the problem immediately, and if you can’t do it immediately, you need to revert all your

changes.

Writing tests

A Services application must be tested in various ways:

• unit and functional tests that are run locally

• functional tests that are run on the dev, stage and production cluster

Writing unit tests

XXX explain unittest

Writing functional tests

XXX will demo WebTest

Running functional tests on a real server

XXX

Conﬁguring Jenkins

XXX

Conﬁguring the application

The application is conﬁgured via two ﬁles:

• the Paster ini-like ﬁle, located in etc/

• the Services conﬁguration ﬁle.

XXX more on ﬁle location

Paster ini ﬁle

All Services projects provide a built-in web server that may be used to run a local instance.

For example in server-full, once the project is built, you can run it:

$ bin/paster serve development.ini

This will run a server on port 5000.

Paster reads an ini-like ﬁle that deﬁnes among other things:

• where the wsgi application is located

• where the application conﬁguration is located

• the logging conﬁguration

3.1. Python Server Development Guide 129

Mozilla Services Documentation, Release

• etc.

The main sections we want to conﬁgure in this ﬁle are:

• DEFAULT

• server:main

• app:main

• logging

DEFAULT

The default section deﬁnes four optional values (all are set to False by default):

1. debug: if set to True, will activate the debug mode.

2. client_debug: if set to True, will return in the body of the response any traceback when a 500 occurs.

3. translogger: will display in the stdout all requests made on the server.

4. proﬁle: will activate a proﬁler and generate cachegrind infos.

Example:

[DEFAULT]

debug = True

translogger = False

profile = False

server:main

Deﬁnes the web server to use to run the app with Paster. See Paster documentation for more info.

Example:

[server:main]

use = egg:Paste#http

host = 0.0.0.0

port = 5000

use_threadpool = True

threadpool_workers = 60

app:main

Deﬁnes the server entry point. See Paster documentation for more info.

conﬁguration can point to a conﬁguration ﬁle for the server. It uses a ﬁle: preﬁx.

Example:

[app:main]

use = egg:SyncServer

configuration = file:%(here)s/etc/sync.conf

130 Chapter 3. Server-side development guide

Mozilla Services Documentation, Release

logging

Logging is done using the logging conﬁguration. See Python’s logging documentation for more details.

The Sync server uses the syncserver logger everywhere.

In the following example, all Sync errors are logged in a speciﬁc ﬁle as long as DEFAULT:debug is activated. Other

logs are in a separate ﬁle:

[loggers]

keys = root,syncserver

[handlers]

keys = global,errors

[formatters]

keys = generic

[logger_root]

level = WARNING

handlers = global

[logger_syncserver]

qualname = syncserver

level = INFO

handlers = global, errors

propagate = 0

[handler_global]

class = handlers.RotatingFileHandler

args = ('/var/log/sync.log',)

level = DEBUG

formatter = generic

[handler_errors]

class = handlers.RotatingFileHandler

args = ('/var/log/sync-error.log',)

level = ERROR

formatter = generic

[formatter_generic]

format = %(asctime)s,%(msecs)03d %(levelname)-5.5s [%(name)s] %(message)s

datefmt = %Y-%m-%d %H:%M:%S

Conﬁguring the proﬁler

Activates the repoze.proﬁle middleware.

XXX

Conﬁguration ﬁle

The server uses a global conﬁguration ﬁle. The ﬁle location is conﬁgured in the Paster ini ﬁle (as the conﬁguration

setting in the [app:main] section), and it is loaded when the application starts. See app:main.

The conﬁguration ﬁle has one section for each module loaded by the application. The conﬁguration data will be

available on the application object as a dictionary-like object stored in the conﬁg attribute. The settings from the

3.1. Python Server Development Guide 131

Mozilla Services Documentation, Release

[global] section will be stored as the simple key name, while the settings from the other sections will be keyed as

<section_name>.<key>.

So:

[global]

foo = bar

baz = bawlp

[storage]

bing = boom

[auth]

snaf = foo

Would produce the following app.conﬁg:

{'foo': 'bar',

'baz': 'bawlp',

'storage.bing': 'boom',

'auth.snaf': 'foo'}

Additionally, conﬁg.get_section will return a dictionary containing only the settings from the speciﬁed section, without

the preﬁx. So continuing the example above...

app.conﬁg.get_section(‘global’) would return:

{'foo': 'bar',

'baz': 'bawlp'}

app.conﬁg.get_section(‘storage’) would return:

{'bing': 'boom}

Multi-Conﬁg Sections

In addition to supporting standard “INI” ﬁle conventions, Server-Core based applications provide an ability to use

namespaced section headers to allow for multiple, similar variations of a conﬁg section to be registered simultaneously.

The storage service used by the sync server is an example of where this is useful. Each storage server instance is

associated with a speciﬁc set of back end storage nodes, usually all of the nodes that are located in the same location

as the storage server itself. The storage server loads a separate conﬁguration for each storage node, all similar save for

the database connection URL.

Example:

[storage]

backend = memcached

cache_servers = 127.0.0.1:11211

192.168.1.13:11211

sqluri = mysql://localhost/sync

standard_collections = false

use_quota = true

quota_size = 5120

pool_size = 100

pool_recycle = 3600

[host:node0]

132 Chapter 3. Server-side development guide

Mozilla Services Documentation, Release

storage.sqluri = mysql://sync:sync@db1.example.com/sync

[host:node1]

storage.sqluri = mysql://sync:sync@db1.example.com/sync

[host:node2]

storage.sqluri = mysql://sync:sync@db2.example.com/sync

[host:node3]

storage.sqluri = mysql://sync:sync@db2.example.com/sync

The generated conﬁg object would then have sections named storage, host:node0, host:node1, etc., as you might

expect. In addition to being available as separate sections, however, these conﬁgurations can be merged with the

sections they refer to to generate “override” conﬁgs. These are made available via the merge(*sections) method on the

conﬁg object.

So, app.conﬁg would produce:

{'storage.backend': 'memcached',

'storage.cache_servers': ['127.0.0.1:11211', '192.168.1.13:11211'],

'storage.sqluri': 'mysql://localhost/sync',

'storage.standard_collections': False,

'storage.use_quota': True,

'storage.quota_size': 5120,

'storage.pool_size': 100,

'storage.pool_recycle': 3600,

'host:node0.storage.sqluri': 'mysql://sync:[email protected]/sync',

'host:node1.storage.sqluri': 'mysql://sync:[email protected]/sync',

'host:node2.storage.sqluri': 'mysql://sync:[email protected]/sync',

'host:node3.storage.sqluri': 'mysql://sync:[email protected]/sync',

}

while app.conﬁg.merge(‘host:node0’) would give:

{'storage.backend': 'memcached',

'storage.cache_servers': ['127.0.0.1:11211', '192.168.1.13:11211'],

'storage.standard_collections': False,

'storage.use_quota': True,

'storage.quota_size': 5120,

'storage.pool_size': 100,

'storage.pool_recycle': 3600,

'storage.sqluri': 'mysql://sync:[email protected]/sync',

'host:node0.storage.sqluri': 'mysql://sync:[email protected]/sync',

'host:node1.storage.sqluri': 'mysql://sync:[email protected]/sync',

'host:node2.storage.sqluri': 'mysql://sync:[email protected]/sync',

'host:node3.storage.sqluri': 'mysql://sync:[email protected]/sync',

}

Global

Global settings for the applications, under the global section.

Available options (o: optional, m: multi-line, d: default):

• retry_after [o, default:1800]: Value in seconds of the Retry-After header sent back when a 503 occurs in the

application.

3.1. Python Server Development Guide 133

Mozilla Services Documentation, Release

• heartbeat_page [o, default:__heartbeat__]: deﬁnes the path of the heartbeat page. The heartbeat page is used

by an HTTP Monitor to check that the server is still running properly. It returns a 200 if everything works, and a

503 if there’s an issue. A typical issue is the inability for the application to reach a backend server, like MySQL

or OpenLDAP.

• debug_page [o, default:None]: deﬁnes the path of the debug page. The debug page displays environ informa-

tion.

Warning: This page may expose private data. Once activated, it is not password-protected. If you use it, make

sure the web server (Apache, Nginx) protects it from anonymous access.

This feature is disabled by default to avoid any security issue.

• shared_secret [o, default: None]: deﬁnes a secret string that can be used by the client when creating users, to

bypass the captcha challenge.

• graceful_shutdown_interval [o, default: 1]: Number of seconds before the app starts to shutdown. New

requests are still accepted but the heartbeat page starts to return 503.

• hard_shutdown_interval [o, default: 1]: Number of seconds before the app is shut down. Any new call returns

a 503, and pending requests have that time to ﬁnish up the work before the app dies.

Notice that an event is triggered when the process ends, giving a chance for apps to cleanup things.

Example:

[global]

retry_after = 60

heartbeat_page = __another_heartbeat_url_

debug_page = __debug__

Storage

The storage section is storage. It contains everything needed by the storage server to read and write data.

Available options (o: optional, m: multi-line, d: default):

• backend: backend used for the storage. Takes the fully qualiﬁed name of the class.

Existing backends :

– syncstorage.storage.sql.SQLStorage

– syncstorage.storage.memcached.MemcachedSQLStorage.

• cache_servers [o, m]: list of memcached servers (host:port)

• sqluri: uri for the DB. see RFC-1738 for the format. driver://username:password@host:port/database. Sup-

ported drivers are: sqlite, postgres, oracle, mssql, mysql, ﬁrebird

• standard_collections [o, default: true]: if set to true, the server will use hardcoded values for collections.

• use_quota [o, default:false]: if set to false, users will not have any quota.

• quota_size [o, default:none]: quota size in KB

• pool_size [o, default:100]: deﬁne the size of the SQL connector pool.

• pool_recycle [o, default:3600]: time in ms to recycle a SQL connection that was closed.

Example:

134 Chapter 3. Server-side development guide

Mozilla Services Documentation, Release

[storage]

backend = memcached

cache_servers = 127.0.0.1:11211

192.168.1.13:11211

sqluri = mysql://sync:sync@localhost/sync

standard_collections = false

use_quota = true

quota_size = 5120

pool_size = 100

pool_recycle = 3600

Authentication

The authentication section is auth. It contains everything needed for authentication and registration.

Available options (o: optional, m: multi-line, d: default):

• backend: backend used for the storage.

Existing backends :

– services.auth.sql.SQLAuth

– services.auth.ldap.LDAPAuth

– services.auth.dummy.DummyAuth

• ldapuri [o]: uri for the LDAP server when the ldap backend is used.

• ldap_use_pool [o, default:False]: If True, a pool of connectors is used.

• ldap_pool_size [o, default:10]: Size of the ldap pool when used.

• use_tls [o, default:false]: If set to true, activates TLS when using LDAP.

• bind_user [o, default:none]: user for common LDAP queries.

• bind_password [o, default:none]: password for the bind user.

• admin_user [o, default:none]: user with extended rights for write operations.

• admin_password [o, default:none]: password for the admin user.

• users_root [o, default:none]: root for all ldap users. If set to md5 will generate a speciﬁc location based on the

md5 hash of the user name.

• cache_servers [o, m]: list of memcached servers (host:port)

• sqluri: uri for the DB. see RFC-1738 for the format. driver://username:password@host:port/database. Sup-

ported drivers are: sqlite, postgres, oracle, mssql, mysql, ﬁrebird

• pool_size [o, default:100]: deﬁne the size of the SQL connector pool.

• pool_recycle [o, default:3600]: time in ms to recycle a SQL connection that was closed.

Example:

[auth]

backend = ldap

ldapuri = ldap://localhost:390

3.1. Python Server Development Guide 135

Mozilla Services Documentation, Release

ldap_timeout = -1

ldap_use_pool = true

ldap_pool_size = 100

use_tls = false

bind_user = "cn=admin,dc=mozilla"

bind_password = admin

admin_user = "cn=admin,dc=mozilla"

admin_password = admin

users_root = "ou=users,dc=mozilla"

sqluri = mysql://sync:sync@localhost/sync

pool_size = 100

pool_recycle = 3600

cache_servers = 127.0.0.1:11211

Captcha

The captcha section enables the re-captcha feature during user registration.

Available options (o: optional, m: multi-line, d: default):

• use: if set to false, all operations will be done without captcha.

• public_key: public key for reCaptcha.

• private_key: private key for reCaptcha.

• use_ssl: if set to true, will use SSL when connecting to reCaptcha.

Example:

[captcha]

use = true

public_key = 6Le8OLwSAAAAAK-wkjNPBtHD4Iv50moNFANIalJL

private_key = 6Le8OLwSAAAAAEKoqfc-DmoF4HNswD7RNdGwxRij

use_ssl = false

Warning: The keys provided in this example work, as they were generated to provide a realistic example. But do

not use them in your applications.

Instead, you should generate a new set of keys for you own domain.

See: https://www.google.com/recaptcha/admin/create

SMTP

The smtp section conﬁgures the SMTP connection used by the application to send e-mails.

Available options (o: optional, m: multi-line, d: default):

• host [o, default:localhost]: SMTP host

136 Chapter 3. Server-side development guide

Mozilla Services Documentation, Release

• port [o, default:25]: SMTP port

• username [o, default:none]: SMTP user

• password [o, default:none]: SMTP password

• sender [o]: E-mail used for the sender ﬁeld.

Example:

[smtp]

host = localhost

port = 25

sender = weave@mozilla.com

CEF

The cef section conﬁgures how CEF security alerts are emitted.

Available options (o: optional, m: multi-line, d: default):

• use: if set to true, CEF alerts are emitted.

• ﬁle: location of the CEF log ﬁle. Can be a ﬁle path or syslog to use the syslog facility.

• syslog.options [o, default:none]: comma-separated values for syslog. Authorized values are: PID, CONS,

NDELAY, NOWAIT, PERROR

• syslog.priority [o, default:INFO]: priority level. Authorized values: EMERG, ALERT, CRIT, ERR, WARN-

ING, NOTICE, INFO, DEBUG.

• syslog.facility [o, default:LOCAL4]: facility. Authorized values: KERN, USER, MAIL, DAEMON, AUTH,

LPR, NEWS, UUCP, CRON and LOCAL0 to LOCAL7.

• vendor: CEF-speciﬁc option.

• version: CEF-speciﬁc option.

• device_version: CEF-speciﬁc option.

• product: CEF-speciﬁc option.

Example:

[cef]

use = true

file = syslog

syslog.options = PID,CONS

syslog.priority = DEBUG

syslog.facility = USER

vendor = mozilla

version = 0

device_version = 1.3

product = weave

Coding guidelines

3.1. Python Server Development Guide 137

Mozilla Services Documentation, Release

Style

All the code must follow PEP 8. You can use Flake8 to check for compliance. Flake8 is installed by default to all

server environments, via MoPyTools.

Flake8 provides a Mercurial hook, to perform a code check on every commit or qrefresh. Here’s an example of

Mercurial ~/.hgrc ﬁle:

[hooks]

commit = python:flake8.run.hg_hook

qrefresh = python:flake8.run.hg_hook

[flake8]

strict = 0

If strict option is set to 1, any warning will block the commit. When strict is set to 0, warnings are just displayed in

the standard output.

Read these:

• Flake8 documentation: http://pypi.python.org/pypi/ﬂake8

• PEP 8 - The ofﬁcial style guideline: http://www.python.org/dev/peps/pep-0008

Unicode vs Str

All internal strings in our server applications must use the unicode type unless stated otherwise.

Exceptions are:

• Rendered e-mails

• LDAP or SQL speciﬁc values

When the str type is used, the utf-8 encoding must be used:

>>> uni = u'Café'

>>> uni

u'Caf\xe9'

>>> uni.encode('utf-8')

'Caf\xc3\xa9'

>>> print uni.encode('utf-8')

Café

>>> print uni

Café

More on Unicode : http://docs.python.org/howto/unicode.html

Benching & Proﬁling

“We should forget about small efﬁciencies, say about 97% of the time: premature optimization is the root

of all evil”

– D. Knuth

138 Chapter 3. Server-side development guide

Mozilla Services Documentation, Release

Proﬁling

The base application provides a Proﬁler that knows how to generate proﬁling data that you’ll be able to visualize in

KCacheGrind.

To enable it you can toggle the proﬁle ﬂag to True in your paster ﬁle. See Conﬁguring the Proﬁler.

Restart your application using the built-in Paster server and start to use it or run a load test on it.

Remember that once enabled, your application will be really slow because the proﬁling is done by wrapping all calls

in the Python interpreter.

You can display the proﬁle information in real-time by visiting http://localhost:5000/__proﬁle__

But the real work should be done in KCacheGrind. Once you have ﬁnished testing the application, stop it and you

should ﬁnd at the root of your application a cachegrind.out ﬁle you can open.

3.1. Python Server Development Guide 139

Mozilla Services Documentation, Release

Benching

XXX

Releasing an application

Preparing a release

This section is a summary of the release process. The rest of the chapter explains everything in detail.

To cut a new release the process is:

1. start a release branch

2. pin the external dependencies versions

3. update RELEASE.txt

4. tag a release candidate

5. build a release and all rpms

140 Chapter 3. Server-side development guide

Mozilla Services Documentation, Release

6. ﬁx your release !

7. backport your changes

1. Start a release branch

The ﬁrst thing to do is to start a branch to do the releasing work.

Let’s say you want to release 1.2, create a 1.2-release branch:

$ git checkout -b 1.2-release

Go back to the default branch, then change the release to 1.3.dev1, so any further change will be in the 1.3 train.

The version is located in the .spec ﬁle under the version ﬁeld. Extract of a spec ﬁle:

%define version 1.3.dev1

$ git checkout master

... edit the spec file so version=1.3.dev1...

$ git commit -m 'starting the 1.3 developement'

$ git push origin master

Once everything is committed, go back to the release branch:

$ git checkout 1.2-release

Change the .spec ﬁle version to 1.2 (it’s probably 1.2.devX right now)

2. Pin the dependencies versions

When you release an application, you must make sure that all the dependencies are pinned. If you don’t do this, you

can’t be sure the application will be run in stage or production with the same versions that the ones you’ve tested.

This can be done in the prod-reqs.txt and stage-reqs.txt ﬁles. They contain all externals dependencies the project

should be using.

They usually have the same versions unless stage needs something speciﬁc.

Example:

cef == 0.2

WebOb == 1.0.7

Paste == 1.7.5.1

PasteDeploy == 1.3.4

PasteScript == 1.7.3

Mako == 0.4.1

MarkupSafe == 0.12

Beaker == 1.5.4

python-memcached == 1.47

simplejson == 2.1.6

Routes == 1.12.3

SQLAlchemy == 0.6.6

MySQL-python == 1.2.3

WSGIProxy == 0.2.2

recaptcha-client == 1.0.6

After you’ve tried a release, you can decide to raise the version to the latest stable version.

3.1. Python Server Development Guide 141

Mozilla Services Documentation, Release

3. Update RELEASE.txt

The RELEASE.txt ﬁle is a high-level changelog that’s used by other teams to know what the release contains.

Each release has a section with the date, containing three parts:

• Impacts: which teams are impacted by the release (Ops, QA, Infrasec, etc)

• Dependencies: list internal dependencies and their versions.

• Relevant changes: lists relevant changes with Bugzilla numbers.

Example:

1.2 - 2011-02-28

================

Impacts:

- Ops

Dependencies:

- None

Relevant changes:

- Bug 636294 - Prevent the automatic creation of the tables

- now using the standalone cef lib

4. Tag the release

Our tags are following this scheme: “rpm-MAJOR.MINOR.[MICRO]” where MAJOR.MINOR[.MICRO] is the ver-

sion of the Python package.

Examples:

$ git tag rpm-1.2

$ git tag rpm-1.2.1

Note: The rpm- preﬁx is a legacy preﬁx we’re keeping to avoid any conﬂict with the old PHP version tags.

5. Build the app and all RPMS

Building the app can now be done, by providing the tag value for your app, and if needed a tag value for internal

dependencies.

For example for account-portal (uses server-core), a call can look like this:

$ make build_rpms SERVER_CORE=rpm-2.0 ACCOUNT_PORTAL=rpm-1.2 RPM_CHANNEL=stage

The syntax for the options is: PROJECT_NAME=rpm-X.X. When used, will checkout the given project at the

mentioned tag. The tag can be a release tag, or master.

142 Chapter 3. Server-side development guide

Mozilla Services Documentation, Release

PROJECT_NAME refers to the name of the repository, after it has been upper-cased and all the dashes (“-”) replaced

by underscores (“_”).

For example, server-core becomes SERVER_CORE.

6. Fix your release

Sorry but your 1.2 release is a brown bag! You need to ﬁx the spec ﬁle and maybe a few Python bugs.

We will use the MICRO version to do this.

• Increment the release to 1.2.1

• Do your ﬁxes

• Tag 1.2.1

• Repeat and increment the MICRO version until the release works

7. Backport your changes

If you did a few micro releases, check if you need to backport them to the default branch.

More details

Naming convention

To avoid any conﬂict with another Python project – even if the project will not be released to PyPI, let’s use these

conventions:

• The project name should start with MozSvc

• Ideally a project contains a single package with a mozsvc preﬁx as well

Note: MozSvc is pronounced Mozz-Vikk, which is an ancient Irish Gaelic word that literally means “Viking Mice”.

Versioning scheme

Final Releases

For ﬁnal releases, projects are versioned using the MAJOR.MINOR scheme.

Examples:

• 1.0

• 1.1

• 2.1

The MINOR part is incremented in the day-to-day work and the MAJOR part is incremented on important updates.

The deﬁnition of important is left to the judgment of the releaser.

3.1. Python Server Development Guide 143

Mozilla Services Documentation, Release

We don’t really have any strategy here, like incrementing MAJOR only on backward incompatible changes: all Python

packages we use are part of a server application and the only public facing API is documented web services that have

their own versioning scheme.

That said, if a library is published at PyPI, it has supposedly reached a stable state, and incrementing the MAJOR

version should occur on backward incompatible changes.

When a release fails in stage or prod, we can use a MAJOR.MINOR.MICRO scheme to ﬁx it.

Development Releases

The master should always have a version with a .devN sufﬁx. That is, the next version to be released, with N being an

integer. Examples:

• 1.5.dev1

• 1.4.dev23

Full example

Here’s a full scenario of versioning usage:

• 1.2 is in production, tagged as “rpm-1.2”

• we want to push a 1.3

• we change the default branch version to 1.4.dev1

• we branch “1.3-release”

• a 1.3 is tagged there as “rpm-1.3”

• 1.3 is pushed on stage

• it’s not working

• devs ﬁx and tag 1.3.1 in the branch

• 1.3.1 is pushed on stage, it’s working

• 1.3.1 is pushed in production

• it breaks !!!

• production is rolled back to 1.2

• devs ﬁx the problems and tag 1.3.2

• 1.3.2 is pushed on stage, it’s working

• 1.3.2 is pushed in production

• it works, congrats. Now working on 1.4.dev1 in master

The Makeﬁle

Releases are driven by the Makefile ﬁle contained in the project.

It should contain these targets:

• build: builds the project in-place

144 Chapter 3. Server-side development guide

Mozilla Services Documentation, Release

• tests: runs the tests.

• build_rpms: build the RPM collection. The collection must include the project RPM but also all direct and

indirect dependencies.

• mock: builds the RPMs, install them in a chroot, then make sure the app can be imported in Python

In more details:

The build target does the following:

1. install a local virtualenv

2. install MoPyTools in it

3. set the project to a speciﬁc channel (prod, dev or stage)

4. build the application and pull internal and external dependencies

The test target runs Nose against the project.

The build_rpms target generates the RPM for the project and for all its internal and external dependencies, using

pypi2rpm

The mock target calls build_rpms then installs everything in a chroot using Mock, then runs an import. That ensures

the spec ﬁle dependencies are error free, and the Python app main module is importable. Notice that this target is run

only under Centos5.

Here’s an extract of a typical Makeﬁle:

APPNAME = server-key-exchange

DEPS = server-core

BUILDAPP = bin/buildapp

BUILDRPMS = bin/buildrpms

CHANNEL = dev

RPM_CHANNEL = prod

VIRTUALENV = virtualenv

NOSE = bin/nosetests -s --with-xunit

TESTS = keyexchange/tests

INSTALL = bin/pip install

build:

$(VIRTUALENV) --no-site-packages --distribute .

$(INSTALL) MoPyTools

$(BUILDAPP) $(PYPIOPTIONS) -c $(CHANNEL) $(DEPS)

test:

$(NOSE) $(TESTS)

build_rpms:

$(BUILDRPMS) -c $(RPM_CHANNEL) $(DEPS)

mock: build build_rpms

mock init

mock --install python26 python26-setuptools

cd rpms; wget http://mrepo.mozilla.org/mrepo/5-x86_64/RPMS.mozilla-services/

˓→gunicorn-0.11.2-1moz.x86_64.rpm

cd rpms; wget http://mrepo.mozilla.org/mrepo/5-x86_64/RPMS.mozilla/nginx-0.7.65-4.

˓→x86_64.rpm

mock --install rpms/

mock --chroot "python2.6 -m keyexchange.run"

3.1. Python Server Development Guide 145

Mozilla Services Documentation, Release

Channels

We deﬁne three channels:

• dev: development channel, most dependencies are unpinned, so the latest PyPI release is taken

• prod: all dependencies should be pinned default one

• stage: all dependencies should be pinned – might vary from production versions. This channel is most of the

time the same as production but can be useful in case the staging environment needs to be different.

Requirement ﬁles

All dependencies are listed in requirement ﬁles. A requirement ﬁle is a text ﬁle with a list of dependencies. One per

line. Each dependency can have a version information. The ﬁle follows Pip’s standard. See http://www.pip-installer.

org/en/latest/requirement-format.html

Example:

cef

WebOb == 1.0.7

Paste

PasteDeploy

PasteScript

Mako

MarkupSafe

Beaker

python-memcached

simplejson

Routes

SQLAlchemy <= 0.6.99

MySQL-python

WSGIProxy

recaptcha-client

There should be three requirement ﬁles located at the root of the project, one for each channel:

1. dev-reqs.txt: requirements for the dev channel

2. stage-reqs.txt: requirements for the stage channel

3. prod-reqs.txt: requirements for the prod channel

stage and prod ﬁles should have pinned versions, since those ﬁles will be used to build applications to be released in

production.

Example:

cef == 0.2

WebOb == 1.0.7

Paste == 1.7.5.1

PasteDeploy == 1.3.4

PasteScript == 1.7.3

Mako == 0.4.1

MarkupSafe == 0.12

Beaker == 1.5.4

python-memcached == 1.47

simplejson == 2.1.6

Routes == 1.12.3

146 Chapter 3. Server-side development guide

Mozilla Services Documentation, Release

SQLAlchemy == 0.6.6

MySQL-python == 1.2.3

WSGIProxy == 0.2.2

recaptcha-client == 1.0.6

When a build or a build_rpms is invoked, it receives a channel option and picks the corresponding requirement ﬁles to

decide which version to pick. Unpinned versions will make the build process pick the latest release at PyPI. (Even if

it’s not stable!)

For the build target the default value is dev and for the build_rpms option it’s prod.

You can also force a speciﬁc channel for build with the CHANNEL variable:

$ make build CHANNEL=prod

And for build_rpms, RPM_CHANNEL:

$ make build RPM_CHANNEL=stage

When the channel option is provided, the Makeﬁle will use the dependencies list from the CHANNEL-reqs.txt ﬁle.

Conﬁguration ﬁles speciﬁcation

All Services applications need to use the same conﬁguration ﬁle format. This document speciﬁes it.

Syntax

The conﬁguration ﬁle is a ini-based ﬁle. (See http://en.wikipedia.org/wiki/INI_ﬁle for more details.) Variable names

can be assigned values, and grouped into sections.

A line that starts with “#” is commented out. Empty lines are also removed.

Example:

[section1]

# comment

name = value

name2 = "other value"

[section2]

foo = bar

Ini readers in Python, PHP and other languages understand this syntax. Although, there are subtle differences in the

way they interpret values and in particular if/how they convert them.

Values conversion

Here are a set of rules for converting values:

• If value is quoted with ” chars, it’s a string. This notation is useful to include “=” characters in the value. In case

the value contains a ” character, it must be escaped with a “” character.

• When the value is composed of digits and optionally preﬁxed by “-”, it’s tentatively converted to an integer or a

long depending on the language. If the number exceeds the range available in the language, it’s left as a string.

3.1. Python Server Development Guide 147

Mozilla Services Documentation, Release

• If the value is “true” or “false”, it’s converted to a boolean, or 0 and 1 when the language does not have a boolean

type.

• A value can be an environment variable : “${VAR}” is replaced by the value of VAR if found in the environment.

If the variable is not found, an error must be raised.

• A value can contain multiple lines. When read, lines are converted into a sequence of values. Each new line for

a multiple lines value must start with at least one space or tab character.

Examples:

[section1]

# comment

a_flag = True

a_number = 1

a_string = "other=value"

another_string = other value

a_list = one

two

three

user = ${USERNAME}

Extending a ﬁle

An INI ﬁle can extend another ﬁle. For this, a “DEFAULT” section must contain an “extends” variable that can point

to one or several INI ﬁles which will be merged into the current ﬁle by adding new sections and values.

If the ﬁle pointed to in “extends” contains section/variable names that already exist in the original ﬁle, they will not

override existing ones.

ﬁle_one.ini:

[section1]

name2 = "other value"

[section2]

foo = baz

bas = bar

ﬁle_two.ini:

[DEFAULT]

extends = file_one.ini

[section2]

foo = bar

Result:

[section1]

name2 = "other value"

[section2]

foo = bar

bas = bar

To point to several ﬁles, the multi-line notation can be used:

148 Chapter 3. Server-side development guide

Mozilla Services Documentation, Release

[DEFAULT]

extends = file_one.ini

file_two.ini

When several ﬁles are provided, they are processed sequentially. So if the ﬁrst one has a value that is also present in

the second, the second one will be ignored. This means that the conﬁguration goes from the most specialized to the

most common.

Implementations

There’s one implementation in the core package of the Python server, but it could be moved to a standalone distribution

if another project wants to use it.

http://bitbucket.org/tarek/sync-core/src/tip/services/conﬁg.py

Glossary

WSGI Web Server Gateway Interface. It is a speciﬁcation for web servers and application servers to communicate

with web applications (though it can also be used for more than that). It is a Python standard, described in detail

in PEP 333.

Basic Authentication The client authenticates by sending an Authorization header of the form: “Basic

base64(username + ‘:’ + password)

services-builds The Build mailing list at [email protected]g receives all Jenkins failures.

You can subscribe to it at: https://mail.mozilla.org/listinfo/services-builds

services-dev The Dev mailing list at [email protected]g is the place where we discuss the development of

Mozilla Services applications.

You can subscribe to it at: https://mail.mozilla.org/listinfo/services-dev

Feedback is welcome at [email protected] or at services-dev.

Indices and tables

• genindex

• search

3.1. Python Server Development Guide 149

Mozilla Services Documentation, Release

150 Chapter 3. Server-side development guide

CHAPTER 4

Client Development

Firefox Health Report

Content on this page has moved to https://hg.mozilla.org/mozilla-central/ﬁle/default/services/healthreport/docs/.

Metrics Collection

Content on this page has moved to https://hg.mozilla.org/mozilla-central/ﬁle/default/services/docs/.

Sync Client Documentation

This section is intended to provide a comprehensive guide to how Firefox Sync clients interact with the server and

ultimately with each other to provide the functionality of syncing browser data between clients.

It is a somewhat technical document, but should require no in-depth knowledge. Links to more detailed API docs offer

an opportunity to dig deeper.

Overview

Introduction

The purpose of Sync is to exchange browser data (bookmarks, history, open tabs, passwords, add-ons, and the like)

between “clients” in a manner that respects a user’s security and privacy.

Syncing is facilitated through the use of a server, where data is centrally stored. This allows for syncing to occur

without pairwise interaction between network-connected clients.

151

Mozilla Services Documentation, Release

Sync Server

Client 1 Client 2

Sync is different from most storage-in-the-cloud services in that data is encrypted locally - that is it cannot be read by

other parties - before it is sent to the cloud. While many services encrypt data as it is being transmitted, Sync keeps

your data encrypted even after it has arrived at the server.

This means that the Sync server operators can’t read your data - even if they wanted to. The only way your data can be

read is if someone possesses your secret Sync Key (sometimes referred to as a Recovery Key). This can occur if your

device is lost or hacked or if you reveal it to another party. The important fact to note is that the Sync Key is never

made available to the Sync Server and without it, your encrypted data is statistically impossible to recover.

That being said, the server operators do have access to some limited data. This includes logs of when you connected

and the types, number, and rough size of items being synchronized. This type of information is “leaked” for practically

every network-connected service, so it shouldn’t come as a surprise.

The Sync Server

The Sync server performs the vital role of storing data, tracking elementary metadata, and providing authenticated

access. The Sync server is effectively a dumb shared whiteboard - a bit bucket if you will. It plays a very small role in

the actual syncing process. And, it must be this way: since data is encrypted before being sent to the server, there is

not much the server can do to help.

The Sync server infrastructure exposes a secure HTTP interface for user management and node assignment as well

as storage. The storage server is actually a generic service that isn’t Sync-speciﬁc. Sync just uses it with speciﬁc

semantics for how and where to store data. These semantics are fully described at Sync Storage Formats.

Per-user access to the Sync server is protected via authentication at the HTTP layer. This can be whatever the server

operator desires. Since the bulk of Sync’s security model resides in client-side encryption (read below) and since a

Sync server is typically made available behind transport-level encryption (like SSL/TLS), primitive forms of security

such as HTTP Basic Authentication are adequate. In fact, Mozilla’s hosted Sync server that is used by default by

Firefox has used HTTP basic auth.

Collections and records

The primary concept behind the Sync server’s storage part is that of the collection. Clients can store objects called

records into collections.

Sync clients take their data, convert it to records, then send them to the Sync server. Receiving data does the same, but

in reverse.

152 Chapter 4. Client Development

Mozilla Services Documentation, Release

Records contain basic public metadata, such as the time they were last modiﬁed. This allows clients to selectively

retrieve only the records that have changed since the last sync operation.

An important observation is that the server has no notion of a “sync” as understood by the client. From the server’s

perspective, there is simply a series of HTTP requests arriving from various IP addresses performing storage operations

on a stateful backing store. The client has a well-deﬁned sequence of actions that take place within a notional session,

which can succeed or fail as a whole; the server does not.

Sync Clients

A Sync Client is an entity that talks to servers providing Sync functionality.

Sync clients come in many different ﬂavors with different levels of support for different features. For example, some

clients may be read-only.

A speciﬁc client often targets speciﬁc versions of the storage service and Sync storage formats.

The Life of a Sync

This document essentially describes how to write a Sync client.

Because the Sync server is essentially a dumb storage bucket, most of the complexity of Sync is the responsibility of

the client. This is good for users’ data security. It is bad for people implementing Sync clients. This document will

hopefully alleviate common issues and answer common questions.

Strictly speaking, information in this document applies only to a speciﬁc version of the Sync server storage format. In

practice, client behavior is similar across storage versions. And, since we wish for clients to support the latest/greatest

versions of everything, this document will target that.

Initial Client Conﬁguration

The process of performing a sync starts with conﬁguring a fresh client. Before you can even think about performing a

sync, the client needs to possess key pieces of information. These include:

• The URL of the Sync server.

• Credentials used to access the Sync server.

Depending on the versions of the Sync server and global storage version, you may also need a Sync Key or similar

private key which is used to access encrypted data on an existing account.

Obtaining these pieces of information is highly dependent on the server instance you will be communicating with, the

client in use, and whether you are creating a new account or joining an existing one.

How Mozilla and Firefox Does It

For reference, this section describes how Mozilla and Firefox handle initial client conﬁguration.

Inside Firefox there exists a UI to Set up Firefox Sync. The user chooses whether she is setting up a new account or

whether she wants to connect to an existing account.

For completely new accounts, the user is presented with a standard sign-up form. The user enters her email address

and selects a password. Behind the scenes Firefox is talking to a user provisioning service and the account is created

there and a Sync server is assigned (Mozilla exposes many different Sync server instances to the Internet and the client

connects directly to just one of them). At this time, a new Sync Key encryption key is generated and stored in Firefox’s

credential manager (possibly protected behind a master password).

4.3. Sync Client Documentation 153

Mozilla Services Documentation, Release

If the user selects an existing account, the user is presented 12 random characters. These are entered on another device

and the two devices effectively pair and share the login credentials, Sync Key, and server info. This is done with J-

PAKE, so the data is secure as it is transported between devices. Even the intermediary agent bridging the connection

between the two devices can’t decrypt the data inside.

Performing a Sync

Settings and State Pre-check

To perform a sync, a client will ﬁrst need to perform some basic checks:

• Do we have all necessary credentials? - Storage server HTTP credentials - Sync Key

• Are we online (do we have network connectivity)

• Are we prohibited from syncing due to result from a previous sync? - The server may have issued a backoff

telling us to slow down, etc

If these are all satisﬁed, the client can move on to the next phase.

Inspect and Reconcile Client and Server State

The initial requests performed on the Sync server serve to inspect, verify, and reconcile high-level state between the

client and server.

Fetch info/collections

The ﬁrst request to the Sync server should be a GET on the info/collections URI. This is a utility API provided by the

storage service that reveals which collections exist on the server and when they were last modiﬁed.

If the client has synced before, it should issue a conditional HTTP request by adding an X-If-Modiﬁed-Since header to

the request. If the server responds with a 304, it means that no modiﬁcations have been made since the last sync. If the

client has no new data to upload (perhaps it was just checking to see if there was any new data it needed to download),

it can stop the sync right now: there is nothing more for it to do!

The info/collections request also serves as a means to verify that the local credentials can connect with the server. If

the server issues a 401 or 404 response, the client should interpret this as credentials failure. The next steps is this case

are highly dependent on how the Sync server is conﬁgured. If using some kind of cached credentials (such as a token),

the client may want to automatically try to fetch new credentials and try again.

Assuming you have a response from info/collections, you’ll need to process that response and possibly take action. If

you received a 304 and have data to upload, you can potentially skip processing if you have all the required values

cached locally.

154 Chapter 4. Client Development

Mozilla Services Documentation, Release

Prepare HTTP Request

Have Synced Before?

Add X-If-Modified-Since Header

Yes

Perform HTTP Request

Check Response

Wait for Response

Have Outgoing Changes?

304

Reauthenticate

401, 403

End Sync

Next Step

Yes

Validate meta/global

The client needs to validate the meta/global record on every request. Upon successful completion of the

info/collections request, the following outcomes are possible:

1. The meta collection does not exist.

2. The meta collection has been modiﬁed since the last sync.

3. The meta collection has not been modiﬁed since the last sync.

4.3. Sync Client Documentation 155

Mozilla Services Documentation, Release

If the meta collection does not exist, the global record inside of it cannot exist. This means no client has synced yet.

If info/collections reveals any collection exists, the client should issue a request to delete all data from the server to

ensure the server is in a fresh state. If there are no collections on the server, you don’t need to issue a delete.

Before we talk about uploading a new meta/global record, let’s talk about processing existing ones.

If the meta collection has not been modiﬁed since the last sync and we have all of the data from a previous fetch of the

meta/global cached locally (scenario 3), the client doesn’t need to do anything.

If the meta collection has been modiﬁed or if the client doesn’t have a cached copy of the metaglobal data, the client

will need to fetch the meta/global record. Simply issue a GET request to the appropriate URI and decode the payload

according to the rules for the storage version the client is using.

If you can’t decode the payload, that’s bad and should never happen. But, it is possible, so you need to handle it.

One solution is to delete all data from the server and upload a new record. However, data on the server could be from

a newer client this one just can’t understand, so it shouldn’t do this lightly. The storage versions have been deﬁned

such that the decoding format of the meta/global are backwards compatible with prior versions. So, if there is an error

decoding, there is almost certainly something wrong going on.

From the decoded payload, the client should ﬁrst inspect the storage version number. If the client supports this storage

version, all is well. Carry on. If not, the client has a few choices to make. If the version is older than what the client

supports, the client can upgrade the server’s data to the new version. These semantics are highly speciﬁc to the speciﬁc

version change. If the version is newer than what the client supports, the client should likely interpret this as “there is

a newer client out there - I’m too old and need to upgrade.” If clients see a new storage format, they should probably

stop what they are doing. Under no circumstances should clients attempt to modify data belonging to a newer storage

version. Instead, delete all data and perform a fresh start (if this is really what you want to do).

This section is incomplete. There is more that needs to be described. The graph below is also incomplete.

Check info/collections

Any Collections Exist?

No 'meta' collection

Delete all Server Collections

Yes

Modified Since Last Sync?

Have 'meta' collection

Fresh Start

Process Response

Wait for Response

204 No Content

Start New Sync

401, 403

TODO

156 Chapter 4. Client Development

Mozilla Services Documentation, Release

Validate crypto/keys

Have Keys Cached?

crypto Collection Exists?

Nocrypto Collection Exists?

Yes

Generate Keys

Fetch Keys

Yes

crypto Collection Modified?

Yes

WUT?

Yes

Next Step

Upload Keys

Keys Valid?

200 OK

Abort

401, 403

201 No Context 401, 403

Not OK

Collections Pre-Sync

Once meta/global and the cryptographic keys are in a good state, it is time to start syncing the regular collections.

The ﬁrst thing the client does is record the last modiﬁed times from the info/collections record. The client will ask the

server for records that changed between the last time it synced and the last modiﬁed time of the collection.

Clients Collection

The clients collection, while a regular collection, is special. Clients always cache all records in the clients collection.

This collection is also used to send commands between clients. Some commands tell a client to do important things,

like clear data. Because of this, commands need to be processed before other collections.

4.3. Sync Client Documentation 157

Mozilla Services Documentation, Release

OLD CONTENT

Don’t read below this. It is old and needs to be formalized.

Perform sync

// - update engine last modified timestamps from info/collections record

// - sync clients engine

// - clients engine always fetches all records

// - process reset/wipe requests in 'firstSync' preference

// - process any commands, including the 'wipeClient' command

// - infer enabled engines from meta/global

// - sync engines

// - only stop if 401 is encountered

// - if meta/global has changed, reupload it

Syncing an engine

// TODO WRITEME

// - meta/global

// - syncID

// - engine storage format

// - fetch incoming records

- GET .../storage/<collection>?newer=<last_sync_server_timestamp>&full=1

- optional but recommended for streaming: Accept: application/newlines

- deserialize and apply each record:

- JSON parse WBO

- JSON parse payload

- verify HMAC

- decrypt ciphertext witH IV

- JSON parse cleartext

- apply to local storage

- TODO deduping

- fetch outgoing records (e.g. via last sync local timestamp,

or from list of tracked items, ...)

- serialize each record

- assemble cleartext record and JSON stringify

- assemble payload and JSON stringify

- generate random IV and encrypt cleartext to ciphertext

- compute HMAC

- assemble WBO and JSON stringify

- upload in batches of 100 or 1 MB, whichever comes first

- POST .../storage/<collection>

[{record}, {record}, ...]

- process repsonse body

Sync Storage Formats

The way that Sync clients store data on a Storage Server is deﬁned by sets of integer storage versions. Each storage

version deﬁnes speciﬁc semantics for how clients are supposed to behave.

158 Chapter 4. Client Development

Mozilla Services Documentation, Release

Global Storage Version

There exists a global storage version which deﬁnes global semantics. This global version typically deﬁnes the follow-

ing:

• What special records exist on the server and what they contain

• The payload format of encrypted records on the server

• How cryptography of data works

Each Sync client is coded to support 1 or more global storage formats. If a client encounters a storage format it doesn’t

support, it should probably stop trying to consume data. Under no normal circumstances should a client modify data

on a server that is deﬁned with an unknown, newer storage format. Even if an old client wipes all server data and

uploads data in its format, newer clients may transparently upgrade to the global storage version they support.

Because changing storage formats can cause clients to not be able to use Sync because all clients may not be upgraded

to support a newer storage format at the same time, new global storage versions are rarely introduced.

Versions 1, 2, and 3

These were used by an old version of Sync which was deprecated in early 2011. Historical information is available

here.

These versions should not be in active use and should all be upgraded to a newer storage format.

Version 4

This version initially made the switch to a new crypto model based fully on AES. Because of a faulty implementation

of the crypto, version 5 was created to force alpha clients created with the faulty implementation to upgrade. Version

4 and version 5 are therefore practically identical.

Version 5 (Spring 2011 - Current)

Version 5 replaces version 3’s cryptographic model with one based purely on AES.

A full overview is available for reference.

Historical notes are available.

Version 6 (???)

PROPOSAL

In terms of cryptography, version 6 is a natural evolution of version 5. It makes minor changes to low-level cryptogra-

phy details.

One driving force behind version 6 was the need to support storage of the encrypted Sync Key on the storage server.

This was required in order to support BrowserID integration.

Another driving force was the transition to version 2.0 of the Storage Service.

Strictly speaking, neither of these require a new global storage version. However, they presented an enticing opportu-

nity to ﬁx minor issues with version 5.

Version 6 is fully documented.

4.3. Sync Client Documentation 159

Mozilla Services Documentation, Release

Collection/Object Format Versions

The formats of unencrypted records stored on the server are also versioned. For example, records in the bookmarks

collection are all deﬁned to be of a speciﬁc type. Strictly speaking, these versions are tied to a speciﬁc global version.

However, since all storage formats to date have stored the per-collection version in a special record, these versions in

effect apply across all global storage versions.

These versions are fully documented.

Global Storage Version 5

This document describes version 5 of Sync’s global storage format. This describes not only the technical details of the

storage format, but also some semantics for how clients supporting version 5 should interact with the Sync server.

Overview

A single unencrypted record called the metaglobal record (because it exists in the meta collection with the id global)

stores essential data used to instruct clients how to behave.

A special record called the cryptokeys record (because it exists in the crypto collection with the id keys) holds

encrypted keys which are used to encrypt, decrypt, and verify all other encrypted records on the server.

Cryptography

Overview

Every encrypted record (and all but one record on the server is encrypted) is encrypted using symmetric key encryption

and veriﬁed using HMAC hashing. The symmetric encryption and HMAC veriﬁcation keys are only available to client

machines: they are not transmitted to the server (at least in any form the server can read). This means that the data on

the server cannot be read by anyone with access to the server.

The aforementioned symmetric encryption key and and HMAC key constitute what’s called a key bundle. Each key

is 256 bits.

Individual records are encrypted with AES 256. The encryption key from a key bundle is combined with a per-record

16 byte IV and a user’s data is converted into ciphertext. The ciphertext is signed with the key bundle’s HMAC key.

The ciphertext, IV, and HMAC value are then uploaded to the server.

When Sync is initially conﬁgured, that client generates a random 128 bit sequence called the Sync Key. This private

key is used to derive a special key bundle via HKDF. This is called the Sync key bundle. The Sync key bundle is used

to encrypt and decrypt a special record on the server which holds more key bundles. Key bundles inside this record are

what’s used to encrypt and decrypt all other records on the server.

Terminology

Sync Key 128 bit random value which effectively serves as the master key to Sync.

Key Bundle A pair of 256 bit keys. One key is used for symmetric encryption. The other is used for HMAC

hashing.

Sync Key Bundle A Key Bundle derived from the Sync Key via HKDF.

HKDF Cryptographic technique to create values derived from another.

160 Chapter 4. Client Development

Mozilla Services Documentation, Release

Bulk Key Bundle A collection of Key Bundles used to secure records. This collection is encrypted with the Sync

Key Bundle.

Cleartext The plain/clear representation of a piece of data. This is the underlying data that will be exchanged via

Sync. It could contain personal and sensitive data.

Ciphertext The encrypted version of Cleartext. Ciphertext cannot be turned back into Cleartext without an En-

cryption Key.

Encryption Key A key in a Key Bundle used for symmetric encryption. This helps turn Cleartext into Ciphertext.

HMAC Key A key in a Key Bundle used for HMAC hashing.

Symmetric Encryption Process by which Cleartext is converted into Ciphertext and back again with the help of a

secret key.

HMAC Hashing A technique used to verify that messages (Ciphertexts) haven’t been tampered with. A HMAC

Key is applied over a Ciphertext to produce a HMAC Value.

The Sync Key

The Sync Key is the master private key for all of Sync. A single Sync Key is shared between all clients that wish to

collaborate with each other using the server. It is important to state that the Sync Key should never be transmitted to an

untrusted party or stored where others could access it. This includes inside the storage server.

The Sync Key is a randomly generated 128 bit sequence. Generation of this value is left to the client. It is assumed that

the chosen random sequence is cryptographically random.

For presentation purposes, the Sync Key should be represented as 26 characters from the friendly base32 alphabet with

dashes after the 1st, 6th, 11th, 16th, and 21st characters. Our friendly base32 alphabet uses lower case characters and

substitutes 8 for l and 9 for o. This prevents ambiguity between 1 and l and 0 and o. In addition, we strip off padding

that may be at the end of the string.

In pseudo-code:

sync_key = randomBytes(16)

sync_key_ui = encodeBase32(sync_key).lowerCase().substr(0, 26).replace('l', '8').

˓→replace('o', '9')

sync_key_dashes = sync_key_ui.replaceRegEx(/{.{1,5})/g, "-$1")

Example:

# Generate 16 random bytes

\xc7\x1a\xa7\xcb\xd8\xb8\x2a\x8f\xf6\xed\xa5\x5c\x39\x47\x9f\xd2

# Base32 encode

Y4NKPS6YXAVI75XNUVODSR472I======

# Lower case and strip

y4nkps6yxavi75xnuvodsr472i

# Perform friendly string substitution (note 'o' to '9')

y4nkps6yxavi75xnuv9dsr472i

# Add dashes for user presentation

y-4nkps-6yxav-i75xn-uv9ds-r472i

4.3. Sync Client Documentation 161

Mozilla Services Documentation, Release

Sync Key Bundle

The Sync Key Bundle is a key bundle derived from the Sync Key via SHA-256 HMAC-based HKDF (RFC 5869).

Remember that a key bundle consists of a 256 bit symmetric encryption key and a HMAC key.

In pseudo-code:

info = "identity.mozilla.com/picl/v1/oldsync"

T(1) = HMAC-SHA256(sync_key, info + 0x01)

T(2) = HMAC-SHA256(sync_key, T(1) + info + 0x02)

encryption_key = T(1)

hmac = T(2)

Example:

sync_key = \xc7\x1a\xa7\xcb\xd8\xb8\x2a\x8f\xf6\xed\xa5\x5c\x39\x47\x9f\xd2

info = "identity.mozilla.com/picl/v1/oldsync"

# Perform HKDF Expansion (1)

encryption_key = HKDF-Expand(sync_key, info + "\x01", 32)

-> 0x8d0765430ea0d9dbd53c536c6c5c4cb639c093075ef2bd77cd30cf485138b905

# Second round of HKDF

hmac = HKDF-Expand(sync_key, encryption_key + info + "\x02", 32)

-> 0xbf9e48ac50a2fcc400ae4d30a58dc6a83a7720c32f58c60fd9d02db16e406216

NB1: The Sync Key is stored in Firefox Accounts. It is referred to as ‘kB’ in https://github.com/mozilla/

fxa-auth-server/wiki/onepw-protocol#-fetching-sync-keys (kA is not used).

NB2: In earlier versions, “Sync-AES_256_CBC-HMAC256” + username was used as the hkdf info string instead of

“identity.mozilla.com/picl/v1/oldsync”, for instance “Sync-AES_256_CBC-HMA[email protected]”.

Record Encryption

Individual records are encrypted using the AES algorithm + HMAC “signing” using keys from a key bundle.

You take your cleartext input (which is typically a JSON string representing an object) and feed it into AES. You

Base64 encode the raw byte output of that and feed that into HMAC SHA-256.

The AES cipher mode is CBC.

In pseudo-code:

cleartext = "SECRET MESSAGE"

iv = randomBytes(16)

ciphertext = AES256(cleartext, bundle.encryptionKey, iv)

hmac = SHA256HMAC(bundle.hmacKey, base64(ciphertext))

Example:

encryption_key = 0xd3af449d2dc4b432b8cb5b59d40c8a5fe53b584b16469f5b44828b756ffb6a81

hmac_key = 0x2c5d98092d500a048d09fd01090bd0d3a4861fc8ea2438bd74a8f43be6f47f02

cleartext = "SECRET MESSAGE"

162 Chapter 4. Client Development

Mozilla Services Documentation, Release

iv = randomBytes(16)

-> 0x375a12d6de4ef26b735f6fccfbafff2d

ciphertext = AES256(cleartext, encryption_key, iv)

-> 0xc1c82acc436de625edf7feca3c9deb4c

ciphertext_b64 = base64(ciphertext)

-> wcgqzENt5iXt9/7KPJ3rTA==

hmac = HMACSHA256(hmac_key, ciphertext_b64)

-> 0xb5d1479ae2019663d6572b8e8a734e5f06c1602a0cd0becb87ca81501a08fa55

The ciphertext, IV, and HMAC are added to the record and uploaded to the server.

Record Decryption

When you obtain a record, that record will have attached its ciphertext, HMAC, and IV. The client will also have a key

bundle (with an encryption key and HMAC key) that is associated with that record’s collection.

The ﬁrst step of decryption is verifying the HMAC. If the locally-computed HMAC does not match the HMAC on

the record, the record could either have been tampered with or it could have been encrypted with a different key

bundle from the one the client has. Under no circumstances should a client try to decrypt a record if the HMAC

veriﬁcation fails.

Once HMAC veriﬁcation is complete, the client decrypts the ciphertext using the IV from the record and the encryption

key from the key bundle.

In pseudo-code:

ciphertext = record.ciphertext

iv = record.iv

record_hmac = record.hmac

encryption_key = bundle.encryption_key

hmac_key = bundle.hmac_key

local_hmac = HMACSHA256(hmac_key, base64(ciphertext))

if local_hmac != record_hmac:

throw Error("HMAC verification failed.")

cleartext = AESDecrypt(ciphertext, encryption_key, iv)

Example:

TODO

New Account Bootstrap

When a new Sync account is initially conﬁgured or when an existing Sync account is reset, we perform an initial

bootstrap of the cryptographic components.

1. The Sync Key is generated.

2. The Sync key bundle is derived from the Sync Key.

4.3. Sync Client Documentation 163

Mozilla Services Documentation, Release

3. New key bundles are created.

4. The new key bundles are assembled into a bulk key bundle/record and uploaded to the server after being encrypted

by the Sync key bundle.

At this point, the client is bootstrapped from a cryptography perspective.

Metaglobal Record

The meta/global record is a special record on the Sync Server that contains general metadata to describe the

state of data on the Sync Server. This state includes things like the global storage version and the set of available

engines/collections on the server.

The meta/global record is different from other records in that it is not encrypted.

The payload of this record is a JSON string that deserializes to an object (i.e. a hash). This object has the following

ﬁelds:

• storageVersion: Integer version of the global storage format used

• syncID: Opaque string that changes when drastic changes happen to the overall data. Change of this string can

cause clients to drop cached data. The Firefox client uses 12 randomly generated base64url characters, much

like for WBO IDs.

• engines: A hash with ﬁelds of engine names and values of objects that contain version and syncID ﬁelds, which

behave like the storageVersion and syncID ﬁelds on this record, but on a per-engine level.

In Protocol 1.5, an additional ﬁeld is present:

• declined: engines that are not present in engines, and are not present in this array, can be presumed to be neither

enabled nor explicitly declined. If a user has explicitly declined an engine, rather than e.g., not having the option

due to missing functionality on the client, then it should be added to this list in the uploaded meta/global record.

No engine should be present in both engines and declined; if an error results in this situation, engines takes

precedent.

Example

{

"syncID":"7vO3Zcdu6V4I",

"storageVersion":5,

"engines":{

"clients": {"version":1,"syncID":"Re1DKzUQE2jt"},

"bookmarks": {"version":2,"syncID":"ApPN6v8VY42s"},

"forms": {"version":1,"syncID":"lLnCTaQM3SPR"},

"tabs": {"version":1,"syncID":"G1nU87H-7jdl"},

"history": {"version":1,"syncID":"9Tvy_Vlb44b2"},

"prefs": {"version":2,"syncID":"8eONx16GXAlp"}

"declined": ["passwords"]

}

Semantics and Behavior

Clients should fetch the metaglobal record after it has been determined that a full sync should be performed. If the

metaglobal record does not exist, the client should issue a request to delete all data from the server and then create and

upload a new metaglobal record.

164 Chapter 4. Client Development

Mozilla Services Documentation, Release

In the common scenario where the metaglobal record exists, the client should ﬁrst check that the storage version from

the record is supported. If it is, great. If the storage version is older than what the client supports, the client may choose

to upgrade server data to a new storage version. Keep in mind this may break older clients! If the storage version is

newer than what the client supports, all bets are off and the client should infer that a new version is available and that

the user should upgrade. Clients should not modify any data on a server if the global storage version is newer

than what is supported.

crypto/keys record

In storage version 5, the public/private key layer has been dropped. All bulk keys are now stored in this one WBO.

Encryption and HMAC keys are separate keys and kept in key pairs.

Encrypting and decrypting

The `crypto/keys` WBO is encrypted and veriﬁed just like any other WBO, except the Sync Key bundle is used

instead of a bulk key bundle.

Format

The inner payload of the crypto/keys record contains the following ﬁelds:

• default: Array of length 2 containing the default key pair (encryption key, HMAC key).

• collections: Object mapping collection name to collection-speciﬁc key pairs which are arrays of length 2 (en-

cryption key, HMAC key).

• collection: String stating the collection of the record. Currently ﬁxed to “crypto”.

Each key is Base64 encoded.

Example

{"id":"keys",

"collection":"crypto",

"collections":{},

"default:['dGhlc2UtYXJlLWV4YWN0bHktMzItY2hhcmFjdGVycy4=',

'eWV0LWFub3RoZXItc2V0LW9mLTMyLWNoYXJhY3RlcnM=']}

Collection Records

All records in non-special collections have a common payload format.

The payload is deﬁned as the JSON encoding of an object containing the following ﬁelds:

• ciphertext: Base64 of encrypted cleartext for underlying payload.

• IV: Base64 encoding of IV used for encryption.

• hmac: Base64 encoding of HMAC for this message.

Here is an example:

4.3. Sync Client Documentation 165

Mozilla Services Documentation, Release

{

"payload": "{\"ciphertext\":\

˓→"K5JZc7t4R2DzL6nanW+xsJMDhMZkiyRnG3ahpuz61hmFrDZu7DbsYHD77r5Eadlj\",\"IV\":\

˓→"THPKCzWVX35\\/5123ho6mJQ==\",\"hmac\":\

˓→"78ecf07c46b12ab71b769532f15977129d5fc0c121ac261bf4dda88b3329f6bd\"}",

"id": "GJN0ojnlXXhU",

"modified": 1332402035.78

}

The format of the unencrypted ciphertext is deﬁned by the collection it resides in. See the Object Formats documen-

tation for speciﬁcs. That being said, the cleartext is almost certainly a JSON string representing an object. This will

be assumed for the examples below.

Encryption

Let’s assume you have the following JSON payload to encrypt:

{

"foo": "supersecret",

"bar": "anothersecret"

}

Now, in pseudo-code:

# collection_name is the name of the collection this record will be inserted

# into. bulk_key_bundle is an object that represents the decrypted

# crypto/keys record. The called function simply extracts the appropriate

# key pair for the specified collection.

key_pair = bulk_key_bundle.getKeyPair(collection_name);

# Just some simple aliasing.

encryption_key = key_pair.encryption_key

hmac_key = key_pair.hmac_key

iv = randomBytes(16)

# cleartext is the example JSON above.

ciphertext = AES256(cleartext, encryption_key, iv)

ciphertext_b64 = Base64Encode(ciphertext)

hmac = HMACSHA256(ciphertext_b64, hmac_key)

payload = {

"ciphertext": ciphertext_b64,

"IV": Base64Encode(iv),

"hmac": Base64Encode(hmac)

}

record.payload = JSONEncode(payload)

Decryption

Decryption is just the opposite of encryption.

Let’s assume we get a record from the server:

166 Chapter 4. Client Development

Mozilla Services Documentation, Release

{

"payload": "{\"ciphertext\":\

˓→"K5JZc7t4R2DzL6nanW+xsJMDhMZkiyRnG3ahpuz61hmFrDZu7DbsYHD77r5Eadlj\",\"IV\":\

˓→"THPKCzWVX35\\/5123ho6mJQ==\",\"hmac\":\

˓→"78ecf07c46b12ab71b769532f15977129d5fc0c121ac261bf4dda88b3329f6bd\"}",

"id": "GJN0ojnlXXhU",

"modified": 1332402035.78

}

To decrypt it:

fields = JSONDecode(record.payload)

# The HMAC is computed over the Base64 version of the ciphertext, so we

# leave the encoding intact for now.

ciphertext_b64 = fields.ciphertext

remote_hmac = Base64Decode(fields.hmac)

iv = Base64Decode(fields.IV)

key_pair = bulk_key_bundle.getKeyPair(collection_name)

encryption_key = key_pair.encryption_key

hmac_key = key_pair.hmac_key

local_hmac = HMACSHA256(ciphertext_b64, hmac_key)

if local_hmac != remote_hmac:

throw Error("HMAC verification failed.")

ciphertext = Base64Decode(ciphertext_b64)

cleartext = AESDecrypt(ciphertext, encryption_key, iv)

object = JSONDecode(cleartext)

Global Storage Version 6

Attention: This document is a proposal. It will likely change.

This document describes version 6 of Sync’s global storage format. This describes not only the technical details of the

storage format, but also some semantics for how clients supporting version 6 should interact with the Sync server.

Cryptographic Model

All data on the server (with the exception of a single record containing non-private metadata used to help clients

perform sanity checking) is encrypted on the client using AES symmetric encryption with 256 bit keys. Encrypted

data is secured against tampering by employing HMAC-SHA256 hashing.

The cryptographic model frequently relies on pairs of 256 bit keys. One key is used for AES symmetric encryption;

the other for HMAC veriﬁcation. We refer to such a pair of keys as a Key Bundle.

Data on the server is organized into collections (e.g. history, bookmarks). Every collection has a single Key Bundle

associated with it. We refer to a Key Bundle that is afﬁliated with a collection as a Collection Key Bundle. A single

4.3. Sync Client Documentation 167

Mozilla Services Documentation, Release

Collection Key Bundle is used to perform cryptographic operations on every record in the collection to which it is

associated.

It is recommended, but not technically required, that each Collection Key Bundle be associated with at most a single

collection.

Special records on the server hold mappings of collection names to their respective Collection Key Bundles. The

Collection Key Bundles are encrypted using another higher-level Key Bundle before they are stored on the server.

We refer to these higher-level Key Bundles as Key-Encrypting Key Bundles. And, the entity that holds the mapping

to Collection Key Bundles is referred to as a crypto record (because it is a record stored in the crypto collection).

In the simple case, we have a single Key-Encrypting Key Bundle used to encrypt the collection of all Collection

Key Bundles. Each Collection Key Bundle is used to encrypt every record in the collection to which it is associated.

In other words, we have a master key used to unlock other keys which in turn unlock data on the server.

In graph form:

Key-Encrypting Key Bundle

Bookmarks Collection Key Bundle History Collection Key Bundle

Bookmark A Bookmark B History 1 History 2

This speciﬁcation establishes no rules for which crypto records exist or for how Key-Encrypting Key Bundles are

managed. This is entirely up to the client. In other words, key management is a convention between clients. If you are

interested in interoperating with Firefox Sync, see Mozilla’s Sync Service.

This is the essence of the cryptographic model. More details are explained below.

Representation of Key Pairs

While Key Bundles consist of two separate keys, they should be thought of as a single immutable entity. To enforce

this, a Sync Key Bundle is represented as a single blob of data.

The blob consists of the 256 bits of the encryption key followed by the 256 bits of the HMAC key followed by

optional metadata. The format of the metadata is not currently deﬁned. If no metadata is present, no extra information

is recorded and the Key Bundle is represented as a single 512 bit (64 byte) blob.

In pseudo-code:

key_bundle = encryption_key + hmac_key + metadata

When encoded in JSON, key bundles are Base64 encoded. Note that keys are not stored in plaintext, so the Base64

encoding will apply to the ciphertext. See below.

168 Chapter 4. Client Development

Mozilla Services Documentation, Release

Encrypted Records

Most encrypted records share a common payload format and method for encryption and decryption.

The payload of an encrypted records effectively consists of the following ﬁelds:

• ciphertext: The encrypted version of the underlying data.

• IV: Initialization vector used by AES encryption.

• HMAC: HMAC for the encrypted message.

Since these 3 items are all related and all are needed to decrypt and verify individual messages, they are represented

by a single entity - a buffer containing all 3 ﬁelds concatenated together.

Each binary buffer holds the raw bytes constituting the HMAC signature, followed by the raw bytes of the IV, followed

by the raw bytes of the ciphertext.

In pseudo-code:

data = hmac + iv + ciphertext

The HMAC signature is always the length of the HMAC key. Since Sync uses 256-bit HMAC keys, the HMAC

signature is 256 bits, or 32 bytes.

The IV is ﬁxed-width at 16 bytes.

The ciphertext is variable length.

We refer to this 3-tuple of encryption matter as Encrypted Data.

When represented in JSON, the raw bytes constituting the Encrypted Data are Base64 encoded.

Encryption

Encryption is the process of taking some piece of data, referred to as cleartext, and converting it to Encrypted Data.

We start with a Key Bundle and cleartext.

In pseudo-code:

# collection_name is the name of the collection this record will be inserted

# into. The called function obtains the appropriate key bundle depending on

# the destination collection of the record.

bundle = getBundleForCollection(collection_name)

# Just some aliasing for readability.

encryption_key = bundle.encryption_key

hmac_key = bundle.hmac_key

iv = randomBytes(16)

ciphertext = AES256Encrypt(encryption_key, iv, cleartext)

# Now compute the HMAC. Be sure to include the IV in the computation.

message = iv + ciphertext

hmac = HMACSHA256(hmac_key, message)

encrypted_data = hmac + message

4.3. Sync Client Documentation 169

Mozilla Services Documentation, Release

# When going to JSON, the binary payload buffer is Base64-encoded first.

record.payload = Base64Encode(encrypted_data)

Decryption

Decryption is the process of taking Encryted Data and turning it into cleartext.

Decryption requires Encrypted Data and a Key Bundle.

In pseudo-code:

bundle = getBundleForCollection(collection_name)

encryption_key = bundle.encryption_key

hmac_key = bundle.hmac_key

# If grabbing the record from JSON, it will Base64 encoded.

payload_b64 = record.payload

encrypted_data = Base64Decode(payload_b64)

# HMAC is first 32 bytes of payload.

hmac_remote = encrypted_data[0:31]

# IV is the 16 bytes after the HMAC

iv = encrypted_data[32:47]

# ciphertext is everything that's left.

ciphertext = encrypted_data[48:]

# The first step of decryption is verifying the HMAC. The HMAC is computed

# over both the IV and the ciphertext.

hmac_local = HMACSHA256(hmac_key, iv + ciphertext)

if hmac_local != hmac_remote:

throw new Error("HMAC verification failed!");

cleartext = AESDecrypt(encryption_key, iv, ciphertext)

Global Metadata Record

The meta/global record exists with the same semantics as version 5, the only difference being that the storageVersion

is 6 and the engines key has been renamed to repositories.

TODO carry version 5’s documentation forward.

Example:

{

"syncID": "7vO3Zcdu6V4I",

"storageVersion": 6,

"repositories":{

"clients": {"version":1,"syncID":"Re1DKzUQE2jt"},

"bookmarks": {"version":2,"syncID":"ApPN6v8VY42s"},

"forms": {"version":1,"syncID":"lLnCTaQM3SPR"},

"tabs": {"version":1,"syncID":"G1nU87H-7jdl"},

"history": {"version":1,"syncID":"9Tvy_Vlb44b2"},

"passwords": {"version":1,"syncID":"yfBi2v7PpFO2"},

170 Chapter 4. Client Development

Mozilla Services Documentation, Release

"prefs": {"version":2,"syncID":"8eONx16GXAlp"}

}

crypto Collection

There exists a special collection on the server named crypto. This collection holds records that contain mappings of

collections to Collection Key Bundles.

Each record in the crypto collection has associated with it speciﬁc semantics. This speciﬁcation is intentionally vague

as to what records and semantics are deﬁned, as it is up to clients to deﬁne those. In other words, the set of records

on the server and the speciﬁcs of which Collection Key Bundles they contain and/or which Key Encrypting Key

Bundle is used to secure them is left to the purview of the client.

The rationale for this is that users may wish to manage their Collection Key Bundles with different levels of access or

security. For example, the record containing all the keys may only be decrypted with a highly secure parent key, while

another record may contain keys for less-sensitive collections, which can be unlocked using a key derived from a less

secure method, such as PBKDF2.

Clients should support the ability to intelligently use different sets of Collection Key Bundles, depending on what

the user has provided them access to. This means clients should not be eager to delete collections for which it doesn’t

have the Collection Key Bundle, as the user may have purposefully withheld access to that speciﬁc collection.

Record Format

The exact format of records in this collection has yet to be decided. We have a few options.

Option 1

The payload of every record is an object containing the following ﬁelds:

• collections - (required) A cleartext wrapping of collection names to Encrypted Data. The decrypted values are

Key Bundles used to encrypt the collection to which it is tied.

• encryptingKey - (optional) An encrypted* Key Bundle used to encrypt other encrypted data in this record.

For example:

{

"collections": {

"bookmarks": "ENCRYPTED KEY 0",

"history": "ENCRYPTED KEY 1"

"encryptingKey": "ENCRYPTED KEY-ENCRYPTING KEY"

}

The client would – if not delivered out-of-band – decrypt the encrypting key. This would require its parent key and the

contents of this record.

The client would then take the decrypted key-encrypting key and decrypt the individual Collection Key Bundles.

Pros:

• Simple

Cons:

4.3. Sync Client Documentation 171

Mozilla Services Documentation, Release

• Server data reveals which encrypting keys can be used to unlock which collections.

Option 2

This is similar to Option 1 except that the mapping info is itself encrypted.

For example:

{

"data": "ENCRYPTED DATA",

"encryptingKey": "ENCRYPTED KEY-ENCRYPTING KEY"

}

The decrypted key encrypting key would ﬁrst decrypt the data ﬁeld. This would expose the mapping of collection

names to encrypted Key Bundles, just as in Option 1. From there, the same key-encrypting key would decrypt each

individual Key Bundle.

Yes, the Key Bundles are encrypted with the same key twice. We do not want the Key Bundles unencrypted after the

ﬁrst unwrapping because we want clients to be designed such that they never have to touch unencrypted key matter.

In the case of Firefox, this means Sync can operate in FIPS mode since NSS will be the only entity handling the

unencrypted keys.

Pros:

• Server data does not reveal which keys can unlock which collections

Cons:

• More complicated than version 1

• Double encryption involves extra work.

No encryptingKey Variation

There is a variation of the above options where the encrypted key encrypting key is not stored in the record. Instead, it

is stored in another record on the server or not stored on the server at all. These variations differ only in the speciﬁcs

of the record payload.

Changes Since Version 5

Sync Keys Consolidated

The Sync Key has traditionally been 128 bits (often encoded as 26 “friendly” Base32 characters). The historical

reason for it being 128 bits is that in early versions of Sync (before J-PAKE), people would need to manually enter the

Sync Key to pair other devices. Even with J-PAKE, people may need to manually enter the Sync Key (known as the

Recovery Key in UI parlance) into their client. From the 128-bit Sync Key, two 256-bit keys were derived via HKDF.

With the intent to use BrowserID’s key wrapping facility, we feel Sync no longer has the requirement that the Sync Key

be manageable to enter from UI. This is because your Sync Key will be accessible merely by logging into BrowserID

(your BrowserID credentials will unlock a BrowserID user key and that user key can unwrap an encrypted Sync Key

stored on the server).

(We expect that users not using BrowserID will use some other mechanism for key exchange other than keyboard

entry.)

172 Chapter 4. Client Development

Mozilla Services Documentation, Release

Therefore, in version 6, the Sync Key will consist of a pair of 256-bit keys. Each key will be generated from a

cryptographically secure random number generator and will not be derived from any other source. This effectively

replaces the single 128-bit random key and two 256-bit HKDF-derived keys with two completely random 256-bit keys.

Sync Key Stored on Server

Version 6 supports storing the encrypted Sync Key on the Storage Server.

Key Pair Encoding

In version 5, key pairs (the two 256-bit keys used for symmetric encryption and HMAC veriﬁcation) were represented

in payloads as arrays consisting of two strings, each representing the Base64 encoded version of the key.

In version 6, key pairs are transmitted as a a single string or byte array. The two keys are merely concatenated together

to form one 512-bit data chunk. Version 6 also supports additional metadata after the keys. However, the format of

this metadata is not yet deﬁned.

IV Included in HMAC Hash

In version 6, the IV is included in the HMAC hash. In previous versions, the IV was not included. This change adds

more theoretical security to the veriﬁcation process.

HMAC Performed Over Raw Ciphertext

In version 6, the HMAC is performed over the raw ciphertext bytes. In version 5, HMAC was performed over the

Base64 encoding of the ciphertext.

Representation of Crypto Fields in Records

In version 6, the representation of cryptographic ﬁelds has been hidden from the record payload.

In version 5, the payload of encrypted records was the Base64 encoding of the JSON encoding of an object with the

ﬁelds ciphertext, hmac, and IV.

In version 6, we embed all 3 elements in one opaque ﬁeld. While the client will need to know how to extract the

individual cryptographic components, the transport layer happily deals with a single string of bytes. In the case of

JSON encoding, the payload is now the Base64 representation of the single string, not a JSON string.

Requiring Storing Separate Key Pairs for Collections

Version 6 requires that separate Key Bundles be used for each collection.

The previous version had a default Key Bundle that could be used to decrypt multiple collections. Clients would

look for a collection-speciﬁc key in the crypto/keys record then fall back to the default. In practice, clients (notably

Firefox), did not generate multiple keys by default.

Version 6 is dropping support for the default key and requiring that each collection use a separate key.

This change is being made in an effort to be forward compatible with future data recovery and sharing scenarios.

The requirement of separate keys per collections effectively requires an extra link in the crypto chain where extra

functionality can be inserted for one collection without impacting other collections.

4.3. Sync Client Documentation 173

Mozilla Services Documentation, Release

Metaglobal Record Format Change

The engines key in the metaglobal record has been renamed to repositories. Semantics are preserved.

Firefox object formats

Decrypted data objects are cleartext JSON strings.

Each collection can have its own object structure. This page documents the format of each collection.

The object structure is versioned with the version metadata stored in the meta/global payload.

The following sections, named by the corresponding collection name, describe the various object formats and how

they’re used. Note that object structures may change in the future and may not be backwards compatible.

In addition to these custom collection object structures, the Encrypted DataObject adds ﬁelds like id and deleted. Also,

remember that there is data at the Weave Basic Object (WBO) level as well as id, modiﬁed, sortindex and payload.

Add-ons

Version 1

Version 1 is likely only afﬁliated with storage format 5 clients.

• addonID string: Public identiﬁer of add-on. This is the id attribute from an Addon object obtained from the

AddonManager.

• applicationID string: The application ID the add-on belongs to.

• enabled bool: Indicates whether the add-on is enabled or disabled. true means enabled.

• source string: Where the add-on came from. amo means it came from addons.mozilla.org or a trusted site.

Bookmarks

Version 1

One bookmark record exists for each bookmark item, where an item may actually be a folder or a separator. Each item

will have a type that determines what other ﬁelds are available in the object. The following sections describe the object

format for a given type.

Each bookmark item has a parentid and predecessorid to form a structure like a tree of linked-lists to provide a

hierarchical ordered list of bookmarks, folders, etc.

bookmark

This describes a regular bookmark that users can click to view a page.

• title string: name of the bookmark

• bmkUri string uri of the page to load

• description string: extra description if provided

• loadInSidebar boolean: true if the bookmark should load in the sidebar

• tags array of strings: tags for the bookmark

174 Chapter 4. Client Development

Mozilla Services Documentation, Release

• keyword string: alias to activate the bookmark from the location bar

• parentid string: GUID of the containing folder

• parentName string: name of the containing folder

• predecessorid string: GUID of the item before this (empty if it’s ﬁrst)

• type string: “bookmark”

microsummary

Microsummaries allow pages to be summarized for viewing from the toolbar. This extends bookmark, so the usual

bookmark ﬁelds apply.

• generatorUri string: uri that generates the summary

• staticTitle string: title to show when no summaries are available

• title string: name of the microsummary

• bmkUri string uri of the page to load

• description string: extra description if provided

• loadInSidebar boolean: true if the bookmark should load in the sidebar

• tags array of strings: tags for the bookmark

• keyword string: alias to activate the bookmark from the location bar

• parentid string: GUID of the containing folder

• parentName string: name of the containing folder

• predecessorid string: GUID of the item before this (empty if it’s ﬁrst)

• type string: “microsummary”

query

Place queries are special bookmarks with a place: uri that links to an existing folder/tag. This extends bookmark, so

the usual bookmark ﬁelds apply.

• folderName string: name of the folder/tag to link to

• queryId string (optional): identiﬁer of the smart bookmark query

• title string: name of the query

• bmkUri string place: uri query

• description string: extra description if provided

• loadInSidebar boolean: true if the bookmark should load in the sidebar

• tags array of strings: tags for the query

• keyword string: alias to activate the bookmark from the location bar

• parentid string: GUID of the containing folder

• parentName string: name of the containing folder

• predecessorid string: GUID of the item before this (empty if it’s ﬁrst)

4.3. Sync Client Documentation 175

Mozilla Services Documentation, Release

• type string: “query”

folder

Folders contain bookmark items like bookmarks and other folders.

• title string: name of the folder

• parentid string: GUID of the containing folder

• parentName string: name of the containing folder

• predecessorid string: GUID of the item before this (empty if it’s ﬁrst)

• type string: “folder”

livemark

Livemarks act like folders with a dynamic list bookmarks, e.g., a RSS feed. This extends folder, so the usual folder

ﬁelds apply.

• siteUri string: site associated with the livemark

• feedUri string: feed to get items for the livemark

• title string: name of the livemark

• parentid string: GUID of the containing folder

• parentName string: name of the containing folder

• predecessorid string: GUID of the item before this (empty if it’s ﬁrst)

• type string: “livemark”

separator

Separators help split sections of a folder.

• pos string: position (index) of the separator

• parentid string: GUID of the containing folder

• parentName string: name of the containing folder

• predecessorid string: GUID of the item before this (empty if it’s ﬁrst)

• type string: “separator”

Version 2

Same as engine version 1, except:

• the predecessorid is removed from all records,

• instead folder and livemark records have a children attribute which is an array of child GUIDs in order of their

appearance in the folder:

• children array of strings: ordered list of child GUIDs

176 Chapter 4. Client Development

Mozilla Services Documentation, Release

• the special folders ‘menu’ and ‘toolbar’ now have records that are synced, purely to maintain order within them

according to their ‘’‘children’‘’ array.

Version 3

Note: Proposal corresponding with storage format 6.

Same as version 2 except:

• Support for microsummaries is removed

• We use the ASCII URL

TODO document full format here since diffs are inconvenient to read.

Clients

Version 1

Client records identify a user’s one or multiple clients that are accessing the data. The existence of client records can

change the behavior of the Firefox Sync client – multiple clients and/or mobile clients result in syncs to happen more

frequently.

• name string: name of the client connecting

• type string: type of the client: “desktop” or “mobile”

• commands array: commands to be executed upon next sync

In Protocol 1.5, client records additionally include:

• version string: a version indicator for this client, such as “29.0a1”. Optional.

• protocols array: an array of Sync protocol versions supported by this client, such as [”1.1”, “1.5”]. Optional.

In Bug 1097222 additional optional ﬁelds were added:

• os string: an OS name, most likely one of “Darwin” (Mac OS X), “WINNT” (Windows), “Android”, or “iOS”.

• appPackage string: an unambiguous identiﬁer for the client application. For Android, this is the package (e.g.,

org.mozilla.ﬁrefox_beta). For desktop this is the value of Services.appinfo.ID.

• application string: a human-readable application name, such as “Nightly” or “Firefox”.

• formfactor string: a value such as “phone”, “tablet” (or the more speciﬁc “largetablet”, “smalltablet”), “desk-

top”, “laptop”, “tv”.

• device string: a description of the hardware that this client uses. Currently only supported by Android; returns

values like “HTC One”.

If these ﬁelds are missing, clients are expected to fall back to behaviors that do not depend on the missing data.

Clients should preserve existing ﬁelds if possible when sending commands to another client.

4.3. Sync Client Documentation 177

Mozilla Services Documentation, Release

Version 2 (never deployed)

Note: Proposal corresponding with storage format 6.

Each client has its own record which it is authoritative for. No other client should modify another client’s record except

in the case where records are deleted.

The payload of a client record has the following ﬁelds:

• name string: The name of the client. This is a user-facing value and may be provided by the user.

• formfactor string: The form factor of the client. Recognized values include phone, tablet, laptop, desktop.

• application string: String identifying the application behind the client. This should only be used for presentation

purposes (e.g. choosing what logo to display).

• version string: The version of the client. This is typically the version of the application. Again, this should only

be used for presentation purposes.

• capabilities object: Denotes the capabilities a client possesses. Keys are string capability names. Values are

booleans indicating whether the capability is enabled. Modifying the capabilities of another client’s record

should not change the enabled state on that client.

• mpEnabled bool: Whether master password is enabled on the client. If master password is enabled on any

client in an account, the current client should hesitate before downloading passwords if master password is not

enabled locally, as this would decrease the security of the passwords locally since they wouldn’t be protected

with a master password.

Commands

Version 1

Note: Proposal corresponding with storage format 6.

This collection holds commands for clients to process. The ID of command records is randomly generated.

Command records contain an extra unencrypted ﬁeld in the BSO that says which client ID they belong to. The value

is the hash of the client ID with the commands engine salt.

Command data is an object with the following ﬁelds:

• receiverID string: Client ID of the client that should receive the command. This is duplicated inside the payload

so it can be veriﬁed by the HMAC.

• senderID string: Client ID of the client that sent the command.

• created number: Integer seconds since Unix epoch that command was created.

• action string: The action to be performed by the command. Each command has its own name that uniquely

identiﬁes it. It is recommended that actions be namespaced using colon-delimited notation. Sync’s commands

are all preﬁxed with sync:. e.g. sync:wipe. If a command is versioned, the name is the appropriate place to

convey that versioning.

• data object: Additional data associated with command. This is dependent on the speciﬁc command type being

issued.

178 Chapter 4. Client Development

Mozilla Services Documentation, Release

Forms

Form data is used to give suggestions for autocomplete for a HTML text input form. One record is created for each

form entry.

• name string: name of the HTML input ﬁeld

• value string: value to suggest for the input

History

Version 1

Every page a user visits generates a history item/page. One history (page) per record.

• histUri string: uri of the page

• title string: title of the page

• visits array of objects: a number of how and when the page was visited

• date integer: datetime of the visit

• type integer: transition type of the visit

Version 2 (never deployed)

Note: Proposal corresponding with storage format 6.

History visits are now stored as a timeline/stream of visits. The historical information for a particular site/URL is

spread out of N>=1 records.

Payloads have the structure:

{

"items": [

"uri": "http://www.mozilla.org/",

"title": "Mozilla",

"visits": {

1: [1340757179.82, 184],

2: [1340341244.31, 12, 4]

}

]

}

The bulk of the payload is a list of history items. Each item is both a place and a set of visits.

• uri string: URI of the page that was visited.

• title string: Title of the page that was visited.

• visits object: Mapping of visit type to visit times.

The keys in visits deﬁne the transition type for the visit. They can be one of the following:

• 1: A link was followed.

• 2: The URL was typed by the user.

4.3. Sync Client Documentation 179

Mozilla Services Documentation, Release

• 3: The user followed a bookmark.

• 4: Some inner content was loaded.

• 5: A permanent redirect was followed.

• 6: A temporary redirect was followed.

• 7: The URL was downloaded.

• 8: User follows a link that was in a frame.

These correspond to nsINavHistoryService’s transition type constants <https://developer.mozilla.org/en/nsINavHistoryService#Constants>.

The values for each visit type are arrays which encode the visit time. The initial array element is the wall time of the

ﬁrst visit being recorded in seconds since epoch, typically with millisecond resolution. Each subsequent value is the

number of seconds elapsed since the previous visit. The values:

[100000000.000, 10.100, 5.200]

Correspond to the times:

100000000.000

100000010.100

100000015.300

The use of deltas to represent times is to minimize the serialized size of visits.

Passwords

Saved passwords help users get back into websites that require a login such as HTML input/password ﬁelds or HTTP

auth.

• hostname string: hostname that password is applicable at

• formSubmitURL string: submission url (GET/POST url set by <form>)

• httpRealm string: the HTTP Realm for which the login is valid. if not provided by the server, the value is the

same as hostname

• username string: username to log in as

• password string: password for the username

• usernameField string: HTML ﬁeld name of the username

• passwordField string: HTML ﬁeld name of the password

If possible, clients should also write ﬁelds corresponding to nsILoginMetaInfo:

• timeLastUsed unsigned long: local Unix timestamp in milliseconds at which this password was last used. Note

that client clocks can be wrong, and thus this time can be dramatically earlier or later than the modiﬁed time of

the record. Consuming clients should be careful to handle out of range values.

• timeCreated unsigned long: as with timeLastUsed, but for creation.

• timePasswordChanged unsigned long: as with timeLastUsed, but for password change.

• timesUsed unsigned long: the number of uses of this password.

Note that these ﬁelds are not required as part of the record, so clients should expect them to be missing. Clients

that don’t use this data locally are encouraged to pass through when it makes sense (timeCreated), or wipe when

invalidation is the best option (e.g., timePasswordChanged).

180 Chapter 4. Client Development

Mozilla Services Documentation, Release

Note also that clients should use judgment when updating these ﬁelds. It is typically not feasible to upload new

records each time a password is used on the web. Neither would it make sense during download to treat a non-

matching timestamp, or a missing ﬁeld in an otherwise matching local record, as a record collision. The handling of

these ﬁelds introduces new complexities in reconciliation.

The Firefox desktop client began recording this data in Bug 555755.

Preferences

Version 1

Some preferences used by Firefox will be synced to other clients. There is only one record for preferences with a

GUID “preferences”.

• value array of objects: each object describes a preference entry

• name string: full name of the preference

• type string: the type of preference (int, string, boolean)

• value depends on type: value of the preference

Version 2

There is only one record for preferences, using nsIXULAppInfo.ID as the GUID. Custom preferences can be synced

by following these instructions.

• value object containing name and value of the preferences.

Note: The preferences that determine which preferences are synced are now included as well.

Tabs

Version 1

Tabs describe the opened tabs on a given client to provide functionality like get-up-n-go. Each client will provide one

record.

• clientName string: name of the client providing these tabs

• tabs array of objects: each object describes a tab

• title string: title of the current page

• urlHistory array of strings: page urls in the tab’s history

• icon string: favicon uri of the tab

• lastUsed string or integer: string representation of Unix epoch (in seconds) at which the tab was last accessed.

Or the integer 0. Your code should accept either. This is ghastly; we apologize.

Version 2

Note: Proposal corresponding with storage format 6.

4.3. Sync Client Documentation 181

Mozilla Services Documentation, Release

In version 2, each tab is represented by its own record. (This is a change from version 1.)

The payload of the BSO is a JSON object containing the following ﬁelds:

• clientID string: ID of the client this tab originated on.

• title string: Title of page that is active in the tab.

• history array of strings: URLs in this tab’s history. Initial element is the current URL. Subsequent URLs were

previously visited URLs.

• lastUsed number: Time in seconds since Unix epoch that tab was last active.

• icon string: Base64 encoded favicon image.

• groupName string: Name of tab group this tab is associated with. This is usually used for presentation purposes

and is typically the same string across all records in a particular tab group.

Mozilla’s Sync Service

This document describes how Sync is deployed at Mozilla.

Attention: This document is not complete. Consult the Services team for authoritative information.

Architecture

Mozilla’s Sync service is comprised of the following services:

• Storage Service 1.1

• User Registration Service

• Secure-Registration Service

• Easy Setup Key Exchange Service

Mozilla operates many instances of the Storage Service. We call these nodes. Each node is independent from the

others and has no knowledge that other nodes exist.

When a new account is provisioned, the Registration Service assigns a user to a node. The node is chosen based

on which nodes have capacity, etc. After node assignment, clients connect directly to that speciﬁc node. All Sync

operations are performed against that client’s assigned Sync node.

The user registration service is hosted on https://auth.services.mozilla.com/. If you download Firefox from Mozilla

and set up Sync, this is where it will connect by default.

Easy Setup Service

Mozilla hosts an instance of the Easy Setup service at https://setup.services.mozilla.com/. When you pair two devices

by entering codes, they communicate through this service.

Crypto Record Semantics

Storage Format 6 does not explicitly deﬁne semantics for how crypto records are managed, leaving it up to the clients

to agree on behavior. This section documents the behavior in Mozilla clients.

182 Chapter 4. Client Development

Mozilla Services Documentation, Release

Clients and Key Management

Sync clients can differ in their abilities to manage keys and their associated crypto records on the server. There exist 2

tiers of clients:

• Tier 1 Client - Supports key generation and management.

• Tier 2 Client - Supports key consumption only.

Tier 1 clients are full Sync clients. They can provision accounts from empty servers, reset server data, change keys

around, etc. Tier 2 clients are simpler clients that only support reading of keys (no writing).

The main reason why different tiers of clients exist is that cryptography, security, and the management of keys is hard

and that these problems should be left to professionals. It is extremely easy for a client to introduce subtle differences

that could compromise the integrity of data security. By providing a facility for clients that don’t modify keys, we are

reducing the surface area on which a new client may error as well as decreasing the number of clients which need to

be validated for proper behavior.

Mozilla provides the following Tier 1 Clients:

• Firefox (on desktop)

• Firefox (on mobile - aka Fennec)

Tier 2 Client Behavior

Tier 2 clients never perform updates to the crypto collection. Instead, they read records and get the data they need. If

the data they need is unavailable (i.e. the record it wants isn’t found), it gives up and tries again later.

Tier 2 clients do support creating new collections on the server. When a Tier 2 client wishes to create a new collection,

it will need to use a Collection Key Bundle for that collection. Normally, a new Collection Key Bundle would be

created and uploaded. However, since Tier 2 clients must not modify the crypto collection, they resort to other means.

4.3. Sync Client Documentation 183

Mozilla Services Documentation, Release

184 Chapter 4. Client Development

185

Mozilla Services Documentation, Release

CHAPTER 5

Miscellaneous

Overview of the services

186 Chapter 5. Miscellaneous

Mozilla Services Documentation, Release

• 1. Firefox Sync registers a new user using the Registration server.

• 2. Firefox Sync or Firefox Home interacts with a Storage Service server.

• 3. & 4) Firefox Sync uses Easy Setup to transfer a user proﬁle.

• 5. Internally, Mozilla Corp uses Secure-Registration (Mozilla speciﬁc) to defer writes to a secured server.

Glossary

Auth Token Used to identify the user after starting a session. Contains the user application id and the expiration

date.

Cluster Group of webheads and storage devices that make up a set of Service Nodes.

Colo Physical datacenter, may contain multiple clusters

Generation Number An integer that may be included in a BrowserID identity certiﬁcate. The issuing server in-

creases this value whenever the user changes their password. By rejecting BrowserID assertions with a gen-

eration number lower than the previously-seen maximum for that user, the Login Server can reject assertions

generated using an old password.

Hawk Auth An HTTP authentication method using a message authentication code (MAC) algorithm to provide

cryptographic veriﬁcation of portions of HTTP requests.

See https://github.com/hueniverse/hawk/

HKDF HMAC-based Key Derivation Function, a method for deriving multiple secret keys from a single master

secret.

See https://tools.ietf.org/html/rfc5869

Master Secret A secret shared between Login Server and Service Node. Never used directly, only for deriving other

secrets.

Node An URL that identiﬁes a service, like http://phx345

Node Assignment Server A service that can attribute to a user a node.

Service A service Mozilla provides, like Sync or Easy Setup.

Service Node a server that contains the service, and can be mapped to several Nodes (URLs)

Signing Secret Derived from the master secret, used to sign the auth token.

Token Secret Derived from the master secret and auth token, used as secret. This is the only secret shared with the

client and is different for each auth token.

User DB A database that keeps the user/node relation

Response codes

These are the error response codes used by the various services.

5.2. Glossary 187

Mozilla Services Documentation, Release

Server-produced response codes

Code Description

1 Illegal method/protocol

2 Incorrect/missing CAPTCHA

3 Invalid/missing username

4 Attempt to overwrite data that can’t be overwritten (such as creating a user ID that already exists)

5 User ID does not match account in path

6 JSON parse failure

7 Missing password ﬁeld

8 Invalid Weave Basic Object

9 Requested password not strong enough

10 Invalid/missing password reset code

11 Unsupported function

12 No email address on ﬁle

13 Invalid collection

14 (1.1 and up) User over quota

15 The email does not match the username

16 Client upgrade required

17 Size limit exceeded

Infrastructure-produced response codes

These response codes are generated by the Mozilla Services infrastructure, particularly load balancers. They will not

occur in self-hosting scenarios.

If you observe these values in a 503 response in Sync logs, please contact Services Operations.

These codes are temporarily a mixture of strings and numeric values. This inconsistency will be resolved at a future

date.

Code Description

“server issue: pool exhausted” An unexpected server error occurred: pool is empty.

“server issue: getVS failed”

“server issue: preﬁx not set”

“server issue: host header not received from client”

“server issue: database lookup failed”

“server issue: database is not healthy”

“server issue: database not in pool”

“server issue: database marked as down”

Term of Services

By accessing or using the Firefox Sync APIs in connection with the development of your own client software to access

the Firefox Sync services (a “Third Party Client”), you acknowledge that you will need to install and use a local version

of the Firefox Sync server for multiple account testing and that any use of Mozilla’s hosted Firefox Sync services is

subject to Mozilla’s Firefox Sync Terms of Service at http://docs.services.mozilla.com/tos

Further, you agree (a) to maintain and link to (including on websites from which your Third Party Client may be down-

loaded) a separate, conspicuous, and reasonably detailed privacy policy detailing how data collected or transmitted by

your Third Party Client is managed and protected; (b) that your Third Party Client will only store data in encrypted

form on the Firefox Sync servers operated by Mozilla; (c) that you and your Third Party Client will use the Firefox

188 Chapter 5. Miscellaneous

Mozilla Services Documentation, Release

Sync APIs solely for their intended purpose; (d) that your Third Party Client will not hide or mask its identity as it

uses the Services and/or Firefox Sync APIs, including by failing to follow required identiﬁcation conventions; and (e)

that you and your Third Party Client will not use the Firefox Sync APIs for any application or service that replicates

or attempts to replicate the Services or Firefox Sync experience unless such use is non-confusing (by non-confusing,

we mean that people should always know with whom they are dealing and where the information or software they are

downloading came from).

You may not imply, either directly or by omission, that your Third Party Client is produced or endorsed by Mozilla.

By providing access to the Firefox Sync APIs, Mozilla is not granting you a license to any of our trademarks.

– The Services Team

About this Website

This website is created using Sphinx.

The source of this website is under version control at https://github.com/mozilla-services/docs. (It also lives in a

Mercurial repository at https://hg.mozilla.org/services/docs. However, the GitHub repository is favored as it is simpler

for non-Mozilla people to contribute.)

If you want to change the content of this website, changes will need to be made to the master branch of the aforemen-

tioned Git repository.

This can be done one of several ways:

• Fork the repository on GitHub and create a pull request.

• Send a patch to the [email protected]g mailing list.

• Create a Bugzilla issue at https://bugzilla.mozilla.org/ under the Mozilla Services product for the component

the docs impact.

Generating Documentation

To generate the docs from source, you’ll need to obtain Sphinx along with some extensions.

Assuming you are using Virtualenv:

$ virtualenv sphinx-env

$ source sphinx-env/bin/activate

# You are now in the fresh virtualenv for Sphinx.

# Install dependencies.

$ pip install sphinx sphinxcontrib-seqdiag mozilla_sphinx_theme

# Build HTML docs.

$ make html

By default, the Makeﬁle looks for sphinx-build in your PATH. If you have sphinx-build elsewhere, just pass the path

to the Makeﬁle:

$ make html SPHINXBUILD=/path/to/sphinx-build

Directory Structure

The source is located in the source directory and contains:

5.5. About this Website 189

Mozilla Services Documentation, Release

• howtos: a directory with How Tos

• server-devguide: server development guide, guidelines, how to release a server app etc.

• one directory per Server application (reg, sreg, etc.)

Update process

The website is located in a svn repository which gets regular snapshots of the HTML structure generated by Sphinx.

This should run automatically (i.e. changes checked into GitHub should automatically propagate to the ofﬁcial web

site). If this does not happen, please send a message to the [email protected]g mailing list and request an

update.

190 Chapter 5. Miscellaneous

Index

Auth Token, 187

Basic Authentication, 149

Cluster, 187

Colo, 187

Generation Number, 187

Hawk Auth, 187

HKDF, 187

Master Secret, 187

Node, 187

Node Assignment Server, 187

Service, 187

Service Node, 187

services-builds, 149

services-dev, 149

Signing Secret, 187

Token Secret, 187

User DB, 187

WSGI, 149

191