REST
It's not just for servers
Mark Lavin - OSCon 2014
Introduction
- Technical Director at Caktus Group in Carborro, NC
- Co-Author of "Lightweight Django" for O'Reilly
- Homebrewer
- Runner/Triathelete
History
REpresentational State Transfer.
Defined by Roy Fielding's doctoral thesis at UC Irvine in 2000.
Abused by webservices and marketing teams every day since.
REST is ...
“an architectural style for distributed hypermedia systems. REST provides a set of architectural constraints that, when applied as a whole, emphasizes scalability of component interactions, generality of interfaces, independent deployment of components, and intermediary components to reduce interaction latency, enforce security, and encapsulate legacy systems.” - Roy Fielding
Constraints
What are these constraints?
- Client-Server
- Stateless
- Cacheable
- Layered
- Uniform Interface
- Code On Demand*
Rewards
What do you get in return?
- Performance
- Scalability
- Simplicity
- Modifiability
Reach/Popularity
Name a popular website/application and it probably has a REST* API.
- Social Media: Twitter, Facebook, LinkedIn, Pintrest, Instragram
- Open Source: Github, Bitbucket
- Cloud: AWS, Rackspace, OpenStack, Digital Ocean
- PAAS: Heroku, DotCloud, OpenShift, CloudFoundry
- Cloud Storage: S3, Cloudfiles, Dropbox, Box
- Payment Processors: Paypal, Stripe, CoinBase
- Personal Fitness: Fitbit, Strava
Public APIs ❤ Open Source Clients
Servers
That's another talk.
Clients
How do you talk to one of these services?
It's just HTTP...how hard can it be?
Challenges
Writing clients is not hard.
Maintaining them is hard.
Why?
- Changing APIs
- Faux-RESTful Servers
- Weak HTTP Support
- Managing State
Changing APIs
APIs are going to change.
You won't always know the changes coming.
Sorry.
Example: Twitter
https://api.twitter.com/1/account/verify_credentials.json
is now
https://api.twitter.com/1.1/account/verify_credentials.json
Faux-RESTful Servers
The Uniform Interface contraint defines the concept of Hypermedia As The Engine Of Application State (HATEOAS).
Sadly this is rarely implemented.
HATEOAS
"REST APIs must be hypertext-driven"
“A REST API must not define fixed resource names or hierarchies (an obvious coupling of client and server). Servers must have the freedom to control their own namespace. Instead, allow servers to instruct clients on how to construct appropriate URIs, such as is done in HTML forms and URI templates, by defining those instructions within media types and link relations.” - Roy Fielding
HATEOAS TL;DR
Clients should not have to build resource URLs.
Resource URLs should be provided by the server so that the server can control them.
Example: Bitbucket v1
https://bitbucket.org/api/1.0/users/mlavin
{
"user":{
"username":"mlavin",
"first_name":"Mark",
"last_name":"Lavin",
"display_name":"Mark Lavin",
"is_staff":false,
"avatar":"https://secure.gravatar.com/avatar/de77fafcd2dd1e7d5a159793ea51b563?d=https%3A%2F%2Fd3oaxc4q5k2d6q.cloudfront.net%2Fm%2F069454acc352%2Fimg%2Fdefault_avatar%2F32%2Fuser_blue.png&s=32",
"resource_uri":"/1.0/users/mlavin",
"is_team":false
},
"repositories":[
{
"scm":"hg",
"has_wiki":false,
"last_updated":"2012-06-26T05:32:29.490",
"no_forks":null,
"created_on":"2011-03-22T00:06:37.631",
"owner":"mlavin",
"logo":"https://d3oaxc4q5k2d6q.cloudfront.net/m/069454acc352/img/language-avatars/python_16.png",
"email_mailinglist":"",
"is_mq":false,
"size":30648,
"read_only":false,
"fork_of":null,
"mq_of":null,
"state":"available",
"utc_created_on":"2011-03-21 23:06:37+00:00",
"website":"",
"description":"A basic photo upload/gallery application for Django.",
"has_issues":true,
"is_fork":false,
"slug":"django-photobook",
"is_private":false,
"name":"django-photobook",
"language":"python",
"utc_last_updated":"2012-06-26 03:32:29+00:00",
"email_writers":true,
"no_public_forks":false,
"creator":null,
"resource_uri":"/1.0/repositories/mlavin/django-photobook"
},
{
"scm":"hg",
"has_wiki":false,
"last_updated":"2012-06-26T08:06:30.579",
"no_forks":null,
"created_on":"2011-01-19T03:01:06.792",
"owner":"mlavin",
"logo":"https://d3oaxc4q5k2d6q.cloudfront.net/m/069454acc352/img/language-avatars/python_16.png",
"email_mailinglist":"",
"is_mq":false,
"size":606374,
"read_only":false,
"fork_of":null,
"mq_of":null,
"state":"available",
"utc_created_on":"2011-01-19 02:01:06+00:00",
"website":"http://www4.ncsu.edu/~kaltofen/courses/Languages/Spring11/syllabus.html",
"description":"Lecture slides for MA792K",
"has_issues":false,
"is_fork":false,
"slug":"python-lectures",
"is_private":false,
"name":"python-lectures",
"language":"python",
"utc_last_updated":"2012-06-26 06:06:30+00:00",
"email_writers":true,
"no_public_forks":false,
"creator":null,
"resource_uri":"/1.0/repositories/mlavin/python-lectures"
},
{
"scm":"hg",
"has_wiki":true,
"last_updated":"2012-07-03T04:03:09.870",
"no_forks":null,
"created_on":"2010-01-10T18:07:46.612",
"owner":"mlavin",
"logo":"https://d3oaxc4q5k2d6q.cloudfront.net/m/069454acc352/img/language-avatars/default_16.png",
"email_mailinglist":"",
"is_mq":false,
"size":810046,
"read_only":false,
"fork_of":null,
"mq_of":null,
"state":"available",
"utc_created_on":"2010-01-10 17:07:46+00:00",
"website":"http://bethandmark.net/",
"description":"This is the source a personal site for me and my wife.",
"has_issues":true,
"is_fork":false,
"slug":"personal",
"is_private":false,
"name":"personal",
"language":"",
"utc_last_updated":"2012-07-03 02:03:09+00:00",
"email_writers":true,
"no_public_forks":false,
"creator":null,
"resource_uri":"/1.0/repositories/mlavin/personal"
},
{
"scm":"hg",
"has_wiki":true,
"last_updated":"2011-09-17T03:03:44.627",
"no_forks":null,
"created_on":"2011-01-26T17:36:30.691",
"owner":"mlavin",
"logo":"https://d3oaxc4q5k2d6q.cloudfront.net/m/069454acc352/img/language-avatars/default_16.png",
"email_mailinglist":"",
"is_mq":false,
"size":12854,
"read_only":false,
"fork_of":{
"scm":"hg",
"has_wiki":true,
"last_updated":"2011-09-17T03:03:44.647",
"no_forks":null,
"created_on":"2010-06-21T21:23:35.033",
"owner":"copelco",
"logo":"https://d3oaxc4q5k2d6q.cloudfront.net/m/069454acc352/img/language-avatars/default_16.png",
"email_mailinglist":"",
"is_mq":false,
"size":14264,
"read_only":false,
"creator":null,
"state":"available",
"utc_created_on":"2010-06-21 19:23:35+00:00",
"website":"",
"description":"Simple Django app to model committees and members",
"has_issues":true,
"is_fork":false,
"slug":"django-committees",
"is_private":false,
"name":"django-committees",
"language":"",
"utc_last_updated":"2011-09-17 01:03:44+00:00",
"email_writers":true,
"no_public_forks":false,
"resource_uri":"/1.0/repositories/copelco/django-committees"
},
"mq_of":{
"scm":"hg",
"has_wiki":true,
"last_updated":"2011-09-17T03:03:44.647",
"no_forks":null,
"created_on":"2010-06-21T21:23:35.033",
"owner":"copelco",
"logo":"https://d3oaxc4q5k2d6q.cloudfront.net/m/069454acc352/img/language-avatars/default_16.png",
"email_mailinglist":"",
"is_mq":false,
"size":14264,
"read_only":false,
"creator":null,
"state":"available",
"utc_created_on":"2010-06-21 19:23:35+00:00",
"website":"",
"description":"Simple Django app to model committees and members",
"has_issues":true,
"is_fork":false,
"slug":"django-committees",
"is_private":false,
"name":"django-committees",
"language":"",
"utc_last_updated":"2011-09-17 01:03:44+00:00",
"email_writers":true,
"no_public_forks":false,
"resource_uri":"/1.0/repositories/copelco/django-committees"
},
"state":"available",
"utc_created_on":"2011-01-26 16:36:30+00:00",
"website":null,
"description":"",
"has_issues":true,
"is_fork":true,
"slug":"django-committees",
"is_private":false,
"name":"django-committees",
"language":"",
"utc_last_updated":"2011-09-17 01:03:44+00:00",
"email_writers":true,
"no_public_forks":false,
"creator":null,
"resource_uri":"/1.0/repositories/mlavin/django-committees"
},
{
"scm":"hg",
"has_wiki":true,
"last_updated":"2014-02-06T23:03:47.962",
"no_forks":false,
"created_on":"2010-12-12T17:34:14.390",
"owner":"mlavin",
"logo":"https://d3oaxc4q5k2d6q.cloudfront.net/m/069454acc352/img/language-avatars/python_16.png",
"email_mailinglist":"",
"is_mq":false,
"size":630785,
"read_only":false,
"fork_of":null,
"mq_of":null,
"state":"available",
"utc_created_on":"2010-12-12 16:34:14+00:00",
"website":"http://django-selectable.readthedocs.org/",
"description":"The development of django-selectable has moved to Github: https://github.com/mlavin/django-selectable",
"has_issues":false,
"is_fork":false,
"slug":"django-selectable",
"is_private":false,
"name":"django-selectable",
"language":"python",
"utc_last_updated":"2014-02-06 22:03:47+00:00",
"email_writers":true,
"no_public_forks":false,
"creator":null,
"resource_uri":"/1.0/repositories/mlavin/django-selectable"
},
{
"scm":"hg",
"has_wiki":false,
"last_updated":"2014-01-14T20:01:18.504",
"no_forks":false,
"created_on":"2014-01-14T20:01:18.262",
"owner":"mlavin",
"logo":"https://d3oaxc4q5k2d6q.cloudfront.net/m/069454acc352/img/language-avatars/python_16.png",
"email_mailinglist":"",
"is_mq":false,
"size":982779,
"read_only":false,
"fork_of":{
"scm":"hg",
"has_wiki":false,
"last_updated":"2014-07-08T14:50:52.317",
"no_forks":null,
"created_on":"2012-04-24T17:16:27.952",
"owner":"Manfre",
"logo":"https://d3oaxc4q5k2d6q.cloudfront.net/m/069454acc352/img/language-avatars/python_16.png",
"email_mailinglist":"\[email protected]",
"is_mq":false,
"size":1049967,
"read_only":false,
"creator":null,
"state":"available",
"utc_created_on":"2012-04-24 15:16:27+00:00",
"website":"",
"description":"Microsoft SQL server backend for Django running on windows.",
"has_issues":true,
"is_fork":false,
"slug":"django-mssql",
"is_private":false,
"name":"django-mssql",
"language":"python",
"utc_last_updated":"2014-07-08 12:50:52+00:00",
"email_writers":true,
"no_public_forks":false,
"resource_uri":"/1.0/repositories/Manfre/django-mssql"
},
"mq_of":{
"scm":"hg",
"has_wiki":false,
"last_updated":"2014-07-08T14:50:52.317",
"no_forks":null,
"created_on":"2012-04-24T17:16:27.952",
"owner":"Manfre",
"logo":"https://d3oaxc4q5k2d6q.cloudfront.net/m/069454acc352/img/language-avatars/python_16.png",
"email_mailinglist":"\[email protected]",
"is_mq":false,
"size":1049967,
"read_only":false,
"creator":null,
"state":"available",
"utc_created_on":"2012-04-24 15:16:27+00:00",
"website":"",
"description":"Microsoft SQL server backend for Django running on windows.",
"has_issues":true,
"is_fork":false,
"slug":"django-mssql",
"is_private":false,
"name":"django-mssql",
"language":"python",
"utc_last_updated":"2014-07-08 12:50:52+00:00",
"email_writers":true,
"no_public_forks":false,
"resource_uri":"/1.0/repositories/Manfre/django-mssql"
},
"state":"available",
"utc_created_on":"2014-01-14 19:01:18+00:00",
"website":null,
"description":"Working copy of django-mssql for contributing fixes.",
"has_issues":false,
"is_fork":true,
"slug":"django-mssql",
"is_private":false,
"name":"django-mssql",
"language":"python",
"utc_last_updated":"2014-01-14 19:01:18+00:00",
"email_writers":true,
"no_public_forks":false,
"creator":null,
"resource_uri":"/1.0/repositories/mlavin/django-mssql"
}
]
}
Example: Bitbucket v2
https://bitbucket.org/api/2.0/users/mlavin
{
"username":"mlavin",
"website":"",
"display_name":"Mark Lavin",
"links":{
"self":{
"href":"https://bitbucket.org/!api/2.0/users/mlavin"
},
"repositories":{
"href":"https://bitbucket.org/!api/2.0/repositories/mlavin"
},
"html":{
"href":"https://bitbucket.org/mlavin"
},
"followers":{
"href":"https://bitbucket.org/!api/2.0/users/mlavin/followers"
},
"avatar":{
"href":"https://secure.gravatar.com/avatar/de77fafcd2dd1e7d5a159793ea51b563?d=https%3A%2F%2Fd3oaxc4q5k2d6q.cloudfront.net%2Fm%2F069454acc352%2Fimg%2Fdefault_avatar%2F32%2Fuser_blue.png&s=32"
},
"following":{
"href":"https://bitbucket.org/!api/2.0/users/mlavin/following"
}
},
"created_on":"2009-07-06T14:39:20.563539+00:00",
"location":"Raleigh, North Carolina",
"type":"user"
}
Weak HTTP Support
This is mostly a challenge for JS (browser-based) clients where DELETE/PUT/PATCH support can be lacking.
Managing State
HTTP - Stateless
Server - Stateless
Guess who has to manage state?
API Client Best Practices
- Build Useful Objects
- Use Cache Headers
- Avoid Hardcoding Paths
Useful Objects
Resource responses should be translated into useful objects by the client.
Example: Twilio Python
# twilio/rest/resources/phone_numbers.py
class AvailablePhoneNumber(InstanceResource):
...
def purchase(self, **kwargs):
return self.parent.purchase(phone_number=self.phone_number,
**kwargs)
Example Usage: Twilio Python
from twilio.rest import TwilioRestClient
client = TwilioRestClient('<ACCOUNT_SID>', '<AUTH_TOKEN>')
numbers = client.phone_numbers.search(area_code=919)
if numbers:
numbers[0].purchase()
Cache Headers
These resource objects should track the cache state and send the proper headers.
Example: github3.py
# github3/models.py
class GitHubCore(GitHubObject):
...
def refresh(self, conditional=False):
...
headers = {}
if conditional:
if self.last_modified:
headers['If-Modified-Since'] = self.last_modified
elif self.etag:
headers['If-None-Match'] = self.etag
headers = headers or None
json = self._json(self._get(self._api, headers=headers), 200)
if json is not None:
self.__init__(json, self._session)
return self
Example Usage: github3.py
from github3 import login
client = login('mlavin', password='<password>')
me = client.user()
print(me.etag)
# Does a conditional fetch
me.refresh(conditional=True)
Avoid Hardcoding Paths
The client should use URLs and links returned from the server (when available).
One place where this is typically implemented is in pagination.
Example: pyrax
# pyrax/clouddns.py
class CloudDNSManager(BaseManager):
...
def _reset_paging(self, service, body=None):
...
links = body.get("links")
uri_base = self.uri_base
if links:
for link in links:
href = link["href"]
pos = href.index(uri_base)
page_uri = href[pos - 1:]
if link["rel"] == "next":
svc_dct["next_uri"] = page_uri
elif link["rel"] == "previous":
svc_dct["prev_uri"] = page_uri
Example: pyrax (Cont.)
# pyrax/clouddns.py
class CloudDNSManager(BaseManager):
...
def _list(self, uri, obj_class=None, list_all=False):
...
self._reset_paging("domain", resp_body)
if list_all:
dom_paging = self._paging.get("domain", {})
while dom_paging.get("next_uri"):
next_uri = dom_paging.get("next_uri")
ret.extend(self._list(uri=next_uri, obj_class=obj_class,
list_all=False))
return ret
Example Usage: pyrax
import pyrax
pyrax.set_setting('identity_type', 'rackspace')
pyrax.set_credentials('{username}', '{apiKey}')
# Internally this handles paging through all search results which may
# span more than one page
for record in pyrax.cloud_dns.search_records('example.com', 'CNAME'):
# Do something with the records...
Stop Making Glorified URL Builders
If your REST client library only builds URLs then both the server and client have failed.
Anything claiming to be a "general REST API" client is a lie.
Example: Slumber
import slumber
## Connect to http://slumber.in/api/v1/
## with the Basic Auth user/password of demo/demo
api = slumber.API("http://slumber.in/api/v1/", auth=("demo", "demo"))
## GET http://slumber.in/api/v1/note/
api.note.get()
## POST http://slumber.in/api/v1/note/
new = api.note.post({
"title": "My Test Note",
"content": "This is the content of my Test Note!"
})
## PUT http://slumber.in/api/v1/note/{id}/
api.note(new["id"]).put({
"content": "I just changed the content of my Test Note!"
})
## DELETE http://slumber.in/api/v1/note/{id}/
pi.note(new["id"]).delete()
Summary
REST is a client-server model.
API servers are nothing without clients.
API Creators
Try writing a meaningful client before releasing your server.
Show larger examples than a single GET/POST/PUT/DELETE request.
Resources
Thanks!
Slides - http://caktus.github.io/talks/oscon/2014/rest/
