Creating a new REST server

November 2, 2016

During the last few years while working at Kaltura I have written, with my team members, many REST client libraries in many languages (including Python, Java, PHP, Javascript, ActionScript, C#, Erlang, ObjectiveC, and Ruby) against different servers. We also implemented REST servers in different languages (PHP, C#, NodeJS).
The REST standard is wide, and yet the common implementation of REST is very narrow and repeatedly follows the same mistakes already made by others.
I want to share Kalturians’ criticism of the common implementation and to offer a few solutions and tips that we developed through the years at Kaltura.

The problem: Complicated nested paths complicate the data model and make it hard to understand.
The solution: We do not believe in complex nested paths. Instead, have a flat list of services with a flat list of actions for each service.

The problem: Advanced HTTP methods are not supported by all firewalls and platforms.
The solution: Avoid using custom HTTP methods such as PUT, HEAD, and PATCH. Instead, we specify the action in the URL and support both GET and POST with all actions.

The problem (part 1): If the same error codes could be relevant both to the HTTP protocol and to the application, it might be difficult to understand which one of them raised the error.
The problem (part 2): A single HTTP error prevents sending multiple applicative requests in a single HTTP request so that each one of them may have different responses or different error codes.
The problem (part 3): A single error code prevents complex responses with partial success or warnings.
The solution: HTTP error codes should be used for HTTP protocol, while applicative errors should be returned in the response content (JSON or XML).

The problem: Sending XML or JSON and upload files in the same HTTP request.
The solution: Use multipart/form-data for the files and send XML or JSON field in the same request.

The problem: Serving content binary and textual content as part of REST API.
The solution: You can use your API, just make sure to use the right content-type in the response.

The problem (part 1): Keeping the documentation up to date.
The problem (part 2): Enabling easy development of client libraries for different languages.
The problem (part 3): In micro-services architecture, different servers are written in different languages. A cross-platform solution requires many different client libraries in different coding languages to communicate with the same servers that the API is changing and developing consistently.
The solution: The XML descriptor generated from the code describes the entire API. Based on that XML, it’s easy to generate documentation and client libraries

Complex paths

REST implementers commonly try to reflect nested objects using hierarchic paths.
For example, if you want to get the devices of a user that belongs to a household that belongs to an account, you might need to use a URL that looks something like this:

/api/account/[account-id]/household/[household-id]/user/[user-id]/device/[device-id]

I believe that if each device has a unique id, it’s better to use a simple form such as:

/api/device/[device-id]

If your system supports more than one type of device (for example, users’ devices and household devices), I still recommend one simple form such as:

/api/userDevice/[device-id]

Flattening your data model to a list of services, when each service represents a single data object type, will force you to choose simple data model for your server side as well.
Of course, flat models will be much easier for your integrators and customers to use.

HTTP methods

Most implementations of REST are counting on different HTTP methods for different CRUD operations on the same object.
Examples include GET for list, PUT for update, POST for create, and DELETE for delete. I also saw implementations that use OPTIONS, HEAD, PATCH, INSERT, UPDATE and other non-standard methods.
I see few problems with this approach.

Not all firewalls enable these custom methods, whereas POST and GET are always enabled.
Associating API call with a single HTTP method will prevent any future wrapping of several API calls in a single HTTP round-trip.
Because different implementations use different methods for the same actions, it’s not really self-explanatory and documentation on the right method for each operation is still needed.
Assuming that you want to enable deleting a user by its ID in one API action and deleting user by its e-mail in a different action, how would you differentiate between the two using HTTP methods?

The solution I prefer is to add the operation to the URL and the HTTP request. This way, you can support both GET and POST with plain form data, in XML or JSON.
Here are a few examples (the examples are self-explanatory):
Using path (GET):

/api/device/get/id/[device-id]
/api/device/delete/id/[device-id]
/api/device/list
/api/device/list/filter:householdIdEqual/[household-id]/filter:statusEqual/[status]
/api/device/add/name/[device-name]/description/[device-description]/tags/[device-tags]
/api/device/update/id/[device-id]/name/[device-name]/tags/[device-tags]

Using query string (GET):

/api?service=device&action=get&id=[device-id]
/api?service=device&action=delete&id=[device-id]
/api?service=device&action=list
/api?service=device&action=list&filter[householdIdEqual]=[household-id]&filter[statusEqual]=[status]
/api?service=device&action=add&name=[device-name]&description=[device-description]&tags=[device-tags]
/api?service=device&action=update&id=[device-id]&name=[device-name]&tags=[device-tags]

Using POST (form data combined with path):

/api/service/device/action/get
id=[device-id]
/api/service/device/action/delete
id=[device-id]
/api/service/device/action/list
/api/service/device/action/list
filter[householdIdEqual]=[household-id]&filter[statusEqual]=[status]
/api/service/device/action/add
name=[device-name]&description=[device-description]&tags=[device-tags]
/api/service/device/action/update
id=[device-id]&name=[device-name]&tags=[device-tags]

Using POST (JSON combined with query string):

/api?service=device&action=get
```
{id: [device-id]}
```
/api?service=device&action=delete
```
{id: [device-id]}
```
/api?service=device&action=list
```
{}
```

/api?service=device&action=list

{
filter: {
householdIdEqual: [household-id],
statusEqual: [status]
}
}

/api?service=device&action=add

{
    device: {
        name: [device-name],
        description: [device-description],
        tags: [device-tags]
    }
}

/api?service=device&action=update

{
id: [device-id],
    device: {
        name: [device-name],
        tags: [device-tags]
    }
}

Using POST (XML):

/api
```
<[device-id]
```
/api
```
[device-id]
```
/api
/api
```
[household-id]
[status]
```

/api



[device-name]
[device-description]
[device-tags]

/api


[device-id]

[device-name]
[device-tags]

Looking at the examples, you can see how self-explained the actions are and how it’s simple to add a new action (deleteByEmail for example).
Assuming that you want to create a user and a device that associated with that user, you could perform both actions in a single HTTP request.
Note the token I entered into the example requests. {results:1:id} means use the ID attribute from result number one.
This empowers the multi-request by enabling the client to use the output of one action as the input of another action in the same HTTP request.
Here are a few examples:
Using POST (form data combined with path):

/api/service/multirequest

1[service]=user&1[action]=add&1[user][firstName]=[user-first-name]&1[user][lastName]=[user-last-name]&1[user][email]=[user-email]&2[service]=device&2[action]=add&2[device][userId]={results:1:id}&2[device][name]=[device-name]&2[device][description]=[device-description]&2[device][tags]=[device-tags]

Using POST (JSON combined with query string):

/api?service=multirequest

[{
service: "user",
action: "add",
    user: {
        firstName: [user-first-name],
        lastName: [user-last-name],
        email: [user-email]
    }
},{
service: "device",
action: "add",
    device: {
        userId: "{results:1:id}",
        name: [device-name],
        description: [device-description],
        tags: [device-tags]
    }
}]

Using POST (XML):

/api




[user-first-name];
[user-last-name]
[user-email];




{results:1:id}
[device-name]
[device-description]
[device-tags]

HTTP Error codes

Usually, the HTTP protocol is used to return HTTP error code that reflects the status of the request.
For example, 400 for bad request, 422 for unprocessable entity, 404 for entity not found, 403 for forbidden service or action, 401 for unauthorized object or action.
Also, the success error codes could be different. For example, 200 for successful list, 201 for successful creation, 202 for successful update, 204 for no content, 205 for successful deletion.
These are the issues I see with this approach:

Numbers are never self-explained. You always need to read the textual description and the documentation to react correctly to each error code.
HTTP protocol enables only one error code (preventing you from returning warnings) or a few codes in case of a few requests (such as in case of multi-request that suggested above).
One textual description doesn’t offer detailed error message with additional attributes such as “Device filter is invalid, at least on of the properties userIdEqual or householdIdEqual is required”.
Multilinguality and localization is impossible if the error message may contain different attribute names or real data, for example “Device id [1234] not found”.

The error codes that I want to reflect in the HTTP protocol are always HTTP errors, such as 200 OK, 404 Not Found, 500 Internal Server Error.
All other applicative errors will be returned with HTTP status 200 and the response content will reflect the error.
Few JSON request and response:

Request: /api/user/add

{
user: {
firstName: [user-first-name],
lastName: [user-last-name],
email: [user-email]
}
}

Good response:

{
    user: {
        id: [user-id],
        createdAt: [user-create-time],
        updatedAt: [user-update-time],
        firstName: [user-first-name],
        lastName: [user-last-name],
        email: [user-email]
    },
warnings: [
    {
        code: 123456,
        message: "User first name [user-first-name] is too long",
        attributes: {
        firstName: [user-first-name]
    }
}],
executionTime: 0.6555
}

Bad response:

{
    error: {
        code: 123457,
        message: "User e-mail [user-email] is not valid",
        attributes: {
        email: [user-email]
    }
},
executionTime: 0.6555
}

/api/multirequest

[{
service: "user",
action: "add",
    user: {
        firstName: [user-first-name],
        lastName: [user-last-name],
        email: [user-email]
    }
},{
service: "device",
action: "add",
    device: {
        userId: "{results:1:id}",
        name: [device-name],
        description: [device-description],
        tags: [device-tags]
    }
}]

Response:

{
[{
user: {
id: [user-id],
createdAt: [user-create-time],
updatedAt: [user-update-time],
firstName: [user-first-name],
lastName: [user-last-name],
email: [user-email]
},
warnings: [{
code: 123456,
message: "User first name [user-first-name] is too long",
attributes: {
firstName: [user-first-name]
}
}
},{
error: {
code: 123458,
message: "Device name [device-name] may not contain special charecters [&]",
attributes: {
name: [device-name]
charecters: "&"
}
},
}],
executionTime: 0.6555
}

As you can see, using error code and attributes you can translate the error message to any language in any structure.
Also, returning the errors in the response body you can support complex responses with detailed explanation for each failure.

File uploads

Most REST API implementations do not support file uploads.
If you followed my recommendations above to always support both GET and POST, then when it comes to actions that include uploaded file, you’re required to use POST only.
In addition, the content-type of your request can’t be application/json, application/xml, text/xml, or even application/x-www-form-urlencoded. It must be always multipart/form-data.
If you still want to support JSON or XML content, you can always submit them as a form field, for example:

Content-Type: multipart/form-data; boundary=---------------------------90519140415448433659727646345634
-----------------------------90519140415448433659727646345634
Content-Disposition: form-data; name="json"
user: {
    firstName: [user-first-name],
    lastName: [user-last-name],
    email: [user-email]
}
-----------------------------90519140415448433659727646345634
Content-Disposition: form-data; name="homepageContent"; filename="myHomepage.html"
Content-Type: text/html

Content of my homepage HTML.
-----------------------------90519140415448433659727646345634--

Content

Most systems are required to also serve images, video files, textual content, and other binary content.
Many binary content entities are used in browsers such as images or used as URL to integrate with UI and external systems.
Therefore I believe it’s important to support also plain GET with path attributes or query string.
A few examples:

/api/user/serveImage/id/[user-id]
/api/user/generateThumbnail/id/[user-id]/width/400/height/300
/api?service=user&action=generateThumbnail&id=[user-id]&width=400&height=300

Note that I encountered few STB devices that accept feeds for video content that do not support query strings. That’s why supporting variables in path was important to me.
If your content should be protected, you can always add authorization token to the URL or tokenize the URL using a CDN token that expires after a configurable time.

Documentation and client libraries generation

At Kaltura, we easily generate different client libraries using clients generator we wrote. It’s open source and generates client libraries in many coding languages: Python, Java, PHP, Javascript, ActionScript, C#, Erlang, ObjectiveC, and Ruby.