The Ably Realtime protocol operates over a connection-oriented, framed, transport. It supports a single connection (and thus a single application) at a time, and enables traffic belonging to multiple channels to be sent over that single connection.
In the current Ably service the backend supports the Realtime protocol over a WebSocket transport and over a Comet transport. Other transports are possible and this document aims to define the protocol in a transport-independent manner.
The protocol supports both a text-based format using JSON and a binary format using MessagePack. All structures passed on the wire are defined in a JSON-like format. Other encodings may be considered in the future.
The unit of any protocol transmission is a ProtocolMessage
key value Hash object, referred to in this description as a Protocol Message.
The WebSocket transport transmits Protocol Messages in a single WebSocket data frame. In the binary transport, these are sent as binary data frames, with the Protocol Message being encoded with MessagePack. In the text transport these are sent as text data frames, the text itself being the white-space-free JSON encoding of the Protocol Message. Empty fields may be encoded as null, or may be handled as undefined (and thus absent from the JSON encoding) and clients should handle both possibilities. Handling empty fields as undefined is clearly preferable, however, since the encoded text is shorter and the encode and decode overhead is minimized. Binary data payloads are handled in the same was as JSON encoded payloads are handled.
In the Comet protocol, all request and response bodies contain an array of one or more protocol messages. In the binary protocol this is standard MessagePack binary encoding of a MessageSet Hash object. In the text (JSON) protocol request and response bodies are simply JSON-encoded arrays containing the Protocol Message content, again either in binary or text encoding.
Separate sections provide more detailed information on the WebSocket and Comet transports.
Each Protocol Message has an action
that indicates the nature of the message.
DISCONNECT
message or if a connection has been disconnected because of an error state with the error reason included in the DISCONNECT
Protocol Message.CLOSED
message will be sent by the service and the transport will be closed, if it has not already been closed by the client, and all connection state for that connection will be disposed. No connection state recovery is possible.CLOSE
request from the client. The transport will be closed by the service immediately after, and all connection state for that connection will be disposed. No connection state recovery is possible.ERROR
messages that apply to a channel have the channel field populated with the channel name; ERROR
messages that apply to the connection have no channel field value.ERROR
messages will have a value in the error field that provides information about the failure condition. Client libraries must make a transition (either of the connection or the relevant channel) to the failed state, and pass the contained error information to the client.CONNECTED
and ATTACHED
response messages may also have the error field populated with information about those non-fatal conditions. These are handled differently from the unrecoverable conditions indicated with an ERROR
message.ATTACHED
(success) response or an ERROR
(failed) response.ATTACHED
messages may also include a presence flag (right most bit value 1), which if present, indicates that members are currently present on the channel and a presence SYNC
is about to commence. Equally, the service might send an ATTACHED
message with no presence flag thus indicating that at the time of attach no members are present on the channel and as such the client can assume the presence set is empty.ATTACHED
messages are not required to be send in any order relative to their corresponding ATTACH
requests, and any other Protocol Message may intervene between the ATTACH
and ATTACHED
messages.ATTACHED
response if the channel is not in the pending state.DETACH
request.PRESENCE
messages may be sent in either direction. The channel associated with these presence updates is indicated in the channel field.msgSerial
field (in the client → service direction) or in the connectionSerial
field (in the service → client direction). Further information on message serial numbering is given below.MESSAGE
messages may be sent in either direction. The channel associated with these messages is indicated in the channel field.msgSerial
field (in the client → service direction) or in the connectionSerial
field (in the service → client direction). Further information on message serial numbering is given below.ATTACHED
Protocol Message being received by the client. Once a channel becomes attached, the server will automatically send a list of all members present on the channel to the client. Every SYNC
Protocol Message received will contain a channelSerial value and one or more PresenceMessages for each member currently present on the channel i.e. they are in the PRESENT
state. The channelSerial serves two purposes, it provides a way for the client library to resume a SYNC
should the transport be disconnected, and it also provides a means for the client library to know when the sync is complete. A channelSerial that is being synced will contian an ID with followed by a cursor after a colon such as “cf30e75054887:psl_7g:client:189”, however the final page of SYNC
messages will have a serial with an empty cursor such as “cf30e75054887:”. A client can explicitly request a SYNC with an optional channelSerial; if no channelSerial is provided the server will send a complete set of members on the channel; if a channelSerial is provided, the server will resume the SYNC
operation.CONNECTED
message being sent back to the client confirming the authentication succeeded. The server can request that the client authenticates by sending an AUTH
protocol message to the client, and the client must respond with a new token in an AUTH
protocol message.ProtocolMessages are populated with one or more of the following fields.
action
id
auth
channel
channelSerial
count
ACK
and NACK
actions. See message acknowledgement protocol.connectionId
CONNECTED
Protocol Messages from the service to the client. The connection ID is a public identifier used to uniquely identify each connected client.connectionKey
ConnectionDetails#connectionKey
is present, it should be considered the definitive connectionKey
for the current connectionconnectionDetails
connectionSerial
MESSAGE
and PRESENCE
protocol messages sent from the service to the client. The connectionSerial
is a zero-based, serially increasing number which, in combination with the connectionId
, uniquely identifies an attempted delivery of a Protocol Message to the client. The client uses the connection serial to track the receipt of messages, and may specify the connectionSerial
when performing connection state recovery.error
Error
type description for details of the contained information. The error field is populated in an ERROR
message and may also be populated to provide supplementary information (eg for non-fatal errors) in various other message types (CONNECTED
, ATTACHED
, DETACHED
, ACK
, NACK
).flags
ATTACHED
ProtocolMessage. See client library spec TR4imsgSerial
msgSerial
is a zero-based, serially increasing number which, in combination with the connectionId
, uniquely identifies the message across the system. The msgSerial
is also used in the message acknowledgement protocol.messages
MESSAGE
action contains one or more messages belonging to a channel. The messages field of the ProtocolMessage contains a collection of messages.presence
PRESENCE
action contains one or more presence updates belonging to a channel. The presence field of the ProtocolMessage contains a collection of presence messages.timestamp
Message
or PresenceMessage
which is an indication of the timestamp of receipt of that message by the system.The protocol relies on a number of structs embedded in Protocol Messages.
An object containing error information. This is used for unrecoverable error conditions (relating to connections, channels or individual messages) and also “informatively” for non-fatal errors in ATTACHED
or CONNECTED
responses.
Error contains the following fields.
In transports that support JSON encoding, Strings and the JSON Object and Array types are represented as their natural JSON value in the enclosing type. For example, a Message with a string payload would be encoded in JSON (with white-space added here for clarity) as:
{
"name": "my_event",
"data": "my_string_payload"
}
For string and binary payload types, an encoded
member is optionally added to the enclosing type. The only supported encodings are “utf8” for strings, “json” for JSON and “base64” for binary. If the encoding member is omitted it defaults to “utf8”.
Therefore, the following encoded messages each have string type:
{
"name": "my_event",
"data": "my string payload"
}
{
"name": "my_event",
"data": "my string payload",
"encoding": "utf8"
}
The following encoded message has binary type:
{
"name": "my_event",
"data": "bXkgYmluYXJ5IHBheWxvYWQ=",
"encoding": "base64"
}
The base64 encoding used is RFC4648 and clients must accept and process values with or without linefeeds.
The following encoded message has JSON type:
{
"name": "my_event",
"data": "{\"id\":\"value\"}",
"encoding": "json"
}
This is an individual channel message.
The members are as follows.
This is an individual channel presence update, as defined in the Thrift TPresence
struct.
The members are as follows.
ABSENT
, PRESENT
, ENTER
, LEAVE
, or UPDATE
) encoded as the ordinal in the PresenceMessage Action enum.ENTER
events.UPDATE
of the already-entered member, with a change in the value of connectionId.The Ably client API allows a caller to provide a success callback when publishing messages, or presence updates, to the Ably service. The callback is called, either with success or a failure code, once the Ably service has indicated whether or not it has processed the message successfully. The callback is not simply an indication that the message sent without error; it is confirmation that the service has processed the message sufficiently that its onward delivery to relevant attached clients is now guaranteed.
In the Comet transport, success or failure is indicated on a per-call basis with an ACK
or NACK
Message body in the HTTP response to the send
API call.
In the WebSocket transport, the service indicates success or failure to the client with the message acknowledgement protocol. This is a simple series of ACK
or NACK
responses, each addressing a contiguous sequence (by msgSerial
) of messages.
An ACK
message contains a msgSerial
and count
value. Receipt of this message signifies that the messages whose serial numbers are:
{ msgSerial ... msgSerial + count - 1 }
have been processed successfully.
Similarly, a NACK
message contains a msgSerial
and count
value and usually also an error
value. Receipt of this message signifies that the messages whose serial numbers are:
{ msgSerial ... msgSerial + count - 1 }
have encountered processing failures. The client library must call the callback, if supplied, with the contained error value, or with an error value that indicates an internal error.
ACK
responses are sent so as to be responsive to the client but also to avoid sending a response for every single published message; when the client is publishing at a high rate the ACK
responses will be sent periodically with an interval of 500ms (say) so many hundreds of messages my be covered in a single ACK
response.
NACK
responses are sent as soon as an error has arisen, and in any event only cover multiple messages to the extent that those messages are subject to the same underlying failure (since the error information is provided once for the group of @NACK@’d messages, not individually.
It is a protocol error if the system sends an ACK
or NACK
that skips past one or more msgSerial
without there having been either and ACK
or NACK
; but a client in this situation should treat this case as implicitly @NACK@ing the skipped messages.
It is also a protocol error if the system sends an ACK
or NACK
that covers a msgSerial
that was covered by an earlier ACK
or NACK
; in such cases the client library must silently ignore the response insofar as it relates to @msgSerial@s that were covered previously (whether the response is the same now or different).