an incredibly simple protocol for reliably and correctly downloading large immutable files.

(this is a follow-up and successor to my RTP proposal)

the ritp url scheme will be described in a follow-up post.

Problem Statement

Downloading large files is hard. Networks are messy, and while there are plenty of protocols that aim to abstract away this messiness, they all have their own problems.

HTTP

If implemented properly, HTTP can handle essentially anything you throw at it... however, to my knowledge, there is no full http implementation that correctly implements every corner-case of error handling. (wget is probably the closest)

The main difficulty of HTTP is that it's too flexible for its own good: all the error recovery and detection mechanisms are optional headers, so a complete implementation has to have several layers of fallbacks, and testing all these rarely used code paths is a significant challenge. RITP addresses this complexity by limiting its scope and choosing a small number of mandatory fields that all servers must support.

One problem with http is that of mid-air collisions: it is entirely possible for a server to implement Range but not provide a Last-Modified or ETag field. RITP solves this by not allowing files/resources to be modified.

Gemini

the gemini protocol is essentially a response to the complexity of http, but it limits its scope to “serving small markup files”, and explicitly recommends using another protocol to handle large downloads.

RITP can be used with gemini as this “protocol for large file downloads”. this especially works well with gemini's cross-protocol redirects, giving a layer of mutable identifiers above the immutable RITP.

BitTorrent

BitTorrent is a great protocol at what it does, but it has one major downside: poor single-peer performance. Currently it solves this with webseeds, but then you have to deal with the issues of HTTP.

RITP can be used with BitTorrent as an alternative protocol for webseeds, reducing the overhead of all the http headers.

9p

The main problem with 9p is poor performance over high-latency links.

Core Protocol Description

RITP is a stateful request/response protocol, similar to 9p.

all messages (both requests and responses) start with the same fields: * a 32 bit length field that stores the length of the entire message, including the length field itself * an 8 bit type field that identifies the type of message * a 24 bit token field that identifies the message batch the message belongs to. this is described in further detail in the batches section.

whenever a field specifies a length or offset of/within a file, this length/offset is in units of bytes.

message types

the high bit (0x80) of the type field marks a message as a response (server-sent).

depending on the type of a message, a message has any number of fixed-size fields, followed by a variable-length “tail field”, which consists of all following bytes of the message. the interpretation of the tail field depends on the type of the message. if no special interpretation of tail field is specified, it should be ignored. the tail field takes advantage of the fact that messages already describe their total length.

Requests

0x01 OPEN

the remaining bytes (tail field) of the message are interpreted as a multihash, which is then looked up by the server and the corresponding file is associated with the message batch.

if the batch id has been used before, the previous batch is closed and replaced.

0x02 READ

fixed-size fields: * (64 bits) offset * (32 bits) length

requests a byte slice of the file associated with the message batch.

Responses

0x81 OPENED

used to indicate that an OPEN request has been successful.

fixed-width fields: * (64 bits) length: the total length of the file that is now associated with this batch.

0x82 DATA

given in response to a READ.

fixed-size fields: * (64 bits) offset: the offset field of the corresponding READ request.

tail fields: * payload: a substring of the overall file, starting at offset.

the payload should be the empty string if (and only if) offset is greater than or equal to the length of the associated file. this mirrors the UNIX method of signaling end-of-file.

0x83 ERROR

fixed-size fields: * (8 bits) code: an error code, possible values listed in the errors section.

tail field: * description: a human readable utf8 string providing a description of the error

when an error response is given, all future messages sent to that batch are canceled, and will not be given a response.

Errors

• 0x00 other error: an error occured that cannot be accurately described by any of the other error codes.
• 0x01 not found: no file with the given hash is stored on the server. this error code is also used if the hash is malformed or uses an unsupported hash algorithm.
• 0x02 unknown request type: the type was not recognized by the server. this allows the protocol to be extended in the future by adding message types.
• 0x03 batch does not exist: the type was recognized, and the batch unexpectedly does not exist.

Additional Explanation

Batches

A batch is a group of messages that represent operations on a single file. A batch is identified by it's token.

A batch begins with an OPEN request and is followed by any number of READ requests.

If the OPEN request encounters an error, other requests with the same batch token will be ignored.

If other message types are introduced in the future, they will be specified as either introducing a new batch (like OPEN) or being part of an existing batch (like READ).

Future Compatibility

future versions of this specification may define new message types.

clients may attempt to use these new message types with any servers, and servers that do not support them MUST respond with error 0x03 unknown request type.

servers MUST NOT respond with new response codes unless the client previously sent a request that indicates that response code is supported (what requests imply support for what responses will be defined in future specifications)

Rationale

“batch does not exist” errors are only returned for recognized message types to potentially allow adding new message types that create a batch.

batch tokens are client-assigned so that a client can “pipeline” an OPEN and a READ in a single transmission, reducing round-trip delays.

errors cancel a batch to prevent an error in a large pipelined batch from causing a bunch of error responses.

there is no need to identify which request in a batch caused the error, as properly formed (i.e. part of an OPENed batch) READ requests are infallible. if they were not infallible, a READ ERROR response would exist, which would contain the offset of the read that caused the error.

the server is allowed to respond with short reads to allow for servers that use fixed-size buffers or have limited memory.

OPEN requests originally used urns, but this was changed to multihash after discovering that sha256 and md5 are not actually in the list of urn namespaces

messages describe their total length to allow new request codes to be added.

---

#networking