Many Rust programmers don't know the exact semantics of let:

A variable in let must be assigned at least once in each control-flow path it is used in (exactly once if it is not declared mut).

This means we don't have to initialize variables if we don't use them:

fn main() {
    let _x: u8; // never assigned a value
}

On its own, this isn't very helpful, but it gets slightly less useless when we combine it with conditional control flow:

use rand;

fn main() {
    let x: u8; 
    
    if rand::random::<bool>() {
        x = rand::random();
        // the borrow checker knows that
        // `x` is always assigned before
        // it is read.
        println!("x is {x}");
    }
}

Ok, still not very useful yet. Where this gets actually useful is when we have a reference that we want to sometimes point to some data we allocated on the heap.

use rand;

fn main() {
    let x_string: String; 
    let mut x = "nothing";
    
    if rand::random::<bool>() {
        let n: u8 = rand::random();
        x_string = n.to_string();
        // because `x_string` is declared
        // in the same scope as `x`,
        // we can do this.
        // if `x_string` was
        // declared within this `if` block,
        // it would not live long enough.
        x = &x_string;
    }
    
    // pretend this is really complicated code
    // that we don't want to repeat.
    println!("x is {x}");
}

Many less-experienced Rust programmers would just use "nothing".to_string() here, always storing x on the heap, but this has the downside of a needless allocation.

Of course, in this example, we could just initialize x_string with a dummy value and make it mut, like let mut x_string = String::new(), but this has a few issues: 1. it implies the string may be mutated several times in-place. 2. if we forget the x_string = line, the compiler won't warn us, and we'll end up printing the empty string. 3. not all types have such a cheap/easy way to create a dummy value like this.

Another alternative would be using MaybeUninit, which may be necessary if your control flow is significantly more complex, but this should be avoided if possible due to the potential to cause Undefined Behavior.

If you're interested in a more real-world example of this pattern, rustdoc uses this in several places, such as in Type::attributes.

#rust #programming

You can follow this blog via its RSS feed or by searching for @binarycat@paper.wf on your Mastodon/ActivityPub instance.

it can either mean that a system is hard to see (like a pane of glass) or easy to see into (like a transparent gameboy).

these two meanings are pretty much exact opposites, and it's frequently difficult to know which one is being spoken about.

FORTH systems often pride themselves as following “radical transparency” due to their lack of encapsulation allowing powerful language extensions to be written in user code, while other programming languages advertising “transparent abstractions” that you can barely notice due to strong encapsulation.

it's also a metaphor, and as a general rule, metaphors should be avoided in technical writing, due to their potential for being misinterpreted.

#programming #terminology

You can follow this blog via its RSS feed or by searching for @binarycat@paper.wf on your Mastodon/ActivityPub instance.

NOTE: this document has been obsolesced by RITP

Goals

1. as simple as possible without sacrificing our other goals (less complex than http)
2. performs well under poor network conditions, even when payloads are large (unlike gemini)
3. performs well under high latency (unlike 9p)
4. good single-source performance (unlike BitTorrent)

Non-Goals

1. content negotiation (http Accept headers)
2. mutable identifiers
3. version negotiation

all of these can be trivially handled via an outer protocol (eg. mutable identifiers can be handled with gemini cross-protocol redirects)

Strategies

a stateful request/response protocol similar to 9p, but with hash-identified urls similar to magnet links (these encode length and content hash)

streams are encoded over tcp, tcp/tls, or quic to ensure reliable delivery.

Notation

each request has a number of fields. each field is marked either as a fixed number of bits, or as a “string” field.

“string” fields consist of a 64 bit length value, followed by that many bytes.

“string” fields do not have to be valid utf-8 unless specified.

all integers are little-endian. note that tokens are not integers, they are opaque client chosen identifiers with a length of 63 bits (followed by a 1 bit flag, which takes the place of the LSB of the final bytes). servers must take to mask the correct bit.

Requests and Responses

there are 2 request types: * OPEN * READ

there are 2 response types: * OK * ERROR

each request has the following structure: * (63 bits) new_token: a token identifying the results of the request * (1 bit) request_type: integer representing type of request * (n bits) type-dependent fields

each response has the following structure: * (63 bits) request_token: the client-chosen token found in the request that generated this responce * (1 bit) is_error: set if the corresponding request generated an error (such as the server being unable to find the requested resource) * (string) payload: if is_error is set, then a human and machine readable UTF-8 string representing an error. otherwise, its interpretation depends on the type of the request.

OPEN request

open requests one extra “string” field: * (string) uri: this field represents the uri of the resource to be downloaded. what schemes are supported depends on the server, but it is recommended to support at least urn:sha256:* uris.

the payload of non-error responses to OPEN requests is ignored, and SHOULD be empty.

READ request

read requests have two extra fields: * (64 bits) offset: at what point to begin reading * (64 bits) length: the maximum number of payload bytes the server is allowed respond with.

the payload of non-error responses to READ requests MUST be the empty string if and only if offset is greater than or equal to the total number of bytes in the resource, or if length is 0. if neither of these conditions are met, the payload MUST NOT be the empty string.

when the server generates an error in response to a token, all further READ requests targeted at that token are canceled. this allows a client to begin sending READ requests before it has received a response to the OPEN request.

URL schemes

rtp

much like the magnet url scheme, the rtp scheme consists entirly of predefined query parameters:

• (one or more) u: the uri/urn of the underlying resource. can be specified multiple times to specify multiple hashes for the resource (the client is expected to verify the hash, so specifying multiple u allows slowly migrating to a new hash algo). if multiple values are specified, they must correspond to the same resource.
• (up to one) l: the length of the resource
• (one or more) s: the server(s) that the resource can be retrieved from. if specified multiple times, the client may choose one, or perform a swarm download from several at once. these servers take the form of proto!addr!port, for example, tcp!example.com!7777. (this is based off of plan9 dial strings, since it seems to be the only well-specified way of specifying a method of transport)
• (zero or more) t: list of mime types that the resource may be interpreted as. clients MAY ignore values other than the first.
• (up to one) v: protocol version. if not specified, it defaults to version 1, which is the version specified in this document. clients MUST reject urls with an unrecognized version.

it is RECOMMENDED that every value of u is recognized by every server s. if a client encounters an error when downloading from one server, it SHOULD try downloading from another server.

clients SHOULD NOT use rtp urls in OPEN requests, instead they should choose a value listed in u.

unrecognized fields SHOULD be ignored.

gemini+rtp

use the Gemini Protocol in order to implement mutable identifiers.

gemini is used in “proxy mode”, that is, the sent url has a scheme of gemini+rtp and not gemini. the gemini server then uses a cross-protocol redirect to return an rtp url.

Rationale

READ.length is defined as a maximum so that clients that do not know the length of the resource they are downloading can use a value of 2^64 - 1 to request as much of the rest of the resource as the server is able to provide.

READ.offset exists both so that downloads can be resumed, and also to allow seeking within complex formats (eg. allowing you do download just one file out of a zip archive). It also allows doing swarm downloads from multiple equally trusted sources.

Appendix A: error strings

an error string consists of a machine readable string representing the kind of error, optionally followed by a colon, and then a human-readable string further clarifying the error.

the following predefined error strings: * error: a generic error kind usable when nothing else is applicaple * not found: no resource with the given uri is known to the server. * unsupported scheme: the given uri or urn scheme is not supported

Errata 2024-10-28

1. READ needs an additional field, open_token, which corresponds to the the request_token of an OPEN request.
2. RTP is commonly used as an abbreviation for the Real-time Transport Protocol, and is not descriptive enough.
3. this is a protocol for reliably downloading large files. it is not designed to be a drop in replacement for http or BitTorrent.

I am currently drafting a successor proposal that addresses these issues.

#networking #programming

You can follow this blog via its RSS feed or by searching for @binarycat@paper.wf on your Mastodon/ActivityPub instance.

imagine you have two devices, a client and a server, connected in a peculiar way:

1. the server cannot send messages to the client without the client asking for them
2. there are two channels, a request on one channel can only be responded to on the same channel
3. the first channel has infinite bandwith and is perfectly reliable, but each message is obscenely expensive.
4. the second channel is free, but can only send messages with a single payload bit, and the message has a 50% chance of being dropped

You've just been tasked with building a layer on top of this so that the server can send messages to the client, what do you do?

[DISCLAIMER: read the errata section before you go out and implement this]

How

Informal

the client sends 1-bit tokens. when the server receives that token, if it has generated new messages since it last sent that token, it responds with the boolean negation of that token.

the client always sends the last token it received. when it receives a token that is different from the one it sent, it retrieves a message from the side channel.

Formal

When the system start up, the devices initialize themselves to the following state: 1. the client sets it's client_token variable to false. 2. the server has an empty message queue 3. the server sets its server_token to false, and new_messages to false.

the client periodically sends its client_token value along the unreliable channel.

when the server receives client_token, it performs the following operation:

if client_token = server_token then
  set new_messages false
end if

it then responds with server_token.

whenever the server generates a new message, it performs the following operation:

if not new_messages then
  set server_token (not server_token)
  set new_messages true
end if

when the client receives server_token, it performs the following operation:

if server_token ≠ client_token then
  pull messages
  set client_token server_token

Why

turns out this absurd hypothetical isn't actually that far off of how TCP+TLS and UDP behave, especially in the context of mobile and embedded devices.

this system also has the nice property that the 1-bit messages don't depend on the side-channel messages, useful if there are actually multiple clients, and perhaps multiple secure servers the client wants to be able to receive messages from, and the presence of new messages is multiplexed through a single unsecured server.

Errata 2024-10-25: one bit isn't enough, but two is

as shown here, using a single bit means the server can't tell the difference between a client sending the previous token and the next token.

however, if you replace the 1-bit bools with 2-bit unsigned integers, and replace the negation with a wrapping increment, the protocol works as intended.

additionally, i should have added a timing requirement to the infallible channel, otherwise long polling is a simpler and equally valid solution to the posed problem.

#networking #programming

You can follow this blog via its RSS feed or by searching for @binarycat@paper.wf on your Mastodon/ActivityPub instance.

Copy and Clone can diverge

#[derive(Debug)]
struct OhNo(u32);

impl Clone for OhNo {
    fn clone(&self) -> Self {
        OhNo(self.0 + 1)
    }
}

impl Copy for OhNo { }

fn main() {
    let oh = OhNo(3);
    dbg!(oh.clone());
    dbg!(oh);
}

Even worse, according to the docs, this isn't even a logic error, unlike inconsistent implementations of PartialOrd and Ord

Really long place expression

a place expression is an expression you can take the address of.

turns out if statements are place expressions. this is so obscure that not even the reference knows this, despite it being true since rust 1.0

fn main() {
    let arr = [10, 20];
    let arr_ref = &if true {
        arr[0]
    } else {
        arr[1]
    };
    dbg!(arr_ref)
}

krate vs crate_

t-compiler says to use krate, t-style says to use crate_

sidenote: how many of you have actually read the style guide? how many actually know that it exists?

Rust has reference variables! kinda..

use std::cell::Cell;

fn main() {
    let x = Cell::new(1);
    let ref y = x;
    x.set(2);
    let ref z = x;
    assert_eq!(y, z);
}

This is mostly just silly.

&* is actually useful

previous entries are here because they are weird and obscure. &* is here because it is actually useful and meaningful, despite the fact that it would be a very silly no-op in most languages.

• invoking Deref without importing the trait
• reborrowing a mutable reference as shared
• turning raw pointers into references

Addendum: it's not just if

A reader asked me if a loop + break could also be a place expression. I thought do myself “well obviously that won't work”, before testing it out and realizing in horror that it does:

fn main() {
    let arr = [10, 20];
    let arr_ref = &loop { break arr[0]; };
    dbg!(arr_ref);
}

now this is weird.

Errata 2024-10-08: temporary lifetime extension is complicated

if is not a place expression. I was under the impression that the previous examples would not work unless it was, but actually my understanding of temporary lifetime extension is simply incomplete.

Addendum 2024-11-05: + is overloaded on strings

rust typically avoids C++ style operator overloading, favoring the haskell style of always having operators represent the same semantic operation.

nonetheless, you can use + for string concatenation, and you can use += as an alternative to push_str.

Addendum 2024-11-08: Calling methods of unnameable traits

Ever wanted an api that's impossible to use without glob imports? well now you can!

mod greet_ext {
    mod inner {
        pub trait GreetExt {
            fn greet(&self);
        }
        
        impl<T> GreetExt for T {
            fn greet(&self) {
                println!("hello, world!");
            }
        }
    }
    pub use inner::GreetExt as _;
}

pub use greet_ext::*;

fn main() {
    1.greet();
}

#rust #programming

You can follow this blog via its RSS feed or by searching for @binarycat@paper.wf on your Mastodon/ActivityPub instance.

background: what is an ocap

put simply, an object-capability is an object that represents a capability to do something.

for example, on unix systems, having a file descriptor gives you the capability to read and/or write that file, depending on how it was created.

where this becomes useful is when the ability to create these objects is restricted. returning to the fd example, a setguid program might open all the relevant files as root, then drop it's privileges to that of a regular user. if this is done before interpreting user input, it means whatever that input causes the program to do, it can't do anything that requires root, other than accessing to those specific files.

step 1: implicit parameters

when calling functions, implicit parameters are automatically filled in with a value from the surrounding context unless it is explicitly provided.

implicit parameters are primarily identified by their type, so their name can be omitted if they are not named directly.

def add(a: Integer)(using b: Integer) = a + b;

def add_more(a: Integer)(using Integer) = add(a) + 1

given Integer = 2;

add(0) // 2
add_more(1) // 4
add_more(5)(using 10) // 16

implicit parameters are useful for programs that are going to be passing around the same arguments to a lot of functions.

step 2: using implicit parameters for ergonomic object-capabilities

the main problem with ocaps is simple: passing all those capability tokens around is tedious and annoying.

without implicit parameters, if all the functions in your codebase log and access the filesystem, then ever function call in your code will have two extra parameters for the logging and filesystem handles.

with implicit parameters, you can omit those params from the call, and they will automatically be filled, in if the containing function has implicit parameters of a compatible type.

step 3: inferred implicit parameters for even less boilerplate, while maintaining control

while implicit parameters exist to repetition in function calls, inferred parameters exist to reduce repetition in function definitions

inferring implicit parameters is opt-in per-function (via a simple syntax such as implicit ... at the end of the argument list), and inferred parameters will be expanded in auto-generated docs.

the logic for inferring parameters is simple: if the containing function opts in, any function calls that would report missing implicit parameters will instead result in those parameters being added to the containing function.

note that if you want to manipulate a parameter directly, you would still have to name it as an implicit parameter, it could not be inferred. this is to prevent functions from containing references to identifiers they do not declare.

ideas for proof-of-concept implementations

it should be fairly easy to implement a prototype of this via code transformation.

a good candidate would be Julia, since it has a macro system, types, and is dynamic.

potential use in a faster and more secure operating system

modern OSes have many security features that require dynamic checking, often with hardware support.
one obvious example is process memory isolation, which prevents one program from accessing the memory of another.

there are a few past attempts to replace these dynamic security checks with static analysis (i'm sure they exist, but unfortunately i can't find them right now), however these were mostly held back by the compiler technology of the time (most of them used a modified version of C)

the main limitation of this would be having to distribute programs as source code in order to get substantial benefit, but with modern JIT compiler technology (as well as global compiler caches), this should be much more manageable.

instead of having an external manifest that tells what permissions an application needs, these permissions would be obvious based on the signature of that program's main function.

more performant?

ring transitions and syscalls are expensive, and there are other costs associated with process memory isolation, such as having to have a general-purpose heap allocator within every process. these in-process allocators can only receive whole pages from the OS, and more importantly, the can only return whole pages to the OS, and only under certain circumstances.

having a single shared address space has the potential to improve performance significantly.

more secure?

wouldn't removing the kernel/userspace split make things less secure?

well, if that's all you were doing, then sure!

but there's a few more pieces of the puzzle: 1. most important stuff is in userspace anyways 2. we're not just removing it, we're replacing it, and we're replacing it with something much easier to use.

filesystems

the biggest security hole of modern desktop operating systems is the filesystem, where keeping files secure requires creating fake “users” for a program and using setuid hacks.

mobile operating systems manage this by simply locking down the filesystem and requiring the use of alternate apis for inter-process communication.

our operating system would take a different approach, where a program can only access parts of the filesystem given to it as objects, either by via a drag&drop file manager, or via the system shell, which would logically be a repl for our new programming language.

an example of ocap filesystem access is in the experimental FileSystemDirectoryHandle browser api.

related work

• guile paramaters

Addendum 2024-09-30: isolation blocks

it has come to my attention that i have neglected to specify a useful concept: isolation blocks.

any function calls inside an isolation block will not resolve implicit parameters to a declaration outside the block.

this is not strictly necessary for the model to work, the same effect can be accomplished by using wrapper functions and passing parameters explicitly, but it makes things much easier to use.

this is expecially useful for interactive use, where it essentially allows you to enter a sandbox just by starting an isolation block. specific ocaps can be imported into the sandbox with the equivalent of scala's given keyword.

Addendum 2024-10-08: $PWD as an implicit parameter

one concept that is very pervasive in UNIX systems is that of a “current directory”.

there is one big problem with this: it's a process-wide variable.

one solution to this that has emerged is the openat() family of functions, these allow you to open a file relative to a file descriptor.

unfortunately, this is almost never used, due to the simple fact that passing around dir handles everywhere is kinda annoying.

under this new paradigm, everything that would care about the “current directory” instead takes an implicit DIR handle which all paths are opened relative to.

this removes pitfalls such as chdir() in multi-threaded applications, while also making it clear in auto-generated docs which functions care about the current directory.

if we require all paths to be simple relative paths (i.e. not starting with / and not containing ..), then we get the ocap-like isolation described before, and each function would essentially run in its own chroot.

#programming #rust

You can follow this blog via its RSS feed or by searching for @binarycat@paper.wf on your Mastodon/ActivityPub instance.