Irdest developer manual

Welcome to the Irdest developer manual. This manual is a collection of documents explaining irdest internal concepts, structures, and protocols. Its source files can be found in the main irdest code repo.

Outline

Social Documentation

This section outlines the social structure of the irdest project.

Communication

The irdest project uses Matrix as a development and social chat. Feel free to drop by to ask questions or hang out!

Code of Conduct

TLDR: be nice!

We want to foster an open and engaging atmosphere for irdest and the development community around it. Because of this we follow the "Contributor Covenant" code of conduct. A copy of it should have been included in the sources for this book.

How to contribute?

First of all: thank you for wanting to help out :)

The irdest source can be found in our mono repo. We accept submissions via GitLab merge requests, or via patches sent to our mailing list!

Submitting an MR

  • If a relevant issue exists, please tag in your description
  • Include a short description of the accumulative changes
  • If you want your history to be rebased/ merged, please clean it up to be useful. Otherwise we will probably squash it.
  • Feel free to open a work-in-progress MR as a place to have a discussion about changes or to get feedback.

Submitting an e-mail patch

If you can't contribute via GitLab , you're very welcome to submit your patch via our community mailing list.

The easiest way of doing this is to configure git send-email.

Without git send-email

  • Send an e-mail with the title [PATCH]: <your title here>.
  • Format your patch with git diff -p
  • Don't send HTML e-mail!
  • Make sure your line-wrapping is wide enough to allow the patch to stay un-wrapped!

Lorri & direnv

You can enable automatic environment loading when you enter the irdest repository, by configuring lorri and direnv on your system.

 ❤ (uwu) ~/p/code> cd irdest
direnv: loading ~/projects/code/irdest/.envrc
direnv: export +AR +AR_FOR_TARGET +AS +AS_FOR_TARGET +CC
        // ... snip ...
 ❤ (uwu) ~/p/c/irdest> cargo build                           lorri-keep-env-hack-irdest
 ...

Contributor Covenant Code of Conduct

Our Pledge

In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, gender identity and expression, level of experience, nationality, personal appearance, race, religion, or sexual identity and orientation.

Our Standards

Examples of behavior that contributes to creating a positive environment include:

  • Using welcoming and inclusive language
  • Being respectful of differing viewpoints and experiences
  • Gracefully accepting constructive criticism
  • Focusing on what is best for the community
  • Showing empathy towards other community members

Examples of unacceptable behavior by participants include:

  • The use of sexualized language or imagery and unwelcome sexual attention or advances
  • Trolling, insulting/derogatory comments, and personal or political attacks
  • Public or private harassment
  • Publishing others' private information, such as a physical or electronic address, without explicit permission
  • Other conduct which could reasonably be considered inappropriate in a professional setting

Our Responsibilities

Project maintainers are responsible for clarifying the standards of acceptable behavior and are expected to take appropriate and fair corrective action in response to any instances of unacceptable behavior.

Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful.

Scope

This Code of Conduct applies both within project spaces and in public spaces when an individual is representing the project or its community. Examples of representing a project or community include using an official project e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event. Representation of a project may be further defined and clarified by project maintainers.

Enforcement

Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting the project team at kookie@spacekookie.de. All complaints will be reviewed and investigated and will result in a response that is deemed necessary and appropriate to the circumstances. The project team is obligated to maintain confidentiality with regard to the reporter of an incident. Further details of specific enforcement policies may be posted separately.

Project maintainers who do not follow or enforce the Code of Conduct in good faith may face temporary or permanent repercussions as determined by other members of the project's leadership.

Attribution

This Code of Conduct is adapted from the Contributor Covenant, version 1.4, available at http://contributor-covenant.org/version/1/4

Technical Documentation

Welcome to the irdest technical documentation. Following is an outline of the system, which will hopefully give you an initial understanding of the project stack.

This manual is relevant for both irdest hackers, and irdest application developers.

Note to future editors: make sure every major component has an internals section, so to make it convenient to skip.

Introduction

Fundamentally, irdest is a highly distributed system. It is not accountable to a single set of rules across all devices that are part of this system. Each device can be home to many users, that can all send messages to each other, but have separate states. Furthermore, connections in this system are not always real time.

A lot of traditional networking infrastructe is built up in layers. Similarly, the irdest project replicates these layers. To give you a better understanding of what parts of the irdest project do what, here's an overview of the layers in a full irdest application stack.

LayerComponent nameDescription
Network driversnetmod, netmod-tcp, ...Network drivers to establish and manage connections with peers via different underlying transport mechansms
Irdest RouterratmanA decentralised packet router using cryptographic public keys as addresses for users
Service API coreirdest-coreThe main irdest state handler, managing an encrypted database, user profiles, and more
Irdest RPCirpc-sdk, irp-brokerA generic RPC protocol to connect different irdest services together
Servicesirdest-chat, irdest-groups, ...A service is a micro application (or actor) that communicates with other services via the irpc bus
Clientsirdest-hubd, irdest-gtkUser facing applications that uses a collection of services and the RPC layer to implement different features

Services & clients

A service is a client on the irdest RPC interface. A service MAY expose an API to other services. An example of such a service is irdest-groups, which implements encrypted group management logic.

A service that does not expose an API of its own can be referred to as a "client service". Technically all user-interfaces are client services.

The irdest project develops and ships a set of core services, that can all be used via the irdest-sdk library. Check out the Rust documentation to find out how you can write a simple client service using these existing APIs.

Irdest RPC

The irdest RPC interface uses a central broker (called irpc-broker) which listens for connections over local TCP connections. This broker is included in the irdest-hubd client. A development utility client is available: irpc-client.

TODO: link to example service section

Irdest core

The irdest-core manages user identities.

Ratman

This is arguably the heart of the irdest application stack. Ratman is a fully decentralised, delay tolerant gossip router. It provides an API that takes messages to send to peers on the network, and returns messages received from the network.

It handles announcements, network segmentation, message journaling, route updates, and networked archive storage.

Addresses on a Ratman network are 32-byte ed25519 public keys, meaning that all direct messages are automatically encrypted.

Network Modules

...

Development Interfaces

This page is outdated!

This document will change quite a lot during the development, especially because there are several layers of interfaces that need to be documented. In the future we might want to move some of them into their dedicated books.

For now, this document is an outline for the json RPC interface of irdest-core. It's being made available via the irdest-core-http crate.

Envelope

Every message is packed into an envelope. It contains metadata that might be present in some transports, but not others. The envelope was generally built to be streamed via sockets, where message association isn't handled via the transport protocol (like http).

{
  "id": "1",
  "auth": {
    "id": "7F9B BAA9 DF90 0F48 CD41 7FD4 2C9C E432 1309 FB79 8CF9 C56F FB9A 8F2C 29C1 12D9",
    "token": "56C0 F4C0 BD0D E822 B21D 4120 CC43 5C73 1F92 B246 C988 3335 72F5 8F1A 2701 3FD7"
  },
  "page": "1",
  "method": "query",
  "kind": "messages",
  "data": {
    "query": {
      "tag": {
        "key": "room-id",
        "val": "A88A 7CC6 DF68 CE60 2D63 64CB 7393 8924 653D 52FB DBC6 2A39 D892 A9BD 6FA4 FD5D"
      }
    },
    "service": "net.irdest.chat"
  }
}

This type can be found in irdest-core/rpc/src/json/mod.rs. The general structure of requests stays the same: there's an ID, some auth data, the page (which isn't implemented yet), method and kind the request operates on.

There are several types available (kind):

  • users, either a local or remote user
  • contacts, some user-specific contact data for another user
  • chat-rooms, a chat room via service (requires irdest feature)
  • chat-messages, a chat room message (requires irdest feature)

Possible methods are (not all combinations exist though!):

  • list, get a list of all, if available
  • create, create a new (with side-effects, such as sending)
  • delete, remove from local stores
  • get, return single item by ID
  • query, return a set of items, according to some query
  • modify, change an existing item in some way
  • subscribe, (not implemented yet)
  • unsubscribe, (not implemented yet)

Three special methods only exist for users:

  • login, validate password and receive auth tokens
  • logout, end session token
  • repass, change user passphrase

Following is a list of examples of how to construct valid requests. If in doubt, the code that parses these is in irdest-core/rpc/src/json/parser.rs. You can always look it up there.

irdest services

Following is a list of services that are bundled with irdest, and what they can do for you.

Services have an internal ID that is used to identify them in irdest-core, using reverse fully qualified domain specifiers (for example org.irdest.my-app).

Fundamental services

As outlined in the technical introduction, some services are fundamental, meaning they provide functionality to other services, instead of being user-facing.

Type Messages

This service provides an interface to send strongly typed messages, with extension hooks for external services to provide their own type data. In irdest-core all payloads are raw binary which is done to save space, and not complicate the API unecessarily. However sometimes type information is useful, and an app service might want to rely on it. That's what this fundamental service is for!

  • Service ID: org.irdest.fundamental.messages
  • Crate name: irdest-type-messages

Group abstractions

This service provides a group management abstraction. While many app services might want to manage groups, doing so in an encrypted system, with no direct group leader can be tricky. To avoid having to re-implement this more than once, all irdest bundled app services use this fundamental service, and it is recommended that your app services do too!

  • Service ID: org.irdest.fundamental.groups
  • Crate name: irdest-groups

App services

These are services that provide some user-facing functionality. Some will ship with their own UI, while others will use a UI bundle (such as the main irdest services).

Feed

  • Service ID: org.irdest.feed
  • Crate name: irdest-feed

Public message feed, similar to micro-blogging applications such as mastodon or twitter. Users can compose short-medium length messages that get shared with the whole network. You have the choice of filtering by "all messages" and "only following", where you can only display messages sent by friends or people you trust.

Messaging

  • Service ID: org.irdest.messaging
  • Crate name: irdest-messaging

Private (1-to-1 or groups) text messaging, that can optionally (by default) be encrypted. Conversations can either be displayed as a feed (like chat), or threaded (like e-mail). Files can be sent either in-line (if the payload is small enough), or via file-announce messages that use the "files" service.

Filesharing

  • Service ID: org.irdest.files
  • Crate name: irdest-files

Filesharing via announce-links similar to how torrents get announced on trackers. You can announce a file to either a group of people, a singe user, or the whole network. Optionally this service can be configured to download all files that were publicly announced to allow servers to replicate a "public archive" that users can have access to, if the original source of a file disappears.

Voices

  • Service ID: org.irdest.voices
  • Crate name: irdest-voices

Integrates with various platform features to allow voice call streaming over the irdest, as well as preparing audio messages that can be inlined into messages, or sent to many people via announce links.

Radio

  • Service ID: org.irdest.radio
  • Crate name: irdest-radio

Similarly to voices, it integrates into platform features to provide audio capture and playback, but for one-to-many streams. This way people can broadcast themselves into the network, while others can tune into a program, without being able to respond. Similar to files, this service can be configured to automatically archive radio broadcasts for community servers.

Writing an irdest service

This document outlines how to write a service using the irdest development libraries. Make sure to read the technical overview first to gain a better understanding of how the system works, and what functionality your service can rely on.

irdest-core

The core of irdest, providing an API for user profile management, services, and message creation. It bundles an encrypted database to persist data to disk.

irdest-core Internals

This section of the manual covers parts of irdest-core not exposed directly via the API. It's primarily useful to learn to debug behaviour, and for future contributors to get up to speed with the internal structure.

Most of the components exposed via the API have an internal component that acts as a strongly typed store (messages, users, contacts, files, ...). Some more interesting components are those that are not exposed.

Key Store

Similar to the user and message store, irdest-core stores public keys that it comes across, for later. This is meant to opportunisticly fill the store with keys that can be user later on to send encrypted messages or verify signatures.

The store is shared between users, but never exposed to users so that keys don't have to be fetched multiple times, but it's not neccessarily possible for a user to figure out the social cycle of one of their peers.

Seeding

When a user is active on the network, this module sends out regular "announcements" that are used to keep the routing tables up with topology changes, as well as updates to local users. Some seeds are opportunistic, such as throwing out a public key, others are reactive, driven by data requests that are picked up by the discovery module.

There are several strategies that can be chosen when seeding data into the network, which should be given their own page soon.

Discovery

The inverse of seeding to the network is the discovery module which reacts to user seeds. It will insert new users into the user store, relay messages to the appropriate service, or dropping them if no service handler was found.

Persistence

The presistence module is implemented mostly by wrapping internal irdest-core types with alexandria storage callbacks, which is the library which implements all of the persistence logic and at-rest encryption. It is developed as part of irdest, but pulled out of the main tree to make it easier to use in other projects.

Routing

All calls to actually send data into a irdest network are implemented by Ratman, the delay tolerant, distance vector based routing protocol built specifically for irdest. You can find documentation specifically for it in later sections of this manual.

Service API

This section of the manual covers the service API. Please check the irdest-core Rust API docs for actual function docs. This document will go into the concepts more than actual code.

The API is written in Rust and uses async/await futures, built with the async-std runtime. If you build a service in Rust, we recommend you use the same runtime to keep the application binary as small as possible.

Code and function docs for irdest-core can be found here!

Users

At the heart of almost every call to the service API is a user session. You can use the login and logout functions to manipulatio a session. The user authentication object, containing the user's ID, and a token assigned with your session, is required for every subsequent call into the API. Tokens will also expire with inactivity, so building your code to be resistent to Error::NoAuth errors is always a good idea.

Attached to a user comes a message, contacts and file store. A global user store also exists. These are provided by function scope endpoints, that encapsulate various functions in a type/namespace to make interacting with them easier.

User store

An instance of irdest-core keeps track of users on the network, and general user announcements, which are saved in the user store. These user profiles are slowly filled with metadata, as more information becomes available about a user: did they set an avatar recently, do they have a preferred nickname or pronouns, etc.

It's also possible to search by any attribute in the store, even optional user generated fields like location.

Mesage store

For each user, a message store overlay is kept, meaning that an instance only keeps one copy of the actual messages, but what user has access to them changes what part of the store they can see. Similar to the user store, it's possible to send queries to the store to get messages sent to the current user and service, according to a parametric search.

Contact book

Each user also keeps a private contact overlay, in which they can annotate users they have interacted with. Available fields are trust, if two users have met, and free-form additional metadata.

Storage

Because irdest-core already has a mechanism in place to encrypt data for a user at rest, this mechanism is exposed in the API to external services. On one hand this API can be used to store additional metadata for a service, that is required to make it all work, or to get files that the user has discovered on the network and that were downloaded previously.

This API scope is complemented by the "files" endpoint, that provides a way to announce, pack, update and send files into a network. Note maybe it would make more sense to join "files" and "storage", and move the "files" functionality entirely into the files service.

Ratman

This is the documentation tree for the decentralised, delay-tolerant vector routing protocol "Ratman". It's basic design is influenced heavily by protocols such as B.A.T.M.A.N., serval, and others.

In Ratman, all routing is done on IDs that are represented as the blake2 hash of a user's public key. It has a real-time and buffered routing component: routing tables are built in userspace over a list of interfaces and interface specific IDs (not the pubkey!), to associate a pubkey with an interface route. This relationship is further explained in the netmod documentation.

If Ratman isn't able to find a route to a target, or based on various parameters, it will cache a message in the local journal, periodically checking if it can be delivered at a later point in time.

Design Guide

This document outlines the basic design guide for irdest components.

Code design

  • Use async only if required
  • Use named generics instead of impl T in public functions
  • Avoid shared state, if a channel will do the same thing

Visual design

irde.st website

The irdest website is built via the static site generator hugo. Its contents and sources are part of the irdest mono repo.

Building the website

You need to have hugo installed on your system to build the website.

$ hugo build # build the website for deployment
$ hugo serve # serve the website for development

Build with Nix

Alternatively you can use the Nix package manager to build the website. The build process will create a result symlink to the generated site data.

$ nix build -f nix/ irdest-website
...

Website structure

The website structure is somewhat non-linear and uses a lot of hugo template features to support easy text translations. Following is a breakdown of the structure.

  • Template
    • irdest-theme folder contains base HTML and CSS templates. The only page not generated via these files is the root page.
    • The root page template can be found in layouts/index.html
  • Content
    • Markdown section content can be found in the content directory
    • The root page content is content/indemd and the content/home directory (to allow multi-language versions).