# ROSE - RDMA Over SPI Engine
A zero-copy, RDMA-inspired transport layer built over SPI, designed
for embedded systems and affordable hardware experimentation.

ROSE is part of the larger FLORA project, see [link here] for details.

## Table of Contents
- [What is a rose?](#what-is-a-rose)
- [What is ROSE?](#what-is-rose)
- [The goal](#the-goal)
- [Getting started](#getting-started)
  - [Requirements](#requirements)
  - [Deploying the masters/controllers on the SBCs](#deploying-the-masterscontrollers-on-the-sbcs)
  - [Deploying the slaves/peripheral modules on the FPGA
    fabric](#deploying-the-slavesperipheral-modules-on-the-fpga-fabric)
- [Coding in style](#coding-in-style)
- [Protocol specifications](#protocol-specifications)
- [The planning](#the-planning)
- [The action](#the-action)
- [Why ROSE?](#why-rose)
  - [The inspiration](#the-inspiration)
- [Special thanks](#special-thanks)

## What is a rose?
> Roses are elegant. So must be the systems we grow.

This is the underlying philosophy of ROSE since the day the idea
popped into my head.

## What is ROSE?
ROSE is an open-source RDMA-inspired data transfer protocol and engine
built over SPI, originally using Raspberry Pi devices and an FPGA
fabric. It simulates key properties of RDMA — low-latency,
memory-mapped, zero-copy semantics — on affordable, widely accessible
hardware.

## The goal
To build a mock system of real-world scale-up networks with cheap
hardware.  Aiming at better performance than non-RDMA enabled
Ethernet systems.

And as a school project, to explore systems-level design, test
automation and test-driven development cycles, and high-
performance data movement through hardware-software co-design.

## Getting started
### Requirements
1. Some SBCs with SPI interfaces, preferably running a DMA-enabled SPI
   controller (Raspberry Pi's will do just fine)
2. An FPGA with enough pins (5 per device connection)
3. The willingness and courage to tinker with Linux and FPGAs.
4. Some SystemVerilog knowledge (unless you're using the Tang Primer
   20K, which is what I use)
5. A good terminal emulator, a good shell, and a good code editor
   (emacs preferred).
### Deploying the masters/controllers on the SBCs
TBD
### Deploying the slaves/peripheral modules on the FPGA fabric
TBD

## Coding in style
Coding style is important.  Code is for both the human and the
machine, the machine doesn't care about style, but humans do.  A good
style would help development a lot.

See [`style.md`](style.md) for details.

## Protocol specifications
ROSE was designed to embrace newer possibilities as development continues.

See [`protocol.md`](protocol.md) for details.

## The planning
See the plan in [`plan.md`](plan.md).  This file also contains short summaries of
what I did at each step.

Most of ROSE's behaviors and features have been planned *before* the
first source file was even created.  A good plan serves both as a good
guideline and a good reward mechanism.  You'd know early when you're
running into trouble, and you'd know when you've made a solid step in
realizing the project, even if it's simple shift registers to send
back what your device sent.

> Plans turn fear into focus, risk into reach, and steps into a path.

When you dream big, use a plan to ground it with smaller, more
manageable structures.  And most people like it when their dreams come
true.

## The action
See the [`devlog/`](devlog/) directory for a detailed record of the development
process.

Writing down what I did, how I did them, what walls I ran into, and
how I can learn from my mistakes helped me realize this project.

## Why ROSE?
RDMA hardware is quite inaccessible to the average person, at least
when this project started.  I want to show that with a few hundred
dollars and some tinkering, I can build a close simulation of
industrial-grade RDMA networks with lower bandwidth and higher
latency, but still outperforming (in specific use cases) the
Ethernet + TCP/IP stack we use day to day.  The best way to learn the
ways of the industry is to try to build a miniature version in your
garage.

### The inspiration
Before planning the project and laying out the roadmap of weeks and
even months of development, I did a co-op at Nokia as a SR Platform
Testing Dev.  That allowed me to learn how the networks worked, how
data is transmitted from one end of the world to another (I mostly
worked the network layer and data link layer).  That co-op helped me
land me an offer as a Data Center Network Engineer (intern), and also
piqued my interests in the lower levels of how the internet runs as it
is today.

I did my own research on how to quickly get into the world of data
centers, and came across RDMA and InfiniBand and RoCE.  It was love at
first sight.  The idea of accessing a piece of data on a remote
machine as if it was on the requesting machine felt like magic.  Then
came the cold, harsh truth that my PC simply could not run anything
like that due to hardware and software constraints.  So, I sought
another path, picking a good protocol with DMA support and RDMA
potential, and ended up finding SPI.  It's not as powerful as messing
around with PCIe lanes, but more cost-effective and easier to
implement, after all, with the right design, this can easily be
migrated to use SERDES or other connections.

The idea bloomed like a flower, extending its petals, and before I
knew it, I had designed an entire framework with RDMA running on SPI
connections and a test-driven development process (which, with some
more planning, became the foundations of THORN).

But designing is not enough.  Anyone could come up with an idea, some
would even come up with brilliant ideas.  I need something concrete to
keep me anchored to my initiative of learning how this technology
works.  And ROSE came into being.

## Special thanks
I'd like to share my gratitude to ChatGPT and other AI-driven tools in
helping to realize this project.  I didn't use them to write the
actual code, but I used them to explore my ideas, to plan my path, and
to catch anything that I overlooked in the process.  They are powerful
in that way, they can help expand what you have in mind, they can
offer insights into areas that you've never even heard of.  And in
that, I'm thankful to the ever-evolving world of technology, and the
countless researchers and their effort to making us live in a better
world.
