diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..bb6e1cd --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,21 @@ +# Changes + +## 0.1.0, 2026-10-09 + +The first release. + +- Ground Control's RockBLOCK 9704 library v1.0.1, pinned untouched, built + as a Zephyr module with a serial port on the devicetree node's UART. +- One worker thread owns the library; it sleeps until a request or a byte + from the modem arrives. +- ``: begin, end, power, send, info, signal, each with a + timeout; callbacks for a sent message, a received one, the signal and + the provisioning; a status snapshot. +- Provisioning asked again when the library's cached answer is missing a + topic, or after the modem refuses one. +- The `rock7,rockblock-9704` binding with optional power, enable and + booted pins. +- The `rb9704` shell command and the shell sample, with overlays for + native_sim, NUCLEO-U575ZI-Q, XIAO ESP32S3 and ESP32-S3-DevKitC. +- `python/`: the `rb9704` command line, the `rb9704-fake` fake modem, and + tests for the Python binding, native_sim, boards and a real modem. diff --git a/README.md b/README.md new file mode 100644 index 0000000..a5b902d --- /dev/null +++ b/README.md @@ -0,0 +1,180 @@ +# rockblock9704-zephyr + +The RockBLOCK 9704, Ground Control's Iridium Certus modem, on Zephyr: a +module that builds Ground Control's own C library for any board with a +UART, gives it the one thread it needs, and offers a small API, a shell +command and a devicetree binding. With it come host tools in Python and a +fake modem, so the same tests run against Ground Control's Python binding, +against Zephyr on `native_sim`, and against Zephyr on a board. + +The 9704 sends and receives IMT messages of up to 100 kB through the +Iridium network, and speaks JSPR, JSON lines over a 230400 8N1 UART. There +is no AT mode; it is not a 9602 or 9603. + +## Ground Control's library + +This module does not reimplement or fork the protocol. Ground Control's +repository, [rock7/RockBLOCK-9704](https://github.com/rock7/RockBLOCK-9704) +(MIT), is pinned untouched in `west.yml` at `modules/lib/rockblock9704`, +and `CMakeLists.txt` compiles its portable core. What each part does: + +| | Ground Control's library | This module | +| --- | --- | --- | +| JSPR, segmenting, CRC, provisioning check | yes | uses it unchanged | +| Platforms | Linux, Windows, Arduino serial presets | a Zephyr serial port, on the UART in devicetree | +| Threads | none: no locking, shared static buffers | one worker thread owns every call into it | +| API | blocking calls, or async calls with callbacks and `rbPoll()` | requests with a timeout each, callbacks from the worker, a status snapshot | +| Polling | the caller calls `rbPoll()`, one line per call | the worker wakes on a byte from the modem, so an idle modem costs nothing | +| Power pins | `rbBeginGpio` with libgpiod, Linux only | `power-gpios`, `iridium-enable-gpios`, `booted-gpios` in devicetree | +| Python | `rockblock9704` on PyPI, a binding of the C | the `rb9704` command line uses it; the fake modem tests it | + +The library's source shows several things a caller has to work around; +`docs/JSPR.md`, section 8, lists them all. The ones this module handles: + +- **Provisioning is asked once and kept for good**, a miss included. A + modem provisioned after the first send would refuse every topic until + a reboot. Before a send whose topic is not in what the library kept, + and after the modem refuses one, the module clears the library's cache + so it asks again: one free JSPR round trip. +- **The blocking calls stop working once callbacks are registered.** The + module uses the async path only. +- **A query during a send swallows the modem's segment requests**, since + the reader discards lines it is not waiting for. Info and signal + requests are refused with `-EBUSY` while a message is moving. +- **No locking.** Every library call is on the worker thread. + +Others are for upstream to fix, and are listed in `docs/JSPR.md` for that: +`setState` waits for a misspelt target, a line with no status code reads +past its buffer, MT segment data is cut at 1446 characters, and the +Python binding never releases the GIL. + +## Using it + +In the application's `west.yml`; the import brings Ground Control's +library, pinned, at the path the module looks in, and nothing else: + +```yaml + - name: rockblock9704-zephyr + url: https://github.com/SH3D/rockblock9704-zephyr + revision: v0.1.0 + path: modules/lib/rockblock9704-zephyr + import: + name-allowlist: + - rockblock9704 +``` + +In the board overlay, a node on the modem's UART. The pins are optional; +without them the modem is taken to be powered: + +```dts +&usart3 { + current-speed = <230400>; + status = "okay"; + + rockblock { + compatible = "rock7,rockblock-9704"; + power-gpios = <&gpiob 0 GPIO_ACTIVE_HIGH>; /* P_EN */ + iridium-enable-gpios = <&gpiob 1 GPIO_ACTIVE_HIGH>; /* I_EN */ + booted-gpios = <&gpiob 2 GPIO_ACTIVE_HIGH>; /* I_BTD */ + }; +}; +``` + +`CONFIG_RB9704` follows the node. Then ``: + +```c +static void mo_done(uint16_t topic, uint16_t id, int result, void *user) +{ + /* result 0: the network has it */ +} + +static void mt(const struct rb9704_mt *m, void *user) +{ + /* m->data, m->len: copy what you need, it is acknowledged after */ +} + +static const struct rb9704_callbacks cb = { .mo_done = mo_done, .mt = mt }; + +rb9704_set_callbacks(&cb); +rb9704_begin(K_SECONDS(30)); +rb9704_send(RB9704_TOPIC_RAW, buf, len, K_SECONDS(10)); /* costs airtime */ +rb9704_send(RB9704_TOPIC_RAW, buf, len, K_NO_WAIT); /* queues, never blocks */ +``` + +Every request takes a timeout and waits no longer; with `K_NO_WAIT` a send +queues and returns `-EINPROGRESS`, and its outcome, a refusal included, +comes through `mo_done`. Callbacks run on the worker thread. One modem per +system: the library keeps global state. + +| Kconfig | Default | | +| --- | --- | --- | +| `CONFIG_RB9704_MESSAGE_MAX` | 2048 | largest message each way; three static buffers of this size | +| `CONFIG_RB9704_WORKER_STACK_SIZE` | 8192 | measured: the library parks 1.5 kB on the stack in `rbPoll()` | +| `CONFIG_RB9704_WORKER_PRIORITY` | 8 | | +| `CONFIG_RB9704_RX_RING_SIZE` | 8192 | the library's longest JSPR line | +| `CONFIG_RB9704_POWER` | with `power-gpios` | drive the pins | +| `CONFIG_RB9704_SHELL` | with `CONFIG_SHELL` | `rb9704` on the shell | +| `CONFIG_RB9704_UPSTREAM_DIR` | empty | where Ground Control's library is, if not where `west.yml` puts it | + +## The shell + +``` +rb9704 begin | end | power on|off +rb9704 status what is known, without asking the modem +rb9704 info IMEI, serial, versions, ICCID, SIM, temperature +rb9704 signal bars now +rb9704 send | sendhex costs airtime +rb9704 spy on|off every JSPR line, both ways +``` + +`samples/shell` is the shell plus callbacks that print what arrives. + +## Host tools + +`python/` is `rb9704` (the modem and Cloudloop from a computer), `rb9704-fake` +(a fake modem on a pty or a serial port), and the tests. `python/README.md` +has every verb; `docs/CLOUDLOOP.md` is the account, the provisioning and +the order to test a new modem in. + +## Testing + +```sh +python/install.sh +python/.venv/bin/pytest -q python/tests # Python +python/.venv/bin/pytest -q python/tests --zephyr-exe +python/.venv/bin/pytest -q python/tests/test_zephyr.py \ + --board-console /dev/ttyACM0 --board-modem /dev/ttyUSB0 # a board +python/.venv/bin/pytest -q python/tests/test_real.py --real-modem /dev/ttyUSB0 +``` + +`docs/TESTING.md` has the levels, the two-board rig that needs no +USB-serial adapter, and what has passed: the Python level, `native_sim`, +an ESP32-S3 and an STM32U5 against the fake, and free checks against a +real 9704 from Python and from `native_sim`. + +## Boards + +Any board with a free UART. The sample has overlays for `native_sim`, +NUCLEO-U575ZI-Q, Seeed XIAO ESP32S3 and ESP32-S3-DevKitC, and has run on +the first three. + +## Layout + +``` +CMakeLists.txt, Kconfig, zephyr/module.yml the module +dts/bindings/rock7,rockblock-9704.yaml the devicetree binding +include/rb9704/rb9704.h the API +src/ serial port, shims, worker, power, shell +samples/shell/ the shell sample, with board overlays +tests/rig/ the two-board test rig +python/ rb9704, rb9704-fake, the tests +docs/JSPR.md the protocol as the library speaks it +docs/CLOUDLOOP.md account to first message +docs/TESTING.md the tests and how to run them +west.yml the pins +``` + +## Licence + +MIT, as Ground Control's library is. The serial port and worker began in +the OGLAS satellite bench and were carried through WII6 before moving here. diff --git a/docs/CLOUDLOOP.md b/docs/CLOUDLOOP.md new file mode 100644 index 0000000..00b3339 --- /dev/null +++ b/docs/CLOUDLOOP.md @@ -0,0 +1,192 @@ +# Cloudloop and a RockBLOCK 9704, from the box to a message + +How to get a RockBLOCK 9704 from its box to a message seen on the +internet, from a computer over its USB-C port: the account, the token, +topics, provisioning, and the order to test things in. `rb9704` is the +command line in `python/` (`python/README.md` has every verb). + +Written 2026-10-09 from Ground Control's and Cloudloop's documentation +and the library's source. `rb9704 info` and the free JSPR verbs have run +against a real 9704 since; the account, provisioning and airtime steps +have not been tried yet. + +## What there is + +- **The modem.** The 9704 is an Iridium IMT modem, the successor to SBD, + with messages of up to 100 kB. It speaks JSPR, JSON over serial at a + fixed 230400 8N1; there is no AT mode (`JSPR.md`). +- **Its USB-C port.** An FTDI FT234XD (USB id `0403:6015`) bridges it to + the modem's UART, and the port also powers the board. It needs a port + that supplies 500 mA at 4.0 to 5.3 V; an unpowered hub is not supported. + The supercapacitor takes about 40 s to charge from USB. LEDs: red PWR is + input voltage present, green RDY is the modem powered and booted, blue + XMT is transmitting. +- **The library.** Ground Control's C library (MIT) is what this + repository builds into Zephyr, pinned at v1.0.1 in `west.yml`. The + `rockblock9704` Python package is a binding of the same C code, pinned + at 1.0.2 in `python/pyproject.toml`. What works from the host is + therefore the library the firmware runs. +- **Cloudloop Data.** Ground Control's service between the Iridium network + and us. A message the modem sends (MO) lands in Cloudloop, which + delivers it to a destination; a message to the modem (MT) is queued in + Cloudloop through its web interface or its API. + +## The accounts + +There is one account to open, Cloudloop, and one token to ask for. + +### 1. The Cloudloop account and the airtime + +The 9704 arrives with an activation card. + +1. Go to and create an account. The account + holds the billing card and every modem on it, so make it under the + business and an address more than one person reads. +2. Follow the prompts to add the RockBLOCK 9704 from its activation card, + choose a data package and activate the airtime. +3. A **Thing** is made for the modem when it is enrolled. A Thing is + Cloudloop's name for one device, whatever its network, and it has a 32 + character case-sensitive id. `rb9704 cloud things` lists + them once the token below is in place. + +The data packages and their prices change, so they are not copied here; +the Ground Control store and the Cloudloop billing page are where to read +them. Check whether the package charges by the month while the line is +active, and if it does, ask Ground Control how to suspend it for a modem +that will sit unused between bench sessions. + +### 2. A Thing Group + +Destinations attach to Thing Groups and never to a Thing alone. In the +Cloudloop web interface, make a group (say `bench`) and add the +modem's Thing to it. A fleet can be one group per project, so a destination +serves a project. + +### 3. The API token + +The API is how the tool reads messages and sends MT, and how the shore +side will. A token is asked for, not made: + +1. Email for a Cloudloop **platform API token** + for the account. Ask for one that can send MT: `DoSendImtMessage` needs + the Admin permission, reading needs only Read Only. +2. When it arrives, replace it with a fresh one that nobody else has seen, + with `User/DoGenerateToken`, and keep the new one: a token cannot be + recovered once generated. +3. Store it outside any repository, readable by you alone: + + ```sh + mkdir -p ~/.config/rb9704 + printf '%s\n' '' > ~/.config/rb9704/cloudloop.token + chmod 600 ~/.config/rb9704/cloudloop.token + rb9704 cloud ping # {"ping": "pong"} + ``` + + `CLOUDLOOP_TOKEN` in the environment overrides the file. + `CLOUDLOOP_THING` saves typing `--thing` on every cloud verb. + +The token goes in the query string of every call, so it is in any proxy +log on the path; the API is HTTPS only, and nothing here logs it. + +### 4. A destination, later + +On the bench a destination is not needed: every MO is kept in Cloudloop +and `cloud mo` and `cloud msg` read it back. For the shore side, +Cloudloop pushes each MO to an **HTTP webhook** destination: + +- URL: your server's route for messages from the modem. +- Format: **JSON (Lingo)**, the one Cloudloop recommends. The message + bytes are in it base64-encoded, beside the identity and the receive + time. The first MO will show it with `rb9704 cloud msg `, which is + the sample for whoever writes the receiving route. +- Certificate validation on. +- The receiver must answer HTTP 200 within 5 seconds. Cloudloop retries + with exponential backoff, from 2 s, for about 12 hours, from + 35.178.100.117 and 52.56.155.169. Those two addresses are the only + authentication the documentation describes, so the route should check + them and the IMEI against the modems it expects. + +Add the destination to the Thing Group. The web interface shows its +health (grey unknown, green OK, orange some failures, red all failing). + +## Topics + +All IMT traffic goes through a **topic**, a numbered channel. 244 is the +raw topic, the default for everything here. Cloudloop also provisions the +colour topics, 313 purple to 317 yellow, which Cloudloop can route +separately. A modem may only send on topics it has been provisioned +with. The library checks this before it transmits and refuses anything +else, including any id below 64, without spending airtime. + +In the Cloudloop API the same topics have names: `IMT_TOPIC_RAW` for 244, +`IMT_TOPIC_PURPLE` and so on. + +## Provisioning + +A new modem knows no topics. After the airtime is activated it needs a +clear view of the sky for 10 to 30 minutes, or more, while it collects +its provisioning from the network. Indoors this never finishes. Do it +once, outside, with the modem on a laptop: + +```sh +rb9704 --log provision.jsonl provision +``` + +This asks the modem for its signal and its provisioning every 30 s and +stops when topics appear, printing them. It talks JSPR directly, because +the library only asks for provisioning when it is about to send, and +asking this way costs nothing. + +## The order to test in + +Each step needs the one before it. Only the steps marked airtime cost +money; the tool asks before each one, and `--yes` skips the question. + +1. **Install.** `python/install.sh --udev`. The `--udev` rule + keeps ModemManager off the port and names it `/dev/rockblock9704`; it + needs sudo once. +2. **Find it.** Plug the USB-C in, wait for green RDY, then + `rb9704 ports`. The 9704 is the FT234XD, listed first. +3. **Identity, indoors.** `rb9704 info` gives hardware and + firmware versions, serial, IMEI, ICCID, whether the SIM is present and + talking, and the board temperature. Record the IMEI and ICCID against + the Thing in Cloudloop. +4. **Signal, outdoors.** `rb9704 signal --watch 600` reads + the bars (0 to 5) every 5 s for ten minutes and ends with how often + each was seen. Bars move a lot as satellites pass; zero for a while is + normal. +5. **Provision**, as above. +6. **One MO** (airtime). `rb9704 send --text "hello"`. + It should appear in the Cloudloop web interface and in + `rb9704 cloud mo`. Then `cloud msg ` shows the Lingo + form, which is the sample for the receiving route. +7. **One MT** (airtime). With `rb9704 listen --out mt` running beside + the modem, `rb9704 cloud mt --text "ping"` from any machine. The MT + arrives in `mt/` and `cloud status ` reports + its delivery. The modem has no store for MT: one that arrives while + nothing is listening is lost, so listen first. +8. **A session** (airtime). Queue two or three MT with `cloud mt`, then + `rb9704 --log session.jsonl session --text status --quiet 90`. It + sends one MO and collects MT until 90 s pass with nothing new. The log + says how long after an MO the queued MT arrive and how far apart, and + so what quiet window a firmware needs after it sends. + +`--log` on any verb appends one JSON line per event, with UTC times, for +the bench log. + +## When begin fails + +`begin()` failing means the library could not finish the setup (API +version, SIM, operational state). In order of likelihood: + +- **The port is held.** ModemManager probes a new USB serial port for + a few seconds after it appears, and any open terminal on the port + blocks it. The `--udev` rule stops the first. +- **Not in `dialout`.** `install.sh` checks; `sudo usermod -aG dialout + $USER` and log in again. +- **Not booted.** No green RDY: the port gives too little current (a hub, + a front-panel port), or the supercapacitor is still charging. +- **Something odd.** `rb9704 jspr --setup 'GET hwInfo {}' + 'GET simStatus {}'` shows the modem's own answers; a 4xx code names + the problem (`409` unauthorized, `410` SIM not configured, `418` not + provisioned). diff --git a/docs/JSPR.md b/docs/JSPR.md new file mode 100644 index 0000000..e98b354 --- /dev/null +++ b/docs/JSPR.md @@ -0,0 +1,835 @@ +# JSPR as Ground Control's library speaks it + +JSPR is the RockBLOCK 9704's serial protocol: JSON lines at 230400 8N1. +This is what Ground Control's C library (v1.0.1, commit `ab37e8a`, the pin +in `../west.yml`) sends and what it expects back, taken from its source +with a file and line for each claim. Paths are relative to that library's +repository. It is what `python/rb9704tools/fakemodem.py` implements, and +section 8 is the list of the library's surprises, several of which this +module works around (README, "Ground Control's library"). + +It describes the library, not the modem: where the two might differ (what +a real modem sends unprompted, how large its MT segments are) the source +cannot say, and the text says so. Field names the library never reads are +marked as placeholders. + +"Library" means the C library. The Python binding (`rockblock9704/rockblock.c`, +`rockblock9704/rockblock9704.py`) is a thin wrapper over the same C files; it +is built without `KERMIT` (`CMakeLists.txt:88-109`), so firmware update is not +in it. + +Notation: `>>` is a line the library writes to the modem, `<<` a line the modem +(or a fake) writes. `\r` is shown where it matters. + +--- + +## 1. Line framing + +### 1.1 Requests (library to modem) + +- Format: ` \r`. VERB is `GET` or `PUT`, one space each + side of the target, JSON always present (`{}` for a bare GET). +- Terminator is a single CR, no LF. Every format string ends `\r` + (`src/jspr_command.c:24,40,89,106,166,184,207,236,257,272,286,325,350,370,386,407`). +- Built with `snprintf` into a 2048-byte buffer (`src/jspr_command.h:12`, + `src/jspr_command.c:8`). The spacing inside the JSON is not uniform between + commands (some `"k": v`, some `"k":v`), so parse the JSON, do not string-match it. +- Written with one `write()` loop until the whole line has gone + (`src/serial_presets/serial_linux/serial_linux.c:188-228`). + +### 1.2 Responses and unsolicited lines (modem to library) + +Format: `<3-digit code> \r` + +Reader: `receiveJspr()` (`src/jspr.c:35-160`). + +- Reads one byte at a time (`src/jspr.c:59`). Each byte read is a `select()` + with a 0.5 s timeout (`serial_linux.c:153-166`). If no byte arrives within + 0.5 s the read returns -1 and `receiveJspr` gives up for this call + (`src/jspr.c:60-64`). So: + - the first byte of a reply must arrive within 0.5 s of the request; + - a pause of 0.5 s or more inside a line loses the bytes already read + (they were consumed and the call returns false). Write each line in one go. +- **Terminator is CR** (`'\r'`), and only when it is at index > 2 of the + line (`src/jspr.c:65`). LF is not a terminator and is not stripped at the + end; a trailing LF becomes the first byte of the *next* line, where the + code-scan below skips it (atoi skips leading whitespace). Safest: end every + line with `\r` only. `\r\n` also works in practice. +- Lines shorter than 9 bytes before the CR (`JSPR_MIN_RESPONSE`, + `src/jspr.h:18`, test at `src/jspr.c:85`) are not parsed and not discarded: + `pos` is not reset, so the next byte overwrites the NUL and the short line + is glued onto the front of the next one. Do not send short lines. +- Maximum line: 8191 bytes (`RX_BUFFER_SIZE 8192`, `src/jspr.h:13`, + `src/jspr.c:57,74`). A line that fills the buffer without CR ends the call. +- **Code**: the reader scans 3-character windows from the start of the line, + `atoi()` on each, until it finds a value in 200..500 inclusive + (`src/jspr.c:89-108`); leading junk (the source mentions a DC1 before + `bootInfo`, `src/jspr.c:87-88`) is then cut off with `memmove` + (`src/jspr.c:110-113`). The code window is the first 3 bytes after that. +- **Target**: from byte 4 (after `NNN `) up to the next space + (`src/jspr.c:115-127`). No space after the target means the call returns + false (`src/jspr.c:117-121`). The target is copied only if shorter than 30 + characters (`JSPR_MAX_TARGET_LENGTH`, `src/jspr.h:16`, `src/jspr.c:123`). +- **JSON**: from the first `{` after the code to the end of the line + (`src/jspr.c:140-151`). No `{` means the call returns false. JSON of 3500 + characters or more (`JSPR_MAX_JSON_LENGTH`, `src/jspr.h:20`) is left empty + while the call still returns true, so the parse that follows fails. + Always send a JSON object, `{}` at minimum. + +### 1.3 Matching a response to a request + +There is no request id on the wire. Matching is by target name only: + +- `receiveJspr(&response, "target")`: every complete line whose target is + not exactly `target` is **thrown away** and reading continues + (`src/jspr.c:129-138`). The first line with the right target is returned, + whatever its code. The caller then checks the code (almost always `== 200`). +- `waitForJsprMessage(&response, target, code, seconds)` + (`src/jspr.c:162-188`): calls `receiveJspr` with the target filter in a loop + until a line has both the target and the code, or the timeout passes + (checked after each `receiveJspr`, then `delay(10)`). Lines with the right + target and the wrong code are consumed and lost. +- `receiveJspr(&response, NULL)` (only from `rbPoll`, `src/rockblock_9704.c:716`): + accepts any target, returns one line. + +Consequence for a fake: while the library is waiting on a reply to a request, +**any other line you send (including `299 messageTerminate`, `299 +messageOriginateSegment`, `299 constellationState`) is silently discarded.** +Send the solicited `200` line first and unsolicited lines after it. + +### 1.4 Unsolicited lines + +Recognised by code `299` (`JSPR_RC_UNSOLICITED_MESSAGE`, `src/jspr.h:40`) +and the target, in `rbPoll()` (`src/rockblock_9704.c:704-906`). `rbPoll` does +anything only if `serialPeek()` (`FIONREAD`, `serial_linux.c:230-241`) +reports bytes waiting (`src/rockblock_9704.c:714`), and then reads **one +line per call**. Targets it acts on: + +| Code | Target | Condition | Section | +| --- | --- | --- | --- | +| 299 | `messageOriginateSegment` | an MO is at the head of the queue | 4 | +| not 200, not 299 | `messageOriginateSegment` | an MO is at the head | 4 | +| 299 | `messageOriginateStatus` | an MO is at the head | 4 | +| 299 | `messageTerminate` | always | 5 | +| 299 | `messageTerminateSegment` | always | 5 | +| 299 | `messageTerminateStatus` | always | 5 | +| 299 | `constellationState` | always | 6 | + +Every other line read by `rbPoll` (any `200` reply, `299 operationalState`, +`299 simStatus`, `299 messageProvisioning`, `299 bootInfo`, ...) is consumed +and ignored. + +### 1.5 Response codes + +`src/jspr.h:37-63`. Only 200 means success anywhere in the library; 299 is +unsolicited. Error codes the enum names: 400 api version not selected, +401 unsupported request type, 402 configuration already set, 403 command too +long, 404 unknown target, 405 command malformed, 406 operation not allowed, +407 bad JSON, 408 request failed, 409 unauthorised, 410 SIM not configured, +411 wake xcvr in invalid, 412 invalid channel, 413 invalid action, +414 hardware not configured, 415 invalid radio path, 416 crash dump not +available, 417 feature not supported by hardware, 418 not provisioned, +419 invalid transmit power, 420 invalid burst type, 500 serial port error. +None of them is handled individually; any non-200 is a failure. + +### 1.6 Serial setup (matters for a pty) + +- 230400 8N1 (`src/rockblock_9704.h:119`). A pty ignores the speed. +- `setContextLinux` opens and immediately closes the port as a test + (`serial_linux.c:27-33`), then `rbBegin` opens it again + (`src/rockblock_9704.c:277-281`). On a pty the master will see `EIO` on + read while no slave fd is open; the fake must tolerate `EIO` and keep the + master open. +- Opened `O_RDWR | O_NOCTTY | O_SYNC | O_NONBLOCK` (`serial_linux.c:41`). + Termios: CS8, no parity, 1 stop, `ICRNL`/`IXON`/`IXOFF`/`IXANY` off, + `ICANON`/`ECHO`/`ECHOE`/`ISIG` off (`serial_linux.c:90-98`). `c_oflag` is + untouched (the library never writes LF, so `ONLCR` does not matter). +- Immediately after opening, `clearLeftoverData()` drains whatever is already + waiting (`src/rockblock_9704.c:135-151,283`). Lines sent before `rbBegin` + are lost. + +--- + +## 2. Every request, and the reply the library expects + +All replies are `200 {...}` unless noted. Fields not listed are +ignored. "valueint" means a JSON number read through cJSON's int. + +### 2.1 GET apiVersion + +``` +>> GET apiVersion {}\r +<< 200 apiVersion {"supported_versions":[{"major":1,"minor":0,"patch":0}],"active_version":{"major":1,"minor":0,"patch":0}}\r +``` +Source: `src/jspr_command.c:21-33`, parse `src/jspr.c:281-338`. + +- `supported_versions`: array of objects with numeric `major`, `minor`, + `patch`. At most 2 are kept, **taken from the end of the array backwards**: + `supportedVersions[0]` is the last element (`src/jspr.c:294-312`). +- `active_version`: object with numeric `major`, `minor`, `patch`. Present + and complete means `activeVersionSet = true`; absent (not an object) means + false (`src/jspr.c:314-331`). +- The version values are not checked against anything. + +### 2.2 PUT apiVersion + +Sent only if the GET reply had no `active_version` (`src/rockblock_9704.c:171-175`). +``` +>> PUT apiVersion {"active_version": {"major": 1, "minor": 0, "patch": 0}}\r +<< 200 apiVersion {"active_version":{"major":1,"minor":0,"patch":0}}\r +``` +Source: `src/jspr_command.c:35-56`. The version sent is the last element of +the GET's `supported_versions`. Only the code 200 and target are checked; +the JSON is not parsed. + +### 2.3 GET simConfig (the "simInterface" getter) + +``` +>> GET simConfig {}\r +<< 200 simConfig {"interface":"internal"}\r +``` +Source: `src/jspr_command.c:86-98`, parse `src/jspr.c:430-473`. +`interface` string, one of `none`, `local`, `remote`, `internal`. + +### 2.4 PUT simConfig + +Sent if the GET did not say `internal` (`src/rockblock_9704.c:200-215`). +The library always asks for `internal` (`src/rockblock_9704.c:202`, +`src/jspr_command.c:58-84`). +``` +>> PUT simConfig {"interface": "internal"}\r +<< 200 simConfig {"interface":"internal"}\r +<< 299 simStatus {"card_present":true,"sim_connected":true,"iccid":"8988169712345678901"}\r +``` +The library needs the `200 simConfig` and then, **within 1 s**, a +`299 simStatus` line (`src/rockblock_9704.c:203-213`). Order matters: a +`299 simStatus` sent before the `200 simConfig` is discarded by the target +filter. The JSON of the `299 simStatus` is not parsed here. + +### 2.5 GET operationalState + +``` +>> GET operationalState {}\r +<< 200 operationalState {"state":"active","reason":0}\r +``` +Source: `src/jspr_command.c:163-176`, parse `src/jspr.c:475-541`. + +- `state` string: `inactive`, `active`, `cal_test`, `hw_self_test`, + `rf_scan`, `loopback`, `fault`. +- `reason` number 0..7 (normal, hw self test failure, temperature fault, + RF power protection fault, VAM app failed ack, invalid hardware, low supply + voltage, mfrtest used incorrectly; `src/jspr.h:141-151`). Optional. + +### 2.6 PUT operationalState + +``` +>> PUT operationalState {"state": "active"}\r +<< 200 operationalState {"state":"active"}\r +``` +Source: `src/jspr_command.c:123-200`. States written: `inactive`, `active` +(others exist in `putOperationalState` but the library only uses these two). +In `rbBegin` only the 200 is needed. In `rbResyncServiceConfig` the library +instead waits for a `299 operationalState` carrying the new state (2.12). + +### 2.7 GET simStatus (card present, SIM connected, ICCID) + +``` +>> GET simStatus {}\r +<< 200 simStatus {"card_present":true,"sim_connected":true,"iccid":"8988169712345678901"}\r +``` +Source: `src/jspr_command.c:367-379`, parse `src/jspr.c:1048-1078`, callers +`src/rockblock_9704.c:983-1026` (`rbGetCardPresent`, `rbGetSimConnected`, +`rbGetIccid`; each sends its own GET). + +- `card_present` bool, `sim_connected` bool (JSON `true`/`false`; numbers are + ignored). +- `iccid` string, **truncated to 19 characters** (`JSPR_ICCID_MAX_LENGTH 20` + including the NUL, `src/jspr.h:35`, `src/jspr.c:1071`). + +### 2.8 GET hwInfo (IMEI, serial number, hardware version, board temperature) + +``` +>> GET hwInfo {}\r +<< 200 hwInfo {"hw_version":"1.0","serial_number":"123456","imei":"300434030000000","board_temp":27}\r +``` +Source: `src/jspr_command.c:283-295`, parse `src/jspr.c:1009-1046`, callers +`src/rockblock_9704.c:927-981` (`rbGetImei`, `rbGetHwVersion`, +`rbGetSerialNumber`, `rbGetBoardTemp`; each sends its own GET). + +| Field | Type | Kept | +| --- | --- | --- | +| `hw_version` | string | first 6 chars (`src/jspr.h:31`) | +| `serial_number` | string | first 6 chars (`src/jspr.h:32`) | +| `imei` | string | first 15 chars (`src/jspr.h:33`) | +| `board_temp` | number | cast to int8 (-128..127) | + +`rbGetBoardTemp` returns -100 on failure (`src/rockblock_9704.c:974`). + +### 2.9 GET firmware (firmware version) + +``` +>> GET firmware {"slot": "primary"}\r +<< 200 firmware {"slot":"primary","validity":1,"version":{"major":1,"minor":2,"patch":3,"build_info":"fake"},"hash":"0000000000000000000000000000000000000000000000000000000000000000"}\r +``` +Source: `src/jspr_command.c:317-340`, parse `src/jspr.c:340-428`, caller +`src/rockblock_9704.c:1028-1062`. `rbGetFirmwareVersion` returns +`"v.."` (`src/rockblock_9704.c:1051`). + +- `slot` must be a **string** if present: the parser calls `strcmp` on + `valuestring` without a type check (`src/jspr.c:366-377`), so a non-string + crashes the host. The same holds for `version.build_info` and `hash` + (`stpncpy` on `valuestring`, `src/jspr.c:409,417`). +- `validity` is read as `valueint > 0` (`src/jspr.c:382`). cJSON does not set + `valueint` for `true`, so JSON `true` reads as false; send `1` if it matters. + Nothing in the public API reads `validity`. +- `build_info` kept to 49 chars, `hash` to 64. +- On a non-200 the library prints `Failed\n` to stdout (`src/rockblock_9704.c:1042`). + +### 2.10 GET constellationState (signal) + +``` +>> GET constellationState {}\r +<< 200 constellationState {"constellation_visible":true,"signal_bars":4,"signal_level":-105}\r +``` +Source: `src/jspr_command.c:254-266`, parse `src/jspr.c:753-788`, caller +`src/rockblock_9704.c:908-925` (`rbGetSignal`). + +- Waits up to 1 s for code 200 with this target (`src/rockblock_9704.c:912`); + `299 constellationState` lines arriving meanwhile are consumed and ignored. +- `constellation_visible` bool; `signal_level` number, read only when + visible is true; `signal_bars` number 0..5 (out of range ignored). +- Returns `signal_bars`, or -1 on failure. + +### 2.11 GET messageProvisioning + +Sent by `checkProvisioning()` the first time any send is attempted, then +never again in the life of the process (cached in a global, +`src/rockblock_9704.c:40,1150-1206`). `rbBegin` does not send it and does not +clear the cache. +``` +>> GET messageProvisioning {}\r +<< 200 messageProvisioning {"provisioning":[{"topic_id":244,"topic_name":"RAW","priority":"Low","discard_time_seconds":604800,"max_queue_depth":8}]}\r +``` +Source: `src/jspr_command.c:269-281`, parse `src/jspr.c:934-1007`. + +- `provisioning`: array, at most 20 entries used (`JSPR_MAX_TOPICS`, + `src/jspr.h:29`). `topicCount` is set to the array size even above 20. +- Per entry: `topic_id` number 64..65535 (below 64 is ignored and the entry's + id is left as stack garbage); `topic_name` string shorter than 57 chars; + `priority` string, one of `Safety-1`, `Safety-2`, `Safety-3`, `High`, + `Medium`, `Low`. +- The struct has `discardTimeSeconds` and `maxQueueDepth` (`src/jspr.h:259-266`) + but **the parser never reads them**; the field names above are placeholders, + not known from the source. +- `provisioningSet` becomes true whenever the JSON parses at all + (`src/jspr.c:1001`), with or without the array. The library caches the + result even when the requested topic is missing, so a fake cannot change + provisioning later without restarting the host process. +- The `messageProvisioning` callback (Python: `set_message_provisioning_callback`) + fires only here, after the GET (`src/rockblock_9704.c:1181-1187`). + +### 2.12 PUT serviceConfig (resync), with its state dance + +`rbResyncServiceConfig()` (`src/rockblock_9704.c:1064-1120`): + +``` +>> GET operationalState {}\r +<< 200 operationalState {"state":"active","reason":0}\r (within 1 s) +>> PUT operationalState {"state": "inactive"}\r +<< 200 operationalState {"state":"inactive"}\r (consumed and ignored) +<< 299 operationalState {"state":"inactive","reason":0}\r (required, within 1 s) +>> PUT serviceConfig {"resync": true}\r +<< 200 serviceConfig {}\r (required, within 1 s) +>> PUT operationalState {"state": "active"}\r +<< 200 operationalState {"state":"active"}\r (consumed and ignored) +<< 299 operationalState {"state":"active","reason":0}\r (required, within 1 s) +``` +If the modem was already inactive, the first PUT and the last two exchanges +are skipped and the call succeeds on `200 serviceConfig`. Any other state +fails without sending anything more. Request text: `src/jspr_command.c:381-400`. + +### 2.13 PUT messageOriginate, PUT messageOriginateSegment, PUT messageOriginateStatus + +See section 4. Request text: `src/jspr_command.c:201-252,402-421`. + +### 2.14 GET/PUT firmware for update (C only, `KERMIT` builds) + +`rbUpdateFirmware` (`src/rockblock_9704.c:1208-1372`) does the +`GET operationalState` / `PUT operationalState inactive` / wait for +`299 operationalState` sequence of 2.12, then +`>> PUT firmware {"slot": "primary"}\r`, needs `200 firmware {...}` and +then switches the line to Kermit. Not in the Python binding; out of scope +for the fake. + +### 2.15 What the library never sends + +There is no request for messageTerminate, messageTerminateSegment or +messageTerminateStatus, no MT acknowledgement of any kind, and no +`GET bootInfo`. `parseJsprBootInfo` exists (`src/jspr.c:198-279`) and is +never called. + +--- + +## 3. rbBegin + +`rbBegin(port)` (`src/rockblock_9704.c:273-300`); the Arduino one is the same +(`src/rockblock_9704_arduino.cpp:9-26`): + +1. Open, close, open the port (1.6). Drain input. +2. `setApi()` (`src/rockblock_9704.c:153-186`), up to 2 attempts, each after + a 5 ms sleep: + ``` + >> GET apiVersion {}\r + << 200 apiVersion {"supported_versions":[{"major":1,"minor":0,"patch":0}],"active_version":{"major":1,"minor":0,"patch":0}}\r + ``` + Succeeds if the reply is 200 and `active_version` is present. If + `active_version` is absent it sends `PUT apiVersion` (2.2) and succeeds on + a `200 apiVersion`. +3. `setSim()` (`src/rockblock_9704.c:188-224`): + ``` + >> GET simConfig {}\r + << 200 simConfig {"interface":"internal"}\r + ``` + Succeeds at once if `internal`. Otherwise PUT, `200 simConfig`, then + `299 simStatus` within 1 s (2.4). No retry. +4. `setState()` (`src/rockblock_9704.c:226-271`): + ``` + >> GET operationalState {}\r + << 200 operationalState {"state":"active","reason":0}\r + ``` + - `active`: success. + - `inactive`: `>> PUT operationalState {"state": "active"}\r`, needs + `200 operationalState`. A `299 operationalState` sent **before** that + 200 is taken as the reply (same target) and fails the begin, because + its code is not 200. Send the 200 first. + - any other state: `PUT ... "inactive"`, needs `200 operationalState`, + then `PUT ... "active"` and reads the reply with the misspelt target + `"operationState"` (`src/rockblock_9704.c:259`). No real line matches, + so this path always fails (see 8). + - `state` missing or not a string: fails. +5. `imtQueueInit()` empties both queues (`src/imt_queue.c:170-201`). + +Minimal fake for a clean begin: answer the three GETs with `active_version` +set, `"interface":"internal"`, and `"state":"active"`. That is three +request/response pairs and nothing unsolicited. + +--- + +## 4. Mobile originated (MO) + +### 4.1 checkProvisioning (all send functions start with it) + +`src/rockblock_9704.c:1150-1206`. + +- Topic must be 64..65535 (`IMT_MIN_TOPIC_ID`/`IMT_MAX_TOPIC_ID`, + `src/rockblock_9704.c:20-21`). Below 64 fails with nothing on the wire. +- First call: `GET messageProvisioning` (2.11). Needs `200` and the target. + The topic must appear as a `topic_id` in the `provisioning` array. +- Later calls: checked against the cache only; nothing on the wire. +- Failure means the send returns false and nothing else is sent. + +`rbSendMessage` (Python `send_message(msg)` with no topic) uses topic 244 +(`RAW_TOPIC`, `src/rockblock_9704.h:160`, `src/rockblock_9704.c:347`). +Other named Cloudloop topics: 313..317 (`src/rockblock_9704.h:163-175`). + +### 4.2 Payload and CRC + +- Payload length 1..100000 bytes on Linux (`IMT_PAYLOAD_SIZE 100000U + + IMT_CRC_SIZE`, `src/imt_queue.h:36-38`; check at + `src/rockblock_9704.c:353,375,397,595`). 5000 on Arduino + (`src/imt_queue.h:31-34`). Both overridable at build time + (`CMakeLists.txt:14-20`). +- A 2-byte CRC is appended to the payload in the library's buffer + (`appendCrc`, `src/rockblock_9704.c:331-341`): + CRC-16 with polynomial 0x1021, initial value 0, no reflection, no final + XOR (table at `src/rockblock_9704.c:99-133`, loop at + `src/rockblock_9704.c:1122-1137`). This is CRC-16/XMODEM. Appended **high + byte first**. + - Check value: `"123456789"` gives `0x31C3`. + - `"hello"` gives `0xC362`, so the 7 bytes sent are + `68 65 6c 6c 6f c3 62`, base64 `aGVsbG/DYg==`. +- `message_length` on the wire is payload + 2. + +### 4.3 The exchange + +Example: `rbSendMessageAsync(244, "hello", 5)`, first send of the process. + +``` +>> GET messageProvisioning {}\r +<< 200 messageProvisioning {"provisioning":[{"topic_id":244,"topic_name":"RAW","priority":"Low"}]}\r +>> PUT messageOriginate {"topic_id":244, "message_length":7, "request_reference":1}\r +<< 200 messageOriginate {"topic_id":244,"request_reference":1,"message_id":17,"message_response":"message_accepted"}\r + ... library returns; from here on it only reacts inside rbPoll() ... +<< 299 messageOriginateSegment {"topic_id":244,"message_id":17,"segment_length":7,"segment_start":0}\r +>> PUT messageOriginateSegment {"topic_id":244, "message_id":17, "segment_length":7, "segment_start":0, "data":"aGVsbG/DYg=="}\r +<< 200 messageOriginateSegment {"topic_id":244,"message_id":17,"segment_length":7,"segment_start":0}\r (ignored by the library) +<< 299 messageOriginateStatus {"topic_id":244,"message_id":17,"final_mo_status":"mo_ack_received"}\r +``` + +Step by step: + +1. **PUT messageOriginate** (`src/jspr_command.c:201-228`). `request_reference` + is a global counter starting at 1, incremented after every PUT + messageOriginate and wrapped from 101 to 1 (`src/jspr.c:16`, + `src/jspr_command.c:212-216`). +2. **Reply** read with `receiveJspr(..., "messageOriginate")` + (`src/rockblock_9704.c:429,562`), so it must come within 0.5 s and before + any `299` line. Parse: `src/jspr.c:543-602`. + - `topic_id` 64..65535, `request_reference` 1..100 (both stored, neither + checked against the request). + - `message_response` string: `message_accepted`, `subscription_invalid`, + `message_discarded_on_overflow`. + - `message_id` 0..255, read **only** when `message_response` is + `message_accepted`. + - **Code 200 alone counts as started** (`src/rockblock_9704.c:431-436, + 564-569`); `message_response` is never tested. Always send + `message_accepted` with a `message_id` on 200, or the library keeps a + garbage id. To refuse a message send a non-200 code, e.g. + `406 messageOriginate {}`, and the library drops it. + - The `moMessageStarted(id)` callback fires here in the async path + (`src/rockblock_9704.c:571-574`). +3. **Segment request**, unsolicited, from the modem: + `299 messageOriginateSegment` with `topic_id`, `message_id`, + `segment_length` (1..1446), `segment_start` (0..100001) + (`src/jspr.c:604-650`). The library answers only if both `message_id` and + `topic_id` match the head of its MO queue (`src/rockblock_9704.c:721-727`). + Out-of-range values leave the field as stack garbage. +4. **Segment reply** from the library (`src/rockblock_9704.c:728-739`, + `src/jspr_command.c:230-252`): the bytes + `buffer[segment_start .. segment_start+segment_length)` of payload+CRC, + standard base64 with padding and no line breaks (vendored mbedtls, + `src/third_party/base64/base64.c`), into a 2048-byte buffer + (`BASE64_TEMP_BUFFER`, `src/rockblock_9704.h:112`). 1446 bytes encode to + 1928 characters, which fits. The library does **not** wait for a reply to + this PUT; it is fire and forget. + - The segment size is chosen by the modem. The library imposes the + 1..1446 range only. Nothing in the library computes segments; the fake + must request every range covering `0 .. message_length` itself, in any + order, one `299` per segment. Each request is served by one `rbPoll`. + - The library does not check that the range lies within the message. + Asking past `message_length` returns stale buffer bytes (zeros after + init). +5. **Reply to the segment PUT**: a `200 messageOriginateSegment` is read by a + later `rbPoll` and ignored. Any code other than 200 and 299 with target + `messageOriginateSegment` and a matching `message_id` **fails the MO** + (`src/rockblock_9704.c:742-761`): `moMessageComplete(id, -1)` or the + sync failure flag, and the message is dropped. To exercise this: + `<< 405 messageOriginateSegment {"topic_id":244,"message_id":17}\r` + (it must carry `message_id`; the topic is not checked). +6. **Final status**, unsolicited: `299 messageOriginateStatus` with + `message_id` matching the head (topic not checked) and `final_mo_status` + (`src/jspr.c:790-880`, `src/rockblock_9704.c:762-795`). + - `mo_ack_received` is the only success. + - Failures: `message_discarded_on_overflow`, `message_expired`, + `message_transfer_timeout`, `segment_not_supplied`, `segment_incorrect`, + `network_error`, `message_cancelled_pre_transit`, + `message_cancelled_in_transit`, `subscription_invalid`, + `protocol_error`, `message_dropped_local_crc_error`, + `crc_error_in_transfer`, `user_supplied_crc_error`. + - Either way the message leaves the queue, and the next queued async MO + (if any) is started at once with its own `PUT messageOriginate` + (`checkMoQueue`, `src/rockblock_9704.c:691-702`). + +The library never checks a CRC on the MO side itself; a fake that wants to +emulate `user_supplied_crc_error` should reassemble the segments, check the +last two bytes as CRC-16/XMODEM big-endian over the rest, and choose the +final status. + +### 4.4 rbSendMessageAsync (Python `send_message_async(msg, topic)`) + +`src/rockblock_9704.c:589-617`, `548-587`. + +- Returns after the `200 messageOriginate`. Segments and status are handled + only when the caller runs `rbPoll()` (Python `poll()`), one line per call. + The upstream example polls every 10 ms (`examples/python/async_send_receive.py`). +- Outcome arrives through `moMessageComplete(id, 1 | -1)`. +- The MO queue holds 1 message (`IMT_QUEUE_SIZE 1`, `src/imt_queue.h:17-19`) + and starts **locked** (`moLock = true`, `src/imt_queue.c:10`), so a second + async send while one is in flight fails at once with nothing on the wire. +- The Python wrapper returns `None` and sends nothing if `topic` is omitted + (`rockblock9704/rockblock9704.py:114-122`). + +### 4.5 rbSendMessage / rbSendMessageAny (Python `send_message(msg[, topic][, timeout=30])`) + +`src/rockblock_9704.c:343-363`, `387-407`, `409-470`. + +- Same wire exchange. After the `200 messageOriginate` the library spins on + `rbPoll()` with no sleep until success, failure or `timeout` seconds + measured from before the PUT (`src/rockblock_9704.c:413,437-457`). +- The fake must deliver segment requests and the final status within that + timeout, unprompted, because the library sends nothing in the meantime + except segment PUTs. +- Success is seen only through a flag that is set **only when no + `moMessageComplete` callback is registered** (`src/rockblock_9704.c:771-778`). + See 8. + +--- + +## 5. Mobile terminated (MT) + +### 5.1 The exchange + +Everything is unsolicited from the modem. **The library sends nothing at +all for an MT**: no request, no segment acknowledgement, no final ack. +`rbAcknowledgeReceiveHeadAsync` only removes the message from the +library's own queue (`src/rockblock_9704.c:681-689`, `src/imt_queue.c:149-168`). + +Example: delivering `"hello"` on topic 244 as message id 5. + +``` +<< 299 messageTerminate {"topic_id":244,"message_id":5,"message_length_max":7}\r +<< 299 messageTerminateSegment {"topic_id":244,"message_id":5,"segment_length":7,"segment_start":0,"data":"aGVsbG/DYg=="}\r +<< 299 messageTerminateStatus {"topic_id":244,"message_id":5,"final_mt_status":"complete"}\r +``` + +1. **`299 messageTerminate`** (`src/jspr.c:652-690`, `src/rockblock_9704.c:798-818`): + - `topic_id` 64..65535, `message_id` 0..255, `message_length_max` 3..100002. + - The library adds an entry to its MT queue (one slot). If the queue is + full and **unlocked** (the default, `mtLock = false`, + `src/imt_queue.c:9`) the old entry is dropped to make room + (`src/imt_queue.c:39-60`). If locked (`rbReceiveLockAsync`) the new MT is + refused and `mtMessageComplete(id, -1)` fires. + - `message_length_max` is used only as "length > 0" at this point; the + final length comes from the segments. +2. **`299 messageTerminateSegment`** (`src/jspr.c:692-751`, + `src/rockblock_9704.c:819-850`), zero or more: + - Applied to the **last** MT queued, and only if its `message_id` + matches. Topic not checked. + - `segment_start` 0..100001, `segment_length` 1..1446. + - `data`: standard base64 of the segment bytes. Decoded straight into the + message buffer at `segment_start`, with `segment_length` as the output + limit. If the decoded size exceeds `segment_length`, or the base64 has + an invalid character, the MT fails (`mtMessageComplete(id, -1)`) and is + removed. + - **The `data` string is truncated to 1446 characters** before decoding + (`JSPR_MAX_SEGMENT_LENGTH 1447`, `src/jspr.h:21`, `src/jspr.c:736-744`). + So in practice keep each MT segment to **at most 1083 bytes** (1444 + base64 characters), even though the parser accepts `segment_length` up to + 1446. A 1446-byte segment (1928 characters) would be silently cut and + decoded short. This is the library's limit; whether a real modem sends + larger MT segments is not answerable from the source. + - Each segment adds its `segment_length` to a running total + (`messageLengthAsync`, `src/rockblock_9704.c:834`). Do not repeat or + overlap segments, the total would be wrong. +3. **`299 messageTerminateStatus`** (`src/jspr.c:882-932`, + `src/rockblock_9704.c:851-892`): + - `message_id` must match the last MT. `final_mt_status`: `complete` + (success), `message_timed_out`, `message_cancelled`, + `crc_error_in_transfer`. + - On `complete`: message length = sum of `segment_length`s, counter reset, + `ready = true`, then `mtMessageComplete(id, 1)` or the sync flag. + - On anything else: `mtMessageComplete(id, -1)` or the sync failure flag; + the entry stays in the queue and the running total is **not** reset. + +### 5.2 CRC on MT + +The library treats the **last 2 bytes of the reassembled MT as a CRC and +strips them** without checking them (`src/rockblock_9704.c:484-485, +506-507, 650-651`). So the fake must send payload + 2 bytes. For realism +use the same CRC-16/XMODEM, high byte first, as MO (`"hello"` gives +`aGVsbG/DYg==` as above). The library writes a NUL over the first CRC byte +and returns `length - 2` bytes. Total length below 2 underflows; keep MTs to +at least 1 byte of payload (the parser's minimum `message_length_max` is 3). + +### 5.3 rbReceiveMessageAsync and rbAcknowledgeReceiveHeadAsync + +- `rbReceiveMessageAsync(&buf)` (`src/rockblock_9704.c:636-659`, Python + `receive_message_async()` with no topic): does not touch the serial line. + Returns the head MT if `ready`, otherwise 0. It can be called repeatedly and + returns the same message until it is acknowledged. +- `rbAcknowledgeReceiveHeadAsync()` (Python `acknowledge_receive_head_async()`): + local only, zeroes and removes the head. Nothing is sent to the modem. +- The lines themselves are processed by `rbPoll()`, one per call. + +### 5.4 rbReceiveMessage (sync, Python `receive_message()`) + +`src/rockblock_9704.c:472-546`. Calls `rbPoll()` once. If the head MT is +waiting for segments (`readyToProcess`), it then spins on `rbPoll()` **with +no timeout** until the MT succeeds or fails. Otherwise it returns 0 at once. +So the usual pattern is the caller looping on `receive_message()`; the call +that reads the `299 messageTerminate` then blocks until the status line. + +`rbReceiveMessageWithTopic(&buf, topic)` (`src/rockblock_9704.c:494-515`) +does not filter by topic; the parameter is assigned to and discarded. + +--- + +## 6. Unsolicited constellationState and messageProvisioning + +### 6.1 `299 constellationState` + +``` +<< 299 constellationState {"constellation_visible":true,"signal_bars":3,"signal_level":-110}\r +``` +Handled in `rbPoll` (`src/rockblock_9704.c:893-903`) with the same parser as +the GET (`src/jspr.c:753-788`): `constellation_visible` bool, +`signal_level` number (read only when visible), `signal_bars` 0..5. Calls the +`constellationState` callback (Python dict `constellationVisible`, +`signalBars`, `signalLevel`, `rockblock9704/rockblock.c:106-132`). Nothing is +stored; `rbGetSignal` always sends a fresh GET. The struct is uninitialised +(`src/rockblock_9704.c:895`), so with `constellation_visible:false` the +`signal_level` handed to the callback is garbage, and a missing +`signal_bars` is garbage too. Always send `signal_bars`. + +### 6.2 `299 messageProvisioning` + +**Not handled.** No code path looks at an unsolicited `messageProvisioning`; +`rbPoll` consumes and ignores it. Provisioning is learned only through the +one `GET messageProvisioning` in 2.11. + +### 6.3 Other unsolicited lines + +`299 simStatus` matters only inside `setSim` (2.4). `299 operationalState` +matters only inside `rbResyncServiceConfig` and `rbUpdateFirmware` (2.12). +Everything else is ignored. + +--- + +## 7. Constants a fake must respect + +| Constant | Value | Where | Meaning for the fake | +| --- | --- | --- | --- | +| line terminator | `\r` | `src/jspr.c:65`, `src/jspr_command.c` | both directions | +| first-byte / inter-byte timeout | 0.5 s | `serial_linux.c:153` | reply fast, write lines whole | +| `RX_BUFFER_SIZE` | 8192 | `src/jspr.h:13` | max line to the library | +| `JSPR_MIN_RESPONSE` | 9 | `src/jspr.h:18` | min line length | +| `JSPR_MAX_TARGET_LENGTH` | 30 | `src/jspr.h:16` | target < 30 chars | +| `JSPR_MAX_JSON_LENGTH` | 3500 | `src/jspr.h:20` | JSON < 3500 chars | +| `COMMAND_MAX_LEN` | 2048 | `src/jspr_command.h:12` | longest request line | +| `BASE64_TEMP_BUFFER` | 2048 | `src/rockblock_9704.h:112` | MO segment base64 | +| MO `segment_length` | 1..1446 | `src/jspr.c:624` | request no more than 1446 | +| MT `segment_length` | 1..1446 parsed; data cut at 1446 chars | `src/jspr.c:712,738-741` | use <= 1083 bytes | +| `segment_start` | 0..100001 | `src/jspr.c:632,720` | | +| `message_length_max` | 3..100002 | `src/jspr.c:672` | | +| `message_id` | 0..255 | `src/jspr.c:590,640,680,728,810,902` | | +| `request_reference` | 1..100, wraps | `src/jspr_command.c:212-216` | | +| topic id | 64..65535 | `src/rockblock_9704.c:20-21` | | +| `IMT_PAYLOAD_SIZE` | 100000 + 2 (Linux), 5000 + 2 (Arduino) | `src/imt_queue.h:31-39` | max payload + CRC | +| `IMT_CRC_SIZE` | 2 | `src/imt_queue.h:25` | CRC-16/XMODEM, big-endian | +| `IMT_QUEUE_SIZE` | 1 | `src/imt_queue.h:17-19` | one MO and one MT in flight | +| `JSPR_MAX_TOPICS` | 20 | `src/jspr.h:29` | provisioning entries used | +| `JSPR_TOPIC_NAME_MAX_LENGTH` | 57 | `src/jspr.h:28` | name < 57 chars | +| baud | 230400 | `src/rockblock_9704.h:119` | | + +--- + +## 8. Surprises and quirks a fake should know + +1. **Lines that arrive while the library waits for a reply are discarded** + if their target differs (1.3). Unsolicited MT or segment requests sent + during a GET/PUT exchange are lost for good. Order every exchange as: + solicited `200` first, then `299`s. +2. **A line with no 3-digit code in 200..500** makes the code scan run off + the end of the line through the zeroed buffer, then `memmove` with a + negative length cast to `size_t` (`src/jspr.c:89-113`). Expect a crash. + Never send banners, prompts or blank-ish lines of 9 bytes or more. +3. **`resultCodeIndexStart` is not reset** after a line is discarded for the + wrong target (`src/jspr.c:48,133-136`): if that line had leading junk, + the next line is read from the same offset. Do not send leading junk. +4. **A pause of 0.5 s inside a line** after a discarded line makes the + partial line be parsed as a response, because `validResponse` stays true + (`src/jspr.c:80` after the `continue` at 136). +5. **setState misspells its target** as `"operationState"` + (`src/rockblock_9704.c:259`), so `rbBegin` from any state other than + `active` or `inactive` always fails. +6. **`299 operationalState` before `200 operationalState`** fails `rbBegin` + from `inactive` (3, step 4). +7. **Sync send with a callback registered never reports success.** `moSent` + and `mtReceived` are set only when the matching callback is absent + (`src/rockblock_9704.c:771-778,868-875`). In Python, once + `set_mo_message_complete_callback` has been called, `send_message()` always + runs to its timeout and returns False, even though the message went. Same + for `receive_message()` with an MT callback: it never returns. +8. **After a sync send times out the MO stays in the queue** (started, never + removed; `src/rockblock_9704.c:452-456,464-467`), and the queue is locked + by default. The next `rbSendMessage` cannot add (returns false with only the + provisioning check, if any, on the wire); the next `rbSendMessageAny` + ignores the failed add (`queued >= 0`, `src/rockblock_9704.c:400`) and + **re-sends the old stuck message** with a new `PUT messageOriginate`. A late + `299 messageOriginateStatus` for the old id, read by any later `rbPoll`, + clears it. +9. **`200 messageOriginate` alone means started**, whatever + `message_response` says (4.3 step 2). +10. **`rbReceiveMessage` has no timeout** once an MT has been announced. + If the fake announces an MT and never sends a status, the host blocks + forever (with the GIL held, see 11). +11. **The Python binding never releases the GIL** (no + `Py_BEGIN_ALLOW_THREADS` in `rockblock9704/rockblock.c`). Each blocking + C call (`begin`, `send_message` for up to `timeout` s, `receive_message`, + every 0.5 s read timeout) freezes every Python thread in that process. + **Run the fake in a separate process**, not a thread beside the code + under test. +12. **MT running length leaks** between messages: `messageLengthAsync` is + reset only on a `complete` status (`src/rockblock_9704.c:865-866`). After a + failed or abandoned MT, the next MT's reported length includes the old + segments' lengths. +13. **MT CRC is stripped, never checked** (5.2). A fake that sends no CRC + loses the last 2 bytes of every message. +14. **MT data longer than 1446 base64 characters is truncated silently** (5.1). +15. **Provisioning is fetched once per process** and cached even on a miss + (2.11). Unsolicited `messageProvisioning` is ignored (6.2). +16. **Uninitialised structs everywhere.** Every parser fills a stack struct + the caller did not zero (`src/rockblock_9704.c:169,197,235,433,566,723, + 800,826,858,895,915,1069,1178`). A missing or out-of-range field leaves + garbage that may still be acted on (for example a garbage `message_id` + that happens to match). Always send every field listed above, in range. +17. **Non-string where a string is expected crashes** in the firmware parser + (2.9). +18. **`rbPoll` reads one line per call** and only when `FIONREAD > 0`. A burst + of N lines needs N polls. Multi-segment MOs and MTs need the host to poll + repeatedly; the sync paths do that themselves in a busy loop. +19. **The 9-byte minimum and `pos > 2` rule** mean short lines merge into the + next one (1.2). Any real JSPR line is longer than 9 bytes, so this only + bites junk. +20. **`clearLeftoverData` reads up to 32 bytes into a 32-byte buffer** and the + Linux read writes a NUL after them (`src/rockblock_9704.c:137-149`, + `serial_linux.c:178`), one byte past the end. Avoid sending anything + before `rbBegin`. +21. **Cancellation** (`rbCancelMessage`, Python `cancel_message(topic, id)`): + `>> PUT messageOriginateStatus {"topic_id": 244, "message_id": 17, "action": "cancel"}\r` + (`src/rockblock_9704.c:619-634`). The reply is not read; the `200` is + swallowed by a later `rbPoll`. The outcome must come as + `299 messageOriginateStatus` with, for example, + `"final_mo_status":"message_cancelled_pre_transit"`. + +--- + +## 9. A minimal session, end to end + +``` +(host opens port, drains) +>> GET apiVersion {}\r +<< 200 apiVersion {"supported_versions":[{"major":1,"minor":0,"patch":0}],"active_version":{"major":1,"minor":0,"patch":0}}\r +>> GET simConfig {}\r +<< 200 simConfig {"interface":"internal"}\r +>> GET operationalState {}\r +<< 200 operationalState {"state":"active","reason":0}\r +(rbBegin returns true) + +>> GET hwInfo {}\r +<< 200 hwInfo {"hw_version":"1.0","serial_number":"123456","imei":"300434030000000","board_temp":27}\r + +>> GET constellationState {}\r +<< 200 constellationState {"constellation_visible":true,"signal_bars":5,"signal_level":-100}\r + +(send_message(b"hello"), topic 244, sync) +>> GET messageProvisioning {}\r +<< 200 messageProvisioning {"provisioning":[{"topic_id":244,"topic_name":"RAW","priority":"Low"}]}\r +>> PUT messageOriginate {"topic_id":244, "message_length":7, "request_reference":1}\r +<< 200 messageOriginate {"topic_id":244,"request_reference":1,"message_id":1,"message_response":"message_accepted"}\r +<< 299 messageOriginateSegment {"topic_id":244,"message_id":1,"segment_length":7,"segment_start":0}\r +>> PUT messageOriginateSegment {"topic_id":244, "message_id":1, "segment_length":7, "segment_start":0, "data":"aGVsbG/DYg=="}\r +<< 200 messageOriginateSegment {"topic_id":244,"message_id":1}\r +<< 299 messageOriginateStatus {"topic_id":244,"message_id":1,"final_mo_status":"mo_ack_received"}\r +(send_message returns True, if no MO callback is registered) + +(receive_message() in a loop) +<< 299 messageTerminate {"topic_id":244,"message_id":2,"message_length_max":7}\r +<< 299 messageTerminateSegment {"topic_id":244,"message_id":2,"segment_length":7,"segment_start":0,"data":"aGVsbG/DYg=="}\r +<< 299 messageTerminateStatus {"topic_id":244,"message_id":2,"final_mt_status":"complete"}\r +(receive_message returns b"hello") +``` + +The JSON field order and spacing in the `<<` lines are free; the library +looks fields up by name. Field names the library reads are exactly those +listed in section 2; any extra fields a real modem sends are ignored. diff --git a/docs/TESTING.md b/docs/TESTING.md new file mode 100644 index 0000000..749a9d9 --- /dev/null +++ b/docs/TESTING.md @@ -0,0 +1,100 @@ +# Testing + +One fake modem, three places to run the code against it, and a few free +checks against a real modem. + +| Level | What runs | Against | Needs | +| --- | --- | --- | --- | +| Python | Ground Control's Python binding, the `rb9704` command line, the fake itself | the fake on a pty | Python 3.10 to 3.14 | +| Host Zephyr | this module in the shell sample on `native_sim` | the fake on a pty, or a real modem on USB | a Zephyr workspace | +| Board | the same sample on an ESP32-S3, STM32U5 or any board | the fake on a USB-serial port wired to the board's modem pins | a board and a wire | + +The native_sim and board runs are the same tests (`python/tests/test_zephyr.py`) +through a different console, so a behaviour seen on one is checked on the +other. + +## The fake modem + +`python/rb9704tools/fakemodem.py`, `rb9704-fake` on the command line. It +implements JSPR as `JSPR.md` describes it: the three exchanges `rbBegin` +needs, the identity and signal getters, provisioning, MO in segments of +any size with the CRC checked, MT in segments the library can take, and +on command a refusal, a failed MO, a signal change or a change of +provisioning. It writes every whole MO it receives as an event line, so a +test checks the bytes that reached "the network". + +What it cannot tell you is what a real modem does that the library's +source does not show: unsolicited lines at odd moments, timing, MT +segment sizes. That is what `test_real.py` and the hardware are for. + +## Running them + +```sh +python/install.sh + +# Python level: the fake, the command line, Ground Control's binding +python/.venv/bin/pytest -q python/tests + +# Host Zephyr: build the sample for native_sim, then +west build -b native_sim/native/64 samples/shell -d build/native +python/.venv/bin/pytest -q python/tests --zephyr-exe build/native/zephyr/zephyr.exe + +# Board: flash the sample with its modem UART wired to a USB-serial +# adapter (3.3 V), then the board's shell port and the adapter's port +python/.venv/bin/pytest -q python/tests/test_zephyr.py \ + --board-console /dev/ttyACM0 --board-modem /dev/ttyUSB0 + +# A real modem, free checks only (identity and signal, never a message) +python/.venv/bin/pytest -q python/tests/test_real.py --real-modem /dev/ttyUSB0 \ + [--zephyr-exe build/native/zephyr/zephyr.exe] +``` + +Every option has an environment variable as well: `RB9704_ZEPHYR_EXE`, +`RB9704_BOARD_CONSOLE`, `RB9704_BOARD_MODEM`, `RB9704_REAL_MODEM`. + +On native_sim the modem UART is a `zephyr,native-tty-uart` node +(`samples/shell/boards/native_sim.overlay`); the tests give it the fake's +pty with `-modem_uart_port=`. The same option takes a real modem's +`/dev/ttyUSB0`, which is how host Zephyr talks to real hardware. + +## The rig: two boards, one wire + +No USB-serial adapter to hand? Two boards with a UART wired between them +will do: one runs the sample, the other runs `tests/rig/bridge`, which +copies bytes between that UART and its own USB serial port, where the +fake answers. + +``` +host --USB-- bridge board --UART-- board under test --USB-- host + (tests/rig/bridge) (samples/shell) + fake modem on this port pytest drives this shell +``` + +The overlays in `tests/rig/` are for a NUCLEO-U575ZI-Q (USART2 on PD5 and +PD6) and a Seeed XIAO ESP32S3 (D8 and D9) wired to each other, either way +round: + +| Under test | Bridge | `--board-console` | `--board-modem` | +| --- | --- | --- | --- | +| XIAO ESP32S3, `-DDTC_OVERLAY_FILE=tests/rig/xiao_esp32s3_rig.overlay` | Nucleo, `tests/rig/bridge` | the XIAO's USB | the Nucleo's ST-LINK port | +| Nucleo, `-DDTC_OVERLAY_FILE=tests/rig/nucleo_u575zi_q_rig.overlay` | XIAO, `tests/rig/bridge` | the Nucleo's ST-LINK port | the XIAO's USB | + +The ESP32-S3 sits in its ROM loader after `west flash` until it is reset; +toggle RTS on its USB port, or press reset, before the tests. + +## What has passed + +On 2026-10-09, Zephyr v4.4.0, Ground Control's library v1.0.1, the +binding 1.0.2: + +| Level | Tests | Result | +| --- | --- | --- | +| Python: fake, command line, binding | 16 | all pass | +| `native_sim/native/64` against the fake | 12 | all pass | +| XIAO ESP32S3 through the rig | 12 | all pass | +| NUCLEO-U575ZI-Q through the rig | 12 | all pass | +| a real 9704 (firmware v1.5.14), binding and native_sim, free checks | 2 | all pass | + +Not yet: a message through a real modem and satellite, which costs +airtime and needs the sky (`CLOUDLOOP.md`, "The order to test in"), and +the power pins on a board that has them wired.