# ArkNet Architecture

ArkNet is the WAN-native transport and site-connectivity layer for Ark
Fabric. It carries storage payloads, replication traffic, metadata movement,
and site-to-site overlay traffic across independent Internet and private WAN
paths.

This document distills the transport and site-connectivity architecture
developed by Ark Sites.

## Role In Ark Fabric

Ark Fabric is the umbrella platform. ArkNet is one included core
subsystem within it; ArkTxn, ArkLog, and ArkObj are sibling
subsystems, while S3, BK-QCOW, and ArkFS are service views.

```mermaid
flowchart TB
    Fabric["Ark Fabric"] -->|includes| Object["ArkObj"]
    Fabric -->|includes| FDB["ArkTxn"]
    Fabric -->|includes| Log["ArkLog"]
    Fabric -->|includes| Transport["ArkNet"]
    Fabric -->|presents| Views["Service views<br/>S3, BK-QCOW,<br/>ArkFS, and others"]

    Views -->|object payloads| Object
    Views -->|transactions<br/>and metadata| FDB
    Object -->|WAN payload<br/>movement| Transport
    FDB -->|routing and<br/>replication<br/>traffic| Transport
    Log -->|segment<br/>dissemination| Transport
```

ArkNet provides the network substrate the sibling subsystems and service
views use when they communicate across sites. It should be usable by storage
reads, storage writes, replication fanout, ArkTxn hot-replica catch-up, ArkLog
segment movement, operator access, and site health probes.

## Goals

ArkNet is designed to:

* use UDP/443 QUIC as the normal WAN transport;
* support regular QUIC fallback when multipath support is unavailable;
* use MPQUIC only when both peers support compatible multipath behavior;
* model IPv4, IPv6, tunnels, relays, and provider paths as separate path
  candidates;
* preserve IPv4 reachability while adding IPv6 as a first-class path;
* use MASQUE CONNECT-IP when a general L3 overlay is required;
* expose telemetry for RTT, loss, congestion, PMTU, eligibility, and scheduler
  decisions;
* keep public ingress constrained to explicitly authorized TCP/443 and UDP/443
  paths.

## Initial QUIC Implementation

The first production transport target is standard single-path QUIC using
[upstream Cloudflare quiche](https://github.com/cloudflare/quiche). This matches
the existing Rust and quiche integration in Ark Sites while preserving an
upstream-supported interoperability baseline.

The first MPQUIC lab target remains the research fork selected for Ark Sites:

```text
repository: IPNetworkingLab/flexicast-quic
branch: per-uc-path-cca
commit: 61350d346c205ac22c68a95cf003c64baff35f53
```

That fork is feature-gated and is not the default production runtime. The IETF
[multipath QUIC specification](https://datatracker.ietf.org/doc/draft-ietf-quic-multipath/)
remains standards-track work, and upstream quiche does not currently document a
production MPQUIC API. ArkNet must therefore preserve standard QUIC fallback.

MPQUIC is eligible only when:

* both peers advertise the same compatible multipath capability;
* at least two validated path candidates exist;
* traffic-class policy permits aggregation or standby failover;
* PMTU, RTT skew, loss, and congestion telemetry pass policy thresholds;
* fallback to standard QUIC has passed no-regression tests.

MPQUIC is never required for correctness, durability, or baseline
interoperability. Revisit the research fork when upstream quiche gains suitable
multipath support, the IETF wire format changes, or compatibility and
performance tests fail.

## Network Stack

```mermaid
flowchart TB
    Service["Ark service or edge"] --> Scheduler["Ark scheduler"]
    Scheduler --> Session["QUIC or MPQUIC session"]
    Session --> Masque["Optional HTTP/3<br/>MASQUE CONNECT-IP<br/>datagrams"]
    Masque --> Path["IPv4, IPv6, tunnel,<br/>relay, or private<br/>WAN path"]
```

QUIC is the secure stream transport. MPQUIC is an optional accelerator and
resilience mechanism. MASQUE CONNECT-IP is the target packet overlay when Ark
must carry arbitrary IPv4 and IPv6 packets through an authenticated HTTP/3
connection.

## Path Inventory

Every site should classify its usable paths before enabling advanced
scheduling.

| Path | Use |
|---|---|
| Native IPv6 | Preferred when available and healthy. |
| Native IPv4 | Baseline reachability and fallback. |
| IPv6-over-IPv4 tunnel | Optional path when native IPv6 is absent and protocol 41 works. |
| Relay or VPS path | Fallback for CGNAT, restrictive CPE, blocked protocol 41, or mobile ISPs. |
| Private WAN path | Enterprise, embassy, consulate, branch-office, or datacenter connectivity. |

The scheduler should treat each path as a candidate with its own health and
policy state, not as an interchangeable route through one abstract Internet.

## Scheduling Policy

ArkNet policy should be explicit per traffic class:

| Policy | Behavior |
|---|---|
| Interactive | Prefer the lowest healthy RTT path; avoid striping ordered latency-sensitive flows across skewed paths. |
| Control | Prefer reliable low-loss paths and keep a standby path ready. |
| Bulk | Aggregate across eligible paths when RTT skew, PMTU, and loss are acceptable. |
| Retransmit | Retry lost or delayed data on another healthy path when congestion state permits. |
| Backup | Keep a path validated with bounded keepalives while mostly idle. |

Blind round-robin should not be a production policy because it can make ordered
streams slower than single-path QUIC when path latency or loss differs.

## MASQUE Overlay

MASQUE CONNECT-IP is the target L3 overlay for IPv4 and IPv6 packets. It is
useful for Ark Sites connectivity, remote operator access, and private service
reachability. It is not the storage object format and does not replace ArkObj.

Endpoint roles:

| Endpoint | Responsibility |
|---|---|
| Client or branch agent | Own a TUN interface, capture selected packets, authenticate, and encapsulate packets into CONNECT-IP datagrams. |
| Site edge agent | Terminate authenticated HTTP/3 sessions, authorize routes and prefixes, decapsulate packets, and forward to local networks. |

## Security Requirements

ArkNet must:

* authenticate every overlay session;
* authorize prefixes and routes per peer;
* disable unauthenticated relay behavior;
* constrain private, link-local, and metadata-service targets by policy;
* enforce tunnel, packet, byte, and idle limits per peer;
* disable 0-RTT for tunnel establishment until replay behavior is reviewed;
* log metadata and counters, not raw packet payloads.

## Relationship To Ark Sites

Ark Sites supplies the transport lab, MPQUIC experiments, MASQUE overlay plan,
topology validation, and local management surface.

Ark Fabric should use the architectural vision from Ark Sites, but it
should not blindly copy every sprint record, runbook, generated artifact, or
lab-specific implementation note. The canonical Ark Fabric docs should retain
the reusable architecture and link back to Ark Sites for lab execution details.