pacing_rate had been computed on every ACK since the controller was written
and never read by anything: admission was decided purely by cwnd. But cwnd
bounds how much may be UNACKNOWLEDGED, not how fast it reaches the wire, so a
full window went out back-to-back. On a bottleneck with a deep buffer that
burst is absorbed rather than dropped, and it lands as standing queue — the
mechanism behind the multi-second RTT this protocol has been showing on
mobile. It is also why BBR could not simply be dropped in: BBR's whole model
is "send at the estimated bottleneck rate", which is meaningless without a
pacer underneath it.
Adds a token bucket to CongestionController, charged in on_send so every byte
that reaches the wire pays exactly once — retransmits included, since those
are precisely what must not bypass the limit and pile into a full queue.
Burst allowance is 10ms-at-rate rather than one packet. Pacing intervals here
are fractions of a millisecond, so strict per-packet release would need a
sub-millisecond timer per packet; sizing the burst to the loop's existing
~10ms wakeup lets the configured rate still be saturated. There is a floor of
4 MTU so a cold or collapsed rate estimate can never wedge sending entirely.
Wired into both directions. The client gates its proxy-event branch on it,
and the server's per-session backpressure snapshot reports zero headroom when
the bucket is empty. The download path matters most here — that is the one
carrying video — and it was also still clamped to the old 16384-packet
ceiling (~20 MB outstanding), now aligned with MAX_CWND_PACKETS.
Tested that the bucket denies once drained and refills over time; the second
is what keeps a stalled bucket from wedging sending permanently.
DnsServer::run_local_udp_listener binds 0.0.0.0 and answers every UDP
datagram by resolving it and replying to the packet's (unverified,
spoofable) source address - a textbook DNS reflection/amplification
primitive. An attacker spoofing a victim's IP as the query source turns
any server with this listener running into a free amplifier against that
victim, with zero authentication gating it (unlike the main OSTP port,
there's no Noise handshake here).
Nothing in the codebase currently calls this function - the live DNS path
is router.route_dns(), reached only through the authenticated OSTP tunnel
relay (relay.rs). But the doc comment describes this as an intended,
not-yet-wired entry point for clients that point their OS resolver
directly at the server, so it's a real latent risk for whoever connects it
without realizing the implication. Added a global (not per-source-IP -
per-IP limiting doesn't help against a reflection attack, since the
attacker never sees the replies and can spread queries across arbitrary
spoofed sources) token bucket capping total replies/sec, so connecting
this later can't silently reintroduce unbounded amplification.
check_token() and handle_login() compared bearer tokens, session tokens,
and the password hash with plain ==, which short-circuits on the first
differing byte - a textbook remote timing side-channel against exactly
the long-lived secrets these gates exist to protect. Added subtle (already
in the dependency tree transitively via chacha20poly1305) as a direct
dependency and route every secret comparison through a small secure_eq()
wrapper over ConstantTimeEq. Username comparison in handle_login is left
as-is: it isn't treated as a secret in this threat model (one fixed admin
username), matching standard practice of only constant-timing the
password/token side of an auth check.
Added tests for secure_eq() itself (equal, different, different-length,
empty) alongside the existing check_token coverage.
GET/POST/DELETE /api/audit were the only three handlers in the whole
Management API that never called check_token() - every other endpoint
(status, users, rules, config) does. Concretely, with the panel's
credentials configured, an unauthenticated request could still:
- read the full audit log (GET)
- inject arbitrary forged entries, e.g. fake "success" events to cover
tracks (POST)
- wipe the entire audit log (DELETE) - the exact mechanism meant to
detect and investigate unauthorized actions, erasable with zero auth
Added the same check_token() gate the rest of the file uses, and fixed
these three handlers' raw .unwrap() on the audit_logs lock to the
poison-recovery pattern (unwrap_or_else(|e| e.into_inner())) used
everywhere else, for consistency.
Added focused unit tests on check_token() itself (missing header, correct/
wrong bearer, raw token, session token, and the documented open-panel
mode when no credentials are configured) - it's the single gate every
sensitive handler depends on, worth pinning down independently of any one
handler.
Matches a real report: traffic counters move (the OSTP tunnel handshakes
fine) but sites don't open, or take very long - on a freshly deployed
DigitalOcean droplet in Amsterdam.
connect_target's fallback path handed the raw "host:port" string straight
to TcpStream::connect, which resolves and tries addresses internally but
shares ONE 10s timeout across the WHOLE attempt (all resolved addresses,
not per-address). Some VPS hosts assign the machine an IPv6 address that
the OS prefers by RFC 6724 ordering but that has no actually-working
outbound route - the connect doesn't get refused, it just hangs. With a
single shared budget, that one dead IPv6 candidate eats the entire 10s and
the working IPv4 candidate is never even attempted: every dual-stack
destination (i.e. most popular sites) times out, while IPv4-only
destinations work fine.
New connect_direct() resolves target itself via lookup_host, sorts IPv4
candidates first, and tries each with its own 3s budget (still bounded
overall by the original 10s outer timeout as a backstop) so a hung IPv6
attempt can't starve the IPv4 fallback of a chance.
Added tests: IPv4-first sort ordering (and stability within a family), a
successful connect against a live local listener, and a refused-port
connect failing well under the timeout (proving failures aren't
needlessly slow). Scoped to connect_target's direct-connect paths; the
SOCKS5/HTTP outbound-proxy paths and the fallback/camouflage TCP proxy
(which targets a fixed admin-configured local address, not arbitrary
dual-stack hostnames) are unaffected.
Matches a real user report after 2 weeks on this version: bandwidth is low,
the client reconnects every 10-20 minutes, sites randomly stop loading or
crawl, and ping visibly jitters. Root cause: the per-target-connection
reader task (handle_relay_message's Connect handler) read from the
upstream target as fast as it would send and forwarded every chunk
straight to send_relay_to_stream -> an immediate UDP datagram, with ZERO
awareness of the client-facing OSTP session's actual congestion window.
The client already gates its own uplink on cwnd (bridge.rs's proxy_ev
select arm); the server's download direction had no equivalent.
On a real (lossy/jittery mobile or Wi-Fi) client path, a fast target (a
CDN, say) gets blasted at the client far beyond what the path can sustain.
That's a self-inflicted loss burst: it wrecks the RTT/RTO estimate (the
"ping jitters" symptom), can push the session into a stall bad enough that
the client's 25s/180s keepalive stall-detection gives up and reconnects
(the "every 10-20 minutes" symptom), and produces exactly the "randomly
stops loading or crawls" experience while it's happening.
Fix: Dispatcher::snapshot_backpressure() computes each session's headroom
(clamped cwnd - in_flight, same clamp(16,16384) the client uses) from the
existing 10ms retransmit tick - no new polling loop. Published to a
lock-free-reader Arc<AtomicI64> per session in a shared map so relay reader
tasks (which don't have access to Dispatcher; it lives on the main loop
task) can check it without touching a lock/mutex on every read. Before
each read, a reader task waits (capped at 2s, so a wedged read can't stall
forever) while headroom is <= 0. New sessions default to a healthy 32
packets until their first snapshot lands (worst case 10ms), so this can't
stall the very first bytes of a fresh connection.
Scoped to the primary TCP CONNECT/proxy path (the one actual web browsing
uses); UDP relay/TURN is unaffected. No load test against real network
jitter was possible in this environment - the mechanism directly targets
the identified cause, but real-world confirmation is still needed.
Brings the crypto security audit into alpha alongside the already-merged
reconnection/UoT/Flutter fixes:
- CRITICAL: transport keys now come from Noise Split() over ck (DH-
inclusive), not the handshake hash — restores forward secrecy.
Wire-breaking, PROTOCOL_VERSION 4->5.
- Rate-limit + cache the O(N_keys) handshake trial path (CPU DoS).
- Quiet hot-path logging; access keys no longer logged verbatim.
- Removed dead 0-RTT resumption module (unsafe XOR ticket crypto).
- Karn's algorithm RTT fix, 32-bit frame-length overflow guard,
replay-cache eviction instead of global reject.
- Updated EN/RU specification docs to match.
No conflicts: protocol-hardening's bridge.rs commit was an independent
duplicate of the same reconnection fix already on alpha, so git merged it
as a no-op on that file.
Two UoT (UDP-over-TCP) correctness issues:
- The accepted UoT stream never had TCP_NODELAY set (the client sets it on
its end, the server didn't). Nagle's algorithm then batched server->client
writes and interacted with the client's delayed ACKs, adding tens-to-
hundreds of ms of stall per burst — throttling the download direction
badly for streaming/video. Every TCP-tunnel proxy disables Nagle; now the
server matches the client.
- handle_tcp_connection join!ed the reader and writer tasks, so a half-open
connection (client's read side gone, no outbound data pending) parked the
writer on rx.recv() forever, leaking the task and a stale tcp_map entry.
Rewrote it with select! so either half closing cancels the other and the
tcp_map entry is always removed. Added duplex-stream tests covering
inbound reassembly across segment boundaries, outbound framing, and
teardown-on-close.
- Karn's algorithm: drop_acked_frames no longer samples RTT from frames
that were retransmitted (last_sent is bumped on each retransmit, so an
ACK for the original transmission would measure a spuriously small RTT
and drag SRTT/RTO down). Added CongestionController::on_ack_no_rtt for
the case where every acked frame was ambiguous, so the window still
advances without polluting the RTT estimator. Refactored the shared
window-growth into grow_window.
- Frame decode: header+payload+pad length now uses checked_add. payload_len
is a u32 from the header and on 32-bit targets (MIPS/ARMv7 routers are
supported) the sum could wrap usize and slip past the truncation check.
- Replay cache: a full cache used to reject ALL new handshakes globally
until the next tick, letting one flooding key-holder deny service to
everyone. Now it reclaims expired entries and, if still full, evicts the
single oldest — new handshakes always get in. Fixed the mislabelled
"100000" log (cap is 50000) and named it REPLAY_CACHE_MAX.
Two classes of issue:
- Hot-path/attacker-triggerable events logged at info/error with internal
detail: a per-handshake info! byte dump (raw_vec[0..6]) and a per-packet
error! on session-id mismatch that dumped expected/got session ids.
Both are log-flood + info-leak surfaces; downgraded to debug and
stripped of the sensitive detail. Close/Resume frame handling likewise
moved from info to debug.
- The access key (a shared secret) was written to logs verbatim in three
places (session drop, key-created UI event, API create-user) and as an
8-char prefix in one. Added key_fp() — a short SHA-256 fingerprint — and
routed all key logging through it so operators can still correlate
events without the secret ever hitting the log.
Every datagram from an unrecognized source ran the full key-trial loop:
for each registered access key, an HKDF (derive_all_secrets) plus two
HMACs (junk markers) plus a Noise read. A garbage flood from spoofed
sources could therefore force unbounded O(N_keys) crypto per packet — a
CPU-amplification DoS with no throttle (the existing token bucket only
guarded the roaming path, not this one).
Two mitigations:
- Memoize the per-key derived secrets (pure function of key+version) and
the per-window junk markers, so the trial loop is now cheap comparisons
plus one Noise read per key instead of HKDF+2*HMAC per key per packet.
Also speeds up every legitimate new connection. Caches are pruned in
on_tick when keys are deleted.
- Gate the trial path behind a global token bucket (TRIAL_RATE=100/s,
same burst). The established-session fast path and roaming are not
gated, so live sessions are unaffected; only unknown-datagram trials
are bounded. Over-budget datagrams are dropped silently.
The junk marker was a per-key CONSTANT sent in plaintext at a fixed offset in
junk frames. Junk is meant to look like random noise (zapret-style), but a
constant prefix is a recognizable per-user structure: an on-path observer
watching one user sees the same 4 bytes on every junk packet, i.e. an OSTP
fingerprint. (The earlier fix only removed the GLOBAL constant.)
Now the marker rotates every 60s window: junk_marker = HKDF(key, ver, 0x04 ||
window). To an observer the prefix changes each window (no fixed signature),
and a captured marker is only valid for ~1 window — the "bit of protection"
against a leaked marker. Only a key holder can compute it, so an outsider still
can't forge a silently-dropped junk packet (and silent-drop is cheaper than
normal processing anyway, so junk spam was never a DoS lever to begin with).
- core: derive_junk_marker(key, window) + current_junk_window() (60s window),
same version-gated HKDF scheme; junk_marker dropped from DerivedSecrets.
- client: stamps junk with the current window's marker.
- server: checks current AND previous window per key (absorbs ~1 window of
clock skew) before falling through to unauthorized-probe handling.
- Not a wire break: only junk framing changes; real handshake/data untouched.
During mixed rollout, unmatched junk merely logs as a probe (cosmetic).
security / protocol:
- Derive a PER-KEY junk marker (obfuscation.rs, info byte 0x04) instead of the
global constant [0x88,0x1A,0x93,0x5D]. A fixed marker was a universal DPI
signature identifying ALL OSTP users at once — exactly what the HKDF version
gate avoids for the handshake. Server drops junk via a new DispatchOutcome::Junk
inside the existing key-trial loop (secrets already derived → zero extra cost);
client stamps its own key's marker.
- §E: configurable junk/fragmentation params (junk_pc / junk_ps / frag_chunk / frag_sleep).
GUI (desktop):
- Light theme + toggle, GUI version footer in Settings.
- Fix mouse-wheel scroll on Settings (flex child needed min-height: 0).
- Drop the false "process exclusions unsupported in TUN mode" warning — they DO
work (native_handler maps port->process via GetExtendedTcpTable).
release / infra:
- build.ps1: add -PreRelease (tag CURRENT version as v<ver>-beta.N, no bump, no
master commit); guard the panel build when ostp-control ships no source; bump
the real ostp-gui/package.json instead of the nonexistent ostp-control one.
- release.yml: mark hyphenated tags as GitHub pre-releases; don't hard-fail the
web-panel step when there is no source (use committed dist/).
- Versions aligned to 0.4.1; README license badge BSL 1.1 -> AGPL v3.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Ported only the fixes that actually apply to the pre-refactor base
(the handshake fixes 6eb7b36/d65af35 fix bugs the 0.3.1 multi-server
refactor introduced into the new outbounds/ostp.rs; the base bridge.rs
already waits for the handshake response with retransmit + NAT64
fallback, so they are intentionally skipped).
- EMFILE (922cf0b): rlimit::increase_nofile_limit at CLI startup.
- Logs (1151726): UoT connect/disconnect → debug; rate-limit the
unauthorized-probe log to one line / ~30s so a junk/probe flood can't
spam the log (and a client running junk-over-UDP can't self-ban).
- Helper lifecycle + bypass routes (b6e78c1):
* ostp-tun-helper forces std::process::exit(0) after run_server so the
WinTun adapter and its metric-0 default route are reclaimed instead
of lingering as a zombie that breaks the next connect.
* windows_route: delete_routes_for_dest() purges stale /32s, dedupe
bypass IPs, and log add failures at warn!.
* windows: retry tun::create through the transient ERROR_INVALID_
PARAMETER window and widen the adapter-index lookup to ~15s.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The core bug: server sent 5 TLS records in server_hello but client only
read the first one (ServerHello), then passed remaining bytes (CCS + fake
records) into RealityStream. RealityStream saw 0x14 (CCS) != 0x17 and
immediately returned an error, killing the connection.
Changes:
- reality.rs: append ChangeCipherSpec after ClientHello (RFC 8446 D.4)
export REALITY_SERVER_HANDSHAKE_RECORDS=5 constant
- xhttp.rs: drain all 5 server handshake records before creating RealityStream
- uot.rs: rebuild server_hello as proper 5-record TLS 1.3 flight:
ServerHello + CCS + fake EE (108B) + fake Cert (812B) + fake Fin (52B)
drain client CCS from raw stream before wrapping in RealityStream
- Flutter: Hide 'DNS Server' field and force '10.1.0.1' if connection link contains owndns=true
- Flutter: Remove 'Use Provider DNS' toggle to eliminate client-side choice
- Server (relay.rs): Intercept DNS queries targeting '10.1.0.1:53' and process them via internal DnsServer if DNS is enabled
- Server (api.rs): Continue appending owndns=true to subscription links to enforce internal DNS logic on clients
- ostp-server/relay.rs: remove DNS port 53 interception — DNS queries
now pass through to the actual DNS server as regular TCP connections
- ostp-client/native_handler.rs (Windows): add explicit gateway/32 route
via real interface BEFORE setting default route via TUN to prevent loop
- ostp-client/native_handler.rs (Linux): properly detect real gateway and
add default route via TUN with metric 10 after server IP exclusion
- Remove redundant extra DNS host routes from Windows setup script
ospab
ospab