ADR-0017: User Notifications¶
Status: Accepted
Date: 2026-10-02
Deciders: nguyenhuuca
Related: ADR-0003 Use Cache (no Redis until horizontal scaling), ADR-0016 User Settings (notify_email, notify_new_content), plan-comment-replies, plan-admin-comment-moderation
Context¶
Users now interact with each other (threaded replies) and with moderators (comment removal), but nothing tells them it happened. They must reopen the comment panel and look.
Current state (verified in code):
| Fact | Where | Consequence |
|---|---|---|
user_settings.notify_email and notify_new_content exist, are editable in Settings, but nothing reads them |
UserSettings, UserSettingsServiceImpl |
Preferences already promised to users; notifications are the missing consumer |
Comment author is stored as email (video_comments.user_id VARCHAR), users are keyed by users.id BIGINT |
VideoComment, User |
Recipient must be resolved email → user id |
| Guests comment with a token, no account | VideoCommentServiceImpl |
Guests cannot receive notifications |
video_sources has no owner column |
VideoSource |
"Your video was liked/commented" has no recipient today |
Mail via JavaMailSender (Gmail SMTP), some methods @Async, EmailCacheLimiter exists |
MailServiceImpl |
Email channel exists but is slow and quota-limited |
Production is one VM, one JVM instance behind nginx (CI: SCP jar → deploy-funny-app.sh, systemctl reload nginx); Helm chart also pins 1 replica; virtual threads on |
.github/workflows/funnyapp-ci.yml, helm/funny-app/values.yaml |
Every SSE connection and every event live in the same JVM → an in-memory registry is sufficient today |
| No event bus, no WebSocket/SSE anywhere | grep | Greenfield for real-time |
Decision Drivers¶
- Ship value fast; fit Golden Paths (Spring Boot, JPA, Liquibase, React Query) — no Redis/Kafka (ADR-0003).
- Never block or fail the user action (posting a reply) because of notification work.
- Respect existing preferences (
notify_email). - Survive the future move to >1 replica without a rewrite.
- No email spam; stay within Gmail SMTP limits.
Scope (v1)¶
| Type | Trigger | Recipient | Channels |
|---|---|---|---|
COMMENT_REPLY |
Someone replies in a thread | Root comment author and the author of the comment being replied to (if different); never the actor | In-app; email if notify_email (digest) |
COMMENT_REMOVED |
Admin removes a comment (single or bulk) | Comment author | In-app (+ reason); email if notify_email |
Out of scope v1: video likes/comments on "your video" (no owner column), @mention parsing, new-content broadcast (notify_new_content), browser/mobile push, guest notifications.
Options¶
A. Delivery to the browser¶
| Option | How | Pros | Cons |
|---|---|---|---|
| A1. Polling | GET /notifications/unread-count via React Query refetchInterval 60 s + refetch on window focus |
Trivial, stateless, works with any replica count and proxy, cacheable | Up to 60 s delay; ~1 req/min per open tab |
| A2. SSE | GET /notifications/stream (SseEmitter, one virtual thread per connection), in-memory userId → emitters |
Near real-time; cheap with virtual threads; one-way fits | In-memory registry breaks with >1 replica (needs Redis pub/sub, prohibited now); proxy buffering/timeouts; reconnect logic; auth needs care (see A′ below) |
| A3. WebSocket/STOMP | Spring WebSocket + broker | Bi-directional | Overkill (one-way data), new dependency, same scaling issue |
Why "multi-instance" matters for push: an SSE connection lives in the JVM the browser connected to. If the event (Bob's reply) is handled by a different instance than the one holding the recipient's (Alice's) connection, that instance has no emitter to push to. Sticky sessions do not help (actor ≠ recipient). Fix when >1 instance: cross-instance pub/sub — PostgreSQL LISTEN/NOTIFY (no new infra) first, Redis later.
Scoring for the actual deployment (single VM, single JVM), 1–5, higher better:
| Criterion (weight) | A1 Polling | A2 SSE (in-memory) | A3 WS |
|---|---|---|---|
| Simplicity (3) | 5 | 4 | 1 |
| Latency (2) | 2 | 5 | 5 |
| Load at scale (2) — idle users cost nothing | 2 | 5 | 5 |
| Fits current deployment (3) | 5 | 5 | 3 |
| Golden-path fit (2) | 5 | 4 | 2 |
| Weighted total | 48 | 54 | 34 |
Load comparison (60 s polling vs SSE), estimates to be validated by load test:
| Online users | Polling unread-count |
SSE |
|---|---|---|
| 1,000 | ~17 req/s | 1,000 idle connections |
| 10,000 | ~170 req/s | 10,000 connections → above Tomcat default max-connections (8192); needs tuning/more RAM |
| 100,000 | ~1,700 req/s | needs multiple instances + pub/sub |
A′. Client transport & auth for the SSE stream¶
EventSource (the browser's built-in SSE API) cannot set request headers, so it cannot send Authorization.
| Option | How | Pros | Cons |
|---|---|---|---|
A′1. EventSource + ?token=<jwt> |
JWT in the query string | Built-in auto-reconnect; least FE code | JWT written to nginx/Cloudflare logs (CWE-598/532) — exactly the leak just fixed for video URLs (commit 9e1fb36); needs custom query-token auth on the server |
A′2. fetch + Authorization header, read response.body as a stream |
Parse event:/data: blocks from a ReadableStream |
URL carries nothing sensitive; reuses the existing JWTAuthenticationFilter (endpoint is a normal authenticated route); no new dependency |
~40 lines of FE code: SSE line parsing + reconnect/backoff written by hand |
| A′3. One-time ticket | POST /notifications/stream-ticket (JWT header) → 30 s single-use ticket → EventSource(?ticket=) |
Built-in reconnect; logs only hold expired tickets | Extra endpoint + ticket store; every reconnect needs a new ticket |
Chosen: A′2. Security first and zero new server-side auth code.
B. Producing notifications¶
| Option | How | Pros | Cons |
|---|---|---|---|
| B1. Inline call | CommentService calls NotificationService.create(...) in the same transaction |
Simple | Couples domains; a failure in notification code fails the reply |
| B2. Spring application events | Publish CommentRepliedEvent / CommentRemovedEvent; @TransactionalEventListener(phase = AFTER_COMMIT) + @Async (virtual-thread executor) writes notifications |
Decoupled; only fires after the reply is committed; failure isolated; no new infra | Lost if the JVM dies between commit and handler (rare; acceptable for v1) |
| B3. Transactional outbox | Write outbox row in same tx; scheduler dispatches |
Guaranteed delivery | More tables + job; premature at this scale |
C. Email¶
| Option | Pros | Cons |
|---|---|---|
| C1. Immediate email per event | Simple | Spam on busy threads; Gmail quota |
C2. Digest job (AppScheduler, every 15 min): unread & not-yet-emailed notifications per user → one email |
Bounded volume; respects notify_email |
Up to 15 min delay |
Decision¶
- Storage: new table
notifications(in-app inbox is the source of truth). - Producing: B2 — Spring events,
@TransactionalEventListener(AFTER_COMMIT)+@Asyncon the virtual-thread executor. Upgrade path to B3 (outbox) if delivery guarantees become required. - Delivery: A2 SSE with an in-memory emitter registry (
userId → emitters, one user may have several tabs), emitting only{"unread": n}hints — the browser then fetches via REST. Polling stays as a fallback (React QueryrefetchInterval120 s while SSE is connected, 30 s while it is down, plus on window focus), so a dropped stream or a restart never loses data, only latency.- Push goes through an interface
NotificationPublisher { void unreadChanged(long userId); }:InMemorySsePublishernow;PgNotifyPublisher(LISTEN/NOTIFY) when a second instance appears;RedisPublisheronly beyond that (ADR-0003). Swapping the implementation does not touch the API, table or FE. - Auth (A′2): the browser opens the stream with
fetch+Authorizationheader and readsresponse.bodyas a stream; no token in the URL. The endpoint is a normal authenticated route (not inALLOW_ALL_METHOD/OPTIONAL_AUTH_PATH), so the existingJWTAuthenticationFilterreturns 401 without a valid JWT.JwtProviderdoes not exposeexp(TTL 30 days), so instead of closing on expiry the emitter times out after 30 min; every reconnect re-runs the JWT filter, so an expired or revoked token is rejected within ≤ 30 min. The client reconnects with the current token (or stops after logout). - Heartbeat comment every 25 s; emitter timeout 30 min (client auto-reconnects); per-user cap 5 connections (oldest closed).
- Deploy restarts the JVM → all streams drop; the client reconnects with exponential backoff (1 s → 30 s cap) plus 1–5 s jitter; on 401 it stops and waits for login. Hidden tabs keep the stream (cheap);
AbortControllercloses it on logout/unmount.
- Push goes through an interface
- Email: C2 digest every 15 minutes, only for users with
notify_email = true, only for unread notifications not yet emailed; at most one email per user per run. - Recipient resolution: comment
user_id(email) →users.idviaUserRepoat event-handling time; guests and deactivated users are skipped. - Grouping: multiple replies in the same thread for the same recipient while unread are collapsed into one row (
actor_count, latest actor,updated_atbumped) — avoids a flood from busy threads.
Data Model¶
CREATE TABLE notifications (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
user_id BIGINT NOT NULL REFERENCES users(id) ON DELETE CASCADE,
type VARCHAR(30) NOT NULL, -- COMMENT_REPLY | COMMENT_REMOVED
group_key VARCHAR(120) NOT NULL, -- e.g. 'REPLY:<rootCommentId>', 'REMOVED:<commentId>'
video_id VARCHAR(100),
comment_id UUID, -- comment to deep-link to
actor_display VARCHAR(100), -- e.g. 'bob' (never full email)
actor_count INT NOT NULL DEFAULT 1,
payload JSONB NOT NULL DEFAULT '{}', -- snippet (≤140 chars), reason code
read_at TIMESTAMPTZ,
emailed_at TIMESTAMPTZ,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX idx_notifications_user_unread ON notifications (user_id, read_at, updated_at DESC);
CREATE UNIQUE INDEX uq_notifications_unread_group
ON notifications (user_id, group_key) WHERE read_at IS NULL; -- enables collapsing via upsert
CREATE INDEX idx_notifications_digest ON notifications (emailed_at, read_at) WHERE emailed_at IS NULL;
Retention: a scheduled job deletes read notifications older than 90 days.
API Contract¶
All endpoints require JWT; a user only ever sees their own rows.
GET /v1/funny-app/notifications?page=&size=20 → ResultObjectInfo<Page<NotificationDto>> (updated_at DESC)
GET /v1/funny-app/notifications/unread-count → ResultObjectInfo<{ count: number }>
GET /v1/funny-app/notifications/stream → text/event-stream (Authorization header required; 401 otherwise)
event: unread data: {"unread": 3} (on connect and on every change)
: ping (every 25 s)
PATCH /v1/funny-app/notifications/{id}/read → 204 (404 if not owner — no existence leak)
PATCH /v1/funny-app/notifications/read-all → 204
// NotificationDto
{
"id": "uuid",
"type": "COMMENT_REPLY",
"videoId": "22",
"commentId": "uuid",
"actorDisplay": "bob",
"actorCount": 3,
"snippet": "haha same here…",
"reason": null,
"read": false,
"updatedAt": "2026-10-02T10:00:00Z"
}
FE renders text from type + fields (i18n stays in FE), e.g. "bob và 2 người khác đã trả lời bình luận của bạn", "Bình luận của bạn đã bị gỡ: SPAM".
Flow¶
POST /videos/{id}/comments (reply)
└─ VideoCommentServiceImpl.createComment ──tx commit──►
publish CommentRepliedEvent(videoId, rootId, parentId, newCommentId, actorEmail, snippet)
└─ NotificationEventListener (@TransactionalEventListener AFTER_COMMIT, @Async)
recipients = {root.author, parent.author} − {actor} − guests
for r: upsert notifications ON CONFLICT (user_id, group_key) WHERE read_at IS NULL
DO UPDATE SET actor_count = actor_count + 1, actor_display = EXCLUDED.actor_display,
payload = EXCLUDED.payload, updated_at = now(), emailed_at = NULL
notificationPublisher.unreadChanged(r) → SSE "unread" to r's open tabs (if any)
mark read / read-all → notificationPublisher.unreadChanged(currentUser) (sync other tabs)
AdminCommentServiceImpl.moderate / bulkModerate (REMOVE) ──► CommentRemovedEvent(commentIds, reason) ──► same listener
AppScheduler.notificationDigest (every 15 min, ShedLock-free: single replica; guard with DB row lock later)
users with notify_email && unread && emailed_at IS NULL → 1 email each → set emailed_at
FE:
useNotificationStream(): fetch('/notifications/stream', { headers: { Authorization: jwt }, signal })
→ read res.body (TextDecoderStream), split on blank line, parse `event:` / `data:`
→ setQueryData(['notifications','unread']) + invalidate ['notifications','list']
→ on end/error: backoff + jitter reconnect; on 401: stop until login
AppShell header: bell icon + badge ← useUnreadCount() (fallback refetch 120 s / 30 s when stream down, on focus)
Bell click → dropdown (latest 10) ← useNotifications(page)
Item click → mark read → open video + CommentPanel scrolled to commentId
"Đánh dấu đã đọc tất cả"
Security & Privacy¶
- Owner-only access enforced in the repository query (
WHERE user_id = :currentUserId);PATCHon someone else's id → 404 (CWE-639). - SSE stream: authenticated with the
Authorizationheader (A′2) — no credential ever appears in a URL, so nothing sensitive reaches nginx/Cloudflare logs. The stream only carries the caller's own unread count. - Rule (project-wide): never put JWTs or guest tokens in URLs (fixed for video streaming in
9e1fb36; documented inwebapp/CLAUDE.md). actor_displayis the email local part / display name, never the full email (consistent with the pending public-email fix).- Snippet truncated to 140 chars and rendered as text (no HTML) — XSS-safe in FE and email (plain-text email).
- Removed/deleted comments: notification shows reason only, never the removed content.
- Email digest uses existing
EmailCacheLimiter; unsubscribe = Settings toggle (link in email footer).
Operations (single VM)¶
Verified on the server (nginx -T, nginx 1.24, 2026-10-02). Chain: Browser → Cloudflare (proxied) → nginx *.canh-labs.com → localhost:8081.
Existing location /api/ (proxies to http://localhost:8081/, strips /api) already has:
| Setting | Value | Effect on SSE |
|---|---|---|
proxy_buffering |
off |
✅ events flushed immediately — no extra location needed for buffering |
proxy_http_version |
1.1 |
✅ keep-alive to upstream |
proxy_read_timeout |
default 60s |
✅ OK only because the 25 s heartbeat resets it; never raise heartbeat above ~50 s |
gzip |
on, default gzip_types (text/html only) |
✅ text/event-stream not compressed (compression would buffer) |
access_log |
default format, full $request incl. query string |
✅ irrelevant for the stream: auth is in a header (A′2). Note: video stream URLs carried ?token=<jwt> until 9e1fb36 — older log files still contain tokens (cleanup tracked separately) |
| Cloudflare | proxied; idle timeout ~100 s | ✅ heartbeat 25 s keeps the connection open; Cloudflare streams text/event-stream without buffering |
So SSE works through the current config without changes. Optional hardening — a dedicated location with a longer read timeout and a log format that drops query strings (defence in depth):
# http {} : log format without query string
log_format no_query '$remote_addr - $remote_user [$time_local] "$request_method $uri $server_protocol" '
'$status $body_bytes_sent "$http_referer" "$http_user_agent"';
# server *.canh-labs.com {} : before "location /api/"
location /api/v1/funny-app/notifications/stream {
proxy_pass http://localhost:8081/v1/funny-app/notifications/stream;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_buffering off;
proxy_read_timeout 1h; # heartbeat still sent; this just avoids 60 s cut if a ping is late
access_log /var/log/nginx/access.log no_query;
}
The backend also sets X-Accel-Buffering: no and Cache-Control: no-cache. Spring: server.tomcat.max-connections default 8192; expose notifications_sse_active_connections on /actuator/prometheus.
Connection capacity (measured 2026-10-03)¶
An SSE stream does not occupy port 443 — it occupies one TCP connection / file descriptor per hop:
Browser ══HTTP/2══► Cloudflare ──HTTP/1.1──► nginx :443 ──HTTP/1.1──► Tomcat :8081
(multiplexed with the (1 conn per stream) (1 conn per stream)
page's other requests)
nginx holds 2 connections per stream (downstream + upstream). Server facts: 1 vCPU → worker_processes auto = 1 worker; JVM nofile 65535, heap -Xmx256m.
| Setting | Before | After (applied 2026-10-03, backup /etc/nginx/nginx.conf.bak-20261003-sse) |
|---|---|---|
events.worker_connections |
768 → ≈ 380 SSE streams max, shared with video/API traffic; exceeding it rejects all traffic (worker_connections are not enough) |
8192 → ≈ 4,000 streams |
worker_rlimit_nofile |
unset (soft limit 1024) | 16384 |
Applied with nginx -t && systemctl reload nginx (graceful, no dropped connections). Next limits after nginx: Tomcat max-connections 8192 and the 256 MB heap — revisit both before ~4,000 concurrent streams.
Scaling Path¶
| Concurrent online users | Setup |
|---|---|
| < 4k (today, after nginx tuning) | One JVM, InMemorySsePublisher, polling fallback |
| 4k – 8k | Same, raise nginx worker_connections, JVM heap and VM size (more vCPU = more nginx workers); watch the SSE gauge |
| > 1 instance needed | PgNotifyPublisher (PostgreSQL LISTEN/NOTIFY, one dedicated non-pooled connection per instance, payload = userId) — no new infra |
| 50k+ or high event rate | RedisPublisher (revisit ADR-0003) or a dedicated push gateway |
Email will hit limits first: Gmail SMTP allows roughly 500–2,000 messages/day. Once the digest exceeds that, move to a transactional provider (SES / SendGrid / Mailgun) behind the existing MailService interface.
Consequences¶
Positive
- Activates the existing
notify_emailsetting; users learn about replies and moderation. - Near-instant badge updates; idle users cost one idle connection instead of a request per minute.
- Zero new infrastructure; transport is behind
NotificationPublisher, so moving to more instances is a new implementation, not a redesign. - Event-driven producer keeps comment/admin code unaware of notifications.
Negative / trade-offs
- SSE adds moving parts: hand-written stream parsing and reconnect/backoff in the FE (A′2), heartbeat, reconnect after each deploy.
- In-memory registry is correct only while there is exactly one JVM; adding a second instance requires switching to
PgNotifyPublisher(polling fallback keeps it correct-but-slow if forgotten). - Up to 15 min email latency (digest).
- AFTER_COMMIT +
@Asynccan drop a notification if the process crashes in between — accepted for v1.
Follow-ups
PgNotifyPublisherwhen a second instance is introduced.- v1.2:
video_sources.owner_user_idto enable "your video got a comment/like". @mentionnotifications once display names exist.- Outbox (B3) if guaranteed delivery becomes a requirement.
- Transactional email provider when the digest volume approaches Gmail limits.
Implementation Outline (for /swarm-plan)¶
| ID | Task | Est. |
|---|---|---|
| NT-1 | Migration notifications, entity, repo (owner-scoped queries, native upsert) |
0.5 d |
| NT-2 | Events + NotificationEventListener (reply, removed incl. bulk), recipient resolution, grouping |
1 d |
| NT-3 | NotificationController (list, unread-count, read, read-all) + tests (owner isolation) |
0.5 d |
| NT-4 | SSE: NotificationPublisher + InMemorySsePublisher, /notifications/stream (header JWT via existing filter, heartbeat, per-user cap, 30-min emitter timeout), SSE gauge, tests (401 without header) |
1 d |
| NT-5 | Digest job + plain-text email template + retention job | 0.5 d |
| NT-6 | FE: useNotificationStream (fetch + Authorization header, stream parser, backoff + jitter reconnect, stop on 401, AbortController) + polling fallback, bell + badge + dropdown + deep link + tests (parser split across chunks, reconnect, 401) |
1.5 d |
| NT-7 | (optional) nginx stream location: proxy_read_timeout 1h + no_query log format (manual on VM, nginx -t then reload) |
0.25 d |
| NT-8 | E2E on prod (two browsers: reply → badge updates instantly; restart → reconnect); update docs/tracking.md |
0.25 d |
Total ≈ 5.5 days.