Plan: Admin Video Import (YouTube / Facebook → Google Drive)¶
Overview¶
Status: Done Author: nguyenhuuca Date: 2026-10-10 Related PRD: PRD-admin-video-import Related ADR: ADR-0019 (proposed)
Verified Facts (exploration 2026-10-10)¶
| Area | Fact | Impact |
|---|---|---|
| Drive client | ConfigBean.googleDrive() Service Account, scope DriveScopes.DRIVE (full) |
Upload API available; quota risk → VI-0 PoC |
| Folder | AppConstant.FOLDER_ID hardcoded |
Upload into the same id |
| Cron | AppScheduler.syncDriveVideos 0 */15 * * * *, window createdTime >= now-60m, skip if video-cache/{id}.full exists |
Uploaded file is ingested by the next run |
| Title | saveInfo(id, file.getName().replaceFirst("[.][^.]+$", ""), …) |
Drive file name = title contract |
| Dedupe | videoSourceRepository.existsBySourceId(driveFileId) |
Use for "Đã lên app" |
| Runtime | Prod = JAR on host via startServer.sh (no container); yt-dlp installed on host by owner |
ProcessBuilder("yt-dlp", …) works |
| Process pattern | FfmpegServiceImpl uses ProcessBuilder, no timeout |
New runner adds timeouts / destroy |
| Scheduling | @EnableScheduling + @EnableAsync, no ShedLock, no job table |
New table + FOR UPDATE SKIP LOCKED claim |
| Admin BE | AdminController class-level @PreAuthorize("hasRole('ADMIN')"), @HasPermission(perm = Permission.ADMIN), @AuditLog, Page<T> + @PageableDefault |
Same conventions |
| Admin FE | AdminView TABS (VIDEO_VAULT, USER_ACCOUNTS, FLAGGED_LOGS, COMMENTS); api/admin.js; hooks/useAdmin.js (no polling yet); .admin-* CSS |
New tab IMPORT, first refetchInterval use |
| Last migration | 202610070001-users-password-login.sql |
New 202610100001-create-video-import-jobs.sql |
Implementation Steps¶
Phase 0 — PoC Drive upload (0.25 d) — VI-0 ⚠️ gate¶
- [ ] Small test (dev profile / one-off runner) uploading a 1 MB file into
FOLDER_IDwith the Service Account - [ ] Success →
ServiceAccountDriveUploader;403 storageQuotaExceeded→OAuthDriveUploader(owner creates OAuth client, consents once, stores refresh token in.env) — record result in ADR-0019 D2 and accept the ADR - [ ] Delete the test file afterwards
Phase 1 — Data & config (0.5 d) — VI-1¶
- [ ] Migration
202610100001-create-video-import-jobs.sql(ADR-0019 D3 columns, indexes, partial unique on activenormalized_url) - [ ] Entity
VideoImportJob(+@Builder.Defaultfor status/progress/attempts), enumImportStatus,ImportPlatform;VideoImportJobRepositorywith nativeclaimNextDue()(FOR UPDATE SKIP LOCKED … RETURNING), throttledupdateProgress,resetInterrupted - [ ]
AppProperties.videoImport:enabled,ytDlpPath(yt-dlp),workDir(/opt/data/imports),maxConcurrent(1),maxFileSize(1G),downloadTimeout(30m),metadataTimeout(60s),minFreeDisk(2GB),cookiesPath(empty),bulkMax(20) - [ ]
ImportUrlValidator: https, host allowlist, length, normalization (YouTube id fromwatch?v=,youtu.be/,/shorts/; Facebookreel/{id},videos/{id},watch?v=,fb.watch/…kept as-is) + unit tests (incl.javascript:,http:,file:,youtube.com.evil.com, userinfo@, IP hosts) - [ ]
TitleSanitizer+ tests
Phase 2 — Process runner & uploader (1 d) — VI-2 (after VI-0, VI-1)¶
- [ ]
YtDlpClient(interface + impl):fetchMetadata(url)(-J --skip-download, 60 s) →{id, title, duration, filesize?};download(url, dir, listener, cancelToken)with ADR-0019 D4 args; parsedownload:progress lines; bounded stderr ring buffer; map exit/stderr →error_code(UNSUPPORTED_URL,LOGIN_REQUIRED,TOO_LARGE,EXTRACTOR_ERROR,TIMEOUT,CANCELLED,NOT_INSTALLED) - [ ]
DriveUploaderinterface + impl chosen in VI-0: resumable upload (8 MB chunks),parents=[FOLDER_ID], name{title}.mp4,appProperties.importJobId, progress listener → returns file id - [ ] Startup check:
yt-dlp --versionlogged; missing binary → feature reportsNOT_INSTALLEDinstead of crashing - [ ] Tests: argument list (no shell,
--before URL,--ignore-config,--use-extractors), progress parsing, timeout → destroy, error mapping (fake process via injectableProcessFactory)
Phase 3 — Service, worker, API (1 d) — VI-3 (after VI-1, VI-2)¶
- [ ]
VideoImportService:create(items, scheduledAt, adminId)(≤ bulkMax, per-line result{line, jobId | errorCode}),list(status, page),preview(url),runNow(id),cancel(id),retry(id);ingestedflag viaexistsBySourceId(drive_file_id)(batchedfindSourceIdsIn) - [ ]
VideoImportWorker:@Scheduled(cron = "0 */5 * * * *")(owner, 2026-10-11: schedules on 5-minute slots) +wakeUp()on virtual thread;Semaphore(maxConcurrent); flow: claim → disk guard → resolve title (if none) → download (status DOWNLOADING) → UPLOADING → DONE +drive_file_id→ delete work dir (always,finally); cancel = flag + destroy process; startupresetInterrupted(attempts < 3) - [ ] Endpoints under
AdminControlleror newAdminVideoImportController(/admin/video-imports):POST(create,@RateLimited(permit = 10)),GET(page, status filter),POST /preview,POST /{id}/run-now,POST /{id}/cancel,POST /{id}/retry— all@HasPermission(ADMIN)+@AuditLog; errors viaCustomException(400INVALID_URL, 409DUPLICATE_ACTIVE, 409INVALID_STATE) - [ ] No retention: job history is kept forever (owner, 2026-10-10); list is paginated, newest first
- [ ] Tests: service rules, state transitions, claim (repository
@DataJpaTestif available, else mocked), worker happy path / failure / cancel / interrupted reset, controller security (non-admin 403)
Phase 4 — Frontend (1.25 d) — VI-4 (contract-first, parallel with VI-2/3)¶
- [ ]
api/admin.js:createImports,listImports,previewImport,runImportNow,cancelImport,retryImport - [ ]
hooks/useAdmin.js:useVideoImports(page, status)withrefetchInterval: data => hasActive(data) ? 2000 : false; mutations invalidate['admin','imports'] - [ ]
AdminViewnew tab Import Video →AdminImportPanel:- Form: textarea (one URL per line, optional
URL | Tiêu đề), single-URL mode shows a title input + "Lấy tiêu đề" (preview); radio Tải ngay / Hẹn giờ +datetime-local(Asia/Ho_Chi_Minh, must be future); submit shows per-line results - Table: title, platform icon, status badge, phase + progress bar (%, MB), scheduled/started time, "Đã lên app"/"Chờ đồng bộ", error message (Vietnamese map), actions run-now / cancel / retry
- Form: textarea (one URL per line, optional
- [ ] Tests: line parsing, validation, schedule-in-past blocked, polling on/off by status, actions, error mapping
- [ ]
npm run lint && npx vitest run && npm run build
Phase 5 — Ops, verification, docs (0.5 d) — VI-5¶
- [ ] Host:
ffmpeg,yt-dlppresent;/opt/data/importsowned by the app user; dailyyt-dlp -Ucron; optional cookies filechmod 600 - [ ] Config
.env:VIDEO_IMPORT_ENABLED=true(+ OAuth vars if VI-0 chose OAuth) - [ ] Prod E2E: 1 YouTube now → Drive file appears → within 15 min in app with the right title; bulk 3 (1 FB reel, 1 Short, 1 invalid) → per-line result; schedule +5 min; cancel during download; retry failed
- [ ]
/security-auditoron the diff (process execution, SSRF, authz) - [ ]
docs/tracking.md, mkdocs nav (ADR-0019, PRD, plan),TASKS.md
Dependency Graph¶
VI-0 (PoC upload) ─┐
VI-1 (data/config) ┼──► VI-2 (yt-dlp + uploader) ──► VI-3 (service/worker/API) ──┐
│ ├──► VI-5
VI-4 (FE, contract-first) ──────────────────────────────────────────────────────┘
Suggested swarm: BE builder VI-0 → VI-1 → VI-2 → VI-3 (shared AppProperties, AdminController); FE builder VI-4 in parallel from the API contract below.
Estimate: ≈ 4.5 days (critical path ≈ 3 days).
API Contract (for VI-4)¶
POST /admin/video-imports
{ "items": [{ "url": "...", "title": "optional" }], "scheduledAt": "2026-10-11T02:00:00Z" | null }
→ 200 { data: { results: [{ "line": 1, "jobId": 12 } | { "line": 2, "errorCode": "INVALID_URL" }] } }
GET /admin/video-imports?status=&page=&size= → Page<VideoImportJobDto>
POST /admin/video-imports/preview { "url": "..." } → { title, platform, durationSec }
POST /admin/video-imports/{id}/run-now | /cancel | /retry → VideoImportJobDto
VideoImportJobDto {
id, sourceUrl, platform, title, status, phase, progressPct, downloadedBytes, totalBytes,
scheduledAt, startedAt, finishedAt, driveFileId, ingested, errorCode, errorMessage, attempts, createdAt
}
Risks¶
| Risk | Mitigation |
|---|---|
| Upload rejected by Drive quota | VI-0 gate before building the uploader |
| Import starves streaming (CPU/disk/network) | Concurrency 1, 720p cap, ffmpeg merge only |
| Stuck child process | Timeouts + destroyForcibly, finally cleanup |
| Option / command injection via URL | Allowlist + normalization, --, arg list, no shell |
| Title with path chars breaks Drive name / cron title | TitleSanitizer |
| Cron 60-min window misses a file if the cron is down > 1 h | Out of scope (same as manual uploads today); note in ops |