Multi-Platform Publishing Pipeline: Queue Architecture, Local Dispatcher, and Platform Adapters

1. System Overview

Managing short-form video releases across multiple social networks often breaks down due to fragile browser automation or bulky third-party SaaS scheduling platforms that lock files behind recurring fees. To solve this locally, I designed a decoupled publishing pipeline built around a filesystem-backed state queue, an isolated Python HTTP gateway, and modular API upload adapters.

The system isolates three concerns:

• Filesystem State Machine: Clean directory separation (queue/ for pending items and done/ for completed runs) guarantees that file state is inspectable using standard Unix tools.
• Central Local Dispatcher: A lightweight Python HTTP server listening on 127.0.0.1:8765 that abstracts thumbnail generation, job logging, streaming endpoints, and review gates.
• Decoupled Upload Adapters: Dedicated platform adapters for Facebook, Instagram, Threads, and YouTube that interact strictly with official platform APIs and handle authentication independently.

2. Directory Pipeline & State Management

Instead of relying on an external database, the pipeline manages execution lifecycle through structured directories on the local machine:

Redacted Directory Layout

Below is the verified directory layout of the local repository with sensitive account identifiers and credentials redacted:

[REDACTED_REPO_ROOT]/auto-publish-v1/
├── adapters/
│   ├── upload_facebook.py       # Meta Graph API v21.0 (Reels)
│   ├── upload_instagram.py      # Meta Graph API v21.0 (Resumable upload)
│   ├── upload_threads.py        # Threads API v1.0 (Container flow)
│   └── upload_youtube.py        # YouTube Data API v3 (OAuth2)
├── credentials/                 # Local secrets [REDACTED]
├── done/
│   ├── [REDACTED_VIDEO_1].mp4   # 1,048,576 bytes
│   └── [REDACTED_VIDEO_2].mp4   # 5,538,005 bytes
├── logs/
│   ├── jobs.jsonl               # 61 recorded audit events
│   └── jobs_registry.json       # 80 lines active registry
├── queue/
│   ├── [REDACTED_VIDEO_1]_thumb.jpg
│   └── [REDACTED_VIDEO_2]_thumb.jpg
├── tools/
│   ├── daily_scheduler.py       # 18:30 queue scanner
│   ├── extract_thumbnail.py     # ffmpeg frame extractor
│   ├── local_dispatcher.py      # HTTP gateway on :8765
│   └── sync_google_sheet.py     # Audit reporting
└── RUNBOOK.md

3. Platform Adapters

Each social network exposes different upload semantics, binary chunk limits, and container initialization requirements. The architecture encapsulates these differences into independent Python modules:

4. Local Gateway Dispatcher

At the heart of the pipeline is tools/local_dispatcher.py, a Python HTTP server binding to 127.0.0.1:8765. It functions as an orchestration gateway providing dedicated endpoints:

• /extract-thumbnail: Uses local ffmpeg to capture a high-quality frame (< 500 KB) for preview and review gates.
• /register-job: Registers an incoming job, tracks state in logs/jobs_registry.json, and dispatches approval messages.
• /publish: Dispatches upload commands to specific platform adapters or orchestrates fan-out across multiple adapters.
• /videos/<filename>: Serves range-based streaming video chunks needed by web container ingestion APIs.

Redacted Job Audit Log Excerpt

Every action produces an append-only audit record in logs/jobs.jsonl. Below is a representative record with sensitive account and token details redacted:

{
  "job_id": "[REDACTED_JOB_ID]",
  "video_name": "[REDACTED_FILENAME].mp4",
  "status": "COMPLETED",
  "platforms": {
    "youtube": {"status": "SUCCESS", "id": "[REDACTED]"},
    "facebook": {"status": "SUCCESS", "id": "[REDACTED]"},
    "instagram": {"status": "SUCCESS", "id": "[REDACTED]"},
    "threads": {"status": "SUCCESS", "id": "[REDACTED]"}
  },
  "timestamp": "2026-10-[REDACTED]T[REDACTED]"
}