Direct-to-object-storage upload service with application-enforced authorization

Architecture

React authenticates users through Keycloak and calls FastAPI over TLS. FastAPI validates the authenticated principal and requested upload metadata, applies application-specific permission and quota rules, creates a pending-upload record in PostgreSQL, generates an object key that is not user-controlled, and returns a narrowly scoped, short-lived Amazon S3 signed upload URL. The browser uploads directly to the signed URL, then calls a completion endpoint. FastAPI verifies expected object metadata/state with storage before marking the record available. For downloads, FastAPI checks PostgreSQL-backed ownership or sharing policy and issues a short-lived signed read URL only when authorized. PostgreSQL is the system of record for ownership, object state, retention/deletion requests, audit-relevant events, and logical object references; S3 holds file bytes. Lifecycle/retention rules must be configured deliberately in storage and reconciled with application records. If an upload is abandoned or a completion request fails, the object remains pending and is later reconciled or removed by an operational cleanup process.

Implementation sequence

• Define a PostgreSQL file model before implementing endpoints. Include an internal file ID, storage bucket and generated object key, owner subject ID, size, declared and detected content type fields, checksum if adopted, status (for example pending, available, rejected, delete_requested, deleted), creation/expiry timestamps, and a version or optimistic-lock field.
• Define authorization rules in the application: who may create an upload, maximum size and allowed types per tenant or role, who may read, delete, share, or extend retention, and whether shared access is user-specific or time-bound. Do not rely on object-key secrecy as authorization.
• Deploy and configure Keycloak as future implementation work: establish a realm/client for the application, redirect URIs, token audience and issuer settings, role/group mapping, administrator access controls, session policy, and key-rotation procedures. Configure FastAPI to verify token signature, issuer, audience, expiry, and relevant authorization claims.
• Create a private S3 bucket and a least-privilege server-side storage identity. Configure it so browsers do not receive general bucket credentials. Restrict the API identity to the exact buckets, prefixes, object operations, and lifecycle configuration actions required by the design.
• Implement an upload-initiation endpoint. Authenticate the caller, validate filename-independent metadata, select a server-generated opaque key, create a pending database row transactionally, and issue a short-expiry signed upload URL constrained to the expected object and, where supported by the chosen signing/configuration approach, expected method, size, and content-type-related conditions.
• In React, upload directly to the returned URL and report progress locally. Treat a successful browser transfer as provisional; call a completion endpoint rather than immediately presenting the file as available.
• Implement completion verification in FastAPI. Re-authorize the caller, retrieve object metadata from S3, compare it with the pending record and expected constraints, and atomically transition the record to available only on success. Mark mismatches rejected and schedule deletion rather than exposing them.
• Implement download and deletion endpoints that always authorize against PostgreSQL first. Generate short-lived signed read URLs only for available files. For delete requests, transition to delete_requested first, attempt storage deletion, and record the final result; make repeated requests idempotent.
• Configure retention/lifecycle behavior as future deployment work. Express retention dates in PostgreSQL and configure matching S3 lifecycle rules where appropriate. Add a scheduled reconciliation job or operational runbook to identify expired records, pending uploads beyond their allowed window, missing objects, and objects with no database record.
• Create recovery procedures. On failed upload or client disconnect, leave the record pending and expire it through reconciliation. On a database failure after storage upload, reconcile orphaned objects against database records. On storage failure after a delete request, retain delete_requested and retry safely. Design and test backup restoration for PostgreSQL metadata and a procedure to validate object/metadata consistency after restoration.
• Add automated tests for token rejection, cross-user access denial, URL expiry, tampered completion metadata, repeated completion/delete calls, oversized and disallowed-type requests, lifecycle edge cases, and reconciliation behavior. Perform deployment-specific integration and security testing before relying on the controls.

Security and operations

• Keep the S3 bucket private. Signed URLs are bearer credentials for their validity period; issue them only after authorization, scope them to one object and operation, use short expirations, and avoid logging full URLs, query strings, or credentials.
• Do not expose long-lived storage credentials in React bundles, browser storage, source control, or client-side configuration. Store server-side secrets in an appropriate deployment secret-management mechanism, which is not part of this required stack.
• Validate authorization server-side on every initiation, completion, download, delete, and share action. Client-side file checks are usability features, not a security boundary.
• Treat declared MIME types, filenames, and browser-supplied sizes as untrusted. Verify available storage metadata during completion and normalize filenames for display. Content-based type inspection and malware scanning require additional tooling or services not included in this stack.
• Use TLS for the React application, FastAPI API, Keycloak, PostgreSQL connections where supported by the deployment, and storage access. Certificate issuance, network segmentation, firewall rules, and reverse-proxy configuration are deployment prerequisites.
• Prevent unauthorized overwrite by generating object keys server-side and avoiding reuse. Decide whether versioning, object lock, or legal-hold behavior is required; these are policy and storage-configuration choices that must be validated for the target account and retention requirements.
• Minimize personal data in object metadata and API logs. Define audit retention, user deletion, retention exceptions, and access-review policies appropriate to the data classification.
• Back up PostgreSQL because it contains the authorization and object-reference record. Define object-data recovery expectations separately; database backups alone do not restore deleted or corrupted object bytes. Test restore and reconciliation procedures rather than assuming backups are usable.

Limitations

• The catalogue does not include a malware-scanning, content-disarmament, or file-type-inspection product. This design can restrict declared metadata and verify storage metadata, but it does not by itself establish that uploaded content is safe.
• No catalogue component is dedicated to scheduled job execution, worker orchestration, logs, alerting, or metrics. Reconciliation, deletion retries, lifecycle auditing, alerting, and incident response need an implementation/runtime plan outside the required component list.
• Amazon S3 signed-URL behavior, supported request constraints, eventing options, lifecycle semantics, regional requirements, retention features, and account policy must be verified against the selected S3 configuration before implementation. This design does not claim those settings have been configured or tested.
• A storage lifecycle policy can remove bytes independently of PostgreSQL. The application needs reconciliation and a clear user-facing policy for expired, unavailable, or partially deleted files.
• Keycloak, FastAPI, PostgreSQL, and React require hosting, scaling, patching, TLS termination, secret handling, and high-availability decisions. Those operating prerequisites are not supplied by this stack.
• Direct browser uploads depend on correctly configured cross-origin behavior and client network reliability. CORS policy, resumable/multipart upload behavior, and large-file limits must be designed and tested for the selected storage configuration.
• The proposal has no cross-region disaster-recovery design, immutable retention design, or regulatory-compliance assessment. Add these only after defining recovery objectives and data obligations.

Alternatives to consider

• Replace Amazon S3 (51) with Cloudflare R2 (52), Backblaze B2 (53), Wasabi Hot Cloud Storage (95), or self-hosted MinIO (94) if their evaluated signed-transfer, lifecycle, security, residency, and operational characteristics meet requirements. This requires validating API and policy differences rather than assuming interchangeability.
• Replace self-hosted Keycloak (41) with Clerk (43), Auth0 (42), or an authentication capability evaluated within Appwrite (40), Supabase (38), or Firebase (39) when managed identity better fits operational constraints. Revalidate token validation and authorization mapping in FastAPI.
• Use Django (22) rather than FastAPI (21) when the implementation team prefers its integrated application conventions; preserve the same server-side authorization and direct-upload boundary.
• Add a dedicated scanning service and asynchronous worker/queue only if the threat model requires malware scanning, deep inspection, transformations, or high-volume post-upload processing. These are intentionally excluded from the minimal required stack.