photoprism/CODEMAP.md

PhotoPrism — Backend CODEMAP

Purpose

• Give agents and contributors a fast, reliable map of where things live and how they fit together, so you can add features, fix bugs, and write tests without spelunking.
• Sources of truth: prefer Makefile targets and the Developer Guide linked in AGENTS.md.

Quick Start

• Inside dev container (recommended):
  • Install deps: make dep
  • Build backend: make build-go
  • Run server: ./photoprism start
  • Open: http://localhost:2342/ or https://app.localssl.dev/ (Traefik required)
• On host (manages Docker):
  • Build image: make docker-build
  • Start services: docker compose up -d
  • Logs: docker compose logs -f --tail=100 photoprism

Executables & Entry Points

• CLI app (binary name across docs/images is photoprism):
  • Main: cmd/photoprism/photoprism.go
  • Commands registry: internal/commands/commands.go (array commands.PhotoPrism)
  • Catalog helpers: internal/commands/catalog (DTOs and builders to enumerate commands/flags; Markdown renderer)
• Web server:
  • Startup: internal/commands/start.go → server.Start (starts HTTP(S), workers, session cleanup)
  • HTTP server: internal/server/start.go (compression, security, healthz, readiness, TLS/AutoTLS/unix socket)
  • Routes: internal/server/routes.go (registers all v1 API groups + UI, WebDAV, sharing, .well-known)
  • API group: APIv1 = router.Group(conf.BaseUri("/api/v1"), Api(conf))

High-Level Package Map (Go)

• internal/api — Gin handlers and Swagger annotations; only glue, no business logic
• internal/commands/catalog — DTOs (App, Command, Flag, Node), builders (BuildFlat/BuildNode, CommandInfo, FlagsToCatalog), and a templated Markdown renderer (RenderMarkdown) for the CLI commands catalog. Depends only on urfave/cli/v2 and stdlib.
• internal/server — HTTP server, middleware, routing, static/ui/webdav
• internal/config — configuration, flags/env/options, client config, DB init/migrate
• internal/entity — GORM v1 models, queries, search helpers, migrations
• internal/photoprism — core domain logic (indexing, import, faces, thumbnails, cleanup)
• internal/workers — background schedulers (index, vision, sync, meta, backup)
• internal/auth — ACL, sessions, OIDC
• internal/service — cluster/portal, maps, hub, webdav
• internal/event — logging, pub/sub, audit
• internal/ffmpeg, internal/thumb, internal/meta, internal/form, internal/mutex — media, thumbs, metadata, forms, coordination
• pkg/* — reusable utilities (must never import from internal/*), e.g. pkg/fs, pkg/log, pkg/service/http/header

HTTP API

• Handlers live in internal/api/*.go and are registered in internal/server/routes.go.
• Annotate new endpoints in handler files; generate docs with: make fmt-go swag-fmt && make swag.
• Do not edit internal/api/swagger.json by hand.
• Swagger notes:
  • Use full /api/v1/... in every @Router annotation (match the group prefix).
  • Annotate only public handlers; skip internal helpers to avoid stray generic paths.
  • make swag-json runs a stabilization step (swaggerfix) removing duplicated enums for time.Duration; API uses integer nanoseconds for durations.
• Common groups in routes.go: sessions, OAuth/OIDC, config, users, services, thumbnails, video, downloads/zip, index/import, photos/files/labels/subjects/faces, batch ops, cluster, technical (metrics, status, echo).

Configuration & Flags

• Options struct: internal/config/options.go with yaml:"…" (for defaults.yml/options.yml), json:"…" (clients/API), and flag:"…" (CLI flags/env) tags.
  • For secrets/internals: json:"-" disables JSON processing to prevent values from being exposed through the API (see internal/api/config_options.go).
  • If needed: yaml:"-" disables YAML processing; flag:"-" prevents ApplyCliContext() from assigning CLI values (flags/env variables) to a field, without affecting the flags in internal/config/flags.go.
  • Annotations may include edition tags like tags:"plus,pro" to control visibility (see internal/config/options_report.go logic).
• Global flags/env: internal/config/flags.go (EnvVars(...))
  • Available flags/env: internal/config/cli_flags_report.go + internal/config/report_sections.go → surfaced by photoprism show config-options --md/--json
  • YAML options mapping: internal/config/options_report.go + internal/config/report_sections.go → surfaced by photoprism show config-yaml --md/--json
  • Report current values: internal/config/report.go → surfaced by photoprism show config (alias photoprism config --md).
  • CLI commands catalog: internal/commands/show_commands.go → surfaced by photoprism show commands (Markdown by default; --json alternative; --nested optional tree; --all includes hidden commands/flags; nested help subcommands omitted).
• Precedence: defaults.yml < CLI/env < options.yml (global options rule). See Agent Tips in AGENTS.md.
• Getters are grouped by topic, e.g. DB in internal/config/config_db.go, server in config_server.go, TLS in config_tls.go, etc.
• Client Config (read-only)
  • Endpoint: GET /api/v1/config (see internal/api/api_client_config.go).
  • Assembly: Built from internal/config/client_config.go (not a direct serialization of Options) plus extension values registered via config.Register in internal/config/extensions.go.
  • Updates: Back-end calls UpdateClientConfig() to publish "config.updated" over websockets after changes (see internal/api/config_options.go and internal/api/config_settings.go).
  • ACL/mode aware: Values are filtered by user/session and may differ for public vs. authenticated users.
  • Don’t expose secrets: Treat it as client-visible; avoid sensitive data. To add fields, extend client values via config.Register rather than exposing Options directly.
  • Refresh cadence: The web UI (non‑mobile) also polls for updates every 10 minutes via $config.update() in frontend/src/app.js, complementing the websocket push.

Database & Migrations

• Driver: GORM v1 (github.com/jinzhu/gorm). No WithContext. Use db.Raw(stmt).Scan(&nop) for raw SQL.
• Entities and helpers: internal/entity/*.go and subpackages (query, search, sortby).
• Migrations engine: internal/entity/migrate/* — run via config.MigrateDb(); CLI: photoprism migrate / photoprism migrations.
• DB init/migrate flow: internal/config/config_db.go chooses driver/DSN, sets gorm:table_options, then entity.InitDb(migrate.Opt(...)).

AuthN/Z & Sessions

• Session model and cache: internal/entity/auth_session* and internal/auth/session/* (cleanup worker).
• ACL: internal/auth/acl/* — roles, grants, scopes; use constants; avoid logging secrets, compare tokens constant‑time.
• OIDC: internal/auth/oidc/*.

Media Processing

• Thumbnails: internal/thumb/* and helpers in internal/photoprism/mediafile_thumbs.go.
• Metadata: internal/meta/*.
• FFmpeg integration: internal/ffmpeg/*.

Background Workers

• Scheduler and workers: internal/workers/*.go (index, vision, meta, sync, backup, share); started from internal/commands/start.go.
• Auto indexer: internal/workers/auto/*.

Cluster / Portal

• Node types: internal/service/cluster/const.go (cluster.RoleInstance, cluster.RolePortal, cluster.RoleService).
• Instance bootstrap & registration: internal/service/cluster/instance/* (HTTP to Portal; do not import Portal internals).
• Registry/provisioner: internal/service/cluster/registry/*, internal/service/cluster/provisioner/*.
• Theme endpoint (server): GET /api/v1/cluster/theme; client/CLI installs theme only if missing or no app.js.
• See specs cheat sheet: specs/portal/README.md.

Logging & Events

• Logger and event hub: internal/event/*; event.Log is the shared logger.
• HTTP headers/constants: pkg/service/http/header/* — always prefer these in handlers and tests.

Server Startup Flow (happy path)

1. photoprism start (CLI) → internal/commands/start.go
2. Config init, DB init/migrate, session cleanup worker
3. internal/server/start.go builds Gin engine, middleware, API group, templates
4. internal/server/routes.go registers UI, WebDAV, sharing, well‑known, and all /api/v1/* routes
5. Workers and auto‑index start; health endpoints /livez, /readyz available

Common How‑Tos

• Add a CLI command

  • Create internal/commands/<name>.go with a *cli.Command
  • Add it to PhotoPrism in internal/commands/commands.go
  • Tests: prefer RunWithTestContext from internal/commands/commands_test.go to avoid os.Exit

• Add a REST endpoint

  • Create handler in internal/api/<area>.go with Swagger annotations
  • Register it in internal/server/routes.go
  • Use helpers: api.ClientIP(c), header.BearerToken(c), Abort* functions
  • Validate pagination bounds (default count=100, max 1000, offset>=0) for list endpoints
  • Run make fmt-go swag-fmt && make swag; keep docs accurate
  • Tests: go test ./internal/api -run <Name> and focused helpers (NewApiTest(), PerformRequest*)

• Add a config option

  • Add field with tags to internal/config/options.go
  • Register CLI flag/env in internal/config/flags.go via EnvVars(...)
  • Expose a getter (e.g., in config_server.go or topic file)
  • Append to rows in *config.Report() after the same option as in options.go
  • If value must persist, write back to options.yml and reload into memory
  • Tests: cover CLI/env/file precedence (see internal/config/test.go helpers)

• Touch the DB schema

  • Use GORM auto-migration, or add a custom migration in internal/entity/migrate/<dialect>/... and run go generate or make generate (runs go generate for all packages)
  • Bump/review version gates in migrate.Version usage via config_db.go
  • Tests: run against SQLite by default; for MySQL cases, gate appropriately

Testing

• Full suite: make test (frontend + backend). Backend only: make test-go.
• Focused packages: go test ./internal/<pkg> -run <Name>.
• CLI tests: PHOTOPRISM_CLI=noninteractive or pass --yes to avoid prompts; use RunWithTestContext to prevent os.Exit.
• SQLite DSN in tests is per‑suite (not empty). Clean up files if you capture the DSN.
• Frontend unit tests via Vitest are separate; see frontend/CODEMAP.md.

Security & Hot Spots (Where to Look)

• Zip extraction (path traversal prevention): pkg/fs/zip.go

  • Uses safeJoin to reject absolute/volume paths and .. traversal; enforces per-file and total size limits.
  • Tests: pkg/fs/zip_extra_test.go cover abs/volume/.. cases and limits.

• Force-aware Copy/Move and truncation-safe writes:

  • App helpers: internal/photoprism/mediafile.go (MediaFile.Copy/Move with force).
  • Utils: pkg/fs/copy.go, pkg/fs/move.go (use O_TRUNC to avoid trailing bytes).

• FFmpeg command builders and encoders:

  • Core: internal/ffmpeg/transcode_cmd.go, internal/ffmpeg/remux.go.
  • Encoders (string builders only): internal/ffmpeg/{apple,intel,nvidia,vaapi,v4l}/avc.go.
  • Tests guard HW runs with PHOTOPRISM_FFMPEG_ENCODER; otherwise assert command strings and negative paths.

• libvips thumbnails:

  • Pipeline: internal/thumb/vips.go (VipsInit, VipsRotate, export params).
  • Sizes & names: internal/thumb/sizes.go, internal/thumb/names.go, internal/thumb/filter.go.

• Safe HTTP downloader:

  • Shared utility: pkg/service/http/safe (Download, Options).
  • Protections: scheme allow‑list (http/https), pre‑DNS + per‑redirect hostname/IP validation, final peer IP check, size and timeout enforcement, temp file 0600 + rename.
  • Avatars: wrapper internal/thumb/avatar.SafeDownload applies stricter defaults (15s, 10 MiB, AllowPrivate=false, image‑focused Accept).
  • Tests: go test ./pkg/service/http/safe -count=1 (includes redirect SSRF cases); avatars: go test ./internal/thumb/avatar -count=1.

Performance & Limits

• Prefer existing caches/workers/batching as per Makefile and code.
• When adding list endpoints, default count=100 (max 1000); set Cache-Control: no-store for secrets.

Conventions & Rules of Thumb

• Respect package boundaries: code in pkg/* must not import internal/*.
• Prefer constants/helpers from pkg/service/http/header over string literals.
• Never log secrets; compare tokens constant‑time.
• Don’t import Portal internals from cluster instance/service bootstraps; use HTTP.
• Prefer small, hermetic unit tests; isolate filesystem paths with t.TempDir() and env like PHOTOPRISM_STORAGE_PATH.
• Cluster nodes: identify by UUID v7 (internally stored as NodeUUID; exposed as uuid in API/CLI). The OAuth client ID (NodeClientID, exposed as clientId) is for OAuth only. Registry lookups and CLI commands accept uuid, clientId, or DNS‑label name (priority in that order).

Filesystem Permissions & io/fs Aliasing

• Use github.com/photoprism/photoprism/pkg/fs permission variables when creating files/dirs:
  • fs.ModeDir (0o755), fs.ModeFile (0o644), fs.ModeConfigFile (0o664), fs.ModeSecret (0o600), fs.ModeBackupFile (0o600).
• Do not use stdlib io/fs mode bits as permission arguments. When importing stdlib io/fs, alias it (iofs/gofs) to avoid fs.* collisions with our package.
• Prefer filepath.Join for filesystem paths across platforms; use path.Join for URLs only.

Cluster Registry & Provisioner Cheatsheet

• UUID‑first everywhere: API paths {uuid}, Registry Get/Delete/RotateSecret by UUID; explicit FindByClientID exists for OAuth.
• Node/DTO fields: uuid required; clientId optional; database metadata includes driver.
• Provisioner naming (no slugs):
  • database: photoprism_d<hmac11>
  • username: photoprism_u<hmac11> HMAC is base32 of ClusterUUID+NodeUUID; drivers currently mysql|mariadb.
• DSN builder: BuildDSN(driver, host, port, user, pass, name); warns and falls back to MySQL format for unsupported drivers.
• Go tests live beside sources: for path/to/pkg/<file>.go, add tests in path/to/pkg/<file>_test.go (create if missing). For the same function, group related cases as t.Run(...) sub-tests (table-driven where helpful).
• Public API and internal registry DTOs use normalized field names:
  • database (not db) with name, user, driver, rotatedAt.
  • Node-level rotation timestamps use rotatedAt.
  • Registration returns secrets.clientSecret; the CLI persists it under config NodeClientSecret.
  • Admin responses may include advertiseUrl and database; non-admin responses are redacted by default.

Frequently Touched Files (by topic)

• CLI wiring: cmd/photoprism/photoprism.go, internal/commands/commands.go
• Server: internal/server/start.go, internal/server/routes.go, middleware in internal/server/*.go
• API handlers: internal/api/*.go (plus docs.go for package docs)
• Config: internal/config/* (flags.go, config_db.go, config_server.go, options.go)
• Entities & queries: internal/entity/*.go, internal/entity/query/*
• Migrations: internal/entity/migrate/*
• Workers: internal/workers/*
• Cluster: internal/service/cluster/*
• Headers: pkg/service/http/header/*

Downloads (CLI) & yt-dlp helpers

• CLI command & core:
  • internal/commands/download.go (flags, defaults, examples)
  • internal/commands/download_impl.go (testable implementation used by CLI)
• yt-dlp wrappers:
  • internal/photoprism/dl/options.go (arg wiring; FFmpegPostArgs hook for --postprocessor-args)
  • internal/photoprism/dl/info.go (metadata discovery)
  • internal/photoprism/dl/file.go (file method with --output/--print)
  • internal/photoprism/dl/meta.go (CreatedFromInfo fallback; RemuxOptionsFromInfo)
• Importer:
  • internal/photoprism/get/import.go (work pool)
  • internal/photoprism/import_options.go (ImportOptionsMove/Copy)
• Testing hints:
  • Fast loops: go test ./internal/photoprism/dl -run 'Options|Created|PostprocessorArgs' -count=1
  • CLI only: go test ./internal/commands -run 'DownloadImpl|HelpFlags' -count=1
  • Disable ffmpeg when not needed: set FFmpegBin = "/bin/false", Settings.Index.Convert=false in tests.
  • Stub yt-dlp: shell script that prints JSON for --dump-single-json, creates a file and prints path for --print.
  • Avoid importer dedup: vary file bytes (e.g., YTDLP_DUMMY_CONTENT) or dest.

Useful Make Targets (selection)

• make help — list targets
• make dep — install Go/JS deps in container
• make build-go — build backend
• make test-go — backend tests (SQLite)
• make swag — generate Swagger JSON in internal/api/swagger.json
• make fmt-go swag-fmt — format Go code and Swagger annotations

See Also

• AGENTS.md (repository rules and tips for agents)
• Developer Guide (Setup/Tests/API) — links in AGENTS.md → Sources of Truth
• Specs: specs/dev/backend-testing.md, specs/dev/api-docs-swagger.md, specs/portal/README.md

Fast Test Recipes

• Filesystem + archives (fast): go test ./pkg/fs -run 'Copy|Move|Unzip' -count=1
• Media helpers (fast): go test ./pkg/media/... -count=1
• Thumbnails (libvips, moderate): go test ./internal/thumb/... -count=1
• FFmpeg command builders (moderate): go test ./internal/ffmpeg -run 'Remux|Transcode|Extract' -count=1