Provider-neutral CI/CD delivery for Elixir applications
  • Python 31.9%
  • Shell 28.6%
  • Jinja 20.9%
  • Elixir 18.6%
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
2026-11-10 13:10:19 -05:00
.github/workflows perf: bound derived thumbnail cache and prepare 0.4.56 #ci 2026-11-10 13:05:40 -05:00
docs Keep consumer delivery guidance to one short instruction 2026-10-26 17:45:15 -04:00
lib Keep consumer delivery guidance to one short instruction 2026-10-26 17:45:15 -04:00
priv fix: keep thumbnail caching opt-in for existing callers #ci 2026-11-10 13:10:19 -05:00
scripts Restore shared intensive command lock 2026-08-23 19:04:15 -04:00
templates perf: bound derived thumbnail cache and prepare 0.4.56 #ci 2026-11-10 13:05:40 -05:00
test fix: keep thumbnail caching opt-in for existing callers #ci 2026-11-10 13:10:19 -05:00
.formatter.exs Extract provider-neutral CI/CD delivery into a versioned package 2026-08-07 15:34:49 -04:00
.gitignore Provide production-only promotion of staging-verified Phoenix artifacts 2026-09-12 20:01:26 -04:00
CHANGELOG.md fix: keep thumbnail caching opt-in for existing callers #ci 2026-11-10 13:10:19 -05:00
mix.exs perf: bound derived thumbnail cache and prepare 0.4.56 #ci 2026-11-10 13:05:40 -05:00
README.md fix: keep thumbnail caching opt-in for existing callers #ci 2026-11-10 13:10:19 -05:00

ci_cd_harness

Provider-neutral CI/CD delivery for Elixir applications, consumed as a tagged git dependency.

Thumbnail content cache

Thumbnail generation reuses identical image bytes across filenames and runs. The cache key includes SHA-256, the converter/version and the actual rendering options. Cached output checksums are verified; incomplete or damaged entries are rendered again. Originals, test results and acceptance gates are unchanged. Metrics distinguish rendered images and cache hits.

Caching is opt-in through ATDD_THUMBNAIL_CACHE_DIR, for example _build/test/atdd-thumbnail-cache, outside published evidence. Existing callers retain uncached rendering. Consumers preserve the chosen directory in their private CI cache to reuse it across jobs. Derived entries expire after seven days without reuse. Oldest entries are trimmed to 256 MiB after generation. Cleanup only handles checksum-named cache entries and leaves originals and unrelated paths intact. Python 3 is required when opting into bounded cache maintenance. Ecojeux adoption is a separate consumer change.

Application error pages and compact guidance

Each application supplies its own styled 404 page, with useful navigation and an actual HTTP 404 response. The delivery harness does not impose a brand, language, or shared page design. JSON/API errors retain their machine-readable format. See the application error-page contract.

At adoption and after a package upgrade, explicitly run:

mix cd.update_agents

This inserts or refreshes one small managed block in the consumer's AGENTS.md, preserving all surrounding instructions. It does not run on compilation, tests, or ordinary use. mix cd.update_agents --check detects stale guidance without writing. If the consumer intentionally has no AGENTS.md, keep that policy: read the linked contract during adoption/upgrades instead of creating a file.

GitHub: build once, stage automatically, promote manually

Copy and configure the CI/staging caller and the manual production caller. They use the shared Phoenix workflows: no consumer-local deployment script is needed. Enable run-acceptance and your existing test command in the CI caller when your application has browser acceptance tests.

Push to main builds/tests one release, deploys it to staging, verifies its health identity and public HTML, and uploads a staging-verification receipt. After reviewing staging, open Actions → Promote production → Run workflow, select main, and enter the successful staging CI run's numeric ID from its /actions/runs/<id> URL. This production-only workflow does not rerun CI or touch staging. It downloads that run's release and receipt, checks repository, workflow path, main-branch event, successful run/attempt and staging job, then checks the archive SHA-256, version, release ID and source commit.

Configure the GitHub production environment with required reviewers and a main-branch deployment policy. The workflow uses that environment; approval is not created automatically by YAML. Its caller must grant actions: read and contents: write. Map production SSH secrets explicitly, or define matching secret names on the protected environment. Configure staging separately.

Both artifacts expire after 14 days. Old runs without a receipt are deliberately ineligible: adopt this version and complete a fresh staging CI run first. Missing, expired, failed, PR, wrong-workflow or stale-attempt sources fail closed. An explicitly selected older eligible run is an intentional rollback; promotion does not infer "latest" or require it to be the release currently on staging. The selected commit is checked out before tag preflight and deployment; tag publication follows successful production health verification. Release health retains the source run identity; the job summary records the separate promotion run and actor. Production jobs share a non-cancelling concurrency group (GitHub concurrency is not a FIFO deployment queue).

phoenix-delivery.yml still supports combined staging/production delivery for compatibility. Do not expose its deploy-production input as the normal manual production action: use the separate promotion caller instead. The shared core still owns SSH/rsync transport, migrations, health verification and release tags; only GitHub provenance/artifact selection lives in the GitHub adapter.

Why this exists

The predecessor was shared through GitLab's include: project:file:ref, which Forgejo has no equivalent for. Rather than copy its scripts into each application — leaving several divergent implementations of the same delivery logic — the shared behaviour moves here and is versioned through mix.lock, with the same immutability as the predecessor's pinned tag.

The canonical name carries no forge, deliberately. A provider-specific name became misleading the moment a second forge appeared; renaming it after each provider migration would repeat the mistake.

Design

Forge-specific code is thin. An adapter answers three questions — what commit, what run, and where do artifacts live — and everything downstream is written once against the normalized answers:

lib/ci_cd_harness.ex          forge detection and normalization
lib/mix/tasks/cd.*.ex         provider-neutral entry points
priv/core/                    shared shell, identical on every forge
                              (delivery, tagging, acceptance, test budgets)
priv/forges/forgejo/          adapter scripts, where shell is the right tool

priv/core/acceptance_evidence.sh runs the acceptance suite, builds the evidence site, evaluates the gate and writes the notification message. It is the GitLab template's logic verbatim, with one input made explicit: ACCEPTANCE_REPORT_URL, because each forge addresses run artifacts differently. The GitLab adapter derives an immutable job-artifact URL when a consumer does not provide an explicit live evidence URL. Evidence is produced even when scenarios fail — the gate is a separate step.

CiCdHarness.normalized_env/0 maps Forgejo's GitHub-compatible variables onto the CI_* names the existing deployment shell already understands. That is the seam that lets proven scripts run unchanged under a new forge instead of being rewritten.

Usage

{:ci_cd_harness, git: "https://git.agile-u.com/olivierg/ci-cd-harness.git", tag: "v0.4.56", only: [:dev, :test], runtime: false}

Build a release with a traceable identity:

mix cd.build_release

Writes _build/RELEASE_ID, VERSION, RELEASE_ARCHIVE and RELEASE_SHA256. The release ID is v<version>-<short sha>-<run id>, stable across build, deploy and verify so an artifact traces back to the run and commit that produced it.

The bootstrap exception

priv/forges/forgejo/setup_private_deps.sh installs the SSH keys that let mix deps.get reach any remaining private dependencies. This package itself is publicly readable over HTTPS, so consumers do not need a Forgejo credential to fetch it. Applications that still have private dependencies keep a copy of the bootstrap script in their own repository and run it before deps.get; everything after dependency resolution comes from this package.

Delivery scripts

Deployment reporting contract

Serialize deployments per repository and target so the observed previous release remains the deployment boundary. Shared GitLab jobs use a target resource group; GitHub jobs use non-cancelling target concurrency groups. Forgejo consumer jobs must keep their equivalent target serialization. These controls are not FIFO queues and do not coordinate another repository deploying to the same target.

deploy_release_fast.sh and notify_deployment.sh use one shared deployment report. The change boundary is the commit observed on the target environment before deployment, not the previous push or the most recent release tag. Staging and production each have their own boundary. Repeating the same commit reports zero commits; a rollback labels the removed commits; divergent history or missing provenance reports Commits: unavailable with a reason.

The collector resolves the health response's full git_sha/commit_sha, or the commit embedded in the standard release ID. CI_CD_PREVIOUS_DEPLOYED_SHA is an explicit override for callers that obtain the target identity another way. Shallow checkouts are hydrated from their existing origin with a bounded Git fetch. An unreachable boundary is never replaced by the latest commit or an arbitrary date window. Set CI_CD_COMMIT_HISTORY_FETCH=false to disable fetches; incomplete history then remains visibly unavailable.

The report in _build/deployment/<target>/commits.json retains every commit's full SHA, subject and complete message, including markers in commit bodies. It records the target, baseline, candidate, range direction and execution run. Deployment metadata and notifications consume that same report. Telegram gets the exact total and an escaped preview capped by a UTF-16 character budget, with an explicit omitted count. It does not cap the collected commit list. Keep the report and adjacent timing.env as CI artifacts, including on failure. The shared GitLab/GitHub workflows do this; Forgejo consumers add the same two paths to their deployment-job artifact upload.

Actual deployment duration comes from the shared deployer's entry/exit clock, not the job start. Job wait is creation-to-start only when both provider timestamps exist; dependency wait, deployment time and runner queue are not interchangeable. Pipeline age uses the executing run's creation timestamp. Unknown timings remain unknown.

The shared core automatically loads the Forgejo metadata adapter. Supply the temporary job-scoped FORGEJO_TOKEN: ${{ forgejo.token }} in the workflow/job environment. No personal token or consumer-local API script is needed. The adapter validates repository, SHA, run, job and UI URL, disables redirects, and retains only normalized metadata. The API run ID, UI run number and job ID are separate identities. Forgejo's current job API does not expose job-created or runner-queue timing, so those values are not invented. GitLab retains its native fields; GitHub retains its existing workflow identity.

CI_CD_EXECUTION_RUN_ID, CI_CD_EXECUTION_RUN_URL and CI_CD_EXECUTION_CREATED_AT describe the workflow executing this job. An explicit, different CI_PIPELINE_ID remains the staged artifact's source run during promotion; normalizing a promotion must not change deployed provenance. Reports validate execution identity when read by a later notification step.

Infrastructure repositories pin the shared Ansible roles; application workflows pin the delivery scripts. Updating an infrastructure submodule alone does not update an application's CI scripts. This reporting change uses the existing remote GIT_MESSAGES contract and does not require a production reprovision.

priv/core/ carries the delivery layer: blue/green deploy, health identity verification, SemVer tag preflight and publication, acceptance evidence, remote evidence copy and evaluation, and the ExUnit budget and value audits.

These previously lived in a repository each consumer cloned at a fixed directory name, and referred to each other through that name — so a consumer that cloned it elsewhere got a deploy that ran, reported success, and then failed on a missing sibling after deploying. They now resolve siblings relative to themselves, and a test enforces that.

Consuming them through this package also removes a network fetch from every job. Cloning them per job made CI depend on a forge that rate-limits SSH, which failed builds intermittently.

The executable scripts/resource_preflight.sh guards intensive local commands against active Linux memory pressure. The executable scripts/phoenix_test_database_preflight.sh requires linked worktrees to use a stable MIX_TEST_PARTITION and verifies that partition's test database before the test command starts. They remain at these public paths so application Mix aliases can share the checks without copying them. scripts/intensive_command_lock.sh serializes resource-heavy local commands with flock on Linux and an atomic-directory fallback on macOS; callers may choose immediate refusal or bounded operator-visible waiting.

Ansible roles

Reusable infrastructure roles live under priv/ansible/roles so both Mix dependency consumers and repositories that pin this project as a submodule use the same provider-neutral implementation. The package includes common, phoenix_postgres, phoenix_blue_green, web, phoenix_backup, and system_toolbox_identity.

Deployment lifecycle transport

The phoenix_blue_green role can call an optional consumer-owned executable configured with deploy_lifecycle_hook. It sends begin-deployment before shared migrations, then deployment-complete after verified cutover or deployment-aborted when the deploy fails. A failed begin hook aborts before migration; terminal hooks are best-effort because cutover may already have occurred.

The hook receives the event as its first argument and the same value in CI_CD_DEPLOYMENT_EVENT. Deployment ID, release ID, pipeline ID, environment, current and target colors, and an expiry epoch are available through CI_CD_DEPLOYMENT_* variables. Consumers must make quiescence expire no later than CI_CD_DEPLOYMENT_EXPIRES_AT_EPOCH; terminal events may clear it earlier. The contract does not require a shared filesystem marker.

Application-aware drain and fencing

Consumers whose application owns durable or long-running work can opt into deploy_application_lifecycle_enabled. The blue/green role then uses the configured localhost lifecycle path to drain the old slot, poll until it reports safe_to_stop, fence its ownership, and wait until the candidate reports active ownership. It never substitutes the legacy fixed sleep for an application safety decision.

Authentication is supplied through a root-readable curl config file named by deploy_application_lifecycle_curl_config, so bearer credentials do not enter process arguments or deployment output. If the bounded pre-fence wait expires, the harness restores the old nginx route, asks the old owner to resume claims, leaves it alive, and fails the deployment visibly. After fencing, rollback is not guessed: candidate promotion must succeed before the color is committed.

When release distribution is disabled, set deploy_lifecycle_state_file to an absolute host path readable by the application. The shared deploy service then writes the deployment ID and bounded expiry before migrations and removes only its own marker after completion or abort. This host-managed path does not invoke release RPC and may be used without a consumer hook. deploy_lifecycle_hook remains available for consumers with another supported transport; both may be enabled when both effects are intentional.

Environment badging

Non-production deployments should say so. See docs/environment_badging.md for the logo sticker, the per-environment favicon and its cache-busting trap, and the X-Robots-Tag the web role emits when app_is_production is false.

GitLab adapter

GitLab consumers can include the tagged public adapter directly:

include:
  - remote: "https://git.agile-u.com/olivierg/ci-cd-harness/raw/tag/v0.4.56/templates/gitlab/permit.yml"
  - remote: "https://git.agile-u.com/olivierg/ci-cd-harness/raw/tag/v0.4.56/templates/gitlab/acceptance.yml"
  - remote: "https://git.agile-u.com/olivierg/ci-cd-harness/raw/tag/v0.4.56/templates/gitlab/cd.yml"
  - remote: "https://git.agile-u.com/olivierg/ci-cd-harness/raw/tag/v0.4.56/templates/gitlab/quality.yml"

The adapter is deliberately thin: it defines GitLab's job graph and variable mapping, then fetches the same tag and runs the provider-neutral scripts under priv/core. Pin the include URL and CI_CD_HARNESS_REF to the same release. Artifact-producing reusable jobs expire their artifacts after two weeks. GitLab projects must also disable "Keep artifacts from most recent successful jobs" so that this maximum applies to the latest successful pipeline on each ref.

Status

Extracted from Agile-U's Forgejo delivery scripts, which remain in that repository until it migrates onto this package. Until then the logic exists in two places; close that before a third consumer appears.

System-toolbox infrastructure role

The package also ships the provider-neutral Ansible role at priv/ansible/roles/system_toolbox_identity. Infrastructure repositories keep their host inventory and rollout decisions, add this directory to roles_path, and opt in one host with system_toolbox_dedicated_user_enabled: true.

The role migrates an already deployed system-toolbox checkout from the deploy user's service to a locked system-toolbox system identity. It synchronizes the controller's selected source and dependencies under that identity, installs only the toolbox's root-owned maintenance helper and sudoers policy, preserves deploy-user source synchronization through ACLs, grants the identity only execute traversal on otherwise-private source parents, verifies the new /health service-manager identity, and restores the legacy service automatically when that verification fails. Exact privileged system units and deploy-user units remain consumer-owned allowlists.

Retention policies

The reusable Phoenix roles provide tiered retention for database/file snapshots and immutable releases. Consumers must set the policy explicitly after budgeting the entire physical host, including every co-located app, legacy release tree, logs, caches, and enough working space to create the next snapshot or unpack the next release. See docs/retention.md.

GitHub mirror

Forgejo olivierg/ci-cd-harness is the canonical repository. GitHub ogourment/github-ci-cd-harness is a push mirror so GitHub consumers can call the reusable workflows under .github/workflows. Develop and release only from the canonical repository; mirror commits and tags must resolve to the same objects.

Existing TLS certificates

The web role preserves HTTP-01 challenges at /.well-known/acme-challenge/ before the HTTPS redirect or application proxy. For an existing webroot-based certificate, set certbot_webroot to the path in its renewal configuration (default /var/www/letsencrypt). Other custom paths are not inferred. The role enables certbot.timer and reloads nginx after successful renewal of its certificate, validating configuration first.

After upgrading from 0.4.42 or earlier, review an Ansible web-role dry run and reapply that role on the consumer host. Then run a certificate-specific certbot renewal dry run and verify the served expiration date after actual renewal. Merely updating a Git pin does not change a host.

Artifact-only acceptance gate

AcceptanceHarness 0.11 supplies a Python standard-library gate reader. The evidence job retains it as tmp/atdd/acceptance_gate.py; the gate job uses that exact reader when present and otherwise uses the legacy Mix task. Preserve this file when overriding artifact paths. Upgraded consumers can remove PostgreSQL and Mix dependency setup from the gate job, but must provide Python 3 and preserve its evidence dependency and production gating. The evidence-producing job still runs its selected scenarios once. This optimization does not relax acceptance outcomes.

Watchdog public-failure handling (0.4.46)

After valid direct app readiness, a failing public endpoint records its reason (including curl exit code for transport failures) in the systemd journal and structured failure history. It retains the healthy service instead of restarting or failing over. One healthy probe does not erase the action cooldown. Public incidents require 60 seconds of observed health to close; unconfirmed notification sends remain retryable. Direct app failure still permits cooldown-limited recovery.

Consumers can apply just phoenix_blue_green tasks_from watchdog_script, using their normal inventory and role defaults. This installs only the managed script; it does not provision other roles or restart applications. Review --check --diff before applying. A dry run or template pin alone does not prove live behaviour.

Run the isolated regressions with python3 test/watchdog_recovery_test.py in an environment containing Jinja2 and jq. All network, service, proxy and notification operations are faked. Public outage, notification retry, stable recovery, direct app recovery and standby failover are checked against the actual managed template.

Operations-monitor error-log recovery

The independent operations monitor reports application error logs immediately, even when readiness and PostgreSQL are healthy. It closes that error-log episode only after three consecutive valid zero-error observations spanning at least 60 seconds. A new error or an unknown probe breaks the clean streak while retaining the active episode. The clean streak survives observer invocations; failed event delivery retries the identical payload and idempotency key. Readiness, database and disk failures retain their immediate alerts, and observer transport still requires three failed observations spanning 60 seconds.

Run the isolated monitor regressions with python3 -m unittest discover -s test -p phoenix_operations_monitor_test.py. They use synthetic probe processes and a local receipt receiver.

Security-only hotfixes (0.4.54)

The CI caller must grant actions: read alongside its existing permissions, because the reusable verifier reads run provenance. Main-only delivery remains the default. A consumer may explicitly set security-hotfix-policy: .github/security-hotfixes.json in its staging caller and its main-only promotion caller. A dispatched security/ branch is eligible only when a policy fetched from an immutable current-main revision approves its exact source SHA, branch, production base SHA, binary diff SHA-256 and expiry. The approved base must be an ancestor and identify a successful production-source run; application version must stay within a PATCH line and migrations are refused. Review the entire exact diff before adding an approval. The policy is a release authorization record, not an advisory scanner or automatic proof of security scope.

Schema1 policy: {"schema":1,"approvals":[{"purpose":"security-only","sha":"<40hex>", "base_sha":"<40hex>","branch":"security/fix","diff_sha256":"<64hex>", "expires_at":"<UTC timestamp>","production_base":{"version":"1.2.3", "release_id":"app-1.2.3-42","pipeline_id":"42"}}]}.

The same successful CI run must produce the original release and a genuine staging-verification receipt. Promotion rechecks the approval and all existing artifact/run/attempt/identity gates, and refuses a production base whose healthy version, release ID or source pipeline has moved. Existing deployment concurrency, health verification, rollback and release-tag protections remain in force. Remove expired or superseded approvals. A new SHA always needs a new reviewed approval.

Bounded optional GitLab history

Source priv/forges/gitlab/history_http.sh and call history_http_init once before historical reporting. history_http_get PATH buffers a successful response; HTTP errors, redirects and timeouts return nonzero without partial/error output. history_http_run COMMAND... applies the same deadline to local collection. The exported absolute deadline survives command substitutions and nested shells.

HISTORY_HTTP_BUDGET_SECONDS defaults to 120 and HISTORY_HTTP_REQUEST_SECONDS to 15 (both 1..300). Connect time is at most 5s; forced termination has a 2s grace. Once the shared budget is exhausted, callers must stop collecting and retain unavailable/NA values, preserving current quality job results. Rendering existing results is outside the historical collection budget.

Use trusted HISTORY_HTTP_API_ROOT, HISTORY_HTTP_AUTH_HEADER (PRIVATE-TOKEN or JOB-TOKEN) and HISTORY_HTTP_TOKEN server configuration. Headers pass through stdin rather than argv or disk. Redirects never receive credentials. Without an API root the helper bounds glab api instead. No credentials are provisioned.