- Python 31.9%
- Shell 28.6%
- Jinja 20.9%
- Elixir 18.6%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
| .github/workflows | ||
| docs | ||
| lib | ||
| priv | ||
| scripts | ||
| templates | ||
| test | ||
| .formatter.exs | ||
| .gitignore | ||
| CHANGELOG.md | ||
| mix.exs | ||
| README.md | ||
ci_cd_harness
Provider-neutral CI/CD delivery for Elixir applications, consumed as a tagged git dependency.
Thumbnail content cache
Thumbnail generation reuses identical image bytes across filenames and runs. The cache key includes SHA-256, the converter/version and the actual rendering options. Cached output checksums are verified; incomplete or damaged entries are rendered again. Originals, test results and acceptance gates are unchanged. Metrics distinguish rendered images and cache hits.
Caching is opt-in through ATDD_THUMBNAIL_CACHE_DIR, for example
_build/test/atdd-thumbnail-cache, outside published evidence. Existing callers
retain uncached rendering. Consumers preserve the chosen directory in their
private CI cache to reuse it across jobs. Derived entries expire after seven days without reuse. Oldest entries are
trimmed to 256 MiB after generation. Cleanup only handles checksum-named cache
entries and leaves originals and unrelated paths intact. Python 3 is required
when opting into bounded cache maintenance. Ecojeux adoption is a separate consumer change.
Application error pages and compact guidance
Each application supplies its own styled 404 page, with useful navigation and an actual HTTP 404 response. The delivery harness does not impose a brand, language, or shared page design. JSON/API errors retain their machine-readable format. See the application error-page contract.
At adoption and after a package upgrade, explicitly run:
mix cd.update_agents
This inserts or refreshes one small managed block in the consumer's AGENTS.md,
preserving all surrounding instructions. It does not run on compilation, tests,
or ordinary use. mix cd.update_agents --check detects stale guidance without
writing. If the consumer intentionally has no AGENTS.md, keep that policy:
read the linked contract during adoption/upgrades instead of creating a file.
GitHub: build once, stage automatically, promote manually
Copy and configure the CI/staging caller and
the manual production caller.
They use the shared Phoenix workflows: no consumer-local deployment script is
needed. Enable run-acceptance and your existing test command in the CI caller
when your application has browser acceptance tests.
Push to main builds/tests one release, deploys it to staging, verifies its
health identity and public HTML, and uploads a staging-verification receipt.
After reviewing staging, open Actions → Promote production → Run workflow,
select main, and enter the successful staging CI run's numeric ID from its
/actions/runs/<id> URL. This production-only workflow does not rerun CI or
touch staging. It downloads that run's release and receipt, checks repository,
workflow path, main-branch event, successful run/attempt and staging job, then
checks the archive SHA-256, version, release ID and source commit.
Configure the GitHub production environment with required reviewers and a
main-branch deployment policy. The workflow uses that environment; approval is
not created automatically by YAML. Its caller must grant actions: read and
contents: write. Map production SSH secrets explicitly, or define matching
secret names on the protected environment. Configure staging separately.
Both artifacts expire after 14 days. Old runs without a receipt are deliberately ineligible: adopt this version and complete a fresh staging CI run first. Missing, expired, failed, PR, wrong-workflow or stale-attempt sources fail closed. An explicitly selected older eligible run is an intentional rollback; promotion does not infer "latest" or require it to be the release currently on staging. The selected commit is checked out before tag preflight and deployment; tag publication follows successful production health verification. Release health retains the source run identity; the job summary records the separate promotion run and actor. Production jobs share a non-cancelling concurrency group (GitHub concurrency is not a FIFO deployment queue).
phoenix-delivery.yml still supports combined staging/production delivery for
compatibility. Do not expose its deploy-production input as the normal manual
production action: use the separate promotion caller instead. The shared core
still owns SSH/rsync transport, migrations, health verification and release tags;
only GitHub provenance/artifact selection lives in the GitHub adapter.
Why this exists
The predecessor was shared through GitLab's include: project:file:ref, which
Forgejo has no equivalent for. Rather than copy its scripts into each
application — leaving several divergent implementations of the same delivery
logic — the shared behaviour moves here and is versioned through mix.lock,
with the same immutability as the predecessor's pinned tag.
The canonical name carries no forge, deliberately. A provider-specific name became misleading the moment a second forge appeared; renaming it after each provider migration would repeat the mistake.
Design
Forge-specific code is thin. An adapter answers three questions — what commit, what run, and where do artifacts live — and everything downstream is written once against the normalized answers:
lib/ci_cd_harness.ex forge detection and normalization
lib/mix/tasks/cd.*.ex provider-neutral entry points
priv/core/ shared shell, identical on every forge
(delivery, tagging, acceptance, test budgets)
priv/forges/forgejo/ adapter scripts, where shell is the right tool
priv/core/acceptance_evidence.sh runs the acceptance suite, builds the
evidence site, evaluates the gate and writes the notification message. It is the
GitLab template's logic verbatim, with one input made explicit:
ACCEPTANCE_REPORT_URL, because each forge addresses run artifacts differently.
The GitLab adapter derives an immutable job-artifact URL when a consumer does
not provide an explicit live evidence URL.
Evidence is produced even when scenarios fail — the gate is a separate step.
CiCdHarness.normalized_env/0 maps Forgejo's GitHub-compatible variables onto
the CI_* names the existing deployment shell already understands. That is the
seam that lets proven scripts run unchanged under a new forge instead of being
rewritten.
Usage
{:ci_cd_harness, git: "https://git.agile-u.com/olivierg/ci-cd-harness.git", tag: "v0.4.56", only: [:dev, :test], runtime: false}
Build a release with a traceable identity:
mix cd.build_release
Writes _build/RELEASE_ID, VERSION, RELEASE_ARCHIVE and RELEASE_SHA256.
The release ID is v<version>-<short sha>-<run id>, stable across build, deploy
and verify so an artifact traces back to the run and commit that produced it.
The bootstrap exception
priv/forges/forgejo/setup_private_deps.sh installs the SSH keys that let
mix deps.get reach any remaining private dependencies. This package itself is
publicly readable over HTTPS, so consumers do not need a Forgejo credential to
fetch it. Applications that still have private dependencies keep a copy of the
bootstrap script in their own repository and run it before deps.get;
everything after dependency resolution comes from this package.
Delivery scripts
Deployment reporting contract
Serialize deployments per repository and target so the observed previous release remains the deployment boundary. Shared GitLab jobs use a target resource group; GitHub jobs use non-cancelling target concurrency groups. Forgejo consumer jobs must keep their equivalent target serialization. These controls are not FIFO queues and do not coordinate another repository deploying to the same target.
deploy_release_fast.sh and notify_deployment.sh use one shared deployment
report. The change boundary is the commit observed on the target environment
before deployment, not the previous push or the most recent release tag.
Staging and production each have their own boundary. Repeating the same commit
reports zero commits; a rollback labels the removed commits; divergent history
or missing provenance reports Commits: unavailable with a reason.
The collector resolves the health response's full git_sha/commit_sha, or the
commit embedded in the standard release ID. CI_CD_PREVIOUS_DEPLOYED_SHA is an
explicit override for callers that obtain the target identity another way.
Shallow checkouts are hydrated from their existing origin with a bounded Git
fetch. An unreachable boundary is never replaced by the latest commit or an
arbitrary date window. Set CI_CD_COMMIT_HISTORY_FETCH=false to disable fetches;
incomplete history then remains visibly unavailable.
The report in _build/deployment/<target>/commits.json retains every commit's
full SHA, subject and complete message, including markers in commit bodies.
It records the target, baseline, candidate, range direction and execution run.
Deployment metadata and notifications consume that same report. Telegram gets
the exact total and an escaped preview capped by a UTF-16 character budget,
with an explicit omitted count. It does not cap the collected commit list.
Keep the report and adjacent timing.env as CI artifacts, including on failure.
The shared GitLab/GitHub workflows do this; Forgejo consumers add the same two
paths to their deployment-job artifact upload.
Actual deployment duration comes from the shared deployer's entry/exit clock, not the job start. Job wait is creation-to-start only when both provider timestamps exist; dependency wait, deployment time and runner queue are not interchangeable. Pipeline age uses the executing run's creation timestamp. Unknown timings remain unknown.
The shared core automatically loads the Forgejo metadata adapter. Supply the
temporary job-scoped FORGEJO_TOKEN: ${{ forgejo.token }} in the workflow/job
environment. No personal token or consumer-local API script is needed.
The adapter validates repository, SHA, run, job and UI URL, disables redirects,
and retains only normalized metadata. The API run ID, UI run number and job ID
are separate identities. Forgejo's current job API does not expose job-created
or runner-queue timing, so those values are not invented. GitLab retains its
native fields; GitHub retains its existing workflow identity.
CI_CD_EXECUTION_RUN_ID, CI_CD_EXECUTION_RUN_URL and
CI_CD_EXECUTION_CREATED_AT describe the workflow executing this job. An
explicit, different CI_PIPELINE_ID remains the staged artifact's source run
during promotion; normalizing a promotion must not change deployed provenance.
Reports validate execution identity when read by a later notification step.
Infrastructure repositories pin the shared Ansible roles; application workflows
pin the delivery scripts. Updating an infrastructure submodule alone does not
update an application's CI scripts. This reporting change uses the existing
remote GIT_MESSAGES contract and does not require a production reprovision.
priv/core/ carries the delivery layer: blue/green deploy, health identity
verification, SemVer tag preflight and publication, acceptance evidence, remote
evidence copy and evaluation, and the ExUnit budget and value audits.
These previously lived in a repository each consumer cloned at a fixed directory name, and referred to each other through that name — so a consumer that cloned it elsewhere got a deploy that ran, reported success, and then failed on a missing sibling after deploying. They now resolve siblings relative to themselves, and a test enforces that.
Consuming them through this package also removes a network fetch from every job. Cloning them per job made CI depend on a forge that rate-limits SSH, which failed builds intermittently.
The executable scripts/resource_preflight.sh guards intensive local commands
against active Linux memory pressure. The executable
scripts/phoenix_test_database_preflight.sh requires linked worktrees to use a
stable MIX_TEST_PARTITION and verifies that partition's test database before
the test command starts. They remain at these public paths so application Mix
aliases can share the checks without copying them.
scripts/intensive_command_lock.sh serializes resource-heavy local commands
with flock on Linux and an atomic-directory fallback on macOS; callers may
choose immediate refusal or bounded operator-visible waiting.
Ansible roles
Reusable infrastructure roles live under priv/ansible/roles so both Mix
dependency consumers and repositories that pin this project as a submodule use
the same provider-neutral implementation. The package includes common,
phoenix_postgres, phoenix_blue_green, web, phoenix_backup, and
system_toolbox_identity.
Deployment lifecycle transport
The phoenix_blue_green role can call an optional consumer-owned executable
configured with deploy_lifecycle_hook. It sends begin-deployment before
shared migrations, then deployment-complete after verified cutover or
deployment-aborted when the deploy fails. A failed begin hook aborts before
migration; terminal hooks are best-effort because cutover may already have
occurred.
The hook receives the event as its first argument and the same value in
CI_CD_DEPLOYMENT_EVENT. Deployment ID, release ID, pipeline ID, environment,
current and target colors, and an expiry epoch are available through
CI_CD_DEPLOYMENT_* variables. Consumers must make quiescence expire no later
than CI_CD_DEPLOYMENT_EXPIRES_AT_EPOCH; terminal events may clear it earlier.
The contract does not require a shared filesystem marker.
Application-aware drain and fencing
Consumers whose application owns durable or long-running work can opt into
deploy_application_lifecycle_enabled. The blue/green role then uses the
configured localhost lifecycle path to drain the old slot, poll until it
reports safe_to_stop, fence its ownership, and wait until the candidate
reports active ownership. It never substitutes the legacy fixed sleep for an
application safety decision.
Authentication is supplied through a root-readable curl config file named by
deploy_application_lifecycle_curl_config, so bearer credentials do not enter
process arguments or deployment output. If the bounded pre-fence wait expires,
the harness restores the old nginx route, asks the old owner to resume claims,
leaves it alive, and fails the deployment visibly. After fencing, rollback is
not guessed: candidate promotion must succeed before the color is committed.
When release distribution is disabled, set deploy_lifecycle_state_file to an
absolute host path readable by the application. The shared deploy service then
writes the deployment ID and bounded expiry before migrations and removes only
its own marker after completion or abort. This host-managed path does not invoke
release RPC and may be used without a consumer hook. deploy_lifecycle_hook
remains available for consumers with another supported transport; both may be
enabled when both effects are intentional.
Environment badging
Non-production deployments should say so. See
docs/environment_badging.md for the logo
sticker, the per-environment favicon and its cache-busting trap, and the
X-Robots-Tag the web role emits when app_is_production is false.
GitLab adapter
GitLab consumers can include the tagged public adapter directly:
include:
- remote: "https://git.agile-u.com/olivierg/ci-cd-harness/raw/tag/v0.4.56/templates/gitlab/permit.yml"
- remote: "https://git.agile-u.com/olivierg/ci-cd-harness/raw/tag/v0.4.56/templates/gitlab/acceptance.yml"
- remote: "https://git.agile-u.com/olivierg/ci-cd-harness/raw/tag/v0.4.56/templates/gitlab/cd.yml"
- remote: "https://git.agile-u.com/olivierg/ci-cd-harness/raw/tag/v0.4.56/templates/gitlab/quality.yml"
The adapter is deliberately thin: it defines GitLab's job graph and variable
mapping, then fetches the same tag and runs the provider-neutral scripts under
priv/core. Pin the include URL and CI_CD_HARNESS_REF to the same release.
Artifact-producing reusable jobs expire their artifacts after two weeks. GitLab
projects must also disable "Keep artifacts from most recent successful jobs" so
that this maximum applies to the latest successful pipeline on each ref.
Status
Extracted from Agile-U's Forgejo delivery scripts, which remain in that repository until it migrates onto this package. Until then the logic exists in two places; close that before a third consumer appears.
System-toolbox infrastructure role
The package also ships the provider-neutral Ansible role at
priv/ansible/roles/system_toolbox_identity. Infrastructure repositories keep
their host inventory and rollout decisions, add this directory to
roles_path, and opt in one host with
system_toolbox_dedicated_user_enabled: true.
The role migrates an already deployed system-toolbox checkout from the deploy
user's service to a locked system-toolbox system identity. It synchronizes
the controller's selected source and dependencies under that identity, installs only
the toolbox's root-owned maintenance helper and sudoers policy, preserves
deploy-user source synchronization through ACLs, grants the identity only
execute traversal on otherwise-private source parents, verifies the new /health
service-manager identity, and restores the legacy service automatically when
that verification fails. Exact privileged system units and deploy-user units
remain consumer-owned allowlists.
Retention policies
The reusable Phoenix roles provide tiered retention for database/file
snapshots and immutable releases. Consumers must set the policy explicitly
after budgeting the entire physical host, including every co-located app,
legacy release tree, logs, caches, and enough working space to create the next
snapshot or unpack the next release. See
docs/retention.md.
GitHub mirror
Forgejo olivierg/ci-cd-harness is the canonical repository. GitHub
ogourment/github-ci-cd-harness is a push mirror so GitHub consumers can call
the reusable workflows under .github/workflows. Develop and release only
from the canonical repository; mirror commits and tags must resolve to the
same objects.
Existing TLS certificates
The web role preserves HTTP-01 challenges at /.well-known/acme-challenge/ before the HTTPS redirect or application proxy. For an existing webroot-based certificate, set certbot_webroot to the path in its renewal configuration (default /var/www/letsencrypt). Other custom paths are not inferred. The role enables certbot.timer and reloads nginx after successful renewal of its certificate, validating configuration first.
After upgrading from 0.4.42 or earlier, review an Ansible web-role dry run and reapply that role on the consumer host. Then run a certificate-specific certbot renewal dry run and verify the served expiration date after actual renewal. Merely updating a Git pin does not change a host.
Artifact-only acceptance gate
AcceptanceHarness 0.11 supplies a Python standard-library gate reader. The evidence
job retains it as tmp/atdd/acceptance_gate.py; the gate job uses that exact reader
when present and otherwise uses the legacy Mix task. Preserve this file when
overriding artifact paths. Upgraded consumers can remove PostgreSQL and Mix
dependency setup from the gate job, but must provide Python 3 and preserve its
evidence dependency and production gating. The evidence-producing job still runs
its selected scenarios once. This optimization does not relax acceptance outcomes.
Watchdog public-failure handling (0.4.46)
After valid direct app readiness, a failing public endpoint records its reason (including curl exit code for transport failures) in the systemd journal and structured failure history. It retains the healthy service instead of restarting or failing over. One healthy probe does not erase the action cooldown. Public incidents require 60 seconds of observed health to close; unconfirmed notification sends remain retryable. Direct app failure still permits cooldown-limited recovery.
Consumers can apply just phoenix_blue_green tasks_from watchdog_script, using
their normal inventory and role defaults. This installs only the managed script;
it does not provision other roles or restart applications. Review --check --diff
before applying. A dry run or template pin alone does not prove live behaviour.
Run the isolated regressions with python3 test/watchdog_recovery_test.py in an
environment containing Jinja2 and jq. All network, service, proxy and notification
operations are faked. Public outage, notification retry, stable recovery, direct
app recovery and standby failover are checked against the actual managed template.
Operations-monitor error-log recovery
The independent operations monitor reports application error logs immediately, even when readiness and PostgreSQL are healthy. It closes that error-log episode only after three consecutive valid zero-error observations spanning at least 60 seconds. A new error or an unknown probe breaks the clean streak while retaining the active episode. The clean streak survives observer invocations; failed event delivery retries the identical payload and idempotency key. Readiness, database and disk failures retain their immediate alerts, and observer transport still requires three failed observations spanning 60 seconds.
Run the isolated monitor regressions with
python3 -m unittest discover -s test -p phoenix_operations_monitor_test.py.
They use synthetic probe processes and a local receipt receiver.
Security-only hotfixes (0.4.54)
The CI caller must grant actions: read alongside its existing permissions,
because the reusable verifier reads run provenance. Main-only delivery remains the default. A consumer may explicitly set
security-hotfix-policy: .github/security-hotfixes.json in its staging caller
and its main-only promotion caller. A dispatched security/ branch is eligible
only when a policy fetched from an immutable current-main revision approves its
exact source SHA, branch, production base SHA, binary diff SHA-256 and expiry.
The approved base must be an ancestor and identify a successful production-source
run; application version must stay within a PATCH line and migrations are refused.
Review the entire exact diff before adding an approval. The policy is a release
authorization record, not an advisory scanner or automatic proof of security scope.
Schema1 policy: {"schema":1,"approvals":[{"purpose":"security-only","sha":"<40hex>", "base_sha":"<40hex>","branch":"security/fix","diff_sha256":"<64hex>", "expires_at":"<UTC timestamp>","production_base":{"version":"1.2.3", "release_id":"app-1.2.3-42","pipeline_id":"42"}}]}.
The same successful CI run must produce the original release and a genuine staging-verification receipt. Promotion rechecks the approval and all existing artifact/run/attempt/identity gates, and refuses a production base whose healthy version, release ID or source pipeline has moved. Existing deployment concurrency, health verification, rollback and release-tag protections remain in force. Remove expired or superseded approvals. A new SHA always needs a new reviewed approval.
Bounded optional GitLab history
Source priv/forges/gitlab/history_http.sh and call history_http_init once before
historical reporting. history_http_get PATH buffers a successful response;
HTTP errors, redirects and timeouts return nonzero without partial/error output.
history_http_run COMMAND... applies the same deadline to local collection.
The exported absolute deadline survives command substitutions and nested shells.
HISTORY_HTTP_BUDGET_SECONDS defaults to 120 and
HISTORY_HTTP_REQUEST_SECONDS to 15 (both 1..300). Connect time is at most 5s;
forced termination has a 2s grace. Once the shared budget is exhausted, callers
must stop collecting and retain unavailable/NA values, preserving current quality
job results. Rendering existing results is outside the historical collection budget.
Use trusted HISTORY_HTTP_API_ROOT, HISTORY_HTTP_AUTH_HEADER (PRIVATE-TOKEN or
JOB-TOKEN) and HISTORY_HTTP_TOKEN server configuration. Headers pass through
stdin rather than argv or disk. Redirects never receive credentials. Without an
API root the helper bounds glab api instead. No credentials are provisioned.