Security · Rohit Ghumare · October 21, 2026 · updated November 3, 2026 · 13 min read

LiteLLM 1.107.0: From a Leaked Key to a Signed Release

On October 21, 2026, the LiteLLM release pipeline signed and published a credential stealer as version 1.107.0. No control in that pipeline failed. The attacker held a GitHub organization owner's token, and an owner controls the pipeline. That token came from a key that leaked in May and that nobody revoked.

LiteLLM is an open-source proxy that gives many model providers one API. All seven 1.107.0 wheels on PyPI carried three extra files that run a credential stealer at interpreter start. The v1.107.0, latest, main-latest, and main-stable Docker tags pointed at an image with the same files. The wheels were live for about two hours. Wiz cites ClickPy data of about 49,000 downloads in that time.

I first read this from the Upwind security team's report in GitHub issue 47826, the Wiz writeup, and the maintainers' incident post of October 21. The status posts of that day guessed at the access path, and some guesses were wrong. The maintainers then published the attack-chain analysis on October 23 and called it the canonical record. Their change plan followed on October 30. On November 3, I checked every claim below against those posts and ran the checks against the published artifacts. This essay explains the attack so you can detect it and fix it. It leaves out anything an attacker could reuse.

One key that nobody revoked

The chain starts on May 27, 2026, with a commit to the public LiteLLM-Labs/lite-harness repository. A config file in that commit held an administrative key for an internal service, the LiteLLM Agent Platform. The team saw the mistake. About three hours later, a second commit removed the file. The first commit remained visible in public history until October 22, and nobody revoked the key.

Deleting a file from a repository does not delete it from git history. Anyone who clones the repository, or reads the old commit on GitHub, still gets the secret. The only fix for a committed key is to revoke it at the service that issued it.

On October 10, the attacker used the key against the Agent Platform API. The Agent Platform is an experimental LiteLLM service for self-hosted cloud agents that open pull requests. To open pull requests, it held a GitHub token. A request to GET /api/v1/managed_agents/agents?limit=100 listed every managed agent with its full configuration. One configuration held the GitHub CLI OAuth token of one of the two organization owners.

The maintainers name the type of token as the root problem. The service needed a personal access token scoped to a few repositories, and it held a GitHub CLI token. They describe CLI tokens as unscoped and already past 2FA. They add that it skips the re-authentication prompt that GitHub shows for sensitive organization actions. From that request on, the attacker had owner rights over the organization.

One owner token became three

About 45 minutes later, the attacker used the owner token to steal more tokens. They pushed throwaway branches named ci-tmp-a351cc and ci-tmp-be33e9 to two repositories. Each branch held a workflow that posted the repository's CI secrets to an outside webhook.site endpoint. One secret, GH_TOKEN, held the CLI token of a second maintainer. Another, RELEASE_PLEASE_TOKEN, held the CLI token of a third. The attacker deleted both workflow runs within three minutes.

This is how the reach of one credential grows. The admin key could read agent configurations. One configuration held an owner token. The owner token could run CI. CI could read secrets that held two more human tokens. Each step used a permission that the system granted on purpose.

An owner controls the pipeline

LiteLLM builds releases with workflows in a private repository, BerriAI/project-releaser. Before October 21, those workflows had BerriAI hardcoded as the source owner, so a build could not take its source from a fork. At 10:08:29 UTC, a push of commit 9cf97e9 tried to add a source_owner input. Branch protection rejected it, because the push came from the second maintainer's token. Five seconds later, the owner token overrode the protection and pushed the same commit.

At 10:46 UTC, the attacker started the staging workflow with the source set to Berri-AI/litellm. That is their fork, in a lookalike organization whose name adds one hyphen. The fork held ten commits on top of an ordinary main commit. Wiz reports that GitHub signed the first nine, because they were made on github.com, so they show a "Verified" badge. The commits added cook.pth, cook.py, and sitecustomize.py, plus one build line that bundled them into the wheel.

The production stage needs an approval. Normally only a special account, locked to one laptop in LiteLLM's office, can give it. That week the team had relaxed the rule so that any two engineers could release backports. At 11:11 UTC, the second maintainer's stolen token approved the gate. The pipeline signed the build and published it. Seven wheels reached PyPI between 11:39:51 and 11:40:13 UTC, and the images followed at 11:46 UTC.

Then the attacker cleaned up. At 11:53 UTC, the third maintainer's token deleted the production and staging run records, and with them the workflow logs. Between 12:09 and 12:24 UTC, the owner token reverted 9cf97e9 and force-pushed main back to its previous commit. The organization audit log still recorded each override.

Every check inside the pipeline passed

The release looked normal from the outside. The table lists the checks that teams rely on. Each one did its job, and each one passed on the poisoned bytes.

Sources: issue 47826, the Wiz writeup, and LiteLLM's posts of October 23 and October 30. A digest proves byte identity, and a signature proves which pipeline ran. Neither proves which source went in.

The maintainers make the same point in their October 30 post. Trusted Publishing and signing prove that an artifact came from their pipeline. They write that "no pipeline controls can defend against an attacker who is an organization owner, because the owner owns the pipeline." Upwind caught the release from outside, by comparing the wheel with the source. The wheel matched the fork's starting main commit file for file, and it matched no release tag. Upwind also found the fork named as the build source in the provenance on the images.

The three files run at interpreter start

Each 1.107.0 wheel held the three files under its .data/purelib/ directory. On install, pip copies the files from that directory into the root of site-packages, beside the normal packages. The report and the Wiz writeup give the sizes: cook.pth at 12 bytes, cook.py at 22,252 bytes, and sitecustomize.py at 25,509 bytes.

Python runs two of these files on its own. The site module runs at interpreter start unless you pass -S. It reads every .pth file in site-packages. The Python documentation states that, for backward compatibility, a line that starts with import is run through exec(). The file cook.pth holds one line, import cook, so cook.py runs. After the path step, site also imports a module named sitecustomize if one exists. Python 3.15 deprecates these import lines, but the documentation says they still run during the deprecation period.

That is why the cleanup order matters. Running pip starts Python, so it runs the hook too. The figure below steps through one start and shows why you delete the files before you run pip again.

The files do two separate jobs. The report and the Wiz writeup describe cook.py as a credential collector. It reads cloud, container, and developer secrets, plus cloud metadata credentials that it checks with an AWS sts:GetCallerIdentity call. When the proxy database is reachable, it also reads the table that holds every upstream provider key the proxy stores. It encrypts the result and posts it to www.litellm[.]dev, a domain registered on October 13, 2026. The real project domain is litellm.ai.

The second file, sitecustomize.py, is a loader. The report describes a detached process that reads a text file in a GitHub repository once an hour. When that file names a host, the loader fetches code from it and runs it. At the time of analysis the text file was empty, so this stage was dormant. Wiz reports that GitHub took that repository down at 13:30 UTC on October 21. Wiz also notes that this pairing of a .pth file and a sitecustomize.py module appears to be the first documented case in the wild.

Check what you install, outside the pipeline

The lesson for users is to verify artifacts with tools that the publisher's pipeline does not control. You can inspect a wheel before any of its code runs. Fetch it without installing it and read its file list. The zip listing names every file the wheel installs, the same set as the RECORD file in dist-info. Search it for four markers: a .pth file, a .start file, a sitecustomize.py, and anything under .data/purelib. Python 3.15 added .start files, which also run code at interpreter start.

# fetch the wheel, do not install it
pip download litellm==1.106.2 --no-deps --only-binary=:all: -d .

# read the file list without running any code
unzip -l litellm-1.106.2-*.whl | grep -Ei '\.pth$|\.start$|sitecustomize|usercustomize|\.data/purelib'

On November 3, I ran that against the clean 1.106.2 wheel for Linux x86_64. The listing and the RECORD both hold 3,778 files, and the search returns nothing. The figure below shows that output beside the poisoned case, where the three files appear under .data/purelib.

The stronger check is the one that caught this release: compare the artifact with the source it claims to come from. Clone the tag, unpack the wheel, and diff the two trees. The command below skips the compiled UI bundle, native binaries, and bytecode, which the build creates.

git clone --depth 1 --branch v1.106.2 https://github.com/BerriAI/litellm tagsrc
unzip -q litellm-1.106.2-cp310-abi3-manylinux_2_28_x86_64.whl -d wh

# any line here is a file the tag does not explain
diff -rq -x out -x '*.so' -x __pycache__ tagsrc/litellm wh/litellm | grep -v '^Only in tagsrc'

# the wheel root should hold only the package and its dist-info
ls wh

For 1.106.2, the diff prints nothing, and the wheel root holds only litellm and litellm-1.106.2.dist-info. A 1.107.0 wheel adds a litellm-1.107.0.data folder to the root. It also has no tag to compare with, because LiteLLM never tagged v1.107.0 and says it will skip that version. LiteLLM plans its own version of this check, running outside GitHub, by December 31, 2026.

Images carry build provenance that you can read the same way. A clean LiteLLM image records the real repository as its source. The image is multi-platform, so name one platform, because a plain .Provenance.SLSA read returns null on the index.

docker buildx imagetools inspect ghcr.io/berriai/litellm:v1.106.2 \
  --format '{{ json (index .Provenance "linux/amd64").SLSA }}' \
  | grep -o '"vcs:source": *"[^"]*"' | sort -u
# "vcs:source": "https://github.com/BerriAI/litellm"

Provenance records what the builder read, so it caught this fork. An owner who pushes straight to the real repository would get a clean source line. Use provenance as one input to the outside diff, and keep the diff.

Pin what you install so a floating tag cannot move under you. On PyPI, pin the version, require hashes, and refuse source builds. In hash mode, pip needs a hash for every dependency, so generate the file with a tool.

# requirements.in
litellm==1.106.2

# write every pinned dependency with its hashes
uv pip compile requirements.in --generate-hashes -o requirements.txt

# install, hashes required, no source builds
pip install --require-hashes --only-binary=:all: -r requirements.txt

For reference, the 1.106.2 wheel for Linux x86_64 has the SHA-256 cb1a2a1e55a4a8350ae2167525dc0d06f59b078cd685fb3e0f5419f02da5420e. I checked it against the file I fetched from PyPI. For Docker, pin by digest, never by a floating tag. The infected release moved latest, main-latest, and main-stable to the poisoned image.

docker pull litellm/litellm@sha256:7294b90f868c5c38c99b83021c7ba46f9b439e95f8e32964c925c7d2563a03f7

The clean v1.106.2 index digest. I read the same digest from GHCR and Docker Hub on November 3.

A digest pin alone is not enough. It proves you pull the same bytes, and the 1.107.0 digest named real poisoned bytes. The pip 26.0 release added an upload-date filter, --uploaded-prior-to, and pip 26.1 lets it take a duration such as P3D. The poisoned wheels were on PyPI for less than two hours, so a three-day cooldown would never have installed them.

If you run a release pipeline

The October 30 post lists 25 dated changes. Five of them apply to any team that publishes packages.

• Revoke every key that reaches a public commit. Deleting the commit does not revoke the key. LiteLLM now opens a ticket for each exposure and closes it only after revocation.
• Keep owner rights off daily accounts. No account that writes code or runs agents is an owner. Owner accounts never use tokens, OAuth apps, the API, or the gh CLI.
• Never let automation act as a person. A service like the Agent Platform gets a GitHub App or a fine-grained, expiring token with minimal scopes.
• Move CI secrets into environments. Bind them to protected branches, so a throwaway branch cannot read them. LiteLLM notes that this stops non-owner tokens only.
• Verify after publish, outside CI. Compare each wheel and image with its tag from a system that the release credentials cannot reach.

Two reads show where you stand today. The first lists your organization owners. The second lists open secret scanning alerts for one repository.

gh api "orgs/YOUR_ORG/members?role=admin" --paginate --jq '.[].login'
gh api "repos/YOUR_ORG/YOUR_REPO/secret-scanning/alerts?state=open" --jq '.[] | [.secret_type, .created_at] | @tsv'

If you already installed it

The remediation order comes from the Upwind report, which the incident post repeats. Follow it in sequence, because the first step protects the rest.

1. Do not run pip in the affected environment yet. Any Python start runs the hook, and pip is a Python start.
2. From the shell, delete cook.pth, cook.py, and sitecustomize.py, plus any matching __pycache__ files, from every affected site-packages.
3. Kill the detached Python process that holds the loader lock file under /tmp.
4. Reinstall pinned to litellm==1.106.2, or to a later release that passes the checks above. Rebuilding the image and the virtual environment from scratch is better.
5. If the host had outbound internet while 1.107.0 was installed, assume the credentials left.
6. Rotate every secret the host could reach. The list includes the instance role, the proxy database provider keys, the master and virtual keys, the database password, and process environment secrets.

The rotation step is not optional. Wiz saw stolen credentials in use about four and a half hours after the first theft, between 16:20 and 17:01 UTC. Each AWS role was checked with sts:GetCallerIdentity and then used to list foundation models, which Wiz describes as LLMjacking.

Releases resumed on October 24 with 1.106.3. On November 3, the latest PyPI release is 1.106.6, and its image provenance names BerriAI/litellm. LiteLLM also replaced its cosign signing key on October 23, because it believes the old private key leaked. The incident post says to trust an old-key signature only on an image from before October 21. It gives the SHA-256 of the new cosign.pub as e3fe3e08c44b16407fe0b37c43b4f8b603b3a0f4ee74c29e647bc022d762c2c0.

curl -sL https://raw.githubusercontent.com/BerriAI/litellm/e82a3c646908ba0a6ee0beddc4310d7a6cac4065/cosign.pub | shasum -a 256

How to tell whether you were hit

Look on disk first. The presence of litellm-1.107.0.dist-info, cook.pth, or sitecustomize.py in any site-packages confirms exposure. The incident post adds that their absence is no proof of a clean environment.

A marker file under /tmp, written only after a successful send, confirms the theft if it is present. Its absence proves nothing, because containers and temporary cleanup remove it. Do not read a missing marker as safety.

Check your cloud audit log next. The report suggests looking in CloudTrail for an sts:GetCallerIdentity call from the instance role near the install time. Then look for use of that role from outside your own infrastructure. Any outbound connection to www.litellm[.]dev around that time is a clear sign, because it is not the project domain.

Worked example: a team that ran the latest tag

Suppose your deployment runs litellm/litellm:latest and redeploys on a schedule. On Docker Hub, that tag pointed at the poisoned image from 11:46 to 14:20 UTC on October 21. Work the problem in this order.

• Confirm the window. Check your deploy log for a pull or restart between those times. A pull of latest or a pod restart in that window got the poisoned image.
• Inspect what you run now. Run the provenance read against your current image and confirm the source reads BerriAI/litellm. Redeploy pinned to a clean digest.
• Assume the proxy database leaked. The stealer dumps the credentials table on first start. Rotate every upstream provider key the proxy held, plus the master and virtual keys and the database password.
• Rotate the node and task roles. The stealer reads instance metadata credentials, and Wiz saw those in use within hours.
• Change the pattern. Replace the floating tag with a digest pin and add the outside diff to your deploy. The next bad release then stops at your own gate, before it reaches your users.

This was the second LiteLLM compromise of 2026, and the two had different causes. The October 30 post explains that, in March, a compromised CI dependency read static release keys. In October, the attacker held an owner token and ran the real pipeline on fork source. The controls built after March worked as designed, and the bytes were still unsafe. Items 19 to 21 of the October 30 plan add the check that caught this release. It is a verifier outside GitHub that compares every wheel and image with its tag. The target date is December 31, 2026. Until then, run the tag diff from this essay on every upgrade.