LLM API keys get stolen through GitHub commits, misconfigured clouds, and exposed pipelines. Learn why billing alerts fail and how a gateway fixes llm api key management.
An xAI employee committed a private API key to GitHub. That key could have allowed anyone to query private xAI large language models for the past two months, including models that appeared to have been custom made for working with internal data from SpaceX, Tesla, and Twitter/X. GitGuardian found the key had access to at least 60 fine-tuned and private LLMs.
GitGuardian alerted the xAI employee about the exposed API key on March 2. As of April 30, when they escalated directly to xAI's security team, the key was still valid and usable.
This is what broken llm api key management looks like. The detection tooling worked. The organizational response did not. And the standard playbook (store keys in .env, set billing alerts, rotate on breach) would not have caught it either, because billing alerts are a lagging indicator. They fire after the damage is done.
Before LLMs, criminals used to look for AWS or GCP access keys, which could give them access to compute resources, storage, and sensitive company information. LLMjacking, hijacking access to your LLM through stolen or exposed credentials, is a different spin on that familiar attack with steeper consequences.
LLM usage is metered and expensive, so abuse translates directly into financial impact. That is the obvious risk. The deeper one: if a key is tied to custom models, internal prompts, or sensitive data pipelines, the risk is no longer solely about compute resources. A compromised LLM credential can give an attacker access to far more than raw data.
The xAI case illustrates this precisely. The leaked key did not just grant compute. It granted access to models that appeared to have been built on proprietary data from multiple Musk-affiliated companies.
API keys end up in repositories constantly. GitGuardian's systems constantly scan GitHub and other code repositories for exposed API keys, and fire off automated alerts to affected users. The xAI key was one such case: committed by an employee, detected by scanning, but never revoked.
Keys also surface in infrastructure that was designed for convenience, not security. They are frequently stored in environment variables, configuration files, container images, CI/CD pipelines, or logging systems that were never meant to be publicly accessible. Overly permissive S3 buckets, an exposed Kubernetes dashboard, or a poorly secured Git repository can hand attackers everything they need without requiring them to exploit a vulnerability directly.
The infrastructure around LLM deployments creates its own exposure paths. The same environment variables and configuration files that make keys easy to use in development make them easy to leak in production. Container images bake in secrets. CI/CD pipelines pass them as variables that show up in build logs.
Despite increasingly sophisticated AI-assisted attacks, attackers still rely on convincing someone to just hand over their credentials. Phishing campaigns remain a highly effective way to gain access to LLM resources, because they target people rather than technology.
The financial hit from a stolen LLM key is usually the first place where you notice that something is wrong. If you see the impact in your bills, the attack has already done significant damage.
Even usage and billing limits provide incomplete protection. Any downstream agentic processes or automated workflows dependent on the LLM will also be affected. This could have an even bigger financial impact, especially if those processes are tied to critical business functions.
Two follow-on attacks compound the problem:
Data exfiltration through custom models. If an attacker gains access to a model trained on internal documents, they can gain knowledge about your organization that was never intended to be public. This information could be used to extend their network foothold or sold on the dark web.
Training pipeline poisoning. An attacker can gradually influence the model over time to provide misleading or biased responses to employees. While this is a more time-consuming exercise, it allows them to nudge decisions, surface false information, or steer internal processes. These changes are usually very subtle and difficult to detect.
The timeline tells the story.
An xAI employee leaked a private key on GitHub that for the past two months could have allowed anyone to query private xAI LLMs. GitGuardian alerted the xAI employee about the exposed API key nearly two months ago, on March 2. But as of April 30, when GitGuardian directly alerted xAI's security team to the exposure, the key was still valid and usable.
The key had access to at least 60 fine-tuned and private LLMs, including unreleased models of Grok and models that appeared to have been custom made for working with internal data from SpaceX, Tesla, and Twitter/X.
With that kind of access, an attacker could use it for prompt injection, to tweak the model to serve their purposes, or try to implant code into the supply chain.
Security researchers summarized the root cause: this kind of long-lived credential exposure highlights weak key management and insufficient internal monitoring, raising questions about safeguards around developer access and broader operational security.
The detection tooling did its job. The key was found on day one. What failed was the response: no system revoked the key, no monitoring flagged anomalous usage, no policy enforced automatic rotation.
An LLM gateway centralizes fragmented API management and enforces consistent security policies across an entire organization. For llm api key management specifically, that centralization solves three problems the .env-and-billing-alert approach cannot.
Provider keys leave application code entirely. When a gateway holds the provider credentials, individual services authenticate to the gateway with scoped tokens. The provider key never appears in a configuration file, a container image, or a Git commit. An xAI-style leak becomes structurally impossible: even if a developer commits their gateway token, the provider key stays vaulted.
Monitoring moves to the request layer. Billing alerts check usage periodically. A gateway sees every request in real time. A spike in volume, requests to models a service should not be calling, queries from an unexpected source: these patterns surface while they are happening, not weeks later in an invoice.
Secrets detection integrates into the request path. GitGuardian demonstrated building a secure LLM gateway using GitGuardian and AWS Lambda, integrating secrets detection into every request. This moves detection from the repository layer into the runtime layer, catching credentials that scanning alone would miss.
In the xAI scenario, a gateway would have changed three things. The leaked credential would have been a gateway token, and the provider key would have stayed vaulted. Revoking it would have been a single API call. And request-level monitoring would have flagged anomalous queries against private models from an unauthorized source immediately.
Based on how keys actually get stolen, here is what a solid llm api key management setup includes.
A gateway as the single point of egress. Every LLM call routes through it. No service holds a provider key directly. When you need to rotate, you rotate in one place.
Per-service scoped credentials. Each service authenticates to the gateway with the minimum access it needs. A summarization service has no business querying your fine-tuned models. A coding assistant should not reach your customer data pipeline. Least privilege applies here the same way it applies everywhere else.
Request-level anomaly detection. Monitoring that watches request volume, model access patterns, and source identity in real time. When a credential that normally generates 50 requests per hour suddenly generates 5,000, the system should act before the bill arrives.
Automated rotation on a fixed cadence. The xAI key was valid for nearly two months because nobody revoked it. Automated rotation on a schedule, combined with immediate revocation on alert, removes the human bottleneck that let that key sit exposed.
No keys in CI/CD artifacts, container images, or build logs. API keys stored in environment variables, configuration files, container images, CI/CD pipelines, or logging systems are the exposure surface that gateway vaulting eliminates. If your build process can see the provider key, your build logs can leak it.
For teams building out their AI security posture, the gateway is the layer that makes every other control enforceable. Policies in documentation are suggestions. Policies in the request path are constraints.
The xAI incident had detection on day one. What it lacked was infrastructure that could act on that detection automatically. A gateway with request-level monitoring and access controls provides that infrastructure.
shim lets applications authenticate with shim API keys while provider keys are supplied per request or stored for the supported OpenAI, Anthropic, and Google providers. Production stored secrets use a managed secret backend.