SSRF in GCP Vertex AI via Malicious Custom Container URI Enables Metadata Exfiltration
Overview
Researchers from Orca Security disclosed a Server-Side Request Forgery (SSRF) vulnerability in Google Cloud Platform's Vertex AI service, as part of their 'AIPACA' research. The vulnerability allowed an attacker with permissions to create a custom training job to compromise the underlying infrastructure and access sensitive metadata. The attack vector involved specifying a malicious URI for a custom container image when defining a training pipeline. Vertex AI's backend service would fetch and process this container without sufficiently validating that the URI pointed to a legitimate container registry. By providing a URL pointing to an internal GCP metadata service endpoint, an attacker could trick the Vertex AI service into making a request to that endpoint on their behalf. The response, containing sensitive information like service account tokens, would then be logged or returned in an error message accessible to the attacker. A successful exploit would grant the attacker the privileges of the Vertex AI service account, potentially allowing them to access GCP storage buckets, other AI models, and sensitive training data within the victim's project. The findings emphasized the need for cloud providers to implement strict network egress controls and URI validation for services that ingest user-provided resources.
Affected Systems
Testing Guide
1. **Review IAM Policies:** Audit GCP IAM policies for roles like `roles/aiplatform.user`. Ensure that only trusted principals can create custom training jobs. 2. **Check Historical Logs:** In Google Cloud Logging, search for `protoPayload.methodName="google.cloud.aiplatform.v1.JobService.CreateCustomJob"`. Review past job creations for any that used unusual `containerSpec.imageUri` values or failed in a way that suggests an SSRF attempt. 3. **Attempt Controlled Test (with caution and permission):** Create a custom job and set the container URI to a service you control (e.g., a Burp Collaborator or Interactsh URL) to see if the Vertex AI backend makes a request to it. This confirms if the execution environment can make arbitrary outbound requests.
Mitigation Steps
1. **Vendor Patch:** This vulnerability was resolved by Google Cloud. No direct user action is required for the specific flaw. 2. **Principle of Least Privilege:** Grant users the most restrictive IAM roles necessary. Avoid giving developers broad `vertexai.admin` permissions. Use custom roles that restrict the ability to create jobs with custom containers. 3. **VPC Service Controls:** Implement VPC Service Controls to create a service perimeter around your Vertex AI resources, preventing data exfiltration by restricting communication with external or unauthorized services. 4. **Audit Logging:** Regularly review Cloud Audit Logs for suspicious activity related to Vertex AI job creation, especially jobs that fail with unusual network-related error messages.
Patch Details
Google Cloud Platform patched the vulnerability in their backend infrastructure after being notified by the researchers. The fix involves stricter validation of container image URIs.