Synopsis
Tenable Research has identified and responsibly disclosed a critical vulnerability in Google Vertex AI Workbench. This flaw enabled a cross-tenant Full Account Takeover by exfiltrating the managed End User Credentials (EUC) of any GCP user with minimal interaction.
The vulnerability exploits the Single User Mode with managed End User Credentials (EUC) in Vertex AI Workbench. This feature allows a notebook instance to run with the actual identity of a single specified GCP user by injecting the user’s credentials into the Instance Metadata. Naturally, the security of these End User Credentials (EUC) instances is tighter, blocking SSH access and any custom containers or scripts to keep the user's credentials safe. For a user's credentials to appear in the Instance Metadata, the user must click "Open JupyterLab" in the Google Cloud Console. This triggers a CheckAuthorization request that applies their OAuth token to the instance if they authorized Vertex AI Workbench OAuth.
The attack leverages a built-in startup script named runtime-config-post-result, which is part of the "Click to Deploy" (c2d) suite used to initialize these instances. This script is designed to report the instance startup result by sending an authenticated status update request to the URL specified in the instance metadata via the status-config-url attribute. The update request's authentication is based on the identity associated with the instance.
An attacker can create a Workbench instance in their own project, but sets a victim from any tenant as the "Single User." They can then update the status-config-url metadata attribute to point to an attacker-controlled server.
The runtime-config-post-result script typically runs only during the first boot (when credentials haven't been loaded yet). However, upgrading Workbench forces these scripts to re-run while maintaining the existing IMDS state. Therefore, if the attacker can get a victim to press the “Open JupyterLab” button, they can then trigger an upgrade, which will execute the script and send the victim’s credentials to the attacker-controlled server.
Proof of Concept:
Attacker's Project:
- Give the victim’s principal IAM permissions in the attacker’s tenant
- Go to https://console.cloud.google.com/vertex-ai/workbench/instances and create a new Workbench instance
- Click on 'Advanced Options'
- Under 'Environment', choose 'JupyterLab 4.x' and 'Use a previous version', and select an older version.
- Click '+ Add metadata' and add the following attribute: status-config-url, with the value of the attacker's HTTP server
- Under 'IAM and security', choose 'Single user', enter the victim's email, and check 'Enable managed end user credentials'
- Create the instance and wait for it to finish provisioning (this may take a few minutes)
- Once the machine is up, you will see an HTTP request in the attacker's server with "No credentialed accounts" in the Authorization header
- Share either the instance page with the victim and have them click 'Open JupyterLab', or share the JupyterLab URL directly and wait for the pop-up to instruct them to click it
- Once the victim clicks the button, click 'Upgrade' to upgrade the machine, and wait for the victim to close the tab and for the upgrade to complete (this also may take a few minutes)
- The victim's access token will be exfiltrated to the attacker's server
Solution
The fix prevents the status-config-url and status-variable-path keys from being set in the control Plane and also removes the vulnerable start-up script from new Workbench images.
Disclosure Timeline
All information within TRA advisories is provided “as is”, without warranty of any kind, including the implied warranties of merchantability and fitness for a particular purpose, and with no guarantee of completeness, accuracy, or timeliness. Individuals and organizations are responsible for assessing the impact of any actual or potential security vulnerability.
Tenable takes product security very seriously. If you believe you have found a vulnerability in one of our products, we ask that you please work with us to quickly resolve it in order to protect customers. Tenable believes in responding quickly to such reports, maintaining communication with researchers, and providing a solution in short order.
For more details on submitting vulnerability information, please see our Vulnerability Reporting Guidelines page.
If you have questions or corrections about this advisory, please email [email protected]