NannyStateGPT

A big part of my job is infosec. This means I occasionally need to run penetration tests against client infrastructure.

I do this in a controlled fashion that won’t hurt production or otherwise harm my clients.

Last week, when the PHP2Shell thing went live, I wanted to be sure anything patched could not be hit with the attack that was published Wed morning.

For the sake of speed, I asked one of the agentic tools I have access to help me set up a quick test. It refused.

I explained we could run the test locally against a Docker file running on my machine locally, and there was no risk of actually damaging anything.

It still refused.

I tried another tool, and it refused.

The files to replicate the attack were available for the entire world to view, so I just downloaded the files, and I had everything I needed to replicate the vulnerability.

In the end, its refusal resulted in slightly inconveniencing me and wasting my time.

Now this is where I am torn.

I can see the argument that you wouldn’t want the LLMs to help a novice execute an attack.

But contrary to that, “Script kiddies” have been running scripts far beyond their understanding to cause damage for a long, long time.

So it's not actually preventing nefarious behavior.

What if there was a non-technical business owner trying to assess the blast radius of a vulnerability on a WordPress install that they had some kid set up 5 years ago and has since stopped responding to emails?

Should they be deprived of the ability to actually test to see if the vulnerability exists and is fixed?

I understand both sides of the argument.

As a paying customer and a capable pentester that can easily trigger the vulnerability on an unpatched server, I will say that by nerfing their product and creating a NannyStateGPT, these tech companies are making their AI tools less valuable to me.

I’m not going to pay top dollar for a tool that is going to argue with me rather than assist me in my duties.

Btw, if no one has coined the term “NannyStateGPT”, I am going to claim credit for that one.