MagicShel

One of the more frustrating quirks of large language models is something I’ve started calling the Advice–Action Gap. It’s the space between what an AI will say you should do when performing a task and what it will actually do when you have it execute the same task.

If you ask a model to explain the “best practice” for almost anything — writing a secure SQL query, structuring clean React components, writing in noir style — it will correctly explain the principles: avoid SQL injection, keep components small and focused, show rather than tell. It will be a textbook answer.

But if you instead give it a practical request — “Write a login query,” “Build a small form in React,” “Draft a short noir story” — it often ignores its own advice to a frustrating degree. It does this because it follows one stochastic pattern when giving advice and a completely different one when executing the task. This is the Advice–Action Gap in action.

Closing the Gap

The good news is you can close the gap. Here are three ways you can improve the odds of AI following its own best advice based on my experience. You can use one of them or combine them all.

1. Recite

Ask the AI to summarize the best practices before having it perform the task.
This loads the rules into its “working memory” before execution and makes it less likely to wander.

2. Embed

Include the practices you want to emphasize in your request:
Write a Python script to do X, following these practices: [list here].
This gives the model less room to drift away from the standards you’ve set.

3. Verify

After you get the response, ask the model to review its own output against best practices and point out where it followed or broke them.
This won’t catch everything, but it’s a handy way to surface obvious misses before you review it yourself.

None of these tricks make the Advice–Action Gap disappear entirely, but they can help you and the LLM work together to arrive at the best result.

MagicShel

Preface: My area of interest and expertise is largely in text generation. I may touch on image, voice, and music generation here and there but my words are intended to apply primarily to text models and the ability of text models to accept input from other modes. I also try to speak colloquially. Despite my understanding that AI doesn't think and doesn't know anything, those words might slip in from time to time because they are such familiar metaphors.

AI is a fast-moving technology and there is always something new going on either in the industry or the response to it. I got involved in AI through AI Dungeon some years ago where I created and maintained some populate user scripts, and I've been a fan and involved in developing AI tools and interfaces ever since.

However, despite being an enthusiast, I consider myself a realist. You aren't going to hear me tell you AI is sentient, or that a golden age of prosperity and freedom is around the corner while AI does all the work for us. AI is none of that. It is a brilliant, but problematic tool that is hyped with lies because they are easier to sell than the truth. Today, I want to tackle those problems head-on.

Some Possible Problems With AI

Is AI bad for the environment?

Writing a script in seconds rather than minutes or hours might take a lot of energy for those few seconds, but how does it compare to the cost of a developer working at a desk for an hour? You have lighting, heating/cooling, food, transportation. I believe some uses are an environmental net-positive.

On the other hand, generating non-functioning code over and over again for multiple hours each day consumes vastly more energy than a developer doing the same on his own. Running entire departments making thousands of requests per hour is extremely energy-intensive and probably for very little gain.

Does AI lead to unemployment?

It shouldn't. AI cannot effectively replace a human worker. But that's not how it's being sold. That's often not how it is being implemented. Companies are investing heavily in AI because they don't believe their employees are offering enough value for their cost and some of them are jumping before even testing that theory. Many of them have already started regretting it.

Even if AI could hypothetically perform 80% of the work of a human specialist, that 20% failure rate is not going to make customers very happy, and if one isn't careful could even open the door for potential legal action. Even if the failure rate was as low as 1%, that is going to disproportionately impact your most complex accounts.

None of that changes the fact that people are losing jobs because of corporate shortsightedness and, frankly, resentment of their own employees.

Is AI stealing?

I am not a legal expert by any stretch of the imagination. But our Intellectual Property laws allow for “transformative” uses. Taking a thing, such as a novel, and turning it into a completely different thing such as a fractional part of a text generator is completely transformative by any understanding of the word.

That being said, legal cases are wending their way through the courts and the results may in fact surprise me. Intellectual Property laws are extremely complex and a surface understanding of them might lead to the wrong conclusion. So all I can say is, I don't think so, but we'll see.

In any event, IP is a civil matter, not criminal. AI is not stealing, as such, but it's possible the model owners might be found liable for damages and if they are, I hope the creators are adequately compensated.

Is AI output shallow or lacking in quality?

Beauty is in the eye of the beholder. You can ask AI to draw you a picture or write you a love poem or a term paper, and the result might impress you. It may even move you. I've had AI put a string of words together that brought me to tears —despite being assembled by an algorithm. But generate enough text or enough pictures, and you'll begin to see that a given model has a limited palette to draw from. Phrases begin to repeat. Paragraphs take on the same cadences over and over. The emotional resonance will be muted or over the top. And ultimately it will sound uninspired and flat, like trying to connect with someone who is trying to keep you at arms reach (or is trying to grope you while you just want a simple answer to a question). You are far better off writing something yourself, having AI review it and offer pointers, and then incorporate the suggestions you agree with yourself rather than allowing AI to rewrite you until your unique voice is lost.

I must confess, this all leaves me a bit perplexed as to how some folks become convinced that they are in love with an AI. How can you fall in love with something without noticing how shallow and empty it's responses to you are? This is a question I will leave to others, because the only answers that come to mind are uncharitable.

Does AI Lie?

Yes! But also no, because to lie is an act of intention. AI is not designed to be a source of factual information. In fact, often enough these days, truth is a political question rather than factual.

AI looks at the text in front of it and, given pretty much all of the text generated throughout human experience, completes that text in a way that actual human people might. But people are flawed. We are sometimes wrong, or misleading, or choose a bad metaphor. Also because AI doesn't know anything, I also can't know when it doesn't know.

Everything AI writes is a story. And like most stories, it contains a lot of real facts, but it isn't shy about embellishing or inventing facts to make the story better. This being the case, you can never completely trust the output of AI. The facts must be vetted and verified.

Is AI Actually Any Good?

After all that, you might be wondering why everyone is so hyped over AI. Is it all just a lie?

No. The key to using AI effectively is to use it for things where veracity is either trivial to confirm or irrelevant. If you tell AI to write a script, run it. Did it work? Great! Did it fail? The only thing you lost was the time spent explaining the requirements to the AI, which is actually valuable in its own right.

Another good use is to ask it to review something and give feedback. Read the feedback and decide what applies and what doesn't. Every sentence doesn't have to be gold in order to extract some value.

MagicShel