In my previous post, I experimented with implementing a poor man’s testing tool for my Claude Code configuration. One option I explored while researching the space was promptfoo. At the time of the initial research, promptfoo did support Claude Code as a provider, but with a catch. It only supported API billing, which was undesirable in my case.
In this post, I will explore the options for using promptfoo with Claude Code and a subscription:
I wrote an MCP server to talk to a service. The point was to give the assistant a clean, typed way in, instead of leaving it to improvise.
Most of the time it was invoked. Sometimes it resorted to curl against the service’s API. Occasionally it went looking for a local CLI tool instead. Same task, three different routes.
Another case where Claude does its own thing. My context files say where things live on my machine, specifically so nothing has to be looked up. And yet Claude often runs find across my home directory, hunting for a path that was sitting in the file it had just loaded.
Back in April 2020 I shared how I built a nutrition tracker in Emacs that leveraged org-capture templates and or-ql to record foods, recipes, and meals. At that time, I relied on an org-mode based database and manual updates to keep track of calories, protein, carbs, and fat. While the system worked, maintaining that data was both tedious and error-prone. Each time I needed to insert a new food, I had to do an internet search to find the nutritional information and then manually update my org-mode files.
A couple of weeks ago I came across Roberto Carratalla’s blog post on Function calling on OpenShift AI. At the time I was preparing for RedHat Summit Connect Zurich 2025 where I was meant to run a workshop on Quarkus and Langchain4j with Kyra Goud, Dimitris Andreadis and Codrin Bucur. We had an issue however, related to enabling functions on OpenShift AI.
Roberto pointed us to the blog post, but I couldn’t spot what I was doing wrong. So, I decided to port the examples in the blog post to Java to make sure that I was comparing apples to apples.