Can you trust what
the system just did?
That question has been my job for nineteen years. I build test, measurement and control systems, instruments whose whole purpose is to report what actually happened and to be right about it. AI has been my singular focus since 2021, and in 2024 I architected the backend of Jake AI at JKI, a production LLM assistant for LabVIEW engineers. An AI feature that demos well and an AI feature that survives your real traffic are two different things, and the distance between them is a measurement problem before it is a modeling one.
Two situations tend to bring people to me. One is a team with an AI feature that works in a demo and is now answerable for it in production, with no way to say what it does on the long tail of real requests. The other is someone holding a vendor's benchmark who cannot tell whether the number means anything for their own traffic.
What I do
Six areas. The thread running through all of them is instrumentation. Before I change anything I want to know how you will tell whether it worked, and what the measurement would look like if it quietly did not.
Evaluation and measurement for AI systems
Does the thing do what you think it does, measured against your traffic instead of a demo corpus. I build the harness. Then I write the invariant down before the run, so the result cannot quietly reshape what I said I was testing for. Most of the effort goes into the parts nobody enjoys: negative controls, cells where the effect cannot exist, and contrastive pairs, meaning benign inputs that closely resemble the thing you are trying to catch. That is how you check the arm nobody is watching, and it is where the three cases below came from.
Self-hosted and private inference
Open-weights models on your own hardware, in your VPC, or fully disconnected. Model and quantization selection, VRAM and throughput budgets, the serving stack, and the unglamorous part where it has to keep running after I leave.
Retrieval and model adaptation
A pipeline can score well on retrieval recall and still answer from the model's own weights instead of the documents you retrieved. That is a common shape and an expensive one, and you only see it if you measure grounding separately from relevance. I work on RAG pipelines, chunking, reranking, and evaluating whether the retrieved context is being used at all. Fine-tuning and adaptation where the domain warrants it.
Agent scoping, tool-call boundaries and policy enforcement
What an agent is allowed to reach, what it must never do, and how you would know it obeyed. Sometimes the answer is that a boundary you want cannot be enforced at all, and I will show you the numbers that say so.
AI-assisted development
I use coding agents daily and ship production code from them. The practical parts I can help with are harness setup, keeping context useful as a session runs long, and review practice for code nobody typed. There are tasks where agents cost more than they save. Working out which ones, in your codebase, is most of the value.
Test, measurement and control systems
The original practice, still offered. Automated test benches, data acquisition, instrument control, hardware in the loop, and the software architecture around them. Deep LabVIEW background, Certified LabVIEW Architect.
Three times the check said fine and the system was not.
These come from my own calibration work while building PRIOR, not from client engagements. The corpora are mine and the bank in the third one is fictional. I am saying that first because the point is the method, and a method has to survive you asking where the number came from.
The brand-token trap
A support scope for a storage product passed every test I had. Then I deleted the product name from the test questions, which is how customers write when they are already talking to you. In-scope questions served fell from 88% to 0%, and hard refusals of legitimate users rose from 0% to 94%. The pack had learned the brand rather than the domain, and my test set had the brand in every row. That is a 17-row in-scope set on one pack, so read it as the shape of the failure and not as an effect size.
The confirm-band collapse
A composed policy held its safety invariant perfectly. Zero legitimate requests blocked, every run. It was also pushing 49 of 55 real customer questions into a confirmation step before anyone got an answer, because the lower threshold had landed below the median benign score. Questions served cleanly went from 58% to 11%. The invariant I was checking measured the one thing that had not gone wrong.
The boundary that was not expressible
"Don't be a free coding assistant" is a reasonable rule for a banking support agent, and I could not enforce it. Adding the technical-sounding questions a bank actually receives to the benign set inverts the scores: highest benign 0.4877 against lowest prohibited probe 0.3286, a margin of negative 0.159. "I'm getting an error code when I try to make a transfer" reads as more of a coding request than "write me a SQL query to analyse my transaction history" does. Trimming the two exemplars responsible narrowed the margin to negative 0.026 and still missed 2 of 7 probes. No threshold separates them, by me or by anyone else, because the thing that separates them is not in the text being scored.
I found that in an afternoon. The alternative was finding it after a fine-tuning run, with a week of refusing the customers whose payments had failed.
I would rather tell you a boundary is not enforceable than sell you a policy that fails quietly on your highest-stakes traffic.
How engagements work
Three shapes, and most people start with the first one.
Assessment
Fixed price, one to two weeks. You send a sample of real traffic and a plain statement of what the system is supposed to do. I come back with what holds, what does not, the numbers, and how I got them, in a document that will stand up to someone who wants to argue with it.
Build
Hands on, either fixed scope or a weekly rate. Evaluation harnesses, retrieval pipelines, self-hosted deployments, fine-tuning runs, agent boundaries, test systems. Your team owns the code and can run it without me.
Advisory
A standing block of hours each month. Design review, a second opinion before you commit to an architecture, or help reading a benchmark someone has put in front of you.
An assessment is the cheapest way to find out whether the rest of the work is worth doing.
Background
I have been a consultant since 2007, which covers most of my career. At Viewpoint Systems (2007 to 2014) I wrote software for clients across a wide range of industries and supported deployments in over 170 labs worldwide. I earned the Certified LabVIEW Developer and Certified LabVIEW Architect credentials there.
At Nuvera Fuel Cells (2014 to 2018) I was the software architect on interdisciplinary teams spanning R&D labs, manufacturing test stands, and engineering design and modeling tools. I built NBase, a dependency-injection plugin architecture that went on to be used across projects there, and I presented at the CLA Americas Summit in 2016 and 2018.
At Composed Systems (2018 to 2022) I was a partner, working with biomedical, materials research and alternative energy clients.
At JKI (2023 to 2026) I was a Senior and then Staff Software Engineer, and I architected the backend of Jake AI, their production LLM assistant for LabVIEW engineers. It went to beta in November 2024 and is public at jki.net/jake.
In 2026 I founded Eagle Logic and built PRIOR, a self-hosted behavioral control plane for open-weights models, which is where the calibration work above comes from.
ctx is a Rust code-intelligence CLI I maintain under MIT. Its documentation includes a section on how its call-graph analysis reports the cases it cannot resolve. That habit matters more to me than the tool does. PRIOR ships the same way: its capabilities and limitations page lists the boundaries I could not make work, with the numbers.
If you hire me to measure something and the measurement says your idea does not work, that is the result you get, early enough to be worth having.
Tell me what it is supposed to do, and how you would know.
Send me the system, the traffic it sees, and the thing you are not sure about. If I am not the right person for it, I will tell you that and point you at who is.