I Trained an AI to Predict Gold Prices. It Flipped Coins. Here’s the Autopsy.

After 819 trades of pure rules, the tempting next step was AI. I built the full ML filter — then let walk-forward testing kill it.

In Part 2, I published the complete backtest of Setup B: +196R over 11.75 years, profit factor 1.37, every number public. The obvious next move was to bolt on machine learning and squeeze out more. So I did it properly — 10 clean features, no leakage, walk-forward validation — and the models told me, to my face, that they had nothing.

This is the autopsy of an honest negative result.

The setup: built to not fool myself

The ML filter’s job was simple: look at each Setup B signal and predict whether it would win. The design rules were strict:

• 10 features, all known at signal time: direction, distance from EMA in ATR units, Stochastic K/D, reward-to-risk to first target, leg size in ATR units, hour and day-of-week encoded as sine/cosine. Nothing a trader couldn’t see on the chart at that moment.
• Aggressively removed anything from the future: the outcome, the R-multiple, how long the trade lasted, the maximum favorable/adverse excursion — all banned. Also removed three features that turned out to be constants for Setup B (they’d only add noise dressed as signal).
• Label: 1 if the trade won (195 of 819 = 23.8%), 0 otherwise.
• Validation: 5-fold TimeSeriesSplit — train on the past, test on the future, never shuffled. 680 out-of-sample predictions pooled.

If the features carried no real information, this design would show it. That’s the point of doing it this way.

The result: two models, one verdict

• Logistic Regression: walk-forward AUC 0.514 — no signal
• HistGradientBoosting: walk-forward AUC 0.523 — no signal

A coin flip scores 0.50. My models scored 0.51–0.52. Per fold, the AUC swung between 0.43 and 0.60 — pure noise around the coin line, not a stable edge hiding in one regime.

The features simply could not tell winners from losers.

The illusion I almost fell for

Here is the dangerous part. Slice the predictions by confidence threshold and one number looks delicious: gradient boosting at threshold 0.3 gives 194 trades, profit factor 1.54 — better than the 1.37 baseline.

I almost wrote the “AI improves my system” post. Then I checked where the threshold came from: it was picked on the evaluation set itself. Choose the cutoff after seeing the answers, and any random model can look skilled. That’s not forecasting — that’s the threshold-selection illusion.

Discipline means deciding your pass criteria before you look.

The kill rule — set in advance, honored in public

Before training, I wrote down the bar a model had to clear to ship:

• Profit factor ≥ 1.3, and beats the 1.37 baseline
• At least 100 trades
• Walk-forward AUC ≥ 0.55

The models failed the AUC bar (0.523 < 0.55). So the training pipeline — built to enforce this automatically — refused to export the model. No ONNX file was created. Nothing shipped. No human override.

That automation matters: in the moment, with a 1.54 staring at you, a human would have rationalized. The script didn’t.

What survived the autopsy

A negative result is still a result — and this one had real value:

1. A reusable pipeline: when I have a longer broker feed, I re-run one command and get a fresh verdict instantly.
2. A verified export path: Logistic Regression exported to ONNX and reproduced identical results (error ~6e-08). Gradient boosting hit a converter incompatibility — documented so I don’t step on it again.
3. An honest saving: I now know that bolting ML onto this rule system adds complexity with zero skill. I won’t spend months operating a useless layer — or worse, trading one.

And the likely reason for the failure is instructive: my features describe each signal in isolation. They carry no market-regime context — no sense of whether gold is trending, choppy, or sitting under a news event. If I ever retry ML, that’s where the new features have to come from, on a longer broker-native feed.

The point

Adding AI to a good rule-based system did not make it better. It made it more complicated and no more predictive. So Setup B stays exactly as it is — pure rules, profit factor 1.37, no AI.

Publishing a negative result is easy to promise and hard to do. But “my AI filter failed” is exactly the kind of sentence this series exists to print.

---

This is Part 3 of a 4-part series. In Part 4, the EA went live on a VPS — and I promised not to touch it for 6 months.

Missed the numbers? Read Part 2: 819 trades, every number public. Start at the beginning: Part 1: the bug that lied to me.