BiLSTM with attention on market data. The hard part is an honest experiment, not the network

Recurrent networks with an attention mechanism are the usual starting point for time-series modelling. We ran a research project on them aimed at diagnosing equity markets: a next-session direction classifier, detectors for support and resistance levels, and an agent trained with PPO in a simulated market environment. The repository now has 186 commits and roughly 780 automated tests.

The main lesson is not about any architecture. With financial data, picking the network is the easy part. The hard part is building an experiment that will not lie to you — no leakage from the future, an honest baseline, and the willingness to write down "this does not work". Most of our hypotheses ended exactly that way, and each of those results was worth more than another layer in the model.

The models

Every model shares the same temporal block:

a bidirectional LSTM reads a window of recent sessions (typically 30 or 60) and builds a representation of each day using context on both sides of it — but only inside a window that ends today, so bidirectional does not mean looking ahead;
multi-head self-attention (4 heads) with a residual connection and layer norm lets the model weight individual days instead of treating them equally;
mean pooling over time produces one vector per instrument for the task heads.

The first version had multi-horizon heads: one shared encoder with separate classification outputs for several horizons. The same encoder was later reused in:

a next-session direction classifier (down / sideways / up, with a band scaled by ATR volatility),
a next local extremum predictor — when it happens and at what level, using quantile regression,
a support/resistance detector on hourly data, where we also compared a convolutional TCN encoder, PatchTST-style tokens and a set transformer with a time-gap-aware attention bias,
an RL policy in a Gymnasium environment: one decision per day (D1), execution simulated on hourly bars (H1), trained with PPO.

In the RL policy attention appeared twice: across time (per instrument, with shared weights) and across instruments — a small transformer block over all instruments' vectors, meant to let the policy reason about correlated exposure. On top of that sat a Wide & Deep split: the deep part is the temporal encoder, the wide part holds flat fundamental features and an instrument embedding.

Data: point-in-time first

A model is only as honest as the data it can see on a given simulated day. The rules we settled on:

One source of prices. All OHLCV data comes from a shared time-series database refreshed hourly by a separate process. The research code only reads from it and mirrors results to local Parquet files.
H1 is derived, not fetched. The database's native resolution is 30-minute bars; hourly bars are built by merging pairs of them. D1, H1 and M30 stay consistent instead of being stitched from different sources.
Fundamentals by filing date. On a simulated day the model sees only reports filed on or before that day — never by period end, which is information the market did not have yet. This has its own tests, including "never returns a report filed after the simulation date" and timezone edge cases.
Extrapolation from the past only. Daily-resolution fundamental ratios are extrapolated linearly from the last two known reports, capped at 180 days. Instruments without filings, such as ETFs, get an explicit "no fundamentals" flag so that zeros do not masquerade as real values.
A report calendar built from history. With no calendar of future filings available, we estimate a company's next report date from its own filing cadence, again restricted to what was known on the simulated day.
Scale-free features. Prices expressed as distance from the close in ATR units, MACD divided by ATR, RSI, volume relative to its average. The model cannot recognise a company by its price level.

The supervised experiments use a chronological split: training before 2024, validation on 2024, test from 2025 onward, across 90 instruments with daily data from 2020. Between splits we purge — a row is dropped if its label window crosses into the next split. For RL we anchored the split boundaries on actual market regimes, so that training, validation and test each contain a different drawdown period rather than whichever one a percentage split happens to land on.

Method: placebos and gates set in advance

Two habits changed how we work more than any architecture did.

Label-shuffle placebos. Every model is also trained on the same data with randomly permuted labels and then scored against the real test set. If the placebo comes close to the real model, the "signal" is an artefact of the procedure.

Pre-registered gates. Before a run we write down what counts as success — for direction: test log-loss below the best naive baseline in every seed and AUC of at least 0.52 for both extreme classes; for extrema: at least +5% Brier skill and AUC of at least 0.58. The bar does not move once the results are in.

Simple baselines are always included: training-set class frequencies, a Markov "tomorrow looks like today" table, gradient boosting on last-day features, and an MLP with no time window.

What the results showed

Next-session direction: no signal. On the test set (2025 onward, classes split 31/36/34%) no model beat the naive baselines on log-loss. The best was the trivial Markov table (1.0973); BiLSTM with attention scored 1.0978 and the placebos 1.0971–1.0978. Network AUC sat between 0.50 and 0.52. The 60-day sequence models were no better than an MLP that sees only the last day, and every model peaked on validation in epoch one or two.

Timing of extrema: only the trivial part is learnable. The networks predicted whether a local high or low would occur within a few sessions with AUC around 0.74–0.80, against roughly 0.5 for placebos — so they learned something real. But a one-line statistic, "sessions since the 11-day high or low", already scored 0.74/0.76, and gradient boosting on the last bar 0.77/0.81, better than any network. The 60-session window, attention, convolutions and fundamentals added nothing beyond what today's bar already says. The level of the extremum carried no usable signal.

The rare "confident" subset was an illusion. We tested whether the flat average hid a small set of days on which the model is unusually confident. The top confidence decile improved on the baseline by 1–2% — the same as a placebo put through the identical procedure. Six further candidate features (day of week, distance to the next report, relative-strength rank, index context, volume, a longer window) failed the same test. A regularisation sweep over nine weight-decay and dropout combinations found only two regimes, overfitting or not fitting at all, with nothing in between.

Attention across instruments destabilised training. In RL, removing just the cross-instrument attention block, with no hyperparameter change, cut the KL divergence of the first PPO update from 0.21–0.38 to 0.03–0.05 — a bigger effect than a fivefold learning-rate cut. The time-only attention variant behaved like the MLP baseline. Later ablations confirmed the instability came from mixing information across instruments, not from the mechanisms we built on top of it.

Attention as a falsification tool

Attention weights have one useful property: you can measure them directly. We built a probe that checks whether the model pays more attention to days that are the high or low of the window, or to high-volume days.

The answer was instructive. On two well-trained checkpoints of the same model, the effect for window highs was +5.3% and −4.4%, and for lows −6.0% and +7.5%. The sign flipped between seeds. If attention had developed a stable way of spotting turning points, two independent training runs should agree on the direction. The support/resistance detector told the same story: attention was indistinguishable from picking a token at random, even after adding a time-gap bias and swapping the encoder.

A separate probe showed that the BiLSTM-with-attention encoder can encode the window's high and low (test R² 0.955–0.985 regardless of pooling). Representation was never the bottleneck. The questions were whether the data contains anything worth learning and whether the training signal actually reaches the part of the model that needs it.

So attention did not give us a neat story about what the model is looking at. It gave us something more useful: a cheap way to kill a hypothesis before building the next floor on it.

Traps we found in our own code

Some of the most important findings were bugs that, without tests and probes, would have produced plausible-looking results:

Training never saw a quarter of the data. The RL environment reset to the first day of the split at the start of every update and stepped forward a fixed number of days. Counted exactly: 197 days, 26.7% of the training split, never reached any update, however long training ran. Validation did not have the bug, so nothing flagged it.
A head that never trained. The parameters controlling the continuous part of the action ended training at their initial values — what looked like a learned constant was initialisation noise. The causes: one shared advantage for dozens of simultaneous decisions, and episode truncation treated as termination. For about 75% of training trades those parameters had no causal effect on the reward.
A metric scored against the wrong column. The first extrema report scored level predictions against the proximity columns, yielding a correlation of about −0.2. Once spotted, the metrics were recomputed from the saved checkpoints and a regression test was added; training and the other metrics were unaffected.
A seed set too late. The random seed was set after the model was built, so weight initialisation was not reproducible — while comparisons across seeds are the backbone of the whole method.
Validation that rewards fitting to itself. Sampling training days by regime "novelty" (Random Network Distillation) reached a better validation score sooner and a clearly worse test result. The short validation window was correlated with exactly what the sampling scheme favoured. Across many checkpoints, the validation ranking carried over to test only weakly.
Reproducing the literature. A level-detection method implemented exactly as published produced a single cluster for an instrument's entire history on our data. Published hyperparameters do not travel between datasets — worth knowing before using someone else's method as a baseline.

The limits of models trained on history

A model like a BiLSTM learns patterns from the past — prices, volumes, financial statements. That assumption breaks down when markets are driven by information rather than history. Recent years have produced an unusual number of such periods: a pandemic, wars, sudden oil-price spikes, political announcements made overnight. Markets then become unpredictable both technically and fundamentally — especially when market participants actively shape and exploit the flow of information.

Under those conditions no model trained on history alone has anywhere to find a signal — the events moving the market were never in its training data. Language models (LLMs) with access to fresh news feeds do better: within seconds they can gather and cross-reference context that will show up in the time series only with a delay. In our experience, during periods of information shocks they are often the most valuable analytical tool — and we treat networks trained on history as a complement, not an oracle.

What we took away

A cheap supervised probe first, RL second. "Is there information about X in these features?" can be settled in minutes with a supervised model and a placebo. Reinforcement learning answers the same question after hours, and with far more noise.
Placebos and pre-set gates matter more than architecture. BiLSTM, convolutional TCNs, attention and a plain MLP were statistically indistinguishable on our features. The procedure made the difference.
Best epoch = 1 is a diagnosis, not an early-stopping success. If every model peaks on validation right after initialisation, there is nothing generalisable to learn.
Point-in-time needs tests. "We do not use future data" is a statement. A test that checks it on boundary dates and timezones is a guarantee.
Negative results get the same write-up as positive ones. Every experiment has a report in the repository with its setup, tables, verdict and the follow-ups not yet run. That stops us from revisiting ideas that have already failed once.

The next methodological step is walk-forward validation across many rolling windows instead of a single chronological split. Some of our conclusions rest on one split and one placebo draw, and the reports say so explicitly.

When diagnosing training problems we used AI agents in an "ideator and critic" loop, but every claim marked as measured was checked against the code and the logs. Why that way of working raises quality is covered in AI does not ruin code. More on how we work with models: AI services.

This article describes machine-learning research and is not investment advice or a recommendation regarding any financial instrument.