FaroQuant / How it's tested
Methodology
How FaroQuant is tested: grounding, metrics and their limits
FaroQuant's AI copilot is told to take every number from a tested quant engine, and a checker looks for numbers that didn't come from there. This page says exactly what that check does, what it can't do, what we've measured, and what each metric leaves out.
Short answer
Grounding here means one thing: every number in an answer should trace to a tool result from the same question. The checker flags numbers that don't, and the app shows the flag. With no chat history, the local test model's answers were clean 59 times out of 60. With history on, as the shipped chat works, it often answered from memory: 24 of 60 answers were clean and 243 numbers were flagged. A number that passes is traceable, which isn't the same as correctly explained.
What the checker does
The copilot is instructed to answer by calling five tools that run FaroQuant's quant engine: a portfolio summary, a single metric, a position's detail, a price-shock scenario and a benchmark comparison. After it answers, the checker:
- pulls every number out of the reply;
- collects every number in that turn's tool outputs;
- accepts a reply number if it matches a tool number as written, with the sign flipped, or multiplied or divided by 100 (so −0.1795 can appear as 17.95%), within 1% or 0.005;
- flags everything else and shows the flags to the user.
This follows the definition the major providers use: grounding ties output to verifiable sources, and reduces invented content rather than eliminating it.[1][2] NIST calls invented content confabulation and lists it as a named risk of generative AI.[3]
What it can't catch
- Everyday numbers are never flagged. 0, 1, 2, 3, 4, 5, 10, 20, 50, 95, 99, 100, 252 and 500 are skipped, as is anything from 1900 to 2100, which is treated as a year.
- Loose matching. An invented number can pass by landing within 1% of a real one.
- Right number, wrong meaning. The checker doesn't know which metric a number belongs to. In the recorded demo, a Spanish answer passed while calling a 95% Value at Risk the loss “in the worst scenario”. That's wrong, as the VaR section below explains.
- Words aren't checked. Calling a Sharpe ratio “moderate” passes, because only numbers are compared.
- Tool calls aren't forced. The copilot is instructed to call the tools every turn, but the code doesn't force a call; under the default setting the model decides.[4] The checker catches the misses afterwards.
What we've measured
The eval asks 20 fixed questions, 14 in English and 6 in Spanish, including two “should I buy or sell?” questions that must be refused, about one demo portfolio. It runs in two modes: fresh, where each question stands alone, and with history, which is how the shipped chat sends requests. On September 24, 2026 we ran each mode three times on the local model qwen2.5:7b, at the code's default temperature of 0.2.
| Run | Fresh: clean answers | Fresh: flagged numbers | With history: clean answers | With history: flagged numbers |
|---|---|---|---|---|
| 1 | 20 / 20 | 0 | 7 / 20 | 59 |
| 2 | 19 / 20 | 1 | 8 / 20 | 87 |
| 3 | 20 / 20 | 0 | 9 / 20 | 97 |
| Total | 59 / 60 | 1 | 24 / 60 | 243 |
Raw logs: docs/eval-logs in the repository (files starting run7-2026-09-24).
With history on, 33 of the 60 answers used no tool at all, and 32 of those were flagged. Chat history keeps earlier answers but not their tool outputs, so a number repeated without a new tool call can only have come from an earlier answer. Of the 243 flagged numbers, 26% matched an earlier answer that had passed the check, 27% matched only an earlier flagged answer, and 47% appeared nowhere in the conversation the model could see.
A July 2026 run of the same eval found 3 of 20 clean with history on. We don't read today's better numbers as an improvement: the English and Spanish instructions haven't changed since then, and three runs of 20 questions is a small sample.
What hasn't been measured
- Claude. FaroQuant uses Claude when an API key is configured, and that path has never been through this eval. Whether it re-calls tools more reliably is untested.
- Other languages. The app speaks Portuguese, French and German too; only English and Spanish questions were tested.
- Other portfolios and questions. One demo portfolio and 20 questions are not a general accuracy rate.
- Whether explanations are right. Nothing automated checks that.
The metrics, and what each one doesn't tell you
Every metric below is computed by FaroQuant's engine from about two years of daily closing prices, against SPY as the benchmark and a constant 4.3% risk-free rate. The unit tests check the arithmetic against independent libraries. They don't show that any metric predicts anything.
Sharpe ratio
How it's computedThe average daily return above the risk-free rate, divided by its standard deviation, then multiplied by √252 to annualize.
What it doesn't tell youHow a holding fits with the rest of your portfolio, or what the ratio will be in the future. Sharpe himself warned against treating historical ratios as forecasts.[5] The number is also an estimate with its own error, and the √252 scaling assumes returns are independent from day to day, which isn't always true.[6] FaroQuant reports no confidence interval.
Sortino ratio
How it's computedLike Sharpe, but it only counts volatility below a minimum acceptable return, here the risk-free rate, measured over all days and annualized by √252.
What it doesn't tell youTotal risk. The result depends on the minimum return you choose and on which version of the formula is used, and when few days fall below the target it can understate risk badly. The CFA Institute paper we rely on also reports, citing Sortino and Forsey, that annualizing a downside deviation built from historical returns this way is incorrect.[7]
Beta and Jensen's alpha
How it's computedBeta is the covariance of the portfolio's returns with SPY's, divided by SPY's variance. Alpha is the annualized return left over after accounting for that beta.
What it doesn't tell youA stable, precise number. Beta measures market risk only, relative to the index you picked, and estimates move with the time window, the index and whether returns are daily, weekly or monthly; standard errors of 0.5 or more are common.[8] FaroQuant reports no standard error.
Value at Risk (VaR)
How it's computedHistorical VaR is the loss at the 5th (or 1st) percentile of past daily returns. Parametric VaR assumes returns are normally distributed. Because the inputs are daily returns, both are one-day figures.
What it doesn't tell youThe worst case. VaR is an estimate of potential loss at a confidence level.[9] For historical VaR at 95%, by construction, about 1 day in 20 in the sample lost more than the VaR figure. That's why “the loss in the worst scenario” is the wrong way to describe it.
CVaR (expected shortfall)
How it's computedThe average of the losses beyond the 95% historical VaR.
What it doesn't tell youWhat the next bad day will look like. It describes the tail of one past sample. Bank regulators moved from VaR toward expected shortfall because it captures tail risk better, which says something about VaR's blind spot, not about CVaR's accuracy.[10]
Maximum drawdown
How it's computedThe largest fall from a peak in the portfolio's value to a later low, over the two-year window.
What it doesn't tell youA figure you can compare across windows. The longer you measure, the larger the maximum drawdown you should expect,[11] and daily data can show deeper troughs than month-end data. It also says nothing about how long recovery took; regulators that define it for commodity pools require the dates separately.[12]
Where the data comes from
Daily closing prices come through yfinance, an open-source library that is not affiliated with or endorsed by Yahoo; its documentation says it's intended for research and educational use and points to Yahoo's terms, which restrict automated data collection.[13][14] So the data is not official, licensed or real-time. Prices are cached for 24 hours, a stale cache is labeled as stale, and if a backup provider is used its prices are split-adjusted but not dividend-adjusted.
What FaroQuant doesn't do
No valuation or discounted-cash-flow models, no price targets, no forecasts, no trade execution, no brokerage links and no personalized advice. The copilot is instructed to refuse “should I buy?” questions and explain the relevant metrics instead; that's an instruction, not a guarantee, which is why the eval tests it. FaroQuant is an educational tool, not a registered investment adviser.
Sources
- Google Cloud, “Grounding overview”, Vertex AI documentation. docs.cloud.google.com
- Anthropic, “Reduce hallucinations”, Claude documentation. platform.claude.com
- NIST, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1), July 2024. doi.org/10.6028/NIST.AI.600-1
- Anthropic, “Tool use with Claude”, Claude documentation. platform.claude.com
- William F. Sharpe, “The Sharpe Ratio”, Journal of Portfolio Management 21(1), 1994. Author's copy
- Andrew W. Lo, “The Statistics of Sharpe Ratios”, Financial Analysts Journal 58(4), 2002. doi.org/10.2469/faj.v58.n4.2453
- Deborah Kidd, “The Sortino Ratio: Is Downside Risk the Only Risk that Matters?”, CFA Institute, 2012. PDF
- Aswath Damodaran, “Estimating Risk Parameters”, NYU Stern working paper. PDF
- SEC Rule 18f-4, 17 CFR 270.18f-4(a), definition of value-at-risk. eCFR
- Basel Committee on Banking Supervision, “Minimum capital requirements for market risk”, January 2019. bis.org
- Malik Magdon-Ismail, Amir F. Atiya, Amrit Pratap and Yaser S. Abu-Mostafa, “On the Maximum Drawdown of a Brownian Motion”, Journal of Applied Probability 41(1), 2004. doi.org/10.1239/jap/1077134674
- CFTC rule, 17 CFR 4.10(k)–(l), definitions of drawdown and worst peak-to-valley drawdown. eCFR
- yfinance README, GitHub. github.com/ranaroussi/yfinance
- Yahoo Terms of Service. legal.yahoo.com