"""
Make arbfree markets
=====================
This module contains :class:`MkArbFree`, a utility class
has two main modes:
* ``expiry``: fits per expiry an arbitrage-free mid price within the bid-ask spread,
and drops any expiries for which this is not possible as it implies actionable arbitrage.
This is a pre-processor for any benchmark which attempts to fit arbitrage-free models
to otherwise arbitrageable data. It is also a useful tool to detect and handle arbitrage in the data, which is a common issue
in WIDB data.
* ``surface``: fits a fully arbitrage free surface, on a strike grid which theoretically means
that a mid-price fit exists if and only if the
resulting fit is within bid/ask spread.
This function can become very slow.
The :meth:`MkArbFree.__call__` function returns an :class:`ArbFreePricer` class which contains the fit to the market and can also compute option
prices for arbitrary strikes and expiries using the fitted model.
*Developer note:*
A really diffult day for testing (likely very wrong data) is ``SPX 2005-11-21``.*
"""
from __future__ import annotations
import math
import warnings as warnings
from dataclasses import asdict, dataclass, field
import cvxpy as cp
import numpy as np
from cdxcore.bs import bs
from cdxcore.err import verify, verify_inp
from cdxcore.pretty import PrettyObject
from cdxcore.verbose import Context
from cdxcore.version import version
from cvxpy.error import SolverError
from quantdata.biz_calendar import DAYS_PER_YEAR, ONE_MINUTE
from quantdata.sanos import IMPOSE_FLOOR_ON_DENSITY
from quantdata.sanos.sanos_tools import (
SanosLPSolver,
_compute_market_fit,
repair_density_nonnegative,
warn_and_fix_density,
)
from quantdata.utils.backbone import BackboneNumpy
[docs]
@version("0.81")
class ArbFreePricer(PrettyObject):
"""
Arbitrage-Free Option pricer based on DLV/SANOS.
This class contains the result of :meth:`MkArbFree.__call__`.
This function call may drop expiries if they have arbitrage which cannot be resolved.
To map the expiries stored in this class back to the input expiries, use the ``input_index`` field.
The class attributes are:
* ``expiries``: array ``(T,)`` of expiries used. This may be fewer expiries than provided as input.
* ``input_index``: array of size ``(T,)`` which maps each returned expiry to the index of the input expiry it corresponds to. In other words ``output_expiries == [ input_expiries[j] for j in input_index ]``.
* ``masks``: a list of ``T`` boolean masks by expiry indicating which strikes were considered during the no-arb fit.
The mask for expory ``j`` has the same length ``Nj`` as the ``strikes[ input_expiries[j] ]``.
The mask is False for strikes which were dropped because they had zero implied density in the per-expiry fit, and for strikes which were dropped because they had too steep bid-ask slopes at the boundaries.
The fit minimizes ``max( 0, fit[mask] - ask[mask] ) + max( 0, bid[mask] - fit[mask] )``.
* ``strikes``: a copy of the ``T`` input strike arrays for the expiries used.
* ``bids``: a copy of the ``T`` arrays of input bids for the expiries used.
* ``asks``: a copy of the ``T`` arrays of input asks for the expiries used.
* ``fits1``: list of ``T`` per-expiry fitted arbitrage-free prices within bid/ask (where masked) for the input ``strikes`` used. The element ``fit1[j]`` is a vector of length ``strikes[j]``.
Within numerical accuracy we have ``bids[j][mask[j]] <= fit1[j][mask[j]] <= asks[j][mask[j]]``.
* ``fitsX``: list of ``T`` surface fitted arbitrage-free prices for the input ``strikes``.
These prices are globally arbitrage-free across the surface, but they are not necessarily within bid/ask spread; they are the closest fit
among strikes with ``mask`` True.
Up to machine precision existence
of ``fitsX`` within bid/ask spread in the linear case ``smoothness=0`` is equivalent to absence of arbitrage.
This field is None if ``no_arb_mode`` is "expiry".
Besides the pure fits, this class also contains the ingredients of the DLV/SANOS model.
* For the **"expiry"** ``no_arb_mode``
the function :meth:`MkArbFree.__call__` will generate internal model ``xstrikes`` around the provided market strikes. For those strikes it will
compute a ``xdensities`` per expiry.
* ``xstrikes``: an extended strike list with two boundary strikes add. Boundary strikes are widening in time; c.f. https://arxiv.org/pdf/2601.11209 remark 2.7.
* ``xdensities``: list of densities for ``xstrikes``, per expiry. Up to machine precision, each density is positive, sums to one, intergates to one
in the sense that `` xstrikes[t]@xdensities[t] == 1``
and reprices ``fits``.
* ``run_time_expiry``: the time taken for the per-expiry first, in seconds.
* For the **"surface"** ``no_arb_mode``
the function :meth:`MkArbFree.__call__` will generate internal model ``mstrikes`` around the provided market strikes which are a superset of the extended ``xstrike`` grid.
This surface grid is constructed by adding strikes from the next expiry if they are not too close to existing strikes.
This is typically a much denser set than the input strikes. Each expiry is guaranteed to have their ``xstrikes`` among the respective model strikes.
These fields are None if ``no_arb_mode`` is not "surface".
* ``mstrikes``: the list of internal model strikes per expiry used.
* ``mdensities```: the list of densities for ``mstrikes``. Up to machine precision, each density is positive, sums to one, intergates to one
in the sense that `` mstrikes[t]@mdensities[t] == 1``
and reprices ``fits``. Moreover, the family of ``mdensities`` is martingale density, each later density evaluaes a convex function
higher than the a previous density.
* ``xix_to_mix``: a list of arrays which gives for every ``xstrike[t]`` the position in ``mstrikes[t]``, i.e. ``mstrikes[t][xix_to_mix[t]] == xstrikes[t]``.
The element ``xix_to_mix[t]`` is ``None`` if no additional strikes were added to expiry ``t``, i.e. the strikes coincide.
* ``run_time_surface``: the time taken for the surface fit, in seconds.
"""
def __init__(self, smoothness : float|None = None):
self.smoothness : float|None = smoothness
self.expiry_linear: bool = False
self.min_half_spread : float = 1E-4
self.expiries : np.ndarray = []
self.variances : np.ndarray = []
self.input_index : np.ndarray = []
self.masks : list[np.ndarray] = []
self.strikes : list[np.ndarray] = []
self.asks : list[np.ndarray] = []
self.bids : list[np.ndarray] = []
self.fits1 : list[np.ndarray] = []
self.fitsX : list[np.ndarray]|None = None
self.xstrikes : list[np.ndarray] = []
self.xdensities : list[np.ndarray] = []
self.xix_to_mix : list[np.ndarray]|None = None
self.mstrikes : list[np.ndarray]|None = None
self.mdensities : list[np.ndarray]|None = None
self.run_time_expiry : float|None = None
self.run_time_surface : float|None = None
[docs]
def to_dict(self) -> dict[str, object]:
"""Return an HDF5-compatible representation of the pricer."""
def encode(value):
if value is None:
return self._none_sentinel
if isinstance(value, list):
return [encode(item) for item in value]
return value
return {key: encode(value) for key, value in self.items()}
[docs]
@staticmethod
def from_dict(d: dict[str, object]) -> ArbFreePricer:
"""Reconstruct a pricer serialized by :meth:`to_dict`."""
verify_inp(isinstance(d, dict), lambda: f"Expected 'd' to be a dict, found {type(d)}")
def decode(value):
if isinstance(value, str) and value == ArbFreePricer._none_sentinel:
return None
if isinstance(value, list):
return [decode(item) for item in value]
return value
decoded = {key: decode(value) for key, value in d.items()}
pricer = ArbFreePricer(smoothness=decoded.get("smoothness"))
expected_fields = set(pricer)
verify_inp(
set(decoded) == expected_fields,
lambda: (
"Serialized ArbFreePricer fields do not match the current implementation. "
f"Missing fields: {sorted(expected_fields - set(decoded))}; "
f"unexpected fields: {sorted(set(decoded) - expected_fields)}"
),
)
pricer.update(decoded)
pricer._make_numpy()
return pricer
_none_sentinel = "__ARB_FREE_PRICER_NONE__"
_eps_price = 1E-6
_eps_density = 1E-7
[docs]
def price( self, expiries : float|np.ndarray, strikes : float|np.ndarray, *, and_vols: bool = False, mode : str|None = None ) -> np.ndarray | tuple[np.ndarray, np.ndarray]:
"""
Price options at ``strikes`` with expiries ``expiries``.
Use ``mode`` to chose which pricing kernel to use: "expiry" uses the per-expiry fit
which may not be free of arbitrage in time; "surface" uses the full surface
fit which is free of arbitrage in time. ``None`` will chose the latter
if available, otherwise the former.
If ``and_vols`` is ``True`` this function returns call prices and their implied volatilities;
if ``False``, it will return only call prices.
Parameters
----------
expiries : float | :class:`np.ndarray`
The expiry or expiries to price at.
* If ``expiries`` is a float, all strikes are evaluated at that expiry.
* If ``expiries`` is a 1D array of shape ``(N,)``, then ``strikes`` must also be a 1D array of shape ``(N,)``, and the function will price each strike at their respective expiry.
* If ``expiries`` is a 2D array of shape ``(M,1)``, then ``strikes`` must be a 2D array of shape ``(1,N)``, and the function will price each strike at each expiry, returning an array of shape ``(M,N)``.
strikes : float | :class:`np.ndarray`
The strikes to price at. The shape of this array must be compatible with the shape of ``expiries`` as described above.
and_vols : bool, default ``False``
Whether to return implied volatilities in addition to call prices. If ``True``, the function returns a tuple of (call prices, implied volatilities).
If ``False``, it returns only call prices.
mode : str | None, default ``None``
Which pricing kernel to use: "expiry" uses the per-expiry fit which may not be free of arbitrage in time; "surface" uses the full surface fit which is free of arbitrage in time. ``None`` will chose the latter if available, otherwise the former.
"""
assert isinstance(self.expiries, np.ndarray), "Internal error - forgot to make my 'expiries' a numpy array"
verify_inp( mode is None or mode in ["surface","expiry"], lambda : f"Expected 'mode' to be one of 'surface', 'expiry' or None, but found {str(mode)}" )
per_expiry = mode=="expiry" if not mode is None else ( self.fitsX is None )
verify_inp( per_expiry or not self.fitsX is None, f"Cannot use 'mode' \"surface\" since no surface fit is available")
strikes = np.asarray(strikes) if not isinstance(strikes, (int,float,np.number)) else np.array([float(strikes)])
def x_price( expiry, strikes, and_vols ):
verify_inp( isinstance( expiry, (float, int, np.number)) and expiry>=0, lambda : f"Expected 'expiry' to be a non-negative number, but found {str(expiry)}")
if self.smoothness is None or expiry<ONE_MINUTE or (self.expiry_linear and per_expiry):
def call( spots, strikes ):
return np.maximum( 0., spots - strikes )
else:
vol = self.interpolate_atm_vols( expiries=expiry ) * math.sqrt(self.smoothness)
def call( spots, strikes ):
return bs.price( k=strikes/spots, vol=vol, sqrtT=math.sqrt(expiry), eps=self._eps_price )*spots
def prices():
if expiry<ONE_MINUTE:
return np.maximum( 0., 1 - strikes )
ix = np.searchsorted( self.expiries, expiry )
if ix==0:
strikes_0 = self.xstrikes[0] if per_expiry else self.mstrikes[0]
p_0 = self.xdensities[0] if per_expiry else self.mdensities[0]
return call( spots=strikes_0[None,:], strikes=strikes[:,None] ) @ p_0
if ix==len(self.expiries):
strikes_T = self.xstrikes[-1] if per_expiry else self.mstrikes[-1]
p_T = self.xdensities[-1] if per_expiry else self.mdensities[-1]
return call( spots=strikes_T[None,:], strikes=strikes[:,None] ) @ p_T
diff_ixm1 = abs( expiry - self.expiries[ix-1] )
diff_ix = abs( expiry - self.expiries[ix] )
diff = diff_ix if diff_ix < diff_ixm1 else diff_ixm1
if diff<ONE_MINUTE:
ix_mind = ix if diff_ix < diff_ixm1 else ix-1
strikes_ix = self.xstrikes[ix_mind] if per_expiry else self.mstrikes[ix_mind]
p_ix = self.xdensities[ix_mind] if per_expiry else self.mdensities[ix_mind]
return call( spots=strikes_ix[None,:], strikes=strikes[:,None] ) @ p_ix
strikes_ixm1 = self.xstrikes[ix-1] if per_expiry else self.mstrikes[ix-1]
p_ixm1 = self.xdensities[ix-1] if per_expiry else self.mdensities[ix-1]
c_ixm1 = call( spots=strikes_ixm1[None,:], strikes=strikes[:,None] ) @ p_ixm1
strikes_ix = self.xstrikes[ix] if per_expiry else self.mstrikes[ix]
p_ix = self.xdensities[ix] if per_expiry else self.mdensities[ix]
c_ix = call( spots=strikes_ix[None,:], strikes=strikes[:,None] ) @ p_ix
return ((expiry - self.expiries[ix-1])*c_ix + (self.expiries[ix]-expiry)*c_ixm1)/(self.expiries[ix]-self.expiries[ix-1])
c = prices()
if not and_vols:
return c
v = bs.implied( k=strikes, prices=c, sqrtT=math.sqrt(expiry), eps=self._eps_price )
return c,v
if hasattr(expiries,"shape"):
if len(expiries.shape)==0:
expiries = float( expiries.item() )
elif expiries.shape == (1,):
expiries = float( expiries[0] )
if isinstance(expiries, (float, int, np.number)):
return x_price( expiry=expiries, strikes=strikes, and_vols=and_vols )
expiries = np.asarray(expiries)
if len(expiries.shape) == 1:
verify_inp( expiries.shape == strikes.shape, lambda : f"If 'expiries' is a 1D vector, then 'strikes' must have the same shape, but found {expiries.shape} and {strikes.shape}" )
uttys = np.unique(expiries)
results = np.zeros_like(strikes)
vols = np.zeros_like(strikes) if and_vols else None
for utty in uttys:
mask = expiries==utty
if not and_vols:
results[mask] = x_price( expiry=utty, strikes=strikes[mask], and_vols=False )
else:
results[mask], vols[mask] = x_price( expiry=utty, strikes=strikes[mask], and_vols=True )
if not and_vols:
return results
return results, vols
if strikes.shape == (1,):
strikes = strikes.reshape(1,1)
verify_inp( len(expiries.shape) == 2 and expiries.shape[1] == 1, lambda : f"'expiries' can be a vector or a matrix if size '(M,1)'. Found shape {expiries.shape}" )
M = expiries.shape[0]
verify_inp( len(strikes.shape) == 2 and strikes.shape[0] in (M,1), lambda : f"If 'expiries' is a matrix of shape '(M,1)', then 'strikes' must be a matrix of shape '(M|1,N)'. Found expiries of shape {expiries.shape} and strikes of shape {strikes.shape}" )
N = strikes.shape[1]
results = np.zeros((M,N),dtype=strikes.dtype)
vols = np.zeros((M,N),dtype=strikes.dtype) if and_vols else None
for ix, tty in enumerate(expiries[:,0]):
if not and_vols:
results[ix,:] = x_price( expiry=tty, strikes=strikes[0 if strikes.shape[0] == 1 else ix,:], and_vols=False )
else:
results[ix,:], vols[ix,:] = x_price( expiry=tty, strikes=strikes[0 if strikes.shape[0] == 1 else ix,:], and_vols=True )
if not and_vols:
return results
return results, vols
[docs]
def interpolate_atm_vols( self, expiries: float|np.ndarray ) -> float|np.ndarray:
""" Interpolates the (ATM) variances (not smoothed) """
assert not self.variances is None, "Internal error - variances are missing"
is_float = not hasattr(expiries,"shape")
expiries = np.asarray([float(expiries)]) if is_float else np.asarray(expiries)
expiries = np.clip( expiries, self.expiries[0], self.expiries[-1] )
vols = np.sqrt( np.interp( expiries, self.expiries, self.variances ) / expiries )
assert np.all(np.isfinite(vols)), "Internal error: infinite vol"
return vols if not is_float else vols[0]
[docs]
def compute_market_fit( self, expiries : np.ndarray, strikes : list[np.ndarray], bids : list[np.ndarray], asks : list[np.ndarray], masks : list[np.ndarray]|None = None, *, and_vols : bool = True ) -> PrettyObject:
"""
Compute standardized fitting error stats
Returns
-------
Errors : :class:`PrettyObject`
A dictionary-lik object with the following fields:
* ``fits``: list of fitted prices for the input strikes, per expiry.
* ``fit_vols``: list of fitted vols for the input strikes, per expiry.
* ``fit_errs``: list of fitting errors for the input strikes, per expiry, where the error is defined as the distance to the bid/ask spread in units of half-spread, i.e. ``max( 0, fit - ask ) + max( 0, bid - fit) / max( 0.5*(ask-bid), self.min_half_spread )``.
* ``pcnt_fit``: the percentage of strikes for which the fit is outside the bid/ask spread, i.e. ``sum( fit_errs > 0 ) / len(fit_errs)``.
* ``median_fit_err``: the median fitting error among strikes for which the fit is outside the bid/ask spread, i.e. ``median( fit_errs[fit_errs>0.] )``.
* ``mean_fit_err``: the mean fitting error among strikes for which the fit is outside the bid/ask spread, i.e. ``mean( fit_errs[fit_errs>0.] )``.
"""
return _compute_market_fit( lambda expires, strikes, and_vols : self.price( expiries=expires, strikes=strikes, and_vols=and_vols ),
self.min_half_spread, expiries=expiries, strikes=strikes, bids=bids, asks=asks, masks=masks, and_vols=and_vols )
@property
def T(self) -> int:
return len(self.expiries)
def _make_numpy(self):
self.smoothness = float(self.smoothness) if not self.smoothness is None else None
self.expiries = np.asarray(self.expiries)
self.variances = np.asarray(self.variances)
self.input_index = np.asarray(self.input_index, dtype=np.int32)
self.masks = [ np.asarray(x) for x in self.masks ]
self.strikes = [ np.asarray(x) for x in self.strikes ]
self.asks = [ np.asarray(x) for x in self.asks ]
self.bids = [ np.asarray(x) for x in self.bids ]
self.fits1 = [ np.asarray(x) for x in self.fits1 ]
self.fitsX = [ np.asarray(x) for x in self.fitsX ] if not self.fitsX is None else None
self.xstrikes = [ np.asarray(x) for x in self.xstrikes ]
self.xdensities = [ np.asarray(x) for x in self.xdensities ]
self.xix_to_mix = [ (np.asarray(x,dtype=np.int32) if not x is None else None) for x in self.xix_to_mix ] if not self.xix_to_mix is None else None
self.mstrikes = [ np.asarray(x) for x in self.mstrikes ] if not self.mstrikes is None else None
self.mdensities = [ np.asarray(x) for x in self.mdensities ] if not self.mdensities is None else None
self.run_time_expiry = float(self.run_time_expiry) if not self.run_time_expiry is None else None
self.run_time_surface = float(self.run_time_surface) if not self.run_time_surface is None else None
T = len(self.expiries)
assert self.variances.shape == (T,), f"Internal error: 'variances' must be of shape (T,), but found {self.variances.shape}"
assert self.input_index.shape == (T,), f"Internal error: 'input_index' must be of shape (T,), but found {self.input_index.shape}"
assert len(self.masks) == T, f"Internal error: 'masks' must be a list of length {T}, but found {len(self.masks)}"
assert len(self.strikes) == T, f"Internal error: 'strikes' must be a list of length {T}, but found {len(self.strikes)}"
assert len(self.asks) == T, f"Internal error: 'asks' must be a list of length {T}, but found {len(self.asks)}"
assert len(self.bids) == T, f"Internal error: 'bids' must be a list of length {T}, but found {len(self.bids)}"
assert len(self.fits1) == T, f"Internal error: 'fits1' must be a list of length {T}, but found {len(self.fits1)}"
assert len(self.xstrikes) == T, f"Internal error: 'xstrikes' must be a list of length {T}, but found {len(self.xstrikes)}"
assert len(self.xdensities) == T, f"Internal error: 'xdensities' must be a list of length {T}, but found {len(self.xdensities)}"
if not self.mstrikes is None:
assert self.fitsX is None or len(self.fitsX) == T, f"Internal error: 'fitsX' must be a list of length {T}, but found {len(self.fitsX)}"
assert len(self.mstrikes) == T, f"Internal error: 'mstrikes' must be a list of length {T}, but found {len(self.mstrikes)}"
assert len(self.mdensities) == T, f"Internal error: 'mdensities' must be a list of length {T}, but found {len(self.mdensities)}"
for t in range(T):
mask_t = self.masks[t]
strikes_t = self.strikes[t]
asks_t = self.asks[t]
bids_t = self.bids[t]
fits1_t = self.fits1[t]
fitsX_t = self.fitsX[t] if not self.fitsX is None else None
assert mask_t.shape == strikes_t.shape == asks_t.shape == bids_t.shape == fits1_t.shape, f"Internal error: 'masks[{t}]', 'strikes[{t}]', 'asks[{t}]', 'bids[{t}]' and 'fits1[{t}]' must all have the same shape, but found {mask_t.shape}, {strikes_t.shape}, {asks_t.shape}, {bids_t.shape} and {fits1_t.shape}"
xstrikes_t = self.xstrikes[t]
xdensities_t = self.xdensities[t]
assert xstrikes_t.shape == xdensities_t.shape, f"Internal error: 'xstrikes[{t}]' and 'xdensities[{t}]' must have the same shape, but found {xstrikes_t.shape} and {xdensities_t.shape}"
if not self.mstrikes is None:
assert fitsX_t is None or fitsX_t.shape == mask_t.shape, f"Internal error: 'fitsX[{t}]' must have the same shape as 'masks[{t}]', but found {fitsX_t.shape} and {mask_t.shape}"
mstrikes_t = self.mstrikes[t]
mdensities_t = self.mdensities[t]
assert mstrikes_t.shape == mdensities_t.shape, f"Internal error: 'mstrikes[{t}]' and 'mdensities[{t}]' must have the same shape, but found {mstrikes_t.shape} and {mdensities_t.shape}"
[docs]
@version("0.51", dependencies=[ArbFreePricer])
@dataclass(frozen=True)
class MkArbFree:
"""
A utility class which generates a "closest" arbitrage free market using the ideas from the paper https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2642630.
Given a market with expiries, strikes, bid and asks this functional:
1) Expands the strike range following https://arxiv.org/pdf/2601.11209 remark 2.7, and drops expiries with too few strikes or with too steep bid-ask slopes at the boundaries.
2) Fits per expiry a linear no-arbitrage mid price within the bid-ask spread. It drops any expiries for which this is not possible as it implies actionable arbitrage. The function
then drops all strikes which have an implied probability density of zero (which means the fitted mid is linear at those strikes).
3) The function then fits a fully arbitrage free surface, on a strike grid which theoretically means that a mid-price fit exists if and only if the
resulting fit is within bid/ask spread. The strike grid used is returned as ``mstrikes``.
See :meth:`MkArbFree.__call__` for more details on the input and output of this functional.
"""
no_arb_mode : str = "surface" # Whether to enforce no-arbirage: "expiry" or "surface".
smoothness : float|None = None # Whether to impose SANOS smoothness.
surface_expiry_linear : bool = True # In 'surface' mode, whether to run 'expiry' in linear mode.
# control over process parameters
min_half_spread : float = 1E-4 #: Minimum bid/ask spread. The optimization is weighted by inverse half spread.
mid_fit_weight : float = 0.1 #: Weight for fitting to the mid prices of options, where 1 means mid-price fit only is active, while 0 means to fit only a linear penalty for exceeding bid-ask.
min_atm_vol : float = 0.05 #: Minimum ATM vol.
min_dlv : float|None = None #: Minimum DLV (difference of log-volatilities between consecutive expiries) to consider an expiry as non-degenerate.
max_dlv : float|None = None #: Maximum DLV (difference of log-volatilities between consecutive expiries) to consider an expiry as non-degenerate.
# timeout
timeout_seconds : float|None = 60.*5. #: Time limit for the optimization solver, in seconds. This is passed to the CVXPY solver options as "time_limit".
# micro paramters
min_strikes_per_expiry : int = 5 #: (expert use only) Minimum strikes per expiry. Expiries with fewer strikes will be dropped.
min_density : float = 1E-4 #: (expert use only) Minimum density value to consider in the optimization. Strikes with lower values will be dropped.
min_dC_eps : float = 1E-3 #: (expert use only) Minimum dC/dK slope at the boundaries. Expiries with steeper slopes will be dropped.
# protected variables do not alter the ID of this field (because they start with '_')
_debug_verbose_cvxpy : set[str]|None = None
_remove_no_arb_strikes: bool = True
_debug_super_dense : bool = False
_eps_price : float = 1E-6
_eps_density : float = 1E-7
_eps_dens_1 : float = 1E-4 #: (expert use only) What passes as ``sum(p)==1`` and ``p@K==1`` for the fitted density.
_eps_no_arb : float = 0.01 #: % half spread to tighten fit.
def __post_init__(self):
verify_inp( self.no_arb_mode in ["expiry", "surface"], lambda : f"'no_arb_mode' must be one of 'expiry' or 'surface', but found '{self.no_arb_mode}'" )
verify_inp( self.min_strikes_per_expiry >= 3, lambda : f"'min_strikes_per_expiry' must be at least 3, but found {self.min_strikes_per_expiry}" )
verify_inp( self.min_dC_eps > 0., lambda : f"'min_dC_eps' must be positive, but found {self.min_dC_eps}" )
verify_inp( self.min_half_spread > 0., lambda : f"'min_half_spread' must be positive, but found {self.min_half_spread}" )
verify_inp( self.min_density > 0., lambda : f"'min_density' must be positive, but found {self.min_density}" )
verify_inp( self.timeout_seconds is None or self.timeout_seconds >= 0., lambda : f"'timeout_seconds' must be positive or None, but found {self.timeout_seconds}" )
verify_inp( self.smoothness is None or self.smoothness >= 0., lambda : f"'smoothness' must be positive or None, but found {self.smoothness:.4g}")
verify_inp( self.mid_fit_weight >= 0. and self.mid_fit_weight <= 1., lambda : f"'mid_fit_weight' must be between 0 and 1, but found {self.mid_fit_weight:.4g}" )
verify_inp( self.min_atm_vol >= 0., lambda : f"'min_atm_vol' must be non-negative, but found {self.min_atm_vol:.4g}" )
def __str__(self) -> str:
""" Very compact but still readable summary """
surface_expiry_linear = self.surface_expiry_linear and self.no_arb_mode=="surface" and not self.smoothness is None
return f"{self.no_arb_mode},{f'std={self.smoothness:.2g},' if self.smoothness is not None else ''}{'elin' if surface_expiry_linear else ''}mhs={self.min_half_spread:.3g},"+\
f"mw={self.mid_fit_weight:.2f},mav={self.min_atm_vol:.2g},"+\
( f"midl={self.min_dlv:.3g}," if not self.min_dlv is None else "" ) + ( f"mxdv={self.max_dlv:.3g}," if not self.max_dlv is None else "" ) +\
f"mik={self.min_strikes_per_expiry},mdn={self.min_density:.3g},dC={self.min_dC_eps:.3g},dps={self._eps_dens_1:.1g}"
def __unique_hash__( self, unique_hash, debug_trace ) -> str:
""" Overwrite hash function to ignore non-functional parameters """
r = asdict(self)
r.pop("timeout_seconds", None)
if self.no_arb_mode!="surface" or self.smoothness is None:
r.pop("surface_expiry_linear", None)
return unique_hash(r, debug_trace=debug_trace)
[docs]
def __call__(self,
expiries : np.ndarray,
strikes : list[np.ndarray],
bids : list[np.ndarray],
asks : list[np.ndarray], *,
vols : np.ndarray|float|None = None, # forward volatilities per expiry, used for the strike expansion step. If a float is provided, it is used for all expiries. If None, it is estimated from the data as the median of the implied vols.
message_context: str|callable[[], str] | None = None,
verbose : Context = Context.all ) -> ArbFreePricer:
"""
Make market data arbitrage-free using ideas from the paper https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2642630.
Given a market with expiries, strikes, bid and asks this functional:
1) Expands the strike range following https://arxiv.org/pdf/2601.11209 remark 2.7, and drops expiries with too few strikes or with too steep bid-ask slopes at the boundaries.
2) Fits per expiry a linear no-arbitrage mid price within the bid-ask spread. It drops any expiries for which this is not possible as it implies actionable arbitrage. The function
then drops all strikes which have an implied probability density of zero (which means the fitted mid is linear at those strikes).
3) The function then fits a fully arbitrage free surface, on a strike grid which theoretically means that a mid-price fit exists if and only if the
resulting fit is within bid/ask spread. The strike grid used is returned as ``mstrikes``.
This function is usually very fast, but it can get stuck when the overall surface is close to arbitrage in areas which have very little
room to adjust.
This function returns a :class:`ArbFreePricer` object which contains the result of the fit, and can be used to price options at arbitrary strikes and expiries using the fitted model.
Parameters
----------
expiries : :class:``numpy.ndarray`
Increasing array of expiries (in years).
strikes : list[:class:``numpy.ndarray`]
List of strike arrays, one per expiry. Each strike array must be strictly increasing and positive, and must have the same shape as the corresponding bid and ask arrays.
bids : list[:class:``numpy.ndarray`]
List of bid price arrays, one per expiry.
asks : list[:class:``numpy.ndarray`]
List of ask price arrays, one per expiry.
vols : :class:``numpy.ndarray`` | float | None, default ``None``
Volatilities per expiry used when ``smoothness`` is not None.
If left to ``None``, the function will compute an approximate mid-atm volatility from the provided bid and ask prices per expiry.
message_context : str | callable[[], str] | None, default None
Optional context to prepend to error and warning messages.
This can be a string or a callable which returns a string, e.g. a lambda which returns a string with dynamic information such as the current expiry being processed. If None, no context is prepended.
verbose : :class:`Context`, default ``Context.all``
Verbose context for logging.
Returns
-------
Extended market: ``dict``
See above.
"""
def message(with_cols : bool = True) -> str:
if message_context is None:
return ""
if callable(message_context):
msg = message_context()
else:
msg = str(message_context)
if msg == "":
return ""
msg = msg[0].upper() + msg[1:]
if not with_cols:
return msg
return msg + ": "
T = len(expiries)
verify_inp( T > 0, lambda : f"{message()}Expected at least one expiry, but found {T}" )
verify_inp( len(strikes) == T, lambda : f"{message()}Expected number of strike arrays {T} to match number of expiries {len(expiries)}" )
verify_inp( len(bids) == T, lambda : f"{message()}Expected number of bid arrays {T} to match number of expiries {len(expiries)}" )
verify_inp( len(asks) == T, lambda : f"{message()}Expected number of ask arrays {T} to match number of expiries {len(expiries)}" )
verify_inp( expiries[0] > 0. and (len(expiries)==1 or np.min(np.diff(expiries)) > 0.), lambda : f"{message()}Expiries must be positive and strictly increasing, but found {expiries}" )
def make_variances():
if vols is None:
atm_calls = []
for t in range(T):
ixs = np.argsort( strikes[t] )
atm_calls.append( np.interp( 1., strikes[t][ixs], (bids[t][ixs]+asks[t][ixs])/2. ) )
atm_calls = np.asarray(atm_calls)
atm_vols = bs.implied( k=np.ones_like(atm_calls), prices=atm_calls, sqrtT=np.sqrt(expiries), eps=self._eps_price )
variances = expiries * (np.maximum(atm_vols, self.min_atm_vol)**2)
dts = np.diff(expiries, prepend=0.)
fwd_vars = np.maximum(0.001**2, np.diff(variances, prepend=0.) / dts ) * dts
variances = np.cumsum(fwd_vars)
del atm_calls, atm_vols, dts, fwd_vars
elif isinstance( vols, np.ndarray ):
verify_inp( vols.shape == expiries.shape, lambda : f"{message()}If 'vols' is a numpy array, it must have the same shape as 'expiries', but found {vols.shape} and {expiries.shape}" )
verify_inp( np.all(vols > 0.), lambda : f"{message()}All volatilities must be positive, but found {vols}" )
variances = expiries * (vols**2)
verify_inp( variances[0] > 0. and (len(variances)==1 or np.min(np.diff(variances)) > 0.), lambda : f"{message()}Variances derived from 'vols' must be positive and strictly increasing, but found {variances}" )
else:
variances = expiries * (float(vols)**2)
verify_inp(vols > 0., lambda : f"{message()}Volatility must be positive, but found {vols}" )
return variances
variances = make_variances()
del vols
# generate strike range, fit per-expiry
# -------------------------------------
# Add strikes to the left and right which are consistent with the no-arb condition.
r = ArbFreePricer(self.smoothness)
r.min_half_spread = self.min_half_spread
r.expiries = []
r.variances = []
r.xstrikes = []
r.xdensities = []
r.masks = []
r.strikes = []
r.asks = []
r.bids = []
r.fits1 = []
tme = verbose.timer()
r.expiry_linear = self.surface_expiry_linear and self.no_arb_mode=="surface" and not self.smoothness is None
for t, expiry in enumerate(expiries):
def local_message():
msg = message(False)
ext = f"expiry #{t} {expiry*DAYS_PER_YEAR:.1f}d"
if msg == "":
msg = ext[0].upper() + ext[1:]
else:
msg += " " + ext
return msg
verbose.write(lambda : f"\r{local_message()}: Generating extended strikes... ", end='')
# strikes
# -------
strikes_t = strikes[t]
if len(strikes_t) < self.min_strikes_per_expiry:
verbose.write(lambda : f"\r{local_message()}: Skipping expiry since it has fewer than {self.min_strikes_per_expiry} strikes.")
continue
xstrikes_t, mask_t, notice = self.make_extended_strikes( strikes_t,
bids=bids[t],
asks=asks[t],
prev_lower = r.xstrikes[-1][0] if len(r.xstrikes) > 0 else None,
prev_upper = r.xstrikes[-1][-1] if len(r.xstrikes) > 0 else None,
message_context = local_message)
assert not xstrikes_t is None, "Internal error: xstrikes_t is None."
if not notice is None:
verbose.write(lambda : notice() + ": ", head=False, end='')
assert np.all( xstrikes_t[1:-1] == strikes_t[mask_t] ), "Mask error...?"
verbose.write(lambda : f"using {len(xstrikes_t)} extended strikes based on {len(strikes_t)} market strikes.", head=False)
# linear fit per expiry
# ---------------------
strikes_t = strikes[t]
bid_t = bids[t]
ask_t = asks[t]
verify_inp( bid_t.shape == ask_t.shape == strikes_t.shape, lambda : f"{local_message()}: 'strikes[{t}]', 'bids[{t}]' and 'asks[{t}]' must all have the same shape, but found {strikes_t.shape}, {bid_t.shape} and {ask_t.shape}" )
verify_inp( np.min( ask_t - bid_t ) > 0., lambda : f"{local_message()}: All options must have positive bid/ask spread, but found minimum spread {np.min(ask_t - bid_t):.4g}" )
p_t, fit_t = self.expiry_density(
expiry = expiry,
variance = variances[t],
strikes_t = strikes_t,
bid_t = bid_t,
ask_t = ask_t,
mask_t = mask_t,
xstrikes_t = xstrikes_t,
force_linear = r.expiry_linear,
message_context = local_message,
verbose = verbose)
assert xstrikes_t.shape == p_t.shape, f"Internal error: {local_message()}Shape of xstrikes {xstrikes_t.shape} does not match shape of density {p_t.shape}"
fit_mask = ( fit_t >= bid_t - self._eps_price ) & ( fit_t <= ask_t + self._eps_price ) & mask_t
fit_mask_live = fit_mask[mask_t]
if not np.all(fit_mask_live):
if np.sum(fit_mask_live) < len(fit_mask_live)//2:
verbose.write(lambda : f"\r{local_message()}: skipping expiry since we cannot fit an arbitrage-free mid within bid/ask spread for more than half of the strikes.")
continue
if self._remove_no_arb_strikes:
assert np.all( xstrikes_t[1:-1] == strikes_t[mask_t] ), "Mask error...?" # only true up here.
xfit_mask = np.concatenate(([True], fit_mask_live, [True]))
xstrikes_R_t = xstrikes_t[xfit_mask]
pR_t, fitR_t = self.expiry_density(
expiry=expiry,
variance=variances[t],
strikes_t=strikes_t,
bid_t=bid_t,
ask_t=ask_t,
mask_t=fit_mask,
xstrikes_t=xstrikes_R_t,
force_linear = r.expiry_linear,
message_context=local_message,
verbose=verbose,
)
fitR_mask = ( fitR_t >= bid_t - self._eps_price ) & ( fitR_t <= ask_t + self._eps_price ) & fit_mask
fitR_mask_live = fitR_mask[fit_mask]
if np.all(fitR_mask_live):
verbose.write(lambda : f"\r{local_message()}: fit achived after removing {np.sum(~fit_mask)} of {len(fit_mask)} strikes.")
xstrikes_t = xstrikes_R_t
p_t = pR_t
fit_t = fitR_t
else:
verbose.write(lambda : f"\r{local_message()}: removing {np.sum(~fit_mask)} of {len(fit_mask)} strikes did not achieve fit; keeping all strikes but masking the options out.")
del pR_t, fitR_t, fitR_mask, fitR_mask_live
mask_t = fit_mask
del fit_mask, fit_mask_live
m_bid_t = bids[t][mask_t]
m_ask_t = asks[t][mask_t]
m_fit_t = fit_t[mask_t]
assert np.all( m_fit_t >= m_bid_t - self._eps_price ) and np.all( m_fit_t <= m_ask_t + self._eps_price ), f"Internal error: {local_message()}Expiry #{t} {expiry*DAYS_PER_YEAR:.1f}d: fitted mid is not within bid/ask spread after attempted fixes. This should not happen."
assert xstrikes_t.shape == p_t.shape, f"Internal error: {local_message()}Shape of xstrikes {xstrikes_t.shape} does not match shape of density {p_t.shape}"
# store result
r.expiries.append( expiry )
r.variances.append( variances[t] )
r.input_index.append( t )
r.xstrikes.append( xstrikes_t )
r.xdensities.append( p_t )
r.masks.append( mask_t )
r.strikes.append( strikes[t] )
r.bids.append( bids[t] )
r.asks.append( asks[t] )
r.fits1.append( fit_t ) # fit is computed for all strikes, also those unmasked
r.expiries = np.array(r.expiries, dtype=np.float64)
r.variances = np.array(r.variances, dtype=np.float64)
r.run_time_expiry = tme.seconds
def expiry_message():
msg = message(False)
if msg == "":
return "Expiry fit"
else:
return msg + " expiry fit"
verbose.write(lambda : f"\r{expiry_message()}: Generated {len(r.expiries)} {'linear' if self.smoothness is None else f'{self.smoothness:.2f}-smooth'} {'arbitrage-free ' if self.no_arb_mode != 'off' else ''}expiries with extended strikes from {T} market expiries. This took {tme}.")
if self.no_arb_mode != "surface":
r._make_numpy()
return r
# backward-dense grid
# -------------------
# Carry grid forward as long as the strike range isn't too tight.
# We are using the xstrike_t grid from which we removed all strikes
# for which the per-expiry density is zero
T = len(r.expiries)
r.mstrikes = [None]*T
r.mdensities = None #set below
r.xix_to_mix = [None]*T
r.fitsX = []
min_dx = None
mguess_p = [None]*T
tme = verbose.timer()
for t in range(T-1,-1,-1):
# fix strikes
# 1) Ensure outside strike range is increasing.
# 2) Add strikes of next expiry if not too close to existing strikes. Always keep current strikes
xstrikes_t = r.xstrikes[t]
mstrikes_t = xstrikes_t
mp_t = r.xdensities[t]
project = None
if t<T-1:
next_mstrikes = r.mstrikes[t+1]
assert next_mstrikes[0] <= xstrikes_t[0] and next_mstrikes[-1] >= xstrikes_t[-1], f"Internal error: {message()}next_mstrikes[0] {next_mstrikes[0]:.3f} > xstrikes_t[0] {xstrikes_t[0]:.3f} or next_mstrikes[-1] {next_mstrikes[-1]:.3f} < xstrikes_t[-1] {xstrikes_t[-1]:.3f}"
min_dx = np.diff(mstrikes_t).min() if not self._debug_super_dense else min( np.diff(mstrikes_t).min(), np.diff(next_mstrikes).min() )
collect = []
project = []
mgpcl = []
inext = 0
add_any = False
# skip next strikes until relevant
while next_mstrikes[inext] <= xstrikes_t[0]+min_dx:
inext += 1
project.append( len(collect) )
collect = [ xstrikes_t[0] ]
mgpcl = [ mp_t[0] ]
for ithis in range(1, xstrikes_t.shape[0]):
k = xstrikes_t[ithis]
# add next strikes up to 'k'
while next_mstrikes[inext] <= k-min_dx:
if next_mstrikes[inext] - collect[-1] >= min_dx:
collect.append( next_mstrikes[inext] )
mgpcl.append( 0. )
add_any = True
inext += 1
project.append( len(collect) )
collect.append( k )
mgpcl.append( mp_t[ithis] )
# skip next_mstrikes which are too close to k
while next_mstrikes[inext] <= k+min_dx:
inext += 1
if inext >= next_mstrikes.shape[0]:
break
assert len(collect) == len(mgpcl), f"Internal error: {message()}collect length {len(collect)} does not match mgpcl length {len(mgpcl)}"
if add_any:
mstrikes_t = np.array(collect, dtype=xstrikes_t.dtype)
mp_t = np.array(mgpcl, dtype=np.float64)
assert np.min( np.diff( mstrikes_t ) ) >= min_dx-1E-6, f"Internal error: {message()}min dx {np.diff(mstrikes_t).min():.6g} < min_dx {min_dx:.6g}"
assert mstrikes_t[0] == xstrikes_t[0] and mstrikes_t[-1] == xstrikes_t[-1], f"Internal error: {message()}mstrikes_t[0] {mstrikes_t[0]:.3f} != xstrikes_t[0] {xstrikes_t[0]:.3f} or mstrikes_t[-1] {mstrikes_t[-1]:.3f} != xstrikes_t[-1] {xstrikes_t[-1]:.3f}"
project = np.array( project, dtype=np.int32 )
else:
project = None
assert project is None or np.allclose( xstrikes_t, mstrikes_t[project] ), f"Internal error: {message()}projection error, xstrikes_t {xstrikes_t} should be equal to mstrikes_t[project] {mstrikes_t[project]}"
r.mstrikes[t] = mstrikes_t
r.xix_to_mix[t] = project
mguess_p[t] = mp_t
# solve surface no-arb
# --------------------
def surface_message():
msg = message(False)
if msg == "":
return "Surface fit"
else:
return msg + " surface fit"
surface = self.surface_densities(
expiries = r.expiries,
variances = r.variances,
strikes = r.strikes,
asks = r.asks,
bids = r.bids,
masks = r.masks,
xstrikes = r.mstrikes,
xguessed_p = mguess_p,
hard_fit = False,
message_context = surface_message,
verbose = verbose
)
r.fitsX = surface.fits
r.mdensities = surface.xdensities
r.run_time_surface = tme.seconds
# make numpy
r._make_numpy()
return r
[docs]
def surface_densities( self, *,
expiries : np.ndarray,
variances : np.ndarray,
strikes : list[np.ndarray],
asks : list[np.ndarray],
bids : list[np.ndarray],
masks : list[np.ndarray]|None,
xstrikes : list[np.ndarray],
xguessed_p : list[np.ndarray]|None = None,
force_linear : bool = False,
hard_fit : bool = False,
message_context : callable|str|None = None,
verbose : Context = Context.quiet
):
"""
Stand-alone smooth or linear fit to a given surface
"""
def message(with_cols : bool = True) -> str:
if message_context is None:
return ""
if callable(message_context):
msg = message_context()
else:
msg = str(message_context)
if msg == "":
return ""
assert msg[-2:] != ": ", f"Internal error: message_context should not end with ': ' found '{msg}'"
msg = msg[0].upper() + msg[1:]
if not with_cols:
return msg
return msg + ": "
T = len(expiries)
assert T == len(strikes) == len(xstrikes) == len(asks) == len(bids) == len(variances), f"Internal error: length of expiries {len(expiries)}, strikes {len(strikes)}, xstrikes {len(xstrikes)}, asks {len(asks)}, bids {len(bids)}, and variances {len(variances)} must all be the same."
assert xguessed_p is None or len(xguessed_p) == T, f"Internal error: length of xguessed_p {len(xguessed_p)} must be the same as length of expiries {T}."
assert masks is None or len(masks) == T, f"Internal error: length of masks {len(masks)} must be the same as length of expiries {T}."
with verbose.write_t(f"\r{message()}Finding {'linear' if self.smoothness is None else f'{self.smoothness:.2f}-smooth'} arbitrage-free prices {'between bid/ask ' if hard_fit else ''}across {sum( len(xt) for xt in xstrikes )} strikes... ", end='') as tme:
variables = []
constraints = []
objective = 0.
xdensities = []
CXs = []
avhs_sum = 0.
avhs_count = 0
for t in range(T):
mask_t = masks[t] if masks is not None and masks[t] is not None else slice(None,None)
avhs_sum += np.sum( asks[t][mask_t] - bids[t][mask_t] )/2.
avhs_count += len(asks[t][mask_t])
average_hspread = avhs_sum / max(1, avhs_count)
eff_mid_fit_weight = self.mid_fit_weight if not hard_fit else 1.
del avhs_sum, avhs_count
for t in range(T):
expiry = expiries[t]
strikes_t = strikes[t]
xstrikes_t = xstrikes[t]
mask_t = masks[t] if not masks is None else slice(None,None)
ask_t = asks[t][mask_t]
bid_t = bids[t][mask_t]
xguess_t = np.clip( xguessed_p[t], 0., None ) if not xguessed_p is None else None
assert xguess_t is None or xguess_t.shape == xstrikes_t.shape, f"Internal error: length of xguessed_p[{t}] {len(xguess_t)} must be the same as length of xstrikes[{t}] {len(xstrikes_t)}."
# density
q_t = cp.Variable(len(xstrikes_t), bounds=[0.,1.])#nonneg=True)
assert q_t.shape == (len(xstrikes_t),), f"Internal error: shape of q_t {q_t.shape} must be the same as length of xstrikes_t {len(xstrikes_t)}."
assert xguess_t is None or q_t.shape == xguess_t.shape, f"Internal error: shape of q_t {q_t.shape} must be the same as shape of xguess_t {xguess_t.shape}."
variables += [(q_t, xguess_t)]
constraints += [ cp.sum( q_t ) == 1.,
q_t @ xstrikes_t == 1., # martingale condition
q_t <= 1.
]
# martingale
Cm = np.maximum( 0., xstrikes_t[None,:] - xstrikes_t[1:-1,None] ) @ q_t # model strikes - do not use "call"
if t>0:
assert xstrikes[t-1].shape == xdensities[t-1].shape, f"Internal error {message()}: shape mismatch between previous and current mstrikes, {xstrikes[t-1].shape} vs {xdensities[t-1].shape}"
Cm1 = np.maximum( 0., xstrikes[t-1][None,:] - xstrikes_t[1:-1,None] ) @ xdensities[t-1]
constraints += [ Cm >= Cm1 ]
else:
Cm1 = np.maximum( 0., 1. - xstrikes_t[1:-1] ) # model strikes, previous density is dirac at 1
# dlv constaints
# dC = 1/2 * dlvsq * x^2 * dt * K^2 / (1/2 (k^+ - k^-)) q
box = xstrikes_t[1:-1]**2 * ( expiries[t] - ( expiries[t-1] if t>0 else 0. ) ) / ( xstrikes_t[2:] - xstrikes_t[:-2])
if not self.min_dlv is None:
constraints += [ Cm - Cm1 >= cp.multiply( box * (self.min_dlv**2), q_t[1:-1] ) ]
if not self.max_dlv is None:
constraints += [ Cm - Cm1 <= cp.multiply( box * (self.max_dlv**2), q_t[1:-1] ) ]
del box, Cm1
# fit
def call( spots, strikes ):
if force_linear or self.smoothness is None or self.smoothness == 0. or expiry < ONE_MINUTE:
return np.maximum( 0., spots - strikes )
expiry_ = np.clip( expiry, expiries[0], expiries[-1] )
vol = math.sqrt( self.smoothness * np.interp( expiry_, expiries, variances ) / expiry_ )
assert np.isfinite(vol), f"Internal error: {message()}infinite vol"
if vol == 0.:
return np.maximum( 0., spots - strikes )
return bs.price( k=strikes/spots, vol=vol, sqrtT=math.sqrt(expiry), eps=self._eps_price )*spots
CX = call( spots=xstrikes_t[None,:], strikes=strikes_t[:,None] )
fit_mid = CX[mask_t,:] @ q_t # market strikes
half_spread = (ask_t - bid_t)/2.
mid_t = (bid_t + ask_t)/2.
whalf_spread = np.maximum( half_spread, self.min_half_spread ) / average_hspread
if hard_fit:
constraints += [ fit_mid <= ask_t - half_spread * (1.-self._eps_no_arb),
fit_mid >= bid_t + half_spread * (1.-self._eps_no_arb) ]
# slack variable version *should* be better..?
guess_fit = CX[mask_t,:] @ xguess_t if not xguess_t is None else None
if eff_mid_fit_weight < 1.:
e_init = np.clip( np.maximum( guess_fit - ask_t, bid_t - guess_fit ) / whalf_spread, 0., None) if guess_fit is not None else None
e_box = cp.Variable(len(bid_t), bounds=[0.,1E6])#nonneg=True)
assert e_init is None or e_box.shape == e_init.shape, f"Internal error: shape of e_box {e_box.shape} must be the same as shape of e_init {e_init.shape}."
variables += [ (e_box, None) ]
constraints += [
fit_mid <= ask_t - half_spread * (1.-self._eps_no_arb) + cp.multiply(whalf_spread, e_box),
fit_mid >= bid_t + half_spread * (1.-self._eps_no_arb) - cp.multiply(whalf_spread, e_box)
]
objective += (1. - eff_mid_fit_weight) * cp.sum(e_box)
del e_box
if eff_mid_fit_weight > 0.:
z_init = np.clip( np.max( np.abs( guess_fit - mid_t ) / whalf_spread ), 0., None) if guess_fit is not None else None
z_mid = cp.Variable(bounds=[0.,1E6])#nonneg=True)
assert z_init is None or z_mid.shape == z_init.shape, f"Internal error: shape of z_mid {z_mid.shape} must be the same as shape of z_init {z_init.shape}."
variables += [ (z_mid, None) ]
constraints += [
fit_mid - mid_t <= cp.multiply(whalf_spread, z_mid),
mid_t - fit_mid <= cp.multiply(whalf_spread, z_mid),
]
objective += eff_mid_fit_weight * z_mid
del z_mid
xdensities.append(q_t)
CXs.append(CX)
success, status, fell_back = SanosLPSolver(timeout_seconds=self.timeout_seconds,
_debug_verbose_cvxpy=self._debug_verbose_cvxpy)(
objective=objective,
variables=variables,
constraints=constraints,
message_context=lambda : message(False),
verbose=verbose )
if fell_back:
verbose.write(lambda : f"had to fall back to a slower solver... ", head=False, end='' )
if not success:
if status in ["infeasible", "infeasible_inaccurate"]:
if hard_fit:
return None
raise RuntimeError(f"{message()}Cannot find no-arbitrage surface mids: problem infeasible??")
if status in ["unbounded", "unbounded_inaccurate", "infeasible_or_unbounded"]:
raise RuntimeError(f"{message()}Cannot find no-arbitrage surface mids: problem unbounded? This should not happen. Please contact the authors with an example.")
if status == "user_limit":
raise RuntimeError(f"{message()}Cannot find no-arbitrage surface mids: reached timeout {self.timeout_seconds} and/or iteration limit.")
raise RuntimeError(f"{message()}Cannot find no-arbitrage surface mids: solver returned status '{status}'.")
if status != "optimal":
verbose.write(lambda : f"solver returned '{status}' -- proceeding anyway... ", head=False, end='' )
# compute call prices for the original strikes
fits = []
xdensities = [ q.value for q in xdensities ]
for t in range(T):
expiry = expiries[t]
strikes_t = strikes[t]
xstrikes_t = xstrikes[t]
CX = CXs[t]
q_t = xdensities[t]
if np.min(q_t) < -self._eps_density or abs( np.sum( q_t )-1. ) > self._eps_dens_1 or abs( q_t@xstrikes_t - 1. ) > self._eps_dens_1:
warnings.warn(f"{message()}Warning: no-arbitrage density does not satisfy basic properties after optimization: min {q_t.min():.3e}, sum {q_t.sum():.6f}, integral {q_t@xstrikes_t:.6f} -- attempting fix.")
q_t = warn_and_fix_density( xstrikes_t, q_t, eps=self._eps_density, message = lambda : f"{message()}expiry {expiry*DAYS_PER_YEAR:.2f}d: surface density after optimization" )
# zero out some density
floor = max( self.min_density / float(len(q_t)), self._eps_density ) if IMPOSE_FLOOR_ON_DENSITY else 0.
is_zero = q_t < floor
if np.any(is_zero):
q_t[is_zero] = 0.
q_t = warn_and_fix_density( xstrikes_t, q_t, eps=self._eps_density, message = lambda : f"{message()} expiry {expiry*DAYS_PER_YEAR:.2f}d: surface density" )
xdensities[t] = q_t
fit_t = CX @ q_t
assert fit_t.shape == strikes_t.shape, f"Internal error: {message()}Shape of fit_t {fit_t.shape} does not match shape of strikes_t {strikes_t.shape}"
fits.append( fit_t )
verbose.write(lambda : f"done. This took {tme}.", head=False )
# make numpy
return PrettyObject(
xdensities = xdensities,
fits = fits,
CXs = CXs)
[docs]
def expiry_density(self, *,
expiry : float,
variance : float,
strikes_t : np.ndarray,
bid_t : np.ndarray,
ask_t : np.ndarray,
mask_t : np.ndarray|None,
xstrikes_t : np.ndarray,
force_linear : bool = False,
message_context : str|callable|None = None,
verbose : Context = Context.all,
end : str = "\n") -> tuple[np.ndarray, np.ndarray]:
"""
Fit a single expiry given market and model strikes.
This function will first attempt a soft fit. If that soft fit is within bid/ask for all ``strikes_t``
the function returns. If this fit is not within bid/ask, then the
function attempts a hard fit in the soft fit starting point. It successful, it returns
the hard fit. If not successful it returns the soft fit, which is not all within bid/ask.
Parameters
-----------
expiry: float
Time to expiry in years.
variance : float | None
Variance for the expiry, used only if ``self.smoothness`` is not None.
If ``variance`` is zero or ``None``, the function attempts a linear fit.
strikes_t : np.ndarray
Strikes for the expiry.
bid_t : np.ndarray
Bid prices for the strikes.
ask_t : np.ndarray
Ask prices for the strikes.
xstrikes_t : np.ndarray
Strikes for the density grid. The first and last point should be outside the range of strikes_t, and the rest should include all strikes_t.
force_linear : boo,l, default ``False``
If ``True``, the function attempts a linear fit even if ``self.smoothness`` is not None or the variance is positive.
message_context : str|callable|None, default ``None``
Optional context for error and debug messages. If a string is provided, it is used as a prefix for all messages.
If a callable is provided, it is called to generate the message prefix at each call site.
verbose : Context, default ``Context.all``
Verbosity context for logging messages.
end : str, default ``"\n"``
For ``verbose`` output: how to end the output. Using ``end=''`` will surpess a line feed.
Returns
-------
Fit : ``tuple[np.ndarray, np.ndarray, bool]``
Returns a tuple with:
* ``xdensity``: the fitted density on the xstrikes_t grid
* ``fit``: the fitted call prices on the strikes_t grid
"""
def message(with_cols : bool = True) -> str:
if message_context is None:
return ""
if callable(message_context):
msg = message_context()
else:
msg = str(message_context)
if msg == "":
return ""
msg = msg[0].upper() + msg[1:]
if not with_cols:
return msg
return msg + ": "
assert expiry > 0, f"Internal error {message(False)}: expiry must be positive, but found {expiry}"
assert variance >= 0, f"Internal error {message(False)}: variance must be non-negative, but found {variance}"
assert strikes_t.shape == bid_t.shape == ask_t.shape, f"Internal error{message(False)}: strikes_t, bid_t and ask_t must all have the same shape, but found {strikes_t.shape}, {bid_t.shape} and {ask_t.shape}"
# find fit between bid/ask without hard constraints first
# -------------------------------------------------------
tme = verbose.timer()
# soft fit first
# --------------
expiries = np.array([expiry], dtype=np.float64)
variances = np.array([variance], dtype=np.float64) if not variance is None else None
msk_t = mask_t if not mask_t is None else slice(None,None)
r = self.surface_densities(
expiries = expiries,
variances = variances,
strikes = [strikes_t],
asks = [ask_t],
bids = [bid_t],
masks = [mask_t] if not mask_t is None else None,
xstrikes = [xstrikes_t],
xguessed_p = None,
hard_fit = False,
force_linear = force_linear,
message_context = lambda : message(False),
verbose = Context.quiet
)
assert not r is None, "Internal error"
xdensity = r.xdensities[0]
fit = r.fits[0]
does_fit = (fit > bid_t - self._eps_price) & (fit < ask_t + self._eps_price)
does_fit_msk = does_fit[msk_t]
smoothness = self.smoothness if not force_linear else None
if np.all(does_fit_msk):
verbose.write(lambda : f"\r{message()}Soft {'linear' if smoothness is None else f'{smoothness:.2f}-smooth'} per-expiry no-arbitrage fit succeeded within bid/ask. This took {tme}.", end=end)
return xdensity, fit
# hard fit
# --------
r = self.surface_densities(
expiries = expiries,
variances = variances,
strikes = [strikes_t],
asks = [ask_t],
bids = [bid_t],
masks = [mask_t] if not mask_t is None else None,
xstrikes = [xstrikes_t],
xguessed_p = [xdensity],
hard_fit = True,
force_linear = force_linear,
message_context = lambda : message(False),
verbose = verbose
)
if not r is None:
fit = r.fits[0]
does_fit = (fit > bid_t - self._eps_price) & (fit < ask_t + self._eps_price)
does_fit_msk = does_fit[msk_t]
assert np.all( does_fit_msk ), "Internal error: hard fit did not fit within bid/ask??"
verbose.write(lambda : f"\r{message()}Hard {'linear' if smoothness is None else f'{smoothness:.2f}-smooth'} per-expiry no-arbitrage fit succeeded within bid/ask. This took {tme}.", end=end)
return r.xdensities[0], r.fits[0]
verbose.write(lambda : f"\r{message()}Could not hard fit with {'linear' if smoothness is None else f'{smoothness:.2f}-smooth'} per-expiry no-arbitrage -- using masked soft fit instead which is outside bid/ask for {np.sum(~does_fit_msk)} of {len(does_fit_msk)} options. This took {tme}.")
return xdensity, fit
[docs]
def make_extended_strikes( self,
strikes: np.ndarray,
bids: np.ndarray,
asks: np.ndarray, *,
prev_lower : float|None = None,
prev_upper : float|None = None,
message_context : str|None = None ) -> tuple[np.ndarray, np.ndarray, str|None]:
r"""
Utility function to generate an increasing range of extended strikes such that
outside points are sufficiently far away, c.f. https://arxiv.org/abs/2601.11209 remark 2.7.
Essentially, given strikes $K_1,\ldots,K_{n-1}$ and mid-prices $C_1,\ldots,C_n$, we want to find $K_0 < K_1$ and $K_{n+1} > K_n$
with call price values equal to intrinsic value such that the boundary conditions
$-1 < \frac{C_1 - C_0}{K_1 - K_0}$ and $\frac{C_{n+1} - C_n}{K_{n+1} - K_n} < 0$ are satisfied.
**Lower strike**
Let $dC_1 = \frac{C_2 - C_1}{K_2 - K_1}$ be the slope of the line connecting the point $(K_1,C_1)$ to the point $(K_2,C_2)$.
Then we want to find $K_0$ with $C_0 = 1 - K_0$ such that:
$$
-1 <= \frac{ C_1 - C_0 }{ K_1 - K_0 } \leq dC_1 \ .
$$
If $dC_1$ is not greater than -1, we drop $K_1$ and proceed. Once $dC_1$ is greater than -1, we can chose any strike below
$$
K_0 \leq \frac{1 - C_1 + K_1 dC_1}{1 + dC_1}
$$
which is positive. If $K_0$ is non-positive, we will drop $K_1$ from the strike list and proceed to the next strike.
The actual strike used is $K'_0 := \max\{ K_0/2, K_0 - 2 ( K_1 - K_0 ) \}$.
The new lower strike will thenn be the minimum of above calculation and ``prev_lower``, if provided.
**Upper strike**
Let $dC_{n-1} := \frac{C_n - C_{n-1}}{K_n - K_{n-1}}$ be the slope of the line connecting the point $(K_{n-1},C_{n-1})$ to the point $(K_n,C_n)$.
Then we want to find $K_n$ with $C_n = 0$ such that
$$
dC_{n-1} \leq \frac{ C_{n+1} - C_n }{ K_{n+1} - K_n } \leq 0 \.
$$
If $dC_{n-1} >= 0$ we drop $K_n$. Once $dC_{n-1}$ is negative, we can chose any strike above
$$
K_{n+1} \geq K_n - \frac{C_n}{dC_{n-1}} \ .
$$
The actual strike used is $K'_{n+1} := K_n + 2 ( K_{n+1} - K_n )$.
The new upper strike will then be the maximum of above calculation and ``prev_upper``, if provided.
Parameters
----------
strikes : np.ndarray
Array of strikes for the expiry, must be strictly increasing and positive.
bids : np.ndarray
Array of bid prices for the expiry, must have the same shape as strikes.
asks : np.ndarray
Array of ask prices for the expiry, must have the same shape as strikes and be strictly greater than bids.
prev_lower : float | None, default ``None``
Previous lower strike from the previous expiry, if any. The new lower strike will be the minimum of the calculated lower strike and this value, if provided.
prev_upper : float | None, default ``None``
Previous upper strike from the previous expiry, if any. The new upper strike will be the maximum of the calculated upper strike and this value, if provided.
message_context : callable | None, default ``None``
Optional callable that returns a string to be used as a header in error messages.
This can be used to provide context such as the expiry time and other relevant information.
Returns
-------
Extended strikes: tuple[np.ndarray, np.ndarray, str|None]
A tuple containing the extended ``xstrikes``, the corresponding ``mask`` to map from input strikes to extended strikes, and an optional ``notice`` to be printed to the user
in verbose mode.
The relation between input strikes, mask, and output "xstrikes" is
```
xstrikes[1:-1] == strikes[mask]
```
"""
def message(with_cols : bool = True) -> str:
if message_context is None:
return ""
if callable(message_context):
msg = message_context()
else:
msg = str(message_context)
if msg == "":
return ""
msg = msg[0].upper() + msg[1:]
if not with_cols:
return msg
return msg + ": "
input_strikes_t = strikes
input_bid_t = bids
input_ask_t = asks
notice = None
xstrikes_t = input_strikes_t
bid_t = input_bid_t
ask_t = input_ask_t
mid_t = (input_bid_t + input_ask_t)/2.
mask_t = np.ones(input_strikes_t.shape[0], dtype=bool)
verify_inp( len(xstrikes_t) >= self.min_strikes_per_expiry, f"{message()}At least {self.min_strikes_per_expiry} strikes are required for this expiry, but found {len(xstrikes_t)}.")
verify_inp( xstrikes_t.shape == bid_t.shape == ask_t.shape, lambda : f"{message()}Expected strikes, bids and asks to have the same shape for expiry, found {xstrikes_t.shape}, {bid_t.shape}, {ask_t.shape}")
verify_inp( np.all( np.diff(xstrikes_t) > 0 ), lambda : f"{message()}Strikes must be strictly increasing; found minimal difference {np.diff(xstrikes_t).min():.3f}" )
verify_inp( xstrikes_t[0] > 0., lambda : f"{message()}Lowest strike must be positive; found {xstrikes_t[0]:.3f}" )
verify_inp( np.all( ask_t - bid_t >0. ), lambda : f"{message()}All ask prices must be greater than bid prices.")
# extend strike range with boundary strikes
# -----------------------------------------
# At K_0 and K_n call prices are intrinsic
# -1 <= { C_1 - C_0 }/{ K_1 - K_0 } <= dC_1
# Assuming -1 < dC_1 and noting that K_0 < K_1
# { 1 - K_0 - C_1) >= dC_1 (K_0 - K_1) =>
# 1 - C_1 + K_1 dC_1 >= K_0 (1 + dC_1) =>
# K_0 <= (1 - C_1 + K_1 dC_1)/(1 + dC_1)
#
# Assume that this is violated:
# Buy the 0 strike for C_0 (the ask)
assert self.min_dC_eps > 0., f"Internal error: min_dC_eps {self.min_dC_eps} must be positive."
total_input = len(xstrikes_t)
N = xstrikes_t.shape[0]
lower_k = None
for ileft in range(N-2):
C_0 = mid_t[ileft]
C_1 = mid_t[ileft+1]
K_0 = xstrikes_t[ileft]
K_1 = xstrikes_t[ileft+1]
dC_0 = ( C_1 - C_0 )/( K_1 - K_0 )
if dC_0+1. > self.min_dC_eps:
lower_k = (1 - C_1 + K_1 * dC_0)/(1 + dC_0)
if lower_k > 0. and lower_k < K_1:
break
lower_k = None
if ileft == N-2 or lower_k is None:
raise RuntimeError(f"{message()}Could not find consistent initial strike for this expiry.")
upper_k = None
for iright in range(N-2,ileft-1,-1):
C_Nm1 = mid_t[iright]
C_N = mid_t[iright+1]
K_Nm1 = xstrikes_t[iright]
K_N = xstrikes_t[iright+1]
dC_Nm1 = ( C_Nm1 - C_N )/( K_Nm1 - K_N )
if dC_Nm1 < -self.min_dC_eps:
upper_k = K_Nm1 - C_Nm1/dC_Nm1
if upper_k > K_Nm1:
break
upper_k = None
if iright <= ileft or upper_k is None:
raise RuntimeError(f"{message()}Could not find consistent terminal strike for this expiry.")
if ileft > 0 or iright < N-2:
slc = slice( ileft, iright+1 )
xstrikes_t = xstrikes_t[slc]
bid_t = bid_t[slc]
ask_t = ask_t[slc]
mask_t[:] = False
mask_t[slc] = True
assert np.all( xstrikes_t == input_strikes_t[mask_t] ), "Mask error...(2) ?"
if len(xstrikes_t) < total_input:
removed_count = total_input - len(xstrikes_t)
notice = lambda removed_count=removed_count, total_input=total_input: f"removed {removed_count} out of {total_input} strikes to ensure -1 < dC/dK[0] and dC/dK[-1] < 0"
dK0 = K_1 - lower_k
lower_k = max( lower_k/2., K_1 - dK0*2. )
dkN = upper_k - K_Nm1
upper_k = K_Nm1 + dkN*2.
if lower_k >= 1.:
lower_k = 1.-(xstrikes_t[1]-xstrikes_t[0])
if upper_k <= 1.:
upper_k = 1.+(xstrikes_t[1]-xstrikes_t[0])
# previous step?
lower_k = min(lower_k, prev_lower) if not prev_lower is None else lower_k
upper_k = max(upper_k, prev_upper) if not prev_upper is None else upper_k
xstrikes_t = np.concatenate(([lower_k], xstrikes_t, [upper_k]))
assert np.all( bid_t == input_bid_t[mask_t] ), "Mask error...?"
assert np.all( ask_t == input_ask_t[mask_t] ), "Mask error...?"
assert np.all( xstrikes_t[1:-1] == input_strikes_t[mask_t] ), "Mask error...?"
return xstrikes_t, mask_t, notice
[docs]
@staticmethod
def atm_volatilities(
expiries : np.ndarray,
strikes : list[np.ndarray]|np.ndarray,
vols : list[np.ndarray]|np.ndarray,
fwdvol_min : float = 0.01
) -> np.ndarray:
"""
Computes ATM vols using linear interpolation in volatilities, with a floor of ``fwdvol_min``.
The forward volatilities will be floored by ``fwdvol_min``.
Parameters
----------
expiries : :class:`numpy.ndarray`
1D array of expiries of length ``T``.
strikes : list[:class:`numpy.ndarray`]|:class:`numpy.ndarray`
List of ``T`` 1D arrays of strikes or a 2D array of strikes with shape ``(T,N)``.
vols : list[:class:`numpy.ndarray`]|:class:`numpy.ndarray`
List of ``T`` 1D arrays of volatilities or a 2D array of volatilities with shape ``(T,N)``.
For SANOS it is recommended to use ask volatilities.
fwdvol_min : float, default 0.01
Minimum forward volatility.
Returns
-------
Vols: :class:`numpy.ndarray`
1D array of ``T`` volatilities for each expiry, with forward volatility floored by ``fwdvol_min``.
"""
verify_inp( len(expiries.shape) == 1, lambda: f"'expiries' must be a 1D array, found shape {expiries.shape}" )
verify_inp( expiries[0] > 0., lambda: f"First expiry must be positive, found {expiries[0]}" )
verify_inp( np.min( np.diff(expiries)) > 0., lambda: "Expiries must be increasing")
verify_inp( isinstance( fwdvol_min, (int,float) ) and fwdvol_min >= 0., lambda: f"'fwdvol_min' must be a non-negative number, found {fwdvol_min}" )
T = expiries.shape[0]
if hasattr(strikes,"shape"):
verify_inp( len(strikes.shape) == 2, lambda: f"'strikes' must be a 2D array if it has a shape, found shape {strikes.shape}" )
verify_inp( strikes.shape[0] == T, lambda: f"First dimension of 'strikes' must match length of 'expiries', found {strikes.shape[0]} and {T}" )
strikes = [ strikes[t] for t in range(T) ]
verify_inp( len(strikes) == T, lambda: f"Expected {T} strike slices, found {len(strikes)}" )
if hasattr(vols,"shape"):
verify_inp( len(vols.shape) == 2, lambda: f"'vols' must be a 2D array if it has a shape, found shape {vols.shape}" )
verify_inp( vols.shape[0] == T, lambda: f"First dimension of 'vols' must match length of 'expiries', found {vols.shape[0]} and {T}" )
vols = [ vols[t] for t in range(T) ]
verify_inp( len(vols) == T, lambda: f"Expected {T} volatility slices, found {len(vols)}" )
total_vars = []
for t in range(T):
strikes_t = strikes[t]
vols_t = vols[t]
if np.min( np.diff(strikes_t) ) < 0.:
ixs = np.argsort(strikes_t)
strikes_t = strikes_t[ixs]
vols_t = vols_t[ixs]
vol = max( np.interp( 1., strikes_t, vols_t ), fwdvol_min )
var = vol**2 * expiries[t]
if t>0:
var = max( var, total_vars[-1] + fwdvol_min**2 * (expiries[t]-expiries[t-1]) )
total_vars.append( var )
return np.sqrt( np.array(total_vars, dtype=np.float64) / expiries.astype(np.float64, copy=False) )