Financial markets are a blend of underlying trends, cyclical
movements, and high-frequency noise. Traditional filters often use fixed
parameters, struggling to distinguish between these components as market
character shifts. This article explores an advanced technique: an
Adaptive Low-Pass Fourier Filter. The strategy aims to
“listen” to the market’s current volatility (σ) and dynamically adjust a
Fourier filter’s passband. In low volatility, it seeks a clearer,
smoother trend by using a narrow passband (stronger filtering). In high
volatility, it employs a wider passband, allowing more price action
through to remain responsive. We’ll dissect the methodology, its Python
implementation for BTC-USD, and discuss the insights from a
systematic backtest.
Fourier Transform (DFT/FFT): At the heart of the strategy is the Discrete Fourier Transform (typically implemented as the Fast Fourier Transform, FFT). This mathematical tool decomposes a time series (like a window of closing prices) into its constituent sine and cosine waves of different frequencies and amplitudes. This allows us to see how much “energy” or variance is present at each frequency.
Volatility (σ) as an Adaptive Signal: We calculate the historical volatility of the asset (e.g., the standard deviation of daily returns over a 20-day window). This σ becomes our guide for how “nervous” or “calm” the market is.
Adaptive Low-Pass Filter Cutoff: The “passband” of a low-pass filter determines which frequencies are allowed through and which are attenuated (filtered out).
Filtering in the Frequency Domain: For each rolling window of price data:
Trading Signals: A simple price crossover system is used:
Close >
previous day’s Filtered_Price_Fourier. Enter long at
current day’s open.Close <
previous day’s Filtered_Price_Fourier. Enter short at
current day’s open.Let’s look at key code sections that implement this adaptive filtering.
Snippet 1: Calculating Volatility and the Adaptive FFT Cutoff Index
First, we calculate historical volatility and normalize it. This normalized volatility then determines the cutoff index for our Fourier filter.
# --- Parameters from the script for BTC-USD ---
# vol_window = 20
# vol_norm_window = 100 # For normalizing volatility
# dft_window = 64 # For the FFT
# cutoff_freq_idx_min = 2 # Min FFT bin index (narrow passband)
# cutoff_freq_idx_max = dft_window // 4 # Max FFT bin index (wider passband)
# --- Column Names ---
# volatility_col = f"Volatility_{vol_window}d"
# normalized_vol_col = f"Normalized_Vol_{vol_norm_window}d"
# adaptive_cutoff_idx_col = "Adaptive_Cutoff_Idx"
# --- Indicator Calculation (within pandas DataFrame 'df') ---
# Assume df[daily_return_col] is pre-calculated
# 1. Historical Volatility
df[volatility_col] = df[daily_return_col].rolling(window=vol_window).std()
# 2. Normalize Volatility (0-1 range)
rolling_min_vol = df[volatility_col].rolling(window=vol_norm_window).min()
rolling_max_vol = df[volatility_col].rolling(window=vol_norm_window).max()
range_vol = rolling_max_vol - rolling_min_vol
df[normalized_vol_col] = ((df[volatility_col] - rolling_min_vol) / range_vol.replace(0, np.nan)).fillna(0.5).clip(0,1)
# 3. Adaptive Cutoff Frequency Index (based on lagged normalized volatility)
# Low Normalized Vol (0, i.e., low actual vol) -> cutoff_freq_idx_min (narrower passband)
# High Normalized Vol (1, i.e., high actual vol) -> cutoff_freq_idx_max (wider passband)
df[adaptive_cutoff_idx_col] = cutoff_freq_idx_min + df[normalized_vol_col].shift(1) * (cutoff_freq_idx_max - cutoff_freq_idx_min)
df[adaptive_cutoff_idx_col] = np.round(df[adaptive_cutoff_idx_col]).fillna( (cutoff_freq_idx_min + cutoff_freq_idx_max) / 2 ).astype(int)
max_possible_idx = dft_window // 2 # Max index for rFFT output
df[adaptive_cutoff_idx_col] = np.clip(df[adaptive_cutoff_idx_col], 0, max_possible_idx)The adaptive_cutoff_idx_col determines how many
frequency components (beyond DC) are retained. A smaller index means
more aggressive low-pass filtering.
Snippet 2: Iterative Calculation of the Adaptive Fourier Filtered Price
The filtered price is calculated day by day. For each day, a window of past prices is taken, transformed via FFT, filtered in the frequency domain using the day’s adaptive cutoff, and then transformed back via IFFT. The last point of this filtered window is the output.
# --- Parameters from the script ---
# dft_window = 64
# --- Column Names ---
# filtered_price_fourier_col = "Filtered_Price_Fourier"
# adaptive_cutoff_idx_col = ... (from previous snippet)
# --- Iterative Filter Calculation (within pandas DataFrame 'df') ---
df[filtered_price_fourier_col] = np.nan
print(f"Calculating Adaptive Fourier Filter (DFT window: {dft_window})...")
# Determine where calculations can start
first_valid_cutoff_idx = df[adaptive_cutoff_idx_col].first_valid_index()
if first_valid_cutoff_idx is None:
raise SystemExit("Cannot determine a valid start for adaptive cutoff index.")
start_loc_for_dft_loop = df.index.get_loc(first_valid_cutoff_idx)
# Ensure we have a full dft_window of data before this start_loc
if start_loc_for_dft_loop < dft_window -1:
start_loc_for_dft_loop = dft_window -1
for i_loop in range(start_loc_for_d_loop, len(df)): # Corrected variable name
idx_today = df.index[i_loop]
current_price_window = df['Close'].iloc[i_loop - dft_window + 1 : i_loop + 1].values
if len(current_price_window) != dft_window: # Should have full window
df.loc[idx_today, filtered_price_fourier_col] = df.loc[df.index[i_loop-1], filtered_price_fourier_col] if i_loop > 0 else np.nan
continue
# For price series, detrending is often beneficial before FFT
# price_window_to_fft = signal.detrend(current_price_window)
price_window_to_fft = current_price_window # Using raw prices for simplicity in this example
fft_coeffs = np.fft.rfft(price_window_to_fft) # Real FFT
cutoff_idx_val = df.loc[idx_today, adaptive_cutoff_idx_col]
if pd.isna(cutoff_idx_val):
df.loc[idx_today, filtered_price_fourier_col] = df.loc[df.index[i_loop-1], filtered_price_fourier_col] if i_loop > 0 else np.nan
continue
cutoff_idx = int(cutoff_idx_val)
# Zero out frequencies above the adaptive cutoff
if cutoff_idx + 1 < len(fft_coeffs):
fft_coeffs[cutoff_idx + 1:] = 0 # Low-pass filtering
filtered_segment = np.fft.irfft(fft_coeffs, n=dft_window) # Inverse FFT
# The filtered price for idx_today is the last point of the filtered segment
# If detrending was used, the trend would need to be added back here.
df.loc[idx_today, filtered_price_fourier_col] = filtered_segment[-1]
# Fill any initial NaNs from the loop start
df[filtered_price_fourier_col].fillna(method='ffill', inplace=True)
df[filtered_price_fourier_col].fillna(method='bfill', inplace=True) # Ensure no leading NaNs if possibleThis iterative process creates the
filtered_price_fourier_col, which serves as our dynamic
trendline.
The trading logic is a standard crossover:
filtered_price_fourier_col, a long
position is initiated at the current day’s open. A cross below signals a
short entry. Positions are flipped if an opposite signal occurs.The backtest results for BTC-USD from January 2020 to
December 2024, using the script’s parameters (DFT window 64, Vol window
20, Vol norm window 100, Cutoff FFT indices from 2 to 16 (64/4), ATR SL
2.0x), were presented in the script’s output:
Interpreting Backtest Outputs – A Researcher’s Lens:
When evaluating this strategy, a researcher would focus on:
BTC-USD is
essential. The example output you included in the previous message
(for a different strategy) showed very high outperformance. For this
Spectral strategy, we would need to see its specific output.low_vol -> narrow_passband (low_cutoff_idx)?Key Research Considerations & Caveats:
dft_window and spectrum_nperseg impacts
this.dft_window, vol_window,
vol_norm_window, cutoff_freq_idx_min/max, and
ATR settings. Performance can be highly sensitive to these. The specific
mapping of volatility to cutoff indices is a critical design
choice.calculate_spectral_slope_fn uses
signal.detrend, the main adaptive filter loop in the
provided code applies FFT to current_price_window (raw
prices for simplicity in the example). A more rigorous approach might
detrend each window before FFT and add back an estimate of the trend to
the IFFT output, which is more complex.t (like
volatility) is based on data available up to day t-1. The
script correctly uses df[normalized_vol_col].shift(1) for
this.The Spectral-Slope Adaptive Low-Pass Fourier Filter strategy represents a highly sophisticated and intellectually appealing method for adapting to changing market characteristics. By analyzing the frequency content of price movements and dynamically adjusting filter bandwidth based on market volatility, it aims to achieve a more nuanced and potentially more effective form of trend following or smoothing than fixed-parameter filters.
While the theoretical underpinnings are strong, its practical success hinges on robust implementation of the spectral estimation, careful mapping of volatility to filter parameters, and thorough validation against the risks of overfitting and real-world trading frictions. The Python framework allows for such detailed investigation, paving the way for deeper insights into adaptive signal processing for financial trading.