Built and run by one person.
Range Requests on the MFPRO NAV File Are Fixed. DuckDB and Polars Queries No Longer Get Refused Midway.
Top•MFPRO - Mutual Fund Analytics•TREMOR - Macro Stress Signals•Quants Agent

Range Requests on the MFPRO NAV File Are Fixed. DuckDB and Polars Queries No Longer Get Refused Midway.

Published: August 7, 2026

Related toolsMFPRO - Mutual Fund Analytics•TREMOR - Macro Stress Signals•Quants Agent

A number of you have been running duckdb or polars queries against the tigzig MFPRO NAV file over the past couple of weeks and getting blocked midway through, and that is now fixed.

The reason turned out to be simple. One analytical query fires off close to three hundred separate byte range requests in a few seconds, and the rate limiting in front of the API was counting those range reads in the same bucket as ordinary whole file requests, so a legitimate query looked like a burst and got refused halfway through.

Range requests now have their own pathway and their own budget, so you are free to run queries against my server without that happening again. The pathway supports up to 3,000 requests per minute and 25,000 per day per (public) IP. And response headers come with remaining limit details.

The file itself is about 175 MB with close to 37 million NAV records, and it is still there to download whole if you would rather.

The API also continues to support single and batch NAV pulls, search across scheme fields, and the newer snapshot endpoint, which gives the latest NAV for every scheme in one file instead of the full history just for today's number.

Links:

Tips

Library comparison

duckdb and polars are the fastest. Polars slightly faster than duckdb in our tests. pyarrow and pandas (with fsspec) work fine but move roughly 10x more than duckdb/polars for the same query. pandas given a bare URL is the one to avoid outright.

The duckdb snippet

INSTALL httpfs; LOAD httpfs;
SELECT date, nav FROM read_parquet('https://api.tigzig.com/mf/v1/download?format=parquet')
WHERE scheme_code = 122639 ORDER BY date DESC;

About 3,250 rows, full history for one fund since 2013, about 1.3 MB moved, a few seconds.

The filter column

The file is sorted by scheme_code. WHERE on scheme_code costs about 5 requests. A WHERE on date, or no filter at all, costs about 300, because every row group has to be opened.

The pandas gotcha

Don't pass a bare url.

# slow: 372 MB moved, 37 seconds, for 3,245 rows
df = pd.read_parquet(URL, columns=[...], filters=[...])

# fast: 17 MB, 3.5 seconds, same rows
import fsspec
h = fsspec.filesystem("http").open(URL)
df = pd.read_parquet(h, columns=[...], filters=[...])

Query vs download

Full file download: about 36 seconds. A scheme_code-filtered query: a few seconds. An unfiltered scan or a name search: worse than just downloading the whole file. It all depends on the specific use case and your constraints .. rough rule, filter on scheme_code and query remotely, anything else, just download it or use the regular API... unless you have a specific reason to do a range request. Based on the volume and variety of requests that I see on the backend, I might optimize the parquet further.

JavaScript / Node options

Doesn't have to be Python. Sharing below two options which I tested lightly only ..might be more ...