Four moves, roughly in order of payoff. Together they turn a heavy daily job into a few hundred bytes on most days.
1. Take the bulk file, not the loop. If you want the whole dataset, download the prepared file rather than paginating the API a few thousand times. It is one clean transfer instead of thousands of requests, and it does not touch your rate limit. Where several formats are offered they are built from the same run and carry identical row counts, so pick the one your stack actually reads - Parquet is smallest and fastest to load, but a compressed CSV that opens in your tooling beats a Parquet you cannot read. You are not getting a lesser copy.
2. Read only the slice you need, without downloading the file. This is the cleverest thing anyone does with these APIs and almost nobody knows it exists. If the download endpoint supports HTTP range requests, tools like DuckDB and pandas can reach into a remote Parquet file and pull just the columns and row groups your query touches. Measured: one caller moved about 9 MB instead of the full ~170 MB for the same answer.
3. Ask whether anything changed before you download. A HEAD request checks size and availability without moving the body, and there are two cheaper tricks on top. If a manifest endpoint returns an ETag, send it back as If-None-Match and a 304 tells you nothing changed - a few hundred bytes instead of the whole file. Data endpoints often honour If-Modified-Since the same way. When the source publishes once a day, a job that asks first skips most of its transfers.
4. Point your agent at the API catalog first. A machine-readable catalog describes every endpoint, parameter, batch limit and expected file size. What that looks like in the logs is a hit on the catalog followed immediately by a correctly shaped call - right parameter names, sensible window, no fumbling. If you are pointing an assistant at an API, give it the catalog URL and let it work the rest out rather than hand-writing the calls yourself.
One more habit worth building in: after a large download, compare the bytes on disk against the declared Content-Length, and retry on mismatch. Big transfers cross your connection, your cloud provider, a CDN and the origin storage, and any of them can drop - so a partial file is worth detecting rather than trusting. Batching many identifiers into one call is the companion move: https://www.tigzig.com/agents-faq/do-i-need-one-api-call-per-item-or-can-i-batch. Full API list: https://www.tigzig.com/apis.