[output_dir]')\n sys.exit(1)\n xlsx = sys.argv[1]\n out = sys.argv[2] if len(sys.argv) > 2 else None\n extractxlwingscode(xlsx, out)\n`\n\nQuick one-liner\n\nFrom the command line:\n\n`python\npy -c \"\nimport zipfile, xml.etree.ElementTree as ET, json, sys\nz=zipfile.ZipFile(sys.argv[1]); t=ET.parse(z.open('xl/webextensions/webextension1.xml'))\nns={'we':'http://schemas.microsoft.com/office/webextensions/webextension/2010/11'}\n[print(json.loads(p.get('value',''))) for p in t.getroot().findall('.//we:property',ns) if p.get('name')=='main.py']\n\" \"C:\\path\\to\\file.xlsx\"\n`\n\nExample usage\n\nFull extraction:\n\n`\npy extractxlwingscode.py \"C:\\path\\to\\your\\xlwingslitefile.xlsx\"\n`\n\nExtract to specific directory:\n\n`\npy extractxlwingscode.py \"C:\\path\\to\\file.xlsx\" \"C:\\output\\dir\"\n``\n\nNotes\nxlwings Lite stores everything as plain text in XML - fully inspectable\nNo Excel installation needed - pure Python, standard library only\nWorks on Windows, macOS, Linux\n\nxlwings Lite official site: lite.xlwings.org"
},
{
"slug": "fa18de05",
"title": "Mutual Fund Portfolio Analysis with ChatGPT: Merging and analyzing across multiple excel files",
"tags": [
"mutual-funds",
"custom-gpt"
],
"content": "Mutual Fund Portfolio Analysis with ChatGPT: Merging and analyzing across multiple excel files\n\nA very common analysis in the investment world is to track changes in equity portfolio allocations within a scheme over time. This often needs to be conducted monthly, and sometimes even weekly or fortnightly, across varying time periods.\n\nThis type of analysis requires merging multiple Excel files, typically print-formatted and may contain images, as was in this case.\n\nWill ChatGPT be able to do it? With just prompts. In a replicable manner. Even if I provide files for different months / multiple months? That was the specific use case requested.\n\nAmazingly: Yes\n\nHad prepared a prompt for two files. That worked. Asked it to merge another 10 using same logic. And it did. In one shot. 30 sec. And for one of the files, the name format was also different...ChatGPT handled it seamlessly...!\n\nSee for yourself.\n\nPrompt preparation secret: The main prompt might look fancy and intimidating. I did not type out a single line of that. I was voice typing using Google docs when doing for the first time. It was iterative. Used to dump the prompt with uuh, aaah, spellings, and horrible formatting into ChatGPT. At the end just asked ChatGPT to prepare a consolidated prompt that I can use, and it will understand. And voila.. the magic prompt...!\n\nMain Prompt\n\nA very common analysis in the investment world is to track changes in equity portfolio allocations within a scheme over time. This often needs to be conducted monthly, and sometimes even weekly or fortnightly, across varying time periods.\nThis type of analysis requires merging multiple Excel files, typically print-formatted and may contain images, as was in this case.\nWill ChatGPT be able to do it? With just prompts. In a replicable manner. Even if I provide files for different months / multiple months? That was the specific use case requested.\nAmazingly: Yes\nHad prepared a prompt for two files. That worked. Asked it to merge another 10 using same logic. And it did. In one shot. 30 sec. And for one of the files, the name format was also different...ChatGPT handled it seamlessly...!\nSee for yourself.\nPrompt preparation secret: The main prompt might look fancy and intimidating. I did not type out a single line of that. I was voice typing using Google docs when doing for the first time. It was iterative. Used to dump the prompt with uuh, aaah, spellings, and horrible formatting into ChatGPT. At the end just asked ChatGPT to prepare a consolidated prompt that I can use, and it will understand. And voila.. the magic prompt...!\nThis prompt will only work for HDFC Top 100 Monthly Portfolio disclosures. That was the specific use case I was working on. Next, would be trying to generalize it, so that it can work for monthly portfolio disclosure of any scheme of any fund house. And publish as a GPT or a web app so that there is no need to worry about the prompt. And a user can simply upload files and proceed.\nMonthly Portfolio - HDFC Mutual Fund\nWhile doing it first time had to iterate with ChatGPT to get it right. It took me around an hour and a half to iterate and get the main prompt out. And after that it was just copy paste.\nWe need to validate, same as we would do had we processed the data with Excel / Python. For instance, validating the totals against main files. I had also asked ChatGPT to prepare a validation summary to make this a bit easier. And also instructed ChatGPT to apply these validation checks on its end.\nAnd if you would rather do it in Python, then you can ask ChatGPT to provide a base code such that you can pass the file names, time periods etc. as variables....offering flexibility on data engineering side."
},
{
"slug": "fail2ban-server-security-bots-ai-tools",
"title": "Server Meltdown: How Bots Crashed My AI Tools and What I Did About It",
"tags": [
"security",
"infrastructure"
],
"content": "Server Meltdown: How Bots Crashed My AI Tools and What I Did About It\n\nI setup a shiny new shared server to host my shiny new AI tools. Few weeks later the server melted. All my tools went phut.\n\nI had disabled password login, setup SSH keys only. But bots don't know what auth method you're using - they just keep trying. Thousands of login attempts...CPU maxed out...server collapsed.\n\nThat was nearly two years back. I came from a world of analytics, dashboards and ML models - this was like a new universe.\n\nAI (Cursor & ChatGPT that time) helped me understand what is happening and how to fix it.\n\nI had fail2ban running at backend. But settings were too gentle.\n\nfail2ban works like this - somebody tries to login and fails we can put them into an IP Jail. But the jail settings make all the difference.\n\nThe old setting I don't remember now, but in the new one I set maxretry at 5. Means after 5 failed attempts, they get banned. But banned for how long? That's where findtime and bantime come in.\n\nfindtime is the window - I set it at 3600 seconds (one hour). So if you mess up 5 times within that one hour window, you go to jail.\n\nbantime is how long you sit in my jail - I set it at 86400 seconds. That's 24 hours ban. Full day.\n\nWhy 5 attempts and not 1 or 2? Because I might fat-finger SSH attempt myself. Wrong key file, typo when connecting from terminal - those things happen right? I don't want to lock myself out on one mistake. 5 gives me decent margin. But bots hit that limit in seconds anyway.\n\nMy jail stats from last week:\nCurrently banned: 157 IPs\nTotal banned: 1,223 IPs (one week since last restart)\nFailed attempts blocked: 6,082\nServer running fine now.\n\nWas that the only mistake I made? I made 17 more that I recall. Read it here. With the fixes.\n\n2026 Infra Guide Part 3: The 18 Common Security Mistakes and How to Fix Them\n\nToday I run multi level defenses for client tools - OAuth, proxies, rate limits, API keys.\n\nAnd before I release anything to a client, two mandatory checks: Browser F12 -> Console, Network, Application tabs to see what's exposed. Second, I ask Claude Code to do a Security Audit and identify all attack vectors. Almost always, I have missed something. Fix it."
},
{
"slug": "fast-tips-what-is-cors-and-how-to-fix-it",
"title": "Infra Guide for AI Tool Builders - Part 4: CORS in Simple Words: What It Is and How to Fix It",
"tags": [
"infrastructure",
"security"
],
"content": "Infra Guide for AI Tool Builders - Part 4: CORS in Simple Words: What It Is and How to Fix It\r\n\r\nCORS - Cross-Origin Resource Sharing\r\n\r\nIf you are building tools, AI or otherwise, you would have run into CORS issues. If not, you will.\r\n\r\nSay you want to download a file from Google Drive into your browser app or just pull Yahoo Finance data using yfinance (needs Pyodide).\r\n\r\nWhat is CORS?\r\n\r\nWhen you download from a terminal (like using curl or python) - this always works. No restrictions.\r\n\r\nBut inside a browser (JavaScript code running in a web app) - there are restrictions. When your browser code tries to fetch data from another website (like Google Drive), that server must explicitly say \"Yes, browsers are allowed to access this.\" They do this by including a special header in the response: Access-Control-Allow-Origin: *\r\n\r\nThis is CORS - Cross-Origin Resource Sharing. It's a browser security feature.\r\n\r\nWhat's the problem?\r\n\r\nMany services (e.g. Yahoo Finance, GitHub Releases, Google Drive, Dropbox etc) don't include this header. So your browser blocks your JavaScript from reading the response. It says: \"Sorry, this server didn't give me permission to share this data with a browser.\"\r\n\r\nWait - does the data actually reach the browser?\r\n\r\nYes. This is the confusing part. The browser makes the request, the server sends the data back. The data actually arrives at your browser. But before handing it to your JavaScript code, the browser checks for that Access-Control-Allow-Origin header. If it's missing - your JavaScript can't touch the data. The browser's security layer sits between the network and your code.\r\n\r\n\r\nNuance: Sometimes the data doesn't even arrive - Preflight Requests\r\n\r\nEverything above applies to simple requests - basic GET requests that just read data. But for more complex requests - POST with JSON body, requests with custom headers like Authorization, PUT, DELETE - the browser does something different.\r\n\r\nWhat is a preflight request? Before sending the actual request, the browser sends a lightweight OPTIONS request first. It asks the server: \"Would you accept this kind of request from a browser?\" If the server says yes (returns the right CORS headers), the browser sends the real request. If not, it stops right there. The real request never goes out. The data never arrives.\r\n\r\nWhy does the browser send a preflight? It's a safety check. A POST or DELETE can change data on the server - create records, delete data, trigger actions. The browser's logic: \"Before I send something that could modify data on a server that might not be expecting browser requests, let me check first.\" This protects older APIs that were never designed to be called from random websites.\r\n\r\nCan I disable it? No. The browser decides automatically. You cannot tell the browser \"skip the preflight.\" It's not a setting or a flag in your code.\r\n\r\nHow does the browser decide when to send a preflight? Simple rules. The browser skips preflight (sends directly) if ALL of these are true: method is GET, HEAD, or POST; only standard headers (no Authorization, no custom headers); and if POST, content type is only form data or plain text. The moment you break any of these - say you add Content-Type: application/json or an Authorization header - the preflight happens automatically. Most modern API calls use JSON and auth headers, so most calls trigger preflight.\r\n\r\nDoes it slow things down? It's an extra round-trip but it's fast (just headers, no body) - you generally don't notice it.\r\n\r\nThe fix is the same either way - use a proxy. But now you know why sometimes you see that OPTIONS request in your DevTools Network tab before the actual call.\r\n
\r\n\r\nSo where is the data sitting?\r\n\r\nIt's in the browser's network layer. The bytes came in, the HTTP response was parsed. But it's walled off from your JavaScript. Think of it like a customs officer who received your package but won't hand it to you because the paperwork (CORS headers) is missing.\r\n\r\nCan I see it in DevTools?\r\n\r\nYes! This trips people up. Go to DevTools > Network tab, click the request - you can see the full response body with the actual data right there. You can read it. The browser has it.\r\n\r\nBut your fetch() call still throws a CORS error. DevTools has elevated privileges that bypass the same-origin policy. So you're staring at the data in DevTools while your code can't access it. Frustrating.\r\n\r\nWhat's the solution?\r\n\r\nYou add a middleman - also called a Proxy. Instead of fetching directly from the offending API, your browser fetches from a proxy. The proxy fetches the data server-side (which always works - CORS is a browser-only restriction), then sends it to your browser with the proper CORS headers added. Your browser is happy, and you get your data.\r\n\r\nCORS issues are fairly common when building browser-based apps. Knowing how to use a proxy to bypass CORS is an essential skill if you're building deployable apps and data tools.\r\n\r\n\r\nTrap: Don't use mode: 'no-cors' - it doesn't fix anything\r\n\r\nWhen you hit a CORS error, you'll find suggestions online to add mode: 'no-cors' to your fetch call. It looks like a fix. The error goes away. But your data is gone too.\r\n\r\nWhat no-cors actually does: it tells the browser \"I know this will fail CORS, send it anyway but I accept that I can't read the response.\" The browser makes the request, gets the response, and gives you an empty opaque response. No data. No error either. Just nothing.\r\n\r\nIt's useful in very specific cases (like sending analytics pings where you don't care about the response), but for fetching data - it's a trap. The CORS error disappears but so does your data. Use a proxy instead.\r\n
\r\n\r\nHow I handle it - and which one to pick when\r\n\r\nNumerous ways to set up a proxy. I typically end up using one of these 3. Here's the order I think about them.\rCloudflare Workers - The Pure Proxy (Free. No Domain Required)\r\n\r\nThis is the simplest option for pure pass-through. Your browser calls the Cloudflare Worker, the Worker fetches from the actual API, adds CORS headers, sends it back. That's it. No domain needed - you get a free workers.dev subdomain. No server to manage. Free Cloudflare account is enough. You can also add authentication if needed.\r\n\r\nGreat for: any situation where you just need to pass data through without processing it. If the API you're calling works fine but just doesn't have CORS headers - this is the quickest fix. Check the CORS section in xlwings Lite Data Importer - there's a complete text guide there. Copy the guide, paste to your AI coder, AI sets it up for you.\rVercel Serverless Functions - The Mini Backend (Free Tier)\r\n\r\nThis is a very nifty feature. If you have a React app on Vercel, you can create an api/ folder in your project. Any file inside api/ becomes a serverless function - essentially a mini backend endpoint. It runs on Vercel's servers, not in the browser. So no CORS issues because the request goes from Vercel's server (server-to-server), not from the browser.\r\n\r\nIt's like getting a mini FastAPI backend for free - except in JavaScript. You can do fair amount of processing there - not just proxy calls. Anything that can be done in JavaScript, you can do there. Data transformation, API orchestration, auth token handling.\r\n\r\nThe free tier gives you up to 12 serverless functions - more than enough for most apps. If you run into the limit, you can always combine functions. Execution timeout on free tier is currently up to 300 seconds (5 minutes) - so even long-running queries or heavier processing work fine.\r\n\r\nFor local development, you need the Vercel CLI (vercel dev) to run serverless functions locally - they won't work with just npm start since they need Vercel's runtime.\r\n\r\nIn my newer React apps, I've moved all API calls from the frontend to serverless functions. Everything goes through the server side. This adds an extra layer of security (API keys never exposed in browser code) and CORS issues are automatically taken care of. That's my default approach now.\r\n\r\nExample: My IMDb Movie Explorer app. For source code hit 'Docs' on the app site (Press F9 if you don't see top menu).\rFastAPI Backend - For Heavy Python Processing (Not Free)\r\n\r\nThis is not required for all cases. For simple pass-through requests, this is overkill - Cloudflare Workers or Vercel serverless handle that easily.\r\n\r\nWhere this makes sense: when you need serious Python processing that can't be done in JavaScript. Libraries like yfinance, pandas, numpy, scikit-learn - things that only exist in the Python ecosystem. Or when you already have a Python backend doing other things and you just want to add another endpoint.\r\n\r\nExample: Yahoo Finance Data Extractor. A FastAPI backend that runs yfinance. Public and open. Pull Yahoo Finance data from any browser.\r\n\r\nSo which one do I pick?\r\n\r\nTypically: if I'm building a React app, Vercel serverless functions are my first choice by default - everything goes through the server side. If something can't be done there (needs Python), I use my FastAPI backend. For standalone pass-through where I just need to proxy an API call quickly, Cloudflare Workers - doesn't even need a domain.\r\n\r\n\r\nlocalhost:3000 calling localhost:8000 is also cross-origin\r\n\r\nThis catches people during development. Your React app runs on localhost:3000. Your backend runs on localhost:8000. Same machine. Same \"localhost.\" But different ports. To the browser, these are different origins. So yes - you get CORS errors even when both are running on your own laptop.\r\n\r\nThat's why your FastAPI backend needs CORS middleware even during local development. In FastAPI you add CORSMiddleware with allow_origins=[\"*\"] (or specific origins for production). Without it, your local frontend can't talk to your local backend. Same machine, different ports, different origins.\r\n
\r\n\r\nOther Options (haven't used these - worth knowing about)\r\n\r\nThere are other solutions people use. Public CORS proxy services like allorigins.win and corsproxy.io let you just prepend their URL to your target URL - quick for testing but you're routing through someone else's server with no guarantees on uptime or rate limits. Not suitable for production. CORS Anywhere is an open source Node.js proxy you can self-host - popular in tutorials, basically what Cloudflare Workers does but you manage the server. Browser extensions like \"CORS Unblock\" disable CORS checking entirely - fine for local dev, useless for production since your users won't have it installed. Nginx/Caddy reverse proxy can also forward requests and add CORS headers - but if you already have a FastAPI backend, this is redundant. Your backend already handles it.\r\n\r\nBest Way to Use This\r\n\r\nJust copy paste this post to your AI Coder. It will explain the differences and trade-offs and applicability for your specific apps. It can clone the repos and guides if you ask it to - but not really required. This is straightforward stuff for them. They know it blind.\r\n\r\nInfra Guide Series\rPart 1: AI Coder\rPart 2: Deployment & Hosting\rPart 3: The 18 Common Security Mistakes and How to Fix Them"
},
{
"slug": "ff7ee13c",
"title": "Google Colab Data Science Agent vs. Mito-AI Jupyter Copilot. How do they compare? When to use which?",
"tags": [
"ai-coders"
],
"content": "Google Colab Data Science Agent vs. Mito-AI Jupyter Copilot. How do they compare? When to use which?\n\nGoogle Colab recently launched Data Science Agent (DSA)-plans and executes multi-step analysis/modeling in one shot. Meanwhile, Mito-AI Copilot (launched earlier this year) offers a seamless coding experience with Cursor-like chat + built-in data science agent.\n\nSo, which one's the best?\n\nBoth have strengths. I used Colab exclusively, but Mito-AI pulled me back to Jupyter-huge efficiency boost. Plan on using both- depending on the project. Best of both worlds.\n\nGoogle Colab edge\n\nMore powerful agent, code sharing, easy google drive/sheets access, strong processing speeds, free T4-GPU access. Supports in-cell code generation ....but lacks a true copilot with memory.\n\nBut Mito-AI wins big in one area: a seamless copilot experience.\n\nIt auto-sends all data schemas (df structures + sample rows) to AI + retains conversation history for real-time AI awareness-big difference. Smooth vibe/ voice coding (Win+H / Cmd+H). Just pip install mito-ai mitosheet and you're set. In the video, I demo a live voice coding for file processing automation to create a datamart.\n\nRole of Data Science Agent\n\nColab DSA is very powerful ( https://lnkd.in/g3ub_84D), and great for the right projects-especially to run a full multi-step workflow in one shot. But I can't see using it for every project. Many require validation against tally numbers and business logic at each step before moving forward. At the same time, I do foresee cases where I'd prefer to run the entire workflow at one go and refine later-Colab DSA would be my choice.\n\nPricing\n\nColab is free. Mito-AI offers 1 month/500 free chats, then $20/month with unlimited completions and extras. Open-source version available with your own API key.\n\nInsights and Tips\n\nBuild AI Co-Analyst Apps for Analytics& Data Science\n\nExplore 15+ open-source AI analytics apps at tigzig.com-including multi-agent workflows, real-time voice AI, and Python/SQL automation. Connect to any database. Free access, no API key needed. Source codes and build guides included.\n\nAI Advanced Analytics App with Multi Agents(Sequential-LangGraph). https://lnkd.in/g8xPQvb8"
},
{
"slug": "flowise-is-my-goto-platform-for-genai-llm-app-development",
"title": "Flowise is my goto platform for GenAI and LLM apps",
"tags": [
"database-ai",
"custom-gpt"
],
"content": "Flowise is my goto platform for GenAI and LLM apps\r\n\r\nEverything is in one place, with solid functionality, great ease of use, and scalability from small-simple apps to complex & enterprise-level projects.\r\n\r\nHave quite a few live apps on the platform. Love it.\r\n\r\nFascinating interview with Henery Heng, Founder of Flowise AI. Talks about many use cases - including enterprise level implementations involving structured data as well as unstructured data - including querying databases, analytics, real time data.\r\n\r\nOpen Source. Y-Combinator backed. Github Repo has a massive 21K stars.\r\n\r\nHuge functionalities\r\n\r\nEspecially for RAG: integrated with both Langchain and LlamaIndex.\r\n\r\nSuper easy to connect to vector databases. Has quick integrations with 10-15+ vector databases - I typically end up using Pinecone and Vectara.\r\n\r\nConnect to structured data and databases.\r\n\r\nCustom Tool: Easy to make an API call (especially compared to OpenAI's custom GPT).\r\n\r\nPlatform is updated at speed. The new Generic Tool Agent released last week is fantastic - can work across LLM's.\r\n\r\nConnect easily to all major LLM's. Has ready integration with Groq. I use Gemini's Pro / 1.5s free tier for many tasks.\r\n\r\nConnects with Make.com and Zapier for automation workflows - very powerful stuff - numerous examples, tutorials, and guides available on the web.\r\n\r\nI can go on and on...\r\n\r\nNot affiliated with Flowise in any way. Just a regular user of their platform - or rather their forked repo. Just sharing my experiences - highly recommend for developing GenAI / LLM / Automation apps.\r\n\r\nResources\rFlowise Tutorials from Leon Van Zyl - brilliant tutorials\rHas a great discord community\rWhole lot of videos, tutorials, use cases, templates on YouTube, Flowise page, and across the web"
},
{
"slug": "free-production-grade-databases-get-setup-in-minutes-great-for-testing-and-development",
"title": "Free, Production-Grade Databases. Get setup in minutes. Great for testing and development",
"tags": [
"database-ai",
"infrastructure"
],
"content": "Free, Production-Grade Databases. Get setup in minutes. Great for testing and development\n\nNeon: My top choice for AI apps. Database live in <1 sec. Straightforward API for creating / modifying databases and role.\n\nSupabase: Auth integration.\n\nAiven: Most generous free tier (5GB) and supports both Postgres & MySQL.\n\nI use these for creating a 'one-time-use' database for testing to live client projects.\n\nSee live in action on app.tigzig.com -> Database AI & SQL Apps\nDATS-4: temporary on-the-fly databases powered by Neon\nCustom GPT: powered by Supabase with Cricket ODI data\nRealtime Voice AI: powered by Aiven\n\nAll open source. Hit 'Docs' on app site for source code\n\nThe attached image is a quick-glance spec sheet comparing my notes on each\n\nFree Database Providers Comparison"
},
{
"slug": "from-12-second-queries-to-under-1s-optimizing-230-million-row-dashboard",
"title": "From 12 second queries to under 1s: Optimizing a 230 Million Row Dashboard - 14 Bottlenecks I Had to Fix",
"tags": [
"duckdb",
"fastapi",
"infrastructure"
],
"content": "From 12 second queries to under 1s: Optimizing a 230 Million Row Dashboard - 14 Bottlenecks I Had to Fix\n\nCine Pro Timeline\n\nThe first version was taking 9 to 12 seconds. I would click, wait, wonder if something was broken. The database had 230 million rows across multiple tables. The largest single table: 91 million records. Total size: 16GB.\n\nThis is the story of getting that down to under 3 seconds - most pages under a second.\n\nI built this with Claude Code. The process was methodical: add a timer to each component, measure page load box by box, then tackle the slowest one. Claude Code would run investigations - writing test scripts, timing queries with curl and Python, tracing execution paths through frontend and backend. It would come back with a diagnosis: \"This query is scanning 12 million rows because of a JOIN. Here's why the index isn't being used.\"\n\nOne problem at a time. Measure, diagnose, fix, measure again.\n\nThe app now runs on two backends simultaneously - Hetzner in Germany and Oracle Cloud in USA - with a frontend toggle to switch between them. Users can compare latency from different regions. Both backends serve the same 16GB DuckDB database with identical optimization.\n\nHere's everything I learned getting from 9 to 12 seconds to 0.5 to 2.5 seconds.\n\nSection 1: Pre-Compute Everything\n\nThe single biggest performance lever. Four separate bottlenecks, all solved by the same idea: do the expensive work once... create flat tables with pre-aggregations, add index, query the result.\n\nChallenge 1: The JOIN That killed performance\n\nThe Deep Profile page shows a person's complete filmography - every movie, TV show, and short they've worked on. For someone like Morgan Freeman, that's 200+ titles with ratings, vote counts, and genres.\n\nThe original query looked reasonable:\n\n``sql\nSELECT tb.primaryTitle, tb.startYear, tr.averageRating\nFROM title_principals tp\nJOIN title_basics tb ON tp.tconst = tb.tconst\nLEFT JOIN title_ratings tr ON tb.tconst = tr.tconst\nWHERE tp.nconst = 'nm0000151'\n`\n\nTime: 1.6 seconds. Every single time.\n\nThe problem: DuckDB wasn't using the index on titlebasics for the JOIN. It was doing a full sequential scan of 12 million rows to find the ~200 matching titles. The filter on titleprincipals was fast (indexed), but the JOIN to title_basics killed performance.\n\nThe fix: Pre-compute the JOIN once, store it as a denormalized table:\n\n`sql\nCREATE TABLE person_filmography AS\nSELECT tp.nconst, tp.tconst, tp.category,\n tb.primaryTitle, tb.startYear, tb.titleType,\n tr.averageRating, tr.numVotes\nFROM title_principals tp\nJOIN title_basics tb ON tp.tconst = tb.tconst\nLEFT JOIN title_ratings tr ON tb.tconst = tr.tconst\n`\n\nResult: 91 million rows, ~1.8GB storage. But now every person lookup is a simple indexed filter - no JOINs at query time.\n\nBefore: 1.6 seconds (JOIN with full table scan)\nAfter: 0.5 seconds (indexed filter on denormalized table)\nImprovement: 3.4x faster\n\nThe principle: if you're repeatedly JOINing the same tables for the same access pattern, just create the JOIN once and push into a intermediate table\n\nChallenge 2: 15 HTTP Requests for a Static Page\n\nThe main dashboard shows database statistics, rating distributions, top movies by genre, hidden gems. None of this data changes - it's aggregate stats from a dataset that updates monthly at most.\n\nBut the page was making 15 parallel API calls on every load. Each request: DNS lookup, TLS handshake, routing through Vercel serverless, hitting the backend, returning JSON. Even with backend caching, the network overhead alone was 2-3 seconds.\n\nI watched the Network tab: 15 requests, 15 responses, all returning small JSON payloads. Total data: maybe 100 rows across all queries. Total time: 3 seconds.\n\nThe page is static. Every user sees the same thing. Why query 15 times?\n\nFix: Store the entire dashboard response as a single JSON blob in DuckDB. One table, one row, one query returns everything. The JSON gets regenerated when the data pipeline runs (weekly).\n\nBefore: 15 HTTP requests, ~3 seconds\nAfter: 1 HTTP request, ~0.65 seconds\nImprovement: 4.6x faster\n\nChallenge 3: In-Memory Query Cache - Why Most Queries Never Hit DuckDB\n\nAll the pre-computation above makes individual queries fast. But the fastest query is the one you don't run at all.\n\nThe backend has an in-memory cache sitting in front of DuckDB using cachetools (Python library). Every query result gets cached by its SQL hash. Same query comes in again - return the cached result, skip DuckDB entirely.\n\nMost dashboard usage is repetitive. People search for popular actors. The same filmography queries get run over and over. After the first user looks up Morgan Freeman, every subsequent lookup is a cache hit - zero DuckDB work.\n\nCache resets on server restart since it's in-memory, not persisted. Cold starts are slower, but the cache warms up quickly with normal traffic. Config is 2000 entries max per database file with a 30-day TTL. Enough to cover the popular queries without eating too much RAM.\n\nChallenge 4: Same Principle, Applied Three More Times\n\nOnce I saw pre-computation working, I applied it everywhere the pattern fit:\n\nLeaderboards: The \"Most Prolific\" section shows top actors, actresses, and directors by era - six eras, three categories. Each combination was scanning 97 million rows, taking 2-5 seconds per query. Pre-computed all combinations into a summary table.\nBefore: 2-5 seconds. After: ~50ms. Improvement: 40-100x faster.\n\nPerson stats: The stats header (total titles, average rating, career span) was running 6 parallel aggregation queries per person, adding up to ~300ms. Created a person_stats table with one row per person.\nBefore: ~300ms. After: ~5ms. Improvement: 60x faster.\n\nSearch ranking: Search for \"fost\" and Adam Foster appeared before Jodie Foster - alphabetical order, ignoring that Jodie has 700+ credits. Used the pre-computed personstats table to order results by totaltitles. Famous people show up first. And because the stats are pre-computed, the ORDER BY adds ~0.1ms per result instead of ~5-10ms with runtime aggregation.\n\nStorage is cheap. User patience is not.\n\nSection 2: Query Patterns That Backfire\n\nChallenge 5: Searching 15 Million People for Every Keystroke\n\nThe person search queries name_basics - 15 million people. Most are extras, background actors, or people with a single credit. Type \"John\" and the database scans all 15 million rows. About 2 seconds per search.\n\nBut most users search for known actors and directors. Out of 15 million people, only 2.3 million have 3 or more credits. That smaller set covers nearly all searches anyone will actually run.\n\nFix: Created a prolific_people table - just people with 3+ credits. Added a \"Smart Search\" toggle, on by default. Searches scan 2.3 million rows instead of 15 million.\n\nBefore: ~2 seconds (scanning 15M rows)\nAfter: ~200ms (scanning 2.3M rows, 85% less data)\n\nIf someone doesn't show up, uncheck Smart Search to search the full database. In practice, nearly all searches are covered by the smaller set.\n\nChallenge 6: The ORDER BY That Killed My Index\n\nAfter all the pre-computation work, one query was still slow: the filmography list. EXPLAIN ANALYZE showed the problem:\n\n`\nTABLE_SCAN - Type: Sequential Scan\nRows scanned: 91,479,013\n`\n\n91 million rows scanned. But I had an index on nconst. Why wasn't DuckDB using it?\n\nThe query:\n\n`sql\nSELECT * FROM person_filmography\nWHERE nconst = 'nm0000151'\nORDER BY startYear DESC\nLIMIT 500\n`\n\nThe problem: DuckDB's ART indexes have a critical limitation. They only work for single-column equality/IN conditions. The moment you add ORDER BY on a different column, the optimizer decides a full table scan is \"cheaper\" than index lookup + sort. Remove the ORDER BY, and the EXPLAIN output changes to Index Scan.\n\nThe fix: Remove ORDER BY from SQL, sort in JavaScript instead.\n\n`javascript\nfilmography.sort((a, b) => (b.year || 0) - (a.year || 0))\n`\n\nSorting 500 objects in JavaScript: less than 1ms. Saved by removing ORDER BY: 1100ms.\n\nBefore: 2.0 seconds (Sequential Scan because of ORDER BY)\nAfter: 0.9 seconds (Index Scan, client-side sort)\nImprovement: 55% faster\n\nTakeaway: When DuckDB isn't using your index, check for ORDER BY. Remove it and sort client-side if the result set is small.\n\nChallenge 7: Adaptive Query - Check the Count Before You Query\n\nChallenge 6 removed ORDER BY to get Index Scan. But that created a new problem.\n\nLIMIT without ORDER BY doesn't return the first N or the last N rows. It returns any N - whatever DuckDB finds first in storage order. For someone with 200 titles and a LIMIT of 5000, we get all 200. Client-side sort works fine. But for someone with 39,000 titles, LIMIT 5000 returns an arbitrary 5000. Client-side sort then orders this random sample by year. Completely wrong data.\n\nThe fix is an adaptive query. Before running the filmography query, make a separate API call to check how many titles this person has. One row from person_stats. The database lookup is instant, but this is a full round-trip through the network (frontend to Vercel to backend and back), so it costs a few hundred milliseconds.\n\nIf they have 5000 or fewer titles - download everything. No ORDER BY needed because LIMIT doesn't cut anything off. This is the fast path: Index Scan, ~0.9 seconds.\n\nIf they have more than 5000 titles - add ORDER BY to get the most recent 5000. This forces the slower Sequential Scan (~2 seconds), but we need the right data, not random data.\n\n99.98% of people have 5000 titles or less - they get the fast path. Only 513 people in the entire database have more than 5000. The most prolific is Ekta Kapoor with 39,000 entries. The count check adds a network round-trip, but it saves over a second on the main query by keeping Index Scan for nearly everyone.\n\nAn extra API call to check the count costs a few hundred ms, but saves ~4.5 seconds on the filmography query for 99.98% of lookups.\n\nChallenge 8: EXISTS vs CTE - A 15x Performance Gap\n\nThe Discover page lets users filter movies by person - \"show all Morgan Freeman movies.\" The original query used EXISTS:\n\n`sql\nSELECT tb., tr.\nFROM title_basics tb\nJOIN title_ratings tr ON tb.tconst = tr.tconst\nWHERE EXISTS (\n SELECT 1 FROM title_principals tp\n WHERE tp.tconst = tb.tconst AND tp.nconst = 'nm0000151'\n)\n`\n\nTime: 2-3 seconds. The EXISTS subquery runs for every row being evaluated - millions of index lookups.\n\nThe fix: CTE pattern. Get the person's titles first (fast indexed lookup), then JOIN to that small set:\n\n`sql\nWITH person_titles AS (\n SELECT DISTINCT tconst\n FROM person_filmography\n WHERE nconst = 'nm0000151'\n)\nSELECT tb., tr.\nFROM person_titles pt\nJOIN title_basics tb ON pt.tconst = tb.tconst\nJOIN title_ratings tr ON tb.tconst = tr.tconst\n`\n\nThe CTE returns ~200 titles. The main query joins to this tiny set instead of running EXISTS against millions of rows.\n\nBefore: 2-3 seconds (EXISTS subquery on every row)\nAfter: ~200ms (CTE + small join)\nImprovement: 10-15x faster\n\nTakeaway: When filtering by a related entity, get that entity's IDs first with a CTE, then JOIN to the small result set.\n\nChallenge 9: The \"Check First\" Anti-Pattern\n\nThe stats section needed to handle two cases: prolific people (in person_filmography table) and non-prolific people (fallback to slower JOINs). The original code ran a check query first - \"is this person in the fast table?\" - then ran the actual stats queries based on the result.\n\nThe filmography appeared instantly. Stats appeared 300ms later. The check query had to complete before the stats queries could start. Sequential bottleneck.\n\nFix: \"Try fast, fallback if empty\" pattern. Run all the fast-table queries immediately in parallel. If they come back empty, the person isn't in the fast table - run the fallback queries. For the 95% of lookups that hit the fast table, stats load instantly alongside the filmography. For the 5% that don't, there's a small penalty - fast queries return empty quickly, then fallback queries run.\n\nBefore: Check query (300ms) then stats queries (parallel)\nAfter: All queries (parallel), fallback if needed\nResult: Stats appear with filmography, not after\n\nTakeaway: Don't check which path to take. Try the fast path. Fallback if it fails.\n\nSection 3: Stop Querying What You Already Have\n\nChallenge 10: When 5 Database Queries Became Zero\n\nThe Top Stats Band shows highest rated titles, most voted titles, titles per decade, and title types. Original implementation: 5 parallel database queries, ~3-4 seconds total.\n\nThen I realized: the filmography tab loads up to 5000 titles for this person. Those 5000 rows contain everything needed for the stats - ratings, votes, years, title types. The data is already in the browser.\n\nFix: Compute stats from the filmography array in JavaScript. Array.sort on 5000 objects takes less than 5ms. The network round-trip it replaced took 3-4 seconds.\n\nBefore: 5 database queries, 3-4 seconds\nAfter: Array operations, less than 5ms\nImprovement: ~99% faster\n\nI applied this same pattern across the app. The filmography table has Role and Title Type filters - originally each filter change triggered a new database query (1-2 second wait). Now it's client-side Array.filter on the already-loaded data. Instant. The Compare tab showed side-by-side stats for multiple people - originally 5 database queries when you switched tabs. Now it computes comparison stats from the loaded filmography for the current person. Instant.\n\nThe principle: If you've already loaded the source data into the browser, don't make a round-trip to the database for derived views of that same data. Compute them client-side. For datasets under ~10,000 rows, client-side filtering and aggregation is almost always faster than a server round-trip.\n\nSection 4: Infrastructure Realities\n\nChallenge 11: The Container That Thought It Had 24GB of RAM\n\nOracle Cloud backend was 4x slower than Hetzner. Same code, same database, similar specs. OCI queries: ~4 seconds. Hetzner queries: ~1 second.\n\nInvestigated disk latency (ioping showed OCI was 10x slower - red herring, DuckDB caches in RAM). Investigated network (similar). Finally checked memory:\n\n`bash\nContainer limit\ndocker inspect | grep Memory\n8589934592 (8GB)\n\nWhat the container sees\ncat /proc/meminfo | grep MemTotal\n24576000 kB (24GB - host RAM!)\n`\n\nThe problem: Docker containers share /proc/meminfo with the host. DuckDB reads /proc/meminfo to determine available RAM. It saw 24GB, set memory_limit to 80% = ~19GB, then tried to allocate 19GB in an 8GB container. Massive swapping.\n\nThe fix: Explicitly configure DuckDB memory via environment variable:\n\n`python\nDUCKDBMEMORYLIMIT = os.getenv(\"DUCKDBMEMORYLIMIT\") # \"7GB\"\n`\n\nSet to 7GB (leaving 1GB headroom below the 8GB container limit).\n\nBefore: 4 seconds (DuckDB thrashing in swap)\nAfter: 1-1.5 seconds (DuckDB fits in container memory)\nImprovement: 3-4x faster\n\nTakeaway: When running DuckDB in containers, always explicitly set memory_limit. The container's cgroup limit is invisible to /proc/meminfo.\n\nChallenge 12: The Single Worker Bottleneck\n\nBackend was running uvicorn with default settings. Under load, requests could queue up.\n\nBefore: 1 worker, requests serialize under load\nAfter: 2 workers, 2x concurrent capacity\n\nChallenge 13: DuckDB Has No Query Timeout\n\nA badly-written query could block a worker forever. A CROSS JOIN between two large tables - 12M x 1.5M = billions of rows - runs until the server runs out of resources.\n\nFirst attempt: Use DuckDB's timeout setting. SET statementtimeoutms = 30000. Result: error. DuckDB has no statementtimeoutms parameter. Unlike PostgreSQL, there's no built-in query timeout.\n\nFix: Timeout at the Python/asyncio level. Wrap the query execution in asyncio.wait_for with a 30-second timeout. The asyncio timeout doesn't kill the DuckDB query - it just stops waiting and returns a timeout error to the user. The query continues in the background until it finishes, but the thread pool size (8) limits how many runaway queries can accumulate.\n\nTakeaway: When your database doesn't support timeouts, implement them at the application layer.\n\nChallenge 14: The Race Condition in Every Autocomplete\n\nType \"fost\" quickly. Results for \"fo\" arrive. Then \"fos\" results. Then \"fost\" results. The dropdown flashes between different result sets as each async response arrives out of order.\n\nThe problem: Each keystroke fires an async search. Responses arrive in arbitrary order. Old responses overwrite new ones.\n\nFix: Sequence number tracking. Every search increments a counter. When results arrive, check if the counter still matches. If a newer search has started, discard the stale results.\n\n`javascript\nconst thisSeq = ++searchSeqRef.current\nconst results = await executeSearch(term)\nif (thisSeq !== searchSeqRef.current) return // Stale, discard\nsetSearchResults(results)\n``\n\nCost: zero. One integer increment, one comparison. Result: no more flashing, clean dropdown.\n\nTakeaway: This pattern applies to any async operation where newer requests should supersede older ones.\n\nWhat's Working Now\n\nThe dashboard pages that took 9-12 seconds now load in 0.5 to 2.5 seconds. Filters respond immediately. Search shows famous people first.\n\nEvery optimization came from the same process: measure, investigate, diagnose, fix, measure again. Claude Code made this efficient - it could trace execution paths, write test scripts, and propose fixes faster than I could type the problem description.\n\n230 million rows. 14 bottlenecks removed. Sub-second response times. Running on a server that costs a few euros a month. Not because of any single optimization, but because each fix revealed the next bottleneck. The slow query you fix today exposes the medium query that becomes tomorrow's slow query.\n\nFull implementations - every query, every config, every JavaScript pattern - are in the open source repos linked below.\n\nRelated Posts\nArchitecture & Setup for a Dashboard with Hundreds of Millions of Records - Powered by DuckDB\nCinePro - 230M Rows, 16GB Database, Instant Queries with DuckDB\n\nResources\nLive App - Cine Pro Movie Explorer\nFrontend Code\nBackend Code"
},
{
"slug": "full-backend-infrastructure-duckdb-production",
"title": "Full backend Infrastructure for DuckDB production and analytics system",
"tags": [
"duckdb",
"open-source",
"infrastructure"
],
"content": "Full backend Infrastructure for DuckDB production and analytics system\r\n\r\nBasekick Arc - High-Performance Columnar Analytical Database\r\n\r\nDuckDB needs backend infra to run as a production service - REST API, auth, monitoring, storage policies. Came across Arc - a solid backend implementation for DuckDB. Written in Go by Ignacio Van Droogenbroeck. Open source (AGPL-3.0).\rREST API layer with multiple response formats (JSON/Arrow)\rAuth with management API\rMultiple storage backends (Local/S3/MinIO/Azure)\rAutomatic file compaction for query performance\rTime-based partitioning, Retention policies\rConnection pooling, continuous queries\rPrometheus metrics and structured logging\rWrite-Ahead Log (optional)\r\n\r\nBuilt on DuckDB + Parquet + Arrow - all DuckDB SQL. Originally built for time series, but the backend patterns apply to any DuckDB deployments. Wonderful library.\r\n\r\n\nResources\rGitHub Repository\rDocumentation\rBasekick"
},
{
"slug": "gemini-2-0-multimodal-how-to-use",
"title": "Gemini 2.0 - Multimodal - How to use",
"tags": [
"ai-coders"
],
"content": "Gemini 2.0: Google Knocks It Out of the Park\n\nQuick Take: Next-level tech with real-time multimodal power\n\nNote: These are early impressions based on a few hours of testing-Gemini 2.0 launched just yesterday. This is not a detailed evaluation.\n\nExplored AI Studio, tinkered with the React starter app with its full API access to real-time vision, video, and audio. Came in with low expectations. Left genuinely astounded.\n\nWhat Stood Out\nReal-Time vision, video, and audio are remarkable\nToolkit for Devs & Analysts: Full API access, React starter kit, and Python SDK\nGenerous Free Tier: 10 RPM, 4M TPM, and 1,500 requests/day-great for prototyping and testing\n\nWhat's in it for You?\nAI Studio: Realtime multimodal power. For developers as well as non-technical users\nBuild Multimodal AI Apps: Solid set of developer resources and affordable pricing (assumed, if free tier is anything to go by)\n\nAny Minuses?\n\nStill early days. Some latency and breaks were noticeable, but it's Day 0-plenty of time to optimize.\n\nUp Next\n\nIntegrating real-time voice with REX, my open-source decision intelligence app for natural language querying and database integration. Testing alternatives to OpenAI Realtime API-starting with Eleven Labs, Hume and now Gemini 2.0. Planning to share learnings, working apps, and source code soon.\n\nTry REX for free: rex.tigzig.com\n\nFull Breakdown: Releasing REX-2: AI Decision Intelligence\n\nLinks\nAI Studio: aistudio.google.com\nOpen Source: Source code on app site"
},
{
"slug": "gemini-3-pro-added-to-database-ai-suite-tested-against-claude-sonnet-4-5-and-gpt-5-1-results-claud",
"title": "Gemini 3 Pro Added to Database AI Suite. Tested Against Claude Sonnet 4.5 and GPT-5.1.Results: Claude still leads. GPT-5.1 is solid. Gemini 3 Pro lands third.",
"tags": [
"database-ai",
"text-to-sql"
],
"content": "Gemini 3 Pro Added to Database AI Suite. Tested Against Claude Sonnet 4.5 and GPT-5.1.Results: Claude still leads. GPT-5.1 is solid. Gemini 3 Pro lands third.\n\nImage 1: ree\n\nPerformance Scores\n\nMulti-step database analysis workflows:\nClaude Sonnet 4.5: 115\nGPT-5.1: 100\nGemini 3 Pro: 90\n90-tier: Gemini 2.5 Pro, GPT-4.1, KIMI 2 Thinking\n85-tier: Gemini 2.5 Flash, Qwen 3 Max, GLM 4.6, DeepSeek R1\n80-tier: Gemini 2.0 Flash\n\nModel Findings\nClaude Sonnet 4.5: Creates multiple segmentation variables beyond the prompt. Quality leader.\nGPT-5.1: Strong. Token bloat gone vs GPT-5. Clear second.\nGemini 3 Pro: Similar to Gemini 2.5 Pro. Better explanations, same output quality. Doesn't match top two.\n\nWhat Was Tested\n\nCredit card analysis: 1M customer + 10M transaction tables (AWS RDS MySQL). Multi-step workflow - plan, summarize, create derived variables, merge, segment profiles.\n\nWeighted ranking of Indian credit card issuers from RBI data (Postgres).\n\nPractical Choices\nHigh quality → Claude Sonnet 4.5\nBalance → GPT-4.1\nLow cost- great value→ Gemini 2.5 / 2.0 Flash\n\nCost Breakdown\n\nPlanning vs Execution\n\nSingle iteration: reasoning LLM runs once (20% cost), execution agent runs 7-10 queries with debugging (80% cost).\n\nReasoning / Planning Cost (Per 100 Questions)\nHigh: Claude, GPT-5.1, Gemini 3 Pro (approx. $7.00)\nMid: GPT-4.1, KIMI 2, Qwen, GLM, DeepSeek (approx. $2.50)\nBudget: Gemini 2.5 Flash (approx. $1.50), Gemini 2.0 Flash ($0.30)\n\nAvoid: Gemini 2.5 Pro ($10) and GPT-5 ($15) - token bloat.\n\nExecution Cost (Per 100 Questions)\nAdvanced analysis: approx. $12.50 (GPT-4.1, consistent across reasoning models)\nSingle-step: $0.40 (GPT-4.1-mini) to approx. $3.50 (GPT-5.1)\n\nI use OpenAI for SQL execution - more reliable. Multi-step workflows multiply costs fast. Use only when needed.\n\nThese costs relate to my typical uses. Your numbers will vary based on context, architecture & output volume. Always test with your use case. Always check actual billing, not token based estimates.\n\nDATS-4: Database AI Suite- v4\n\nRemote database AI app. Postgres or MySQL. Single-step queries or multi-step analysis. Python charts, table upload, PDF reports. Open source, live, free.\n\nTry It\n\nUse Sample button. Data loads to temp Postgres. Or connect your database.\n\nPublic app routes through my backend - sandbox only. Deploy on your servers for live work\n\nResources\nDatabase AI Field guide- usage, architecture, process flows\nPrevious post: GPT-5.1 + KIMI 2 evaluation with short video\nPosts & guides - AI for Analytics: tigzig.com\n\nDirect access to the main database AI apps:\nChatGPT + Your Database (connect any Postgres/MySQL): ChatGPT here\nChatGPT + Fixed Data (Simultaneous connection across 3 Databases): ChatGPT here\n\nRelated resources - Database AI\nImplementing Database AI: Field Guide - 49-page PDF - architecture, agent setups, cost analysis - Read Here\nIn-Browser Database AI - 1.5 GB files: process locally with DuckDB - Read here\n10 Options for Talking to your databases - 10 micro-apps across 5 categories - Remote, In-Browser, ChatGPT, Voice, Rapid Deploy - Read here"
},
{
"slug": "genai-llm-app-analytics-assistant-aws-azure-mysql",
"title": "GenAI App | LLM Analytics Assistant: Simplifying Data Transformation & Insights. AWS & Azure MySQL DW Example",
"tags": [
"database-ai",
"text-to-sql"
],
"content": "GenAI App | LLM Analytics Assistant: Simplifying Data Transformation & Insights. AWS & Azure MySQL DW Example\n\nNEW\n\nMy open-source platform with a ton of micro-apps and tooling's for AI driven analytics\n\nText to SQL / connect to ANY data-warehouse on the fly/ direct file upload to data-warehouse table / create temporary database on the fly / python charts / statistical analysis\n\nRealtime-voice connected to database - OpenAI new WebRTC API & Eleven Labs\n\nAnd more ....\n\n3rd part of the series on LLM Analytics Assistant Apps\n\nDemonstrating data transformation and analysis on AWS MySQL via an LLM App. The app is deployed on my public website (outside of GPT Store, access-controlled section).\n\nI cover 3 areas:\n\nLLM APP DEMO\n\nData Wrangling & Analysis: prototype customer table and transaction table with a million to 10 million records, creating summaries and merging data into new tables with additional variables... analyzing and creating customer profiles. All instructions in natural language... sometimes fuzzy and unclear... and sometimes with spellos...\n\nBASIC ARCHITECTURE\n\nSimilar to one that I am currently using on a live client project.\n\nLLM App Build and UI: using Flowise AI. Open-source. Allows for rapid deployment. Powerful capabilities. Many other options - e.g. custom build with React/Next.js that can link up to company SSO and authentications.\n\nModel Choice: trade-offs between pricing, speed, response quality, and security/privacy. Premium model vs. open-source on-prem solution.\n\nArchitecture Flexibility: FastAPI processing server. Separate from the main system, making it reusable with different UI apps and backend databases.\n\nCOST CONSIDERATIONS\n\nCost Example: ran 478 API requests/queries over 10 hours with GPT-3.5, costing around $1... working with the 1 million-10 million dataset referred to above... also discuss optimization strategies...\n\nChoosing LLM models: depends on use case. e.g. Multi-LLM option...for difficult tasks, use an expensive model, and for simpler tasks, use a lower cost model.... or On-Prem solution for specific use cases.\n\nFull Data Ingestion by the LLM model is not always necessary... can significantly increase costs... potentially increasing by 100 times or more. For many use cases, processing can be done separately, and the LLM only passes SQL queries/Python commands.\n\nSplit Workflow Approach: for scenarios requiring full data ingestion, split the workflow into multiple modules. LLM to only ingest the necessary and smallest amount of data directly... process the rest of the data separately.\n\nUPCOMING VIDEOS AND POSTS\n\nCurrently preparing detailed tutorials and step-by-step guides covering code, tips, and leveraging GPTs to develop apps. In future videos and posts, I will also cover areas like : processing with on-prem solutions, multiple LLM approaches, segregation of Python processing vs. MySQL processing, machine learning model builds, selective accesses, and more."
},
{
"slug": "go-from-a-200mb-flat-file-with-1-5m-records-to-analysis-in-minutes-with-my-open-source-ai-sql-app",
"title": "Go from a 200MB flat file with 1.5M records to analysis in minutes with my open-source AI-SQL App",
"tags": [
"database-ai",
"text-to-sql"
],
"content": "Go from a 200MB flat file with 1.5M records to analysis in minutes with my open-source AI-SQL App\n\n20 Yrs ODI Cricket stats - I'm providing the data and tools. Go run it.\nGet the Data\nData: 25 years of ODI cricket data from cricsheet.org processed with Tigzig tools.\nFormat: Pipe delimited raw TXT file, approx. 200MB, 1.5M records.\n\nDownload from Google Drive\nGet the Free Database\nPlatform: neon.com\nAction: Go to their site. Get a free, live Postgres database in seconds. No CC required. Copy the credentials. This is your temporary analysis sandbox.\nLoad the Data\nPlatform: app.tigzig.com → Database Apps → DATS-4\nAction: Connect to your Neon DB, then upload the 200MB text file. The app handles the rest. Takes approx. 2 minutes.\nMenu → Connect to Database\nMenu → Choose File\nQuery with Natural Language\nAction: Just type or dictate your question.\nExample 1: \"Show top 10 batsmen by runs off the bat with chart\"\nExample 2: \"For these, show runs, matches, overs, run rate per match & per over, with chart\"\n\nDATS-4\n\nMy open-source SQL multi-agent app. It handles Text-to-SQL, Python charting, stats, instant Postgres creation, PDF outputs, and provides 9 reasoning models (Gemini, Claude, DeepSeek, more).\n\nPractitioner's Warning\n\nThis is a public-facing app. All credentials and API calls run through my backend server.\n\nRule: Use this public version for sandbox testing ONLY, with temporary databases and non-sensitive data.\n\nFor Live Use: Full source code shared. Deploy it on your VPN. Current setup is low-security for open testing; live use must tighten auth and access controls. Basic OAuth module with Auth0 included in source.\n\nWhere it gets messy\n\nThis example uses file I pre-processed for rapid analysis.\n\nReality: is not like click-click and report appears. It's more like bang-head, bang-head and then a drop appears.\n\nThe Work: needs data cleaning, semantic layers, pre-computed metrics, marts and summary tables. AI is a powerful tool, but it doesn't replace solid data engineering - even though I use AI for data engineering too, including pre-processing of this data."
},
{
"slug": "going-beyond-google-login-hardening-entry-points",
"title": "Going beyond Google Login for critical apps. Identifying gaps & hardening your entry points.",
"tags": [
"security",
"infrastructure"
],
"content": "Going beyond Google Login for critical apps. Identifying gaps & hardening your entry points.\r\n\r\nTigZig Command\r\n\r\nJust AI security audit is not enough.. it misses things.\r\n\r\nSo had two Claude's go against each other....one hitting my app the other one fixing it. 130 tests across 3 phases.\r\n\r\nThis is the app that monitors all my databases, servers, deployments. Basically the keys to the kingdom, sitting on the public internet deployed at logs.tigzig.com. I also have client tools deployed on open internet (all with OAuth).\r\n\r\nFor admin-type apps sitting on the public internet (vs a VPN), OAuth (Google etc) alone isn't enough. So now this app has multiple independent gates.. Cloudflare Turnstile for invisible bot detection, a password gate before you even see the login page, Google OAuth with email whitelist, Google Authenticator MFA, JWT verification on every API call.\r\n\r\nDDoS with rotating proxies - my server might remain unavailable for a little while till I turn on the 'under attack' mode in Cloudflare, But my gates won't be breached (hopefully)\r\n\r\nMore gate security possible? yupp.. but for a tool builder for small businesses, I think this is good enough.\r\n\r\nUpdated the security checklist to 78 items with all the new patterns.\r\ntigzig.com/security"
},
{
"slug": "gold-up-2-3x-past-2-years-but-what-about-the-10-years-drawdown-in-between-as-buffett-says",
"title": "Gold up 2.3X past 2 years. But what about the 10 years drawdown in between.",
"tags": [
"portfolio-analytics"
],
"content": "Gold up 2.3X past 2 years. But what about the 10 years drawdown in between.\n\nAs Buffett says \"As 'bandwagon' investors join any party, they create their own truth - for a while\"\n\nIf you had bought it at its previous rising tide (like now) in 2011 you would be waiting for nearly 10 years till 2020 just for it to reach your buying levels.\n\nIf it happens again - and it could - do you have the stomach to wait till 2035 or beyond?\n\nGold has been on a rise since COVID. Central banks loading up, especially emerging markets after the U.S. froze Russian assets. Goldman Sachs and many others say it'll keep rising. Maybe it will. But any of those reports warning you about a potential repeat of the brutal 10-year drawdown? I haven't seen one.\n\nBuffett said this in 2011 when gold was hot:\n\n\"...Tulips, of all things, briefly became a favorite of such buyers in the 17th century. The major asset in this category is gold, currently a huge favorite of investors. Gold, however, has two significant shortcomings, being neither of much use nor procreative.\nWhat motivates most gold purchasers is their belief that the ranks of the fearful will grow. During the past decade that belief has proved correct. Beyond that, the rising price has on its own generated additional buying enthusiasm, attracting purchasers who see the rise as validating an investment thesis. As 'bandwagon' investors join any party, they create their own truth - for a while\"\n\nTo pull the report\n\nHead over to quants agent at quants.tigzig.com. Open source. Free.\n\nPrompt: \"get me SPR for gold futures and S&P 500 past 20 years.\"\n\nGold futures work as a decent proxy for gold prices. The drawdown data is brutal. Agent is free to use - run your own reports across securities and time periods.\n\nReferences\nGoldman Sachs (Mar '25): https://www.goldmansachs.com/insights/articles/why-gold-prices-are-forecast-to-rise-to-new-record-highs\nBuffett's 2011 letter: https://www.berkshirehathaway.com/letters/2011ltr.pdf"
},
{
"slug": "google-antigravity-just-launched-for-analysts-and-data-scientists-worth-adding-to-your-toolkit",
"title": "Google Antigravity just launched. For analysts and data scientists: Worth adding to your toolkit",
"tags": [
"ai-coders"
],
"content": "Google Antigravity just launched. For analysts and data scientists: Worth adding to your toolkit\n\nGoogle Antigravity just launched. For analysts and data scientists: If you're already on Cursor/Claude Code, this gives you an extra option when you hit message limits. If you're just starting with AI coders, this is the easiest entry point I've seen.\n\nFree agentic IDE with Gemini 3 Pro, Claude Sonnet 4.5, and GPT-OSS built in. Built on top of VS Code. One-click install with pretty good rate limits.\n\nI've been testing it for low & medium complexity tasks. Solid performance so far. Interface is clean and pretty much like working with Cursor. Hit rate limit on Claude Sonnet 4.5, so changed model to Gemini 3 Pro and continued. Hit rate limit there too after a while with a reset after 2 hours. After around 25 messages, but they were all context heavy ones\n\nMy primary toolset still is Claude Code & Cursor. And Gemini CLI for lighter tasks. This is another option that's performing well enough to add.\n\nMore complex tasks to be tested. For now, it's earned a spot in my toolkit.\n\nPrevious posts on AI coders\nCoding by hand is obsolete. Domain + execution is the edge\nAI Coders: The 6 rules I follow\n\nLive apps built this way: app.tigzig.com\n\nGuides: tigzig.com"
},
{
"slug": "google-gemini-2-0-flash-api-performance-quality-cheaper-gpt-4o-mini",
"title": "Google Gemini 2.0 Flash — solid API performance, great quality, and cheaper than GPT-4-mini. The new workhorse?",
"tags": [
"database-ai"
],
"content": "Google Gemini 2.0 Flash - solid API performance, great quality, and cheaper than GPT-4-mini. The new workhorse?\n\nLLM usage hierarchy for my day-to-day live use cases\ngpt-4o-mini - default choice, a workhorse with solid pricing\ngpt-4o - when mini struggles or for complex cases\nclaude-3.5-sonnet - when gpt-4o struggles\n\nPoint to note: gpt-4o and sonnet API pricing is approx. 15x mini.\n\nAnd now: gemini-2.0-flash - getting excellent results in early testing for my use cases. Earlier Gemini models had higher API call failures and issues with structured output, but that seems to have changed. Stress testing now. If Gemini's API reliability and structured output consistency hold up, I may move to gemini-2.0-flash as primary and start migrating clients.\n\nTypical use cases\n\nRequiring LLM API calls or agent setups:\nAutomation\nWeb scraping\nStructured outputs\nOCR\nDatabase-connected micro-apps\n\nAs an aside, coding is separate:\nJavaScript and React micro-apps: Cursor\nPython work: Colab, now moving to Mito-AI for its co-pilot and Cursor-like experience\n\nWhat's new with gemini-2.0-flash?\n\nProduction release: first week of Feb. Based on initial live testing with my typical use cases:\nQuality: equal to or better than gpt-4o\nReasoning: better than o3-mini, comparable to deepseek-r1\nCost: generous free tier and paid pricing lower than gpt-4o-mini\n\nAPI pricing\n\n| Model | USD per M input tokens | USD per M output tokens | Free tier |\n|-------|------------------------|-------------------------|-----------|\n| gpt-4o-mini | 0.15 | 0.60 | none |\n| gpt-4o | 2.5 | 10 | none |\n| claude-3.5-sonnet | 3 | 15 | none |\n| gemini-2.0-flash | 0.10 | 0.40 | 15 req/min, 1M tokens/min, 1500 req/day |\n\nFlash 2.0 is now priced below mini.\n\nWant to test yourself?\n\nTry my open-source AI analytics apps, no API key needed, live on rex.tigzig.com. Compare between LLMs. Source code available on the site in the Help and Build sections.\nMulti (sequential) agentic app for advanced analytics\n\nCompare reasoning between o3-mini, flash-2.0, and deepseek-r1. Sample files available on the site. Temporary Postgres databases created on the fly, no login required. Tigzig Analyzer\nAI-enabled mutual fund portfolio processor\n\nCompare file schema identification between gpt-4o and other LLMs with live runs. Use your choice of mutual fund monthly file or a sample file from Google Drive (link in Help section). Tigzig MF Portfolio Processor"
},
{
"slug": "google-on-a-roll-launches-dsa-data-science-agent-on-colab-first-impression-just-brilliant",
"title": "Google on a roll — launches DSA — Data Science Agent on Colab. First impression = just brilliant.",
"tags": [
"ai-coders"
],
"content": "Google on a roll - launches DSA - Data Science Agent on Colab. First impression = just brilliant.\n\nGoogle Colab recently launched Data Science Agent (DSA)-plans and executes multi-step analysis/modeling in one shot. Meanwhile, Mito-AI Copilot (launched earlier this year) offers a seamless coding experience with Cursor-like chat + built-in data science agent.\n\nSo, which one's the best?\n\nBoth have strengths. I used Colab exclusively, but Mito-AI pulled me back to Jupyter-huge efficiency boost. Plan on using both- depending on the project. Best of both worlds.\n\nGoogle Colab edge\n\nMore powerful agent, code sharing, easy google drive/sheets access, strong processing speeds, free T4-GPU access. Supports in-cell code generation ....but lacks a true copilot with memory.\n\nBut Mito-AI wins big in one area: a seamless copilot experience.\n\nIt auto-sends all data schemas (df structures + sample rows) to AI + retains conversation history for real-time AI awareness-big difference. Smooth vibe/ voice coding (Win+H / Cmd+H). Just pip install mito-ai mitosheet and you're set. In the video, I demo a live voice coding for file processing automation to create a datamart.\n\nRole of Data Science Agent\n\nColab DSA is very powerful (https://lnkd.in/g3ub_84D), and great for the right projects-especially to run a full multi-step workflow in one shot. But I can't see using it for every project. Many require validation against tally numbers and business logic at each step before moving forward. At the same time, I do foresee cases where I'd prefer to run the entire workflow at one go and refine later-Colab DSA would be my choice.\n\nPricing\n\nColab is free. Mito-AI offers 1 month/500 free chats, then $20/month with unlimited completions and extras. Open-source version available with your own API key.\n\nInsights and Tips\n\nBuild AI Co-Analyst Apps for Analytics & Data Science\n\nExplore 15+ open-source AI analytics apps at tigzig.com-including multi-agent workflows, real-time voice AI, and Python/SQL automation. Connect to any database. Free access, no API key needed. Source codes and build guides included.\n\nAI Advanced Analytics App with Multi Agents (Sequential-LangGraph). https://lnkd.in/g8xPQvb8"
},
{
"slug": "google-the-old-edge-is-back-by-dec-24-in-ai-i-had-written-google-off-now-the-balance-has-shif",
"title": "Google - The old edge is back.By Dec ’24, in AI, I had written Google off. Now, the balance has shifted",
"tags": [
"ai-coders"
],
"content": "Google - The old edge is back. By Dec '24, in AI, I had written Google off. Now, the balance has shifted\n\nHorrible models. Gemini 2.0 Flash Experimental was a disaster – more than 50% error in my schema detection and structured output tests.\n\nThen in Jan '25, they pushed it into production as Gemini-2.0-Flash-001.\n\nI laughed. Tested it anyway.\nFirst run: 100% accuracy → luck\nSecond run: 100% → coincidence\nThird run: 100% → that's a trend.\n\nThat was the instant that Google changed.\n\nI even published my first post on it: LinkedIn post\n\nAnd just for context – this wasn't casual testing, and I have used Google from day one. My first investment analysis webpage went live the same month Google did.\n\nToday, approx. 20% of my workflow is Google-powered.\n\nCan Gemini become the top LLM, beating Claude Sonnet-4?\n\nA year back I would have laughed. Now – very likely.\n\nIf Google gets this right, it might reign over AI the way it does over Search – lawsuits and congressional committees notwithstanding.\n\nThe old Google is back.\n\nAs much as I love them, I wouldn't want to be in Microsoft, OpenAI, or Anthropic's shoes at this point in time.\n\n2.0-Flash is like the old Nokia 3310\n\nI've tested it across workflows. Schema detection + structured output in my live Mutual Fund Portfolio Processor (India) still runs on Gemini 2.0 Flash. GPT-4.1 and Sonnet-4 both miss at times. Flash hasn't failed.\n\nTry it live: app.tigzig.com/mf-files-ai\n\nThis model is like the old Nokia 3310 - low-cost high-performance. Break it, drown it, throw anything at it, it works. Huge free tier.\n\nGoogle kept shipping\nAI Studio → I built 70% of my Quant Reporting Suite UI there (rest with Cursor). Live at quants-suite.tigzig.com\nGemini CLI → 75K GitHub stars. I use it for FastAPI servers, xlwings Lite, simple web apps. github.com/google-gemini/gemini-cli\nFlash-2.5 & Pro-2.5 → solid upgrades (Pro is a token guzzler).\nNotebookLM → my YouTube time-saver (auto action items + guides). notebooklm.google\nGoogle Search AI Mode → very new. Replaced Perplexity & ChatGPT for me for research tasks. Blazing fast.\nOpal → for mini-apps + n8n-like workflows. opal.withgoogle.com\nDatabase Toolbox → web layer to connect agents to DBs. This could replace my custom connector. Google Cloud Blog\nCode Interpreter → sandboxed Python for LLM. AI Studio Code Interpreter\nURL Context → fetch and analyze live pages. AI Studio URL Context\nGoogle ADK (Agent Development Kit)\nGoogle LangExtract → structured output from long docs. Vertex AI LangExtract\nGemini with browser use → AI Studio Browser\n\nPerformance charts, technicals & CAGR reports → built with my open-source Portfolio Analysis tools at quants.tigzig.com. Valuation metrics → Yahoo Finance."
},
{
"slug": "google-tools-i-use-on-live-projects-analysis-automation-building-micro-apps",
"title": "Google Tools I Use on Live Projects — Analysis, Automation & Building Micro-Apps",
"tags": [
"ai-coders"
],
"content": "Google Tools I Use on Live Projects - Analysis, Automation & Building Micro-Apps\n\nI went from zero Google usage last year → now approx. 20% of my work runs on their stack. Here's my current short list of practical tools.\nGemini Build\n\n(aistudio.google.com → Build)\n\nA fast way to create basic working app with React. Fewer features than say Lovable, but the free tier is significant.\n\nMy Quants Reporting app → 70% of the UI in Gemini Build, 30% Cursor.\n\nJust starting with AI coding app builds? Starter links below.\nAI Studio\n\n(aistudio.google.com → Chat)\n\nGemini 2.5 Pro. Huge context. Big free tier.\n\nI use it for xlwings Lite code gen for when Cursor hits limits.\nBackend APIs\n\nI use these for reliable, low-cost automated processing.\nGemini 2.0 Flash → schema detection, structured outputs. Cheap + reliable. My Mutual Fund Processor app runs on this.\nGemini 2.5 Flash → where you need analysis layered on automation. I run my AI Technical Reports backend on it.\n\nLive App → With agent support at quant-agent.tigzig.com. Form based at Quants Suite\nGemini CLI\n\nCursor is still my primary AI coder, but when limits kick in → Gemini CLI.\n\nWhere I use:\nPython automations\nFastAPI backends\nxlwings Lite code gen\nHTML/JS Apps\n\nEven file cleanup - had 20k small text files, renamed them in seconds via CLI. For more complex stuff, I stick to Cursor + Claude Sonnet 4. The recent integration with the ZED editor is on my list to test for a full IDE workflow.\n\nGetting started with Gemini CLI? Helpful links below.\nNotebookLM\n\n(notebooklm.google.com)\n\nTurns hours of video into notes, checklists, action steps.\n\nFor technical videos → extract exact workflow and step by step guides.\n\nFor info videos → key takeaways\n\nNot for every video. Some still need full watching.\nOpal\n\n(opal.withgoogle.com)\n\nGoogle's new AI micro-app builder. For workflows for automation and AI apps. Very promising. Next on my list.\n\nLinks - AI-assisted coding\n\nMy Top pick - Volo Builds YouTube Channel\n\nYou can start with these:\nSingle-file HTML app\nPrompting for AI coding\nAI coding basics (beginners)\nFull-stack app build\n3-Tier Frontend–Backend - Simple explanation of a 3-Tier architecture.\n\nGemini CLI\nNode.js install (needed for local dev)\nGetting started with Gemini CLI"
},
{
"slug": "gpt-a-force-multiplier",
"title": "GPT-4 is acting like a force-multiplier like I have never experienced before.",
"tags": [
"ai-coders"
],
"content": "GPT-4 is acting like a force-multiplier like I have never experienced before.\n\nAnd spoilt me utterly ...\n\nGPT-4 Force Multiplier\n\n✴️ For coding errors....i've stopped describing errors....takes too long...\n\nI attach screenshots of errors and type: fix it pls.\n\nThat's it....get a new code back. In most cases, it works right off the bat...\n\n✴️ Some days back, I did a post about GPT-4o release vs. higher expectations. I take it all back...spoilt analyst syndrome....\n\n✴️ For just $20 a month, the value I am getting is humongously exponentially more.\n\n✴️ I'm still on the older voice mode on my account.....amazing technical brainstorming conversations. Can't wait to try out the new voice-video mode.\n\n🔍 Can now directly connect to Google Drive / One Drive.\n\n🔍 Connectors to AWS / Azure/ GCP: likely to come sooner than later.\n\n✴️ Recently completed a client project involving process reengineering and automation, followed by analysis. Used VBA (for automation) and Python.\n\nHad totally forgotten VBA\n\nSomething that would have taken me weeks was delivered in a few days.\n\nOne of the best VBA codes 'I' (ahem....) have ever written in my life.\n\n✴️ For another Excel automation for a client: set up a private Custom GPT running Python code. The trick to consistent results? Freeze the Python code, otherwise, it starts to re-engineer the whole thing. ChatGPT-4 acts as a UI: upload the file, process it, and get the output CSVs back.\n\n✴️ Then there are those Annual Reports I am dumping into GPT-4 for (DuPont) Ratio Analysis....some are 500 pages plus ... and also extracting nicely formatted tables of financial statements and schedules\n\n✴️ And Custom GPT and LLM apps for a variety of analytics tasks. Have setup multiple Fast API servers running my python codes ....and am calling those via API calls from within GPT as well as from external LLM apps.\n\n✴️ I doubt I 'chat' with ChatGPT / LLMs too much... except brainstorming conversations... mostly it is coding, debugging, automation, data cleaning, machine learning models, running API calls. ....\n\n...and acting as a UNIVERSAL USER INTERFACE and a MULTI-PURPOSE PROCESSING SERVER\n\n....I could go on and on....\n\nAll GPT Magic\n\nMany of these covered in detail my earlier LinkedIn Posts and Videos.\n\nOpenAI, I bow before thee. 🙏\n\nWith a single stroke...on that fateful day of 30th November 2022....OpenAI changed the world. Definitely mine"
},
{
"slug": "hetzner-coolify-self-hosting-ai-apps-under-10-dollars",
"title": "You are paying ~$3-7 per deployment for your AI Apps. How do you do it in <$10 per month?",
"tags": [
"infrastructure"
],
"content": "You are paying ~$3-7 per deployment for your AI Apps. How do you do it in <$10 per month?\n\nYou are paying ~$3-7 per deployment for your AI Apps. For 40 tools that is $100-200 PM. How do you do it in <$10 per month?\n\nThat's what happened around two years back, when I started deploying my open source tools on tigzig.com and doing client app projects.\n\nComing from analytics, this was new to me. Started with Render - very easy. Then I had 10 apps on free service and 2-3 on paid plans, paying $7 per service for persistent disk. Free tier ones go to sleep after inactivity. Plus, you only get 512MB RAM and half a CPU. Good for small tools, but larger tools tank.\n\nI realized I needed full flexibility. Adding GBs of data, file hosting, higher RAM etc - without worrying about pay-as-you-go surprises, multiple providers or complex pricing.\n\nFinally decided to self-host everything. There was the fear of the unknown, but over time, and with AI help, turned out to be easier than I had thought.\n\nTwo Major Elements\nA shared server - basically a machine\nA deployment tool - to manage multiple apps\n\nHetzner for the Server\n\nDecided on Hetzner for pricing, reliability, and simplicity. For Euro 7.69 you get 8GB RAM, 80GB disk, 4 vCPUs, dedicated IP, full SSH access. It seemed like an ocean at that time. Now filling up fast.\n\nHave a big app? 32GB RAM, 640GB disk, 16 vCPUs - Euro 39 p.m. Not bad, right?\n\nAll NVMe disks, by the way.\n\nVery clean interface vs some cloud providers that feel like an aeroplane cockpit.\n\nCoolify for Deployment\n\nFor deployment using Coolify (Free, Open Source). Provides a Render-like interface. Deploy via GitHub repo or Docker. Everything in its own container. You can customize RAM and swap file sizes.\n\nI used this playlist from RJ Reynolds to set up Coolify: YouTube Playlist\n\nWhen I set it up nearly two years back, I did it one command at a time with ChatGPT. Now with Claude Code, you can share SSH keys and have it do most of the setup. But still do it in small steps, checking every step.\n\nMore Resources\n\nMore on Infra Setup for AI Tool Builders in my 2026 Guide - Part 2"
},
{
"slug": "how-to-build-ai-action-agents-beyond-chat-with-voice-agents",
"title": "How to update Excel, Google Sheet and backend Databases with Natural Language commands with Voice Agents",
"tags": [
"voice-ai",
"database-ai"
],
"content": "How to update Excel, Google Sheet and backend Databases with Natural Language commands with Voice Agents\n\nExecute. Automate. Analyze\n\nChat is just one of the many things LLMs can do...\n\nThey can act and execute.\n\nIn this first of a 5 part series, I show a demo, architecture and process tracing. Next 4 are all detailed hands-on implementation guides.\n\nVTEXER: Voice-Enabled LLM Action Agent App for Task Execution Automation Query and Research. I demonstrate how you can, with natural language voice instructions:\nUpdate Excel Sheet\nUpdate Google Sheet\nUpdate tables in remote Data Warehouses\nGenerate and email print formatted report\nGenerate and email a slide format\nQuery MySQL database, including updating and modifying data\nAnd few additional things shown in the demo.\n\nAnd yes, it can also chat.\n\nThe app uses LLM ReAct Agents, Reasoning and Action Agents. Called as such since they can execute tasks via API calls, also called Function Calls / Tool Calling. Fairly easy to setup in Flowise. FlowiseAI (YC S23) has a marvelous functionality to access an agent / chat flow via an API endpoint. Allows you to fully customize the UI and response flow. Using that extensively here (glad i remembered it from Van Zyl's videos).\n\nThe app has a React.js frontend, with Flowise providing complete middle layer of LLM Agents and ChatFlows. Backend of automation workflows in Make.com, automation scripts in Google Apps Script and database connectivity via a custom-built FastAPI server.\n\nALL coding done by GPT, LLMs & AI Assisted Coding Tools. For Google Scripts and Python I like GPT-4o. For React in general it is Claude Sonnet. For React.js apps like this one, my current favorites are Claude Dev, a VS Code extension and Cursor AI IDE, a VS Code fork. Claude Dev works on Github Codespaces also… I have a bit of preference for cloud over local. Both are able to create new files as well as modify files across whole codebase. Claude Dev even has terminal access, so does pretty much everything.\n\nIn this video:\nDemo\nArchitecture Overview\nTracing of Agent Flow - Component Level\nImplementation Guide - process steps\n\nThe total video content for the implementation guide is more than 2 hours. I will be publishing it in 4 Parts over the coming week or so. Along with the GitHub Repo, Codes, Schemas, Blueprints, etc. Free to use.\nPart 1 : Demo / Architecture / Agent Process Tracing (this one)\nPart 2 : ReAct LLM Agent in Flowise AI and Make.com workflow\nPart 3 : Setup Google App Scripts and connect to Make.com\nPart 4 : Custom Frontend with AI-Assisted Tools. From develop to live deploy.\nPart 5 : Integrate rest of ReAct Agents & Chatflows in Flowise. Deploy to Vercel. Go Live"
},
{
"slug": "how-to-build-voice-based-ai-action-agents-app-to-execute-tasks-automate-reports-and-analyze-data",
"title": "How to build Voice-based AI Action Agents App to Execute Tasks, Automate Reports, and Analyze Data …and more.",
"tags": [
"voice-ai"
],
"content": "How to build Voice-based AI Action Agents App to Execute Tasks, Automate Reports, and Analyze Data …and more.\n\nIn this article I show how to build a voice-powered AI agentic app to automate business tasks and analyze live data warehouses.\n\nNEW\n\nMy open-source platform with a ton of micro-apps and tooling's for AI driven analytics and data science\n\nThis 5 part series has very detailed video guides for numerous reusable components that can be used across many apps\n\nThis app has a non-realtime implementation of voice AI agents. For realtime voice AI agent app implemented with OpenAI's new WebRTC check out the posting below\n\nChat is just one of the many things LLMs can do…They can act and execute.\n\nVTEXER: Voice-Enabled LLM Action Agent App for Task Execution Automation Query and Research. I demonstrate how you can, with natural language voice instructions:\nUpdate Excel Sheet\nUpdate Google Sheet\nUpdate records in remote Data Warehouses\nQuery, Analyze and Transform data in remote Data Warehouses\nGenerate and email print formatted report in PDF format\nGenerate and email deck format\nCarry out custom web search\nProvide user menu option to choose amongst multiple LLM Agents\n\nAnd yes, it can also chat.\n\nDetailed, step-by-step, hands-on implementation guide, source code, GitHub repo, Schemas, Blueprints shared in sections below.\n\nThe source deployed as-is on Vercel etc. will give you a fully functional voice-bot right off the bat (with restricted app features)\n\nBy leveraging platforms like Flowise AI and Make.com, you can interact with and execute tasks in voice and natural language text with: your business data warehouses, thousands of platforms (Zoho, QuickBooks, Salesforce, AWS, Google, Microsoft, Azure, Slack HubSpot, Stripe, Woo Commerce, Shopify….) and most importantly your own existing API's\n\nThe implementations are relatively rapid, with part of it being no-code, and any coding being done by AI-assisted coding tools.\n\nAs part of the series, I show you how to use AI Assisted Coding Tools like Claude Dev and Cursor AI to develop LLM Apps with natural language commands. And deploy to open internet.\n\nThe app uses LLM ReAct Agents, Reasoning and Action Agents. Called as such since they can execute tasks via API calls, also called Function Calls / Tool Calling. Fairly easy to setup in Flowise. Flowise has a marvelous functionality to access an agent / chat flow via an API endpoint. Allows you to fully customize the UI and response flow. Using that extensively here\nCustom Frontend: React.js\nRapid Deploy Frontend: Flowise\nLLM ReAct Agents: Flowise (Flowiseai.com)\nWorkflow: Make.com\nAutomation Scripts: Google Script\nAWS MySQL DB Connection: Flowise ReAct Agent & Custom built FastAPI Server\n\nALL coding done by GPT, LLMs & AI Assisted Coding Tools. For Google Scripts and Python I like GPT-4o. For React in general it is Claude Sonnet. For React.js apps like this one, my current favorites are Claude Dev, a VS Code extension and Cursor AI IDE, a VS Code fork. Claude Dev works on Github Codespaces also… I have a bit of preference for cloud over local. Both are able to create new files as well as modify files across whole codebase. Claude Dev even has terminal access, so does pretty much everything.\n\nHands on Implementation Guide\n\nVideo guide\n📌 Part 1: Demo and Agent Process Flow\n📌 Part 2 : How to update Excel, Google Sheets and Databases with AI / LLM Voice Agents\n📌 Part 3: How To Automate Excel to PDF, Excel to Slides, Automatic Email with AI Agents and Google Scripts\n📌 Part 4: How to build AI App with natural language with Claude Dev and Cursor AI\n📌 Part 5: How to create AI/ LLM Agents that query databases, do web search and take action\n\nSource code, JSON Schemas and blueprints on GitHub:\n\nThe repo, deployed ASIS to Vercel / Netlify etc will give you a fully functional voice-bot (with restricted app features)\n\nTop Flowise AI Resource:\n\nLeon Van Zyl's videos are a must-watch - from simple chatbots to complex agentic flows. Leon van Zyl - YouTube\n\nAI Assisted Coding: Must Read\n\nImplementation Videos with Time-Stamped Sections\n\nComplete Playlist at: YouTube\n\nPart 1: Demo and Agent Process Flow\n00:02 - Introduction and LLM Agent Capabilities\n01:10 - LLM Agent Demo: Updating Excel and Google Sheets\n03:31 - Generating Report and Deck Formats\n05:12 - Data SQL Analyst and Other Functionalities\n08:06 - Overview of Architecture and Flow\n09:47 - Tracing the Flow of the Agent\n14:01 - Details on Data SQL Analyst and General Analyst\n17:50 - Build Process and Implementation Overview\n18:51 - Breakdown of the Four-Part Guide\n\nPart 2 : How to update Excel, Google Sheets and Databases with AI / LLM Voice Agents\n00:00 - Overview and Introduction\n03:55 - Setting up Flowise and Learning Resources\n05:10 - Top Tips for Working with GPT\n10:10 - Building the Chat Flow in Flowise\n11:30 - Custom Tool for Function Calls in Flowise\n17:15 - Connecting Flow Wise to Make.com\n20:10 - Testing the Flow and Updating Google Sheets\n22:54 - Connecting and updating Google Sheets and Excel Sheets\n29:10 - Connecting Flow Wise to AWS RDS MySQL\n35:00 - Adding Voice Functionality to the Flowise Bot\n38:00 - Connecting Flow Wise to a React Frontend\n41:00 - Deploying the React Frontend Using Vercel\n43:00 - Conclusion and Next Steps\n\nPart 3: How To Automate Excel to PDF, Excel to Slides, Automatic Email with AI Agents and Google Scripts\n00:03 - Introduction and Recap\n00:36 - Creating Automation Scripts in Google Script\n07:38 - Deploying Google Script and connecting to Make.com\n10:38 - Configuring Triggers and Filters in Make.com\n14:26 - Demonstrating the Functioning Web App and Chatbot\n17:29 - Connecting React Frontend to Flowwise\n21:10 - Deploy and Go Live with Vercel\n\nPart 4: How to Build AI Voice Action Agent App with natural language with Claude Dev and Cursor AI\n00:04 - Introduction to Part Four and Custom Frontend\n00:36 - Deploying the Voicebot on a Website and Repository Setup\n01:36 - Overview of Voicebot Components: Voice to Text, Chat Completion, Text to Speech\n04:18 - Building the Demo App with Claude Dev and Cursor AI\n15:26 - Deploying the Voicebot App Live on Vercel\n16:38 - Connecting GitHub Repository to Vercel for Deployment\n17:12 - Final Testing of Live Voicebot App\n19:03 - Demonstrating Multilingual Support in Voicebot\n20:21 - Differentiating Between Transcription and Translation in API Usage\n21:22 - Overview of API Endpoints for Speech and Text Processing\n21:49 - Routing Text to LLM Agent for Response Generation\n23:58 - Finalizing API Integrations for Voicebot Functionality\n25:48 - Conclusion and Next Steps for Voicebot Enhancement\n\nPart 5: How to create LLM Agents that query databases, do web search and take action\n00:04 - Introduction and Overview\n00:32 - Analyst Team and LLM Agents\n04:16 - Configuring Perplexity Search Tool\n15:09 - Implementing Session IDs in React\n16:08 - Demonstrating Session Memory\n18:07 - Integrating Data SQL and Document PDF Analysts\n23:50 - Pushing to GitHub and Deploying to Vercel\n25:42 - Conclusion\n\nHope you find the guide useful and enjoy using it as much as I enjoyed making it."
},
{
"slug": "how-to-summarize-analyze-youtube-videos-with-ai",
"title": "How to summarize & analyze YouTube videos with AI: Two FREE and EASY options",
"tags": [
"converters-tools"
],
"content": "How to summarize & analyze YouTube videos with AI: Two FREE and EASY options\n\nAre there YouTube videos where you need highlights or a detailed written report... or maybe both? For instance, earnings calls, tutorials, documentaries, or lectures?\n\nHere are 2 free and easy options:\n\n1️⃣ Gemini Pro 1.5 : 1 Million token context window\n\nJust download an MP4 version of the YouTube video and upload it to Gemini.\n\nShare your instructions/prompts. (Sharing some sample prompts in comments.) Modify as per requirement.\n\nIt has a 1 Million token limit.\n\nFor instance, Google (Alphabet) 2023 10K filing of 97 pages is around 80K tokens... Larsen & Tubro's 700-page Annual Report for FY23 is ~600K tokens.\n\nA million tokens will accommodate a one-hour video.\n\n👉 How to get an MP4 version of a YouTube Video?\n\nMany public websites have download tools, or you can use the YouTube Downloader tool on my website at tigzig.com.\n\n2️⃣ Get raw transcript & upload to AI\n\nGet transcript and upload it to an AI Chatbot with a large enough context window.\n\nCheck out Claude (claude.ai), Mistral AI (chat.mistral.ai), and Google Gemini Pro 1.5. All of them have a large context window and a generous free tier.\n\n👉 How to get transcript of YouTube Video?\n\nDirectly from YouTube / third party sites / Transcript Extractor Tool on my website. These only work if transcript enabled for YouTube video....which is usually the case.\n\n👉 What if transcript not available?\n\nCreate one using a Speech-to-Text Tool. Many websites would have it. Or use the one on my website."
},
{
"slug": "how-to-update-excel-google-sheets-and-databases-with-ai-voice-agents",
"title": "How to update Excel, Google Sheet and backend Databases with Natural Language commands with Voice Agents",
"tags": [
"voice-ai",
"database-ai"
],
"content": "How to update Excel, Google Sheet and backend Databases with Natural Language commands with Voice Agents\n\nWith Flowise and Make.com. Two Hours. Two Go-Lives.\n\nDeployment with Flowise is stand-alone and fast. Does not need any of the custom UI code in the Github Repo.\n\nIn this Part 2 of series on Building AI / LLM Voice Action Agents, I show how to update Excel, Google Sheet and backend Databases with Natural Language commands with Voice Agents. Detailed hands-on implementation guide available on you tube. 45-minute video:\n\nSource code, deployable ASIS, at Github. With JSON Schemas.\n\nThis source code deployed ASIS give a fully functional voice bot (with restricted app features).\n\nDemo App, available at vtexer-shared.tigzig.com\n\nUse the demo OpenAI API key shared at:\nReplicating repo ASIS: 10 mins to Fork/ Clone. 1min to deploy Vercel.\nVideo Time : 45 min\nFlowise Tool Setup with JSON Import: 15 min\nFlowise ReAct Agent Setup with JSON import: 15 min\nMake.com WorkFlow setup and Webhook connect: 30 min\nFIRST GO-LIVE with Flowise Frontend with Voice Input\nCopy ReAct Agent URL, Commit, Push and AutoDeploy= 5 mins\nSECOND Go-Live with Full Voice Bot UI\n\nAdvanced features like latency reduction, voice activity detection, and auto-send-on-pause coming soon\n\nMake.com provides easy integration with over 1000 platforms (like Zoho, Salesforce, AWS). FlowiseAI (YC S23) provides a powerful platform to rapidly deploy agents and agent teams plus ease of integration with Make.com as well as your own existing business API endpoints\n\nWhat this means:\n\nWith voice and text-based natural language, you can now interact with any of these 1000s of platforms, Data Warehouses, as well as your existing business technology API backend, to communicate, interact, and execute tasks. Fairly Rapidly.\n\nWhat's next:\nPart 3: Google Script Automation\nPart 4: Custom React Frontend\nPart 5: More ReAct Agents in Flowise & Deploy to Vercel"
},
{
"slug": "instant-database-setup-for-ai-apps-with-neon-com",
"title": "Instant Database Setup for AI Apps. With Neon.com",
"tags": [
"database-ai",
"infrastructure"
],
"content": "Instant Database Setup for AI Apps. With Neon.com\n\nYou're building an AI app. A user hits 'create'. Your app needs to spin up a database. How long do they wait?\n\nRegular cloud databases take 3-5 minutes or longer to setup, need manual configuration, charge 24/7 even when idle. Doesn't work when you need databases created on-demand, within an API call.\n\nI use neon.com for this. Neon setups up a full Postgres instance in under 1 second via API. Not a sandbox. A real database with connection string, credentials, everything.\n\nI use it in my data app at app.tigzig.com/analyzer. The app works with remote databases, but when users upload CSV files without an existing database, I need to give them something to work with.\n\nMy workflow: User uploads CSV → API call to Neon → Postgres instance created → Username and password generated → Database connection checked → AI schema detection runs on the CSV → Data pushed to Postgres.\n\nThe whole sequence happens in seconds. The user sees a working database, data loaded and AI all ready to answer questions as schema shared with it as part of workflow.\n\nI currently have 6 live client workflows running on free tier. Small businesses getting live analytics and automation with low database admin overhead.\n\nLarge scale use cases\nReplit Databases : uses Neon to instantly setup databases for its AI Agent\nRetool : manages over 300,000 customer databases with a single engineer\nVercel : Vercel Postgres is powered by Neon- instant, serverless databases\n\nGenerous free tier\n\nUp to 30 projects. Each project supports unlimited databases. For small interactive databases, you can fit 30-40 databases in one project. Each project gets 0.5GB storage and 100 CU-hours per month with autoscaling up to 2 CU (2 vCPU / 8 GB RAM). Total: 15GB storage across all projects. More than enough for development and testing as well as live small business use cases.\n\nDatabase provisioning solved. In under a second. Via API."
},
{
"slug": "intelligent-ai-web-scraper-in-excel-with-python-xlwings-lite",
"title": "Intelligent AI Web Scraper in Excel with Python (xlwings Lite)",
"tags": [
"xlwings-lite",
"python-in-excel"
],
"content": "Intelligent AI Web Scraper in Excel with Python (xlwings Lite)\n\nTell it what to extract - fields, filters, rules - all in plain English.\n\nLive Tool. Paste a URL list. The app extracts the fields you asked for, applies your transforms, filters and any data cleaning instructions. And updates structured data in Excel, URL status, error logs, and a 30-KPI scraping dashboard.\n\nWhat Intelligence does\n\nYou can share instructions in conversational English.\nFilter: \"Don't include any records from Mumbai, Chennai and Delhi\"\nSelect: \"If there are multiple phone numbers, just keep the first one.\"\nNormalize: \"If a state is abbreviated (e.g., UP), replace with the full name (e.g., Uttar Pradesh).\"\nDerive: \"Combine first name and city into a unique ID. No spaces. All caps.\"\n\nHow it works\n\nJina.ai fetches the text from URL. Gemini extracts and transforms. xlwings Lite runs it all. Control LLM Intelligence by tweaking LLM parameters: topP, temperature, max tokens, and thinking budget. Configure scrape parameters like request delays, max retries and timeout settings. Non-deterministic by default. For stricter pulls, add a Python parser layer or a hybrid. Select LLMs by use case: cheap-fast for volume, slower-stronger for precision.\n\nHow to customize\n\nI use it as-is for common client scrapes and customize for tougher cases. To customize, hand the code to your AI Coder with change requests. Examples: switch to OpenAI or Anthropic, capture images, follow child URLs, add a python parser step. The foundation is xlwings Lite. I've documented my process and examples in the xlwings Practice Lab: xlwings-lite.tigzig.com. Refer to my blog posts (links below) on choosing an AI Coder.\n\nLive Tool, Source Code & Docs\n\nBuilt with xlwings Lite by Felix Zumstein (lite.xlwings.com)\n\nResources\n\nAI Coder Instruction file for xlwings Lite\n\nxlwings Lite official site: lite.xlwings.org"
},
{
"slug": "large-file-upload-for-database-ai-text-to-sql-apps",
"title": "Large File Upload for Database AI Text-to-SQL Apps: A Practical Guide",
"tags": [
"database-ai",
"text-to-sql",
"fastapi"
],
"content": "Large File Upload for Database AI Text-to-SQL Apps: A Practical Guide\n\nEverything I learned uploading 1.6GB files through a FastAPI backend without crashing the server\n\nLarge File Upload for Database AI\n\nThe Use Case\n\nUpload large CSV/data files to be made available to a Text-to-SQL agent for natural language querying and analysis.\n\nThe Challenge\n\nUploading large files to PostgreSQL (or MySQL) through a web API sounds simple until you try it with a 70MB compressed file that expands to 1.6GB. Your app crashes, memory explodes, connections timeout, and users see cryptic 500 errors. And it gets worse with multiple concurrent users.\n\nThis guide documents 30+ patterns I implemented to handle large file uploads reliably.\n\nLive App and Source Code\n\nApp live here: app.tigzig.com/analyzer. Pick a dataset, click \"Use Temporary Database\". That's it. Post upload, schema automatically sent to the AI Agent. Start querying in natural language. Each dataset has a pre-built prompt. Copy, paste, run. Or use your own. The more specific, the better.\n\nThe app has sample datasets ranging from 64 rows to 11.8 million rows (1.6 GB). Setup takes 30 seconds for small files. Around 2 minutes for the 1.6 GB file. 9 LLMs available: Gemini 2.0 Flash (cheapest) to Claude 4.5 Sonnet (best quality).\n\nFor more info on datasets, how to explore and other app capabilities:\n\nTry Text-to-SQL on Real Data: GB Files, Multi-Million Rows\n\nApp is open source. For source code just hit \"Docs\" on app site.\n\nBACKEND FILE HANDLING\n\n▸ Don't Decompress Large Files in Memory\n\nA 70MB .gz file decompresses to 1.6GB. Loading into RAM crashes your server. Stream to disk with chunked reads (32MB chunks). Read chunk, write to temp file, repeat.\n\n▸ File Size Threshold Detection\n\nYou don't want chunked processing overhead for small files. Detect file size from Content-Length header. Use 100MB as threshold. Below 100MB use memory, above use disk streaming. For compressed files always use chunked (you don't know final size).\n\n▸ Gzip Decompression Streaming\n\nStandard gzip.decompress() loads entire file to memory. Stream decompress in 32MB chunks. Track metrics: original size, decompressed size, compression ratio, speed.\n\nDATABASE INSTANCE CREATION (NEON)\n\n▸ Instant PostgreSQL with Neon API\n\nUser needs a fresh database. Traditional provisioning takes minutes. Use Neon's serverless Postgres API - creates databases in seconds.\n\n▸ Race Condition in Database Creation\n\nWhen creating a Neon database, you create a role then a database. Sometimes database creation fires before the role is ready. Add 500ms delay between role and database creation. Implement retry with exponential backoff (1s, 2s, 4s, 8s). Max total wait 15 seconds. OR use polling for an even more robust process.\n\n▸ Project Lock Handling\n\nNeon returns HTTP 423 (Locked) when a project is temporarily busy. Detect 423, wait with exponential backoff, retry up to 4 times.\n\nASYNC HANDLING\n\n▸ Make Your Endpoints Async\n\nSynchronous endpoints block the event loop. One slow upload blocks all other requests. Use async def for all upload endpoints.\n\n▸ Async File Reading\n\nfile.read() is blocking even in async endpoints. Use await file.read(chunk_size) for chunked async reads.\n\n▸ Thread Pool for Blocking Database Operations\n\npsycopg2 and mysql.connector as well many file I/O and CPU bound tasks are synchronous libraries. Calling them in async endpoints doesn't work and still blocks the event loop. Run blocking operations in thread pool executor using asyncio.geteventloop().runinexecutor().\n\nREMOVE BLOCKING OPERATIONS\n\n▸ Pandas to Polars\n\nPandas operations are slow. pd.readcsv() on a 500MB file freezes your server. Replace Pandas with Polars. Benefits: lazy evaluation with scancsv(), native streaming with collect_chunks(), Rust-based so significantly faster, better memory efficiency.\n\n▸ Identify Every Blocking Operation\n\nEven a single blocking operation in an otherwise async endpoint chokes the entire application. Audit your upload/processing flow for significant blocking calls - file I/O, database operations, external API calls. A 100ms blocking call in a high-traffic endpoint kills concurrency. For low-traffic internal tools, prioritize the big blockers first.\n\nCONNECTION POOLING\n\n▸ Why Connection Pooling\n\nCreating a new database connection per request is slow (100-300ms) and exhausts server resources. Use ThreadedConnectionPool from psycopg2. Set minconn=1, maxconn=10.\n\n▸ Per-Database Pools\n\nDifferent users connect to different databases. One pool per database needed. Create pools keyed by connection details. Key format: host:port:database:user.\n\n▸ Stale Connection Detection\n\nPooled connections go stale (timeout, network issues, server restart). Using a stale connection causes cryptic errors. Validate connection before use with SELECT 1 query.\n\n▸ Stale Connection Retry\n\nEven with validation, you might get multiple stale connections in a row. Retry up to 3 times with validation. If all retries fail, create fresh connection outside pool.\n\n▸ Thread-Safe Pool Access\n\nMultiple async requests accessing pools simultaneously causes race conditions. Use asyncio.Lock() for pool operations.\n\nTIMEOUTS\n\n▸ Connection Timeout\n\nConnecting to an unavailable database hangs forever. Set connect_timeout=30 seconds.\n\n▸ Statement Timeout\n\nLong-running queries block connections and cause cascading failures. Set statement_timeout=900000 (15 minutes in milliseconds) at connection level.\n\n▸ HTTP Request Timeout\n\nExternal API calls (OpenAI, Neon) can hang. Set timeout=60 seconds on HTTP clients.\n\n▸ Upload Timeout\n\nLarge file uploads take time. Default timeouts kill them. Extended timeouts for upload endpoints. Use 15 minutes (900 seconds) for large uploads.\n\nREQUEST BODY LIMITS\n\nDefault body size limits reject large uploads. Custom middleware to allow up to 1.5GB bodies on upload endpoints only.\n\nRATE LIMITING\n\n▸ IP-Based Rate Limiting\n\nOne client can overwhelm your server with too many requests. Use slowapi library for IP-based rate limiting. Set 300/hour (1 request per 12 seconds average).\n\n▸ Rate Limit Hit Logging\n\nYou need visibility into who's hitting rate limits. Middleware that logs rate limit hits with IP, path, and timestamp when HTTP 429 returned.\n\nAPI KEY SECURITY\n\nSensitive endpoints (database creation) need protection. API key in Authorization header with Bearer scheme. Use secrets.compare_digest() for constant-time comparison to prevent timing attacks.\n\nBATCH PROCESSING FOR MYSQL\n\n▸ Chunk Size Selection\n\nInserting millions of rows one-by-one is slow. Inserting all at once exhausts memory. Batch insert in chunks of 25,000 rows. Commit per chunk.\n\n▸ Disable Constraints During Bulk Insert\n\nForeign key checks, unique checks, and autocommit slow down bulk inserts significantly. Disable foreignkeychecks, uniquechecks, autocommit, and sqllog_bin before insert. Re-enable after completion.\n\n▸ MySQL Connection Optimization\n\nUse allowlocalinfile=True, usepure=False (C extension for speed), poolsize=32, maxallowedpacket=1GB.\n\nPOSTGRESQL COPY COMMAND\n\nINSERT statements are slow for bulk data. Use PostgreSQL's COPY command with copy_expert(). Performance is 10-100x faster than equivalent INSERT statements.\n\nSSL/TLS CONFIGURATION\n\n▸ Neon Requires SSL\n\nAlways set sslmode='require' for Neon connections.\n\n▸ Optional SSL for Custom MySQL\n\nSupport optional sslverifycert and sslverifyidentity parameters for custom databases.\n\nTEMPORARY FILE MANAGEMENT\n\n▸ Temporary File Creation\n\nUse tempfile.NamedTemporaryFile with delete=False and suffix='.csv'. Manual cleanup required.\n\n▸ Guaranteed Cleanup\n\nTemp files pile up on errors. Clean up in finally block always. Check if file exists before deleting. Log cleanup errors but don't crash.\n\n▸ Middleware Cleanup for Query Results\n\nMiddleware that automatically deletes temp files after FileResponse is sent.\n\nERROR HANDLING PATTERNS\n\n▸ Try-Except-Finally Structure\n\nCatch HTTPException and re-raise as-is. Catch specific database errors (psycopg2.Error). Catch general Exception as fallback. Always clean up in finally block.\n\n▸ Specific Exception Handling\n\nHandle gzip.BadGzipFile with HTTP 400. Handle psycopg2.OperationalError for connection issues. Provide meaningful error messages.\n\n▸ Fallback Mechanisms\n\nTry primary method, log warning on failure, try fallback method. Example: Polars primary, Pandas fallback.\n\nPROGRESS TRACKING AND LOGGING\n\n▸ Multi-Level Timing\n\nTrack overall time, file processing time, schema detection time, database insert time. Log breakdown at completion.\n\n▸ Chunk Progress Logging\n\nLog chunk number, rows in chunk, and running total after each chunk processed.\n\n▸ Log Tags for Filtering\n\nUse consistent tags like [TIMER] for performance, [POOL] for connections, [GZ] for compression. Makes log filtering easy.\n\nDELIMITER DETECTION\n\nFiles come with different delimiters (comma, tab, pipe). Read first few lines, detect delimiter. Priority order: Tab, Comma, Pipe. Default to comma.\n\nCOLUMN NAME SANITIZATION\n\nColumn names with spaces, special characters, or SQL keywords break queries. Remove special characters, replace spaces with underscores, handle SQL keywords, ensure name starts with letter.\n\nSTREAMING RESPONSE FOR EXPORTS\n\nExporting large query results to file loads everything to memory. Stream results with a generator function. Fetch in chunks (10,000 rows). Yield header row first, then data rows. Clean up cursor and connection in finally block of generator.\n\nLLM SCHEMA DETECTION WITH FALLBACK\n\nOpenAI API might be down or rate-limited. Try OpenAI first, fall back to OpenRouter on failure. Log warning when using fallback.\n\nAPPLICATION SHUTDOWN CLEANUP\n\nConnection pools need proper cleanup on shutdown. Register shutdown event handler that calls closeallpools() on the pool manager.\n\nUVICORN WORKERS AND TIMEOUT CONFIG\n\nDefault uvicorn runs single worker - one slow upload blocks everyone. Use multiple workers for concurrent requests. I use 4 workers depending on server capacity. Set timeout-keep-alive high for large uploads - default 5 seconds kills long-running connections. I use 1800 seconds (30 min).\n\nPOSTGRES OVER MYSQL FOR LARGE FILE OPERATIONS\n\nIn my testing, Postgres handles large file uploads significantly better than MySQL. COPY command is faster and more reliable than batched INSERT. Connection handling is smoother. MySQL was fragile with large files - frequent timeouts, connection drops, inconsistent behavior. If you have a choice, use Postgres for bulk data operations.\n\nQuick Reference\n\nConfiguration I Use (adjust based on your use case):\nChunk size for reading: 32MB\nChunk size for inserting: 25,000 rows (MySQL only)\nConnection timeout: 30 seconds\nStatement timeout: 15 minutes\nRate limit: 300 per hour\nMax body size: 1.5GB\nFile size threshold: 100MB\nUvicorn workers: 4\nUvicorn timeout: 1800 seconds (30 min)\n\nMemory Benchmarks with Chunked Processing (Postgres):\n100MB file: 62% memory reduction (400MB to 150MB)\n500MB file: 90% memory reduction (2GB to 200MB)\n1GB file: 94% memory reduction (4GB to 250MB)\n\nConclusion\n\nLarge file upload is death by a thousand cuts. Each optimization solves a specific problem:\nMemory issues: Chunked streaming\nSlow inserts: Batch processing and COPY command\nConnection issues: Pooling and stale detection\nBlocking issues: Async and thread pool executor\nPandas issues: Polars with fallback\n\nNo single solution works for everything. The key is layered defenses: detect the file characteristics, select the right processing path, handle errors gracefully, and always clean up.\n\nNotes:\nMy tools are deployed with small businesses and small teams. For enterprise grade with hundreds or thousands of users, additional considerations would apply.\nPublic app routes all DB credentials and API calls through my backend. It's a sandbox only for testing. Deploy on your own servers for live work."
},
{
"slug": "leave-all-programming-to-ai-a-data-scientists-perspective",
"title": "Maybe leave programming to AI. Coding by GPTs: A Data Scientist's Perspective",
"tags": [
"ai-coders"
],
"content": "Maybe leave programming to AI. Coding by GPTs: A Data Scientist's Perspective\n\nThe title of the post is a quote from Jensen Huang, CEO of NVIDIA. Couldn't agree more. And a sentiment shared by many fellow analysts.\n\nWith over two decades coding and executing data science and analytics projects, GPTs have significantly increased my efficiency. Particularly in code generation, allowing me to focus more on output quality and deployments.\n\nGPT MAGIC\n\nCoding: Best done by AI. Just can't beat the quality and the speed. Ginormous time savings\n\nPRACTICAL CONSIDERATIONS\nImpact: A beautiful code by itself doesn't create any impact. It's part of a bigger pipeline.\nGrind Still Exists: e.g. API stuff was new to me (FastAPI, GET/POST, transfers). Understanding took time... and then coding was a breeze as I knew exactly what to ask of GPTs\nIt's not instant coffee: We still need to iterate, check outputs, modify code, debug... until it works. No change there. GPTs do make it faster.\nDomain Expertise Matters: e.g. never worked on healthcare analytics. No amount of GPT code would make me an expert quickly\nLanguage understanding matters: Things go very fast with Python. But, React-Node.js web app? I realized quickly that just dumping codes that I don't understand doesn't work well. Though GPTs do speed up learning.\nGPTs make mistakes and get stuck sometimes, needing \"hooman\" help (identifying issues, sharing docs/code samples).\nCode privacy a likely challenge, but solutions are in development.\nThey can't do everything. For instance, deployments (yet. I think..)\n\nWith Devin and other full-stack bots coming up, this might all change soon. Eagerly awaiting.\n\nCODING BY GPT: EXAMPLES\n\nTools I've built using GPT (links in the comments). While small in scale, these tools have helped me understand GPT's application in coding, particularly in areas like LLM apps, APIs, and deployments.\nYFIN Bots: Data transfers via FastAPI endpoints / Flowise AI Platform. Demonstrate GPT's potential for building data pipelines, as well as LLM app deployment within and outside GPT Store.\nLlama Parser Widget: File processing & transfers via API for LLM apps\nBuilding ML Models with ChatGPT: Demonstrates GPT4's ability to generate ML code, as well run ML models based on prompts\nMutual Funds Allocation Analyzer (GPT): Automation and data manipulation across multiple Excel files.\n\nAll coding by ChatGPT3.5, ChatGPT4, and Gemini ~equally. End-to-end time exponentially quicker than what I could have done alone. Free GPTs are also exceptional.\n\nSam Altman, \"AI could boost programmers' productivity by 30X. Totally agree."
},
{
"slug": "live-python-in-excel-with-xlwings-lite",
"title": "Live Python-in-Excel systems - built with xlwings Lite. AI, Scraping, APIs, EDA, DB, Charts, PDFs, Automations",
"tags": [
"xlwings-lite",
"python-in-excel"
],
"content": "Live Python-in-Excel systems - built with xlwings Lite\n\nAI, Scraping, APIs, EDA, DB, Charts, PDFs, Automations\n\nxlwings Lite turns Excel into a live Python app - from simple automations to complex API based workflows. Just a lightweight add-in, no Python install needed. Built-in code editor, console, environment variables, and deep Excel integration. Built by Felix Zumstein - creator of original xlwings.\n\nHere's a quick roundup of apps I've built with xlwings Lite across a range of use cases. Links include files, code, and setup docs. These are live working apps - while the examples shown are around representative use cases, the architecture supports complex, production-grade workflows. Adapt as needed for your own use cases.\nAI Web Scraper - Input URLs, define columns, set rules. API calls behind the scenes: Jina AI pulls pages, Gemini extracts structured data. Output goes straight into Excel tables. tigzig.com/web-scraper\nAI Technical Analyst - Pulls Yahoo Finance data, builds charts and indicators, then sends to Gemini Vision for analysis. Output: clean PDF and web reports. Useful base for API based workflows, FastAPI backends, AI analysis and formatted reports. tigzig.com/technical-analysis-report\nAI Schema Detection and Automation - Handles data with varying columns and types. Uses Gemini API for schema detection, then feeds it into the next automation step. tigzig.com/xlwings-llm-api\nDatabase Connector - Connect to any remote DB via an API layer. Built on a FastAPI server - just pass query params….pull schema, browse tables, fetch records, or run custom queries. All inside Excel. tigzig.com/xlwings-api-db\nEDA + ML Workflow - Builds on schema detection to run dynamic EDA plots and summary tables. Includes an XGBoost response model with evals, decile tables, and gains chart. tigzig.com/xlwings-llm-api\nYahoo Finance Extractor - Punch in a symbol - get price, profile and financials directly in Excel. Powered by a FastAPI backend (also doubles as an MCP server). Great starter for getting into FastAPI backends. tigzig.com/yahoo-finance-analyzer\n\nxlwings Lite official site: lite.xlwings.org"
},
{
"slug": "llama-parse-pdf-analyze-with-chatgpt-rag",
"title": "How to use Llama Parse to convert PDF to text and extract complex table data. For Annual Reports, 10Ks, Research Reports",
"tags": [
"converters-tools"
],
"content": "How to use Llama Parse to convert PDF to text and extract complex table data. For Annual Reports, 10Ks, Research Reports\n\nNEW: TIGZIG: Co-Analyst\n\napp.tigzig.com - my open-source platform with 25+ micro-apps and tooling's for AI driven analytics and data science.\n\nIncluding a Llama Parse PDF to Markdown converter\n\nExtracting data, especially table data, from complex PDFs with tables used to be a challenge. But with the launch of LlamaParse by LlamaIndex, that period is now over.\n\nOriginally published on LinkedIn. Embedded post below.\n\nNote for developers doing the conversion themselves with Python/JS scripts:\nThe API call works faster than the Python package.\nChunking the file before parsing improves speeds.\nCurrently, around 50 pages seems to be the optimal chunk size.\nParsing is faster when done in 50-page chunks versus the full file at once, even for say a 100-page report.\nTested chunk sizes between 25 to 100 pages, with less than 50 or more than 50 pages increasing the conversion time.\nHowever, all this can change rapidly as LlamaParse is evolving quickly. For example, just a few days back they increased file size limit from 200 to 700 pages."
},
{
"slug": "llm-costing-for-database-ai-apps-live-experience-live-app-open-source",
"title": "LLM Costing for Database AI Apps. Live Experience. Live App. Open Source",
"tags": [
"database-ai",
"text-to-sql"
],
"content": "LLM Costing for Database AI Apps. Live Experience. Live App. Open Source\r\n\r\nOn costing, a hard rule: Published API rates are a fantasy. Always test your specific use case and check the actual charged cost. Takeaways from 250+ structured tests and live experiences:\r\n\r\nKey Findings\rThe Workhorse (80% of jobs): Single-step agents. Cost: approx. $0.50 - $2.00 per 100 questions.\rThe Specialist (Critical tasks): Multi-agent setups. Warning: a 10X-50X cost multiplier. Cost $15-$20 per 100 questions. Use surgically.\rThe Top Tier (Reasoning): Claude Sonnet 4 still leads. GPT-5 is close but its costs are volatile. No client migrations till cost and latencies stabilize.\r\n\r\nDatabase AI Suite - v4 (DATS-4)\r\n\r\nDATS-4 is live. It's an open-source full-stack tool for natural language to SQL applications.\r\n\r\nLive Apps. Source Code. Full 49-page Field Guide. All Available now.\r\n\r\nAccess: app.tigzig.com - Path: Database AI & SQL Apps"
},
{
"slug": "mcp-server-bot-attack-security-lessons",
"title": "My Public MCP Server Got Hammered - Security Lessons from a Bot Attack",
"tags": [
"infrastructure",
"security",
"mcp"
],
"content": "My Public MCP Server Got Hammered - Security Lessons from a Bot Attack\r\n\r\nMy public database MCP server got hammered yesterday. Bot attack. Database server CPU at 100%.\r\n\r\nHad protections in place - rate limiting, read-only access, SQL validation. Still got hammered. The gaps were in what I didn't think about.\r\n\r\nWorked with my AI Coder (Claude Code) to get it back up, then went deep into every attack vector. Some 15 odd fixes - from exposed server IPs to open system catalogs to a Supabase REST API I wasn't even thinking about as an entry point.\r\n\r\nWhat I learned fixing this in a few hours - would have taken days of watching tutorials. Getting bashed is the fastest way to learn. And all free.\r\n\r\nTwo things worth noting:\r\n\r\nAll my client apps have mandatory auth. My public demo apps is where I get negligent. This was a good reminder.\r\n\r\nAnd I wasn't listening to my own AI coder. Every app build, I ask Claude Code for a security audit. It gives me a long list. Most times I think .. let me go live and I'll come back to this...well, this is what happens.\r\n\r\nServer is back up. Open to public. No login.\rMCP: https://db-mcp.tigzig.com/mcp\rAPI: https://db-mcp.tigzig.com/api\r\n\r\n... and yes a few things are still remaining.\r\n\r\nDetailed post with all fixes coming soon. And if I get hammered again - more free learning."
},
{
"slug": "mdrift-flexi-cap-focused-fund-composition-analytics",
"title": "MF Composition Analytics With MDRIFT - Interesting Moves in Top Flexi Cap and Focused Funds",
"tags": [
"mutual-funds"
],
"content": "MF Composition Analytics With MDRIFT - Interesting Moves in Top Flexi Cap and Focused Funds\n\nMF composition analytics with MDRIFT throws up some interesting moves in top Flexi Cap and Focused funds. PPFAS holding nearly 12% in US tech, SBI nearly 10% in Google (Alphabet) and HDFC sitting on a 15% cash war chest (Jan 2026 end).\n\nMDRIFT compares mutual fund portfolio compositions against each other - across time periods - entries, exits, weight changes, all precomputed - with filters, drill downs etc. Nifty indices now added to quickly spot off-Index bets and bullish / bearish stances.\n\nPPFAS Flexi Cap\n\nBetting heavily on Google (Alphabet, 4.4%), Meta, Amazon. Nearly 12% in US tech. Sitting on 6% cash (TREPS). Big on Power Grid, Coal India, Bajaj Holdings. Avoiding Reliance, L&T, SBI, HDFC Bank, Infosys.\n\nHDFC Flexi Cap\n\nA very different strategy - 15% cash (TREPS) war chest. No foreign stocks. Bigger bets on Axis Bank, SBI Life, Cipla. Avoiding Reliance, L&T, SBI, HDFC Bank, Infosys, TCS.\n\nSBI Focused\n\nA mix - big on Alphabet with approx. 10%, holding approx. 11% in cash, and heavy on Muthoot Finance.\n\nKotak Flexi Cap\n\nTaking a different angle - nearly fully invested at 2% cash. Betting on Bharat Electronics, Jindal Steel, SRF. Avoiding Reliance, L&T, ITC, Infosys, TCS.\n\nOne Thing in Common\n\nGenerally underweight on the traditional stocks including large IT.\n\nMethodology, validations, ISIN mappings and index comparisons on the app.\nLive app: tigzig.com/mdrift"
},
{
"slug": "mdrift-isin-mapping-process",
"title": "How I Identify and Map Every Holding — The ISIN Mapping Process",
"tags": [
"mutual-funds"
],
"content": "How I Identify and Map Every Holding — The ISIN Mapping Process\n\nISIN Mapping Process\n\nEvery month, each mutual fund publishes an Excel file listing every stock, bond, and instrument they hold. The challenge? These files come from 15 different fund houses, each with different formats, naming conventions, and quirks. The same company can appear under dozens of different spellings. A stock split can change an identifier overnight. And short-term debt instruments like bank CDs can have 10 separate line items for the same bank.\n\nThis document explains — step by step — how I take these messy Excel files and turn them into a clean, comparable dataset where the same company always appears as one row, no matter which fund holds it or how they spell the name.\n\nStep 1: Building the Reference Database (ISIN Master)\n\nEvery security traded in India has a unique 12-character code called an ISIN (International Securities Identification Number), assigned by NSDL (National Securities Depository Limited). Think of it as a universal ID card for every stock, bond, and instrument.\n\nI download two public datasets and merge them into a single reference table:\n\nNSDL Master — The complete register of every ISIN ever issued in India. Over 361,000 records covering equities, bonds, mutual fund units, certificates of deposit, commercial papers, treasury bills, government securities, and more. Each record has the ISIN, the official NSDL name, the issuer, and the security type.\n\nNSE Equity List — The list of all stocks currently listed on the National Stock Exchange, with their trading symbols (like RELIANCE, TCS, HDFCBANK). About 2,000 records.\n\nI merge these by joining on the ISIN code. The result: for every Indian security, I know its official name, and for every NSE-listed equity, I also know its trading symbol.\n\nThis reference table is the backbone of the entire process. When the converter encounters an ISIN in a fund's Excel file, it looks it up here to get the canonical name and trading symbol.\n\nStep 2: Reading the Excel Files\n\nEach AMC (Asset Management Company) uses a different Excel layout. Column positions vary, header rows are in different places, and some AMCs have quirks like putting data across multiple sheets.\n\nThe converter handles this automatically:\nFormat detection — AMC websites sometimes serve files with the wrong extension (a .xlsx file that's actually in the old .xls binary format, or vice versa). The converter reads the first few bytes of the file to detect the actual format, regardless of the file extension.\nSchema auto-detection — The converter scans each sheet row by row, looking for a header row that contains the word \"ISIN\". Once found, it identifies which columns contain the company name, ISIN, quantity, and market value by matching keywords in the header cells.\nGrand total extraction — The converter finds the \"Grand Total\" row in the Excel file and reads the total market value. This number is used later for validation — my calculated total must match this exactly.\n\nFor example, Axis funds have the ISIN in column C, name in column B, and market value in column F. HDFC funds have ISIN in column B, name in column D, and market value in column G. Kotak funds are completely different again. The converter handles all of these automatically, with manual overrides for the few AMCs that don't follow any standard pattern.\n\nStep 3: Identifying Each Holding — ISIN Lookup\n\nFor every row in the Excel file, the converter takes the ISIN and looks it up in the reference database.\n\nIf the ISIN is found — The lookup returns the official NSDL name and NSE symbol. This is the most common case for equity holdings.\n\nFor example, fund houses might spell the same company three different ways:\n\"Infosys Ltd\" (one fund)\n\"Infosys Limited\" (another fund)\n\"INFOSYS LTD.\" (a third fund)\n\nBut they all report the same ISIN: INE009A01021. Looking this up in the NSDL master gives us the canonical name \"INFOSYS LIMITED\" and the NSE symbol \"INFY\". Problem solved — all three are now identified as the same company.\n\nIf the ISIN is not in the master — This can happen for very recent issuances or private placements. The converter falls back to the name as written in the Excel file.\n\nIf there is no valid ISIN at all — Many non-equity items (cash, TREPS, net receivables, futures, margins) don't have standard ISINs. These get assigned a placeholder code IN9999999999 and are handled separately (see Step 6 below).\n\nStep 4: Understanding ISIN Structure — How I Detect Corporate Actions\n\nAn Indian ISIN isn't just a random code. It has a specific structure that encodes information about the security:\n\n``\nINE 238A 01 02 8\n│ │ │ │ └─ Check digit\n│ │ │ └──── Serial number (increments on corporate actions)\n│ │ └─────── Security type (01 = equity, D6/16 = CD, 14 = CP)\n│ └──────────── Issuer code (unique per company)\n└──────────────── Country code + issuer type\n`\n\nReal example — Kotak Mahindra Bank face value split:\nOld ISIN: INE237A01028 — issuer code 237A, security type 01 (equity), serial 02\nNew ISIN: INE237A01036 — same issuer 237A, same security type 01, serial 03 (incremented)\n\nWhen a company does a stock split, bonus issue, or rights issue, NSDL issues a new ISIN with the serial number incremented by 1. The issuer code stays the same because it's still the same company. This is how I detect corporate actions automatically.\n\nMore real examples from my data:\nBEML Limited — INE258A01016 (serial 01) → INE258A01024 (serial 02) after a face value split from Rs 10 to Rs 5\nComputer Age Management Services — INE596I01012 (serial 01) → INE596I01020 (serial 02) after a bonus issue\nMotilal Oswal Financial Services — INE338I01027 (serial 02) → INE338I01035 (serial 03) after a face value split\n\nStep 5: The Name-Cut Process — Finding Potential Duplicates\n\nCorporate action detection relies on two ISINs being from the same company. But how do I even know which ISINs to compare? Comparing every ISIN against every other ISIN would be impractical with 7,000+ holdings.\n\nThis is where the name-cut process comes in.\n\nHow it works:\nTake the company name from the NSDL master (or the Excel file)\nConvert to lowercase and remove all spaces, punctuation, and special characters\nTake the first 7 characters\n\nThis gives us a \"name-cut\" — a short fingerprint of the company name.\n\nExamples:\n\n| Original Name | After Cleanup | Name-Cut (7 chars) |\n|---|---|---|\n| AXIS BANK LIMITED | axisbanklimited | axisban |\n| AXIS BANK CD 08JAN26 | axisbankcd08jan26 | axisban |\n| KOTAK MAHINDRA BANK | kotakmahindrabank | kotakma |\n| KOTAK MAHINDRA PRIME | kotakmahindraprime | kotakma |\n| GOVERNMENT OF INDIA 364D TBILL | governmentofindia364dtbill | governm |\n\nWhy 7 characters? It's a deliberate balance. Short enough to catch the same company across different instrument types (Axis Bank equity and Axis Bank CDs both become \"axisban\"). Long enough to mostly avoid false positives between unrelated companies.\n\nGrouping by name-cut: All ISINs with the same 7-character name-cut are placed into the same group. These are the candidates for being \"the same company under different ISINs.\"\n\nWithin each group, the algorithm does a deeper analysis using the ISIN structure:\n\n5a. Same Issuer Code + Equity Type + Different Serial → Corporate Action\n\nIf two ISINs in the group have the same issuer code (positions 4–7) and both are equity type (01), but different serial numbers — this is a corporate action. The newer serial is the current ISIN; the older one is mapped to it.\n\n5b. Same Issuer Code + CD Type → Certificate of Deposit Aggregation\n\nIf multiple ISINs share the same issuer code but have security type D6 or 16 (Certificate of Deposit), they are separate CD issuances from the same bank. These get aggregated into a single synthetic entry.\n\n5c. Same Issuer Code + CP Type → Commercial Paper Aggregation\n\nSame logic for security type 14 (Commercial Paper).\n\n5d. Different Issuer Codes → No Action\n\nIf the group contains ISINs from different issuer codes, they are genuinely different companies that happen to share a similar name prefix. For example, \"KOTAK MAHINDRA BANK\" (issuer 237A) and \"KOTAK MAHINDRA PRIME\" (issuer 476A) both have name-cut \"kotakma\" but are different entities. These are flagged as NO_ACTION and left as separate holdings.\n\nStep 6: Handling Non-Equity Items — Cash, TREPS, and Foreign Stocks\n\nEquity mutual funds don't just hold stocks. They also hold various non-equity items for cash management, hedging, and regulatory compliance. These items typically don't have standard Indian ISINs.\n\nWhen the converter encounters a holding without a valid Indian ISIN, I apply a three-step classification:\n\n6a. Foreign Stock Check\n\nSome equity funds (notably Parag Parikh Flexi Cap, SBI Contra, SBI Focused, ICICI Value) hold overseas stocks like Alphabet (Google), Meta, Microsoft, and Amazon. These have valid ISINs — just not Indian ones. For example, Alphabet's ISIN is US02079K3059.\n\nThe converter checks the raw ISIN from the Excel file against a curated list of known foreign stocks. If matched, the original US/foreign ISIN is preserved as-is. This allows the dashboard to identify and group foreign holdings correctly.\n\nCurrent foreign stock holdings across my tracked funds:\n\n| Company | ISIN | Held By |\n|---|---|---|\n| Alphabet Inc (Google) | US02079K3059 | PPFAS Flexi Cap, SBI Contra |\n| Meta Platforms | US30303M1027 | PPFAS Flexi Cap |\n| Microsoft Corp | US5949181045 | PPFAS Flexi Cap |\n| Amazon.com Inc | US0231351067 | PPFAS Flexi Cap |\n| EPAM Systems | US29414B1044 | SBI Focused, SBI Contra |\n| Cognizant Technology | US1924461023 | SBI Contra |\n\n6b. TREPS Check\n\nTREPS (Tri-Party Repo) is an overnight lending instrument where mutual funds lend money to other financial institutions through the Clearing Corporation of India. Almost every fund holds TREPS as a cash management tool.\n\nDifferent fund houses report TREPS under different names: \"TREPS\", \"Tri-party Repo\", \"Triparty Repo\", \"CBLO\", \"Collateralized Borrowing\", and others. The converter matches against a list of known patterns and groups all of them under a single synthetic ISIN: SYNTREPS00001.\n\n6c. Everything Else → Cash & Other Assets\n\nItems that are neither foreign stocks nor TREPS — such as net receivables, derivative margins, cash balances, and reverse repos — are grouped under the placeholder IN9999999999 with the display name \"Cash & Other Assets.\"\n\nThe AI-in-the-Loop Review\n\nAfter each monthly conversion, I run a cash classification report that scans all items classified as \"Cash & Other Assets\" for potential misclassifications:\nNew foreign stocks — If a US ISIN appears in the cash bucket, it means a fund has added a new overseas holding that isn't in my list yet. The report flags it for addition.\nNew TREPS variants — If a name containing \"repo\", \"trep\", \"clearing\", or \"cblo\" appears in cash, a fund house may be using a new naming convention for TREPS. The report flags it for review.\n\nThis ensures the classification stays accurate as fund houses change their reporting formats.\n\nStep 7: Synthetic ISINs — Creating Virtual Identifiers\n\nWhen multiple instruments are aggregated into a single entry, I need a new identifier for the group. I call these synthetic ISINs — they follow a naming convention that encodes the source:\n\n| Type | Format | Example | Represents |\n|---|---|---|---|\n| Certificate of Deposit | SYN{issuer}CD01 | SYN238ACD01 | All Axis Bank CDs |\n| Commercial Paper | SYN{issuer}CP01 | SYN296ACP01 | All Bajaj Finance CPs |\n| Treasury Bills | SYNGOITBILL01 | SYNGOITBILL01 | All GOI T-Bills |\n| Government Securities | SYNGOIGSEC01 | SYNGOIGSEC01 | All GOI G-Secs |\n| TREPS | SYNTREPS00001 | SYNTREPS00001 | All TREPS instruments |\n\nThe {issuer} in CD and CP synthetic ISINs is the 4-character issuer code from the original ISIN. This means each bank's CDs get their own synthetic ISIN — Axis Bank CDs, HDFC Bank CDs, and ICICI Bank CDs are separate entries, not lumped together.\n\nWhy aggregate CDs and CPs? A single bank like HDFC Bank might have 5–10 separate CDs with different maturity dates, each with its own ISIN. For an equity fund analysis, knowing that the fund holds \"₹500 Cr in HDFC Bank CDs\" is more useful than seeing 8 separate line items for each maturity date. The individual issuances are still available in the drill-down modal.\n\nWhy aggregate GOI T-Bills and G-Secs? Similarly, a fund might hold treasury bills across 4–5 different auction dates. Grouping them into \"GOI T-BILL\" gives a cleaner picture of the fund's government securities exposure.\n\nStep 8: The Mapping File — Human-in-the-Loop Review\n\nAfter the algorithm generates all mappings, it writes two files:\n\nThe Validation File (for human review)\n\nA detailed text file with every potential duplicate detected, organized by category. Each entry shows:\nThe name-cut group that triggered the analysis\nThe category (CORPORATEACTION, CDAGGREGATE, CPAGGREGATE, TBILLAGGREGATE, GSECAGGREGATE, or NOACTION)\nThe action taken (MAP, AGGREGATE, or NO_ACTION)\nBoth the original and mapped ISINs\nThe original and mapped names\nThe total market value affected\nThe reasoning\n\nThis file is reviewed by a human before the mappings are applied. The reviewer checks:\nAre the corporate action mappings correct? (Is the old ISIN really the same company as the new one?)\nAre the CD/CP aggregations grouping the right instruments? (Is \"AXIS BANK CD 08JAN26\" really from Axis Bank?)\nAre the NO_ACTION items correctly identified as different companies?\n\nThe Mapping File (consumed by the converter)\n\nA compact file with only the actionable mappings — one line per ISIN that needs to be remapped. This is the file that the converter reads on subsequent runs to apply the mappings.\n\nCurrent mapping statistics (as of Feb 2026):\n9 corporate action mappings (face value splits, bonus issues)\n114 CD aggregations (across ~15 banks)\n9 CP aggregations (across ~5 companies)\n61 T-Bill aggregations (various auction dates)\n4 G-Sec aggregations (various coupon rates and maturities)\n62 NO_ACTION records (different companies with similar names — verified and left alone)\n\nStep 9: Applying Mappings — The Re-Conversion\n\nHere's an important detail: ISIN mapping is not incremental. Every time I add a new month's data, I regenerate mappings from the entire database and then re-convert every single month.\n\nWhy? Because a new month can introduce a corporate action that affects older data. For example:\n\nIn December 2025, Fund A holds BEML with ISIN INE258A01016. In January 2026, BEML does a stock split, and Fund B (which bought after the split) reports ISIN INE258A01024. If I only mapped January's data, December's BEML would still be on the old ISIN, and the dashboard would show them as two different companies.\n\nBy re-mapping the full database, December's BEML gets correctly mapped to the new ISIN, and both months show a single BEML entry.\n\nThe re-conversion process:\nGenerate fresh mappings from the full database\nRe-convert every month × every category (e.g., 4 months × 6 categories = 24 runs)\nEach run replaces the data for that fund-month combination in the database\n\nStep 10: Validation — Making Sure Nothing Is Lost\n\nAfter all conversions are complete, I run a validation check on every fund-month combination. For each one:\nRead the \"Grand Total\" market value from the original Excel file\nSum all market values for that fund-month in the database\nCompare the two numbers\n\nThey must match exactly (0.0000% difference). This confirms that no rows were accidentally skipped, no values were misread, and no data was corrupted during processing.\n\nCurrent results (96 fund-month combinations):\n93 exact matches (0.0000% difference)\n3 known expected differences:\nKotak Flexicap Sep 2025: Rs 10,414 difference (0.19%) — Kotak's Grand Total excludes futures mark-to-market adjustments, but I include the futures rows\nKotak Midcap Jan 2026: Rs 13,391 difference (0.23%) — Same Kotak futures issue\nPPFAS Flexicap Sep 2025: Rs 15.06 difference (0.0001%) — PPFAS hardcodes their Grand Total cell instead of using a SUM formula; my sum matches their section sub-totals exactly\n\nThese three exceptions are documented and verified. Every other fund-month is an exact match down to the paisa.\n\nStep 11: Frontend Display — Virtual Grouping in the Browser\n\nThe processed database is served to your browser as a compact SQLite file (~2.5 MB). All the ISIN mapping, name resolution, and aggregation described above has already been applied in the data.\n\nIn the browser, one additional grouping step happens: non-equity items are organized into four category rows at the bottom of the comparison table:\n\n| Category | What's In It | Why Separate |\n|---|---|---|\n| Liquidity (TREPS, Repos) | TREPS, reverse repos, treasury bills | Near-cash instruments for daily cash management |\n| Debt (CD, CP, GSecs) | Certificates of deposit, commercial papers, government securities | Interest-bearing debt instruments |\n| Foreign Stocks (Alphabet, Meta, Amazon etc.) | Overseas equity holdings | Non-Indian equities identified by their US/GB/KY ISIN prefix |\n| Others | Cash balances, net receivables, margins, derivatives | Everything else |\n\nEach row shows the combined value across all selected funds. Clicking any row opens a drill-down modal where you can see:\nBy Fund — How much each fund holds in this category\nBy Instrument Group — Breakdown by type (e.g., CDs vs CPs within Debt)\nBy Individual Instrument — Every single CD issuance, every T-Bill, every cash item\n\nThis means the aggregation keeps the main table clean while preserving full granularity in the drill-down.\n\nStep 12: ISIN Mapping Disclosure in the Frontend\n\nTransparency is a core principle. Every holding where an ISIN mapping has been applied is tagged with an ISN badge in the comparison table. This small grey label appears next to the company name.\n\nClicking the ISN badge (or the company row) opens a modal showing:\nThe original ISIN(s) from each fund's Excel file\nThe mapped ISIN after processing\nThe mapping category (Corporate Action, CD Aggregate, etc.)\nThe specific reason for the mapping\nWhich funds held which ISINs\nThe market values involved\n\nFor example, clicking the ISN badge on \"BEML LIMITED\" would show:\nFund A reported ISIN INE258A01016 (old, pre-split)\nFund B reported ISIN INE258A01024` (new, post-split)\nMapping: Corporate Action — serial 01 → 02, face value split from Rs 10 to Rs 5\nBoth ISINs now grouped under the newer ISIN\n\nSimilarly, clicking \"AXIS BANK CD\" would show all 8 individual CD issuances with their original ISINs, maturity dates, and market values — aggregated into a single row for clarity but fully transparent.\n\nThe complete list of all ISIN mappings is also available in the ISIN Remaps tab in the Methodology section, where every single mapping record can be reviewed.\n\nSummary: The Full Pipeline\n\nHere's the complete flow from raw Excel to dashboard display:\nDownload — Collect monthly portfolio Excel files from 15 AMC websites\nRead — Auto-detect format and schema, extract holdings data\nIdentify — Look up each ISIN in the 361,000-record NSDL master\nClassify cash — Sort non-ISIN items into foreign stocks, TREPS, or other\nName-cut grouping — Group ISINs by 7-character name prefix to find potential duplicates\nDetect corporate actions — Same issuer + same equity type + different serial = stock split or bonus\nAggregate debt instruments — Group CDs, CPs, T-Bills, G-Secs by issuer with synthetic ISINs\nHuman review — Review the validation file, verify mappings are correct\nApply mappings — Re-convert all months with fresh mappings\nValidate — Confirm Excel totals match database totals exactly\nDeploy — Push the ~2.5 MB frontend database to production\nDisplay — Browser groups non-equity items into 4 categories with full drill-down"
},
{
"slug": "mfpro-india-mutual-fund-nav-data-api",
"title": "India Mutual Fund NAV Data Since 2013 - 19.8M+ Records. Download or Query via API. Free. No Auth.",
"tags": [
"mutual-funds"
],
"content": "India Mutual Fund NAV Data Since 2013 - 19.8M+ Records. Download or Query via API. Free. No Auth.\n\nIndia Mutual Fund NAV data since 2013 - 19.8M+ daily records across 17,866 schemes. Updated daily. Download or query via API. Free. No login. No auth.\n\nFull database available as CSV, TSV, Parquet and SQLite. Updated daily from AMFI India.\n\nPick what works for you:\nDownload by fund - search any scheme, pick a period, download CSV\nDownload full database - all 17,866 schemes in one file, four formats\nREST API - query by scheme code, filter by date, get JSON back. Also supports full database download\n104 tracked funds include pre-computed rolling returns (3M to 5Y)\n\nSync runs 3 times a day. CI/CD pipeline updates API server from production server, with download files regenerated after each sync and validations at each step. Latest status visible on the app.\n\nAPI docs with curl, Python and JavaScript examples on site. Swagger available. Rate limit: 100 req/min per IP.\n\nBuilt as part of MFPRO - Mutual Fund Analytics.\n\nmfpro.tigzig.com/data-api\n\nMFPRO Data Download and API Access"
},
{
"slug": "mfpro-v2-mutual-fund-analytics-rolling-returns",
"title": "Now live - MFPRO v2 - Mutual Fund Analytics (India) - now with rolling returns, custom eval periods, multi-period & multi-instrument comparisons",
"tags": [
"mutual-funds",
"portfolio-analytics"
],
"content": "Now live - MFPRO v2 - Mutual Fund Analytics (India) - now with rolling returns, custom eval periods, multi-period & multi-instrument comparisons\r\n\r\nMFPRO Rolling Returns\r\n\r\nFree to use. No login. No ads. Data Downloads. By Analyst. For Analysts.\r\n\r\nv1 just had portfolio comparison of one or more funds. New functionalities now added:\r\n\r\nReturns\rPoint-to-point returns - custom periods\rRolling return statistics: Average, Median, Min, Max across 1Y, 3Y, 5Y windows\rCustom evaluation periods for rolling returns\rCompare returns of multiple funds and indices on a single screen\rCoverage: 95 funds & 9 Indices\r\n\r\nDownloads & Validations\rRaw NAV data & Index data as CSV for independent offline verification\rPre-built Excel templates with CAGR & Rolling return formulas for cross-checking\rRolling return validation: e.g for BSE 100 - 7,485 values verified within 0.50 bps of manual calculation\rStep-by-step guide on how to replicate every number independently\r\n\r\nPortfolio Holdings\rCompare holdings of one or more funds side by side across two time periods (existing)\rCompare fund holdings against index benchmarks (Nifty 50, Nifty 100, Nifty 500, Nifty Midcap)\rMulti-period comparison - track holdings changes across up to 5 time periods\rCoverage: 24 funds across 6 categories: Large Cap, Mid Cap, Small Cap, Flexi Cap, Focused, Value\r\n\r\nIndices\rView constituents and weights of Nifty indices (Nifty 50, 100, 500, Midcap, Midcap 50)\rCompare two indices side by side - see overlapping and unique stocks\r\n\r\nData\rPortfolio composition from MF Monthly Disclosure files\rNAV & Indices from Jan 2013 & sourced from mfapi.in\rAutomated daily sync - data updated twice daily\rView daily update status on app\r\n\r\nDocs\rDocs with detailed explanation of methodology with examples.\r\n\r\nLive now at: mfpro.tigzig.com"
},
{
"slug": "mistakes-i-made-building-text-to-sql-agents-live-projects-2025-learnings",
"title": "Mistakes I Made Building Text-to-SQL Agents in Live Projects. My 2025 Learnings",
"tags": [
"text-to-sql",
"database-ai"
],
"content": "Mistakes I Made Building Text-to-SQL Agents in Live Projects. My 2025 Learnings\n\nI've been setting up Text-to-SQL apps and agents for over a year now. Multi-user systems. Live databases. Made many mistakes. Here's what broke, what it cost, and how I fixed it.\n\nSTRATEGIC & ARCHITECTURE\n\n▸ Built for analytics when main use was operations\n\nDesigned for analysts running complex queries. Actual users: customer service, finance & operations teams pulling raw data for reconciliations and responses.\n\nMost queries are straightforward - ID lookups, transaction history, ad-hoc filters. Don't over-engineer for complexity that doesn't exist.\n\n▸ Using multi-step agents where simple one would do\n\nBuilt a reasoning-heavy agent: plan → execute → validate. Cost went from <$1 per 100 queries to $20+. Queries took 1-3 minutes. Single-step agents work for 80% of SQL tasks. Use multi-step only when genuinely required.\n\nDATA & CONTEXT SHARING\n\n▸ Didn't share schema with the agent\n\nUser asks query. Agent runs blind - wrong joins, hallucinated columns, missed filters.\n\nAlways include: table names, columns, data types, primary/foreign keys, relationships.\n\n▸ Did not provide business context\n\nAgent doesn't know what fields mean, how they're computed, what edge cases exist.\n\nShare: variable definitions, computation logic, common query patterns, odd values to expect, business rules. Schema alone isn't enough.\n\n▸ Used plain text dumps for schema\n\nDumped schema as big text blocks adding things randomly. Agent got confused.\n\nUse YAML - is human-readable, cleaner hierarchy.\n\n▸ Didn't share categorical variable distributions\n\nAgent didn't know valid values for status fields, product categories, regions. Made up values. Spelling inconsistencies broke queries.\n\nShare distributions for cardinal variables upfront. Agent needs to know: what values exist, common misspellings, edge cases. Gives it X-ray vision into the data. Basically share a condensed EDA output with the agent.\n\nAGENT INSTRUCTIONS & CONSTRAINTS\n\n▸ No LIMIT clause rules\n\nAgent would download full datasets for 'exploration'. 1M rows pulled for simple queries.\n\nRule I added: Always LIMIT 100 on raw SELECT queries. No LIMIT on aggregations (COUNT, SUM, GROUP BY). Explicitly tell the agent when and when not to use LIMIT. For stricter control, apply at DB level (if supported) or API layer. This is a tricky one though.\n\n▸ Unlimited query execution\n\nAgent would run say 10 SQL queries for a simple question. Cost and latency go up.\n\nCap it. Simple agent: 2-3 queries max. Complex reasoning agent: 7-10. Specify execution limit upfront based on use case.\n\n▸ No debug protocol\n\nQuery failed. Agent didn't self-diagnose.\n\nGive the agent a debug protocol: That it should try and debug iteratively in case of errors, say up to additional 5 queries.\n\n▸ Assumed agents write perfect SQL\n\nThey don't. Division by zero errors is very common.\n\nTell the agent: wrap divisors with NULLIF(x, 0). Use COALESCE() for fallback values. Identify common error patterns and suggest fixes upfront.\n\nAGENT BEHAVIOR REALITY\n\n▸ Assumed agents follow rules 100%\n\nSet LIMIT rules, query caps, table restrictions. Agents still break them occasionally. Non-deterministic behavior.\n\nYou will never get 100% compliance. You can add additional checks at DB level or API level. But that's the nature of the beast. See what you can live with.\n\nMODEL SELECTION & COST OPTIMIZATION\n\n▸ Used expensive models everywhere\n\nUsed to use GPT-4.1 for everything. 5-10x cost vs cheaper models.\n\nGPT-4.1-mini and even GPT-4o-mini great for routine tasks. Reserve GPT-4.1 / 5.1 / 5.2 for harder tasks.\n\n▸ Upgraded to latest model immediately\n\nSwitched from GPT-4.1 to GPT-5 the day it launched. Cost blew up - reasoning tokens added massive overhead.\n\nLet models stabilize. Test cost and performance before upgrading. Don't chase 'latest' by default.\n\n▸ Didn't specify reasoning parameters\n\nMany models have reasoning controls. Didn't use them. Agent applied full reasoning to simple queries. For simple tasks, set reasoning to zero or minimal. Saves tokens and latency. Only enable reasoning when genuinely needed.\n\n▸ Didn't monitor cache hits\n\nSystem instructions and schemas ran 20K+ tokens. No cache hits = repeated full charge.\n\nMonitor cache hit rates in logs. Key rules: exact prompt match (byte-for-byte), no dynamic content at start, proper cache configuration. This is a complex area with major cost impact - detailed cache optimization guide coming soon.\n\nSECURITY & ACCESS CONTROL\n\n▸ Provided write access inadvertently\n\nUser truncated live tables. My fault - the test ID write permissions went to production.\n\nFix: separate DB users with fine-grained permissions.\n\n▸ No CREATE table functionality\n\nUser pulls 5,000 records. Can't print it. Can't export it.\n\nAdded: write access to 'workdir' schema. Added export functionality. Gave users control over larger result sets.\n\nPERFORMANCE & LIMITS\n\n▸ No rate limits\n\nOne user could trigger 100+ queries in a minute.\n\nRate limits at API level (SlowAPI for FastAPI). Rate limits at agent level.\n\n▸ Async in name only\n\nWrote async def everywhere. One blocking file read killed concurrency for all users.\n\nAudit for hidden blocking calls. Wrap blocking DB operations in thread pools. Replace blocking file reads. Async must be end-to-end or it doesn't work.\n\n▸ Didn't test edge cases\n\nWorked fine with clean data. Broke with null values, special characters in column names, mixed delimiters, inconsistent date formats. Push it to limits. Test with messy real-world data. Observe failures. Adjust instructions.\n\n▸ Temporary tables without cleanup\n\nAllowed agent to create intermediate tables. Never dropped them. Database bloated over time.\n\nHave agent use clear naming (e.g. tempagentxxxx) and purge regularly.\n\nResult\n\nCost per 100 queries: $20+ → under $2 for most tasks\nLatency: 2+ minutes → 10-30 seconds\nError rate: ~30% → under 5%\nUser satisfaction: measurable improvement\n\nTest it\ntigzig.com/analyzer - live Text-to-SQL app with sample datasets up to 1.6GB. No login required. Open Source\ntigzig.com/database-landing - detailed field guides & costing for deploying AI on databases\ntigzig.com/blog - posts and guides on AI for Analytics"
},
{
"slug": "monthly-mf-portfolio-files-hours-wasted-re-formatting-here-s-a-tool-that-fixes-it",
"title": "Monthly MF portfolio files = hours wasted re-formatting. Here’s a tool that fixes it",
"tags": [
"mutual-funds",
"converters-tools"
],
"content": "Monthly MF portfolio files = hours wasted re-formatting. Here's a tool that fixes it\n\nWhy This Exists\n\nIf you track Indian MF portfolios, you know the drill. Monthly disclosure Excel files = same info, different layouts. Spending time re-structuring is dead time.\n\nWhat You Get\nA converter utility built for analysts.\nDrop in raw files.\nGet back clean CSVs with ISIN mapping + standardized names.\nPlug straight into your analysis workflow\n\nHow It Works\nAI-powered schema detection: Automatically figures out data layout\nCross-validation: Runs multiple models to check totals.\nOverride: When in doubt, enter the schema manually - tool re-runs instantly.\n\nExtra Utilities\nAppend multiple funds → one file.\nTranspose holdings → side-by-side view.\nLogs + validation table → quick audit trail.\n\nWhat Next - Analysis Options\nDo it your way → pivots, Jupyter, Colab.\nUse the AI Analyst SQL Agent at app.tigzig.com/analyzer → Choose File Load to a temporary DB → analyze with Claude Sonnet 4 / DeepSeek / Qwen3 Max.\nUpload to ChatGPT (Go/Plus). Have it load into python/ pandas and analyze.\n\nBottom line: A single, focused utility. Converts messy disclosures → clean, structured data. From there, analysis is up to you. Usage guide + build guide included → hit 'Docs' on top.\n\nLive at\n\napp.tigzig.com/mf-files-ai\n\nResources for Analysts\nRBICC Tool: If you also work with RBI Monthly ATM/ POS/ Cards, this Excel to CSV converter might be useful - app.tigzig.com/rbi-cards\nFull Suite: All 25+ open-source analytics apps are here → app.tigzig.com"
},
{
"slug": "movie-similarity-engine-sql-jaccard-duckdb",
"title": "How I Built a Sub-Second Movie Similarity Engine With a 10-Line SQL Query",
"tags": [
"duckdb",
"database-ai"
],
"content": "How I Built a Sub-Second Movie Similarity Engine With a 10-Line SQL Query\r\n\r\nOn my IMDb dashboard (tigzig.com/movie-explorer), you can click a lightning icon next to any movie and get a list of similar movies. Match percentages, shared factors, the whole thing. It runs in under a second.\r\n\r\nNo ML - just SQL\r\n\r\nThe Problem\r\n\r\nIMDb has 12 million titles and 97 million person-to-title credit records. I wanted a 'movies like this' feature that could compare any movie against all others and return results fast enough to feel instant. The constraint was that I only had public metadata - no user ratings or watch history. Just titles, genres, cast, crew, year, and runtime.\r\n\r\nBrainstormed with Claude Code on approaches - from ML-based recommender systems to simpler methods - looking at trade-offs, deployment constraints etc. Here's what I went with.\r\n\r\nThe Approach: Jaccard Similarity With Weighted Tokens\r\n\r\nJaccard similarity is simple. Take two sets. Count what they share. Divide by the total unique items across both sets. Two movies that share 5 out of 20 total unique factors get a 25% match score.\r\n\r\nThe trick is what you put in those sets. I capture each movie's attributes as a list of tokens - text labels that describe its properties. A movie might have tokens like genre:Drama, director:nm0000229, star:nm0000151, decade:1990s, runtime:standard, rating_band:excellent.\r\n\r\nThe problem is that standard Jaccard treats every token equally. Sharing a lead actor should matter more than sharing a runtime category. But Jaccard has no built-in weighting mechanism.\r\n\r\nThe workaround is token duplication. I repeat important tokens multiple times in the list. A top-billed star gets about 10 tokens - star:, lead:, and actor: variants of their ID, repeated. A director gets about 5 tokens. A writer gets 1. Runtime gets 1. Genre gets 1. When two movies share a star actor, those ~10 duplicate tokens all appear in the intersection, pushing the Jaccard score up far more than sharing a single runtime:medium token would. The weighting is baked into the data at table creation time - the similarity query itself is completely unweighted. It just counts matching tokens versus total tokens. The weighting already happened.\r\n\r\nThe factors I settled on: genre, directors, actors (with star, lead, and supporting distinctions), writers, producers, decade, runtime category, and rating band.\r\n\r\nThe Implementation\r\n\r\nI cannot run this comparison on raw data. The cast and crew information lives in titleprincipals - 97 million rows. Genres and runtime are in titlebasics - 12 million rows. Ratings are in title_ratings - 1.6 million rows. To build a token set for one movie, you need to join across all three tables. To compare one movie against all others on the fly, you'd be hitting that 97 million row table repeatedly.\r\n\r\nWhat I did was build a pre-computed table called movie_tokens. One row per movie, with all similarity factors already assembled from those three source tables and encoded as a DuckDB list (array) column.\r\n\r\nThe key design choice: I don't tokenize all 12 million titles\r\n\r\nI filter to titleType = 'movie' (drops TV shows, shorts, episodes, etc.) and then to movies with 10,000+ votes. That takes 12 million titles down to about 12,000 movies. These are movies that enough people have actually watched and rated. Nobody needs a similarity match for a zero-vote short film from 1927. This filter is what makes the whole thing fast - the similarity query compares against 12,000 rows, not 12 million.\r\n\r\nSo instead of joining across 97 million + 12 million + 1.6 million rows at query time, I join them once during the data pipeline and store the result. The movie_tokens table is about 10MB. Each row has a tconst, title, year, rating, votes, genres, and a tokens array.\r\n\r\nThe query that powers the whole feature\r\n\r\n``sql\r\nWITH source AS (\r\n SELECT tokens FROM movie_tokens WHERE tconst = 'tt0111161'\r\n)\r\nSELECT\r\n m.tconst, m.title, m.year, m.rating, m.votes, m.genres,\r\n LEN(list_intersect(m.tokens, s.tokens)) * 1.0 /\r\n LEN(listdistinct(listconcat(m.tokens, s.tokens))) as jaccard_score,\r\n listintersect(m.tokens, s.tokens) as matchingtokens\r\nFROM movie_tokens m, source s\r\nWHERE m.tconst <> 'tt0111161'\r\n AND LEN(list_intersect(m.tokens, s.tokens)) > 0\r\nORDER BY jaccard_score DESC, m.votes DESC\r\nLIMIT 30\r\n``\r\n\r\nBut building the pre-computed table is only half the problem\r\n\r\nWhen a user clicks that lightning icon, the query still has to take one movie's token list and compare it against all 12,000 other movies in real time. That's 12,000 set comparisons - find the intersection, find the union, compute the ratio - all in one query.\r\n\r\nThis is where DuckDB earns its keep. The CTE (WITH clause) grabs the source movie's tokens. Then the main query cross-joins that against every other row in movietokens. For each pair, listintersect finds the shared tokens, listconcat + listdistinct gives the union, and the division produces the Jaccard score. The matching_tokens column returns exactly which tokens matched - those are the tags you see in the \"Why Similar\" column on the UI.\r\n\r\nDuckDB is built for this kind of columnar, analytical workload. It processes all 12,000 comparisons - each involving array intersection and union operations - in under a second. It's not doing 12,000 separate queries. It's one vectorized scan across the entire table. That's the difference between an analytical database like DuckDB and a row-oriented database.\r\n\r\nThe backend cache means repeated lookups for the same movie are instant after the first hit.\r\n\r\nJaccard on token lists is not the most sophisticated similarity algorithm\r\n\r\nBut it has properties I care about: it's explainable (you can show exactly which factors matched), it runs in pure SQL (no external libraries or services), and it's fast enough for a real-time feature.\r\n\r\nThe matching_tokens field is what makes this useful. When a user sees '24% match' next to a movie, they can also see Actor, 1990s, Drama, Lead, Star - the actual reasons. That transparency matters more than a slightly better algorithm that feels like a black box.\r\n\r\nThe full implementation is open source. Frontend code, backend code, the token generation query, the similarity SQL - all in the repos linked on the dashboard. The data pipeline docs walk through exactly how the movie_tokens table is built.\rLive app: tigzig.com/movie-explorer (go to Explore tab, click any lightning icon)\rFrontend: github.com/amararun/shared-imdb-dashboards\rBackend: github.com/amararun/shared-duckdb-dashboards-backend"
},
{
"slug": "multi-agents-sequential-reasoning-connect-database-o3-mini-deepseek-r1-flash-2-0-flowise",
"title": "Multi-Agents (Sequential) with Reasoning – Connect to any database - o3-mini / Deepseek-R1 / Flash-2.0. Built with Flowise.",
"tags": [
"database-ai"
],
"content": "Multi-Agents (Sequential) with Reasoning – Connect to any database - o3-mini / Deepseek-R1 / Flash-2.0. Built with Flowise.\n\nBuilt with Flowise. Leverage as a reusable component\n\nWith agent templates, tool schemas and source codes.\n\n6 database agent options\n\nThe Analyzer micro-app on REX-3 (tigzig.com) offers 6 agent options, all capable of:\nconnecting to any database (with credentials)\nnatural-language-to-SQL\nnatural-language to python (for charts & stats)\n\nApp also allows on-the-fly Postgres database for file uploads and temp use\n\nAI backend is on Flowise AI. Leverage as a reusable component:\ndirectly via Flowise's native UI (Click → Share Chatbot → Live)\nplugged into any UI via API (as in REX-3)\nintegrated into automated workflows via API\n\nEasy to setup, modify and maintain.\n\nStandalone General Analyst\n\nSingle-agent setup for regular queries & simple analysis. Uses o3-mini; for complex cases, I use gpt-4o\n\nMulti-Agents (technically sequential agents in Flowise, Backend: LangGraph)\n\nThree-agent flow: Gemini-Flash-2 / o3-mini / Deepseek-R1. Agent flow:\nRouter Agent → Determines if a query goes to General or Advanced Analyst\nGeneral Analyst → Same as standalone version above\nAdvanced Analyst →\nStep 1: Reasoning LLM (o3-mini / Flash-2 / Deepseek-R1) generates analysis plan & SQL\nStep 2: gpt-4o agent verifies query accuracy, executes, and returns results\n\nMulti (Sequential) Agents with Additional Debugging\n\nStill in testing. Similar to sequential agents but with:\none has and additional error-checking step\nother is a debugger agent for routing based on user request\n\nTake them for a spin\n\nFeel free to test out with different data/query complexity/agent choices-no API key needed. Connect to your database or use the temporary Postgres DB (created on-the-fly for file uploads). Upload your own files or just hit the 'Sample' button for sample file upload.\n\nSource Codes/ Resources\nFlowise agent templates & tool schemas\n\nAll schemas + source codes → app.tigzig.com (Help & Build section)\nDetailed Video Guide – Demo & Build Walkthrough\n\nCovers: Agent architecture, SQL errors, API failures, error handling, cost, performance, quality, latency\nTop Flowise Resource\n\nLeon van Zyl's YouTube → @leonvanzy (best guide out there)\n\nExplore 15+ open-source AI analytics apps at tigzig.com with natural language to SQL, python charts, stats, database connectivity and more. Free access, source codes and build guides\nAI Co-Analyst Advanced Analytics App: https://lnkd.in/g8xPQvb8\nAI Automation Micro App: https://lnkd.in/gxVbCdpv\nRealtime SQL Voice AI: https://lnkd.in/g6cT8ZSM https://lnkd.in/gCT9J_K4\nConnect ChatGPT to Supabase in 10 minutes: https://lnkd.in/g2A9h8f2"
},
{
"slug": "mutual-fund-analysis-custom-gpt-python-multiple-excel",
"title": "Mutual Fund Allocation Analysis with GPT Power Tools. Custom GPT. Custom Python Code. Multiple Excels.",
"tags": [
"mutual-funds",
"custom-gpt"
],
"content": "Mutual Fund Allocation Analysis with GPT Power Tools. Custom GPT. Custom Python Code. Multiple Excels.\n\nLive use case of having the GPT use a custom code that ensures consistent performance, rather than relying on it's adhoc different approach each time. A GPT Power Tool.\n\nIn my previous videos and posts, I have shown how to use GPT Plus to run machine learning (https://rb.gy/o39bvm), LlamaParse complex PDFs to analyze 10K / Annual Reports(deployed at https://www.tigzig.com/). As well as a LLM App,YFIN Bot that will pull multiple-year financials from Yahoo Finance, now live on public internet (https://www.tigzig.com/yfin) as well as a custom GPT (https://rb.gy/fy8lkk)\n\nAll apps, including this one, fully coded by ChatGPT / Gemini (wherever coding is required)\n\nCurrent use case: (a) track changes in equity holdings of a mutual fund over two or more time periods. (b) compare portfolio holdings of two or more different mutual funds. To be done repetitively. Can handle upto 10 excel files, each in different format (GPT Plus limit)\n\nSolution: Custom GPT 'Mutual Funds Portfolio Analyzer' (https://rb.gy/nj51lf). Live now. Feel free to use. It has regular instructions along with a custom Python code generated by GPT itself, to serve as a guideline. I was getting inconsistent results with just prompts and instructions. So now I am having the GPT use a coding approach that ensures consistent performance, rather than relying on it's adhoc different approach each time. A GPT Power Tool.\n\nHow to Use: Upload the excels, share a few details and that's it. Have GPT analyze it or analyze offline, pushing the appended data into a Pivot Table. [See document for details]\n\nWhere do I get the portfolio allocation files ? Available on MF websites. Have uploaded some at https://tigzig.com/files\n\nCan I replicate this for my excel processing use case, nothing to do with equities? Yes. Just replicate the GPT and customize your instructions. Easy to do. No coding required. Only one piece of GPT auto generated code to be copy pasted during setup. (see document)\n\nCan I deploy this outside of GPT store, on public internet or my corporate intranet Yes, you can. See my YFIN bot for pulling multi-year financial statements from Yahoo Finance (https://rb.gy/ca2k5p). It's deployed inside GPT store as well as on the public internet and uses GPT 3.5. However for real-world use, anything requiring GPT4/Code Interpreter has significant cost considerations. Currently exploring lower cost custom build option. Watch this space for more."
},
{
"slug": "new-open-source-tool-mutual-funds-holdings-analyzer-python-in-excel-xlwings-lite-now-live",
"title": "New Open Source Tool. Mutual Funds Holdings Analyzer. Python in Excel (xlwings Lite). Now Live.",
"tags": [
"mutual-funds",
"xlwings-lite",
"python-in-excel"
],
"content": "New Open Source Tool. Mutual Funds Holdings Analyzer. Python in Excel (xlwings Lite). Now Live.\n\nIt's a specific solution to a recurring problem: standardizing, consolidating and comparing mutual fund portfolio data from the monthly statutory disclosures (India Market). The pipeline takes monthly portfolio disclosure files from mutual funds as inputs and outputs formatted summary with key indicators and charts.\n\nName standardization and ISIN merging\n\nThe workflow standardizes multiple security names for same ISIN and allows for merging security names for corporate action via a human-in-the-loop intervention step.\n\nAI-Powered Data Conversion\n\nRaw portfolio disclosure files (Excel) are fed into a separate converter tool (HTML-JS). It uses AI schema detection to identify structure, extract the data, and standardize it into a single text file. This is the input for the xlwings Lite App.\n\nxlwings Lite automation\n\nImport and parse data with a file picker (VBA), with automatic delimiter detection. Run Stage 1 to execute core Python script. It loads the data, auto detects the two reporting periods, and performs the initial analysis. Its primary job is to generate data quality reports that flag all naming and mapping inconsistencies. After a human review and modification of the ISIN_Mapping file, Run Stage 2 - the script re-runs the final aggregation using the cleaned mapping file, with final summary and charts.\n\nHuman-in-the-Loop\n\nThe process separates automated analysis from manual intervention. The Stage 1 reports are designed to expose data quality issues. Quality is enforced by the manual review before the final report is generated.\n\nAI Coder used for the app\n\nGemini CLI\n\nxlwings Lite MF Holdings Analyzer\n\nApp & Docs\n\nResources\n\nMutual Funds Holdings Analyzer"
},
{
"slug": "nifty50-30-day-forward-return-analysis-claude-in-excel",
"title": "NIFTY50 - 30 Day Forward Return Analysis Feb 2008 to 2026 - Claude in Excel with Python, Lambdas and Advanced Formulas",
"tags": [
"claude-in-excel",
"portfolio-analytics"
],
"content": "NIFTY50 - 30 Day Forward Return Analysis Feb 2008 to 2026 - Claude in Excel with Python, Lambdas and Advanced Formulas\r\n\r\nNIFTY50 Forward Returns Analysis\r\n\r\nFile shared + Power User Guide for Claude in Excel.\r\n\r\nAnalysis\r\n\r\nFor each Nifty50 trading day, compute 30 forward returns (Day 0 to Day 1, Day 0 to Day 2... Day 0 to Day 30). From those 30 returns - quintile cuts (P20, P40, P50, P60, P80) plus positive/negative day counts. And confidence intervals assuming normality for the time being. Then 30-day rolling diagnostics - max consecutive positive/negative streaks, max drawdown and runup %, days to max drawdown/runup, recovery days long and short.\r\n\r\nHow Claude handled it\r\n\r\nQuintile calculation in Python first, then validate with manual formula for a single day, then validate with LET + SEQUENCE, then setup a named LAMBDA function for reuse. Trade diagnostics using MAP, REDUCE and SCAN with LAMBDA - these walk through 30-day forward windows item by item. Had it setup documentation sheets ...plus a hyperlinked index sheet at start.\r\n\r\nWhole thing took just ~2.5 hours over two sessions. About an hour brainstorming and discussion, rest in processing, redoing, formatting. All done per instructions, but you need to be very clear ....and very granular sometimes in what you want it to do.\r\n\r\nStill WIP\r\n\r\n3 validations done for quintile cuts. Diagnostics need summarizing with distributions and confidence intervals, then full validation via Python recreation. After that - Gold, Oil & S&P 500 .. then add technicals ...then a regression...mostly quantile ...but yet to think through.\r\n\r\nFor traders\r\n\r\nThese are closing rates. Good for quick diagnostics only. Intraday prices needed for practical work. And for actual setups and backtesting, Claude Code / Scripts / Notebooks might be more efficient.\r\n\r\nBe aware:\rHeavy excel work is also token heavy\rPython code is reconstructed code, not the code snippets it actually ran.\r\n\r\nResources\rWorkbook: NIFTY50 Forward Return Analysis\rPower user guide - Claude in Excel: Power User Guide to Claude in Excel & PowerPoint - 26 Working Tips\rGet aligned price data: Portfolio Analysis Suite"
},
{
"slug": "open-so",
"title": "Live Portfolio Analytics - Powered by MCP Servers - Open Source",
"tags": [
"portfolio-analytics",
"mcp"
],
"content": "Live Portfolio Analytics - Powered by MCP Servers - Open Source\n\nAnalyze any stock, crypto, metal, or oil symbol vs benchmark using 70+ KPIs & 15+ Charts, AI-powered technicals, and clean PDF + web reports.\n\nLive across 6 interfaces: web apps, ChatGPT, chat agents, Excel (xlwings Lite), and forms.\n\nEach interface serves a different use case - from rapid-deploy to full-featured agents. Modular architecture makes it easy to plug components into any flow - agentic or not.\n\nBuilt the backend MCP servers, agent flows, and user interfaces as reusable modular components - easy to plug into different use cases and mix across stacks.\n\nPerformance stats powered by Python QuantStats package (by Ran Aroussi, creator of yfinance). Technical chart analysis with Python Finta and Gemini Vision. MCP servers built with Tadata FastAPI-MCP package, web apps with React and NextJS, and MCP enabled agents on n8n & Flowise.\n\nFully modular and live - clone it, remix it, set up your own stack\n\nMCP Servers are public:\nQuantStats MCP\nYFinance MCP\nTechnical Analysis MCP\n\nFull build breakdown below\nWHAT IT DOES - Live Portfolio Analytics Stack\n\nThis is a working analytics stack delivering live performance reports, AI-driven technicals, and financial data pulls - powered by MCP, FastAPI, and modular agents.\nQuantStats Performance vs. Benchmark - For any Yahoo Finance symbol - stocks, crypto, oil, metals. Over 70 metrics, 15+ charts: drawdowns, Sharpe/Sortino ratios, return distributions, correlations - delivered in clean HTML\nAI Technical Analysis - Technical analysis across two timeframes - Across daily and weekly timeframes. Chart analysis via Gemini Vision API, delivered as PDF and web reports with structured tables.\nFinance Data Pull: Extract prices, profiles, and full financials from Yahoo Finance: 150+ fields of profile info, P&L, balance sheet, cash flow. Excel integration via xlwings Lite\nHOW TO USE - 6 Live Interfaces\n\nThe system runs across 6 interfaces - each tailored for different use cases\nAgentic - Custom UI (NextJS + n8n)\nAgentic - ChatGPT (Custom GPT)\nAgentic - Advanced (React-based with full analysis tools)\nAgentic - Rapid Deploy (Flowise Native)\nExcel - xlwings Lite\nForm UI - HTML-JS-Jinja2\n\nAsk the agent for guidance or start with a prebuilt prompt.\n\nAll support the same analytics setup - just different frontends and feature layers. See next section on interface strategy.\nINTERFACE STRATEGY - Why 6, and Why Modular\n\nThe setup is designed around modular Gen AI-powered components - backend, agent layer, and UI - each one reusable and configurable depending on the use case. Once core processing is in place, it's easy to plug into different interfaces without rebuilding the logic.\n\nThe six interfaces aren't just demos - they show real deployment options. From lightweight forms to full-stack apps and AI agents. The options support a wide range of use cases - Section 6 on User Interfaces goes into details on when to use which and the trade-offs involved\n\nTo support this, I built three MCP-FastAPI servers and one standalone FastAPI server. These connect to agents running on n8n and Flowise, and frontends on React and Next.js. All components are connected via standard APIs, making them portable across tools - including third-party platforms.\n\nIn practice, modularity isn't always necessary. I sometimes deliver integrated solutions where the UI, logic, and agent live in the same build - faster for simpler use cases. But where reusability or scale is a factor, modular saves time, simplifies updates, and isolates risk.\n\nThis isn't about MCP or agents - it's about building practical, reusable analytics solutions that can plug into any interface or automation flow.\n\nAll live. Test it, clone it, build your own.\nARCHITECTURE - Modular, Component based, Reusable\n\nThe full stack is built around reusable components - each layer (frontend, agents, backend) is designed to plug into others with minimal setup. Here's how the architecture breaks down across interfaces and agents.\n\n4.1. Modular - Component based: Why and When?\n\nThere are cases where a non-modular setup makes more sense. For example, in one client project I built a lightweight HTML-JS tool for Excel file import and manipulation, bundled with a browser-based SQLite agent and a simple NL-to-SQL chat interface - all integrated in a single app. In that setup, modularity would've just added unnecessary complexity.\n\nBut when I see components - UI, backend logic, agents - that can be reused, I default to separating them out. In another case, I built a custom UI connected to a Flowise agent and backend SQL service. Later, the client needed a second agent setup pointing to a different database. All I had to do was update the agent config and env variable - no UI rebuild, no backend changes.\n\nModular setups also help with debugging, iteration, and access control. I can isolate issues, restrict backend exposure, and upgrade parts independently.\n\nI started building my technical analysis stack directly inside Excel with xlwings Lite. As it evolved, I split core processing into an MCP-FastAPI server - now the same logic runs across all UIs: web, forms, agents, GPT, Excel.\n\nNone of this is new - tech has done it for decades. Just sharing how modularity speeds up my own analytics builds, and when I choose to keep it simple.\n\n4.2. Frontend options\n\nCovered in detail in a separate section with notes on when to use what, and trade-offs based on my experience.\n\nUI options include: Next.js, React, ChatGPT, form-based UI, Flowise native UI, and Excel\n\nReact and NextJS UIs are set up as reusable modules - they can connect to agent flows on Flowise, n8n, or any API-accessible setup. Just update the API endpoint in the env variable, match input/output formats, and it's live.\n\nIn the current setup, the n8n agent connects to the NextJS UI, and the Flowise agent connects to the React app. Both agents are MCP enabled and are in turn connected to MCP Servers.\n\n4.3. Agent setups\n\nThe agent layer acts as the glue between UIs and backend MCP servers - handling workflows, orchestration, and logic routing. Here's how the n8n and Flowise setups are configured.\n\nn8n and Flowise AI support webhooks and API endpoints - easy to plug into any interface and is great for keeping the Agent layer separate.\n\nBoth include MCP Clients with SSE support for remote MCP Servers. Just drop in the MCP Server URL and you're set. In the current setup, agents connect to multiple MCP Servers and database tools (Flowise).\n\nBoth are production-grade tools built for complex agentic and non-agentic workflows (n8n). Flowise supports Sequential Agents with LangGraph - enabling advanced orchestration with routing and step-wise execution. n8n is superb with its Agent node, wide platform integration, HTTP node for API calls, and a solid set of routing and processing nodes.\n\n4.4. Core engine\n\nThis is the processing brain of the system - everything from calculations to report formatting runs through this Python-based backend, wrapped in modular MCP-FastAPI services.\nQuantStats by Ran Aroussi (creator of yfinance)\nyfinance for market data\nFinta for technical indicators\nMatplotlib for charts\nGemini Vision API for visual chart analysis\nReportLab for PDF formatting\n\n4.5. Backend\n\nThree integrated MCP + FastAPI servers and one standalone FastAPI server (details in next section). All Python logic is wrapped in FastAPI and mounted on an MCP Server in a single deployment - which also serves the form UI.\n\nI keep all reusable logic on FastAPI - easy to automate and connect across UIs, from ChatGPT to Excel to any custom UI. Tadata's FastAPI-MCP package makes it simple to mount MCP on any existing FastAPI setup.\n\nConnections:\nn8n and Flowise agents connect to the MCP server via their native MCP Client nodes\nReact and NextJS UIs connect to agents via API\nChatGPT connected to FastAPI endpoints on the same integrated MCP-FastAPI server via Custom Actions OpenAPI schema\nForm UI connected to FastAPI endpoints on the same MCP-FastAPI server\nExcel connects to FastAPI endpoints through xlwings Lite\nMCP SERVERS\n\nThe backend is split into four focused processing services - each one handles a specific piece of the analytics workflow, from financial data to report generation. All are exposed via MCP or API, built for reuse and quick integration.\n\nThere are three custom MCP Servers and one standalone FastAPI Server. The MCP servers are public - just plug and play. Add the URL to any SSE-enabled MCP client. Both n8n and Flowise have native nodes for this.\n\nThe servers are integrated MCP-FastAPI servers. I used Tadata's FastAPI-MCP package to mount MCP on top of FastAPI - just a few lines of codes - brilliant package. A single deployment runs MCP + FastAPI + Form UI (HTML-JS-Jinja2). Works cleanly with both agentic and non-agentic setups.\n\nAll the user interfaces connect to these integrated MCP-FastAPI servers - explained in the User Interfaces section\n\n5.1. QuantStats MCP Server\n\nThis is an MCP-FastAPI wrapper over the QuantStats package - connects to any frontend via MCP or API. Takes two Yahoo Finance symbols (one for performance, one for benchmark) plus a time range. Returns a formatted HTML report using the QuantStats package. Tables and charts are auto-generated directly by the package.\n\nThe MCP enabled agents connect to MCP Server and rest of UIs to the FastAPI endpoints.\n\n5.2. Technical Analysis MCP Server\n\nThis is a processing server that runs a multi-step workflow to generate technical analysis reports in PDF and web format. Takes a Yahoo Finance symbol and time range, returns AI-generated reports.\n\nWorkflow steps:\nConnects to Yahoo Finance MCP-FastAPI server to pull price data\nConverts daily prices to weekly dataframe\nCalculates technical indicators using Finta\nConnects to Gemini Vision API to get chart and technical analysis\nConnects to ReportLab FastAPI server for generating the final PDF and Web reports.\n\nThe MCP enabled agents connect to MCP Server and rest of UIs to the FastAPI endpoints.\n\n5.3 Yahoo Finance MCP Server\n\nThis is an MCP-FastAPI wrapper over the yfinance package. Connects to any frontend via MCP/ API. Takes a Yahoo Finance symbol and returns:\nPrice data (JSON) - requires date range\nCompany profile with 150+ fields\nFull financials: P&L, balance sheet, cash flow, and quarterly breakdowns\n\nThe MCP enabled agents connect to MCP Server and rest of UIs to the FastAPI endpoints.\n\n5.4 ReportLab Markdown to PDF-HTML FastAPI Server\n\nThis is a FastAPI processing server for generating custom formatted reports. PDF and HTML outputs are customized for this use case but can be adapted for others. ReportLab offers deep customization for PDFs, easily replicated in web reports using standard HTML-JS.\n\nWorkflow:\nConvert Markdown to HTML (using markdown)\nParse HTML with BeautifulSoup for structure\nUse ReportLab to build styled PDF\nStyle HTML output to match PDF\nReference charts from static folder\nAuto-clean old files (older than 24 hrs) using FastAPI startup events + Starlette background tasks\n\nI've set up a separate endpoint for technical analysis, with custom formatting for both PDF and HTML outputs. The same FastAPI server also includes a generic endpoint that takes Markdown content and returns a PDF with simpler formatting. It's deployed as an HTML-JS-Tailwind form UI that calls FastAPI endpoints - all served from a unified FastAPI server using Jinja2 templates.\nUSER INTERFACES\n\nAll processing and agent logic connects to live, working interfaces. Each UI is connected to the same backend and agent layer - just optimized for different workflows, tools, or user preferences. All of these are live, working apps - each built on top of the same backend stack. The UI is just the entry point. Some are lightweight and fast to deploy, others offer more control or customization.\n\n6.1. Custom GPT\n\nCustom GPT is usually my first choice when a UI is needed. You get an out-of-the-box interface, embedded agent, and Python code execution. Just supply a JSON OpenAPI schema to connect to any backend. Faster and cleaner than building even basic HTML-JS forms.\n\nLimitations: no custom UI, and feature set is narrower than a React or NextJS build. You'll need a Plus account ($20/month) to create a Custom GPT, though free users can still access it with rate limits.\n\nLive setup connects to QuantStats and Technical Analysis MCP-FastAPI servers via FastAPI endpoints.\n\n6.2. Flowise native UI\n\nFlowise native UI is a solid option when GPT isn't feasible. No UI build required. You get a ready-to-use chat interface that supports complex agent flows using LangGraph Sequential Agents, custom tools, and APIs. n8n too provides a similar native UI.\n\n6.3. Excel integration - xlwings Lite\n\nxlwings Lite (by Felix Zumstein creator of xlwings) is a lightweight Excel add-in that runs full Python workflows inside Excel - no local Python install needed. It comes with a built-in editor, console, environment variables, and deep Excel integration.\n\nOne of the big benefits is that it allows easy connectivity to any CORS-enabled backend API service. This fits well with my setup - since I use FastAPI servers extensively - and also makes it easy to connect to LLM/AI endpoints and third-party server providers.\n\n6.4. Simple Forms\n\nIn many cases, a simple form works best. I typically use Jinja2 templates (HTML-JS-Tailwind) for tight FastAPI integration. Sometimes Flask. Works well for fairly complex UIs too, with full JavaScript access and server-side rendering - env vars and routes stay hidden.\n\n6.5. Going full stack? React / NextJS\n\nFor complex apps and a polished UI, I use React or NextJS. The REX-3 Co-Analyst app has the QuantStats Agent on a separate tab, connected to the same Flowise agent from earlier - just wrapped inside a React interface.\n\n6.6. NextJS\n\nThe biggest benefit of NextJS: env vars and API routes stay private, and it supports server-side rendering. That's a major security benefit. The NextJS portfolio analysis agent uses a leaner UI, but it's still set up in a modular way - can be connected to any API-enabled agent backend by just changing the environment variable.\n\nNote: Vercel serverless functions time out at 60 seconds (300s on pro plan), so longer API calls need workarounds.\nDEPLOYMENTS\n\nNextJS and React apps are deployed on Vercel. All MCP + FastAPI servers, along with n8n and Flowise, run on a Dockerized setup via Coolify, deployed on a Hetzner VPS.\nAI CODER: CURSOR\n\nI use Cursor as my AI coding assistant across all builds - including this one. Every part of this stack - from UI's to FastAPI servers - was written and iterated using Cursor.\n\nFine print: This is not investment research or financial advice. It's a live working example showing how to stitch together AI, analytics, and infra into real outputs. The logic and analysis structure is based on a general-use setup - fully modifiable to fit your own requirements. Source code, backend, and app stack are open and adaptable. AI and humans can both make mistakes - always validate results."
},
{
"slug": "open-source-asset-comparison-tool-compare-stocks-indices-crypto-commodities-in-one-dashboard",
"title": "Open Source Asset Comparison Tool: Compare Stocks, Indices, Crypto & Commodities in One Dashboard",
"tags": [
"portfolio-analytics"
],
"content": "Open Source Asset Comparison Tool: Compare Stocks, Indices, Crypto & Commodities in One Dashboard\n\nEver tried comparing Apple, TCS, Nifty, Gold & Oil in one report - without juggling multiple platforms or drowning in paid apps?\n\nI use Trading View regularly. Used Amibroker back in my full-time trading days. Koyfin, Portfolio Visualizer, Portfolio Metrics - solid tools.\n\nBut here's my problem: I don't want model portfolios.\n\nI want to compare across markets and assets classes - Nifty vs Gold vs Microsoft vs Brent. Fast.\n\nWhat I wanted\nDaily returns chart - all securities in one chart\nDrawdowns, CAGR, Sharpe, Sortino - single visual\nDate-aligned CSV downloads\nTechnical analysis report\nRun combos of assets and timeframes\nAll in 30 seconds, not hours. In same tab.\n\nTry doing that on free tools - you'll hit limits, create 'portfolios' for single stocks, run into missing securities or juggle multiple tabs.\n\nSo I built TIGZIG Quants\n\nMy workflow now:\nQuick comparison on TIGZIG Quants\nDeeper TA -> Trading View\nDeeper Research -> Platforms and offline work\n\nIs this the ultimate solution? Absolutely not!\n\nBut it solves my problem: fast, no-nonsense comparisons between unrelated assets.\n\nQuick disclosure: I'm a Chartered Accountant and data scientist, not a portfolio manager. I built this for my personal research, and while it's a powerful tool, it isn't investment advice. Do your own due diligence always\n\nThis is build-as-you-go. Right now, CAGR is point-to-point (not a fan - too sensitive to endpoints). Rolling metrics & ML forecasts coming soon.\n\nReady to try it?\nAgent - Smart assistant\nSuite - Faster, no-frills version (needs exact Yahoo symbols)\n\nWant to build your own version? Hit the Doc button in the app for source codes and build guides + methodology, validations and recon report. Built with MCP servers, n8n, and React."
},
{
"slug": "oracle-always-free-arm-vps-retry-script",
"title": "How to get Oracle's 24GB RAM server free - what I call the 'VPS Lottery'. Problem - hard to get. Solution - automated scripts and patience.",
"tags": [
"infrastructure"
],
"content": "How to get Oracle's 24GB RAM server free - what I call the 'VPS Lottery'. Problem - hard to get. Solution - automated scripts and patience.\n\nOracle ARM VPS Monitor\n\nOracle provides one VPS instance of 24GB RAM, 4 ARM CPUs, 200GB storage. Forever free. Like an EC2/Azure VM. Only cloud provider that gives this away always free.\n\nBanged my head past one year on the web console - capacity never available.\n\nFinally asked my AI Coder (Claude Code now) to research how people actually get these instances. Answer: automated retry scripts running 24/7, cycling through availability zones. Takes 1-3 months typically.\n\nHad it build a retry script with monitoring dashboard, deployed on my server. The script will keep retrying until a VM is created. Results are saved to json on server on success. Checks all 3 availability domains each round. Ran 30+ days, over 100K API calls. Finally got the instance. Felt like I won a lottery.\n\nOracle reclaims inactive free instances - so put it to work immediately. Had Claude Code secure the server, install Coolify, deploy my DuckDB dashboards as alternate backend (IMDb data, 230M rows, 16GB database, FastAPI layer) - same day.\n\nResources below - VPS creation tool, infra guides, live dashboard.\n\nBonus: Oracle also gives 2 AMD Micro VMs (E2.1.Micro) - 1 CPU, 1 GB RAM each, x86 (not ARM), always available. No lottery. Great for many tasks. Each uses ~50 GB of your 200 GB quota.\n\nResources\n\nOracle ARM VPS Creator: Real-time Web UI with Live Monitoring\nhttps://github.com/amararun/oci-arm-vm-monitor\nFastAPI + HTML interface with Basic Auth. All OCI credentials stay server-side.\n\n2026 Infra Guide for Tool Builders\nPart 1: AI Coders\nPart 2: Deployment & Hosting\nPart 3: Security\n\nLive Dashboard (DuckDB, IMDb data, 230M rows, 16GB, Open Source)\nhttps://www.tigzig.com/movie-explorer\nToggle between Hetzner/Oracle servers."
},
{
"slug": "perimeter-security-cloudflare-free-plan",
"title": "Tool Builders Infra Guide - Part 5: Set Up Perimeter Security (Edge Defense) for Your Apps on Cloudflare's Free Plan",
"tags": [
"cloudflare",
"security",
"infrastructure"
],
"content": "Infra Guide for AI Tool Builders - Part 5: Set Up Perimeter Security (Edge Defense) for Your Apps on Cloudflare's Free Plan\r\n\r\nEdge Security Cloudflare Free Plan\r\n\r\nPrevious posts covered frontend, backend, and database security. This one is about what happens before traffic even reaches your server. Edge defense. Perimeter security. The stuff that sits between the internet and your application.\r\n\r\nIf you come from an analyst or data science background like me — building tools and small apps for businesses — you may not know about Cloudflare capabilities. I didn't. Even on the free plan, the amount of security and non security functionalities it offers is huge. And it costs nothing.\r\n\r\nHere's what I'm running across 60+ subdomains and 40+ deployed apps. All on the free tier.\r\n\r\nJust note that, with Cloudflare tokens, your AI Coder (Claude Code, Cursor etc) would be able to do any / all of this for you.\r\n\r\nOrange Cloud Everything\r\n\r\nFirst thing. When you add a domain to Cloudflare, every DNS record gets a toggle — orange cloud (proxied) or grey cloud (DNS-only). Orange cloud means traffic flows through Cloudflare. Grey cloud means traffic goes directly to your server, bypassing everything. Every domain that can be orange-clouded should be. The exceptions are third-party CNAMEs — Auth0, Clerk, AWS RDS — those need grey cloud because the third-party service requires direct resolution.\r\n\r\nOne issue I faced here - It has a 100 seconds time out. For my longer running backends - they are temporarily on grey cloud as I migrate to a polling system .. i.e my backend will send a job ID and I'll poll the completion endpoint from my frontend with that job ID.\r\n\r\nImportant thing is - know exactly which domains are grey and why. I maintain an audit list. Everything else — orange.\r\n\r\nJS Challenge on Frontend Domains\r\n\r\nCloudflare gives you 5 free custom WAF (Web Application Firewall) rules. One of mine applies a JavaScript challenge to most of my frontend tools. What this does — when a request hits your site, Cloudflare injects a lightweight JS snippet that the browser solves transparently. Users barely even notice. But automated scanners, bots, and curl scripts can't solve it. They get blocked before touching your app.\r\n\r\nBut here's an issue I faced- tigzig.com is an AI Agent first site with the all domains assets available as text and markdown files and indexed via llms.txt. But with JS Challenge I suddenly saw a drop in AI Agent hits, even though the llms.txt and other assets were excluded from the challenge and were fully accessible. My hypothesis - AI Agents skip the site without bothering to check for llms.txt if they hit a JS challenge on main domain. And as soon as I removed the JS Challenge, they came happily back.\r\n\r\nBrowser Integrity and Security Level\r\n\r\nFree settings. Browser Integrity Check catches requests with fake or missing User-Agent headers. Security Level set to High makes Cloudflare challenge IPs with bad reputation scores. Two toggles. Takes 30 seconds.\r\n\r\nThe Rate Limit Problem\r\n\r\nCloudflare's free WAF gives you one rate limit rule. One threshold across your entire zone. But a frontend React app could send anywhere from 20-40 requests on page load — JS bundles, CSS, images, API calls. For backend API, it's much lesser as it receives only the calls from frontend. So you need different rate limits for frontend and backends.\r\n\r\nThis is where Cloudflare Workers come in. Free plan gives you 100,000 invocations per day. You upload a small script that runs at the edge — before traffic reaches your server. My Worker checks the incoming domain against a config map and applies different rate limits:\rFrontends: higher limit - handles page load plus a few refreshes\rBackends: lower limit - pure API, no page loads\rMCP servers: medium limit - as AI agents can shoot off queries fast\rCatch-all: medium limit for any that might remain unconfigured\r\n\r\nOne Worker. One deployment. Different limits per domain and even per path. When the limit is hit, Cloudflare returns a 429 at the edge. The request never reaches your server. Your AI coder can write and deploy this — it's a single JavaScript file with a configuration map.\r\n\r\nAnd you can monitor your 429 rejects on Cloudflare dashboard and calibrate limits after having your AI Coder investigate their payload. I monitor this using a custom dashboard.\r\n\r\nIs the 100K invocations enough?\r\n\r\nIt looks like a big number but actually not so. Try this - next time you deploy a new app - check the number of requests at Cloudflare. Within first day you'll see thousands of hits on Cloudflare even without a single page view on your web analytics dashboard. Why? All automated bot scanners doing vulnerability scans. For a large public facing app, that's a drop in the ocean. For internal tools for small businesses, that might work....it all depends on what all you got in that particular zone. So yes, monitoring your free quota usage is important.\r\n\r\nWhat if you run out of free quota? Then there is option of Workers paid plan of $5 p.m which covers 100 million requests/month + $0.30 per additional million.\r\n\r\nFree tier is also limited to 10ms CPU time (not wall clock time) per request. For rate limit checks - domain lookup, config match, counter comparison - that's more than enough.\r\n\r\nYou need to monitor usage. A single bot storm could eat through that quota fast.\r\n\r\nZone-Level IP Blocking\r\n\r\nWhen you spot a malicious IP in your logs — vulnerability scanners, credential stuffers, path traversal attempts — you can block the entire IP range at the Cloudflare zone level (either a single IP or an entire subnet say, 256 addresses owned by the same provider). One API call. Applies across all subdomains instantly. I have blocked a few ranges from a bulletproof hosting provider (hosting provider known for harboring malicious traffic) after they hit my MCP server with .env hunting and malicious file upload attempts. Every request from those IP ranges now gets dropped at the edge. Zero impact on my servers. This is only as an exception. You can't really keep blocking IPs as they get rotated. The scalable way is rate limits, JS challenges etc as well as hardened & non-leaky backends. But good to know in case you do need to do that.\r\n\r\n\r\n
Quick Info — IP Address Blocks (CIDR Notation)
\r\n
In IP addressing, ranges are expressed as \"blocks\" using CIDR notation:
\r\n
A /32 is a single IP (1 address)
\r\n
A /24 is a block of 256 addresses (last number varies: 185.177.72.0 to 185.177.72.255)
\r\n
A /16 is a block of 65,536 addresses (last two numbers vary)
\r\n
So in one of my cases I put up a /24 block (256 IPs). In this case the attacker owned two /24 blocks (512 IPs total), so blocking both /24s covered their entire operation. You're not blocking the whole internet — just a specific slice you've identified as malicious. But again, as mentioned earlier, this level of blocking is more of an exception than the norm.
\r\n