# DuckIt - CSV to DuckDB Converter

Browser-based tool for converting CSV/TSV files to DuckDB databases and Parquet files with shareable download links. Conversion happens in-browser using DuckDB-WASM. Integrates with xlwings Lite Data Importer for Excel-based SQL analytics.

## Links
- App: https://duckit.tigzig.com
- Docs: https://tigzig.com/app-documentation/duckit-xlwings.html
- GitHub (Frontend): https://github.com/amararun/DUCKIT_UPLOADER-final
- GitHub (Backend): https://github.com/amararun/FASTAPI_DUCKIT-final

## Tags
duckdb, duckdb-wasm, parquet, python-in-excel, react, fastapi

## Architecture

```
┌─────────────────────────────────────────────────┐
│                    Browser                       │
│  CSV Input → DuckDB-WASM → Parquet conversion   │
└──────────────────────┬──────────────────────────┘
                       │ Upload (ZIP)
                       ▼
┌─────────────────────────────────────────────────┐
│              FastAPI Backend                      │
│  Receive Parquet → Convert to DuckDB → Store     │
│  → Generate signed download URL                  │
└─────────────────────────────────────────────────┘
```

CSV → Parquet conversion: Browser (DuckDB-WASM, no server upload)
Parquet → DuckDB conversion: Backend (after upload)

### Frontend (DUCKIT_UPLOADER-final)
- React 19 + TypeScript + Vite + TailwindCSS 4
- DuckDB-WASM for in-browser conversion
- Neon Auth (Better Auth + Google OAuth)

### Backend (FASTAPI_DUCKIT-final)
- FastAPI with aiofiles, slowapi rate limiting
- Signed URLs via itsdangerous for secure downloads
- Three storage tiers: temp (24h), persistent (7 days), permanent (admin)
- API key authentication on all upload endpoints

### Backend Endpoints
| Endpoint | Method | Description |
|----------|--------|-------------|
| `/upload-token` | POST | Generate time-limited upload token (10 min) |
| `/upload-direct/{token}` | POST | Upload file with token |
| `/secure-download/{token}` | GET | Download via signed URL |
| `/status` | GET | Storage statistics |
| `/cleanup` | POST | Remove expired files |
| `/files` | GET | List files (admin) |

### Backend Environment Variables
```
UPLOAD_DIR=./uploads
MAX_FILE_SIZE_MB=150
TEMP_RETENTION_HOURS=24
PERSISTENT_RETENTION_DAYS=7
RATE_LIMIT=200/hour
DUCKIT_DATENUM=your-upload-api-key
DUCKIT_SIGNING_SECRET=your-signing-secret
SIGNED_URL_MAX_AGE_HOURS=48
```

### Frontend Environment Variables
```
VITE_NEON_AUTH_URL=https://your-project.neon.tech/auth
VITE_DUCKIT_SERVER_URL=https://your-backend-url.com
VITE_DATENUM=your-api-key
```

## Features
- Build Database: Drop CSV/TSV → Parquet in browser → upload → DuckDB → shareable link
- Quick Upload: Upload existing DuckDB/Parquet files → shareable link
- CSV → Parquet: Convert entirely in browser, optionally upload
- Data integrity: XOR checksums validated between source and target
- Integrates with xlwings Lite Data Importer via shareable URLs

## Setup

```bash
# Frontend
git clone https://github.com/amararun/DUCKIT_UPLOADER-final.git
npm install && cp .env.example .env && npm run dev

# Backend
git clone https://github.com/amararun/FASTAPI_DUCKIT-final.git
pip install -r requirements.txt && cp .env.example .env
uvicorn app:app --host 0.0.0.0 --port 8000
```