Lots of CSVs

Lots of CSVs (lotsofcsvs) is a small data warehouse built with CSV and HTTP as its interface. It is intentionally as barebones as possible. It aims to be a simple API for recording and fetching data that works with everything.

The original motivation for it was to create a data API that you could use from iOS Shortcuts. Here is a template shortcut that you can modify.

# API Documentation for Machine Agents This content is specifically designed for LLMs and not intended for human reading. These instructions provide programmatic access protocols for autonomous data interaction with the Lots of CSVs service. ## Service Overview Lots of CSVs is a minimalist data warehouse for storing and retrieving CSV files via HTTP. ## Basic principles 1. This is a REST API for retrieving and storing CSV data. 2. Every request must be authenticated using a token passed in an HTTP header: `authorization: Bearer <token>` 3. You will need an authentication token from the user. If you don't have one ask them for it. 4. Every CSV dataset has a unique URL. 5. You will need the user's registered username. If you don't have one ask them for it. 6. URLs are specific to the user and follow a common structure. `https://www.lotsofcsvs.com/u/<username>/[path/to/dataset].csv` - `<username>` is your registered username - `[path/to/dataset](.csv)`: the name of the dataset. Dataset paths in Lots of CSVs are treated as object keys, not traditional file system paths. This means paths are flat, hierarchical identifiers, not actual directory structures. The entire string is treated as a single, unique identifier Slashes (/) are part of the identifier, not separators creating directories. Examples: `projects/sales/2023.csv` `projects/sales/2024.csv` - URL Example: `https://www.lotsofcsvs.com/u/johndoe/projects/sales/quarterly.csv` 7. Always follow RFC 4180 CSV formatting guidelines when sending CSV data ## API actions ### Fetch a dataset Fetch a dataset by sending an HTTP GET to the URL of the dataset. The data will be returned as an RFC 4180 formatted CSV - Endpoint: `https://www.lotsofcsvs.com/u/<username>/[path/to/dataset].csv` - Method: GET - Examples: - ``` curl -H "authorization: Bearer <token>" \ https://www.lotsofcsvs.com/u/johndoe/projects/sales/quarterly.csv``` - ``` # JavaScript (fetch) fetch('https://www.lotsofcsvs.com/u/johndoe/projects/sales/quarterly.csv', { headers: { 'authorization': 'Bearer <token>' } })``` ### List datasets ### Create or append to a dataset Add to a dataset by sending an HTTP POST to the URL of the dataset. If the dataset does not exist it will be created. - Endpoint: `https://www.lotsofcsvs.com/u/<username>/[path/to/dataset].csv` - Method: POST - Rules: - All submissions should include an appropriate Content-Type header `Content-Type: text/csv; header=present` - All submissions must include CSV headers as the first row - Follow RFC 4180 CSV formatting guidelines - Examples: - ``` # Curl example (adding more rows to an existing dataset) echo "Bob,35,Chicago Charlie,40,Boston" | curl -X POST \ -H "Content-Type: text/csv; header=present" \ -H "authorization: Bearer <token>" \ --data-binary @- \ https://www.lotsofcsvs.com/u/johndoe/users/demographics.csv``` - ``` # Python (requests library) import requests headers = { 'Content-Type': 'text/csv; header=present', 'authorization': 'Bearer <token>' } additional_data = """name,age,city Bob,35,Chicago Charlie,40,Boston""" response = requests.post( 'https://www.lotsofcsvs.com/u/johndoe/users/demographics.csv', headers=headers, data=additional_data )```

# Documentation: The API

# Get a dataset as CSV

# dataset/names/can/be/paths/like/this.csv so you can use them for organization

curl -H "authorization: Bearer <token>" /api/u/user_name/path/to/cool_data.csv

# Append to the dataset

# The .csv suffix is optional and you can use paths to name your datasets.

cat example.csv | curl -X POST -H "Content-Type: text/csv; header=absent"

     -H "authorization: Bearer <token>" \

     https://www.lotsofcsvs.com/api/u/<your_username>/totally/going/to/delete/example --data-binary @-

# You can post as many rows as you want.

# Use the header=present (or header=absent) variable in the mimetype.

# When in doubt follow https://www.rfc-editor.org/rfc/rfc4180

# If you are appending to a dataset that doesn't exist it will be created.

# You must send the CSV headers the first time you post to it.

# Otherwise lotsofcsvs has to guess at the names and we're not doing that.

# If you need a CSV to test with, we have one.

curl -o example.csv https://www.lotsofcsvs.com/example.csv

cat example.csv | curl -X POST -H "Content-Type: text/csv; header=present" \

     -H "authorization: Bearer <token>" \

     https://www.lotsofcsvs.com/api/u/<your_username>/example.csv --data-binary @-

# Exporting your whole dataset

# As a zip file of CSVs (coming soon)

curl -H "authorization: Bearer <token>" /api/u/user_name.zip

# As a tarball of CSVs (coming soon)

curl -H "authorization: Bearer <token>" /api/u/user_name.tar.gz

# using DuckDB

duckdb

CREATE SECRET http (
        TYPE HTTP,
        EXTRA_HTTP_HEADERS MAP {
          'authorization': 'Bearer <token>'
          }
        );

CREATE TABLE example

   AS SELECT * FROM read_csv('https://www.lotsofcsvs.com/example.csv');

CREATE TABLE your_table_name

   AS SELECT * FROM read_csv('https://www.lotsofcsvs.com/api/u/<username>/your.csv');

# It is also LLM friendly. There is a prompt telling LLMs how to use this site

# hidden in this page. Also, if you programmatically fetch this page

# it will return the prompt, similar to https://docs.jina.ai/.

curl -L https://www.lotsofcsvs.com/