Litelink
Python library that captures streaming data durably into Iceberg tables with no broker, daemon, or catalog service to run.
🔗 Visit Litelink
Description
Capturing a continuous stream of events into a data lake table format like Iceberg usually means standing up infrastructure: a message broker (Kafka), a catalog service, maybe a daemon process to batch and write. That's a lot of moving parts for a team that just wants events durably landing in queryable tables. Litelink strips that down to a Python library you import — no separate services to run.
Litelink is a Python library that captures data append-only into Iceberg tables through a three-tier local architecture: writes land first in a SQLite buffer, get promoted to a local Iceberg table, and eventually sync to remote S3. A write is considered durable as soon as append() returns, giving sub-second freshness without sacrificing durability guarantees, and a write-ahead log supports replication for failover if the writing machine goes down. Because the resulting tables are standard Iceberg, they're readable by any standard Iceberg query engine, and Litelink supports snapshot-based reads from other machines. It's open source under Apache 2.0.
💬 Our review
The short version: Litelink's whole value proposition is operational simplicity — durable, queryable Iceberg data capture as a Python import instead of a Kafka cluster plus a catalog service plus a batching daemon, which is a real reduction in infrastructure for teams whose ingest volume doesn't actually need Kafka-scale throughput.
The three-tier local buffer design (SQLite → local Iceberg → S3) is a sensible way to get both immediate durability and eventual centralized queryability without needing a broker in the write path at all. This trades off against Kafka-based pipelines in the obvious way: Litelink is embedded and local-first, so it's a better fit for single-writer or moderate-throughput capture scenarios than for massive distributed ingest where a broker's fan-out and replay guarantees genuinely earn their operational cost. Being Apache 2.0 and outputting standard Iceberg tables avoids any lock-in — you can point any Iceberg-compatible engine (Spark, Trino, DuckDB) at the result without needing Litelink again. Worth adopting for teams wanting durable, queryable capture without new infrastructure to operate; not a replacement for Kafka-scale distributed streaming.
💰 Pricing
📊 Global score
🤖 AI-enriched data
Apache License 2.0, gratuit
Pros
Aucun broker, daemon ou service de catalogue à opérer
Durabilité immédiate au retour de append(), fraîcheur sub-seconde
Sortie en tables Iceberg standard, lisibles par n'importe quel moteur
Réplication WAL pour failover
Cons
Pas adapté à de l'ingestion distribuée à très grande échelle (vs Kafka)
Bibliothèque Python jeune, communauté limitée
Architecture locale-first moins adaptée au multi-writer distribué