#dataengineering

Spotify Engineering @spotify_engineering@robot.villas

· 3mo ago

Indexing the Data Lake for Online Point Queries https://engineering.atspotify.com/2026/7/indexing-the-data-lake-for-online-point-queries/ #DataEngineering #BigData #CloudComputing

Indexing the Data Lake for Online Point Queries | Spotify Engineering

engineering.atspotify.com

Indexing the Data Lake for Online Point Queries | Spotify Engineering

Companies like Spotify need vast quantities of data accessible at low latency for online services and, increasingly, AI Agents acting on behalf of users. Online services — such as portals and personalization features that use or display a user's listening history — need to look up and paginate through per-user data at interactive speeds. AI Agents answering questions like, "what was I listening to last summer?" need to retrieve a user's data quickly so they can reason over it — filtering, aggreg

Lobsters @lobsters@robot.villas

· 3mo ago

A Fast Path for Fixed-Length Lists in Parquet https://www.morling.dev/blog/fast-path-for-fixed-length-lists-in-parquet/ #DataEngineering #Performance #Parquet

morling.dev

A Fast Path for Fixed-Length Lists in Parquet

In its current form Apache Parquet isn’t a great fit for storing fixed-length lists, such as coordinates, RGB(A) colors, or—an increasingly common case—vector embeddings driving search and …

Self-Hosted Feed @selfhosted_bot@fd.mrmave.work

· 3mo ago

🛠️ pracdata/awesome-open-source-data-engineering A curated list of open source tools used in analytics platforms and data engineering ecosystem Compiles software tools across storage, streaming, integration, and metadata management for data architecture platforms. ⭐ Stars: 592 📅 Last Update: Jul 24, 2026 https://github.com/pracdata/awesome-open-source-data-engineering #selfhosted #homelab #selfhost #selfhosting #opensource #dataengineering #databases

github.com

GitHub - pracdata/awesome-open-source-data-engineering: A curated list of open source tools used in analytics platforms and data engineering ecosystem

A curated list of open source tools used in analytics platforms and data engineering ecosystem - pracdata/awesome-open-source-data-engineering

Foodspark @foodspark@mstdn.business

· 3mo ago

Building an in-house scraper? Before investing months in development, compare the real costs with a managed data solution. Learn about: 📊 Cost ⚡ Scalability 🛠 Maintenance 🔒 Reliability 📈 ROI Read here 👇 https://www.foodspark.io/foodspark-vs-in-house-data-scraping/ #WebScraping #FoodData #DataEngineering

Foodspark vs In-House Data Scraping for Food Businesses

foodspark.io

Foodspark vs In-House Data Scraping for Food Businesses

Compare FoodSpark and in-house data scraping to understand cost, speed, accuracy, scalability, maintenance, and data quality for food businesses.

Psycopg @psycopg@fosstodon.org

· 3mo ago

We've launched a GitHub Sponsors page for Psycopg 🐘🐍 Psycopg has been the standard PostgreSQL adapter for Python for 20 years. Your sponsorship goes directly towards maintaining Psycopg 2 and Psycopg 3, and keeping the project active and up to date. Sponsors are also listed on our new sponsors page on the Psycopg website. Links in replies 👇 #PostgreSQL #Python #Psycopg #OpenSource #FOSS #DataEngineering

You've seen all posts