Unified Data Platform – One interface to query, discover, and collaborate on data. Data versioning & federation, built with Postgres, inspired by Docker & Git.splitgraph.com 📍 Remote — USA, UK, EuropeJoined October 2018
📢 Exciting News! We have acquired @splitgraph, a serverless SQL API disruptor. This strategic move enhances our commitment to solving diverse data challenges, & will make data lake/warehousing operations more efficient, streamlined, and collaborative. ⬇ okt.to/271y0b
@WillSpagnoli@OpenAI@langchain You can query data on Splitgraph with any Postgres client: splitgraph.com/connect/query
Caveat is introspection queries to 𝚒𝚗𝚏𝚘𝚛𝚖𝚊𝚝𝚒𝚘𝚗_𝚜𝚌𝚑𝚎𝚖𝚊 don't return 40k+ tables, hence why the demo fetches schemas from GraphQL API to create embeddings (see GitHub link in blog)
Splitgraph indexes 40,000 public datasets and makes them queryable via SQL.
We're experimenting with @OpenAI and @langchain to generate SQL queries from questions, like "Which residential neighborhood of Chicago is the most expensive?"
Read the blog: splitgraph.com/blog/gpt-power…
@WillSpagnoli@OpenAI@langchain Agreed, we'd like to try the GPT functions next, but they were released while we were developing this :)
We also just got access to the @OpenAI plugin developer program, so stay tuned for a Splitgraph ChatGPT plug-in!
@woundedkarma @OpenAI@langchain@AirbyteHQ The GitHub repo in this blog post is a demo that you can run locally to index embeddings for any namespace on Splitgraph (it gets the schemas from our GraphQL API)
We're productionizing the feature and adding it to the Splitgraph Console (the web app) at splitgraph.com/query
@woundedkarma @OpenAI@langchain It's public! We index (not ingest) open datasets and proxy queries to their origin with our Socrata plugin. Other data sources can be live queried via FDW plugins.
You can *ingest* data snapshots e.g. via 100+ @AirbyteHQ plugins, or CSV/SQLite upload, optionally in private repos
@OpenAI@langchain How?
1. Index each repository in a namespace (e.g. 𝚍𝚊𝚝𝚊.𝚌𝚒𝚝𝚢𝚘𝚏𝚌𝚑𝚒𝚌𝚊𝚐𝚘.𝚘𝚛𝚐) to create @OpenAI embeddings via @langchain and PgVector.
2. Compare the question to embeddings
3. Retrieve schemas from Splitgraph API
4. Generate SQL
github.com/splitgraph/spl…
People often assume that "Splitgraph" must have something to do with GraphQL, but nope 🙂We're a serverless data platform!
BUT! We do *use* GraphQL in our web stack...
Here are some best practices we've learned about @apollographql cache invalidation:
splitgraph.com/blog/keeping-a…
Seafowl is a serverless OLAP database for analytics "at the edge." It needs to store metadata (schemas, etc.) in a SQLite or Postgres DB, but how do we avoid that bottlenecking?
We can deploy to @googlecloud and mount the #SQLite catalog with GCS FUSE!
splitgraph.com/blog/deploying…
Seafowl lets you deliver query results for data visualizations, dashboards and notebooks in milliseconds.
The team at Splitgraph published a neat guide on using Dagster to orchestrate these interactive analytical applications:
splitgraph.com/blog/seafowl-d…
SQLite Postgres
🤝
Splitgraph
🔼 Upload SQLite files to the Data Delivery Network
🔐Share tables publicly, or invite collaborators
📈 Query them with SQL
splitgraph.com/blog/sqlite-fi…
Splitgraph indexes 40,000+ open datasets, from 200+ government data portals - like those from @CDCgov and @USAID - and makes them all queryable with #SQL
We made a newsfeed for tracking daily additions (and deletions!) of datasets on those data portals:
open-data-monitor.splitgraph.io
BLOG: Seafowl's storage layer is now built on delta-rs, a Rust native interface to Delta Lake (a protocol pioneered by @databricks for the "Lakehouse" pattern).
It's a perfect fit for Seafowl, a database optimized for cache-friendly queries "at the edge"
splitgraph.com/blog/seafowl-d…
Seafowl is an analytical database for modern data-driven Web applications, written in @rustlang with #Datafusion and optimized for running cache-friendly queries "at the edge."
The storage layer is now based on the Delta Lake protocol, thanks to delta-rs
github.com/splitgraph/sea…
342 Followers 1K FollowingEX Co-Founder && Founding Engineer @databendlabs, focus on #rust and #vibe, opinions my own. (in San Jose) https://t.co/Tg8FssNPLP
90K Followers 8K Followingdespite all my ragie I'm still just a wagie in a cagie
working on DL Software: https://t.co/FVn3NRNrLe
https://t.co/CgaoMfhUHd
577 Followers 4K FollowingMi appassiona di più il come del cosa. I am more passionate about the how than the what.
( "correlation does not imply causation" )
390 Followers 2K FollowingSr Data Scientist @Microsoft. Part-Time Lecturer @UW_iSchool. MSc in Data Science & BS in Informatics @UW. Tweets are my own and not the views of my employers.
907 Followers 2K FollowingPhD Student in Rehabilitation Science - Pitt
+62 🇮🇩 currently living as a registered Alien in +1 🇺🇸 | I heart Coffee, Traveling and Code for living(fun) 🌿
125 Followers 8 FollowingSoftware person, mostly | Co-founder of @splitgraph (acquired by @EDBPostgres) | Prev.: @ManQuantTech, @Cambridge_Uni | Any controversy I start is mine.