![\"A](\"media/foursquare-os-places-density-2024.png\")
My day job involves wrangling a lot of data very fast.\nI've heard a lot of people raving about several technologies like DuckDB,\n(Geo)Parquet, and Apache Arrow recently.\nBut despite being an "early adopter,"\nit took me quite a while to figure out how and why to leverage these practiclaly.
\nLast week, a few things "clicked" for me, so I'd like to share what I learned in case it helps you.
\n(Geo)Parquet is quite possibly the best understood tech in the mix.\nIt is not exactly new.\nParquet has been around for quite a while in the big data ecosystem.\nIf you need a refresher, the Cloud-optimized Geospatial Formats Guide\ngives a great high-level overview.
\nHere are the stand-out features:
\nPractically speaking, parquet lets you can distribute large datasets in one or more files\nwhich will be significantly smaller and faster to query than other familiar formats.
\nThe value proposition is clear for big data processing.\nIf you're trying to get a record of all traffic accidents in California,\nor find the hottest restaurants in Paris based on a multi-terabyte dataset,\nparquet provides clear advantages.\nYou can skip row groups within the parquet file or even whole files\nto narrow your search!\nAnd since datasets can be split across files,\nyou can keep adding to the dataset over time, parallelize queries,\nand other nice things.
\nBut what if you're not doing these high-level analytical things?\nWhy not just use a more straightforward format like CSV\nthat avoids the need to "rotate" back into rows\nfor non-aggregation use cases?\nHere are a few reasons to like Parquet:
\nThe upshot of all this is that it generally gets both easier and faster\nto work with your data...\nprovided that you have the right tools to leverage it.
\nDuckDB describes itself as an in-process, portable, feature-rich, and fast database\nfor analytical workloads.\nDuckDB was that tool that triggered my "lightbulb moment" last week.\nFoursquare, an app which I've used for a decade or more,\nrecently released an open data set,\nwhich was pretty cool!\nIt was also in Parquet format (just like Overture's data sets).
\nYou can't just open up a Parquet file in a text editor or spreadsheet software like you can a CSV.\nMy friend Oliver released a web-based demo\na few weeks ago which lets you inspect the data on a map at the point level.\nBut to do more than spot checking, you'll probably want a database that can work with Parquet.\nAnd that's where DuckDB comes in.
\nI understood the in-process part of DuckDB's value proposition right away.\nIt's similar to SQLite, where you don't have to go through a server\nor over an HTTP connection.\nThis is both simpler to reason about and is usually quite a bit faster\nthan having to call out to a separate service!
\nDuckDB is pretty quick to compile from source.\nYou probably don't need to muck around with this if you're just using the CLI,\nbut I wanted to eventually use it embedded in some Rust code.\nCompiling from source turned out to be the easiest way to get their crate working.\nIt looks for a shared library by default, but I couldn't get this working after a brew install.\nThis was mildly annoying, but on the other hand,\nvendoring the library does make consistent Docker builds easier 🤷🏻♂️
DuckDB includes a mind boggling number of features.\nNot in a confusing way; more in a Python stdlib way where just about everything you'd want is already there.\nYou can query a whole directory (or bucket) of CSV files,\na Postgres database, SQLite, or even an OpenStreetMap PBF file 🤯\nYou can even write a SQL query against a glob expression of Parquet files in S3\nas your "table."\nThat's really cool!\n(If you've been around the space, you may recognize this concept from\nAWS Athena and others.)
\nWriting a query against a local directory of files is actually really fast!\nIt does a bit of munging upfront, and yes,\nit's not quite as fast as if you'd prepped the data into a clean table,\nbut you actually can run quite efficient queries this way locally!
\nWhen running a query against local data,\nDuckDB will make liberal use of your system memory\n(the default is 80% of system RAM)\nand as many CPUs as you can throw at it.\nBut it will reward you with excellent response times,\ncourtesy of the "vectorized" query engine.\nWhat I've heard of the design reminds me of how array-oriented programming languages like APL\n(or less esoteric libraries like numpy) are often implemented.
\nI was able to do some spatial aggregation operations\n(bucketing a filtered list of locations by H3 index)\nin about 10 seconds on a dataset of more than 40 million rows!\n(The full dataset is over 100 million rows, so I also got to see the selective reading in action.)\nThat piqued my interest, to say the least.\n(Here's the result of that query, visualized).
\n
And now for the final buzzword in DuckDB's marketing: analytical.\nDuckDB frequently describes itself as optimized for OLAP (OnLine Analytical Processing) workloads.\nThis is contrasted with OLTP (OnLine Transaction Processing).\nWikipedia will tell you some differences\nin a lot of sweepingly broad terms, like being used for "business reporting" and read operations\nrather than "transactions."
\nWhen reaching for a definition, many sources focus on things like aggregation queries\nas a differentiator.\nThis didn't help, since most of my use cases involve slurping most or all of the data set.\nThe DuckDB marketing and docs didn't help clarify things either.
\nLet me know on Mastodon if you have a better explanatation of what an "analytical" database is 🤣
\nI think a better explanation is probably 1) you do mostly read queries,\nand 2) it can execute highly parallel queries.\nSo far, DuckDB has been excellent for both the "aggregate" and the "iterative" use case.\nI assume it's just not the best choice per se if your workload is a lot of single-record writes?
\nEmbedding DuckDB in a Rust project allowed me to deliver something with a better end-user experience,\nis easier to maintain,\nand saved writing hundreds of lines of code in the process.
\nMost general-purpose languages like Python and Rust\ndon't have primitives for expressing things like joins across datasets.\nDuckDB, like most database systems, does!\nYes, I could write some code using the parquet crate\nthat would filter across a nested directory tree of 5,000 files.\nBut DuckDB does that out of the box!
It feels like this is a "regex moment" for data processing.\nJust like you don't (usually) need to hand-roll string processing,\nthere's now little reason to hand-roll data aggregation.
\nFor the above visualization, I used the Rust DuckDB crate for the data processing,\nconverted the results to JSON,\nand served it up from an Axum web server.\nAll in a single binary!\nThat's lot nicer than a bash script that executes SQL,\ndumps to a file, and then starts up a Python or Node web server!\nAnd breaks when you don't have Python or Node installed,\nyour OS changes its default shell,\nyou forget that some awk flag doesn't work on the GNU version,\nand so on.
\nThe final thing I want to touch on is Apache Arrow.\nThis is yet another incredibly useful technology which I've been following for a while,\nbut never quite figured out how to properly use until last week.
\nArrow is a language-independent memory format\nthat's optimized for efficient analytic operations on modern CPUs and GPUs.\nThe core idea is that, rather than having to convert data from one format to another (this implies copying!),\nArrow defines as shared memory format which many systems understand.\nIn practice, this ends up being a bunch of standards which define common representations for different types,\nand libraries for working with them.\nFor example, the GeoArrow spec\nbuilds on the Arrow ecosystem to enable operations on spatial data in a common memory format.\nPretty cool!
\nIt turns out that copying and format shifting data can really eat into your processing times.\nArrow helps you sidestep that by reducing the amount of both you'll need to do,\nand by working on data in groups.
\nArrow is mostly hidden from view beneath other libraries.\nSo most of the time, especially if you're writing in a very high level language like Python,\nyou won't even see it.
\nBut if you're writing something at a slightly lower level,\nit's something you may have to touch for critical sections.\nThe DuckDB crate\nincludes an Arrow API\nwhich will give you an iterator over RecordBatches.\nThis is pretty convenient, since you can use DuckDB to gather all your data\nand just consume the stream of batches!
So, how do we work with RecordBatches?\nThe Arrow ecosystem, like Parquet, takes a lot of work to understand,\nand using the low-level libraries directly is difficult.\nEven as a seasoned Rustacean, I found the docs rather obtuse.
After some searching, I finally found serde_arrow.\nIt builds on the serde ecosystem with easy-to-use methods that operate on RecordBatches.\nFinally; something I can use!
I was initilaly worried about how performant the shift from columns to rows + any (minimal) serde overhead would be,\nbut this turned out to not be an issue.
Here's how the code looks:
\nserde_arrow::from_record_batch::<Vec<FoursquarePlaceRecord>>(&batch)\nA few combinators later and you've got a proper data pipeline!
\nWhat this ultimately enabled for me was being able to get a lot closer to "scripting"\na pipeline in Rust.\nMost people turn to Python or JavaScript for tasks like this,\nbut Rust has something to add: strong typing and all the related guarantees which can only come with some level of formalism.\nBut that doesn't necessarily have to get in the way of productivity!
\nHopefully this sparks some ideas for making your next data pipeline both fast and correct.
\n", "summary": "", "date_published": "2024-12-08T00:00:00-00:00", "image": "media/foursquare-os-places-density-2024.png", "authors": [ { "name": "Ian Wagner", "url": "https://fosstodon.org/@ianthetechie", "avatar": "media/avi.jpeg" } ], "tags": [ "rust", "apache arrow", "parquet", "duckdb", "big data", "data engineering", "gis" ], "language": "en" }, { "id": "https://ianwwagner.com//quadrupling-the-performance-of-a-data-pipeline.html", "url": "https://ianwwagner.com//quadrupling-the-performance-of-a-data-pipeline.html", "title": "Quadrupling the Performance of a Data Pipeline", "content_html": "Over the past two weeks, I've been focused on optimizing some data pipelines.\nI inherited some old ones which seemed especially slow,\nand I finally hit a limit where an overhaul made sense.\nThe pipelines process and generate data on the order of hundreds of gigabytes,\nrequiring correlation and conflated across several datasets.
\nThe pipelines in question happened to be written in Node.js,\nwhich I will do my absolute best not to pick on too much throughout.\nNode is actually a perfectly fine solution for certain problems,\nbut was being used especially badly in this case.\nThe rewritten pipeline, using Rust, clocked in at 4x faster than the original.\nBut as we'll soon see, the choice of language wasn't even the main factor in the sluggishness.
\nSo, let's get into it...
\nNode.js made a splash in the early 2010s,\nand I can remember a few years where it was the hot new thing to write everything in.\nOne of the selling points was its ability to handle thousands (or tens of thousands)\nof connections with ease; all from JavaScript!\nThe key to this performance is async I/O.\nModern operating systems are insanely good at this, and Node made it really easy to tap into it.\nThis was novel to a lot of developers at the time, but it's pretty standard now\nfor building I/O heavy apps.
\nNode performs well as long as you were dealing with I/O-bound workloads,\nbut the magic fades if your workload requires a lot of CPU work.\nBy default, Node is single-threaded.\nYou need to bring in libuv, worker threads (Node 10 or so), or something similar\nto access parallel processing from JavaScript.\nI've only seen a handful of Node programs use these,\nand the pipelines in question were not among them.
If you ingest data files (CSV and the like) record-by-record in a naïve way,\nyou'll just read one record at a time, process, insert to the database, and so on in a loop.\nThe original pipeline code was fortunately not quite this bad (it did have batching at least),\nbut had some room for improvemnet.
\nThe ingestion phase, where you're just reading data from CSV, parquet, etc.\nmaps naturally to Rust's streams\n(the cousin of futures).\nThe original node code was actually fine at this stage,\nif a bit less elegant.\nBut the Rust structure we settled on is worth a closer look.
\nfn csv_record_stream<'a, S: AsyncRead + Unpin + Send + 'a, T: TryFrom<StringRecord>>(\n stream: S,\n delimiter: u8,\n) -> impl Stream<Item = T> + 'a\nwhere\n <T as TryFrom<StringRecord>>::Error: Debug,\n{\n let reader = AsyncReaderBuilder::new()\n .delimiter(delimiter)\n // Other config elided...\n .create_reader(stream);\n reader.into_records().filter_map(|res| async move {\n let Ok(record) = res else {\n log::error!("Error reading from the record stream: {:?}", res);\n return None;\n };\n\n match T::try_from(record) {\n Ok(parsed) => Some(parsed),\n Err(e) => {\n log::error!("Error parsing record: {:?}.", e);\n None\n }\n }\n })\n}\nIt starts off dense, but the concept is simple.\nWe'll take an async reader,\nconfigure a CSV reader to pull records for it,\nand map them to another data type using TryFrom.\nIf there are any errors, we just drop them from the stream and log an error.\nThis usually isn't a reason to stop processing for our use case.
You should not be putting expensive code in your TryFrom implementation.\nBut really quick things like verifying that you have the right number of fields,\nor that a field contains an integer or is non-blank are usually fair game.
Rust's trait system really shines here.\nOur code can turn any CSV(-like) file\ninto an arbitrary record type.\nAnd the same techniques can apply to just about any other data format too.
\nNow that we've done the light format shifting and discarded some obviously invalid records,\nlet's turn to the heavier processing.
\nlet available_parallelism = std::thread::available_parallelism()?.get();\n// let record_pipeline = csv_record_stream(...);\nrecord_pipeline\n .chunks(500) // Batch the work (your optimal size may vary)\n .for_each_concurrent(available_parallelism, |chunk| {\n // Clone your database connection pool or whatnot before `move`\n // Every app is different, but this is a pretty common pattern\n // for sqlx, Elastic Search, hyper, and more which use Arcs and cheap clones for pools.\n let db_pool = db_pool.clone();\n async move {\n // Process your records using a blocking threadpool\n let documents = tokio::task::spawn_blocking(move || {\n // Do the heavy work here!\n chunk\n .into_iter()\n .map(do_heavy_work)\n .collect()\n })\n .await\n .expect("Problem spawning a blocking task");\n\n // Insert processesd data to your database\n db_pool.bulk_insert(documents).await.expect("You probably need an error handling strategy here...");\n }\n })\n .await;\nWe used the chunks\nadaptor to pull hundreds of items at a time for more efficient processing in batches.\nThen, we used for_each_concurrent\nin conjunction with spawn_blocking\nto introduce parallel processing.
Note that neither chunks nor even for_each_concurrent imply any amount of parallelism\non their own.\nspawn_blocking is the only thing that can actually create a new thread of execution!\nChunking simply splits the work into batches (most workloads like this tend to benefit from batching).\nAnd for_each_concurrent allows for concurrent operations over multiple batches.\nBut spawn_blocking is what enables computation in a background thread.\nIf you don't use spawn_blocking,\nyou'll end up blocking Tokio's async workers,\nand your performance will tank.\nJust like the old Node.js code.
The astute reader may point out that using spawn_blocking like this\nis not universally accepted as a solution.\nTokio is (relatively) optimized for non-blocking workloads, so some claim that you should avoid this pattern.\nBut my experience having done this for 5+ years in production code serving over 2 billion requests/month,\nis that Tokio can be a great scheduler for heavier tasks too!
One thing that's often overlooked in these discussions\nis that not all "long-running operations" are the same.\nOne category consists of graphics event loops,\nlong-running continuous computations,\nor other things that may not have an obvious "end."\nBut some tasks can be expected to complete within some period of time,\nthat's longer than a blink.
\nIn the case of the former ("long-lived" tasks), then spawning a dedicated thread often makes sense.\nIn the latter scenario though, Tokio tasks with spawn_blocking can be a great choice.
For our workload, we were doing a lot of the latter sort of operation.\nOne helpful rule of thumb I've seen is that if your task takes longer than tens of microseconds,\nyou should move it off the Tokio worker threads.\nUsing chunks and spawn_blocking avoids this death by a thousand cuts.\nIn our case, the parallelism resulted in a VERY clear speedup.
The original data pipeline was very careful to not overload the data store.\nPerhaps a bit too careful!\nThis may have been necessary at some point in the distant past,\nbut most data storage, from vanilla databases to multi-node clustered storage,\nhave some level of natural backpressure built-in.\nThe Node implementation was essentially limiting the amount of work in-flight that hadn't been flushed.
\nThis premature optimization and the numerous micro-pauses it introduced\nwere another death by a thousand cuts problem.\nDropping the artificial limits approximately doubled throughput.\nIt turned out that our database was able to process 2-4x more records than under the previous implementation.
\nTL;DR — set a reasonable concurrency, let the server tell you when it's chugging (usually via slower response times),\nand let your async runtime handle the rest!
\nSerde, or serialization + deserialization, can be a silent killer.\nAnd unless you're tracking things carefully, you often won't notice!
\nI recently listened to Recoding America at the recommendation of a friend.\nOne of the anecdotes made me want to laugh and cry at the same time.\nEngineers had designed a major improvemnet to GPS, but the rollout is delayed\ndue to a performance problem that renders it unusable.
\nThe project is overseen by Raytheyon, a US government contractor.\nAnd they can't deliver because some arcane federal guidance (not even a regulation proper)\n"recommends" an "Enterprise Service Bus" in the architecture.\nThe startuppper in me dies when I hear such things.\nThe "recommendation" boils down to a data exchange medium where one "service" writes data and another consumes it.\nThink message queues like you may have used before.
\nThis is fine (even necessary) for some applications,\nbut positively crippling for others.\nIn the case of the new positioning system,\nwhich was heavily dependent on timing,\nthis was a wildly inefficient architecture.\nEven worse, the guidelines stated that it should be encrypted.
\nThis wasn't even "bad" guidance, but in the context of the problem,\nwhich depended on rapid exchange of time-sensitive messages,\nit was a horrendously bad fit.
\nIn our data pipeline, I discovered a situation with humorous resemblance in retrospect.\nThe pipeline was set up using a microservice architecture,\nwhich I'm sure souded like a good idea at the time,\nbut it introduced some truly obscene overhead.\nAll services involved were capable of working with data in the same format,\nbut the Node.js implementation was split into multiple services with HTTP and JSON round trips in the middle!\nDouble whammy!
\nThe new data pipeline simply imports the "service" as a crate,\nand gets rid of all the overhead by keeping everything in-process.\nIf you do really need to have a microservice architecture (ex: to scale another service up independently),\nthen other communication + data exchange formats may improve your performance.\nBut if it's possible to keep everything in-process, your overhead is roughly zero.\nThat's hard to beat!
\nIn the end, the new pipeline was 4x the speed of the old.\nI happened to rewrite it in Rust, but Rust itself wasn't the source of all the speedups:\nunderstanding the architecture was.\nYou could achieve similar results in Node.js or Python,\nbut Rust makes it significantly easy to reason about the architecture and correctness of your code.\nThis is especially important when it comes to parallelizing sections of a pipeline,\nwhere Rust's type system will save you from the most common mistakes.
\nThese and other non-performance-related reasons to use Rust will be the subject of a future blog post (or two).
\n", "summary": "", "date_published": "2024-11-29T00:00:00-00:00", "image": "", "authors": [ { "name": "Ian Wagner", "url": "https://fosstodon.org/@ianthetechie", "avatar": "media/avi.jpeg" } ], "tags": [ "algorithms", "rust", "elasticsearch", "nodejs", "data engineering", "gis" ], "language": "en" }, { "id": "https://ianwwagner.com//searching-for-tiger-features.html", "url": "https://ianwwagner.com//searching-for-tiger-features.html", "title": "Searching for TIGER Features", "content_html": "Today I had a rather peculiar need to search through features from TIGER\nmatching specific attributes.\nThese files are not CSV or JSON, but rather ESRI Shapefiles.\nShapefiles are a binary format which have long outlived their welcome\naccording to many in the industry, but they still persist today.
\nYeah, so this post probably isn't interesting to very many people,\nbut here's a bit of context in case you don't know what's going on and you're still reading.\nTIGER is a geospatial dataset published by the US government.\nThere's far more to this dataset than fits in this TIL post,\nbut my interest in it lies in finding addresses.\nSpecifically, guessing at where an address might be.
\nWhen you type an address into your maps app,\nthey might not actually have the exact address in their database.\nThis happens more than you might imagine,\nbut you can usually get a pretty good guess of where the address is\nvia a process called interpolation.\nThe basic idea is that you take address data from multiple sources and use that to make a better guess.
\nSome of the input to this is existing address points.\nBut there's one really interesting form of data that brings us to today's TIL:\naddress ranges.\nOne of the TIGER datasets is a set of lines (for the roads.\nEach segment is annotated with info letting us know the range of house numbers on each side of the road.
\nI happen to use this data for my day job at Stadia Maps,\nwhere I was investigating a data issue today related to our geocoder and TIGER data.
\nIn case you find yourself in a similar situation,\nyou may notice that the data from the government is sitting in an FTP directory,\nwhich contains a bunch of confusingly named ZIP files.\nThe data that I'm interested in (address features)\nhas names like tl_2024_48485_addrfeat.zip.
The year might be familiar, but what's that other number?\nThat's a FIPS code for the county whose data is contained in the archive.\nYou can find a list here.\nThis is somewhat interesting in itself, since the first 2 characters are a state code.\nTexas, in this case.\nThe full number makes up a county: Wichita County, in this case.\nYou can suck down the entire dataset, just one file, or anything in-between\nfrom the Census website.
\nSo, now you have a directory full of ZIP files.\nEach of which has a bunch of files necessary to interpret the shapefile.\nIsn't GIS lovely?
\nThe following script will let you write a simple "WHERE" clause,\nfiltering the data exactly as it comes from the Census Bureau!
\n#!/bin/bash\nset -e;\n\nfind "$1" -type f -iname "*.zip" -print0 |\\\n while IFS= read -r -d $'\\0' filename; do\n\n filtered_json=$(ogr2ogr -f GeoJSON -t_srs crs:84 -where "$2" /vsistdout/ /vsizip/$filename);\n # Check if the filtered GeoJSON has any features\n feature_count=$(echo "$filtered_json" | jq '.features | length')\n\n if [ "$feature_count" -gt 0 ]; then\n # echo filename to stderr\n >&2 echo $(date -u) "Match(es) found in $filename";\n echo "$filtered_json";\n fi\n\n done;\nYou can run it like so:
\n./find-tiger-features.sh $HOME/Downloads/tiger-2021/ "TFIDL = 213297979 OR TFIDR = 213297979"\nThis ends up being a LOT easier and faster than QGIS in my experience\nif you want to search for specific known attributes.\nEspecially if you don't know the specific area that you're looking for.\nI was surprised that so such tool for things like ID lookps existed already!
\nNote that this isn't exactly "fast" by typical data processing workload standards.\nIt takes around 10 minutes to run on my laptop.\nBut it's a lot faster than the alternatives in many circumstances,\nespecilaly if you don't know exactly which file the data is in!
\nFor details on the fields available,\nrefer to the technical documentation on the Census Bureau website.
\n", "summary": "", "date_published": "2024-11-09T00:00:00-00:00", "image": "", "authors": [ { "name": "Ian Wagner", "url": "https://fosstodon.org/@ianthetechie", "avatar": "media/avi.jpeg" } ], "tags": [ "gis", "shell", "ogr2ogr" ], "language": "en" } ] }