What Spark is actually doing when you call an action: topology, stages and shuffles, the cost of cardinality, a global sort that cost 37.8 GiB, and the three optimisers. Notes and Spark UI screenshots from a Databricks project.
API · lifecycle · FastAPI + PostgreSQL Every API is a conversation with a contract. An API is the structured
How the two slowly changing dimension patterns are actually built: LEAD and LAG for Type 2 intervals and change detection, UPDATE with a delete marker for Type 1, and the failure modes worth remembering.
A working tour of Amazon Aurora: the shared storage design everything else hangs off, then Serverless v2, Blue/Green, zero-ETL, I/O-Optimized, Optimized Reads, Limitless and DSQL, checked against where AWS has actually got to.
Notes on a metadata-driven ELT platform at a large UK general insurer. What the pattern bought us, the seven ways it went wrong, and the one structural property behind all of them.
Soil, drainage, slope and climate decide what a vine does with its energy: how wine grapes are grown, which varieties suit which conditions, and why the best vineyard land often looks like the worst farmland.
PySpark · working reference · DataFrame API Everything is a questionabout the shuffle. Spark builds a plan and does nothing.
Insight into 3NF v Star v Snowflake Schemas: In this article 3NF Star Schema Snowflake Schema 1.Third Normal Form (3NF)
This article is still under review & incomplete with on going research & edits . Please bear that in mind
Understanding Lambda Architecture in Modern Data Engineering: What is Lambda Architecture? I only recently came across this type of data
AWS SAM:Simplifying Serverless Application Development In thisThis article is still under review & incomplete with on going research & edits
The Cognitive Benefits of Playing Chess In this article For the last few years I have been chess everyday for
Snowflake Architecture Demystified When studying for the SnowPro Core Certification, one of the areas that comes up again and again
Introduction & Why Snowflake? In this article When I first started revising for the SnowPro Core Certification, one of the
Inside AWS Aurora: Key Takeaways from an In-Depth Technical Workshop From architecture and Serverless v2 to Blue/Green deployments, Zero-ETL, I/O-Optimized