Exploring the future of financial data architecture: AI-native FICC canonical models, large-scale financial data processing, and enterprise data quality & governance.
This blog post continues to discuss the partitioning and ordering in Spark. In the last blog post, I explain the SortExec operator and the underlying UnsafeExternalSorter for ordering. This blog post focuses on the ShuffleExcahngeExec operator and the Tungsten supported shuffleExternalWriter for partitioning. As explained in the previous blog posts, when the output partitioning of … Continue reading Spark SQL Query Engine Deep Dive (16) – ShuffleExchangeExec & UnsafeShuffleWriter
In the last blog post, I explained the partitioning and ordering requirements for preparing a physical operator to execute. In this and the next blog post, I look into the primary physical operators for implementing partitioning and ordering. This blog post focuses on the SortExec operator and the UnsafeExternalSorter for ordering, while the next blog … Continue reading Spark SQL Query Engine Deep Dive (15) – UnsafeExternalSorter & SortExec
In the last few blog posts, I introduced the SparkPlanner for generating physical plans from logical plans and looked into the details of the aggregation, join and runnable command execution strategies. When a physical plan is selected as the "best" plan by a cost model (not implemented in Spark 3.0 yet though) or other approaches, … Continue reading Spark SQL Query Engine Deep Dive (14) – Partitioning & Ordering
This blog post looks into Spark SQL Cache Commands under the hood, walking through the execution flows of the persisting and unpersisting operations, from the physical execution plan to the cache block storages. Cache Commands Spark SQL ships with three runnable commands for caching operations, including CacheTableCommand, UncacheTableCommand, and ClearCacheCommand. End-user developers or analysts can … Continue reading Spark SQL Query Engine Deep Dive (13) – Cache Commands Internal
In this blog posts, I will dig into the execution internals of the runnable commands, which inherit from the RunnableCommand parent class. The runnable commands are normally the commands for interacting with the Spark session catalog and managing the metadata. Unlike the data query alike operations which are distributed and lazily executed in Spark, the … Continue reading Spark SQL Query Engine Deep Dive (12) – SessionCatalog & RunnableCommand Internal
In this blog post, I am going to explain the Join strategies applied by the Spark Planner for generating physical Join plans. Based on the Join strategy selection logic implemented in the JoinSelection object (core object for planning physical join operations), I draw the following decision tree chart. Spark SQL ships five built-in Join physical … Continue reading Spark SQL Query Engine Deep Dive (11) – Join Strategies
This blog post continues to explore the Aggregate strategy and focuses on the two hash-based aggregation operators provided by Spark SQL, HashAggregateExec and ObjectHashAggregateExec. Hash-based aggregation is the preferred approach to sort-based aggregation which was explained in the last blog post. Compared to the sort-based aggregation, the hash-based aggregation does not need the extra sorting … Continue reading Spark SQL Query Engine Deep Dive (10) – HashAggregateExec & ObjectHashAggregateExec
The last blog post explains the Aggregation strategy for generating physical plans for aggregate operations. I will continue with this topic to look into the details of the physical aggregate operators supported by Spark SQL. As explained in the last blog post, a logical aggregate operator can be transformed into a physical plan consisting of … Continue reading Spark SQL Query Engine Deep Dive (9) – SortAggregateExec
In the last blog post, I gave an overview of the SparkPlanner for planning physical execution plans from an optimised logical plan. In the next few blog posts, I will dig a bit deeper into some important strategies. I will start with the "Aggregation" strategy in this blog post. "Aggregation" strategy plans the physical execution … Continue reading Spark SQL Query Engine Deep Dive (8) – Aggregation Strategy
After logical plans are optimised by the Catalyst Optimizer rules, SparkPlanner takes an optimized logical plan and converts it into one or more candidate physical plans. The best physical plan is expected to be selected based on a cost model approach, i.e. comparing the CPU and IO costs of the candidate physical plans and selecting … Continue reading Spark SQL Query Engine Deep Dive (7) – Spark Planner Overview
Data Ninjago (Financial Data Architecture) 
You must be logged in to post a comment.