As mentioned in the last blog post, a Spark Catalyst query plan is internally represented as a tree, defined as a subclass of the TreeNode type. Each tree node can have zero or more tree node children, which makes a mutually recursive structure. TreeNode type encapsulates a list of methods for traversing the tree and … Continue reading Spark SQL Query Engine Deep Dive (2) – Catalyst Rule Executor →

From this blog post on, I am going to start writing about Spark SQL Catalyst. Catalyst is the core of Spark SQL and there are many topics to cover. I don't have a formal writing plan on this, but instead, I want to keep my blog to be casual, informal, and more importantly stress-free to … Continue reading Spark SQL Query Engine Deep Dive (1) – Catalyst QueryExecution Overview →