Skip to content

Linxiao's technical blog over Data Engineering, BI, Machine Learning, Algo Trading and Data Quality Improvement in Financial Services

Azure Data Ninjago & dqops

About

4

Linxiao Ma
November 12, 2017

Leave a Reply Cancel reply

Enter your comment here...

Please log in using one of these methods to post your comment:

Email (Address never made public)

Name

Website

You are commenting using your WordPress.com account. ( Log Out / Change )

You are commenting using your Facebook account. ( Log Out / Change )

Connecting to %s

Notify me of new comments via email.

Notify me of new posts via email.

Δ

Follow Azure Data Ninjago & dqops on WordPress.com

Author

Linxiao Ma

Categories

*Deep Dive – Parquet for Spark (4)
*Deep Dive – Spark SQL Query Engine (20)
*Deep Dive – Spark Structured Streaming (8)
Azure Data Platform (25)
Data Quality / dqops (11)
Databricks / Spark (30)
Machine Learning / Data Mining (10)
Power BI/DAX/M (18)
Python/R (11)
SSIS/SSAS/SSRS/MDX (17)
The Others (8)

Tags

Azure Batch Azure Data Factory Azure Data Lake Analytics Azure Data Lake Store Azure Stream Analytics Best Practises C# ChartJS Databricks / Spark Data Governance data profiling Data Quality / dqops Data Quality Assessment DAX dqops ETL Feature Engineering ggplot2 M Machine Learning / Data Mining MDX Microsoft Cognitive Services pandas Parquet Power BI Power Query PowerShell Python R scikit-learn Spark Catalyst Spark SQL SQL SSAS/MDX SSDT SSIS SSRS Streaming Tableau U-SQL

Recent Posts

Spark Structured Streaming Deep Dive (8) – Session Window
Spark Structured Streaming Deep Dive (7) – Stream-Stream Join
Spark Structured Streaming Deep Dive (6) – Stateful Operations
Spark Structured Streaming Deep Dive (5) – IncrementalExecution
Spark Structured Streaming Deep Dive (4) – Azure Event Hub Integration
Spark Structured Streaming Deep Dive (3) – Sink
Spark Structured Streaming Deep Dive (2) – Source
Spark Structured Streaming Deep Dive (1) – Execution Flow
Spark SQL Query Engine Deep Dive (20) – Adaptive Query Execution (Part 2)
Spark SQL Query Engine Deep Dive (19) – Adaptive Query Execution (Part 1)
Spark SQL Query Engine Deep Dive (18) -Partitioning & Bucketing
Spark SQL Query Engine Deep Dive (17) – Dynamic Partition Pruning
Spark SQL Query Engine Deep Dive (16) – ShuffleExchangeExec & UnsafeShuffleWriter
Spark SQL Query Engine Deep Dive (15) – UnsafeExternalSorter & SortExec
Spark SQL Query Engine Deep Dive (14) – Partitioning & Ordering
Spark SQL Query Engine Deep Dive (13) – Cache Commands Internal
Spark SQL Query Engine Deep Dive (12) – SessionCatalog & RunnableCommand Internal
Spark SQL Query Engine Deep Dive (11) – Join Strategies
Spark SQL Query Engine Deep Dive (10) – HashAggregateExec & ObjectHashAggregateExec
Spark SQL Query Engine Deep Dive (9) – SortAggregateExec
Spark SQL Query Engine Deep Dive (8) – Aggregation Strategy
Spark SQL Query Engine Deep Dive (7) – Spark Planner Overview
Spark SQL Query Engine Deep Dive (6) – Catalyst Optimizer Rules (Part 3)
Spark SQL Query Engine Deep Dive (5) – Catalyst Optimizer Rules (Part 2)
Spark SQL Query Engine Deep Dive (4) – Catalyst Optimizer Rules (Part 1)
Spark SQL Query Engine Deep Dive (3) – Catalyst Analyzer
Spark SQL Query Engine Deep Dive (2) – Catalyst Rule Executor
Spark SQL Query Engine Deep Dive (1) – Catalyst QueryExecution Overview
Parquet for Spark Deep Dive (4) – Vectorised Parquet Reading
Parquet for Spark Deep Dive (3) – Parquet Encoding
Parquet for Spark Deep Dive (2) – Parquet Write Internal
Parquet for Spark Deep Dive (1) – Table Writing Journey Overview
Databricks Lakehouse Breaks Data Warehousing Performance Record – Time to Forget about Data Warehouse?
dqops Data Quality Rules (Part 2) – CFD, Machine Learning
dqops Data Quality Rules (Part 1) – Basic Rules
Data Quality Improvement – Rule-Based Data Quality Assessment
What is Data Management, actually? – DAMA-DMBOK Framework
How Azure Storage Cheats Over the CAP Theorem
dqops – Query Databricks Database Schema through SQL Connector for Python
Setup a Dockerised Spark Development Environment with VS code and Docker
Why I Prefer Hand-Coded Transformations over ADF Mapping Data Flow
Data Quality Improvement – DQ Dimensions = Confusions
Data Quality Improvement – Conditional Functional Dependency (CFD)
Data Quality Improvement – Data Profiling
Data Quality – 80:20 Rule and 1:10:100 Rule
Data Quality Improvement – Set the Scene Up
Our Data Quality is Good, Nothing Breakdown
What Makes Me Become a Data Quality Enthusiast
Create Custom Partitioner for Spark Dataframe
Configuration-Driven Azure Data Factory Pipelines

Twitter
WordPress
LinkedIn
GitHub

Website Powered by WordPress.com.

Follow Following
- Azure Data Ninjago & dqops
- Already have a WordPress.com account? Log in now.