Hire Me
Services Skills Experience Projects GitHub Reviews Contact
Hire Me
Open to Full-time · Part-time · Freelance

Arslan
Ali.

Senior Data Engineer

50× Performance Optimization  |  Azure · Databricks · PySpark  |  Kaggle Master

4+ Years of Production Experience
50× Query Speedup via SQL→PySpark Migration
Databricks Certified Data Engineer — 95% Score
Kaggle Legacy Master — Rank #1,914 / 9,003 globally
Arslan Ali
50×
Query speedup
via SQL→PySpark
99.9%
Pipeline uptime
in production
10M+
Records per batch
BCBS regulatory
70%
Manual effort saved
via SSIS automation
4+
Years building
cloud-native platforms
Azure Data Factory Apache Spark Snowflake Databricks Apache Airflow Docker Power BI Python Delta Lake Kafka AWS Lambda Tableau Azure Data Factory Apache Spark Snowflake Databricks Apache Airflow Docker Power BI Python Delta Lake Kafka AWS Lambda Tableau
🟢 Currently Available
Open for New
Projects & Clients

Whether you're a startup or enterprise — let's talk data.

Full-Time Employment
On-site or remote · Senior DE roles
Part-Time Engagement
Flexible hours · advisory or pipeline work
Freelance Projects
One-off pipelines · consulting · audits

Need a pipeline built from scratch, an existing system optimised, or expert data consulting? Let's talk.

Send Me a Message
What I Offer

Services for Clients & Companies

01
Cloud Data Pipeline Design
End-to-end ETL/ELT architecture on Azure (ADF, Synapse, Databricks) or AWS (Lambda, S3). Production-ready, governed, and monitored pipelines from raw ingestion to analytics layer.
Azure ADFDatabricksAirflowAWS Lambda
Full-time & Freelance
02
Spark & Big Data Optimisation
Migrate slow SQL workloads to PySpark, tune partitioning, broadcast joins, and caching. Proven 50× speedups and 35% cost reduction. Cloud-agnostic across Azure and AWS.
PySparkDelta LakePolarDBKafka
Consulting & Contract
03
Data Warehouse & Modelling
Design Snowflake, SQL Server or PostgreSQL warehouses with star/snowflake schemas and semantic layers for Power BI and Tableau. Reduce refresh latency from hours to minutes.
SnowflakeSQL ServerT-SQLModelling
Part-time & Freelance
04
BI Dashboards & Reports
Power BI and Tableau dashboards on optimised semantic models serving 50+ business users. Dashboards that load in seconds, with automated refresh and alerting.
Power BITableauDAXReports
Freelance & Part-time
05
Data Governance & Quality
Lineage tracking, access control, encryption, SLA alerts, and automated quality checks. Build systems your team trusts with 99.9% uptime and full audit trails.
GovernanceLineageQualityAudit
Advisory & Consulting
06
ML & Computer Vision Pipelines
Production ML pipelines — XGBoost, ARIMA, Prophet forecasting, and YOLOv8 computer vision. From training to inference at scale in containerised Docker environments.
PythonYOLOv8XGBoostDocker
Project-based & Freelance
Technical Stack

Core Skills & Expertise

Cloud & Infrastructure
Azure Data FactoryAzure SynapseData Lake Gen2DatabricksAWS LambdaAWS S3DockerKubernetesCI/CD
Big Data & Processing
Apache SparkPySparkDelta LakeApache AirflowKafkaHadoop HDFSSSISApache Sqoop
Databases
SQL ServerSnowflakePostgreSQLPolarDBOracleT-SQL
Languages
PythonSQL (T-SQL)ScalaBash Scripting
BI & Reporting
Power BITableauSemantic ModelsDAX
ML / AI & Data Science
XGBoostARIMAProphetYOLOv8OpenCVPandasNumPy
Career History

Work Experience

Tkxel · Full-time · On-site
Senior Data Engineer
Nov 2025 — Present
Azure Data FactoryData Lake Gen2Synapse AnalyticsPythonPower BI
  • Building end-to-end ETL pipelines for Vision Invest (US) — ~80% reduction in manual data handling, with full lineage tracking.
  • Designing ADF data flows integrating 5+ heterogeneous financial sources with real-time quality validation and SLA-compliant scheduling.
  • Architecting Azure Synapse Analytics for near real-time Power BI dashboards for US-based investment stakeholders.
  • Enforcing data governance: lineage tracking, access control, encryption, and audit logging across the full Azure stack.
Techlogix · Promoted · Hybrid
Senior Data Engineer
Nov 2024 — Nov 2025
PySparkPolarDBSnowflakeAWS LambdaDatabricksAirflowDocker
  • Spearheaded SQL-to-PySpark migration on PolarDB: 50× query execution speedup and ~35% reduction in compute costs.
  • Advanced PySpark broadcast joins, partitioning & caching — ~40% memory overhead reduction, cloud-agnostic across Azure and AWS.
  • Architected Snowflake-powered serverless pipeline using AWS Lambda for real-time validation and MoEngage delivery.
  • Promoted to Senior DE — mentored 3 junior engineers on PySpark, code review, and pipeline design standards.
  • Delivered Power BI & Tableau reports on optimised semantic models — reducing refresh latency from hours to under 10 minutes.
Techlogix · Full-time · On-site
Data Engineer
May 2022 — Nov 2024
SQL ServerSSIST-SQLPythonADFDatabricksDockerHadoop HDFS
  • Built SSIS packages automating JSON, PDF & flat-file ingestion — 70% reduction in manual data loading effort.
  • Developed T-SQL stored procedures for BCBS regulatory reporting handling 10M+ records per batch.
  • Designed Airflow + Docker fault-tolerant pipelines with 99.9% production uptime, retry logic, and automated quality checks.
  • Built ADF + Databricks pipeline for BCBS; Azure Synapse + Power BI dashboards serving 50+ business users in near real-time.
  • Engineered YOLOv8 + OpenCV + Pytesseract computer vision — 85% accuracy at <280ms inference per frame, trained on 500k augmented images.
Broadstone · On-site
Python Developer
Aug 2021 — Dec 2021
DjangoREST APIPython
  • Developed production-grade backend services and REST APIs using Django for web applications.
Featured Work

Key Projects

P-001
End-to-End Financial Data Pipeline

Airflow-orchestrated Spark pipeline for large-scale financial file processing. Hadoop HDFS storage, Sqoop RDBMS transfer, and live REST API for front-end BI dashboards. Full Docker containerisation.

DockerApache SparkAirflowHDFSSqoopREST API
P-002
SQL → PySpark Migration · 50× Speedup

Complete migration of legacy SQL workloads to PySpark on PolarDB. Broadcast joins, custom partitioning, and caching strategies delivered 50× query speedup and 35% compute cost reduction.

PySparkPolarDBDatabricksAirflow
P-003
BCBS Azure Telemetry Platform

ADF + Databricks telemetry pipeline for Blue Cross Blue Shield processing 10M+ records/batch. Azure Synapse Analytics with near-real-time Power BI dashboards for 50+ business users.

ADFDatabricksSynapsePower BI
P-004
Serverless Real-Time Validation Pipeline

Snowflake-powered serverless task pipeline using AWS Lambda for event-driven real-time validation with automated MoEngage downstream delivery. Cloud-agnostic across Azure and AWS.

SnowflakeAWS LambdaMoEngage
P-005
YOLOv8 Computer Vision System

Image recognition & OCR system trained on 500k augmented images. YOLOv8 + OpenCV + Pytesseract achieving 85% detection accuracy at under 280ms inference per frame in production.

YOLOv8OpenCVPytesseractPython
P-006
Vision Invest Azure Data Platform

End-to-end ETL for US-based investment firm — ADF flows integrating 5+ financial sources with governance-compliant pipeline stack, Synapse Analytics, and live Power BI investment analytics.

ADFSynapseData Lake Gen2Power BI
Open Source

GitHub Repositories

25
Repositories
30
Stars Earned
32
Followers
🏆
YOLO Achievement
Python
1
azure-kafka-pipeline

Real-time streaming pipeline integrating Azure cloud services with Apache Kafka for event-driven data ingestion and processing at scale.

View Repo
Python
1
data-pipeline-with-docker

Containerised end-to-end pipeline — Spark ingestion, Airflow orchestration, Hadoop HDFS storage, Sqoop transfer, and REST API integration.

View Repo
Jupyter Notebook
1
Data-Engineering-Projects

Collection of practical data engineering implementations covering PySpark, Delta Lake, Apache Airflow, and cloud integrations.

View Repo
HTML
1
Personal-Portfolio

Portfolio website — HTML, CSS, JavaScript showcasing data engineering projects, skills and experience.

View Repo
Jupyter Notebook
5
Subjective-Exam-Checking

FYP: NLP-based auto-grading using semantic similarity for question classification and subjective answer evaluation. MIT License.

View Repo
Python
1
Heart-Disease-Prediction

Neural network for heart disease prediction — data cleaning, feature engineering, training, evaluation and deployment pipeline.

View Repo
View All 25 Repositories on GitHub →
Data Science

Kaggle Profile & Achievements

Arslan Ali
Arslan Ali
@arslanali4343
Datasets
Rank 1,914 / 9,003
Legacy Master
Notebooks
Rank 7,839 / 60,750
Expert
Discussions
Legacy Expert
Expert
View Kaggle Profile
2
🥇 GOLD
Dataset Medals
4
🥈 SILVER
Dataset Medals
7
🏅 TOTAL
Notebook Medals
166
👥 COMMUNITY
Kaggle Followers
Face Recognition Student Attendance System (FaceNet + Kairos)
Python · Tkinter GUI · MySQL Database
COVID-19 Prediction using Machine Learning
Python · Scikit-learn · Data Analysis
Dragon Real Estates — House Price Prediction (Advanced Level)
Python · XGBoost · Feature Engineering
Dask vs Pandas on Big Datasets — Performance Benchmarks
Jupyter Notebook · Dask · Benchmarking
Mastering Apache Spark — End-to-End Implementation
PySpark · Big Data · AI Engineering
Web Scraping Basics with Python (Beautiful Soup)
Python · BeautifulSoup · Data Collection
Recognition

Awards & Certifications

Achiever of the Quarter

Awarded Q1 & Q2 2023 at Techlogix for delivering the 50× pipeline optimisation and measurable infrastructure cost reduction at scale in production.

Databricks Certified (95%)

Data Engineer Associate — scored 95% on first attempt. Expertise in Lakehouse architecture, Delta Lake, Unity Catalog, and production pipelines. Valid through Jun 2026.

Kaggle Legacy Master

Datasets Legacy Master (Rank 1,914/9,003) · Notebooks Expert (7,839/60,750) · 2 Gold + 4 Silver + 7 total medals earned globally.

PEEF Merit Scholarship

Full merit-based scholarship from Punjab Educational Endowment Fund for all 4 years at UET Lahore, 2018–2022.

PM Laptop Scheme

Awarded by the Prime Minister Laptop Scheme for academic distinction, 2015–2018.

LinkedIn Top Data Engineering Voice

Top 5% of global data engineering contributors on LinkedIn. 20K+ followers. Regular posts on Spark, Databricks, Azure, and data architecture.

Databricks Certified Data Engineer Associate
Databricks
Jun 2024 – Jun 2026 · Credential ID: 106026604
✓ Active
Applied Text Mining in Python
University of Michigan
Issued Jul 2020 · Credential ID: RL9P3ZDQGZJB
Verified
BS Computer Engineering
University of Engineering & Technology (UET), Lahore
Sep 2018 – Jun 2022 · PEEF Full Merit Scholarship
Completed
Social Proof

What Colleagues Say

"

Arslan is one of the most reliable and skilled Data Engineers I've collaborated with. He has strong command of data pipelines, ETL/ELT processes, and cloud technologies, and consistently delivers high-quality solutions even under tight deadlines. What sets him apart is his ability to troubleshoot complex issues calmly and provide clear, actionable insights. Any organisation would be lucky to have him — highly recommended!

MK
Muhammad Khawar
Senior Data Engineer · Databricks & Snowflake Certified
LinkedIn
"

Arslan is an exceptionally intelligent individual. He consistently shares valuable insights and offers clear, practical advice. His deep understanding of various subjects and ability to communicate complex concepts in an accessible manner make him an invaluable resource. His passion and dedication are truly inspiring.

NP
Neysha Pagán
Data & Cloud Architect · MS Data Science · Microsoft Certified
LinkedIn
"

A team player, always willing to support others and share his knowledge. He consistently delivers measurable performance and cost improvements. Any organisation would be lucky to have Arslan on their team.

6 LinkedIn Recommendations
Verified recommendations from colleagues and collaborators
LinkedIn
🟢 Open to Work · Worldwide

Let's Build Something Great Together

Whether you need a pipeline built, a system optimised, a dashboard delivered, or expert data consulting — I'm available for full-time roles, part-time engagements, and freelance projects worldwide.

Quick Opportunities
Click to send a pre-addressed message

📍 Based in Lahore, Pakistan · Available for remote work worldwide
Typically responds within 24 hours