Data Engineer at PhonePe / Bengaluru

Hi, I’m Gokul Arumugam.

I build data
pipelines behind
India’s everyday
payments.

From chai breaks to rent day, UPI is part of everyday life. At PhonePe, I build the data platforms that keep analytics and ML teams up to speed. Five-plus years in, still trying not to get paged at 3 AM.

Explore my work The career so far

NOT AT MY DESK, FOR ONCE

Gokul by a river in the mountains — A different kind of stream.This one doesn’t need Kafka.

Yes, I do
go outside.

GitHub X LinkedIn

The toolbox.

Some assembly required

The stuff I reach for. Depends on the problem, not the hype.

toolbox

(01)

Languages

PythonScalaOpenJDKSQLT-SQL

Processing

Apache SparkPySparkSpark Streaming

Streaming & CDC

Apache KafkaDebeziumRabbitMQ

Orchestration

Apache Airflow

Storage & Query

Apache HiveDeltaMariaDBPostgreSQLRedisIcebergTrinodbt

Platform

Apache HadoopDockerKubernetesGitLabAzure

Cloud

AWSSnowflakeDatabricks

File Formats

JSONCSVApache ParquetApache AvroORCDelta

What I’ve been building at PhonePe.

Selected impact / PhonePe

The job behind
the job title.

A few things I’ve built and fixed: getting data in, keeping streams alive, and figuring out why that Spark job is still running.

Selected professional work. Diagrams illustrate the ideas, not internal architecture or live telemetry.

Freshness / Change data capture

400+ tables. Preferably no surprises.

A CDC platform connecting 150+ MariaDB shards and 400+ tables to the teams that need them.

3 TBof changes, every day

MariaDB150+ shards / 400+ tables
DebeziumMulti-tenant Kubernetes
KafkaChange streams
PySparkProcessing
Hive ACIDAnalytical tables

Illustrative flow Schema evolution supported / Sub-hour freshness / 6 teams

Inside the system

Problem: Six teams needed fresh data from a large, sharded MariaDB estate, including tables whose schemas evolve.
Implementation: Built a multi-tenant Debezium deployment on Kubernetes, feeding Kafka, PySpark, and Hive ACID. Supported schema evolution along the ingestion path.
Outcome: 3 TB/day processed across 150+ shards and 400+ tables, with sub-hour data freshness for six teams.

Scale / Data lifecycle

Turns out, not everything needs hot storage.

A cross-environment pipeline that made archival possible, with 50% less hot storage.

50+ PBacross environments

Replicate across environmentsIceberg to HiveManaged to external

1/2the hot storage

Illustrative archival path / 50% less hot storage

Inside the system

Problem: A large data estate needed a path across environments and table formats to enable archival and reduce hot-storage requirements.
Implementation: Built a pipeline spanning replication, Iceberg-to-Hive conversion, and managed-to-external table conversion.
Outcome: Enabled archival across a 50+ PB estate and reduced hot storage by 50%.

Reliability / Stream recovery

Restart the job. Not my sleep schedule.

Foxtrot StreamJobManager: turning job recovery into a system responsibility.

200+streaming jobs managed

Poll YARN
Detect failure
Restart stream

Illustrative control loop / Kafka-to-Hive streams

Inside the system

Problem: Failed Kafka-to-Hive streams needed to be detected and restarted without repeated manual intervention.
Implementation: Built Foxtrot StreamJobManager to poll YARN for job state and automatically restart failed Kafka-to-Hive streaming jobs.
Outcome: Managed 200+ jobs with near-zero manual intervention for stream recovery.

Efficiency / Critical batch

We were counting the expensive way.

Replacing count() with Kafka offset counting cut a critical batch from 60 to 35 minutes.

650+executor-hours saved / day at peak

Before60 min

After35 min

Illustrative comparison / Approximately 40% lower batch runtime

Inside the system

Problem: A critical batch spent unnecessary runtime counting records with count().
Implementation: Replaced count() with Kafka offset counting to avoid the expensive counting work in the batch pipeline.
Outcome: Runtime fell from 60 to 35 minutes, approximately 40% lower, reclaiming 650+ executor-hours/day at peak load.

Knowledge / On-call RAG

30% fewer inbound queries.

Joined Slack, 500+ Confluence pages, and Grafana through a two-stage Ollama/GPT workflow with Redis retrieval. Less searching for answers during on-call.

Performance / Tail latency

40% lower P99 latency.

Debugged Spark memory bottlenecks, JVM garbage-collection pauses, and Kafka partition imbalances; tuned executors and shuffle partitions for faster real-time analytics.

How I got here.

Nov 2020 to today

Commerce degree.
Slight detour into data.

I started with SQL and dashboards, then got curious about everything upstream. A few years later, here we are. Most recent stop first.

Latest chapter first. Bar lengths show time in each role, through Oct 2026.

Chapter 03 / Current
Dec 2024 - Present
Bengaluru
PhonePeData Engineer
Building the systems behind fresh, dependable data at scale.
- CDC platform moving 3 TB/day from 150+ MariaDB shards and 400+ tables, delivering sub-hour freshness to six teams.
- 50+ PB cross-environment pipeline enabling archival and 50% less hot storage; automated recovery for 200+ Foxtrot streaming jobs.
- Critical batch runtime reduced from 60 to 35 minutes, saving 650+ executor-hours/day at peak load; separately, P99 latency reduced by 40% through Spark and Kafka tuning.
- On-call RAG across Slack, 500+ Confluence pages, and Grafana reduced inbound queries by 30%.
Explore the case studies
Chapter 02
Nov 2022 - Dec 2024
Fairfax, VA (Remote)
Bare InternationalAnalytical Engineer
From analytical models to real-time data products.
- Processed 1M+ events per four-hour window with Kafka, Redis, and MariaDB, supporting sub-second queries.
- Reduced Delta Lake query latency by approximately 45% for four analyst teams.
- Built a Neo4j/FastAPI catalog for 40+ datasets and implemented RBAC and row-level security for GDPR across 30 datasets.
- Automated loads for 15+ Domo reports through SQL Server, saving approximately five hours of manual maintenance per week.
Chapter 01
Nov 2020 - Nov 2022
Bengaluru
TTK ServicesData Analyst
Making reporting reliable, from the query to the dashboard.
- Built KPI reporting with SQL Server and Power BI; reduced query latency by 20%.
- Automated validation to reduce manual checks by 60%.

The starting point

Bachelor of Commerce

Hindusthan College of Arts and Science
Coimbatore

Continued learning

Microsoft Certified: Azure Data Engineer Associate (DP-203)
Microsoft Certified: Azure Data Fundamentals (DP-900)
Google Data Engineering Professional Certificate (Coursera)
Google Data Analytics Professional Certificate (Coursera)

Apparently, one day job wasn’t enough.

All projects

Side projects that made it out of the “I should build that” stage. Code, demos, and the bits I learned the hard way.

Atlas — an AI data analyst that respects who's allowed to see what

Text-to-insight with a fail-closed SQL firewall, per-user scoping, and a tamper-evident audit trail. The raw data never leaves your network.

fail-closed governance · 116 tests green

Real-time Wikipedia Edits Pipeline

A local-first lakehouse that turns Wikimedia's live event stream into queryable, replayable tables.

sub-minute freshness

Payments Reconciliation Engine

A deterministic, SQL-first reconciler for switch-ledger attempts and T+1 bank settlements.

1M pairs · <5s

opensource projects

(02)

github.com/GokulArumugam/atlas →

Atlas — governed AI data analyst

Try it out for yourself — atlas-analyst.onrender.com ↗

An AI agent that sits on top of your database: ask questions in plain English and get real insights out of your data — available to everyone on the team, not just analysts, with a firewall that decides who's allowed to see what.

What you just watched: asking the warehouse questions in plain English, the same HR question being allowed for HR and refused for engineering, phone numbers coming back masked, the per-user data map, and every decision landing in a tamper-evident audit trail — all against a Postgres warehouse running in a Podman container. The raw data never leaves the network it belongs to.

Learn more about it →fail-closed governance · 116 tests green

Some evenings look like this.

GitHub activity

69 contributions in the last year · GokulArumugam + gokulguugu · synced 2026-10-15 04:40 UTC · refreshes every 6 hours

Writing it down before I forget.

On Medium

Oct 2023Medium

off the clock

(05)

On repeat

Uptown Funk

Mark Ronson ft. Bruno Mars

Ayrton Senna, Alain Prost, Nigel Mansell, and Nelson Piquet sitting together at the track

race sundays are sacred.

Stephen Curry, Kevin Durant, and LeBron James in Team USA jerseys

NBA

6 AM tip-offs. suddenly I’m a morning person.

Adventure

auto across blr, no destination.

07 / Start a conversation

Talk data to me.
Or Formula 1 / basketball.

Got something interesting to build, a pipeline acting up, or a questionable F1 take? I’m listening.

gokularumugam64@gmail.com Or find me on LinkedIn

I build data
pipelines behind
India’s everyday
payments.

The toolbox.

toolbox