Measuring WhatsApp Pay’s Speed Against Google Pay, PhonePe and Paytm
Case Study - QA Automation & Performance Benchmarking
When WhatsApp entered India's Unified Payments Interface (UPI) market against established players like Google Pay, PhonePe, and Paytm, it needed hard data on how its payment experience actually compared. Auriga IT built an automated benchmarking system that measured end-user payment latency across every major UPI app, turning thousands of test runs into a single decision-ready dashboard.
WhatsApp, installed on more than 2 billion devices worldwide, was expanding into India's fast-growing UPI payments market. As a late entrant competing against Google Pay, PhonePe, Paytm, and Amazon Pay, it needed objective evidence of how its payment speed compared - measured the way a real user would experience it, not in a lab.
Payment latency is one of the clearest differentiators in a UPI app. A transaction that completes a second or two faster, or a QR scanner that locks on more reliably, directly shapes whether a user sticks with an app or switches. WhatsApp Pay needed to know precisely where it stood on each step of the payment journey, against each competitor, under the messy real-world conditions Indian users actually face.
02 - The Challenge
Measuring Real-World Payment Speed at Scale
Benchmarking payment latency credibly meant capturing it the way users experience it, across dozens of variables, repeated enough times to be statistically meaningful, then making the results legible to a product team. Doing this manually was not feasible.
01
Many User Journeys
UPI payments happen through several flows - scan and pay, pick a QR code from chat, pay to a phone number. Each had to be measured separately and consistently.
02
Dozens of Variables
Telecom and network type, first versus repeat transaction, scanning distance and angle, and device type all affect latency. Each combination had to be controlled and captured.
03
Manual Testing Wouldn't Scale
Capturing page-level load times by hand across five apps and dozens of conditions, repeated for statistical reliability, would take impossibly long and introduce human error.
04
Data Had to Be Decision-Ready
Raw timing logs are useless to a product team. The output had to surface clear, filterable, step-by-step comparisons that leadership could act on.
03 - What Auriga Built
An Automated Benchmarking Engine
Auriga IT designed an automation framework that simulated real human payment behaviour, ran it in bulk across every test configuration, and fed the results into a purpose-built analytics dashboard. The approach removed human error, made the study repeatable, and turned raw timing data into clear competitive insight.
1
Automated Test Scripts Simulating Real Users
Auriga's team wrote Java automation scripts that mimicked how a person actually moves through a payment, capturing load time at each page from the first screen to the payment completion status. The same scripts ran identically across WhatsApp Pay, Google Pay, PhonePe, Paytm, and Amazon Pay, so every app was measured on exactly the same basis.
1Identify the flows and pages where load latency needed to be captured.
2Write Java scripts that locate on-screen elements and record the load time for each step.
3Execute the scripts in bulk across many test configurations and conditions.
4Generate detailed timing reports covering the entire payment journey.
2
A Test Matrix Built Around Real Conditions
The team brainstormed the critical UPI user journeys - scan and pay, pick a QR code from chat, pay to a number - and the full set of variables that influence latency: telecom type, network type, first versus repeat transaction, scanning distance and angle, and device type. Bulk iterations were run across these combinations so the benchmark reflected the conditions Indian users actually encounter, not just ideal-case lab results.
Critical UPI user journeys and test parameters were mapped before scripting, covering network type, device, transaction type, and scanning conditions.
3
QR Scanner Efficiency Under Varying Conditions
Beyond end-to-end payment time, Auriga measured how efficiently each app's built-in QR scanner performed. The team recorded the time taken to scan a QR code while systematically varying the angle from 0 to 50 degrees and the distance from 20 to 100 centimetres, revealing how robust each scanner was in the imperfect conditions of real use.
QR scanner efficiency was benchmarked by varying scan angle (0 to 50 degrees) and distance (20 to 100 cm) across each payment app.
4
A Decision-Ready Analytics Dashboard
Auriga built a web-based dashboard for the WhatsApp Pay team to explore the benchmark results. It surfaced page-level loading time for each step of each user journey, set side by side against every major UPI app in India. Built-in filters let the team focus the analysis on a specific flow, condition, or competitor. Each transaction was visualised as a sequence of coloured steps, with the width of each block showing the time taken - making slow steps immediately visible.
The dashboard compared transaction time step by step across apps. Each colour is a step, and each block's width shows the time taken in that step.
04 - Technology
How It Was Built
A pragmatic, automation-first stack: Java for the test scripts, MySQL to store the captured timing data, and Apache Superset to turn it into an interactive, filterable dashboard delivered over the web.
Automation
Java
Scripts that locate on-screen elements and capture page-level load latency, simulating real human payment behaviour across every app.
Data Store
MySQL
Structured storage for thousands of timing data points captured across user journeys, conditions, and competitor apps.
Visualisation
Apache Superset
Interactive analytics layer powering the comparison charts, step-level breakdowns, and filters in the benchmark dashboard.
Delivery
Web Dashboard
A browser-based dashboard giving the WhatsApp Pay team self-serve access to the full study with focused, filterable analysis.
05 - Practical Outcomes
From Raw Timings to a Usable Benchmarking View
The project produced a structured benchmarking workflow and dashboard that helped the team review payment latency data by app, flow, test condition, and journey step. The outcomes below focus on delivered capabilities rather than business impact claims.
Comparable Test Runs
Each app was measured through the same scripted flows, making the collected timing data easier to compare across consistent test conditions.
Step-Level Review
The dashboard separated payment journeys into individual steps, helping reviewers inspect where time was being spent within a flow.
QR Test Coverage
QR scanning was tested across defined angle and distance ranges, adding useful context beyond only total transaction duration.
Repeatable Method
The automated setup created a process that could be run again under similar conditions when fresh benchmark data was needed.
Reduced Manual Timing
Automation reduced reliance on hand-timed observations and helped keep measurement capture more consistent across repeated runs.
Filterable Dashboard
The dashboard gave users a way to narrow the benchmark by flow, app, and test condition without manually sorting through raw logs.
06 - Why This Case Matters
For Any Team Competing in a Latency-Sensitive Market
Any team competing in a data-driven, latency-sensitive market - payments, delivery, ride-hailing, or any app where a few hundred milliseconds shapes user choice - faces the same question WhatsApp Pay faced: how does our product actually perform against named competitors, under real-world conditions, not lab conditions. An automated testing framework that simulates real user journeys across every major competitor, captures step-level timing data, and visualises it in a filterable dashboard turns that open question into evidence a product team can act on, without needing a large in-house QA team to run it by hand.
07 - More Case Studies
Other Engineering Work by Auriga IT
Auriga IT delivers QA automation, performance engineering, data analytics, and custom platforms for fintech, manufacturing, and enterprise clients. Here are a few more from our work.
What was the goal of the WhatsApp Pay benchmarking project?
+
As a new entrant in India's UPI market, WhatsApp Pay needed objective data on how its end-user payment latency compared to Google Pay, PhonePe, Paytm, and Amazon Pay. Auriga IT built an automated system to measure and visualise that comparison across real-world conditions.
How is UPI payment latency benchmarked across apps?
+
Auriga wrote Java automation scripts that simulate a real user moving through a payment, capturing the load time of each page from the first screen to the completion status. The same scripts run identically across every app, and results are stored and visualised so each step can be compared on equal footing. See the Solution section above for the full method.
Which UPI apps were included in the benchmark?
+
WhatsApp Pay was measured against the major UPI apps operating in India: Google Pay, PhonePe, Paytm, and Amazon Pay.
What variables affect UPI payment latency?
+
The study controlled for telecom type, network type, first versus repeat transaction, scanning distance and angle, device type, and the specific user journey - scan and pay, pick QR from chat, or pay to a number. QR scanner efficiency was tested across angles from 0 to 50 degrees and distances from 20 to 100 centimetres.
What technology did Auriga use to build the dashboard?
+
The automation scripts were written in Java, timing data was stored in MySQL, and the interactive benchmark dashboard was built on Apache Superset and delivered through the web.
Can Auriga IT run performance benchmarking for other apps?
+
Yes. Auriga IT builds automated QA and performance benchmarking frameworks for fintech, e-commerce, and enterprise apps, along with the data pipelines and dashboards to make the results actionable. Get in touch to discuss your requirements.
Need Hard Data on How Your App Performs?
Auriga IT builds automated benchmarking and performance testing frameworks that measure your product against the competition the way real users experience it - then turn the data into dashboards your team can act on.
