Fighting Payment Frauds as a Fintech Startup 🕵️
A detailed tale of metric-ising fraud, segmenting fraudsters, organising tech + ops to work in tandem, building an anti-fraud engine and leveraging LLMs
Welcome to another issue of PMF: Perspectives, Markets & Frameworks 🌈, where we endeavour to shed light on the intricacies of product management and delve into the unique aspects of India's market landscape.
If you’re not a subscriber yet, subscribe now (like 120+ PMs, founders and builders already have) and get notified whenever we publish a behind-the-scenes story of building great products.
This article is an expansion of a small section of another article. Here’s a TLDR of the linked article:
In October 2021, FloBiz launched Smart Collect – a powerful reconciliation tool for Indian SMEs within its accounting SaaS product myBillBook.
For those of you aware of Razorpay’s Smart Collect, this was a similar feature, but with a GTM tailored for non-tech-savvy Indian SMEs and deeply integrated within myBillBook.
For those who aren’t, myBillBook’s Smart Collect provided merchants with automatic payment reconciliation by serving as a tech-enabled middleman between merchants and their customers.
The linked article talks about the research behind launching this feature, the product, design and tech choices made, and the success we saw. One part of being a payments middleman was combating fraud in a time-sensitive and UX-friendly manner.
That’s what this article focuses on.
Disclaimer: This is a story from 2021-22, back when fintech middlemen (like us) could act as payment aggregators (PA) themselves (by maintaining a float with their actual payment aggregator). As a consequence, fintechs had to bear the cost of fraud that happened on their platform. This doesn’t happen today. All merchants are onboarded by the licensed PA itself (with the fintech strictly acting as a middleman) and PAs owning the full responsibility of KYC and fighting fraud.
Table of Contents
How did fraudsters misuse Smart Collect?
If you’ve read our article on Designing KYC, you’ll remember that we started with explaining how some fraudsters conned unsuspecting victims by misusing Smart Collect. That’s what this section talks about. If you remember it well, you can jump to the next section.
If not, read on.
Imagine a fraudster collecting money from unsuspecting victims.
He needs to share a QR code with his victims to which they will send money. But in case of a Cyber Crime investigation, he won’t want his bank account to be known to and blocked by authorities.
A cunning fraudster could use myBillBook’s QR code to collect money from victims. By using it as an intermediary, the fraudster could significantly slow down the investigation, as the police would first have to be convinced of myBillBook’s innocence.
The fraudster could use an unsuspecting friend’s account to collect the victim’s money from myBillBook. The friend could be under the impression that he’s helping out someone with bank account-related issues, but in the event of a Cyber Crime investigation, the friend would get implicated.
Why was stopping fraud important?
#1: Our moral stance
As a participant and beneficiary of India’s burgeoning digital payment infrastructure, we were responsible for upholding its safety and sanctity. We were morally obliged to ensure that we allowed no misuse of Smart Collect for fraudulent purposes.
#2: Impact on bottom-line
💡 If a customer accused a merchant of fraud, the Cyber Crime Division would find myBillBook to be the recipient of the contested transaction. After confirming that the transaction was indeed fraudulent 🚨, the beneficiary bank (under orders from the police) would instruct the beneficiary account holder to pay back the contested sum. But myBillBook had already settled the amount to the corresponding merchant (since we made real-time settlements). Recovering this money from the merchant was an unfruitful ordeal (we tried 😞). This meant we had to pay out of pocket. Banks did this by marking lien💸 on our account. We had to maintain a separate pool of money for lien with our payment provider Hypto, which could be used in such situations.
#3: Impact on brand
When victims sent money to fraudsters, they saw myBillBook’s name in the recepient UPI address. In their eyes, myBillBook was an accessory to fraud – a perception we wanted to stifle before it took root.
The goal was clear – minimise the volume of fraud on SC.
How did we measure fraud?
We broke down fraud volume into its components:
We then broke down # of victims for each fraudster:
These equations laid out a set of objectives for us:
# of fraudsters onboarded (variable #1) was not an easy metric to track. Fraud recognition (borrowing from revenue-recognition) was largely a post-facto action – you didn’t know that a transaction was fraudulent, till after it had happened, when someone reported it or you spotted the pattern of fraud. Simply put, we couldn’t look at the previous day’s TPV and say with confidence that x% of TPV was fraudulent. Complaints took time to reach us (passing through Cyber Crime, Yes Bank and Hypto) – empirically 2-3 weeks on average. We couldn’t wait for this long to track fraud volume accurately.
💡 A rising fraud volume would warrant an immediate deceleration of our TPV growth plans. To this end, we split fraud volume into 2 parts:
Confirmed fraud: transactions that we were certain were fraudulent
Predicted fraud: transactions that we thought (but weren’t certain) were fraudulent
Confirmed fraud TPV + x% of predicted fraud TPV helped us get a realistic sense of what our current fraud volume was. This helped give confident lien amount estimates to our Finance team, as well as enabled our Operations team in manual review of blocked merchants, which we’ve discussed in detail in the section titled Empowering Anti-Fraud Operations.
💡 Reducing the time to spot fraudsters (objective #3) had to be a predictive process. We had to act the moment we sensed that something was amiss. But this enthusiasm was a double-edged sword:
If we were too enthusiastic, we would block genuine merchants and restrict TPV growth
If we weren’t enthusiastic enough, we would let fraudsters defraud too many victims
This tradeoff can be encapsulated well in the definitions of precision and recall. For Smart Collect, these terms were defined as:
How did we know if a fraudster was correctly predicted? We assumed that any blocked merchant who didn’t get unblocked was correctly predicted to be a fraudster.
How did we know who was an actual fraudster? We assumed that all blocked merchants who didn’t get unblocked + all merchants who were reported to us by the bank or by the police were actual fraudsters.
The lens of precision and recall added more dimensionality to our objectives:
Relying on the strength of an Operations team armed and ready to assist us, we prioritised recall in our anti-fraud rule engine (blocking as many fraudsters as possible) over precision (blocking only fraudsters).
Like what you’ve read so far? Subscribe (for free!), and we’ll stay in touch
How did we stop (or minimise) fraud?
Q. If you suspect that some merchants may be committing fraud on SC, how do you stop them from causing more harm?
A. You don’t let them collect payments on your product
Q. But what if your suspicion is wrong? What if you block innocent merchants and deny them the value of SC?
A. You let them collect payments, but you don’t settle it to them. You give suspects a chance to contest your judgement.
This approach empowered us to prioritise suspected merchants with large blocked amounts, thus ensuring that genuine whales (merchants with high transaction volumes) were unblocked promptly and that high volume frauds were identified swiftly.
Q. But how do you get better at suspecting that a merchant may be committing fraud?
A. You hypothesise, test and deploy
💡 We followed an ops-first, productise-later approach, when learning to suspect better. This is our approach broken into steps:
Hypothesize how fraudsters use SC based on our conversations with victims and secondary research
Formulate heuristics (by writing SQL queries and making product analytics cohorts) to identify merchants who exhibit such behaviour
Manually evaluate how many shortlisted merchants are fraudulent (often involving phone calls to merchants)
If the precision of the heuristic is high enough, implement and deploy these rules on our anti-fraud engine to block settlements for such merchants
As we got down to business, we quickly realised that suspicion isn’t black or white, it’s grey. So we assigned 4 levels of suspicion to merchants:
Level -1: We know (for certain) that the merchant isn’t fraudulent
We whitelisted such merchants from all fraud rules. Some examples included:
Registered on myBillBook before we launched SC
Collected > 4 payments via bank transfer
Previously blocked, but later verified as genuine and unblocked
Level 0: We don’t know anything about the merchant’s intentions
For merchants early in their myBillBook journeys, we didn’t have enough datapoints to gauge their intentions. While we planned to reduce time to spot fraudsters, for such merchants, we couldn’t do much but wait.
Level 1: We think (but aren’t certain) that the merchant may be fraudulent
We blocked such merchants the moment they collected any suspicious-looking payments, by implementing rules in our anti-fraud engine, but manually reviewed their details to be certain. We considered these transactions to be predicted fraud, mentioned in the previous section. Some examples included:
Collected > 4 UPI payments under 1 minute
Enabled SC without exploring core SaaS offering
This involved loads of real-time vigilance from Operations, who reviewed a bunch of datapoints and collected additional documents from merchants to verify their authenticity. We’ve covered this in detail in the section titled Empowering Anti-Fraud Operations.
Level 2: We know (for certain) that the merchant is fraudulent
We blocked such merchants immediately and considered their transactions to be confirmed fraud. Some examples included:
Company name contained suspicious keywords (eg. Receive Money)
Verified a bank account or KYC, which had been previously verified by a confirmed fraudster
As you can imagine, the charter of spotting and stopping fraud kept us up on many nights. Some moments, however, truly overwhelmed us. Taking inspiration from the recently abated pandemic, we called such moments waves.
1st Wave of Fraud
We launched Smart Collect on 29th October 2021 to 2K merchants and gradually increased rollout. Only 3 weeks later, we ran into trouble.
Modus Operandi
Scores of victims reached out to us at once and we realised that we were hit by a wave of OLX frauds. For those of you lucky enough to not have experienced this deception firsthand, here’s what happens:
Imagine you list a used sofa for sale on OLX for ₹4,000. You get an interested buyer, who offers to pay immediately, even before inspecting the sofa. While on call, he sends you a QR code to collect money from him and insists that you collect it right away. You scan the QR code and enter your UPI pin. But instead of collecting, money has been deducted from you.
In case, you resist the fraudster’s urgency and reason with him that you don’t need to enter your UPI pin to collect money, he asks you to send a test payment of ₹1. You do and find that you receive 2 ₹1 payments right after. The fraudster explains that this is a new mode of payment and asks you to send ₹4,000 for the sofa, after which he would send ₹8,000 – paying you ₹4,000 as a result. After you send the money, he doesn’t send any back.
Some steadfast fraudsters even go as far, as to explain that there is some issue with their PSP app (PhonePe, Google Pay), and that you should send them ₹8,000 again and promise to return all your money.
Victims who were desperate to retrieve their money and fell prey to sunk cost fallacy, gave in to the demand and made bigger losses. Dhruv Rathi even made a video about this type of scam.
To make the convincing easy, fraudsters sent their victims an SC QR code that prominently said ‘Receive Money’ – to convince the victim that they would be able to receive payments by scanning this QR code. They achieved this by changing their ‘Business Name’ in myBillBook to ‘Receive Money’.
Some confidence men (which is where the word con man comes from) told victims that they’re from the army to win their trust and respect. They too changed their business name on myBillBook.
Counter-measures
#1: Showed bank account holder’s name on QR code
Given that a misleading business name was important for the victim to believe the fraudster, we stopped showing business name on QR codes, and instead showed the bank account holder’s name. You may recall from our article on Designing KYC that we had fetched this name at the time of verifying the merchant’s bank account. This ensured that victims couldn’t be fooled as easily. While this made sense from an anti-fraud lens, we had to concede 2 benefits:
Merchants could no longer collect payments before verifying their bank account (which had contributed to 8% of bank account verifications as discussed in our KYC article)
Merchants who used their personal savings account (or their spouse’s) to collect payments had to explain the new beneficiary name to bemused customers
#2: Mandated early KYC
We reduced the limit for KYC-less cumulative payment volume from 50K to 10K. This way, we could curb the total amount of fraud from any unverified merchant to 10K.
2nd Wave of Fraud
This wave escalated significantly faster than the last, rising to a TPV of ₹1 crore in just 2 days.
Modus Operandi
Victims were making payments to phishing pages, under the guise of collecting rewards from PhonePe.
Links to these phishing pages were sent to a massive number of victims via SMS, which led to a high velocity of such payments.
Counter-measures
#1: Blocked settlements to neobanks
We identified that 75% of fraudulent TPV was being settled to neobank accounts (we identified them using their IFSC code). Back in 2021, neobanks had lax KYC measures, thanks to a heavy-handed push for seamless and quick onboarding. Some neobanks allowed as much as 1 year before a customer needed to complete his full KYC. Such relaxations made neobanks a hot target for fraudsters, a grave matter that had been highlighted in mature FinTechs in the west, with N26, Monzo and Revolut finding themselves in regulatory hot water. We promptly blocked settlement to and verification of neobank accounts.
If you’re fearful of neobank account holders too, you can use our list of blacklisted neobank IFSC codes:
PYTM0123456
FDRL0007777
FDRL0000001
FDRL0005555
AIRP0000001
#2: Reduced SC rollout
Given the rapid increase in fraudulent transactions, we temporarily reduced the rollout of Smart Collect to 12.5% of post-launch users
#3: Mandated (even) earlier KYC
We reduced the limit of KYC-less payment volume further to 5K
Have you faced such waves too? How did you handle them?
Empowering Anti-Fraud Operations
We launched SC with only a scant understanding of how fast and intricate frauds could be. This meant that often we had to roll up our sleeves and do things that didn’t scale, to accomplish anti-fraud objectives.
Set up processes and tech to block missed fraudsters
This gave Operations 2 big jobs:
#1: Spot missed fraudsters and formulate new hypotheses
Finding a fraudster in a sea of 3K transacting merchants/day is a daunting task. So we prioritised our search. We set our sights on the most damaging fraudsters – those driving most transactions or volume.
Once we had found a suspect, we would call him and probe him to confirm our suspicion. If we were right, we would analyse his product behaviour on Amplitude (product analytics tool), his transaction history and invoices on an internal tool and call his victims to understand his modus operandi and formulate our hypothesis on how to identify such fraudsters early. Needless to say, this felt like proper investigative work and we hired folks with previous experience in risk and compliance to execute this well.
#2: Manually validate untested hypotheses, by reviewing merchant behaviour and making phone calls
This was doing investigative work from #1, but for a sample size of 40-50 merchants for each hypothesis, till we could hone the hypothesis enough to improve precision and implement as a rule.
We offered 3 blocking functionalities for Operations:
Block SC Settlements – for fraudsters
Block SC Rewards – for cashback exploiters (we discuss this more in section titled Applying anti-fraud learnings to cashback)
Block myBillBook account – for non-SC scamsters using myBillBook invoices
For blocked merchants, we showed the blocking type (suspicion level) on the internal tool for Operations:
Blocked (Predicted) or Level 1 (maybe fraudulent) – which meant that Operations needed to engage with the merchant and review his case again
Blocked (Confirmed) or Level 2 (fraudulent) – which meant that Operations had to extract more information about the fraudster’s modus operandi
Each blocking functionality included options to select reasons for blocking, along with the ability to change the reason (in case of an error) and unblock merchants (if they were found to be innocent later).
Reduce time to verify and unblock incorrectly blocked merchants
For Level 1 suspects (maybe fraudulent), we sent realtime Slack updates for each case that Operations manually reviewed.
💡 Operations followed a set of checks on their end to evaluate if the merchant looked genuine, some of which involved:
Reviewing merchant’s invoices and inventory
Scanning merchant’s event log on Amplitude
Calling the merchant to understand their business
Calling the payer to confirm intent of payment
You can find a doc to the full list of checks in the section titled Giveaways.
We informed merchants on the product that their payments had been blocked, so they could reach out to us and help us resolve their case.
Metrics
With Engineering, Product, Design and Operations working hand-in-hand, to exhaustively cover all aspects of the problem, through multiple instances of learning and executing on the go, we were able to achieve these metrics:
Re-imagining our work with ChatGPT
A bunch of non-scalable Operations measures that we took could have scaled if we had the powers of an LLM back then. Here are some of them:
#1: Calling merchants to determine if they owned a legitimate business
We made phone calls to multiple kinds of merchants with this goal in mind:
Level 1 (maybe fraudulent) merchants who had been blocked by our anti-fraud engine (mentioned here)
Merchants who had transacted high volumes or frequency of payments (mentioned here)
Merchants who fit the mould for a new untested heuristic (mentioned here)
Understanding a merchant’s business in detail, to make sure that his alibi is rational and consistent throughout is a process that requires tact, along with being highly non-deterministic. This is how these conversations would go:
What business do you run?
Where is your shop located?
How many shops do you have?
How long have you run this business?
Do you run this business alone?
What do you use myBillBook for?
Somewhere along this line of questioning, fraudsters would falter. For example, a fraudster might say that he has 3 shops, but his business is only a year old and he runs it alone. Follow up questions on how he was able to achieve this would cause the cracks to start showing. With our Operations team’s easy going and disarming tone, fraudsters would get comfortable and drop their guard.
With GPT-4o’s ability to speak in regional languages with human-like modulations and banter, we could outsource some of the preliminary probing work to ChatGPT, and then let Operations look at the conversation’s summary, ChatGPT’s assessment of the potential fraudster’s mood and tone and an annotated and transcribed audio recording, to make their judgement.
#2: Reviewing myBillBook accounting data
We did this whenever Level 1 merchants (maybe fraudulent) would get blocked by the anti-fraud engine. We found that only 60% of calls made to merchants were answered — probably due to merchants being engaged with customers in their shop or chores around the shop. We couldn’t let this stop us from unblocking genuine merchants. So we looked at their myBillBook accounting data to see if we could make any inferences. A merchant with relevant invoices and inventory (along with metadata like sales price) as per their business and industry would be less likely to be fraudulent and should be followed up with on call.
An LLM-powered agent could swiftly skim through invoices and inventory and save hours of mundane app-switching, clicking and Google searches.
#3: Calling customers of Level 1 merchants to confirm intent of payment
If Level 1 merchants are unreachable, Operations would call their payers to understand if they made the payment in good faith or if they were hoodwinked.
Similar (or in fact more trivial compared) to calling Level 1 merchants, GPT-4o could call their payers and help Operations act fast.
Back then, we did consider applying traditional ML to identify fraudsters, but couldn’t make much progress due to limited number of datapoints, and a class imbalance problem.
Interactions with Cyber Crime and PAs
Before we get into our interactions with the authorities, let’s look at how a victim of online payment scams find recourse.
💡 After a victim is cheated by a fraudster via UPI, he or she has 4 modes of recourse (in order of popularity):
In our case, the victim’s bank, PSP and NPCI all spoke to Yes Bank (which housed our PA Hypto’s nodal account containing our VAs), which asked myBillBook for proof of the transaction (failing which we were asked to block their VA and reverse the transaction), while Cyber Crime directly asked for KYC of the fraudster, along with VA blocking and transaction reversal.
Such a transaction reversal is referred to as chargeback in banking parlance. While most chargebacks were for fraud, some were for failure to deliver products — a merchant collected money from a customer, but didn’t deliver products. In such cases, we worked with merchants to provide proof of delivery (eg. delivery receipt), but found that most cases were fraudulent with no legitimate transaction. In the few cases where the transaction was genuine, such proofs were hard to provide (since businesses in India only have an invoice to show as proof of transaction, but it doesn’t count as proof of delivery).
Every fraudulent or disputed transaction committed on myBillBook, was a lien marked on our account with Hypto. We raised chargebacks ourselves with the banks where amounts were settled (as a way of forcing the fraudster to pay up), but didn’t find much luck.
Interesting episodes from monitoring fraud
#1: Petty penny thieves
We knew that a couple of test ₹1 transactions at the beginning of SC usage were to be expected. But we found merchants making as many as 700-800 ₹1 transactions every day. On further inspection, we uncovered that Bharat Khata (another Bharat-focused fintech) used penny drop for verification, sending ₹1 to the settlement bank account. Fraudsters verified 600-700 VAs on Bharat Khata to earn ₹600-700 every day. We asked the team at Bharat Khata to disallow verification of our VAs.
#2: 1 PAN, 42 merchants
We spotted that the same bank accounts, KYCs and device IDs were being used by different merchants and started reviewing these cases with grave concern. To our shock, we found that 1 PAN was linked to 42 merchants at the same time. This merchant was undoubtedly fraudulent. When we heard that RBI found a case of 1 PAN being linked to a thousand users on Paytm, we were relieved that we had spotted our crook early.
#3: Pasting your QR code on someone else’s
As we delved into the murky world of fraud and their conspirators, we worked to constantly learn about the history of online frauds and the possible ways Smart Collect could be vulnerable. We found out about one particularly heartbreaking type of fraud from the early days of QR codes.
Imagine you run a small shop and stick a QR code on your shop.
While things work smoothly at first, one day you realise that you aren’t getting any payment notifications. Your bank app doesn’t show any credited transactions either. But the rush of customers distracts you and you ignore this concern. Later, you realise that someone pasted an identical looking but different QR code on top of your’s, collecting all your hard earned money.
Applying anti-fraud learnings to cashback
Similar to the dawn of PSPs like Google Pay, when we sent our friends payments back and forth to earn cashbacks, we found that some merchants were doing the same on myBillBook. We built processes and rules (similar to our anti-fraud rule engine) in our anti-exploitation engine to spot and not reward such transactions. Some of the rules were:
Collecting payments from a UPI ID, with same phone number or similar name as the merchant
Collecting payments from a UPI ID, which had previously paid cashback exploiters
You’ll find a complete list of such rules in the Giveaways section. Some cashback exploiters (a bunch of 20 year olds) were so persistent in their attempts that our Operations team had ended up knowing them dearly and befriending them.
We’ll soon be publishing a separate article on how we designed cashback, diving deep into understanding merchant behaviour, incentive design, variable rewards and measuring ROI. Subscribe to make sure you don’t miss it!
How to not get scammed
We’ve covered a fair bit of fraud-fighting from a builder’s lens. From a customer’s len’s, here are our 2 rules to stay safe of UPI fraudsters:
Remember that PIN is never needing for receiving money – same as signing on a cheque is not needed when receiving money
Take your time – don’t let strangers pressurise you into acting immediately on money matters
Giveaways
A list of all:
If you think this’ll be useful to a fellow PM/builder, share it with them!
Thanks for sticking till the end!
Subscribe now and don't miss out on other parts of this series! Feel free to continue exploring the intricacies of:
From Zero to Hero: A Payment Product's 100x Journey (part 1 of this article)
Designing KYC: a bachata feat. Growth and Compliance💃 (part 2 of this article)
How to design a Rewards Program (part 4 of this article, yet to be published)