Depending on what classifies the transactions as fraudulent will determine next steps.
If the fraudulent transactions were due to the cardholder having their card stolen and then being used to make fraudulent purchases you should work with your merchant services provider. Within your gateway or shopping cart you have settings that allow you to request AVS, CVV, CID, Address Match, etc. and you can determine the velocity settings and how strict you want to be for each transaction. Some gateways also have country blockers to eliminate transactions that are coming from known organized crime or hackers specific to fraudulent transactions.
You should always report any suspected fraud to your merchant services provider, gateway or shopping cart provider along with the authorities in your local jurisdiction.
Setting your filters and tolerances really low will allow transactions to go through and generate revenue but when you do not validate AVS, CVV, CID or address match, etc. then you risk the chance of fraudulent transactions.
If you collect all the above information and it still ends up being a fraudulent transaction then it should be the responsibility of the card issuer and not the merchant or merchant services provider.
First we should define what is considered fraudulent. Chargebacks can occur for many reasons. They even have specific reason codes which you can define response procedures for. However there is no centralized version of Cardholder MATCH or TMF available for merchants to filter against and it would be so subjective as to be useless for practical purposes.
There used to be a private site called badcustomer which attempted to police the friendly fraud type customers by creating such a central database. However this was undermined by the creator who used to offer the option for bad actors to pay to have their name removed!
Most merchants handle it in their own way. Your realistic options are.. Implement the anti fraud controls suggested by the previous contributor (AVS, CVV etc..) In addition you could enable 3D-Secure (Verified by Visa or Mastercard Securecode) This would in some cases shift the liability for a chargeback away from the acquirer to the issuer.
In terms of buyers remorse / friendly fraud chargeback situations.. The best thing to do is to try and get ahead of the game. There are services that can now report chargebacks to the merchant before they are received at the acquirer. This enables the merchant to be pro-active and reach out to cardholders to resolve the relevant issue either by the cardholder canceling their chargeback request or the merchant issuing a refund PRIOR to the chargeback being received
Here's what you can do:
if web, then 1) figure out if your anti-fraud tools are operating properly. (it might be ghost accounts (multiple users from same ip/deviceid. if so, ban the ips and device ids. If are unable to identify whether its a common ip or device id, then figure out if they used the same password by checking the hash (provided you have a single salt for all the password hashes). Usually fraud chains will use a scripts that will use the same passwords. If you have visibility on their security questions, then check that. Check other factors like similar times of login or very close to each other. Find out how your anti-fraud tools were abused and fix it.
if app, make sure devices were not compromised. If you don't have multifactor authentication, get it.
What should you do?
if web or app, then lock out the offending account, fence the funds, and make sure that any account that signs up from then on and shares similar parameters to the offending account is flagged and comes under your review. (Ex: same ip/ same device ID/ same password hashes/same responses to security questions)
Who do you report it to?
If it is more than $25k, you can expect that reporting it to the police will get you somewhere. Regardless, report it, but don't expect any effort on their part if less than $25k. Probability of that is pretty low.
If you are using a credit card PSP, then alert them, and tell them what you have done to make sure it doesn't happen. Alert your bank too and let them know how you have made sure it wont occur.
If you are registered as a MSB with FINCEN , file a (suspicious activity report) SAR with FINCEN. Your compliance officer can do that. If you don't have a CO, your legal counsel can help.
Finally, how to automate your fraud detection for future instances? You could get some traditional products that come with your PSP , but I find them very bloated and typically not good. I am now becoming a big fan of "machine learning". You should look into companies that provide that service.
Hope it helps,