The rapid development of information technology coupled with an increase in public activity in electronic financial transactions has provided convenience but has been accompanied by the occurrence of fraudulent financial transactions. The purpose of this research is how to find the best model to be implemented in the banking payment system in detecting fraudulent electronic financial transactions so as to prevent losses for customers and banks. Fraud detection uses machine learning with ensemble and deep learning with SMOTE. Financial transaction data is taken from bank payment simulations built with the concept of Multi Agent-Based Simulation (MABS) by banks in Spain. To build the best model, not only pay attention to the accuracy value, but the value of precision is a value that needs attention. A precision score is very important for fraud prevention. Fraud detection gets the best results without the SMOTE process by using deep learning with an accuracy score of 99.602% and precision score of 90.574%. By adding SMOTE, it will increase the accuracy score and precision score with the best model produced in the Extra Trees Classification with an accuracy score of 99.835% and precision score of 99.786%.