Transcript: Redis + Amazon SageMaker for real-time fraud detection demo

Version 1: Source Transcript

This document represents the raw lecture transcript recorded from the video lecture. For architectural study notes and system design documentation, see Version 2.

• Video URL: https://www.youtube.com/watch?v=kQKfXi7NfWs
• Video ID: kQKfXi7NfWs
• Word Count: 2,106 words

Full Transcript

hello and welcome to another lightning demo my name is Talon Miller I do technical marketing here at redus today we're going to implement a solution for fraud detection using Amazon Sage maker and Rus Cloud let's dive in before we dive into our use case of fraud detection it's worth mentioning that you can use red as cloud and Sage maker together for more than just fraud detection Sage maker is a great service for building training and deploying machine learning models because it contains all of the managed infrastructure tools and workflows you'd want when working with machine learning models right as cloud has best-in-class submillisecond latency and high availability linear scalability and a huge variety of data structures to work with which makes it a great data store for use cases that Implement machine learning more details on that as we dive into the challenges with fraud detection Solutions in production fraud detection is a complex process and it has an incredible impact whether it's working as intended or fails to flag a bad actor so we're working with very high stakes the first of a few reasons why large Financial Services organizations choose r as cloud is it's 59 high availability this means these organizations can have their fraud detection Solutions up and running 99.99% of the time which is imperative the fraud detection solution needs to flag or decide whether a transaction or activity is fraudulent or not and take action within seconds if not milliseconds failing to do so can lead to significant losses and I mean significant like b as in billions of dollars of losses reported each year because of fraud detection frauders or Bad actors are evolving every day and are moving in tandem with digital banking transformation discovering innovative ways to steal or fake customers identities two challenges that yield both unhappy customers and substantial loss false positives and latency false positives is when a system flags a legitimate transaction by the user as fraud I can imagine that could have happened to you before it happens commonly when you travel internationally for example effective machine learning algorithms such as random cut forest and XG boost are effective in combating fraud but as fraud evolves a multi-layer approach needs to be in place in addition to continuous model learning so why latency well latency translates to speed or lack thereof so mature fraud detection Solutions commonly use in memory stores like redis Cloud so that fraud can be flagged ASAP in this video we'll go over the solution architecture and prerequisites you'll need to build this along with me and then go over the steps of implementing this solution with that said let's jump into what the solution looks like first organizations can adopt either a rule based solution or ml based solution or a combination of both to mitigate fraud detection in real time rule based Solutions typically enforce a set of hard-coded rules to determine if the transaction is fraudulent or not for example blacklisting a set of ips using location based rules like my traveling example or using a purchase profile information to determine if a user has purchased in a particular product category before let's dive into how our solution built on sagemaker and do Cloud integrate to provide the Best of Both Worlds when implementing rule-based and ml-based real-time fra detection systems Amazon sagemaker comes with pre-built ml algorithms like the ones I mentioned earlier XG boost and random cut Forest this is one of the benefits of sage as it reduces the amount of development time we would spend on our solution Red Cloud is a fully managed service and will use Advanced data structures such as search Json and time series to handle the various data types we'll be working with like Json documents indexing data while persisting for faster complex SQL like query executions with search and persist the ml inference results data in Time series format to build visualizations at inference time the ml engine needs to quickly read the latest feature values from the online feature store and use them based on the real-time features the ml model will score the transaction this makes building an online and real-time accessible feature store on reddis a strong architectural design Choice we'll use S3 for our historical data sets of credit card transactions Lambda functions process transactions from the historical data sets and invoke the two Sage maker endpoints that assign anomaly scores and classification scores to incoming data points end users whether on mobile or web invoke Amazon API Gateway rest apis for predictions using signed HTTP requests Kinesis is used to capture real-time event data another Lambda function reads the stream and persist transactional data to the search and Json data structures in a redos cloud database another Lambda function further persist the prediction results to the redus cloud database As Time series data for realtime analytics or another optional lamb function can pass the prediction results via Kinesis which persists the data to an S3 bucket so that quick site can consume this data for analytics now that you know how we're going to set it up let's get going on our setup and if you would like to follow along I'll put the GitHub repo in the description of the video now let's go over the prerequisites first an AWS account and a redis cloud account you can start a 14-day trial with redis cloud from the AWS Marketplace or directly on our website for our purposes though we'll need a database larger than the free tier so I'm going to create a flexible subscription with two databases one with Json and search and the other with time series you'll also need the AWS CLI setup lastly Docker and Docker compose should be installed to run containerized applications and AWS Lambda functions all right let's dive in to start off this implementation I'm going to break this section up into six parts first we're going to set up the infrastructure second we're going to train test and deploy our sag maker models and endpoints third we're going to build and deploy Lambda functions fourth we're going to set up Amazon Kinesis fifth we're going to turn on our solution and see the fraud detection in action and finally six we're going to visualize our solution in grafana first we're going to start in the AWS console go to cloud formation create stack select upload a template file choose file and select the template file from the GitHub repo give your sack a name select the correct configurations like IM rolls and then create it this will stand up a lot of our resources like our sagemaker Jupiter notebooks that contains our models and S3 buckets it can take some time for that cloud formation stack to stand up once it's complete we're going to navigate to sagemaker and get into our second section enter Jupiter lab and we're going to execute each of these cells one by one there's a lot to go through here on video but each step is detailed in The Notebook for you to understand how we're training and testing our two models XG boost and random cut Forest to eventually do some supervised learning and use that data to evaluate how our model model are performing I would love to spend another 5 minutes just going through this in detail but we must press on the notebook will generate two model endpoints we'll need before moving on to our next section which is our third section building and deploying our Lambda functions to do this we'll need to set up an ec2 instance I recommend setting up sentos OS using this in the search bar and then switching to the AWS Marketplace Amis select t2. medium for our instance type and then create a key value pair with RSA and PM as the file format and then you'll download the file format and then create a security group with my IP like shown here configure the storage to 30 and then launch the instance once it's set up grab the public IP and SSH then SSH into your ec2 instance there are quite a few things we need to do in our terminal that I'll Breeze through here like installing prerequisite packages and generic utilities like Docker Docker compos get WG python AWS CLI and then finally we need to configure our AWS CLI for that we'll need to go back to our AWS console navigate to IM and then to security credentials scroll down to access keys and create new access key for the CLI grab your access key and secret and we'll execute those with our region after we get back into our ec2 instance install git WG Python and the AWS CLI configure and then use that new key in secret we just created next we'll actually build out the Lambda functions as a Docker container and then return back to the AWS console we're going to build a Docker container and upload it to the ECR or elastic container registry let's create a repository give it a name and keep it private create repo and then click the view push commands these commands are what we're going to commit next on our terminal making sure I'm in the Lambda folder as shown let's execute those commands this can take a little bit of time so we'll speed this up the cool thing is that back on your ECR page you should have a container in your private repositories next let's go use our new Docker image and finally create this Lambda function using a container image name it browse images and select our image drop down the option for execution Rule and select create a new role from AWS policy templates and then create a function time to point our Lambda function to our R database grab the host port and password and we'll input that back into the AWS console in configuration and environment variables We'll add our host port and password and save that's it for this section on to our fourth section setting up Kinesis setting up Kinesis and add it to our function takes less time very little time so let's do it in Kinesis create a data stream after it's up and running back on our function we're going to click add trigger search Kinesis select it search the name of your data stream change the batch size to one instead of 100 and click add now that we have a trigger connected to our function with Reds Cloud as our en ironment or database when a new record arrives in the Kinesis data stream it will immediately start the Lambda function the Lambda function persists the transactions as they are arriving to the redis cloud database it makes inferences to the sagemaker models to detect if a specific event is fraudulent or not finally it persists the inference data back into our Red's Cloud database to drive realtime data visualization and just like that done with our fourth section onto our fifth section which is actually testing our solution let's get back to our terminal and back into our ec2 machine and let's get into our python environment let's install some dependencies and then the exciting part we'll actually simulate thousands of transactions as if they're happening real time check this out this is what an individual record looks like that is getting pushed into our Kinesis data stream let's go look add it in the AWS console looking at our Lambda function let's click the monitor tab as you can see we're processing some data here the data stream we set up is working as intended but this isn't super clear how the fraud detection is performing so let's jump into our final section and see it visualized in grafana we're now looking at our fraud transaction score which is basically a grade of how we're doing detecting fraud from the transactions coming in on the right we have a live view of how we're scoring on the bottom left account of how much fraud is happening within a certain time period and finally on the right fraud versus non-f fraud scores well that covers it for our lightning demo I hope you enjoyed it and I hope you learned something new that redis can do I hope it pushes your curiosity to build something new with redus and with that I'll see you in the next one

Timestamped Segments

• [00:00] hello and welcome to another lightning demo my name is Talon Miller I do technical marketing here at redus today we're going to implement a solution for fraud detection using Amazon Sage maker and Rus Cloud let's dive in before we dive into our use case of fraud detection it's worth mentioning that you can use red as cloud and Sage maker together for more than just fraud detection Sage maker is a great service for building training and deploying machine learning models because it contains all of the managed infrastructure tools and workflows you'd want when working with machine learning
• [00:32] models right as cloud has best-in-class submillisecond latency and high availability linear scalability and a huge variety of data structures to work with which makes it a great data store for use cases that Implement machine learning more details on that as we dive into the challenges with fraud detection Solutions in production fraud detection is a complex process and it has an incredible impact whether it's working as intended or fails to flag a bad actor so we're working with very high stakes the first of a few reasons why large Financial Services organizations choose
• [01:06] r as cloud is it's 59 high availability this means these organizations can have their fraud detection Solutions up and running 99.99% of the time which is imperative the fraud detection solution needs to flag or decide whether a transaction or activity is fraudulent or not and take action within seconds if not milliseconds failing to do so can lead to significant losses and I mean significant like b as in billions of dollars of losses reported each year because of fraud detection frauders or
• [01:39] Bad actors are evolving every day and are moving in tandem with digital banking transformation discovering innovative ways to steal or fake customers identities two challenges that yield both unhappy customers and substantial loss false positives and latency false positives is when a system flags a legitimate transaction by the user as fraud I can imagine that could have happened to you before it happens commonly when you travel internationally for example effective machine learning algorithms such as random cut forest and XG boost are effective in combating
• [02:12] fraud but as fraud evolves a multi-layer approach needs to be in place in addition to continuous model learning so why latency well latency translates to speed or lack thereof so mature fraud detection Solutions commonly use in memory stores like redis Cloud so that fraud can be flagged ASAP in this video we'll go over the solution architecture and prerequisites you'll need to build this along with me and then go over the steps of implementing this solution with that said let's jump into what the solution looks like first organizations can adopt either a rule based solution
• [02:47] or ml based solution or a combination of both to mitigate fraud detection in real time rule based Solutions typically enforce a set of hard-coded rules to determine if the transaction is fraudulent or not for example blacklisting a set of ips using location based rules like my traveling example or using a purchase profile information to determine if a user has purchased in a particular product category before let's dive into how our solution built on sagemaker and do Cloud integrate to provide the Best of Both Worlds when implementing rule-based and ml-based
• [03:19] real-time fra detection systems Amazon sagemaker comes with pre-built ml algorithms like the ones I mentioned earlier XG boost and random cut Forest this is one of the benefits of sage as it reduces the amount of development time we would spend on our solution Red Cloud is a fully managed service and will use Advanced data structures such as search Json and time series to handle the various data types we'll be working with like Json documents indexing data while persisting for faster complex SQL like query executions with search and persist the ml inference results data in
• [03:53] Time series format to build visualizations at inference time the ml engine needs to quickly read the latest feature values from the online feature store and use them based on the real-time features the ml model will score the transaction this makes building an online and real-time accessible feature store on reddis a strong architectural design Choice we'll use S3 for our historical data sets of credit card transactions Lambda functions process transactions from the historical data sets and invoke the two Sage maker endpoints that assign anomaly
• [04:27] scores and classification scores to incoming data points end users whether on mobile or web invoke Amazon API Gateway rest apis for predictions using signed HTTP requests Kinesis is used to capture real-time event data another Lambda function reads the stream and persist transactional data to the search and Json data structures in a redos cloud database another Lambda function further persist the prediction results to the redus cloud database As Time series data for realtime analytics or
• [04:59] another optional lamb function can pass the prediction results via Kinesis which persists the data to an S3 bucket so that quick site can consume this data for analytics now that you know how we're going to set it up let's get going on our setup and if you would like to follow along I'll put the GitHub repo in the description of the video now let's go over the prerequisites first an AWS account and a redis cloud account you can start a 14-day trial with redis cloud from the AWS Marketplace or directly on our website for our purposes
• [05:33] though we'll need a database larger than the free tier so I'm going to create a flexible subscription with two databases one with Json and search and the other with time series you'll also need the AWS CLI setup lastly Docker and Docker compose should be installed to run containerized applications and AWS Lambda functions all right let's dive in to start off this implementation I'm going to break this section up into six parts first we're going to set up the infrastructure
• [06:07] second we're going to train test and deploy our sag maker models and endpoints third we're going to build and deploy Lambda functions fourth we're going to set up Amazon Kinesis fifth we're going to turn on our solution and see the fraud detection in action and finally six we're going to visualize our solution in grafana first we're going to start in the AWS console go to cloud formation create stack select upload a template file choose file and select the template file from
• [06:42] the GitHub repo give your sack a name select the correct configurations like IM rolls and then create it this will stand up a lot of our resources like our sagemaker Jupiter notebooks that contains our models and S3 buckets it can take some time for that cloud formation stack to stand up once it's complete we're going to navigate to sagemaker and get into our second section enter Jupiter lab and we're going to execute each of these cells one by one there's a lot to go through here
• [07:15] on video but each step is detailed in The Notebook for you to understand how we're training and testing our two models XG boost and random cut Forest to eventually do some supervised learning and use that data to evaluate how our model model are performing I would love to spend another 5 minutes just going through this in detail but we must press on the notebook will generate two model endpoints we'll need before moving on to our next section which is our third section building and deploying our Lambda functions to do this we'll need
• [07:49] to set up an ec2 instance I recommend setting up sentos OS using this in the search bar and then switching to the AWS Marketplace Amis select t2. medium for our instance type and then create a key value pair with RSA and PM as the file format and then you'll download the file format and then create a security group with my IP like shown here configure the storage to 30
• [08:24] and then launch the instance once it's set up grab the public IP and SSH then SSH into your ec2 instance there are quite a few things we need to do in our terminal that I'll Breeze through here like installing prerequisite packages and generic utilities like Docker Docker compos get WG python AWS CLI and then finally we need to configure our AWS CLI for that we'll need to go back to our AWS console navigate to IM and then to security
• [09:01] credentials scroll down to access keys and create new access key for the CLI grab your access key and secret and we'll execute those with our region after we get back into our ec2 instance install git WG Python and the AWS CLI configure and then use that new key in secret we just created next we'll actually build out the Lambda functions as a Docker container and then return back to the
• [09:34] AWS console we're going to build a Docker container and upload it to the ECR or elastic container registry let's create a repository give it a name and keep it private create repo and then click the view push commands these commands are what we're going to commit next on our terminal making sure I'm in the Lambda folder as shown let's execute those commands this can take a little bit of time so we'll speed this
• [10:06] up the cool thing is that back on your ECR page you should have a container in your private repositories next let's go use our new Docker image and finally create this Lambda function using a container image name it browse images and select our image drop down the option for execution Rule and select create a new role from AWS policy templates and then create a function time to point our Lambda function to our R database grab the host
• [10:40] port and password and we'll input that back into the AWS console in configuration and environment variables We'll add our host port and password and save that's it for this section on to our fourth section setting up Kinesis setting up Kinesis and add it to our function takes less time very little time so let's do it in Kinesis create a data stream after it's up and running back on our function we're going to click add trigger search Kinesis select
• [11:15] it search the name of your data stream change the batch size to one instead of 100 and click add now that we have a trigger connected to our function with Reds Cloud as our en ironment or database when a new record arrives in the Kinesis data stream it will immediately start the Lambda function the Lambda function persists the transactions as they are arriving to the redis cloud database it makes inferences to the sagemaker models to detect if a
• [11:48] specific event is fraudulent or not finally it persists the inference data back into our Red's Cloud database to drive realtime data visualization and just like that done with our fourth section onto our fifth section which is actually testing our solution let's get back to our terminal and back into our ec2 machine and let's get into our python environment let's install some dependencies and then the exciting part we'll actually simulate thousands of transactions as if they're happening real time check this out this is what an
• [12:23] individual record looks like that is getting pushed into our Kinesis data stream let's go look add it in the AWS console looking at our Lambda function let's click the monitor tab as you can see we're processing some data here the data stream we set up is working as intended but this isn't super clear how the fraud detection is performing so let's jump into our final section and see it visualized in grafana we're now looking at our fraud transaction score which is basically a grade of how we're doing detecting fraud from the
• [12:56] transactions coming in on the right we have a live view of how we're scoring on the bottom left account of how much fraud is happening within a certain time period and finally on the right fraud versus non-f fraud scores well that covers it for our lightning demo I hope you enjoyed it and I hope you learned something new that redis can do I hope it pushes your curiosity to build something new with redus and with that I'll see you in the next one