Real-time Sanctions Screening with Didit and Kafka

Scalable ComplianceIntegrating Didit's AML Screening API with Apache Kafka enables financial institutions and businesses to achieve real-time, high-throughput sanctions screening, essential for modern compliance and risk management.

Architectural EfficiencyLeveraging Kafka's distributed streaming platform allows for asynchronous processing, buffering of requests, and reliable data delivery, ensuring that even under heavy load, screening requests are handled efficiently without impacting user experience.

Intelligent Risk ScoringDidit's advanced two-score system (Match Score and Risk Score) provides granular insights into potential risks, allowing for configurable compliance thresholds and reducing false positives through AI-powered assessment.

Seamless Integration with DiditDidit offers a developer-first approach with clean APIs and a modular architecture, making it straightforward to embed real-time AML screening into existing high-throughput systems, complemented by Free Core KYC and no setup fees.

The Imperative for Real-time Sanctions Screening

In today's fast-paced digital economy, financial institutions, fintech companies, and any business handling transactions or onboarding users face an ever-growing challenge: staying compliant with Anti-Money Laundering (AML) and Counter-Terrorist Financing (CTF) regulations. Traditional batch-based sanctions screening methods are no longer sufficient to combat sophisticated financial crime, which operates in real time. The need for immediate identification of individuals and entities on global watchlists, sanctions lists, and Politically Exposed Persons (PEP) databases is paramount. Delays can lead to significant regulatory penalties, reputational damage, and an increased risk of facilitating illicit activities.

Real-time sanctions screening allows organizations to assess risk instantly at critical junctures, such as account opening, transaction initiation, or even continuous monitoring. This proactive approach minimizes exposure to high-risk individuals and entities, ensuring that businesses remain compliant and secure. However, achieving true real-time screening at scale, especially in high-throughput environments, presents significant architectural and technical challenges. This is where combining powerful, AI-native APIs like Didit's AML Screening with robust message brokers like Apache Kafka becomes a game-changer.

Architecting for Scale: Didit's AML API with Apache Kafka

Building a real-time sanctions screening system capable of handling millions of requests requires a scalable, resilient, and high-performance architecture. Apache Kafka, a distributed streaming platform, is an ideal choice for this purpose due to its ability to handle high volumes of data, provide fault tolerance, and enable asynchronous processing. When integrated with Didit's AML Screening API, it creates a powerful compliance engine.

The architecture typically involves producing screening requests to a Kafka topic. These requests might originate from various sources: new user registrations, transaction processing systems, or periodic re-screening jobs. Consumer applications then read from this topic, call Didit's AML Screening API, and publish the results to another Kafka topic. This decoupled approach offers several advantages:

• High Throughput: Kafka can ingest and process millions of messages per second, ensuring that screening requests are never a bottleneck.
• Scalability: Both Kafka and Didit's API are designed for scale. You can easily add more Kafka brokers or consumer instances to handle increasing load.
• Resilience: Kafka's distributed nature and data replication ensure that messages are not lost, even in the event of system failures.
• Asynchronous Processing: Screening requests can be processed in the background without blocking the originating application, improving user experience.
• Auditability: Kafka provides a durable log of all screening requests and responses, crucial for compliance audits.

Didit's AML Screening API screens users against over 1300 global sanctions, PEP, and watchlist databases in real time, making it perfectly suited for this high-volume, real-time integration. The API provides a comprehensive report, including hit details, risk scores, match scores, and adverse media intelligence, which can then be consumed by downstream systems for automated decision-making or manual review.

Understanding Didit's Two-Score Risk System

Effective AML screening isn't just about identifying a potential match; it's about understanding the nuances of that match to avoid false positives and accurately assess risk. Didit's AML Screening employs a sophisticated two-score system – the Match Score and the Risk Score – providing granular control and intelligence for compliance teams.

The Match Score addresses the question: "Is this potential match the same person or entity we are screening?" It's an identity confidence score, calculated based on factors like name similarity, date of birth, nationality, and document numbers. This score helps to distinguish between a true match and a false positive. For instance, a high Match Score (e.g., above 93, Didit's default threshold) indicates a strong likelihood that the individual being screened is indeed the one on the watchlist. Requests falling below this threshold are often classified as false positives, streamlining the review process.

The Risk Score, conversely, evaluates: "How risky is this entity if it is a true match?" This score assesses the inherent risk level of the matched entity, taking into account factors such as country risk, the specific category of the watchlist (e.g., PEP, sanctions, criminal records), and other relevant intelligence. The Risk Score determines the final AML status – Approved, In Review, or Declined – based on configurable thresholds. For example, a score below the 'approve threshold' (default 80) might lead to automatic approval, while a score above the 'review threshold' (default 100) could trigger an automatic decline. Scores in between typically require manual review by a compliance officer.

This dual-scoring mechanism, configurable via parameters like aml_match_score_threshold, aml_score_approve_threshold, and aml_score_review_threshold in the API request, allows businesses to fine-tune their AML policies to their specific risk appetite and regulatory requirements, significantly reducing manual review burdens while maintaining robust compliance.

Implementing Real-time Screening Workflows

Integrating Didit's AML Screening API into a Kafka-based pipeline involves several key steps. First, define the data structure for your screening requests and responses. Requests typically include full_name, entity_type (person or company), date_of_birth, nationality, and optional parameters like document_number or custom score thresholds.

When a new user registers or a transaction is initiated, a message containing the necessary user data is produced to an 'aml-screening-requests' Kafka topic. A dedicated microservice, acting as a Kafka consumer, reads these messages. For each message, it constructs a request to Didit's /v3/aml/ endpoint. Didit processes the request in real time, performing checks against global watchlists and applying its intelligent two-score risk system. The API response, which includes the overall AML status, match details, and various risk scores, is then received by the microservice.

Upon receiving Didit's response, the microservice can publish the results to an 'aml-screening-results' Kafka topic. Downstream systems, such as a user onboarding service, a transaction processing engine, or a case management system, can then consume these results. For instance, if the AML status is 'Approved', the user onboarding can proceed. If it's 'In Review', a flag can be set for a compliance officer to manually investigate. For 'Declined' statuses, appropriate actions can be triggered, such as blocking a transaction or denying account creation.

This implementation ensures that the core business logic remains decoupled from the compliance checks, allowing each component to scale independently and maintain high availability. The use of Kafka also provides an inherent retry mechanism and backpressure handling, preventing the Didit API from being overwhelmed during peak loads, and ensuring that no screening request is ever missed.

How Didit Helps

Didit is at the forefront of providing AI-native, developer-first identity verification solutions designed for modern, high-throughput systems. Our AML Screening product is a cornerstone of our offering, enabling businesses to screen individuals or companies against 1300+ global sanctions, PEP, and watchlist databases in real time. Our modular architecture means you can seamlessly integrate AML Screening as a standalone API or as part of a broader identity verification workflow, without complex setup or lengthy integration times. The AI-native foundation of Didit ensures that our two-score risk system (Match Score and Risk Score) is constantly optimized for accuracy, reducing false positives and providing actionable intelligence for compliance teams.

Beyond powerful AML Screening, Didit offers a comprehensive suite of identity primitives, including ID Verification (OCR, MRZ, barcodes), Passive & Active Liveness, and 1:1 Face Match & Face Search. Our developer-first approach includes an instant sandbox and clean APIs, making integration straightforward. We stand out with our commitment to making robust identity verification accessible, offering Free Core KYC and absolutely no setup fees, allowing businesses of all sizes to automate trust and ensure compliance globally and at scale.

Ready to Get Started?

Ready to see Didit in action? Get a free demo today.

Start verifying identities for free with Didit's free tier.