Idempotent Webhook Consumers in Ruby on Rails for Didit Events

Ensuring Data IntegrityIdempotency is vital for processing webhook events reliably, preventing duplicate actions and maintaining a consistent state in your application.

Leveraging Webhook SignaturesAlways verify webhook signatures to confirm the authenticity and integrity of incoming requests, protecting against tampering and spoofing.

Utilizing Unique Event IDsStore and check unique event IDs provided in webhook payloads to detect and discard duplicate deliveries effectively.

Didit's Robust Webhook SystemDidit provides secure, versioned webhooks with HMAC signatures and unique event IDs, simplifying the implementation of idempotent consumers and ensuring reliable event delivery for critical identity verification workflows.

The Challenge of Webhook Idempotency

Webhooks are a powerful mechanism for real-time communication between services, enabling your application to react instantly to events occurring in another system. However, the distributed nature of webhooks means that events can sometimes be delivered multiple times. Network glitches, timeouts, or retries can all lead to duplicate webhook payloads. Without proper handling, these duplicates can cause significant issues in your application, such as creating duplicate records, triggering redundant actions, or corrupting data.

This is where idempotency becomes critical. An idempotent operation is one that produces the same result whether it's executed once or multiple times. For webhook consumers, this means designing your system to process a given event only once, even if the webhook payload is received multiple times. When dealing with sensitive data like identity verification results from platforms like Didit, ensuring idempotency is not just good practice—it's essential for maintaining data integrity and system reliability.

Verifying Webhook Signatures for Security and Authenticity

Before processing any webhook payload, the first and most critical step is to verify its authenticity. This ensures that the webhook truly originated from the expected sender (e.g., Didit) and that its content has not been tampered with in transit. Didit's webhooks, for instance, include an HMAC signature in the request headers, generated using a shared secret key.

In your Ruby on Rails application, you'll need to retrieve the shared secret key from your Didit account (which you can access via the Business Console or programmatically using the API). When a webhook arrives, you'll compute your own HMAC signature using the request body and your secret key. This computed signature is then compared to the signature provided in the webhook header. If they don't match, the request should be rejected immediately.

class DiditWebhooksController < ApplicationController
  skip_before_action :verify_authenticity_token

  def create
    # Retrieve the shared secret key from your environment variables
    secret = ENV['DIDIT_WEBHOOK_SECRET']
    payload = request.body.read
    signature = request.headers['X-Didit-Signature'] # Or similar header name

    unless verify_signature(payload, signature, secret)
      head :unauthorized
      return
    end

    # Process the webhook payload after verification
    process_didit_event(JSON.parse(payload))
    head :ok
  end

  private

  def verify_signature(payload, signature, secret)
    digest = OpenSSL::Digest.new('sha256')
    hmac = OpenSSL::HMAC.hexdigest(digest, secret, payload)
    ActiveSupport::SecurityUtils.secure_compare("sha256=#{hmac}", signature)
  end

  def process_didit_event(event_data)
    # ... implementation for processing ...
  end
end

Didit simplifies this by providing a secret_shared_key as part of its webhook configuration, which can be retrieved via the API or rotated as needed. This secure mechanism is fundamental for any identity verification workflow, where the integrity of results, such as those from ID Verification or Liveness checks, is paramount.

Implementing Idempotency with Unique Event Identifiers

Once you've verified the webhook's authenticity, the next step is to ensure idempotency. Most well-designed webhook systems, including Didit's, provide a unique identifier for each event. This ID is crucial for detecting and preventing duplicate processing. For Didit's webhooks (especially v3, which is recommended), each event payload includes a unique ID that you can use for this purpose.

The strategy is to store these event IDs in your database and check against them before processing. A common pattern involves creating an EventLog or WebhookEvent model:

# db/migrate/YYYYMMDDHHMMSS_create_webhook_events.rb
class CreateWebhookEvents < ActiveRecord::Migration[7.x]
  def change
    create_table :webhook_events do |t|
      t.string :event_id, null: false, index: { unique: true }
      t.string :event_type
      t.jsonb :payload
      t.string :status, default: 'pending'
      t.timestamps
    end
  end
end

When a webhook arrives:

1. Extract the unique event_id from the payload.
2. Attempt to create a new WebhookEvent record with this event_id.
3. If the creation fails due to a unique constraint violation (meaning the event_id already exists), then you know it's a duplicate and can safely ignore it or log it as such.
4. If the creation is successful, proceed to process the event, marking its status accordingly.

class DiditWebhooksController < ApplicationController
  # ... (signature verification as above) ...

  def create
    # ... (signature verification) ...

    event_data = JSON.parse(payload)
    event_id = event_data['id'] # Assuming 'id' is the unique event identifier from Didit

    # Use a transaction to ensure atomicity
    ActiveRecord::Base.transaction do
      webhook_event = WebhookEvent.find_or_initialize_by(event_id: event_id)

      if webhook_event.persisted? # If it already exists, it's a duplicate
        Rails.logger.info "Duplicate webhook event received: #{event_id}"
        head :ok
        return
      end

      webhook_event.event_type = event_data['type']
      webhook_event.payload = event_data
      webhook_event.status = 'processing'
      webhook_event.save!

      # Process the event in a background job for long-running tasks
      DiditEventProcessorJob.perform_later(webhook_event.id)
      head :ok
    end
  rescue ActiveRecord::RecordNotUnique # Handle race conditions for event_id
    Rails.logger.warn "Race condition detected for webhook event: #{event_id}. Ignoring."
    head :ok
  rescue JSON::ParserError
    head :bad_request
  rescue => e
    Rails.logger.error "Error processing webhook: #{e.message}"
    head :internal_server_error
  end
end

This approach, combined with background jobs for actual processing, ensures that your webhook endpoint responds quickly, preventing retries from the sender, while reliably handling duplicates.

Handling Race Conditions and Transactions

Even with a unique index, race conditions can occur if two identical webhooks arrive almost simultaneously. Both might attempt to create a new WebhookEvent record before the first one commits its transaction. To mitigate this:

• Use Database Transactions: Wrap the find_or_initialize_by and subsequent processing logic within a database transaction. This ensures that either the entire operation succeeds or fails, maintaining data consistency.
• Handle RecordNotUnique: Be prepared to catch ActiveRecord::RecordNotUnique exceptions. If this exception occurs when trying to save a new WebhookEvent, it signifies a race condition where another process has already inserted the event, and you can safely treat it as a duplicate.

For operations that modify core application data, extending the transaction to cover those changes is crucial, or at least ensuring that the background job processing the event also implements its own idempotency checks on the application data it modifies.

How Didit Helps

Didit is an AI-native, developer-first identity platform designed with reliability and security in mind, making it easier to implement robust webhook consumers. Our modular architecture provides clean APIs and secure webhooks that are inherently designed to support idempotent processing.

• Secure Webhooks: Didit's webhooks provide strong HMAC signatures (using a secret_shared_key you can manage and rotate) to ensure the authenticity and integrity of every event. This crucial first step in idempotency is built-in. You can even specify the webhook_version (v3 recommended) for optimal payload structure.
• Unique Event Identifiers: Every Didit webhook event includes a unique identifier, making it straightforward to implement the deduplication logic discussed above.
• Configurable Data Retention: With Didit, you control how long verification data is stored. You can set data retention policies from 1 month to 10 years, or even null for unlimited, directly in the Business Console or via API. This allows you to meet compliance requirements (like GDPR) while managing your data footprint. This also ties into how your webhook consumer might store and manage event logs.
• Free Core KYC & Modular Design: Didit offers Free Core KYC, allowing you to start building and testing your webhook consumers without upfront costs. Our modular design means you can easily integrate specific identity verification products—like ID Verification for document checks, Passive & Active Liveness for fraud prevention, or Age Estimation for age verification—and receive real-time updates via webhooks.

By leveraging Didit's robust and secure webhook system, developers can focus more on their application's core logic and less on the complexities of securing and deduplicating incoming events, leading to more resilient and trustworthy identity verification workflows.

Ready to Get Started?

Ready to see Didit in action? Get a free demo today.

Start verifying identities for free with Didit's free tier.