Yodlee Soft-Refresh Optimization

Context

Yodlee is the only integration that blocks php-fpm worker threads during data refresh. All refresh operations run synchronously in HTTP request threads, occupying workers for up to 20 minutes (Redis lock timeout). Under high concurrency, this can exhaust the php-fpm worker pool and degrade other API endpoints.

Trigger

• DEV-3438: User cannot see manual refresh button because Yodlee soft-refresh frequently times out
• Parent Epic DEV-3901: Investigate and optimise nightly sync

Optimization Goals

1. Saloon migration - Migrate Yodlee’s custom Guzzle-based Connector to standard Saloon architecture
2. Async conversion - Move synchronous refresh operations to Laravel Queue
3. Notification integration - Add push notifications (reuse Wealthbox/Redtail patterns), including error feedback

Expected Benefits

• Significantly reduce php-fpm resource occupation
• Improve data sync user experience
• Unified, maintainable technical architecture

Current State Analysis

Why Yodlee Is Different

Yodlee is a completely separate legacy system from the standard Saloon-based integration architecture:

Dimension	Standard Integration	Yodlee
Framework	Saloon PHP	Guzzle HTTP direct
Connector	Extends Saloon\Http\Connector with middleware	Static call(), no base class
Authentication	OAuth 2.0 (thread-safe token refresh)	JWT RS512 (25-min cache)
Initiator	Advisor configures connection	Client links bank via FastLink
Data Model	Integration + IntegrationMapping	yodlee_providers + yodlee_provider_accounts + account_yodlees
IntegrationType	In enum (47 types)	Not in enum (uses AccountSource::YODLEE)
Nightly Sync	Supported (scheduled)	Not participating (requires user session context)
Refresh Model	Active scheduling (background jobs)	Synchronous blocking (HTTP thread)
Rate Limiting	Saloon middleware (Redis + HasRateLimits)	Manual Redis counter (9/120s, silent abort)

Current Refresh Flow (Synchronous)

User triggers POST /refresh
    -> Acquire Redis distributed lock (20 min timeout)
    -> Call Yodlee API: PUT providerAccounts?providerAccountIds={id}
    -> Return 202 (in progress) or 200 (complete)
    [php-fpm worker BLOCKED]

User polls GET /refresh
    -> Call Yodlee API: GET providerAccounts/{id}
    -> Check status, update local DB
    -> Return action_required to frontend
    [php-fpm worker BLOCKED on each poll]

Identified Problems

Problem	Impact
Synchronous blocking	php-fpm workers occupied for duration of Yodlee API calls
20 min lock timeout	Too long, masks stuck processes
Silent rate limiting	Users get no feedback when hitting 9-refreshes/120s limit
No error transparency	action_required: null means both “done” and “rate limit abort”
No middleware reuse	Cannot leverage Saloon’s logging, retry, tracing, rate-limit middleware

Connector and Integrator Roles

Understanding the layered architecture is essential for planning the migration scope.

Connector (Transport Layer)

Responsibility: Pure communication with external Vendor API.

Concern	Connector Does	Connector Does NOT
Authentication	Manage tokens, JWT signing, API keys	Understand business meaning of responses
HTTP transport	Send requests, receive responses, timeouts	Transform data to internal models
Error mapping	HTTP status -> domain exceptions	Write to database
Rate limiting	Track and enforce API limits	Decide business retry logic
Observability	Request/response logging, OTel tracing	Manage sync lifecycle state

Standard Saloon Connector hierarchy:

Saloon\Http\Connector
  -> Support\ApiBased\Connectors\Connector (rate limit + timeout + error middleware)
       -> Support\ApiBased\Connectors\OauthConnector (thread-safe token refresh)

Yodlee Connector (Integrations/Yodlee/Connector.php):

• Does NOT extend any base connector class
• Static-only API: call(url, method, headers, params, queries, context)
• Custom JWT RS512 authentication inline
• Custom exception types (not mapped through standard AffectedLevel)

Integrator (Business Orchestration Layer)

Responsibility: Orchestrate the entire data sync process.

Concern	Integrator Does	Integrator Does NOT
Data fetching	Call Connector to retrieve vendor data	Manage HTTP transport details
Data transformation	Vendor Model -> RC internal Model	Handle authentication
Persistence	Create/Update/Delete database records	Care about HTTP status codes
Sync lifecycle	Manage mapping states, detect changes	Handle connection retries
Entity discovery	List available entities from vendor (via EntityProvider interface)	Configure request headers

Standard Integrator base class (integrations-core):

abstract class Integrator {
    abstract public function syncAll(): void;                        // Batch sync
    abstract public static function sync(IntegrationMapping $mapping): ?array;  // Single sync
    abstract protected static function getVendor(): IntegrationType;
}

Yodlee Integrator (Integrations/Yodlee/Integrator.php, ~592 lines):

• Does NOT extend the base Integrator class
• Entry point: saveProviderAccount() (not sync())
• Overloaded responsibilities: status code mapping (30+ codes -> 5 actions), data sync, rate limit checking all in one class

How They Collaborate

Controller (HTTP entry point)
    |
    v
Integrator::sync($mapping)
    |
    |-- $connector->call('/endpoint')       <- Connector: HTTP + auth + error mapping
    |-- VendorModel::fromResponse($data)    <- Integrator: transform vendor data
    |-- Account::updateOrCreate(...)        <- Integrator: persist to database
    |-- $mapping->update(['last_completed_at' => now()])

Options Considered

Option 1: Saloon Connector Only (Transport Layer Migration)

Migrate Yodlee’s Connector.php from raw Guzzle to Saloon Connector. Data model and Integrator logic unchanged.

Pros:

• Smallest blast radius, Integrator untouched
• Gains: Saloon middleware (logging, retry, rate-limit, OTel tracing)
• Can be done independently of async work

Cons:

• Does not solve php-fpm blocking
• Does not address Integrator.php overload

Option 2: Async Queue + Frontend Polling

Move refresh operations into Laravel Queue. Frontend continues polling for results.

Pros:

• Directly solves php-fpm blocking (core goal)
• Leverages existing queue infrastructure
• Minimal frontend change (already polls GET /refresh)

Cons:

• Queue worker still synchronously polls Yodlee API (acceptable - isolated process)
• Needs intermediate state storage (Redis/DB) for job status

Option 3: Yodlee Webhook Integration

Use Yodlee’s webhook notification to avoid polling entirely.

Pros:

• Truly async, no polling needed
• Yodlee proactively notifies on completion

Cons:

• Requires webhook endpoint configuration and verification
• Frontend needs WebSocket/SSE for real-time updates (new infrastructure)
• Yodlee webhook API capability needs verification

Option 4: Hybrid (Short-term Polling + Long-term Webhook)

Queue job polls for first 30 seconds, then falls back to webhook notification.

Pros: Covers both fast and slow refresh scenarios

Cons: Highest complexity, over-engineering for current needs

Decision

Phased approach: Phase 1 -> Phase 2 -> Phase 3 (optional)

Phase 1: Async Queue + Error Transparency (Highest Priority)

Move refresh operations to Laravel Queue. This directly addresses the php-fpm blocking problem.

Key design decisions:

Aspect	Decision	Rationale
Job dispatch	POST /refresh dispatches job, returns 202 immediately	Frees php-fpm worker instantly
Status storage	Persist job state in Redis (keyed by provider account ID)	Frontend polls same GET /refresh endpoint
Lock redesign	Bind lock to Job lifecycle, not HTTP request. Shorten to 5 min with active renewal	Prevent stuck processes
Queue isolation	Dedicated yodlee-refresh queue	Prevent interference with other integration sync jobs
Polling in Job	Job polls Yodlee API with reasonable interval (5s) and max retries (60 = 5 min)	Bounded resource usage
Data sync	Execute saveAccountsWhenSuccessful() within Job on SUCCESS/PARTIAL_SUCCESS	Ensure transaction integrity and idempotency
Error transparency	Replace ambiguous null with explicit states: done / rate_limited / error / in_progress	Users get actionable feedback

Notification scenarios for Yodlee (vs standard integrations):

Scenario	Standard Integration	Yodlee-Specific
Sync success	Supported	Data refreshed notification
Sync failure	Supported	Technical error notification
Credentials expired	Supported	put action - prompt reconnect via FastLink
MFA required	Not applicable	manual_refresh - prompt FastLink Refresh mode
Partial success	Not applicable	PARTIAL_SUCCESS - some accounts have issues
Rate limited	Not applicable	Currently silent, should notify

Notification channel: Frontend polling + toast notification (minimal change). WebSocket/SSE is not justified given low event frequency.

Phase 2: Saloon Connector Migration

Migrate Yodlee’s Connector from raw Guzzle to Saloon architecture.

Scope: Transport layer only. Data model (yodlee_* tables) and IntegrationType enum unchanged.

Component	Before	After
Base class	None (standalone)	Extends Support\ApiBased\Connectors\Connector
Authentication	Inline JWT RS512	Saloon Authenticator (custom JWT authenticator)
Rate limiting	Manual Redis counter	Saloon HasRateLimits middleware
Error handling	Custom exception types	Map to standard ExternalServiceException hierarchy
Logging	Manual file write	Saloon middleware (WriteFatalRequestToFileLog)
Tracing	Manual OTel spans	Saloon APM middleware

Why NOT migrate to IntegrationType enum / Integration model:

• Yodlee’s passive refresh model (requires user session context) is fundamentally incompatible with nightly sync
• Data model migration (yodlee_* tables -> integrations + integration_mappings) would be high cost, low benefit
• AccountSource::YODLEE works correctly for its use case

Phase 3: Notification Enhancement (Optional)

Reuse Wealthbox/Redtail notification patterns for richer Yodlee feedback:

• App-level notifications for long-running refreshes
• Persistent error messages for credential issues
• Consider integration with existing failed_since / failed_biz_days tracking pattern if Yodlee is ever brought closer to the standard model

Open Questions

1. Yodlee Webhook API - Does Yodlee support webhook notifications for refresh completion? Needs verification against latest Yodlee documentation under STG ownership.
2. Frontend polling adjustment - When POST /refresh returns 202 immediately, does the frontend polling interval need tuning?
3. Monitoring metrics - What new metrics are needed post-async? (job duration, success rate, Yodlee API latency, queue depth)
4. Integrator refactoring - Should Integrator.php’s status code mapping be extracted into a dedicated StatusResolver class during Phase 2?

Related

• Yodlee - Vendor documentation
• DEV-3438 - Original ticket investigation
• API-based Integrations - Standard architecture
• Sync Lifecycle - Job flow and error handling
• API-based Integration Patterns - Connector/Integrator patterns
• Yodlee Data Flow - Detailed data flow diagrams