DEV-3438: Yodlee Soft-Refresh Async

Ticket Overview

Field	Value
Key	DEV-3438
Summary	Put Yodlee soft-refresh into async job
Status	To Do
Priority	Medium
Assignee	Heli Yin
Parent Epic	DEV-3901: Investigate and optimise nightly sync
Created	2020-11-19

Background

User (ID: 54618, Household: 226359) cannot see the manual refresh button on Profile > Net Worth page. The root cause is that Yodlee soft-refresh frequently times out during synchronous execution.

Original Proposal (Rejected)

Put Yodlee soft-refresh into a scheduled nightly sync job.

Why Nightly Sync Doesn’t Work

Update (2026-01-19): Nightly sync is not suitable for Yodlee because:

Yodlee’s refresh mechanism is passively triggered
Data only refreshes when active advisors/households trigger API calls
Unlike Morningstar/ByAllAccounts, Yodlee has no proactive scheduled sync API
startProviderAccountRefresh can only be called within user session context

Code Investigation

Key Files

File	Purpose
`retail-api/app/Http/Controllers/.../YodleeProviderAccountRefreshController.php`	Refresh controller (initiate/poll)
`retail-api/app/Integrations/Yodlee/Api.php`	Yodlee API wrapper
`retail-api/app/Integrations/Yodlee/Integrator.php`	Business logic, status mapping, rate limiting
`retail-api/app/Models/YodleeProviderAccount.php`	Model with lock mechanism

Current Implementation Flow

User triggers POST → Acquire Redis distributed lock (20 min)
                   → Call Yodlee API: PUT providerAccounts?providerAccountIds={id}
                   → Return 202 (in progress) or 200 (complete)

User polls GET     → Call Yodlee API: GET providerAccounts/{id}
                   → Return action_required based on status
                   → Release lock when complete

Current Characteristics

Aspect	Implementation
Mode	Synchronous polling (no background jobs)
Lock	Redis distributed lock, 20 min timeout
Rate Limit	Max 9 status checks per 120 seconds
Token	JWT RS512, 25 min cache

Identified Problems

No async processing: All operations block request threads
20 min lock timeout: Too long, masks stuck processes
Silent rate limiting: Errors suppressed when rate limit hit, user unaware
User-activity dependent: Yodlee only refreshes on user activity

Potential Solutions

Option 1: Laravel Queue + Polling

Move refresh operation into Laravel Queue, frontend continues polling for results.

Pros:

Does not block HTTP request threads
Leverages existing queue infrastructure

Cons:

Still requires frontend polling
Queue worker still synchronously waits for Yodlee

Option 2: Yodlee Webhook Integration

Use Yodlee’s webhook notification mechanism to avoid polling.