Integration Architecture Overview

This document provides an overview of the integration module in retail-api, connecting 46+ external financial platforms to synchronize accounts, holdings, and transaction data.

System Positioning

Responsibilities:

Connect to external financial platforms (brokerages, custodians, CRMs, aggregators)
Synchronize account data, holdings, transactions, and insurance policies
Map external entities to internal Households, Accounts, and Persons

Code Locations:

/retail-api/app/Integrations/ - 46+ vendor implementations
/retail-api/app/Http/Controllers/Advisors/Integrations/ - REST API endpoints
/packages/libs/integrations-core/ - Core abstractions (Integrator, EntityProvider, Exceptions)
/packages/libs/integrations-file-based/ - Shared file-based integration logic

Two Integration Types

Dimension	API-based	File-based
Data Source	Real-time external API calls	User-uploaded files (CSV/Excel)
Authentication	OAuth 2.0 / JWT / API Key / SOAP	None (file validation only)
Sync Trigger	Scheduled + Webhook + Manual	Manual upload only
Complexity	High (state machine, error recovery)	Medium (parsing, mapping)
Count	27 integrations	19 integrations
Examples	Yodlee, Schwab API, Orion, Addepar	Fidelity, Pershing, LPL, Schwab (file)

Note: Some vendors have both API and file-based versions (e.g., Schwab, LPL, Betterment).

Core Data Flow

┌─────────────────────────────────────────────────────────────────────────────┐
│                              User Action                                     │
│                    (Link Account / Upload File / Sync)                       │
└─────────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                           Controller Layer                                   │
│   EntityController | IntegrationMappingController | SyncController           │
│                    #[IntegrationScenario(Scenario::*)]                       │
└─────────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                            Job Queue (Async)                                 │
│              Dispatch sync jobs for background processing                    │
└─────────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                          Integrator Layer                                    │
│     Integration::getIntegratorFqcn() → App\Integrations\[Vendor]\Integrator │
│                                                                              │
│     API-based: listEntities(), sync(), syncAll()                            │
│     File-based: parse(), map(), import()                                    │
└─────────────────────────────────────────────────────────────────────────────┘
                                    │
                    ┌───────────────┴───────────────┐
                    ▼                               ▼
┌───────────────────────────────┐   ┌───────────────────────────────┐
│     Connector (API-based)      │   │     Parser (File-based)        │
│  HttpConnector | OAuthConnector│   │  CSV/Excel parsing & mapping   │
│       SoapConnector            │   │                                 │
└───────────────────────────────┘   └───────────────────────────────┘
                    │                               │
                    ▼                               ▼
┌───────────────────────────────┐   ┌───────────────────────────────┐
│       External API             │   │          S3 Storage            │
│   (Vendor REST/SOAP/OAuth)     │   │    (Uploaded files)            │
└───────────────────────────────┘   └───────────────────────────────┘
                    │                               │
                    └───────────────┬───────────────┘
                                    ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                         Vendor Models Layer                                  │
│           Parse response → Vendor-specific Account/Holding models           │
└─────────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                        Internal Models Layer                                 │
│    Map to: Household | Account | Insurance | Person | TargetCategoryMix     │
│                   Create/Update IntegrationMapping                          │
└─────────────────────────────────────────────────────────────────────────────┘

Code Structure

Per-Vendor Implementation Pattern

app/Integrations/
├── [Vendor]/
│   ├── Api.php              # API client wrapper (optional)
│   ├── Connector.php        # Transport: HTTP/OAuth/SOAP
│   ├── Integrator.php       # Core sync logic
│   ├── Importer.php         # Household import logic
│   ├── Models/              # Vendor-specific data models
│   │   ├── Account.php
│   │   ├── Holding.php
│   │   └── ...
│   ├── Exceptions/          # Vendor-specific exceptions
│   └── OAuth2/              # OAuth config (if OAuth-based)
└── Support/
    ├── Scenario.php                    # 10 integration scenarios
    ├── Developer.php                   # Developer ownership map
    ├── LegacyApiBased/
    │   ├── Connectors/                 # Base connector classes
    │   │   ├── HttpConnectorWithIntegration.php
    │   │   ├── OauthConnectorWithIntegration.php
    │   │   └── SoapConnectorWithIntegration.php
    │   ├── Integrators/
    │   └── Exceptions/
    └── ApiBased/                       # Modern API-based infrastructure
        ├── Connectors/
        └── Integrators/

Shared Packages

Package	Location	Purpose
`integrations-core`	`/packages/libs/integrations-core/`	Base classes: Integrator, EntityProvider, CursorPage, Exceptions
`integrations-file-based`	`/packages/libs/integrations-file-based/`	Shared file parsing and mapping logic
`core-models`	`/packages/libs/core-models/`	IntegrationType enum, base models

Key Concepts

Integration

An instance of a configured integration for an advisor.

// Key fields
$integration->type;              // IntegrationType enum (46 types)
$integration->reference;         // Vendor-specific identifier
$integration->credentials;       // Encrypted JSON credentials
$integration->failed_since;      // When sync started failing
$integration->failed_biz_days;   // Consecutive failure days

// Key methods
$integration->getIntegratorFqcn();   // Get Integrator class
$integration->getConnectorFqcn();    // Get Connector class
$integration->getSynchronizer();     // Get Sync handler

IntegrationMapping

Links an external entity (vendor account) to an internal object (Household, Account, etc.).

// Key fields
$mapping->integration_id;     // Parent Integration
$mapping->parent_id;          // Hierarchical parent (for nested mappings)
$mapping->mappable_type;      // HOUSEHOLD | ACCOUNT | INSURANCE | PERSON | CONTACT | TARGET_CATEGORY_MIX
$mapping->mappable_id;        // ID of mapped internal object
$mapping->reference;          // Vendor-specific entity ID
$mapping->sync_status;        // Current sync status
$mapping->last_completed_at;  // Last successful sync

// Key methods
$mapping->sync();             // Synchronize this mapping

Integrator

Core class implementing sync logic for each vendor.

abstract class Integrator
{
    abstract public function syncAll(): void;
    abstract public static function sync(IntegrationMapping $mapping): array|null;
    abstract protected static function getVendor(): IntegrationType;
}

Connector

Transport layer handling API communication.

Type	Class	Use Case
HTTP	`HttpConnectorWithIntegration`	REST APIs with API Key/Token
OAuth	`OauthConnectorWithIntegration`	OAuth 2.0 flow (Schwab, Addepar, Orion)
SOAP	`SoapConnectorWithIntegration`	Legacy SOAP services

EntityProvider

Interface for listing available entities from external service.

interface EntityProvider
{
    public function listEntities(array $conditions, ?string $cursor = null): CursorPage;
}

Integration Scenarios

Defined in Support/Scenario.php, used to track operation context:

Scenario	Description
`CREATE_INTEGRATION`	Initial integration setup
`LINK_ENTITY`	Link vendor entity to household
`LIST_ENTITIES`	Discover available accounts
`SYNC_HOUSEHOLD`	Sync single household
`SYNC_INTEGRATION`	Sync all accounts for integration
`NIGHTLY_SYNC`	Scheduled background sync
`IMPORT_HOUSEHOLD`	Bulk household import
`IMPORT_CLIENTS`	Import client data
`LIST_TAGS`	Get CRM tags/categories
`SEND_PDF`	Document delivery

API Endpoints

GET    /advisors/{advisor}/integrations/{integration}/entities
       List available entities from vendor (with search, cursor pagination)

GET    /advisors/{advisor}/integrations/{integration}/mappings
       List linked accounts with household info

POST   /advisors/{advisor}/integrations/{integration}/mappings
       Link vendor entity to household

DELETE /advisors/{advisor}/integrations/{integration}/mappings/{mapping}
       Unlink entity (soft delete)

POST   /advisors/{advisor}/integrations/{integration}/mappings/{mapping}/sync
       Sync single household mapping

POST   /advisors/{advisor}/integrations/{integration}/sync
       Sync entire integration

GET    /advisors/{advisor}/integrations/{integration}/tags
       List household tags from CRM integration

All 46 Integration Types

API-based (27)

Vendor	Auth Type	Developer
Addepar	OAuth 2.0	Winston Li
Advyzon	OAuth	Kewei Yan
Albridge	API	Tingsong Xu
Allianz API	OAuth	Yan Hu
Asset Book	OAuth	Qianwei Hao
Asset Mark	API	Qianwei Hao
Black Diamond	API	Qianwei Hao
Blueleaf	HTTP	Tingsong Xu
Bridge FT	API	Yan Hu
Capitect	OAuth	Yan Hu
Circle Black	API	Tingsong Xu
Commonwealth	HTTP	Tingsong Xu
DST	API	Yan Hu
FinFolio	API	Yan Hu
Investigo	API	Kewei Yan
LPL (API)	API	Qianwei Hao
Max My Interest	API	Winston Li
Morningstar Advisor	API	Qianwei Hao
Morningstar Office	API	Winston Li
Nationwide	API	Winston Li
Orion	OAuth	Kewei Yan
Panoramix	OAuth	Tingsong Xu
Redtail	API	Qianwei Hao
Riskalyze	API	Tingsong Xu
Schwab API	OAuth 2.0	Yan Hu
Smart Office	API	Winston Li
Tamarac	SOAP	Tingsong Xu
Wealth Access	API	Yan Hu
Wealthbox	API Key	Kewei Yan

File-based (19)

Vendor	File Type	Developer
Allianz	File	Yan Hu
Altruist	File	Yan Hu
Apex	File	Tingsong Xu
Betterment	File	Kewei Yan
Fidelity	CSV	Qianwei Hao
First Clearing	File	Qianwei Hao
Flourish	File	Tingsong Xu
Folio Investing	File	Qianwei Hao
Interactive Brokers	File	Winston Li
Jackson	File	Winston Li
LPL (File)	File	Qianwei Hao
My529	File	Kewei Yan
Pacific Life	File	Winston Li
Pershing	Excel	Kewei Yan
Raymond James	File	Winston Li
RBC	File	Yan Hu
Schwab (File)	CSV	Yan Hu
SEI	File	Winston Li
Trust America	File	Kewei Yan

Next Steps

API-based Patterns - Connector design, authentication patterns
API-based Sync Lifecycle - Job flow, error handling
API-based Vendor Specifics - Yodlee, Schwab details
File-based Patterns - Parser, mapper design
File-based Vendor Specifics - Format details
Technical Debt - Known issues and improvement opportunities