Technical Debt & Improvement Opportunities

This document identifies architectural pain points in the integration system and proposes improvement directions. It serves as a living document to track known issues and refactoring priorities.

Identified Issues from Code Analysis

1. Legacy vs Modern Architecture Split

Location: app/Integrations/Support/

Problem: Two parallel infrastructures exist:

• LegacyApiBased/ - Older integrations
• ApiBased/ - Newer integrations

Support/
├── LegacyApiBased/
│   ├── Connectors/
│   │   ├── HttpConnectorWithIntegration.php
│   │   ├── OauthConnectorWithIntegration.php
│   │   └── SoapConnectorWithIntegration.php
│   ├── Integrators/
│   └── Exceptions/
└── ApiBased/
    ├── Connectors/
    └── Integrators/

Impact:

• Maintenance burden: changes may need to be made in both places
• Confusion for new developers: which pattern to follow?
• Inconsistent behavior between old and new integrations

Recommendation:

• Document which integrations use which infrastructure
• For new integrations, always use ApiBased/
• Gradually migrate legacy integrations as they need updates
• Consider unifying into single infrastructure with backward compatibility

---

2. Yodlee Complexity

Location: app/Integrations/Yodlee/Integrator.php (~1100+ lines)

Problem:

• Single file with 1100+ lines of complex state machine logic
• 30+ status codes with intricate mapping rules
• Hard to test individual paths
• Changes risk breaking other status handling

Impact:

• High risk for bug fixes
• Difficult onboarding for new developers
• Low test coverage due to complexity

Recommendation:

• Extract status mapping to dedicated StatusMapper class
• Extract user action logic to UserActionResolver class
• Add comprehensive unit tests for each status code
• Consider state machine library (e.g., winzou/state-machine)

// Proposed structure
Integrations/Yodlee/
├── Integrator.php           # Simplified, delegates to components
├── StatusMapper.php         # Maps vendor status to internal status
├── UserActionResolver.php   # Determines required user action
├── StateHandler/
│   ├── SuccessHandler.php
│   ├── AuthErrorHandler.php
│   └── TempFailureHandler.php
└── Tests/
    ├── StatusMapperTest.php
    └── UserActionResolverTest.php

---

3. Inconsistent Exception Handling

Problem: Different vendors handle exceptions differently:

• Some throw specific exceptions
• Some return error arrays
• Some silently fail
• Error messages vary in quality

Impact:

• Inconsistent user experience
• Difficult to build unified error monitoring
• Hard to know when to retry vs. fail permanently

Recommendation:

• Enforce use of integrations-core exception hierarchy
• Create vendor-agnostic error codes
• Standardize user-facing error messages
• Add exception mapping layer per vendor

// Proposed: Vendor exception wrapper
class VendorExceptionMapper
{
    public static function wrap(Throwable $e, Integration $integration): ExternalServiceException
    {
        return match(true) {
            $e instanceof VendorUnauthorized => new UnauthorizedException($integration, $e),
            $e instanceof VendorNotFound => new NotFoundException($integration, $e),
            $e instanceof VendorRateLimit => new TooManyRequestException($integration, $e),
            default => new UnrecognizedException($integration, $e),
        };
    }
}

---

4. Missing Unified Abstractions

Problem: Each vendor implements sync logic slightly differently:

• No consistent interface for “sync a household”
• No consistent interface for “list accounts”
• EntityProvider interface exists but not universally implemented

Impact:

• Adding new features requires touching many integrations
• Can’t build unified monitoring/debugging tools
• Hard to implement cross-vendor features

Recommendation:

• Require all integrations to implement EntityProvider
• Create SyncResult DTO for consistent return values
• Add IntegrationCapabilities interface to declare features

// Proposed interfaces
interface IntegrationCapabilities
{
    public function supportsListEntities(): bool;
    public function supportsSync(): bool;
    public function supportsTags(): bool;
    public function supportsWebhook(): bool;
}

class SyncResult
{
    public function __construct(
        public array $created,
        public array $updated,
        public array $deleted,
        public array $errors,
        public Carbon $completedAt,
    ) {}
}

---

5. Lack of Integration Testing

Problem:

• Most integrations lack automated tests
• Manual testing required for each vendor
• Regressions discovered in production
• No mock/sandbox environments documented

Impact:

• Fear of refactoring
• Long release cycles
• Production incidents

Recommendation:

• Create test fixtures with sample API responses
• Implement contract tests for each vendor
• Document sandbox environments where available
• Add integration test suite to CI pipeline

// Example test structure
tests/
└── Integrations/
    ├── Yodlee/
    │   ├── IntegratorTest.php
    │   ├── ConnectorTest.php
    │   └── Fixtures/
    │       ├── success_response.json
    │       ├── auth_error_response.json
    │       └── rate_limit_response.json
    ├── SchwabApi/
    │   └── ...
    └── Fidelity/
        └── ...

---

6. Configuration Scattered

Problem: Integration configuration lives in multiple places:

• Integration model credentials JSON
• Integration model settings JSON
• Environment variables
• Hardcoded in Connector classes
• Some in Config.php files

Impact:

• Hard to audit all configuration
• Easy to miss configuration during setup
• Inconsistent handling of secrets

Recommendation:

• Centralize configuration schema per integration
• Use Config.php consistently for all config
• Separate secrets (credentials) from settings
• Add configuration validation

---

7. Memory-Intensive Operations

Problem: Some operations allocate 2GB+ memory:

ini_set('memory_limit', '2048M');

Impact:

• Resource-intensive
• Can cause OOM in containerized environments
• Slow processing

Recommendation:

• Implement streaming/chunked processing
• Use cursor-based pagination consistently
• Process in batches with explicit memory cleanup
• Consider moving heavy operations to dedicated workers

---

Known Team Pain Points

(To be collected from team)

• Add pain points from team discussions
• Include specific incidents/bugs
• Note recurring support tickets

---

Improvement Priority Matrix

Priority	Improvement	Impact	Effort	Dependencies
P0	Add integration tests	High	Medium	Test fixtures
P0	Document sandbox environments	High	Low	Vendor docs
P1	Extract Yodlee state machine	High	High	None
P1	Standardize exception handling	Medium	Medium	None
P2	Unify Legacy/Modern infrastructure	Medium	High	Migration plan
P2	Implement streaming for large data	Medium	Medium	Architecture review
P3	Centralize configuration	Low	Medium	None

---

Refactoring Principles

When addressing technical debt:

1. Incremental Improvement

   • No big-bang rewrites
   • Each PR should be self-contained and deployable
   • Leave code better than you found it

2. New Integrations Use New Patterns

   • Always use ApiBased/ for new integrations
   • Follow latest interfaces and patterns
   • Add tests from the start

3. Migrate on Touch

   • When fixing a bug in legacy integration, consider migrating
   • When adding features, update to new patterns
   • Document migration path for each integration

4. Test First

   • Before refactoring, add tests for current behavior
   • Tests become safety net for changes
   • Aim for 80%+ coverage on refactored code

5. Document Decisions

   • Record why changes were made in ADRs
   • Update this document as debt is addressed
   • Share learnings with team

---

Progress Tracking

Date	Improvement	Status	Notes
-	-	-	-

(Update this table as improvements are made)

---

Related Documentation

• Architecture Overview - System overview
• API-based Patterns - Current patterns
• File-based Patterns - Current patterns