Files
maestro/MIGRATION_MAP_INFACTORY_SIDE.md
T
Claude 177ec6368c DOCS: Map maestro-infactory integration points for migration
Add comprehensive documentation mapping all integration points between
maestro and in-factory services to facilitate removing the dependency.

Includes:
- All 6 gRPC client configurations using INFACTORY_URL
- 27+ gRPC method calls across 6 services
- 25 REST endpoints that proxy to in-factory
- Configuration files and environment variables
- Proto package dependencies
- Migration strategy options and recommendations
2025-12-17 17:09:50 +00:00

9.8 KiB

In-Factory Side - Migration Requirements

This document outlines what needs to be addressed in the In-Factory service to remove its coupling with Maestro.

Note: This analysis is based on the Maestro codebase. For a complete analysis, the In-Factory repository should also be reviewed.


Current Architecture (In-Factory → Maestro)

Based on the Maestro codebase analysis, In-Factory exposes the following gRPC services that Maestro consumes:

graph LR;
  Maestro -->|gRPC| InFactory;
  subgraph InFactory Services
    CM[ConnectionManager]
    ConM[ConnectorManager]
    IN[Input]
    NC[NetworkConfig]
    CT[ConnectionTest]
    TR[Transformation]
  end
  Maestro --> CM;
  Maestro --> ConM;
  Maestro --> IN;
  Maestro --> NC;
  Maestro --> CT;
  Maestro --> TR;

gRPC Services Exposed by In-Factory

1. ConnectionManager Service

Package: ConnectionManager from @dadosfera/protospack-v2

Write Services (ConnectionManagerWriteServices)

gRPC Method Description Called By
CreateConnection Create a new data connection Maestro POST /connections
UpdateConnection Update an existing connection Maestro PUT /connections/:id
DeleteConnection Delete a connection Maestro DELETE /connections/:id

Read Services (ConnectionManagerReadServices)

gRPC Method Description Called By
GetConnectionDetails Get details of a single connection Maestro GET /connections/:id
GetAllConnections List all connections for a customer Maestro GET /connections
GetConnectorAvailableConnectionsByCustomer Get available connections by connector Maestro GET /connections/available/:connector_id
ValidatePlatformConnections Validate connections against platform Maestro internal

2. ConnectorManager Service

Package: ConnectorManager from @dadosfera/protospack-v2

Write Services (ConnectorManagerWriteServices)

gRPC Method Description Called By
RegisterConnector Register a new connector Maestro POST /connectors
UploadFile Upload connector files Maestro POST /connectors/uploads
RegisterMultipleConnectorsWithoutImage Bulk register connectors Maestro internal
UpdateConnectorByID Update a connector Maestro PUT /connectors/:id
DeleteConnectorById Delete a connector Maestro DELETE /connectors/:id

Read Services (ConnectorManagerReadServices)

gRPC Method Description Called By
GetConnectors List all connectors Maestro GET /connectors

3. Input Service

Package: Input from @dadosfera/protospack-v2

Write Services (InputWriteService)

gRPC Method Description Called By
CreateInput Create a new input configuration Maestro POST /inputs
UpdateInput Update input configuration Maestro PATCH /inputs
DeleteInput Delete an input Maestro DELETE /inputs/:id

Read Services (InputReadService)

gRPC Method Description Called By
GetAvailableEntities Get available entities for a plugin Maestro GET /inputs/available-entities/:plugin

4. NetworkConfig Service

Package: NetworkConfig from @dadosfera/protospack-v2

Write Services (NetworkConfigWriteService)

gRPC Method Description Called By
NetworkConfigCreate Create network configuration Maestro POST /network-configs
NetworkConfigUpdate Update network configuration Maestro PUT /network-configs/:id
NetworkConfigDelete Delete network configuration Maestro DELETE /network-configs/:id

Read Services (NetworkConfigReadService)

gRPC Method Description Called By
NetworkConfigFindAll List all network configs Maestro GET /network-configs
NetworkConfigFindOneById Get network config by ID Maestro GET /network-configs/:id

5. ConnectionTest Service

Package: ConnectionTest from @dadosfera/protospack-v2

Read Services (ConnectionTestReadService)

gRPC Method Description Called By
TestConnection Test a connection's connectivity Maestro POST /connection-test

6. Transformation Service

Package: Transformation from @dadosfera/protospack-v2

Write Services (TransformationWriteService)

gRPC Method Description Called By
CreateTransformation Create a transformation Maestro POST /transformations
UpdateTransformation Update a transformation Maestro PUT /transformations/:id
DeleteTransformation Delete a transformation Maestro DELETE /transformations/:id

Read Services (TransformationReadService)

gRPC Method Description Called By
Various read operations Get transformation details Maestro GET /transformations

Proto Package Ownership

The proto definitions are managed in @dadosfera/protospack-v2:

@dadosfera/protospack-v2/
├── dist/lib/
│   ├── ConnectionManager/
│   │   ├── interfaces/messages.ts
│   │   └── interfaces/entities.ts
│   ├── ConnectorManager/
│   │   ├── interfaces/messages.ts
│   │   └── interfaces/entities.ts
│   ├── Input/
│   │   ├── interfaces/messages.ts
│   │   └── interfaces/entities.ts
│   ├── NetworkConfig/
│   │   ├── interfaces/messages.ts
│   │   └── interfaces/entities.ts
│   ├── ConnectionTest/
│   │   ├── interfaces/messages.ts
│   │   └── interfaces/entities.ts
│   └── Transformation/
│       ├── interfaces/messages.ts
│       └── interfaces/entities.ts

In-Factory Migration Options

Option 1: Expose REST API (Keep In-Factory, Add REST Layer)

Changes Required in In-Factory:

  1. Add REST controllers for all services
  2. Implement HTTP endpoints mirroring gRPC methods
  3. Update deployment to expose HTTP port
  4. Create OpenAPI documentation

Pros:

  • Minimal architectural changes
  • Can run both gRPC and REST in parallel during migration
  • Maestro can switch to REST calls gradually

Cons:

  • Still maintains service dependency
  • Adds another communication layer

Option 2: Move Logic to Maestro (Deprecate In-Factory for these features)

Changes Required in In-Factory:

  1. Export database schema/migrations
  2. Document all business logic
  3. Provide data migration scripts
  4. Deprecate gRPC endpoints after migration

Changes Required in Maestro:

  1. Create database models
  2. Implement repositories
  3. Create service layer with same business logic
  4. Run data migration

Pros:

  • Removes runtime dependency completely
  • Simplifies architecture
  • One less service to maintain

Cons:

  • Significant development effort
  • Risk of business logic divergence during migration
  • Database sharing concerns

Option 3: Merge Services (Combine In-Factory into a larger service)

Changes Required:

  1. Create new combined service
  2. Migrate both In-Factory and relevant Maestro code
  3. Update all clients

Pros:

  • Clean architectural redesign
  • Opportunity to optimize

Cons:

  • Largest effort
  • Risk of disruption

Data Migration Considerations

Entities Managed by In-Factory

Based on proto definitions, In-Factory manages:

  1. Connections

    • Connection credentials
    • Connection metadata
    • Customer associations
  2. Connectors

    • Connector definitions
    • Connector images/files
    • Plugin configurations
  3. Inputs

    • Input configurations
    • Entity mappings
  4. Network Configs

    • Network configuration settings
    • Security settings
  5. Transformations

    • Transformation definitions
    • Transformation scripts

Migration Steps

  1. Export database schema from In-Factory
  2. Create equivalent schema in target database
  3. Write data migration scripts
  4. Validate data integrity
  5. Switch traffic
  6. Decommission old service

  1. Check for Maestro dependencies in In-Factory

    • Does In-Factory call any Maestro APIs?
    • Are there any shared databases?
    • Any shared message queues?
  2. Document database schema

    • All tables related to connections, connectors, inputs, network-configs, transformations
    • Foreign key relationships
    • Indexes and constraints
  3. List all consumers

    • Besides Maestro, who else calls In-Factory?
    • Are there other internal services?
    • Any external integrations?
  4. Business logic documentation

    • Validation rules
    • Business constraints
    • Side effects (events, notifications, etc.)

Timeline Considerations

Phase Description Dependencies
Phase 1 Analysis & Planning Both teams available
Phase 2 Schema/API Design Proto definitions finalized
Phase 3 Implementation Development resources
Phase 4 Data Migration Database access, downtime window
Phase 5 Testing QA resources, test environments
Phase 6 Cutover Deployment coordination
Phase 7 Decommission Monitoring, rollback plan

Questions for In-Factory Team

  1. What database does In-Factory use? (PostgreSQL, MongoDB, etc.)
  2. Are there any async operations? (message queues, event sourcing)
  3. What is the current data volume for each entity type?
  4. Are there any scheduled jobs or background processes?
  5. What monitoring/alerting is in place?
  6. Are there any data retention policies?
  7. What is the backup/recovery strategy?