Files
maestro/MIGRATION_MAP_INFACTORY.md
T
Claude 177ec6368c DOCS: Map maestro-infactory integration points for migration
Add comprehensive documentation mapping all integration points between
maestro and in-factory services to facilitate removing the dependency.

Includes:
- All 6 gRPC client configurations using INFACTORY_URL
- 27+ gRPC method calls across 6 services
- 25 REST endpoints that proxy to in-factory
- Configuration files and environment variables
- Proto package dependencies
- Migration strategy options and recommendations
2025-12-17 17:09:50 +00:00

17 KiB

Maestro ↔ In-Factory Migration Map

This document maps all integration points between Maestro and In-Factory that need to be addressed to remove the dependency.

Architecture Overview

graph TD;
  Frontend<-->Maestro;
  Maestro<-->DUC;
  Maestro<-->pi-factory;
  Maestro<-->in-factory;

Maestro connects to 3 external microservices:

  • DUC (DUC_URL) - User management, authentication, permissions
  • PI-Factory (PIFACTORY_URL) - Pipelines, Catalog services
  • IN-Factory (INFACTORY_URL) - Connections, Connectors, Inputs, Network Config, Transformations

Summary Table

Category Count Impact Level
gRPC Client Configurations 6 HIGH
NestJS Modules 6 HIGH
REST Controllers/Endpoints 6 HIGH
Service Classes 6 HIGH
Configuration Files 5 MEDIUM
Proto Package Dependencies 2 HIGH
Environment Variables 2 LOW

1. gRPC CLIENT CONFIGURATIONS (Files to Migrate)

These files configure gRPC connections to In-Factory services:

1.1 Connection Client (INFACTORY_URL)

File: src/modules/connection/client.config.ts

// Lines 10-11, 18
process.env.INFACTORY_URL.startsWith('in-factory:')
process.env.INFACTORY_URL.includes('0.0.0.0')
url: process.env.INFACTORY_URL

Proto Services Used:

  • ConnectionManager.ProtoPackages.WritePackage
  • ConnectionManager.ProtoPackages.ReadPackage

1.2 Connector Client (INFACTORY_URL)

File: src/modules/connector/client.config.ts

// Lines 10-11, 18
url: process.env.INFACTORY_URL

Proto Services Used:

  • ConnectorManager.ProtoPackages.WritePackage
  • ConnectorManager.ProtoPackages.ReadPackage

1.3 Inputs Client (INFACTORY_URL)

File: src/modules/inputs/inputs-client.config.ts

// Lines 10-11, 17
url: process.env.INFACTORY_URL

Proto Services Used:

  • Input.ProtoPackages.WritePackage
  • Input.ProtoPackages.ReadPackage

1.4 Network Config Client (INFACTORY_URL)

File: src/modules/network-config/network-config-client.config.ts

// Lines 10-11, 19
url: process.env.INFACTORY_URL

Proto Services Used:

  • NetworkConfig.ProtoPackages.WritePackage
  • NetworkConfig.ProtoPackages.ReadPackage

1.5 Connection Test Client (INFACTORY_URL)

File: src/modules/connection-test/connection-test-client.config.ts

// Lines 10-11, 17
url: process.env.INFACTORY_URL

Proto Services Used:

  • ConnectionTest.ProtoPackages.ReadPackage

1.6 Transformations Client (INFACTORY_URL)

File: src/modules/transformations/transformations-client.ts

// Lines 10-11, 18
url: process.env.INFACTORY_URL

Proto Services Used:

  • Transformation.ProtoPackages.WritePackage

2. SERVICE CLASSES (Business Logic to Migrate)

2.1 Connection Client Service

File: src/modules/connection/client.service.ts

gRPC Methods Called:

Method Service Direction
CreateConnection() ConnectionManagerWriteServices Write
UpdateConnection() ConnectionManagerWriteServices Write
DeleteConnection() ConnectionManagerWriteServices Write
GetConnectionDetails() ConnectionManagerReadServices Read
GetAllConnections() ConnectionManagerReadServices Read
GetConnectorAvailableConnectionsByCustomer() ConnectionManagerReadServices Read
ValidatePlatformConnections() ConnectionManagerReadServices Read

Key Methods:

  • createConnection()
  • updateConnection()
  • deleteConnection()
  • getConnections()
  • getConnectionDetails()
  • validatePlatformConnections()

2.2 Connector Client Service

File: src/modules/connector/client.service.ts

gRPC Methods Called:

Method Service Direction
RegisterConnector() ConnectorManagerWriteServices Write
UploadFile() ConnectorManagerWriteServices Write
RegisterMultipleConnectorsWithoutImage() ConnectorManagerWriteServices Write
UpdateConnectorByID() ConnectorManagerWriteServices Write
DeleteConnectorById() ConnectorManagerWriteServices Write
GetConnectors() ConnectorManagerReadServices Read

2.3 Inputs Service

File: src/modules/inputs/inputs.service.ts

gRPC Methods Called:

Method Service Direction
CreateInput() InputWriteService Write
UpdateInput() InputWriteService Write
DeleteInput() InputWriteService Write
GetAvailableEntities() InputReadService Read

2.4 Network Config Service

File: src/modules/network-config/network-config.service.ts

gRPC Methods Called:

Method Service Direction
NetworkConfigCreate() NetworkConfigWriteService Write
NetworkConfigUpdate() NetworkConfigWriteService Write
NetworkConfigDelete() NetworkConfigWriteService Write
NetworkConfigFindAll() NetworkConfigReadService Read
NetworkConfigFindOneById() NetworkConfigReadService Read

2.5 Connection Test Service

File: src/modules/connection-test/connection-test.service.ts

gRPC Methods Called:

Method Service Direction
TestConnection() ConnectionTestReadService Read

2.6 Transformations Service

File: src/modules/transformations/client.service.ts

gRPC Methods Called:

Method Service Direction
CreateTransformation() TransformationWriteService Write
UpdateTransformation() TransformationWriteService Write
DeleteTransformation() TransformationWriteService Write
Various read operations TransformationReadService Read

3. NESTJS MODULES (Module Registration)

These modules register the gRPC clients and export services:

Module File Imports Exports
ConnectionModule src/modules/connection/connection.module.ts ConnectionClientConfiguration ConnectionClientService
ConnectorModule src/modules/connector/connector.module.ts ConnectorClientConfiguration ConnectorClientService
InputsModule src/modules/inputs/inputs.module.ts InputsGrpcClient InputsService
NetworkConfigModule src/modules/network-config/network-config.module.ts NetworkConfigGrpcClient NetworkConfigService
ConnectionTestModule src/modules/connection-test/connection-test.module.ts ConnectionTestClientConfiguration ConnectionTestService
TransformationsModule src/modules/transformations/transformations.module.ts TransformationsClientConfiguration TransformationsService

App Module Registration: src/app.module.ts (Lines 15-21, 50-60)


4. REST CONTROLLERS (API Endpoints to Migrate)

4.1 Connection Controller

File: src/modules/connection/connection.controller.ts

HTTP Method Endpoint Description
POST /connections Create a new connection
PUT /connections/:id Update an existing connection
DELETE /connections/:id Delete a connection
GET /connections Get all connections
GET /connections/:id Get connection details
GET /connections/available/:connector_id Get available connections by connector

4.2 Connector Controller

File: src/modules/connector/connector.controller.ts

HTTP Method Endpoint Description
POST /connectors Register a new connector
POST /connectors/uploads Upload multiple connector files
GET /connectors List all connectors
PUT /connectors/:id Update a connector
DELETE /connectors/:id Delete a connector

4.3 Inputs Controller

File: src/modules/inputs/inputs.controller.ts

HTTP Method Endpoint Description
GET /inputs/available-entities/:plugin Get available entities for a plugin
POST /inputs Create an input
PATCH /inputs Update an input
DELETE /inputs/:id Delete an input

4.4 Network Config Controller

File: src/modules/network-config/network-config.controller.ts

HTTP Method Endpoint Description
GET /network-configs Get all network configurations
GET /network-configs/:id Get network config by ID
POST /network-configs Create network configuration
PUT /network-configs/:id Update network configuration
DELETE /network-configs/:id Delete network configuration

4.5 Connection Test Controller

File: src/modules/connection-test/connection-test.controller.ts

HTTP Method Endpoint Description
POST /connection-test Test a connection

4.6 Transformations Controller

File: src/modules/transformations/transformations.controller.ts

HTTP Method Endpoint Description
POST /transformations Create transformation
GET /transformations Get transformations
PUT /transformations/:id Update transformation
DELETE /transformations/:id Delete transformation

5. CONFIGURATION FILES

5.1 Helm Chart Values (Production)

File: deploy/helm-chart/values.yaml

# Lines 32, 42
maestro:
  in_factory_url: in-factory.dadosfera.ai
  tr_factory_url: in-factory.dadosfera.ai  # Transformation factory also uses in-factory

5.2 Helm Chart Values (Staging)

File: deploy/helm-chart/values-stg.yaml

# Lines 5-6
maestro:
  in_factory_url: in-factory.stg.dadosfera.ai
  tr_factory_url: in-factory.stg.dadosfera.ai

5.3 Helmfiles (Production)

File: deploy/helmfiles/prd.yaml

# Lines 15-18, 39-42 (for both maestro and maestro-unimed releases)
maestro.in_factory_url: in-factory.dadosfera.ai
maestro.tr_factory_url: in-factory.dadosfera.ai

5.4 Deployment Template

File: deploy/helm-chart/templates/deployment.yaml

# Lines 71-72
- name: INFACTORY_URL
  value: {{ .Values.maestro.in_factory_url }}

5.5 Environment Type Definition

File: environment.d.ts

// Line 9
INFACTORY_URL: string;

6. PROTO PACKAGE DEPENDENCIES

6.1 Package.json

File: package.json

{
  "@dadosfera/protospack": "2.5.3",
  "@dadosfera/protospack-v2": "3.38.0-beta.14"
}
Import Path Used In
ConnectionManager connection/client.config.ts, connection/client.service.ts
ConnectionManager/interfaces/messages connection/client.service.ts, connection/dtos/connection.ts
ConnectionManager/interfaces/entities connection/dtos/connection.ts
ConnectorManager connector/client.config.ts, connector/client.service.ts
Input inputs/inputs-client.config.ts, inputs/inputs.service.ts
Input/interfaces/messages inputs/inputs.service.ts
Input/interfaces/entities inputs/inputs.controller.ts, inputs/inputs.service.ts
NetworkConfig network-config/network-config-client.config.ts, network-config/network-config.service.ts
NetworkConfig/interfaces/entities network-config/dto/network-config.ts
ConnectionTest connection-test/connection-test-client.config.ts, connection-test/connection-test.service.ts
Transformation transformations/transformations-client.ts, transformations/client.service.ts

7. MIGRATION STRATEGY OPTIONS

Option A: Move In-Factory functionality INTO Maestro

Pros:

  • Single service to maintain
  • No network latency for these operations
  • Simpler deployment

Cons:

  • Increases Maestro's responsibility/complexity
  • Requires database access from Maestro
  • May require significant refactoring

Files to create/migrate:

  1. Database models for connections, connectors, inputs, network-configs, transformations
  2. Repository layer for database operations
  3. Convert gRPC services to internal services
  4. Remove all gRPC client configurations

Option B: Create REST API wrapper in In-Factory

Pros:

  • Minimal changes to Maestro
  • Can migrate incrementally

Cons:

  • Still maintains dependency
  • Additional REST→gRPC translation layer

Option C: Direct database access from Maestro

Pros:

  • Removes runtime dependency
  • Better performance

Cons:

  • Shared database coupling
  • Complex migration

8. FILES TO MODIFY/DELETE (Summary)

High Priority - Core Integration Files

src/modules/connection/client.config.ts          → DELETE or REPLACE
src/modules/connection/client.service.ts         → REPLACE with local implementation
src/modules/connector/client.config.ts           → DELETE or REPLACE
src/modules/connector/client.service.ts          → REPLACE with local implementation
src/modules/inputs/inputs-client.config.ts       → DELETE or REPLACE
src/modules/inputs/inputs.service.ts             → REPLACE with local implementation
src/modules/network-config/network-config-client.config.ts → DELETE or REPLACE
src/modules/network-config/network-config.service.ts      → REPLACE with local implementation
src/modules/connection-test/connection-test-client.config.ts → DELETE or REPLACE
src/modules/connection-test/connection-test.service.ts    → REPLACE with local implementation
src/modules/transformations/transformations-client.ts     → DELETE or REPLACE
src/modules/transformations/client.service.ts            → REPLACE with local implementation

Medium Priority - Module Registration

src/modules/connection/connection.module.ts      → UPDATE imports
src/modules/connector/connector.module.ts        → UPDATE imports
src/modules/inputs/inputs.module.ts              → UPDATE imports
src/modules/network-config/network-config.module.ts → UPDATE imports
src/modules/connection-test/connection-test.module.ts → UPDATE imports
src/modules/transformations/transformations.module.ts → UPDATE imports
src/app.module.ts                                → UPDATE if module structure changes

Low Priority - Configuration

deploy/helm-chart/values.yaml                    → REMOVE in_factory_url, tr_factory_url
deploy/helm-chart/values-stg.yaml                → REMOVE in_factory_url, tr_factory_url
deploy/helmfiles/prd.yaml                        → REMOVE in_factory_url references
deploy/helm-chart/templates/deployment.yaml      → REMOVE INFACTORY_URL env var
environment.d.ts                                 → REMOVE INFACTORY_URL type
README.md                                        → UPDATE architecture diagram

9. DEPENDENCY COUNT BY MODULE

Module Files gRPC Calls REST Endpoints
Connection 4 7 6
Connector 3 6 5
Inputs 3 4 4
Network Config 3 5 5
Connection Test 3 1 1
Transformations 4 4+ 4
TOTAL 20 27+ 25

10. DATA MODELS (Proto Messages Used)

Connection Manager

  • CreateConnectionRequest / CreateConnectionResponse
  • UpdateConnectionRequest / UpdateConnectionResponse
  • DeleteConnectionRequest / DeleteConnectionResponse
  • GetConnectionDetailsRequest / GetConnectionDetailsResponse
  • GetAllConnectionsRequest / GetAllConnectionsResponse
  • Connection (entity)
  • ConnectionCredential (entity)

Connector Manager

  • RegisterConnectorRequest / RegisterConnectorResponse
  • UploadFileRequest / UploadFileResponse
  • GetConnectorsRequest / GetConnectorsResponse
  • Connector (entity)

Input

  • CreateInputRequest / CreateInputResponse
  • UpdateInputRequest / UpdateInputResponse
  • DeleteInputRequest / DeleteInputResponse
  • GetAvailableEntitiesRequest / GetAvailableEntitiesResponse
  • Info (entity)

Network Config

  • NetworkConfigCreateRequest / NetworkConfigCreateResponse
  • NetworkConfigUpdateRequest / NetworkConfigUpdateResponse
  • NetworkConfigDeleteRequest / NetworkConfigDeleteResponse
  • NetworkConfigFindAllRequest / NetworkConfigFindAllResponse
  • NetworkConfig (entity)

Connection Test

  • TestConnectionRequest / TestConnectionResponse

Transformation

  • CreateTransformationRequest / CreateTransformationResponse
  • UpdateTransformationRequest / UpdateTransformationResponse
  • DeleteTransformationRequest / DeleteTransformationResponse

NEXT STEPS

  1. Decide on migration strategy (Option A, B, or C)
  2. Prioritize modules - Recommend starting with Connection Test (smallest), then Inputs, Network Config, Transformations, Connection, Connector (largest)
  3. Create database schema if moving to Option A
  4. Implement local services one module at a time
  5. Update tests for each migrated module
  6. Update deployment configs to remove INFACTORY_URL
  7. Coordinate with In-Factory team for data migration