Add comprehensive documentation mapping all integration points between maestro and in-factory services to facilitate removing the dependency. Includes: - All 6 gRPC client configurations using INFACTORY_URL - 27+ gRPC method calls across 6 services - 25 REST endpoints that proxy to in-factory - Configuration files and environment variables - Proto package dependencies - Migration strategy options and recommendations
17 KiB
Maestro ↔ In-Factory Migration Map
This document maps all integration points between Maestro and In-Factory that need to be addressed to remove the dependency.
Architecture Overview
graph TD;
Frontend<-->Maestro;
Maestro<-->DUC;
Maestro<-->pi-factory;
Maestro<-->in-factory;
Maestro connects to 3 external microservices:
- DUC (
DUC_URL) - User management, authentication, permissions - PI-Factory (
PIFACTORY_URL) - Pipelines, Catalog services - IN-Factory (
INFACTORY_URL) - Connections, Connectors, Inputs, Network Config, Transformations
Summary Table
| Category | Count | Impact Level |
|---|---|---|
| gRPC Client Configurations | 6 | HIGH |
| NestJS Modules | 6 | HIGH |
| REST Controllers/Endpoints | 6 | HIGH |
| Service Classes | 6 | HIGH |
| Configuration Files | 5 | MEDIUM |
| Proto Package Dependencies | 2 | HIGH |
| Environment Variables | 2 | LOW |
1. gRPC CLIENT CONFIGURATIONS (Files to Migrate)
These files configure gRPC connections to In-Factory services:
1.1 Connection Client (INFACTORY_URL)
File: src/modules/connection/client.config.ts
// Lines 10-11, 18
process.env.INFACTORY_URL.startsWith('in-factory:')
process.env.INFACTORY_URL.includes('0.0.0.0')
url: process.env.INFACTORY_URL
Proto Services Used:
ConnectionManager.ProtoPackages.WritePackageConnectionManager.ProtoPackages.ReadPackage
1.2 Connector Client (INFACTORY_URL)
File: src/modules/connector/client.config.ts
// Lines 10-11, 18
url: process.env.INFACTORY_URL
Proto Services Used:
ConnectorManager.ProtoPackages.WritePackageConnectorManager.ProtoPackages.ReadPackage
1.3 Inputs Client (INFACTORY_URL)
File: src/modules/inputs/inputs-client.config.ts
// Lines 10-11, 17
url: process.env.INFACTORY_URL
Proto Services Used:
Input.ProtoPackages.WritePackageInput.ProtoPackages.ReadPackage
1.4 Network Config Client (INFACTORY_URL)
File: src/modules/network-config/network-config-client.config.ts
// Lines 10-11, 19
url: process.env.INFACTORY_URL
Proto Services Used:
NetworkConfig.ProtoPackages.WritePackageNetworkConfig.ProtoPackages.ReadPackage
1.5 Connection Test Client (INFACTORY_URL)
File: src/modules/connection-test/connection-test-client.config.ts
// Lines 10-11, 17
url: process.env.INFACTORY_URL
Proto Services Used:
ConnectionTest.ProtoPackages.ReadPackage
1.6 Transformations Client (INFACTORY_URL)
File: src/modules/transformations/transformations-client.ts
// Lines 10-11, 18
url: process.env.INFACTORY_URL
Proto Services Used:
Transformation.ProtoPackages.WritePackage
2. SERVICE CLASSES (Business Logic to Migrate)
2.1 Connection Client Service
File: src/modules/connection/client.service.ts
gRPC Methods Called:
| Method | Service | Direction |
|---|---|---|
CreateConnection() |
ConnectionManagerWriteServices | Write |
UpdateConnection() |
ConnectionManagerWriteServices | Write |
DeleteConnection() |
ConnectionManagerWriteServices | Write |
GetConnectionDetails() |
ConnectionManagerReadServices | Read |
GetAllConnections() |
ConnectionManagerReadServices | Read |
GetConnectorAvailableConnectionsByCustomer() |
ConnectionManagerReadServices | Read |
ValidatePlatformConnections() |
ConnectionManagerReadServices | Read |
Key Methods:
createConnection()updateConnection()deleteConnection()getConnections()getConnectionDetails()validatePlatformConnections()
2.2 Connector Client Service
File: src/modules/connector/client.service.ts
gRPC Methods Called:
| Method | Service | Direction |
|---|---|---|
RegisterConnector() |
ConnectorManagerWriteServices | Write |
UploadFile() |
ConnectorManagerWriteServices | Write |
RegisterMultipleConnectorsWithoutImage() |
ConnectorManagerWriteServices | Write |
UpdateConnectorByID() |
ConnectorManagerWriteServices | Write |
DeleteConnectorById() |
ConnectorManagerWriteServices | Write |
GetConnectors() |
ConnectorManagerReadServices | Read |
2.3 Inputs Service
File: src/modules/inputs/inputs.service.ts
gRPC Methods Called:
| Method | Service | Direction |
|---|---|---|
CreateInput() |
InputWriteService | Write |
UpdateInput() |
InputWriteService | Write |
DeleteInput() |
InputWriteService | Write |
GetAvailableEntities() |
InputReadService | Read |
2.4 Network Config Service
File: src/modules/network-config/network-config.service.ts
gRPC Methods Called:
| Method | Service | Direction |
|---|---|---|
NetworkConfigCreate() |
NetworkConfigWriteService | Write |
NetworkConfigUpdate() |
NetworkConfigWriteService | Write |
NetworkConfigDelete() |
NetworkConfigWriteService | Write |
NetworkConfigFindAll() |
NetworkConfigReadService | Read |
NetworkConfigFindOneById() |
NetworkConfigReadService | Read |
2.5 Connection Test Service
File: src/modules/connection-test/connection-test.service.ts
gRPC Methods Called:
| Method | Service | Direction |
|---|---|---|
TestConnection() |
ConnectionTestReadService | Read |
2.6 Transformations Service
File: src/modules/transformations/client.service.ts
gRPC Methods Called:
| Method | Service | Direction |
|---|---|---|
CreateTransformation() |
TransformationWriteService | Write |
UpdateTransformation() |
TransformationWriteService | Write |
DeleteTransformation() |
TransformationWriteService | Write |
| Various read operations | TransformationReadService | Read |
3. NESTJS MODULES (Module Registration)
These modules register the gRPC clients and export services:
| Module | File | Imports | Exports |
|---|---|---|---|
| ConnectionModule | src/modules/connection/connection.module.ts |
ConnectionClientConfiguration | ConnectionClientService |
| ConnectorModule | src/modules/connector/connector.module.ts |
ConnectorClientConfiguration | ConnectorClientService |
| InputsModule | src/modules/inputs/inputs.module.ts |
InputsGrpcClient | InputsService |
| NetworkConfigModule | src/modules/network-config/network-config.module.ts |
NetworkConfigGrpcClient | NetworkConfigService |
| ConnectionTestModule | src/modules/connection-test/connection-test.module.ts |
ConnectionTestClientConfiguration | ConnectionTestService |
| TransformationsModule | src/modules/transformations/transformations.module.ts |
TransformationsClientConfiguration | TransformationsService |
App Module Registration: src/app.module.ts (Lines 15-21, 50-60)
4. REST CONTROLLERS (API Endpoints to Migrate)
4.1 Connection Controller
File: src/modules/connection/connection.controller.ts
| HTTP Method | Endpoint | Description |
|---|---|---|
| POST | /connections |
Create a new connection |
| PUT | /connections/:id |
Update an existing connection |
| DELETE | /connections/:id |
Delete a connection |
| GET | /connections |
Get all connections |
| GET | /connections/:id |
Get connection details |
| GET | /connections/available/:connector_id |
Get available connections by connector |
4.2 Connector Controller
File: src/modules/connector/connector.controller.ts
| HTTP Method | Endpoint | Description |
|---|---|---|
| POST | /connectors |
Register a new connector |
| POST | /connectors/uploads |
Upload multiple connector files |
| GET | /connectors |
List all connectors |
| PUT | /connectors/:id |
Update a connector |
| DELETE | /connectors/:id |
Delete a connector |
4.3 Inputs Controller
File: src/modules/inputs/inputs.controller.ts
| HTTP Method | Endpoint | Description |
|---|---|---|
| GET | /inputs/available-entities/:plugin |
Get available entities for a plugin |
| POST | /inputs |
Create an input |
| PATCH | /inputs |
Update an input |
| DELETE | /inputs/:id |
Delete an input |
4.4 Network Config Controller
File: src/modules/network-config/network-config.controller.ts
| HTTP Method | Endpoint | Description |
|---|---|---|
| GET | /network-configs |
Get all network configurations |
| GET | /network-configs/:id |
Get network config by ID |
| POST | /network-configs |
Create network configuration |
| PUT | /network-configs/:id |
Update network configuration |
| DELETE | /network-configs/:id |
Delete network configuration |
4.5 Connection Test Controller
File: src/modules/connection-test/connection-test.controller.ts
| HTTP Method | Endpoint | Description |
|---|---|---|
| POST | /connection-test |
Test a connection |
4.6 Transformations Controller
File: src/modules/transformations/transformations.controller.ts
| HTTP Method | Endpoint | Description |
|---|---|---|
| POST | /transformations |
Create transformation |
| GET | /transformations |
Get transformations |
| PUT | /transformations/:id |
Update transformation |
| DELETE | /transformations/:id |
Delete transformation |
5. CONFIGURATION FILES
5.1 Helm Chart Values (Production)
File: deploy/helm-chart/values.yaml
# Lines 32, 42
maestro:
in_factory_url: in-factory.dadosfera.ai
tr_factory_url: in-factory.dadosfera.ai # Transformation factory also uses in-factory
5.2 Helm Chart Values (Staging)
File: deploy/helm-chart/values-stg.yaml
# Lines 5-6
maestro:
in_factory_url: in-factory.stg.dadosfera.ai
tr_factory_url: in-factory.stg.dadosfera.ai
5.3 Helmfiles (Production)
File: deploy/helmfiles/prd.yaml
# Lines 15-18, 39-42 (for both maestro and maestro-unimed releases)
maestro.in_factory_url: in-factory.dadosfera.ai
maestro.tr_factory_url: in-factory.dadosfera.ai
5.4 Deployment Template
File: deploy/helm-chart/templates/deployment.yaml
# Lines 71-72
- name: INFACTORY_URL
value: {{ .Values.maestro.in_factory_url }}
5.5 Environment Type Definition
File: environment.d.ts
// Line 9
INFACTORY_URL: string;
6. PROTO PACKAGE DEPENDENCIES
6.1 Package.json
File: package.json
{
"@dadosfera/protospack": "2.5.3",
"@dadosfera/protospack-v2": "3.38.0-beta.14"
}
6.2 Proto Imports from @dadosfera/protospack-v2 (In-Factory related)
| Import Path | Used In |
|---|---|
ConnectionManager |
connection/client.config.ts, connection/client.service.ts |
ConnectionManager/interfaces/messages |
connection/client.service.ts, connection/dtos/connection.ts |
ConnectionManager/interfaces/entities |
connection/dtos/connection.ts |
ConnectorManager |
connector/client.config.ts, connector/client.service.ts |
Input |
inputs/inputs-client.config.ts, inputs/inputs.service.ts |
Input/interfaces/messages |
inputs/inputs.service.ts |
Input/interfaces/entities |
inputs/inputs.controller.ts, inputs/inputs.service.ts |
NetworkConfig |
network-config/network-config-client.config.ts, network-config/network-config.service.ts |
NetworkConfig/interfaces/entities |
network-config/dto/network-config.ts |
ConnectionTest |
connection-test/connection-test-client.config.ts, connection-test/connection-test.service.ts |
Transformation |
transformations/transformations-client.ts, transformations/client.service.ts |
7. MIGRATION STRATEGY OPTIONS
Option A: Move In-Factory functionality INTO Maestro
Pros:
- Single service to maintain
- No network latency for these operations
- Simpler deployment
Cons:
- Increases Maestro's responsibility/complexity
- Requires database access from Maestro
- May require significant refactoring
Files to create/migrate:
- Database models for connections, connectors, inputs, network-configs, transformations
- Repository layer for database operations
- Convert gRPC services to internal services
- Remove all gRPC client configurations
Option B: Create REST API wrapper in In-Factory
Pros:
- Minimal changes to Maestro
- Can migrate incrementally
Cons:
- Still maintains dependency
- Additional REST→gRPC translation layer
Option C: Direct database access from Maestro
Pros:
- Removes runtime dependency
- Better performance
Cons:
- Shared database coupling
- Complex migration
8. FILES TO MODIFY/DELETE (Summary)
High Priority - Core Integration Files
src/modules/connection/client.config.ts → DELETE or REPLACE
src/modules/connection/client.service.ts → REPLACE with local implementation
src/modules/connector/client.config.ts → DELETE or REPLACE
src/modules/connector/client.service.ts → REPLACE with local implementation
src/modules/inputs/inputs-client.config.ts → DELETE or REPLACE
src/modules/inputs/inputs.service.ts → REPLACE with local implementation
src/modules/network-config/network-config-client.config.ts → DELETE or REPLACE
src/modules/network-config/network-config.service.ts → REPLACE with local implementation
src/modules/connection-test/connection-test-client.config.ts → DELETE or REPLACE
src/modules/connection-test/connection-test.service.ts → REPLACE with local implementation
src/modules/transformations/transformations-client.ts → DELETE or REPLACE
src/modules/transformations/client.service.ts → REPLACE with local implementation
Medium Priority - Module Registration
src/modules/connection/connection.module.ts → UPDATE imports
src/modules/connector/connector.module.ts → UPDATE imports
src/modules/inputs/inputs.module.ts → UPDATE imports
src/modules/network-config/network-config.module.ts → UPDATE imports
src/modules/connection-test/connection-test.module.ts → UPDATE imports
src/modules/transformations/transformations.module.ts → UPDATE imports
src/app.module.ts → UPDATE if module structure changes
Low Priority - Configuration
deploy/helm-chart/values.yaml → REMOVE in_factory_url, tr_factory_url
deploy/helm-chart/values-stg.yaml → REMOVE in_factory_url, tr_factory_url
deploy/helmfiles/prd.yaml → REMOVE in_factory_url references
deploy/helm-chart/templates/deployment.yaml → REMOVE INFACTORY_URL env var
environment.d.ts → REMOVE INFACTORY_URL type
README.md → UPDATE architecture diagram
9. DEPENDENCY COUNT BY MODULE
| Module | Files | gRPC Calls | REST Endpoints |
|---|---|---|---|
| Connection | 4 | 7 | 6 |
| Connector | 3 | 6 | 5 |
| Inputs | 3 | 4 | 4 |
| Network Config | 3 | 5 | 5 |
| Connection Test | 3 | 1 | 1 |
| Transformations | 4 | 4+ | 4 |
| TOTAL | 20 | 27+ | 25 |
10. DATA MODELS (Proto Messages Used)
Connection Manager
CreateConnectionRequest/CreateConnectionResponseUpdateConnectionRequest/UpdateConnectionResponseDeleteConnectionRequest/DeleteConnectionResponseGetConnectionDetailsRequest/GetConnectionDetailsResponseGetAllConnectionsRequest/GetAllConnectionsResponseConnection(entity)ConnectionCredential(entity)
Connector Manager
RegisterConnectorRequest/RegisterConnectorResponseUploadFileRequest/UploadFileResponseGetConnectorsRequest/GetConnectorsResponseConnector(entity)
Input
CreateInputRequest/CreateInputResponseUpdateInputRequest/UpdateInputResponseDeleteInputRequest/DeleteInputResponseGetAvailableEntitiesRequest/GetAvailableEntitiesResponseInfo(entity)
Network Config
NetworkConfigCreateRequest/NetworkConfigCreateResponseNetworkConfigUpdateRequest/NetworkConfigUpdateResponseNetworkConfigDeleteRequest/NetworkConfigDeleteResponseNetworkConfigFindAllRequest/NetworkConfigFindAllResponseNetworkConfig(entity)
Connection Test
TestConnectionRequest/TestConnectionResponse
Transformation
CreateTransformationRequest/CreateTransformationResponseUpdateTransformationRequest/UpdateTransformationResponseDeleteTransformationRequest/DeleteTransformationResponse
NEXT STEPS
- Decide on migration strategy (Option A, B, or C)
- Prioritize modules - Recommend starting with Connection Test (smallest), then Inputs, Network Config, Transformations, Connection, Connector (largest)
- Create database schema if moving to Option A
- Implement local services one module at a time
- Update tests for each migrated module
- Update deployment configs to remove INFACTORY_URL
- Coordinate with In-Factory team for data migration