From 177ec6368ca231a4210887ceee8489400d57b30a Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 17 Dec 2025 17:09:50 +0000 Subject: [PATCH] DOCS: Map maestro-infactory integration points for migration Add comprehensive documentation mapping all integration points between maestro and in-factory services to facilitate removing the dependency. Includes: - All 6 gRPC client configurations using INFACTORY_URL - 27+ gRPC method calls across 6 services - 25 REST endpoints that proxy to in-factory - Configuration files and environment variables - Proto package dependencies - Migration strategy options and recommendations --- MIGRATION_MAP_INFACTORY.md | 482 ++++++++++++++++++++++++++++++++ MIGRATION_MAP_INFACTORY_SIDE.md | 320 +++++++++++++++++++++ 2 files changed, 802 insertions(+) create mode 100644 MIGRATION_MAP_INFACTORY.md create mode 100644 MIGRATION_MAP_INFACTORY_SIDE.md diff --git a/MIGRATION_MAP_INFACTORY.md b/MIGRATION_MAP_INFACTORY.md new file mode 100644 index 0000000..e38d5fc --- /dev/null +++ b/MIGRATION_MAP_INFACTORY.md @@ -0,0 +1,482 @@ +# Maestro ↔ In-Factory Migration Map + +This document maps all integration points between Maestro and In-Factory that need to be addressed to remove the dependency. + +## Architecture Overview + +```mermaid +graph TD; + Frontend<-->Maestro; + Maestro<-->DUC; + Maestro<-->pi-factory; + Maestro<-->in-factory; +``` + +Maestro connects to **3 external microservices**: +- **DUC** (`DUC_URL`) - User management, authentication, permissions +- **PI-Factory** (`PIFACTORY_URL`) - Pipelines, Catalog services +- **IN-Factory** (`INFACTORY_URL`) - Connections, Connectors, Inputs, Network Config, Transformations + +--- + +## Summary Table + +| Category | Count | Impact Level | +|----------|-------|--------------| +| gRPC Client Configurations | 6 | HIGH | +| NestJS Modules | 6 | HIGH | +| REST Controllers/Endpoints | 6 | HIGH | +| Service Classes | 6 | HIGH | +| Configuration Files | 5 | MEDIUM | +| Proto Package Dependencies | 2 | HIGH | +| Environment Variables | 2 | LOW | + +--- + +## 1. gRPC CLIENT CONFIGURATIONS (Files to Migrate) + +These files configure gRPC connections to In-Factory services: + +### 1.1 Connection Client (`INFACTORY_URL`) +**File:** `src/modules/connection/client.config.ts` +```typescript +// Lines 10-11, 18 +process.env.INFACTORY_URL.startsWith('in-factory:') +process.env.INFACTORY_URL.includes('0.0.0.0') +url: process.env.INFACTORY_URL +``` +**Proto Services Used:** +- `ConnectionManager.ProtoPackages.WritePackage` +- `ConnectionManager.ProtoPackages.ReadPackage` + +### 1.2 Connector Client (`INFACTORY_URL`) +**File:** `src/modules/connector/client.config.ts` +```typescript +// Lines 10-11, 18 +url: process.env.INFACTORY_URL +``` +**Proto Services Used:** +- `ConnectorManager.ProtoPackages.WritePackage` +- `ConnectorManager.ProtoPackages.ReadPackage` + +### 1.3 Inputs Client (`INFACTORY_URL`) +**File:** `src/modules/inputs/inputs-client.config.ts` +```typescript +// Lines 10-11, 17 +url: process.env.INFACTORY_URL +``` +**Proto Services Used:** +- `Input.ProtoPackages.WritePackage` +- `Input.ProtoPackages.ReadPackage` + +### 1.4 Network Config Client (`INFACTORY_URL`) +**File:** `src/modules/network-config/network-config-client.config.ts` +```typescript +// Lines 10-11, 19 +url: process.env.INFACTORY_URL +``` +**Proto Services Used:** +- `NetworkConfig.ProtoPackages.WritePackage` +- `NetworkConfig.ProtoPackages.ReadPackage` + +### 1.5 Connection Test Client (`INFACTORY_URL`) +**File:** `src/modules/connection-test/connection-test-client.config.ts` +```typescript +// Lines 10-11, 17 +url: process.env.INFACTORY_URL +``` +**Proto Services Used:** +- `ConnectionTest.ProtoPackages.ReadPackage` + +### 1.6 Transformations Client (`INFACTORY_URL`) +**File:** `src/modules/transformations/transformations-client.ts` +```typescript +// Lines 10-11, 18 +url: process.env.INFACTORY_URL +``` +**Proto Services Used:** +- `Transformation.ProtoPackages.WritePackage` + +--- + +## 2. SERVICE CLASSES (Business Logic to Migrate) + +### 2.1 Connection Client Service +**File:** `src/modules/connection/client.service.ts` + +**gRPC Methods Called:** +| Method | Service | Direction | +|--------|---------|-----------| +| `CreateConnection()` | ConnectionManagerWriteServices | Write | +| `UpdateConnection()` | ConnectionManagerWriteServices | Write | +| `DeleteConnection()` | ConnectionManagerWriteServices | Write | +| `GetConnectionDetails()` | ConnectionManagerReadServices | Read | +| `GetAllConnections()` | ConnectionManagerReadServices | Read | +| `GetConnectorAvailableConnectionsByCustomer()` | ConnectionManagerReadServices | Read | +| `ValidatePlatformConnections()` | ConnectionManagerReadServices | Read | + +**Key Methods:** +- `createConnection()` +- `updateConnection()` +- `deleteConnection()` +- `getConnections()` +- `getConnectionDetails()` +- `validatePlatformConnections()` + +### 2.2 Connector Client Service +**File:** `src/modules/connector/client.service.ts` + +**gRPC Methods Called:** +| Method | Service | Direction | +|--------|---------|-----------| +| `RegisterConnector()` | ConnectorManagerWriteServices | Write | +| `UploadFile()` | ConnectorManagerWriteServices | Write | +| `RegisterMultipleConnectorsWithoutImage()` | ConnectorManagerWriteServices | Write | +| `UpdateConnectorByID()` | ConnectorManagerWriteServices | Write | +| `DeleteConnectorById()` | ConnectorManagerWriteServices | Write | +| `GetConnectors()` | ConnectorManagerReadServices | Read | + +### 2.3 Inputs Service +**File:** `src/modules/inputs/inputs.service.ts` + +**gRPC Methods Called:** +| Method | Service | Direction | +|--------|---------|-----------| +| `CreateInput()` | InputWriteService | Write | +| `UpdateInput()` | InputWriteService | Write | +| `DeleteInput()` | InputWriteService | Write | +| `GetAvailableEntities()` | InputReadService | Read | + +### 2.4 Network Config Service +**File:** `src/modules/network-config/network-config.service.ts` + +**gRPC Methods Called:** +| Method | Service | Direction | +|--------|---------|-----------| +| `NetworkConfigCreate()` | NetworkConfigWriteService | Write | +| `NetworkConfigUpdate()` | NetworkConfigWriteService | Write | +| `NetworkConfigDelete()` | NetworkConfigWriteService | Write | +| `NetworkConfigFindAll()` | NetworkConfigReadService | Read | +| `NetworkConfigFindOneById()` | NetworkConfigReadService | Read | + +### 2.5 Connection Test Service +**File:** `src/modules/connection-test/connection-test.service.ts` + +**gRPC Methods Called:** +| Method | Service | Direction | +|--------|---------|-----------| +| `TestConnection()` | ConnectionTestReadService | Read | + +### 2.6 Transformations Service +**File:** `src/modules/transformations/client.service.ts` + +**gRPC Methods Called:** +| Method | Service | Direction | +|--------|---------|-----------| +| `CreateTransformation()` | TransformationWriteService | Write | +| `UpdateTransformation()` | TransformationWriteService | Write | +| `DeleteTransformation()` | TransformationWriteService | Write | +| Various read operations | TransformationReadService | Read | + +--- + +## 3. NESTJS MODULES (Module Registration) + +These modules register the gRPC clients and export services: + +| Module | File | Imports | Exports | +|--------|------|---------|---------| +| ConnectionModule | `src/modules/connection/connection.module.ts` | ConnectionClientConfiguration | ConnectionClientService | +| ConnectorModule | `src/modules/connector/connector.module.ts` | ConnectorClientConfiguration | ConnectorClientService | +| InputsModule | `src/modules/inputs/inputs.module.ts` | InputsGrpcClient | InputsService | +| NetworkConfigModule | `src/modules/network-config/network-config.module.ts` | NetworkConfigGrpcClient | NetworkConfigService | +| ConnectionTestModule | `src/modules/connection-test/connection-test.module.ts` | ConnectionTestClientConfiguration | ConnectionTestService | +| TransformationsModule | `src/modules/transformations/transformations.module.ts` | TransformationsClientConfiguration | TransformationsService | + +**App Module Registration:** `src/app.module.ts` (Lines 15-21, 50-60) + +--- + +## 4. REST CONTROLLERS (API Endpoints to Migrate) + +### 4.1 Connection Controller +**File:** `src/modules/connection/connection.controller.ts` + +| HTTP Method | Endpoint | Description | +|-------------|----------|-------------| +| POST | `/connections` | Create a new connection | +| PUT | `/connections/:id` | Update an existing connection | +| DELETE | `/connections/:id` | Delete a connection | +| GET | `/connections` | Get all connections | +| GET | `/connections/:id` | Get connection details | +| GET | `/connections/available/:connector_id` | Get available connections by connector | + +### 4.2 Connector Controller +**File:** `src/modules/connector/connector.controller.ts` + +| HTTP Method | Endpoint | Description | +|-------------|----------|-------------| +| POST | `/connectors` | Register a new connector | +| POST | `/connectors/uploads` | Upload multiple connector files | +| GET | `/connectors` | List all connectors | +| PUT | `/connectors/:id` | Update a connector | +| DELETE | `/connectors/:id` | Delete a connector | + +### 4.3 Inputs Controller +**File:** `src/modules/inputs/inputs.controller.ts` + +| HTTP Method | Endpoint | Description | +|-------------|----------|-------------| +| GET | `/inputs/available-entities/:plugin` | Get available entities for a plugin | +| POST | `/inputs` | Create an input | +| PATCH | `/inputs` | Update an input | +| DELETE | `/inputs/:id` | Delete an input | + +### 4.4 Network Config Controller +**File:** `src/modules/network-config/network-config.controller.ts` + +| HTTP Method | Endpoint | Description | +|-------------|----------|-------------| +| GET | `/network-configs` | Get all network configurations | +| GET | `/network-configs/:id` | Get network config by ID | +| POST | `/network-configs` | Create network configuration | +| PUT | `/network-configs/:id` | Update network configuration | +| DELETE | `/network-configs/:id` | Delete network configuration | + +### 4.5 Connection Test Controller +**File:** `src/modules/connection-test/connection-test.controller.ts` + +| HTTP Method | Endpoint | Description | +|-------------|----------|-------------| +| POST | `/connection-test` | Test a connection | + +### 4.6 Transformations Controller +**File:** `src/modules/transformations/transformations.controller.ts` + +| HTTP Method | Endpoint | Description | +|-------------|----------|-------------| +| POST | `/transformations` | Create transformation | +| GET | `/transformations` | Get transformations | +| PUT | `/transformations/:id` | Update transformation | +| DELETE | `/transformations/:id` | Delete transformation | + +--- + +## 5. CONFIGURATION FILES + +### 5.1 Helm Chart Values (Production) +**File:** `deploy/helm-chart/values.yaml` +```yaml +# Lines 32, 42 +maestro: + in_factory_url: in-factory.dadosfera.ai + tr_factory_url: in-factory.dadosfera.ai # Transformation factory also uses in-factory +``` + +### 5.2 Helm Chart Values (Staging) +**File:** `deploy/helm-chart/values-stg.yaml` +```yaml +# Lines 5-6 +maestro: + in_factory_url: in-factory.stg.dadosfera.ai + tr_factory_url: in-factory.stg.dadosfera.ai +``` + +### 5.3 Helmfiles (Production) +**File:** `deploy/helmfiles/prd.yaml` +```yaml +# Lines 15-18, 39-42 (for both maestro and maestro-unimed releases) +maestro.in_factory_url: in-factory.dadosfera.ai +maestro.tr_factory_url: in-factory.dadosfera.ai +``` + +### 5.4 Deployment Template +**File:** `deploy/helm-chart/templates/deployment.yaml` +```yaml +# Lines 71-72 +- name: INFACTORY_URL + value: {{ .Values.maestro.in_factory_url }} +``` + +### 5.5 Environment Type Definition +**File:** `environment.d.ts` +```typescript +// Line 9 +INFACTORY_URL: string; +``` + +--- + +## 6. PROTO PACKAGE DEPENDENCIES + +### 6.1 Package.json +**File:** `package.json` +```json +{ + "@dadosfera/protospack": "2.5.3", + "@dadosfera/protospack-v2": "3.38.0-beta.14" +} +``` + +### 6.2 Proto Imports from `@dadosfera/protospack-v2` (In-Factory related) + +| Import Path | Used In | +|-------------|---------| +| `ConnectionManager` | connection/client.config.ts, connection/client.service.ts | +| `ConnectionManager/interfaces/messages` | connection/client.service.ts, connection/dtos/connection.ts | +| `ConnectionManager/interfaces/entities` | connection/dtos/connection.ts | +| `ConnectorManager` | connector/client.config.ts, connector/client.service.ts | +| `Input` | inputs/inputs-client.config.ts, inputs/inputs.service.ts | +| `Input/interfaces/messages` | inputs/inputs.service.ts | +| `Input/interfaces/entities` | inputs/inputs.controller.ts, inputs/inputs.service.ts | +| `NetworkConfig` | network-config/network-config-client.config.ts, network-config/network-config.service.ts | +| `NetworkConfig/interfaces/entities` | network-config/dto/network-config.ts | +| `ConnectionTest` | connection-test/connection-test-client.config.ts, connection-test/connection-test.service.ts | +| `Transformation` | transformations/transformations-client.ts, transformations/client.service.ts | + +--- + +## 7. MIGRATION STRATEGY OPTIONS + +### Option A: Move In-Factory functionality INTO Maestro +**Pros:** +- Single service to maintain +- No network latency for these operations +- Simpler deployment + +**Cons:** +- Increases Maestro's responsibility/complexity +- Requires database access from Maestro +- May require significant refactoring + +**Files to create/migrate:** +1. Database models for connections, connectors, inputs, network-configs, transformations +2. Repository layer for database operations +3. Convert gRPC services to internal services +4. Remove all gRPC client configurations + +### Option B: Create REST API wrapper in In-Factory +**Pros:** +- Minimal changes to Maestro +- Can migrate incrementally + +**Cons:** +- Still maintains dependency +- Additional REST→gRPC translation layer + +### Option C: Direct database access from Maestro +**Pros:** +- Removes runtime dependency +- Better performance + +**Cons:** +- Shared database coupling +- Complex migration + +--- + +## 8. FILES TO MODIFY/DELETE (Summary) + +### High Priority - Core Integration Files +``` +src/modules/connection/client.config.ts → DELETE or REPLACE +src/modules/connection/client.service.ts → REPLACE with local implementation +src/modules/connector/client.config.ts → DELETE or REPLACE +src/modules/connector/client.service.ts → REPLACE with local implementation +src/modules/inputs/inputs-client.config.ts → DELETE or REPLACE +src/modules/inputs/inputs.service.ts → REPLACE with local implementation +src/modules/network-config/network-config-client.config.ts → DELETE or REPLACE +src/modules/network-config/network-config.service.ts → REPLACE with local implementation +src/modules/connection-test/connection-test-client.config.ts → DELETE or REPLACE +src/modules/connection-test/connection-test.service.ts → REPLACE with local implementation +src/modules/transformations/transformations-client.ts → DELETE or REPLACE +src/modules/transformations/client.service.ts → REPLACE with local implementation +``` + +### Medium Priority - Module Registration +``` +src/modules/connection/connection.module.ts → UPDATE imports +src/modules/connector/connector.module.ts → UPDATE imports +src/modules/inputs/inputs.module.ts → UPDATE imports +src/modules/network-config/network-config.module.ts → UPDATE imports +src/modules/connection-test/connection-test.module.ts → UPDATE imports +src/modules/transformations/transformations.module.ts → UPDATE imports +src/app.module.ts → UPDATE if module structure changes +``` + +### Low Priority - Configuration +``` +deploy/helm-chart/values.yaml → REMOVE in_factory_url, tr_factory_url +deploy/helm-chart/values-stg.yaml → REMOVE in_factory_url, tr_factory_url +deploy/helmfiles/prd.yaml → REMOVE in_factory_url references +deploy/helm-chart/templates/deployment.yaml → REMOVE INFACTORY_URL env var +environment.d.ts → REMOVE INFACTORY_URL type +README.md → UPDATE architecture diagram +``` + +--- + +## 9. DEPENDENCY COUNT BY MODULE + +| Module | Files | gRPC Calls | REST Endpoints | +|--------|-------|------------|----------------| +| Connection | 4 | 7 | 6 | +| Connector | 3 | 6 | 5 | +| Inputs | 3 | 4 | 4 | +| Network Config | 3 | 5 | 5 | +| Connection Test | 3 | 1 | 1 | +| Transformations | 4 | 4+ | 4 | +| **TOTAL** | **20** | **27+** | **25** | + +--- + +## 10. DATA MODELS (Proto Messages Used) + +### Connection Manager +- `CreateConnectionRequest` / `CreateConnectionResponse` +- `UpdateConnectionRequest` / `UpdateConnectionResponse` +- `DeleteConnectionRequest` / `DeleteConnectionResponse` +- `GetConnectionDetailsRequest` / `GetConnectionDetailsResponse` +- `GetAllConnectionsRequest` / `GetAllConnectionsResponse` +- `Connection` (entity) +- `ConnectionCredential` (entity) + +### Connector Manager +- `RegisterConnectorRequest` / `RegisterConnectorResponse` +- `UploadFileRequest` / `UploadFileResponse` +- `GetConnectorsRequest` / `GetConnectorsResponse` +- `Connector` (entity) + +### Input +- `CreateInputRequest` / `CreateInputResponse` +- `UpdateInputRequest` / `UpdateInputResponse` +- `DeleteInputRequest` / `DeleteInputResponse` +- `GetAvailableEntitiesRequest` / `GetAvailableEntitiesResponse` +- `Info` (entity) + +### Network Config +- `NetworkConfigCreateRequest` / `NetworkConfigCreateResponse` +- `NetworkConfigUpdateRequest` / `NetworkConfigUpdateResponse` +- `NetworkConfigDeleteRequest` / `NetworkConfigDeleteResponse` +- `NetworkConfigFindAllRequest` / `NetworkConfigFindAllResponse` +- `NetworkConfig` (entity) + +### Connection Test +- `TestConnectionRequest` / `TestConnectionResponse` + +### Transformation +- `CreateTransformationRequest` / `CreateTransformationResponse` +- `UpdateTransformationRequest` / `UpdateTransformationResponse` +- `DeleteTransformationRequest` / `DeleteTransformationResponse` + +--- + +## NEXT STEPS + +1. **Decide on migration strategy** (Option A, B, or C) +2. **Prioritize modules** - Recommend starting with Connection Test (smallest), then Inputs, Network Config, Transformations, Connection, Connector (largest) +3. **Create database schema** if moving to Option A +4. **Implement local services** one module at a time +5. **Update tests** for each migrated module +6. **Update deployment configs** to remove INFACTORY_URL +7. **Coordinate with In-Factory team** for data migration diff --git a/MIGRATION_MAP_INFACTORY_SIDE.md b/MIGRATION_MAP_INFACTORY_SIDE.md new file mode 100644 index 0000000..80d98de --- /dev/null +++ b/MIGRATION_MAP_INFACTORY_SIDE.md @@ -0,0 +1,320 @@ +# In-Factory Side - Migration Requirements + +This document outlines what needs to be addressed in the **In-Factory** service to remove its coupling with Maestro. + +> **Note:** This analysis is based on the Maestro codebase. For a complete analysis, the In-Factory repository should also be reviewed. + +--- + +## Current Architecture (In-Factory → Maestro) + +Based on the Maestro codebase analysis, **In-Factory** exposes the following gRPC services that Maestro consumes: + +```mermaid +graph LR; + Maestro -->|gRPC| InFactory; + subgraph InFactory Services + CM[ConnectionManager] + ConM[ConnectorManager] + IN[Input] + NC[NetworkConfig] + CT[ConnectionTest] + TR[Transformation] + end + Maestro --> CM; + Maestro --> ConM; + Maestro --> IN; + Maestro --> NC; + Maestro --> CT; + Maestro --> TR; +``` + +--- + +## gRPC Services Exposed by In-Factory + +### 1. ConnectionManager Service + +**Package:** `ConnectionManager` from `@dadosfera/protospack-v2` + +#### Write Services (`ConnectionManagerWriteServices`) +| gRPC Method | Description | Called By | +|-------------|-------------|-----------| +| `CreateConnection` | Create a new data connection | Maestro POST /connections | +| `UpdateConnection` | Update an existing connection | Maestro PUT /connections/:id | +| `DeleteConnection` | Delete a connection | Maestro DELETE /connections/:id | + +#### Read Services (`ConnectionManagerReadServices`) +| gRPC Method | Description | Called By | +|-------------|-------------|-----------| +| `GetConnectionDetails` | Get details of a single connection | Maestro GET /connections/:id | +| `GetAllConnections` | List all connections for a customer | Maestro GET /connections | +| `GetConnectorAvailableConnectionsByCustomer` | Get available connections by connector | Maestro GET /connections/available/:connector_id | +| `ValidatePlatformConnections` | Validate connections against platform | Maestro internal | + +--- + +### 2. ConnectorManager Service + +**Package:** `ConnectorManager` from `@dadosfera/protospack-v2` + +#### Write Services (`ConnectorManagerWriteServices`) +| gRPC Method | Description | Called By | +|-------------|-------------|-----------| +| `RegisterConnector` | Register a new connector | Maestro POST /connectors | +| `UploadFile` | Upload connector files | Maestro POST /connectors/uploads | +| `RegisterMultipleConnectorsWithoutImage` | Bulk register connectors | Maestro internal | +| `UpdateConnectorByID` | Update a connector | Maestro PUT /connectors/:id | +| `DeleteConnectorById` | Delete a connector | Maestro DELETE /connectors/:id | + +#### Read Services (`ConnectorManagerReadServices`) +| gRPC Method | Description | Called By | +|-------------|-------------|-----------| +| `GetConnectors` | List all connectors | Maestro GET /connectors | + +--- + +### 3. Input Service + +**Package:** `Input` from `@dadosfera/protospack-v2` + +#### Write Services (`InputWriteService`) +| gRPC Method | Description | Called By | +|-------------|-------------|-----------| +| `CreateInput` | Create a new input configuration | Maestro POST /inputs | +| `UpdateInput` | Update input configuration | Maestro PATCH /inputs | +| `DeleteInput` | Delete an input | Maestro DELETE /inputs/:id | + +#### Read Services (`InputReadService`) +| gRPC Method | Description | Called By | +|-------------|-------------|-----------| +| `GetAvailableEntities` | Get available entities for a plugin | Maestro GET /inputs/available-entities/:plugin | + +--- + +### 4. NetworkConfig Service + +**Package:** `NetworkConfig` from `@dadosfera/protospack-v2` + +#### Write Services (`NetworkConfigWriteService`) +| gRPC Method | Description | Called By | +|-------------|-------------|-----------| +| `NetworkConfigCreate` | Create network configuration | Maestro POST /network-configs | +| `NetworkConfigUpdate` | Update network configuration | Maestro PUT /network-configs/:id | +| `NetworkConfigDelete` | Delete network configuration | Maestro DELETE /network-configs/:id | + +#### Read Services (`NetworkConfigReadService`) +| gRPC Method | Description | Called By | +|-------------|-------------|-----------| +| `NetworkConfigFindAll` | List all network configs | Maestro GET /network-configs | +| `NetworkConfigFindOneById` | Get network config by ID | Maestro GET /network-configs/:id | + +--- + +### 5. ConnectionTest Service + +**Package:** `ConnectionTest` from `@dadosfera/protospack-v2` + +#### Read Services (`ConnectionTestReadService`) +| gRPC Method | Description | Called By | +|-------------|-------------|-----------| +| `TestConnection` | Test a connection's connectivity | Maestro POST /connection-test | + +--- + +### 6. Transformation Service + +**Package:** `Transformation` from `@dadosfera/protospack-v2` + +#### Write Services (`TransformationWriteService`) +| gRPC Method | Description | Called By | +|-------------|-------------|-----------| +| `CreateTransformation` | Create a transformation | Maestro POST /transformations | +| `UpdateTransformation` | Update a transformation | Maestro PUT /transformations/:id | +| `DeleteTransformation` | Delete a transformation | Maestro DELETE /transformations/:id | + +#### Read Services (`TransformationReadService`) +| gRPC Method | Description | Called By | +|-------------|-------------|-----------| +| Various read operations | Get transformation details | Maestro GET /transformations | + +--- + +## Proto Package Ownership + +The proto definitions are managed in `@dadosfera/protospack-v2`: + +``` +@dadosfera/protospack-v2/ +├── dist/lib/ +│ ├── ConnectionManager/ +│ │ ├── interfaces/messages.ts +│ │ └── interfaces/entities.ts +│ ├── ConnectorManager/ +│ │ ├── interfaces/messages.ts +│ │ └── interfaces/entities.ts +│ ├── Input/ +│ │ ├── interfaces/messages.ts +│ │ └── interfaces/entities.ts +│ ├── NetworkConfig/ +│ │ ├── interfaces/messages.ts +│ │ └── interfaces/entities.ts +│ ├── ConnectionTest/ +│ │ ├── interfaces/messages.ts +│ │ └── interfaces/entities.ts +│ └── Transformation/ +│ ├── interfaces/messages.ts +│ └── interfaces/entities.ts +``` + +--- + +## In-Factory Migration Options + +### Option 1: Expose REST API (Keep In-Factory, Add REST Layer) + +**Changes Required in In-Factory:** +1. Add REST controllers for all services +2. Implement HTTP endpoints mirroring gRPC methods +3. Update deployment to expose HTTP port +4. Create OpenAPI documentation + +**Pros:** +- Minimal architectural changes +- Can run both gRPC and REST in parallel during migration +- Maestro can switch to REST calls gradually + +**Cons:** +- Still maintains service dependency +- Adds another communication layer + +--- + +### Option 2: Move Logic to Maestro (Deprecate In-Factory for these features) + +**Changes Required in In-Factory:** +1. Export database schema/migrations +2. Document all business logic +3. Provide data migration scripts +4. Deprecate gRPC endpoints after migration + +**Changes Required in Maestro:** +1. Create database models +2. Implement repositories +3. Create service layer with same business logic +4. Run data migration + +**Pros:** +- Removes runtime dependency completely +- Simplifies architecture +- One less service to maintain + +**Cons:** +- Significant development effort +- Risk of business logic divergence during migration +- Database sharing concerns + +--- + +### Option 3: Merge Services (Combine In-Factory into a larger service) + +**Changes Required:** +1. Create new combined service +2. Migrate both In-Factory and relevant Maestro code +3. Update all clients + +**Pros:** +- Clean architectural redesign +- Opportunity to optimize + +**Cons:** +- Largest effort +- Risk of disruption + +--- + +## Data Migration Considerations + +### Entities Managed by In-Factory + +Based on proto definitions, In-Factory manages: + +1. **Connections** + - Connection credentials + - Connection metadata + - Customer associations + +2. **Connectors** + - Connector definitions + - Connector images/files + - Plugin configurations + +3. **Inputs** + - Input configurations + - Entity mappings + +4. **Network Configs** + - Network configuration settings + - Security settings + +5. **Transformations** + - Transformation definitions + - Transformation scripts + +### Migration Steps +1. Export database schema from In-Factory +2. Create equivalent schema in target database +3. Write data migration scripts +4. Validate data integrity +5. Switch traffic +6. Decommission old service + +--- + +## Recommended Investigation for In-Factory Team + +1. **Check for Maestro dependencies in In-Factory** + - Does In-Factory call any Maestro APIs? + - Are there any shared databases? + - Any shared message queues? + +2. **Document database schema** + - All tables related to connections, connectors, inputs, network-configs, transformations + - Foreign key relationships + - Indexes and constraints + +3. **List all consumers** + - Besides Maestro, who else calls In-Factory? + - Are there other internal services? + - Any external integrations? + +4. **Business logic documentation** + - Validation rules + - Business constraints + - Side effects (events, notifications, etc.) + +--- + +## Timeline Considerations + +| Phase | Description | Dependencies | +|-------|-------------|--------------| +| Phase 1 | Analysis & Planning | Both teams available | +| Phase 2 | Schema/API Design | Proto definitions finalized | +| Phase 3 | Implementation | Development resources | +| Phase 4 | Data Migration | Database access, downtime window | +| Phase 5 | Testing | QA resources, test environments | +| Phase 6 | Cutover | Deployment coordination | +| Phase 7 | Decommission | Monitoring, rollback plan | + +--- + +## Questions for In-Factory Team + +1. What database does In-Factory use? (PostgreSQL, MongoDB, etc.) +2. Are there any async operations? (message queues, event sourcing) +3. What is the current data volume for each entity type? +4. Are there any scheduled jobs or background processes? +5. What monitoring/alerting is in place? +6. Are there any data retention policies? +7. What is the backup/recovery strategy?