gitea_admin/maestro
1
0
Fork 0
mirror of https://github.com/dadosfera/maestro.git synced 2026-11-11 19:39:09 +00:00
Code Issues Packages Projects Releases Wiki Activity

DOCS: Map maestro-infactory integration points for migration

Browse Source 
Add comprehensive documentation mapping all integration points between
maestro and in-factory services to facilitate removing the dependency.

Includes:
- All 6 gRPC client configurations using INFACTORY_URL
- 27+ gRPC method calls across 6 services
- 25 REST endpoints that proxy to in-factory
- Configuration files and environment variables
- Proto package dependencies
- Migration strategy options and recommendations
This commit is contained in:
Claude
2025-12-17 17:09:50 +00:00
parent e7f410831f
commit 177ec6368c
2 changed files with 802 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
MIGRATION_MAP_INFACTORY.md
+482
View File
@@ -0,0 +1,482 @@
# Maestro ↔ In-Factory Migration Map
This document maps all integration points between Maestro and In-Factory that need to be addressed to remove the dependency.
## Architecture Overview
```mermaid
graph TD;
Frontend<-->Maestro;
Maestro<-->DUC;
Maestro<-->pi-factory;
Maestro<-->in-factory;
```
Maestro connects to **3 external microservices**:
- **DUC** (`DUC_URL`) - User management, authentication, permissions
- **PI-Factory** (`PIFACTORY_URL`) - Pipelines, Catalog services
- **IN-Factory** (`INFACTORY_URL`) - Connections, Connectors, Inputs, Network Config, Transformations
---
## Summary Table
| Category | Count | Impact Level |
|----------|-------|--------------|
| gRPC Client Configurations | 6 | HIGH |
| NestJS Modules | 6 | HIGH |
| REST Controllers/Endpoints | 6 | HIGH |
| Service Classes | 6 | HIGH |
| Configuration Files | 5 | MEDIUM |
| Proto Package Dependencies | 2 | HIGH |
| Environment Variables | 2 | LOW |
---
## 1. gRPC CLIENT CONFIGURATIONS (Files to Migrate)
These files configure gRPC connections to In-Factory services:
### 1.1 Connection Client (`INFACTORY_URL`)
**File:** `src/modules/connection/client.config.ts`
```typescript
// Lines 10-11, 18
process.env.INFACTORY_URL.startsWith('in-factory:')
process.env.INFACTORY_URL.includes('0.0.0.0')
url: process.env.INFACTORY_URL
```
**Proto Services Used:**
- `ConnectionManager.ProtoPackages.WritePackage`
- `ConnectionManager.ProtoPackages.ReadPackage`
### 1.2 Connector Client (`INFACTORY_URL`)
**File:** `src/modules/connector/client.config.ts`
```typescript
// Lines 10-11, 18
url: process.env.INFACTORY_URL
```
**Proto Services Used:**
- `ConnectorManager.ProtoPackages.WritePackage`
- `ConnectorManager.ProtoPackages.ReadPackage`
### 1.3 Inputs Client (`INFACTORY_URL`)
**File:** `src/modules/inputs/inputs-client.config.ts`
```typescript
// Lines 10-11, 17
url: process.env.INFACTORY_URL
```
**Proto Services Used:**
- `Input.ProtoPackages.WritePackage`
- `Input.ProtoPackages.ReadPackage`
### 1.4 Network Config Client (`INFACTORY_URL`)
**File:** `src/modules/network-config/network-config-client.config.ts`
```typescript
// Lines 10-11, 19
url: process.env.INFACTORY_URL
```
**Proto Services Used:**
- `NetworkConfig.ProtoPackages.WritePackage`
- `NetworkConfig.ProtoPackages.ReadPackage`
### 1.5 Connection Test Client (`INFACTORY_URL`)
**File:** `src/modules/connection-test/connection-test-client.config.ts`
```typescript
// Lines 10-11, 17
url: process.env.INFACTORY_URL
```
**Proto Services Used:**
- `ConnectionTest.ProtoPackages.ReadPackage`
### 1.6 Transformations Client (`INFACTORY_URL`)
**File:** `src/modules/transformations/transformations-client.ts`
```typescript
// Lines 10-11, 18
url: process.env.INFACTORY_URL
```
**Proto Services Used:**
- `Transformation.ProtoPackages.WritePackage`
---
## 2. SERVICE CLASSES (Business Logic to Migrate)
### 2.1 Connection Client Service
**File:** `src/modules/connection/client.service.ts`
**gRPC Methods Called:**
| Method | Service | Direction |
|--------|---------|-----------|
| `CreateConnection()` | ConnectionManagerWriteServices | Write |
| `UpdateConnection()` | ConnectionManagerWriteServices | Write |
| `DeleteConnection()` | ConnectionManagerWriteServices | Write |
| `GetConnectionDetails()` | ConnectionManagerReadServices | Read |
| `GetAllConnections()` | ConnectionManagerReadServices | Read |
| `GetConnectorAvailableConnectionsByCustomer()` | ConnectionManagerReadServices | Read |
| `ValidatePlatformConnections()` | ConnectionManagerReadServices | Read |
**Key Methods:**
- `createConnection()`
- `updateConnection()`
- `deleteConnection()`
- `getConnections()`
- `getConnectionDetails()`
- `validatePlatformConnections()`
### 2.2 Connector Client Service
**File:** `src/modules/connector/client.service.ts`
**gRPC Methods Called:**
| Method | Service | Direction |
|--------|---------|-----------|
| `RegisterConnector()` | ConnectorManagerWriteServices | Write |
| `UploadFile()` | ConnectorManagerWriteServices | Write |
| `RegisterMultipleConnectorsWithoutImage()` | ConnectorManagerWriteServices | Write |
| `UpdateConnectorByID()` | ConnectorManagerWriteServices | Write |
| `DeleteConnectorById()` | ConnectorManagerWriteServices | Write |
| `GetConnectors()` | ConnectorManagerReadServices | Read |
### 2.3 Inputs Service
**File:** `src/modules/inputs/inputs.service.ts`
**gRPC Methods Called:**
| Method | Service | Direction |
|--------|---------|-----------|
| `CreateInput()` | InputWriteService | Write |
| `UpdateInput()` | InputWriteService | Write |
| `DeleteInput()` | InputWriteService | Write |
| `GetAvailableEntities()` | InputReadService | Read |
### 2.4 Network Config Service
**File:** `src/modules/network-config/network-config.service.ts`
**gRPC Methods Called:**
| Method | Service | Direction |
|--------|---------|-----------|
| `NetworkConfigCreate()` | NetworkConfigWriteService | Write |
| `NetworkConfigUpdate()` | NetworkConfigWriteService | Write |
| `NetworkConfigDelete()` | NetworkConfigWriteService | Write |
| `NetworkConfigFindAll()` | NetworkConfigReadService | Read |
| `NetworkConfigFindOneById()` | NetworkConfigReadService | Read |
### 2.5 Connection Test Service
**File:** `src/modules/connection-test/connection-test.service.ts`
**gRPC Methods Called:**
| Method | Service | Direction |
|--------|---------|-----------|
| `TestConnection()` | ConnectionTestReadService | Read |
### 2.6 Transformations Service
**File:** `src/modules/transformations/client.service.ts`
**gRPC Methods Called:**
| Method | Service | Direction |
|--------|---------|-----------|
| `CreateTransformation()` | TransformationWriteService | Write |
| `UpdateTransformation()` | TransformationWriteService | Write |
| `DeleteTransformation()` | TransformationWriteService | Write |
| Various read operations | TransformationReadService | Read |
---
## 3. NESTJS MODULES (Module Registration)
These modules register the gRPC clients and export services:
| Module | File | Imports | Exports |
|--------|------|---------|---------|
| ConnectionModule | `src/modules/connection/connection.module.ts` | ConnectionClientConfiguration | ConnectionClientService |
| ConnectorModule | `src/modules/connector/connector.module.ts` | ConnectorClientConfiguration | ConnectorClientService |
| InputsModule | `src/modules/inputs/inputs.module.ts` | InputsGrpcClient | InputsService |
| NetworkConfigModule | `src/modules/network-config/network-config.module.ts` | NetworkConfigGrpcClient | NetworkConfigService |
| ConnectionTestModule | `src/modules/connection-test/connection-test.module.ts` | ConnectionTestClientConfiguration | ConnectionTestService |
| TransformationsModule | `src/modules/transformations/transformations.module.ts` | TransformationsClientConfiguration | TransformationsService |
**App Module Registration:** `src/app.module.ts` (Lines 15-21, 50-60)
---
## 4. REST CONTROLLERS (API Endpoints to Migrate)
### 4.1 Connection Controller
**File:** `src/modules/connection/connection.controller.ts`
| HTTP Method | Endpoint | Description |
|-------------|----------|-------------|
| POST | `/connections` | Create a new connection |
| PUT | `/connections/:id` | Update an existing connection |
| DELETE | `/connections/:id` | Delete a connection |
| GET | `/connections` | Get all connections |
| GET | `/connections/:id` | Get connection details |
| GET | `/connections/available/:connector_id` | Get available connections by connector |
### 4.2 Connector Controller
**File:** `src/modules/connector/connector.controller.ts`
| HTTP Method | Endpoint | Description |
|-------------|----------|-------------|
| POST | `/connectors` | Register a new connector |
| POST | `/connectors/uploads` | Upload multiple connector files |
| GET | `/connectors` | List all connectors |
| PUT | `/connectors/:id` | Update a connector |
| DELETE | `/connectors/:id` | Delete a connector |
### 4.3 Inputs Controller
**File:** `src/modules/inputs/inputs.controller.ts`
| HTTP Method | Endpoint | Description |
|-------------|----------|-------------|
| GET | `/inputs/available-entities/:plugin` | Get available entities for a plugin |
| POST | `/inputs` | Create an input |
| PATCH | `/inputs` | Update an input |
| DELETE | `/inputs/:id` | Delete an input |
### 4.4 Network Config Controller
**File:** `src/modules/network-config/network-config.controller.ts`
| HTTP Method | Endpoint | Description |
|-------------|----------|-------------|
| GET | `/network-configs` | Get all network configurations |
| GET | `/network-configs/:id` | Get network config by ID |
| POST | `/network-configs` | Create network configuration |
| PUT | `/network-configs/:id` | Update network configuration |
| DELETE | `/network-configs/:id` | Delete network configuration |
### 4.5 Connection Test Controller
**File:** `src/modules/connection-test/connection-test.controller.ts`
| HTTP Method | Endpoint | Description |
|-------------|----------|-------------|
| POST | `/connection-test` | Test a connection |
### 4.6 Transformations Controller
**File:** `src/modules/transformations/transformations.controller.ts`
| HTTP Method | Endpoint | Description |
|-------------|----------|-------------|
| POST | `/transformations` | Create transformation |
| GET | `/transformations` | Get transformations |
| PUT | `/transformations/:id` | Update transformation |
| DELETE | `/transformations/:id` | Delete transformation |
---
## 5. CONFIGURATION FILES
### 5.1 Helm Chart Values (Production)
**File:** `deploy/helm-chart/values.yaml`
```yaml
# Lines 32, 42
maestro:
in_factory_url: in-factory.dadosfera.ai
tr_factory_url: in-factory.dadosfera.ai # Transformation factory also uses in-factory
```
### 5.2 Helm Chart Values (Staging)
**File:** `deploy/helm-chart/values-stg.yaml`
```yaml
# Lines 5-6
maestro:
in_factory_url: in-factory.stg.dadosfera.ai
tr_factory_url: in-factory.stg.dadosfera.ai
```
### 5.3 Helmfiles (Production)
**File:** `deploy/helmfiles/prd.yaml`
```yaml
# Lines 15-18, 39-42 (for both maestro and maestro-unimed releases)
maestro.in_factory_url: in-factory.dadosfera.ai
maestro.tr_factory_url: in-factory.dadosfera.ai
```
### 5.4 Deployment Template
**File:** `deploy/helm-chart/templates/deployment.yaml`
```yaml
# Lines 71-72
- name: INFACTORY_URL
value: {{ .Values.maestro.in_factory_url }}
```
### 5.5 Environment Type Definition
**File:** `environment.d.ts`
```typescript
// Line 9
INFACTORY_URL: string;
```
---
## 6. PROTO PACKAGE DEPENDENCIES
### 6.1 Package.json
**File:** `package.json`
```json
{
"@dadosfera/protospack": "2.5.3",
"@dadosfera/protospack-v2": "3.38.0-beta.14"
}
```
### 6.2 Proto Imports from `@dadosfera/protospack-v2` (In-Factory related)
| Import Path | Used In |
|-------------|---------|
| `ConnectionManager` | connection/client.config.ts, connection/client.service.ts |
| `ConnectionManager/interfaces/messages` | connection/client.service.ts, connection/dtos/connection.ts |
| `ConnectionManager/interfaces/entities` | connection/dtos/connection.ts |
| `ConnectorManager` | connector/client.config.ts, connector/client.service.ts |
| `Input` | inputs/inputs-client.config.ts, inputs/inputs.service.ts |
| `Input/interfaces/messages` | inputs/inputs.service.ts |
| `Input/interfaces/entities` | inputs/inputs.controller.ts, inputs/inputs.service.ts |
| `NetworkConfig` | network-config/network-config-client.config.ts, network-config/network-config.service.ts |
| `NetworkConfig/interfaces/entities` | network-config/dto/network-config.ts |
| `ConnectionTest` | connection-test/connection-test-client.config.ts, connection-test/connection-test.service.ts |
| `Transformation` | transformations/transformations-client.ts, transformations/client.service.ts |
---
## 7. MIGRATION STRATEGY OPTIONS
### Option A: Move In-Factory functionality INTO Maestro
**Pros:**
- Single service to maintain
- No network latency for these operations
- Simpler deployment
**Cons:**
- Increases Maestro's responsibility/complexity
- Requires database access from Maestro
- May require significant refactoring
**Files to create/migrate:**
1. Database models for connections, connectors, inputs, network-configs, transformations
2. Repository layer for database operations
3. Convert gRPC services to internal services
4. Remove all gRPC client configurations
### Option B: Create REST API wrapper in In-Factory
**Pros:**
- Minimal changes to Maestro
- Can migrate incrementally
**Cons:**
- Still maintains dependency
- Additional REST→gRPC translation layer
### Option C: Direct database access from Maestro
**Pros:**
- Removes runtime dependency
- Better performance
**Cons:**
- Shared database coupling
- Complex migration
---
## 8. FILES TO MODIFY/DELETE (Summary)
### High Priority - Core Integration Files
```
src/modules/connection/client.config.ts → DELETE or REPLACE
src/modules/connection/client.service.ts → REPLACE with local implementation
src/modules/connector/client.config.ts → DELETE or REPLACE
src/modules/connector/client.service.ts → REPLACE with local implementation
src/modules/inputs/inputs-client.config.ts → DELETE or REPLACE
src/modules/inputs/inputs.service.ts → REPLACE with local implementation
src/modules/network-config/network-config-client.config.ts → DELETE or REPLACE
src/modules/network-config/network-config.service.ts → REPLACE with local implementation
src/modules/connection-test/connection-test-client.config.ts → DELETE or REPLACE
src/modules/connection-test/connection-test.service.ts → REPLACE with local implementation
src/modules/transformations/transformations-client.ts → DELETE or REPLACE
src/modules/transformations/client.service.ts → REPLACE with local implementation
```
### Medium Priority - Module Registration
```
src/modules/connection/connection.module.ts → UPDATE imports
src/modules/connector/connector.module.ts → UPDATE imports
src/modules/inputs/inputs.module.ts → UPDATE imports
src/modules/network-config/network-config.module.ts → UPDATE imports
src/modules/connection-test/connection-test.module.ts → UPDATE imports
src/modules/transformations/transformations.module.ts → UPDATE imports
src/app.module.ts → UPDATE if module structure changes
```
### Low Priority - Configuration
```
deploy/helm-chart/values.yaml → REMOVE in_factory_url, tr_factory_url
deploy/helm-chart/values-stg.yaml → REMOVE in_factory_url, tr_factory_url
deploy/helmfiles/prd.yaml → REMOVE in_factory_url references
deploy/helm-chart/templates/deployment.yaml → REMOVE INFACTORY_URL env var
environment.d.ts → REMOVE INFACTORY_URL type
README.md → UPDATE architecture diagram
```
---
## 9. DEPENDENCY COUNT BY MODULE
| Module | Files | gRPC Calls | REST Endpoints |
|--------|-------|------------|----------------|
| Connection | 4 | 7 | 6 |
| Connector | 3 | 6 | 5 |
| Inputs | 3 | 4 | 4 |
| Network Config | 3 | 5 | 5 |
| Connection Test | 3 | 1 | 1 |
| Transformations | 4 | 4+ | 4 |
| **TOTAL** | **20** | **27+** | **25** |
---
## 10. DATA MODELS (Proto Messages Used)
### Connection Manager
- `CreateConnectionRequest` / `CreateConnectionResponse`
- `UpdateConnectionRequest` / `UpdateConnectionResponse`
- `DeleteConnectionRequest` / `DeleteConnectionResponse`
- `GetConnectionDetailsRequest` / `GetConnectionDetailsResponse`
- `GetAllConnectionsRequest` / `GetAllConnectionsResponse`
- `Connection` (entity)
- `ConnectionCredential` (entity)
### Connector Manager
- `RegisterConnectorRequest` / `RegisterConnectorResponse`
- `UploadFileRequest` / `UploadFileResponse`
- `GetConnectorsRequest` / `GetConnectorsResponse`
- `Connector` (entity)
### Input
- `CreateInputRequest` / `CreateInputResponse`
- `UpdateInputRequest` / `UpdateInputResponse`
- `DeleteInputRequest` / `DeleteInputResponse`
- `GetAvailableEntitiesRequest` / `GetAvailableEntitiesResponse`
- `Info` (entity)
### Network Config
- `NetworkConfigCreateRequest` / `NetworkConfigCreateResponse`
- `NetworkConfigUpdateRequest` / `NetworkConfigUpdateResponse`
- `NetworkConfigDeleteRequest` / `NetworkConfigDeleteResponse`
- `NetworkConfigFindAllRequest` / `NetworkConfigFindAllResponse`
- `NetworkConfig` (entity)
### Connection Test
- `TestConnectionRequest` / `TestConnectionResponse`
### Transformation
- `CreateTransformationRequest` / `CreateTransformationResponse`
- `UpdateTransformationRequest` / `UpdateTransformationResponse`
- `DeleteTransformationRequest` / `DeleteTransformationResponse`
---
## NEXT STEPS
1. **Decide on migration strategy** (Option A, B, or C)
2. **Prioritize modules** - Recommend starting with Connection Test (smallest), then Inputs, Network Config, Transformations, Connection, Connector (largest)
3. **Create database schema** if moving to Option A
4. **Implement local services** one module at a time
5. **Update tests** for each migrated module
6. **Update deployment configs** to remove INFACTORY_URL
7. **Coordinate with In-Factory team** for data migration
MIGRATION_MAP_INFACTORY_SIDE.md
+320
View File
@@ -0,0 +1,320 @@
# In-Factory Side - Migration Requirements
This document outlines what needs to be addressed in the **In-Factory** service to remove its coupling with Maestro.
> **Note:** This analysis is based on the Maestro codebase. For a complete analysis, the In-Factory repository should also be reviewed.
---
## Current Architecture (In-Factory → Maestro)
Based on the Maestro codebase analysis, **In-Factory** exposes the following gRPC services that Maestro consumes:
```mermaid
graph LR;
Maestro -->|gRPC| InFactory;
subgraph InFactory Services
CM[ConnectionManager]
ConM[ConnectorManager]
IN[Input]
NC[NetworkConfig]
CT[ConnectionTest]
TR[Transformation]
end
Maestro --> CM;
Maestro --> ConM;
Maestro --> IN;
Maestro --> NC;
Maestro --> CT;
Maestro --> TR;
```
---
## gRPC Services Exposed by In-Factory
### 1. ConnectionManager Service
**Package:** `ConnectionManager` from `@dadosfera/protospack-v2`
#### Write Services (`ConnectionManagerWriteServices`)
| gRPC Method | Description | Called By |
|-------------|-------------|-----------|
| `CreateConnection` | Create a new data connection | Maestro POST /connections |
| `UpdateConnection` | Update an existing connection | Maestro PUT /connections/:id |
| `DeleteConnection` | Delete a connection | Maestro DELETE /connections/:id |
#### Read Services (`ConnectionManagerReadServices`)
| gRPC Method | Description | Called By |
|-------------|-------------|-----------|
| `GetConnectionDetails` | Get details of a single connection | Maestro GET /connections/:id |
| `GetAllConnections` | List all connections for a customer | Maestro GET /connections |
| `GetConnectorAvailableConnectionsByCustomer` | Get available connections by connector | Maestro GET /connections/available/:connector_id |
| `ValidatePlatformConnections` | Validate connections against platform | Maestro internal |
---
### 2. ConnectorManager Service
**Package:** `ConnectorManager` from `@dadosfera/protospack-v2`
#### Write Services (`ConnectorManagerWriteServices`)
| gRPC Method | Description | Called By |
|-------------|-------------|-----------|
| `RegisterConnector` | Register a new connector | Maestro POST /connectors |
| `UploadFile` | Upload connector files | Maestro POST /connectors/uploads |
| `RegisterMultipleConnectorsWithoutImage` | Bulk register connectors | Maestro internal |
| `UpdateConnectorByID` | Update a connector | Maestro PUT /connectors/:id |
| `DeleteConnectorById` | Delete a connector | Maestro DELETE /connectors/:id |
#### Read Services (`ConnectorManagerReadServices`)
| gRPC Method | Description | Called By |
|-------------|-------------|-----------|
| `GetConnectors` | List all connectors | Maestro GET /connectors |
---
### 3. Input Service
**Package:** `Input` from `@dadosfera/protospack-v2`
#### Write Services (`InputWriteService`)
| gRPC Method | Description | Called By |
|-------------|-------------|-----------|
| `CreateInput` | Create a new input configuration | Maestro POST /inputs |
| `UpdateInput` | Update input configuration | Maestro PATCH /inputs |
| `DeleteInput` | Delete an input | Maestro DELETE /inputs/:id |
#### Read Services (`InputReadService`)
| gRPC Method | Description | Called By |
|-------------|-------------|-----------|
| `GetAvailableEntities` | Get available entities for a plugin | Maestro GET /inputs/available-entities/:plugin |
---
### 4. NetworkConfig Service
**Package:** `NetworkConfig` from `@dadosfera/protospack-v2`
#### Write Services (`NetworkConfigWriteService`)
| gRPC Method | Description | Called By |
|-------------|-------------|-----------|
| `NetworkConfigCreate` | Create network configuration | Maestro POST /network-configs |
| `NetworkConfigUpdate` | Update network configuration | Maestro PUT /network-configs/:id |
| `NetworkConfigDelete` | Delete network configuration | Maestro DELETE /network-configs/:id |
#### Read Services (`NetworkConfigReadService`)
| gRPC Method | Description | Called By |
|-------------|-------------|-----------|
| `NetworkConfigFindAll` | List all network configs | Maestro GET /network-configs |
| `NetworkConfigFindOneById` | Get network config by ID | Maestro GET /network-configs/:id |
---
### 5. ConnectionTest Service
**Package:** `ConnectionTest` from `@dadosfera/protospack-v2`
#### Read Services (`ConnectionTestReadService`)
| gRPC Method | Description | Called By |
|-------------|-------------|-----------|
| `TestConnection` | Test a connection's connectivity | Maestro POST /connection-test |
---
### 6. Transformation Service
**Package:** `Transformation` from `@dadosfera/protospack-v2`
#### Write Services (`TransformationWriteService`)
| gRPC Method | Description | Called By |
|-------------|-------------|-----------|
| `CreateTransformation` | Create a transformation | Maestro POST /transformations |
| `UpdateTransformation` | Update a transformation | Maestro PUT /transformations/:id |
| `DeleteTransformation` | Delete a transformation | Maestro DELETE /transformations/:id |
#### Read Services (`TransformationReadService`)
| gRPC Method | Description | Called By |
|-------------|-------------|-----------|
| Various read operations | Get transformation details | Maestro GET /transformations |
---
## Proto Package Ownership
The proto definitions are managed in `@dadosfera/protospack-v2`:
```
@dadosfera/protospack-v2/
├── dist/lib/
│ ├── ConnectionManager/
│ │ ├── interfaces/messages.ts
│ │ └── interfaces/entities.ts
│ ├── ConnectorManager/
│ │ ├── interfaces/messages.ts
│ │ └── interfaces/entities.ts
│ ├── Input/
│ │ ├── interfaces/messages.ts
│ │ └── interfaces/entities.ts
│ ├── NetworkConfig/
│ │ ├── interfaces/messages.ts
│ │ └── interfaces/entities.ts
│ ├── ConnectionTest/
│ │ ├── interfaces/messages.ts
│ │ └── interfaces/entities.ts
│ └── Transformation/
│ ├── interfaces/messages.ts
│ └── interfaces/entities.ts
```
---
## In-Factory Migration Options
### Option 1: Expose REST API (Keep In-Factory, Add REST Layer)
**Changes Required in In-Factory:**
1. Add REST controllers for all services
2. Implement HTTP endpoints mirroring gRPC methods
3. Update deployment to expose HTTP port
4. Create OpenAPI documentation
**Pros:**
- Minimal architectural changes
- Can run both gRPC and REST in parallel during migration
- Maestro can switch to REST calls gradually
**Cons:**
- Still maintains service dependency
- Adds another communication layer
---
### Option 2: Move Logic to Maestro (Deprecate In-Factory for these features)
**Changes Required in In-Factory:**
1. Export database schema/migrations
2. Document all business logic
3. Provide data migration scripts
4. Deprecate gRPC endpoints after migration
**Changes Required in Maestro:**
1. Create database models
2. Implement repositories
3. Create service layer with same business logic
4. Run data migration
**Pros:**
- Removes runtime dependency completely
- Simplifies architecture
- One less service to maintain
**Cons:**
- Significant development effort
- Risk of business logic divergence during migration
- Database sharing concerns
---
### Option 3: Merge Services (Combine In-Factory into a larger service)
**Changes Required:**
1. Create new combined service
2. Migrate both In-Factory and relevant Maestro code
3. Update all clients
**Pros:**
- Clean architectural redesign
- Opportunity to optimize
**Cons:**
- Largest effort
- Risk of disruption
---
## Data Migration Considerations
### Entities Managed by In-Factory
Based on proto definitions, In-Factory manages:
1. **Connections**
- Connection credentials
- Connection metadata
- Customer associations
2. **Connectors**
- Connector definitions
- Connector images/files
- Plugin configurations
3. **Inputs**
- Input configurations
- Entity mappings
4. **Network Configs**
- Network configuration settings
- Security settings
5. **Transformations**
- Transformation definitions
- Transformation scripts
### Migration Steps
1. Export database schema from In-Factory
2. Create equivalent schema in target database
3. Write data migration scripts
4. Validate data integrity
5. Switch traffic
6. Decommission old service
---
## Recommended Investigation for In-Factory Team
1. **Check for Maestro dependencies in In-Factory**
- Does In-Factory call any Maestro APIs?
- Are there any shared databases?
- Any shared message queues?
2. **Document database schema**
- All tables related to connections, connectors, inputs, network-configs, transformations
- Foreign key relationships
- Indexes and constraints
3. **List all consumers**
- Besides Maestro, who else calls In-Factory?
- Are there other internal services?
- Any external integrations?
4. **Business logic documentation**
- Validation rules
- Business constraints
- Side effects (events, notifications, etc.)
---
## Timeline Considerations
| Phase | Description | Dependencies |
|-------|-------------|--------------|
| Phase 1 | Analysis & Planning | Both teams available |
| Phase 2 | Schema/API Design | Proto definitions finalized |
| Phase 3 | Implementation | Development resources |
| Phase 4 | Data Migration | Database access, downtime window |
| Phase 5 | Testing | QA resources, test environments |
| Phase 6 | Cutover | Deployment coordination |
| Phase 7 | Decommission | Monitoring, rollback plan |
---
## Questions for In-Factory Team
1. What database does In-Factory use? (PostgreSQL, MongoDB, etc.)
2. Are there any async operations? (message queues, event sourcing)
3. What is the current data volume for each entity type?
4. Are there any scheduled jobs or background processes?
5. What monitoring/alerting is in place?
6. Are there any data retention policies?
7. What is the backup/recovery strategy?