Add comprehensive documentation mapping all integration points between maestro and in-factory services to facilitate removing the dependency. Includes: - All 6 gRPC client configurations using INFACTORY_URL - 27+ gRPC method calls across 6 services - 25 REST endpoints that proxy to in-factory - Configuration files and environment variables - Proto package dependencies - Migration strategy options and recommendations
9.8 KiB
In-Factory Side - Migration Requirements
This document outlines what needs to be addressed in the In-Factory service to remove its coupling with Maestro.
Note: This analysis is based on the Maestro codebase. For a complete analysis, the In-Factory repository should also be reviewed.
Current Architecture (In-Factory → Maestro)
Based on the Maestro codebase analysis, In-Factory exposes the following gRPC services that Maestro consumes:
graph LR;
Maestro -->|gRPC| InFactory;
subgraph InFactory Services
CM[ConnectionManager]
ConM[ConnectorManager]
IN[Input]
NC[NetworkConfig]
CT[ConnectionTest]
TR[Transformation]
end
Maestro --> CM;
Maestro --> ConM;
Maestro --> IN;
Maestro --> NC;
Maestro --> CT;
Maestro --> TR;
gRPC Services Exposed by In-Factory
1. ConnectionManager Service
Package: ConnectionManager from @dadosfera/protospack-v2
Write Services (ConnectionManagerWriteServices)
| gRPC Method | Description | Called By |
|---|---|---|
CreateConnection |
Create a new data connection | Maestro POST /connections |
UpdateConnection |
Update an existing connection | Maestro PUT /connections/:id |
DeleteConnection |
Delete a connection | Maestro DELETE /connections/:id |
Read Services (ConnectionManagerReadServices)
| gRPC Method | Description | Called By |
|---|---|---|
GetConnectionDetails |
Get details of a single connection | Maestro GET /connections/:id |
GetAllConnections |
List all connections for a customer | Maestro GET /connections |
GetConnectorAvailableConnectionsByCustomer |
Get available connections by connector | Maestro GET /connections/available/:connector_id |
ValidatePlatformConnections |
Validate connections against platform | Maestro internal |
2. ConnectorManager Service
Package: ConnectorManager from @dadosfera/protospack-v2
Write Services (ConnectorManagerWriteServices)
| gRPC Method | Description | Called By |
|---|---|---|
RegisterConnector |
Register a new connector | Maestro POST /connectors |
UploadFile |
Upload connector files | Maestro POST /connectors/uploads |
RegisterMultipleConnectorsWithoutImage |
Bulk register connectors | Maestro internal |
UpdateConnectorByID |
Update a connector | Maestro PUT /connectors/:id |
DeleteConnectorById |
Delete a connector | Maestro DELETE /connectors/:id |
Read Services (ConnectorManagerReadServices)
| gRPC Method | Description | Called By |
|---|---|---|
GetConnectors |
List all connectors | Maestro GET /connectors |
3. Input Service
Package: Input from @dadosfera/protospack-v2
Write Services (InputWriteService)
| gRPC Method | Description | Called By |
|---|---|---|
CreateInput |
Create a new input configuration | Maestro POST /inputs |
UpdateInput |
Update input configuration | Maestro PATCH /inputs |
DeleteInput |
Delete an input | Maestro DELETE /inputs/:id |
Read Services (InputReadService)
| gRPC Method | Description | Called By |
|---|---|---|
GetAvailableEntities |
Get available entities for a plugin | Maestro GET /inputs/available-entities/:plugin |
4. NetworkConfig Service
Package: NetworkConfig from @dadosfera/protospack-v2
Write Services (NetworkConfigWriteService)
| gRPC Method | Description | Called By |
|---|---|---|
NetworkConfigCreate |
Create network configuration | Maestro POST /network-configs |
NetworkConfigUpdate |
Update network configuration | Maestro PUT /network-configs/:id |
NetworkConfigDelete |
Delete network configuration | Maestro DELETE /network-configs/:id |
Read Services (NetworkConfigReadService)
| gRPC Method | Description | Called By |
|---|---|---|
NetworkConfigFindAll |
List all network configs | Maestro GET /network-configs |
NetworkConfigFindOneById |
Get network config by ID | Maestro GET /network-configs/:id |
5. ConnectionTest Service
Package: ConnectionTest from @dadosfera/protospack-v2
Read Services (ConnectionTestReadService)
| gRPC Method | Description | Called By |
|---|---|---|
TestConnection |
Test a connection's connectivity | Maestro POST /connection-test |
6. Transformation Service
Package: Transformation from @dadosfera/protospack-v2
Write Services (TransformationWriteService)
| gRPC Method | Description | Called By |
|---|---|---|
CreateTransformation |
Create a transformation | Maestro POST /transformations |
UpdateTransformation |
Update a transformation | Maestro PUT /transformations/:id |
DeleteTransformation |
Delete a transformation | Maestro DELETE /transformations/:id |
Read Services (TransformationReadService)
| gRPC Method | Description | Called By |
|---|---|---|
| Various read operations | Get transformation details | Maestro GET /transformations |
Proto Package Ownership
The proto definitions are managed in @dadosfera/protospack-v2:
@dadosfera/protospack-v2/
├── dist/lib/
│ ├── ConnectionManager/
│ │ ├── interfaces/messages.ts
│ │ └── interfaces/entities.ts
│ ├── ConnectorManager/
│ │ ├── interfaces/messages.ts
│ │ └── interfaces/entities.ts
│ ├── Input/
│ │ ├── interfaces/messages.ts
│ │ └── interfaces/entities.ts
│ ├── NetworkConfig/
│ │ ├── interfaces/messages.ts
│ │ └── interfaces/entities.ts
│ ├── ConnectionTest/
│ │ ├── interfaces/messages.ts
│ │ └── interfaces/entities.ts
│ └── Transformation/
│ ├── interfaces/messages.ts
│ └── interfaces/entities.ts
In-Factory Migration Options
Option 1: Expose REST API (Keep In-Factory, Add REST Layer)
Changes Required in In-Factory:
- Add REST controllers for all services
- Implement HTTP endpoints mirroring gRPC methods
- Update deployment to expose HTTP port
- Create OpenAPI documentation
Pros:
- Minimal architectural changes
- Can run both gRPC and REST in parallel during migration
- Maestro can switch to REST calls gradually
Cons:
- Still maintains service dependency
- Adds another communication layer
Option 2: Move Logic to Maestro (Deprecate In-Factory for these features)
Changes Required in In-Factory:
- Export database schema/migrations
- Document all business logic
- Provide data migration scripts
- Deprecate gRPC endpoints after migration
Changes Required in Maestro:
- Create database models
- Implement repositories
- Create service layer with same business logic
- Run data migration
Pros:
- Removes runtime dependency completely
- Simplifies architecture
- One less service to maintain
Cons:
- Significant development effort
- Risk of business logic divergence during migration
- Database sharing concerns
Option 3: Merge Services (Combine In-Factory into a larger service)
Changes Required:
- Create new combined service
- Migrate both In-Factory and relevant Maestro code
- Update all clients
Pros:
- Clean architectural redesign
- Opportunity to optimize
Cons:
- Largest effort
- Risk of disruption
Data Migration Considerations
Entities Managed by In-Factory
Based on proto definitions, In-Factory manages:
-
Connections
- Connection credentials
- Connection metadata
- Customer associations
-
Connectors
- Connector definitions
- Connector images/files
- Plugin configurations
-
Inputs
- Input configurations
- Entity mappings
-
Network Configs
- Network configuration settings
- Security settings
-
Transformations
- Transformation definitions
- Transformation scripts
Migration Steps
- Export database schema from In-Factory
- Create equivalent schema in target database
- Write data migration scripts
- Validate data integrity
- Switch traffic
- Decommission old service
Recommended Investigation for In-Factory Team
-
Check for Maestro dependencies in In-Factory
- Does In-Factory call any Maestro APIs?
- Are there any shared databases?
- Any shared message queues?
-
Document database schema
- All tables related to connections, connectors, inputs, network-configs, transformations
- Foreign key relationships
- Indexes and constraints
-
List all consumers
- Besides Maestro, who else calls In-Factory?
- Are there other internal services?
- Any external integrations?
-
Business logic documentation
- Validation rules
- Business constraints
- Side effects (events, notifications, etc.)
Timeline Considerations
| Phase | Description | Dependencies |
|---|---|---|
| Phase 1 | Analysis & Planning | Both teams available |
| Phase 2 | Schema/API Design | Proto definitions finalized |
| Phase 3 | Implementation | Development resources |
| Phase 4 | Data Migration | Database access, downtime window |
| Phase 5 | Testing | QA resources, test environments |
| Phase 6 | Cutover | Deployment coordination |
| Phase 7 | Decommission | Monitoring, rollback plan |
Questions for In-Factory Team
- What database does In-Factory use? (PostgreSQL, MongoDB, etc.)
- Are there any async operations? (message queues, event sourcing)
- What is the current data volume for each entity type?
- Are there any scheduled jobs or background processes?
- What monitoring/alerting is in place?
- Are there any data retention policies?
- What is the backup/recovery strategy?