DOCS: Map maestro-pifactory integration points for migration

Add comprehensive documentation mapping all integration points between
maestro and pi-factory services to facilitate removing the dependency.

Includes:
- 3 gRPC client configurations using PIFACTORY_URL
- 36+ gRPC method calls across Catalog, PipelineV2, and legacy Pipeline services
- 44 REST endpoints that proxy to pi-factory
- Configuration files and environment variables
- Proto package dependencies (protospack and protospack-v2)
- Migration strategy options and recommendations
- Comparison with in-factory migration scope
This commit is contained in:
Claude
2025-12-17 18:21:38 +00:00
parent 177ec6368c
commit e0bfe03898
+486
View File
@@ -0,0 +1,486 @@
# Maestro ↔ PI-Factory Migration Map
This document maps all integration points between Maestro and PI-Factory that need to be addressed to remove the dependency.
## Architecture Overview
```mermaid
graph TD;
Frontend<-->Maestro;
Maestro<-->DUC;
Maestro<-->pi-factory;
Maestro<-->in-factory;
```
PI-Factory (`PIFACTORY_URL`) is responsible for:
- **Pipeline Management** - Create, read, update, delete pipelines
- **Catalog Services** - Data asset management, metadata, previews
- **Platform Interfaces** - Dataset cataloging operations
---
## Summary Table
| Category | Count | Impact Level |
|----------|-------|--------------|
| gRPC Client Configurations | 3 | HIGH |
| NestJS Modules | 3 | HIGH |
| REST Controllers/Endpoints | 3 (~50 endpoints) | HIGH |
| Service Classes | 4 | HIGH |
| Configuration Files | 4 | MEDIUM |
| Proto Package Dependencies | 2 | HIGH |
| Environment Variables | 1 | LOW |
---
## 1. gRPC CLIENT CONFIGURATIONS (Files to Migrate)
These files configure gRPC connections to PI-Factory services:
### 1.1 Catalog Client (`PIFACTORY_URL`)
**File:** `src/modules/catalog/catalog-client.ts`
```typescript
// Lines 11-12, 19
process.env.PIFACTORY_URL.startsWith('pi-factory:')
process.env.PIFACTORY_URL.includes('0.0.0.0')
url: process.env.PIFACTORY_URL
```
**Proto Services Used:**
- `Catalog.ProtoPackages.ReadPackage`
- `Catalog.ProtoPackages.WritePackage`
- `PlatformInterfaces.ProtoPackages.WritePackage`
### 1.2 Pipelines V2 Client (`PIFACTORY_URL`)
**File:** `src/modules/pipelinesV2/pipelines-client.ts`
```typescript
// Lines 13-14, 21
url: process.env.PIFACTORY_URL
```
**Proto Services Used:**
- `PipelineV2.ProtoPackages.ReadPackage`
- `PipelineV2.ProtoPackages.WritePackage`
### 1.3 Pipelines Client (Legacy) (`PIFACTORY_URL`)
**File:** `src/modules/pipelines/pipelines-client.ts`
```typescript
// Lines 10-11, 18
url: process.env.PIFACTORY_URL
```
**Proto Services Used:**
- `PipelinePackages` from `@dadosfera/protospack`
---
## 2. SERVICE CLASSES (Business Logic to Migrate)
### 2.1 Catalog Service
**File:** `src/modules/catalog/catalog.service.ts`
**gRPC Services Initialized:**
- `CatalogReadServices` (from `Catalog.ReadService`)
- `CatalogWriteServices` (from `Catalog.WriteService`)
- `PlatformInterfacesWriteServices` (from `PlatformInterfaces.WriteService`)
**gRPC Methods Called:**
| Method | Service | Description |
|--------|---------|-------------|
| `GetAllDataAssets()` | CatalogReadServices | Search/list data assets |
| `GetOneDataAsset()` | CatalogReadServices | Get single data asset by ID |
| `GetOneDataAssetByPipelineAndObject()` | CatalogReadServices | Get asset by pipeline/object |
| `GetDatasetDoc()` | CatalogReadServices | Get dataset documentation |
| `GetDatasetPreview()` | CatalogReadServices | Get data preview |
| `GetDatasetColumnsMetadata()` | CatalogReadServices | Get column metadata |
| `GetCustomerTags()` | CatalogReadServices | Get all tags for customer |
| `GetDatasetCatalogTask()` | CatalogReadServices | Get catalog task status |
| `GetRlsRules()` | CatalogReadServices | Get RLS rules |
| `GetOneRlsRule()` | CatalogReadServices | Get single RLS rule |
| `GetNimbusDashboards()` | CatalogReadServices | Get Nimbus dashboards |
| `CreateDataAsset()` | CatalogWriteServices | Create new data asset |
| `UpdateDataAsset()` | CatalogWriteServices | Update existing data asset |
| `DeleteDataAsset()` | CatalogWriteServices | Delete data asset |
| `ManagePermission()` | CatalogWriteServices | Manage asset permissions |
| `RevokePermission()` | CatalogWriteServices | Revoke asset permissions |
| `MakeAComment()` | CatalogWriteServices | Add comment to asset |
| `UpdateAComment()` | CatalogWriteServices | Update/delete comment |
| `TriggerDatasetCataloging()` | CatalogWriteServices | Trigger catalog process |
| `AddRlsRule()` | CatalogWriteServices | Add RLS rule |
| `RemoveRlsRule()` | CatalogWriteServices | Remove RLS rule |
| `RemoveRlsRulesByRlsId()` | CatalogWriteServices | Batch remove by RLS ID |
| `RemoveRlsRulesByDashboardId()` | CatalogWriteServices | Batch remove by dashboard |
| `GetPiiReporter()` | CatalogWriteServices | Get PII report data |
| `CatalogDataAssets()` | PlatformInterfacesWriteServices | Catalog datasets |
**Additional HTTP Calls to Nimbus:**
- `POST ${nimbusUrl}/api/catalog/data-docs/` - Create data docs
- `POST ${nimbusUrl}/api/catalog/table-metadata/` - Create table metadata
- `POST ${nimbusUrl}/api/catalog/column-metadata/` - Create column metadata
- `POST ${nimbusUrl}/api/catalog/data-preview/` - Create data preview
### 2.2 Pipelines V2 Service
**File:** `src/modules/pipelinesV2/pipelines.service.ts`
**gRPC Services Initialized:**
- `PipelineV2ReadService` (from `PipelineV2.ReadService`)
- `PipelineV2WriteService` (from `PipelineV2.WriteService`)
**gRPC Methods Called:**
| Method | Service | Description |
|--------|---------|-------------|
| `PipelineV2Create()` | PipelineV2WriteService | Create new pipeline |
| `PipelineV2Update()` | PipelineV2WriteService | Update pipeline |
| `PipelineV2Remove()` | PipelineV2WriteService | Delete pipeline |
| `PipelineV2UploadFile()` | PipelineV2WriteService | Initialize file upload |
| `PipelineV2CompleteUploadFile()` | PipelineV2WriteService | Complete file upload |
| `PipelineV2FindAll()` | PipelineV2ReadService | List all pipelines |
| `PipelineV2FindOne()` | PipelineV2ReadService | Get single pipeline |
| `PipelineV2FindObjects()` | PipelineV2ReadService | Get pipeline objects |
| `PipelineV2DownloadLogs()` | PipelineV2ReadService | Download pipeline logs |
| `PipelineV2GetDashboardUrl()` | PipelineV2ReadService | Get monitoring dashboard URL |
### 2.3 Pipelines Service (Legacy)
**File:** `src/modules/pipelines/pipelines.service.ts`
**Uses:** `PipelinesClientService`
**Methods:**
- `getPipelineStatus()` - Get pipeline execution status
- `runPipeline()` - Trigger pipeline execution
### 2.4 Pipelines Client Service (Legacy)
**File:** `src/modules/pipelines/client.service.ts`
**gRPC Methods Called:**
| Method | Service | Description |
|--------|---------|-------------|
| `getPipelineStatus()` | PipelineService | Get pipeline status |
| `triggerPipeline()` | PipelineService | Trigger pipeline run |
---
## 3. NESTJS MODULES (Module Registration)
| Module | File | Client Configuration | Exports |
|--------|------|---------------------|---------|
| CatalogModule | `src/modules/catalog/catalog.module.ts` | CatalogClientConfiguration | CatalogService |
| PipelinesV2Module | `src/modules/pipelinesV2/pipelines.module.ts` | PipelinesClientConfiguration | PipelinesService |
| PipelinesModule | `src/modules/pipelines/pipelines.module.ts` | PipelinesClientConfiguration | PipelinesService, PipelinesClientService |
**App Module Registration:** `src/app.module.ts` (Lines 20, 23, 25, 53-54, 59)
---
## 4. REST CONTROLLERS (API Endpoints to Migrate)
### 4.1 Catalog Controller
**File:** `src/modules/catalog/catalog.controller.ts`
**Base Path:** `/catalog`
| HTTP Method | Endpoint | Description |
|-------------|----------|-------------|
| GET | `/catalog` | Search data assets |
| GET | `/catalog/download` | Download assets as CSV |
| GET | `/catalog/data-asset` | Get asset by pipeline/object |
| GET | `/catalog/data-asset/:id` | Get single data asset |
| GET | `/catalog/data-asset/rls/:id` | Get data asset RLS info |
| GET | `/catalog/data-asset/:id/columns-metadata` | Get column metadata |
| GET | `/catalog/data-asset/:id/preview` | Get data preview |
| GET | `/catalog/data-asset/:id/docs` | Get documentation |
| GET | `/catalog/tags` | Get all tags |
| PUT | `/catalog/data-asset/:id` | Update data asset |
| PUT | `/catalog/data-asset/:id/manage-permissions` | Manage permissions |
| PUT | `/catalog/data-asset/:id/revoke-permissions` | Revoke permissions |
| POST | `/catalog` | Create data asset |
| POST | `/catalog/data-asset/:id/docs` | Create documentation |
| POST | `/catalog/data-asset/:id/comment` | Add comment |
| POST | `/catalog/dataset-catalog-task` | Trigger catalog task |
| POST | `/catalog/rls-rule` | Add RLS rule |
| POST | `/catalog/register-dataset` | Register dataset with metadata |
| DELETE | `/catalog/data-asset/:id` | Delete data asset |
| DELETE | `/catalog/data-asset/:id/comment` | Delete comment |
| DELETE | `/catalog/rls-rule/:id` | Remove RLS rule |
| DELETE | `/catalog/rls-rule` | Batch remove RLS rules |
| GET | `/catalog/dataset-catalog-task/:session` | Get catalog task status |
| GET | `/catalog/rls-rule/:id` | Get single RLS rule |
| GET | `/catalog/rls-rule` | Get RLS rules |
| GET | `/catalog/nimbus-dashboards` | Get Nimbus dashboards |
| GET | `/catalog/pii-reporter` | Get PII report |
**Total: 27 endpoints**
### 4.2 Pipelines V2 Controller
**File:** `src/modules/pipelinesV2/pipelines.controller.ts`
**Base Path:** `/pipelinesV2`
| HTTP Method | Endpoint | Description |
|-------------|----------|-------------|
| GET | `/pipelinesV2/monitoring-dashboard` | Get monitoring dashboard URL |
| GET | `/pipelinesV2` | List all pipelines |
| GET | `/pipelinesV2/download-logs` | Download pipeline logs |
| GET | `/pipelinesV2/:id` | Get single pipeline |
| GET | `/pipelinesV2/:id/config` | Get pipeline properties |
| GET | `/pipelinesV2/:id/objects` | Get pipeline objects |
| GET | `/pipelinesV2/:id/status` | Get pipeline status (legacy) |
| POST | `/pipelinesV2` | Create pipeline |
| POST | `/pipelinesV2/init-upload` | Initialize file upload |
| POST | `/pipelinesV2/complete-upload` | Complete file upload |
| POST | `/pipelinesV2/file` | Upload file pipeline |
| POST | `/pipelinesV2/start/:id` | Start pipeline |
| PATCH | `/pipelinesV2/:id` | Update pipeline |
| PUT | `/pipelinesV2/:id` | Update pipeline (deprecated) |
| DELETE | `/pipelinesV2/:id` | Delete pipeline |
**Total: 15 endpoints**
### 4.3 Pipelines Controller (Legacy)
**File:** `src/modules/pipelines/pipelines.controller.ts`
**Base Path:** `/pipelines`
| HTTP Method | Endpoint | Description |
|-------------|----------|-------------|
| POST | `/pipelines/start/:id` | Start pipeline (deprecated) |
| GET | `/pipelines/:id/status` | Get pipeline status (deprecated) |
**Total: 2 endpoints (deprecated)**
---
## 5. CONFIGURATION FILES
### 5.1 Helm Chart Values (Production)
**File:** `deploy/helm-chart/values.yaml`
```yaml
# Line 40
maestro:
pi_factory_url: pi-factory.dadosfera.ai
```
### 5.2 Helm Chart Values (Staging)
**File:** `deploy/helm-chart/values-stg.yaml`
```yaml
# Line 4
maestro:
pi_factory_url: pi-factory.stg.dadosfera.ai
```
### 5.3 Helmfiles (Production)
**File:** `deploy/helmfiles/prd.yaml`
```yaml
# Lines 14, 38
maestro.pi_factory_url: pi-factory.dadosfera.ai
```
### 5.4 Deployment Template
**File:** `deploy/helm-chart/templates/deployment.yaml`
```yaml
# Line 85
- name: PIFACTORY_URL
value: {{ .Values.maestro.pi_factory_url }}
```
### 5.5 Environment Type Definition
**File:** `environment.d.ts`
```typescript
// Line 10
PIFACTORY_URL: string;
```
---
## 6. PROTO PACKAGE DEPENDENCIES
### 6.1 Package.json
**File:** `package.json`
```json
{
"@dadosfera/protospack": "2.5.3", // Legacy pipelines
"@dadosfera/protospack-v2": "3.38.0-beta.14" // PipelineV2, Catalog, PlatformInterfaces
}
```
### 6.2 Proto Imports from `@dadosfera/protospack-v2` (PI-Factory related)
| Import Path | Used In |
|-------------|---------|
| `Catalog` | catalog/catalog-client.ts, catalog/catalog.service.ts |
| `Catalog/interfaces/messages` | catalog/catalog.service.ts, catalog/catalog.controller.ts, catalog/dtos |
| `PlatformInterfaces` | catalog/catalog-client.ts, catalog/catalog.service.ts |
| `PipelineV2` | pipelinesV2/pipelines-client.ts, pipelinesV2/pipelines.service.ts |
| `PipelineV2/interfaces/messages` | pipelinesV2/pipelines.service.ts, pipelinesV2/pipelines.controller.ts |
### 6.3 Proto Imports from `@dadosfera/protospack` (Legacy)
| Import Path | Used In |
|-------------|---------|
| `PipelinePackages` | pipelines/pipelines-client.ts |
| `PipelineProtoFilePath` | pipelines/pipelines-client.ts |
| `PipelineServicesNames` | pipelines/client.service.ts |
| `PipelinesServiceInterface` | pipelines/client.service.ts |
---
## 7. FILES TO MODIFY/DELETE (Summary)
### High Priority - Core Integration Files
```
src/modules/catalog/catalog-client.ts → DELETE or REPLACE
src/modules/catalog/catalog.service.ts → REPLACE with local implementation
src/modules/pipelinesV2/pipelines-client.ts → DELETE or REPLACE
src/modules/pipelinesV2/pipelines.service.ts → REPLACE with local implementation
src/modules/pipelines/pipelines-client.ts → DELETE or REPLACE
src/modules/pipelines/client.service.ts → REPLACE with local implementation
src/modules/pipelines/pipelines.service.ts → REPLACE with local implementation
```
### Medium Priority - Module Registration
```
src/modules/catalog/catalog.module.ts → UPDATE imports
src/modules/pipelinesV2/pipelines.module.ts → UPDATE imports
src/modules/pipelines/pipelines.module.ts → UPDATE imports
src/app.module.ts → UPDATE if module structure changes
```
### Low Priority - Configuration
```
deploy/helm-chart/values.yaml → REMOVE pi_factory_url
deploy/helm-chart/values-stg.yaml → REMOVE pi_factory_url
deploy/helmfiles/prd.yaml → REMOVE pi_factory_url references
deploy/helm-chart/templates/deployment.yaml → REMOVE PIFACTORY_URL env var
environment.d.ts → REMOVE PIFACTORY_URL type
README.md → UPDATE architecture diagram
```
---
## 8. DEPENDENCY COUNT BY MODULE
| Module | Files | gRPC Calls | REST Endpoints |
|--------|-------|------------|----------------|
| Catalog | 3 | 24+ | 27 |
| PipelinesV2 | 3 | 10 | 15 |
| Pipelines (Legacy) | 3 | 2 | 2 |
| **TOTAL** | **9** | **36+** | **44** |
---
## 9. COMPARISON: PI-Factory vs In-Factory
| Aspect | PI-Factory | In-Factory |
|--------|------------|------------|
| Environment Variable | `PIFACTORY_URL` | `INFACTORY_URL` |
| Modules | 3 | 6 |
| gRPC Calls | 36+ | 27+ |
| REST Endpoints | 44 | 25 |
| Complexity | HIGH | MEDIUM-HIGH |
| Domain | Pipelines, Catalog | Connections, Connectors, Inputs |
---
## 10. DATA MODELS (Proto Messages Used)
### Catalog Messages
- `CreateDataAssetRequest` / `CreateDataAssetResponse`
- `GetAllDataAssetsRequest` / `GetAllDataAssetsResponse`
- `GetOneDataAssetRequest` / Response
- `UpdateDataAssetRequest` / Response
- `DeleteDataAssetRequest` / Response
- `ManagePermissionRequest` / Response
- `RevokePermissionRequest` / Response
- `MakeACommentRequest` / Response
- `UpdateACommentRequest` / Response
- `TriggerDatasetCatalogingRequest` / Response
- `GetDatasetCatalogTaskRequest` / Response
- `AddRlsRuleRequest` / Response
- `RemoveRlsRuleRequest` / Response
- `GetRlsRulesRequest` / Response
- `GetNimbusDashboardsRequest` / Response
- `PiiMetadata`
- `RegisterDatasetWithMetatadaRequest`
### PipelineV2 Messages
- `PipelineV2CreateRequest` / `PipelineV2CreateResponse`
- `PipelineV2FindAllRequest` / `PipelineV2FindAllResponse`
- `PipelineV2FindOneRequest` / `PipelineV2FindOneResponse`
- `PipelineV2UpdateRequest` / `PipelineV2UpdateResponse`
- `PipelineV2RemoveRequest` / Response
- `PipelineV2UploadFileRequest` / Response
- `PipelineV2CompleteUploadFileRequest` / Response
- `PipelineV2FindObjectsRequest` / Response
- `PipelineV2DownloadLogsRequest` / Response
- `PipelineV2GetDashboardUrlRequest` / Response
### Platform Interfaces Messages
- `CatalogDataAssetsRequest` / Response
---
## 11. MIGRATION STRATEGY OPTIONS
### Option A: Move PI-Factory functionality INTO Maestro
**Pros:**
- Single service to maintain
- No network latency for these operations
- Simpler deployment
**Cons:**
- Significantly increases Maestro's responsibility
- Requires database access from Maestro
- Large refactoring effort (44 endpoints)
### Option B: Create REST API wrapper in PI-Factory
**Pros:**
- Minimal changes to Maestro
- Can migrate incrementally
**Cons:**
- Still maintains dependency
- Additional REST→gRPC translation layer
### Option C: Direct database access from Maestro
**Pros:**
- Removes runtime dependency
- Better performance
**Cons:**
- Shared database coupling
- Complex migration
---
## 12. RECOMMENDED MIGRATION ORDER
Given the complexity, we recommend migrating in this order:
1. **Legacy Pipelines** (2 endpoints, deprecated) - Lowest risk
2. **PipelinesV2** (15 endpoints) - Core pipeline functionality
3. **Catalog** (27 endpoints) - Most complex, migrate last
### Phase 1: Legacy Pipelines (Deprecated)
- Remove `/pipelines/start/:id`
- Remove `/pipelines/:id/status`
- Update all clients to use `/pipelinesV2/*` endpoints
### Phase 2: PipelinesV2
- Migrate pipeline CRUD operations
- Migrate file upload functionality
- Migrate monitoring dashboard
### Phase 3: Catalog
- Migrate data asset CRUD
- Migrate permissions management
- Migrate RLS rules
- Migrate dataset registration
- Migrate PII reporting
---
## NEXT STEPS
1. **Decide on migration strategy** (Option A, B, or C)
2. **Deprecate legacy pipelines module** first
3. **Create database schema** if moving to Option A
4. **Implement local services** one module at a time
5. **Update tests** for each migrated module
6. **Update deployment configs** to remove PIFACTORY_URL
7. **Coordinate with PI-Factory team** for data migration