diff --git a/MIGRATION_MAP_PIFACTORY.md b/MIGRATION_MAP_PIFACTORY.md new file mode 100644 index 0000000..a99ed17 --- /dev/null +++ b/MIGRATION_MAP_PIFACTORY.md @@ -0,0 +1,486 @@ +# Maestro ↔ PI-Factory Migration Map + +This document maps all integration points between Maestro and PI-Factory that need to be addressed to remove the dependency. + +## Architecture Overview + +```mermaid +graph TD; + Frontend<-->Maestro; + Maestro<-->DUC; + Maestro<-->pi-factory; + Maestro<-->in-factory; +``` + +PI-Factory (`PIFACTORY_URL`) is responsible for: +- **Pipeline Management** - Create, read, update, delete pipelines +- **Catalog Services** - Data asset management, metadata, previews +- **Platform Interfaces** - Dataset cataloging operations + +--- + +## Summary Table + +| Category | Count | Impact Level | +|----------|-------|--------------| +| gRPC Client Configurations | 3 | HIGH | +| NestJS Modules | 3 | HIGH | +| REST Controllers/Endpoints | 3 (~50 endpoints) | HIGH | +| Service Classes | 4 | HIGH | +| Configuration Files | 4 | MEDIUM | +| Proto Package Dependencies | 2 | HIGH | +| Environment Variables | 1 | LOW | + +--- + +## 1. gRPC CLIENT CONFIGURATIONS (Files to Migrate) + +These files configure gRPC connections to PI-Factory services: + +### 1.1 Catalog Client (`PIFACTORY_URL`) +**File:** `src/modules/catalog/catalog-client.ts` +```typescript +// Lines 11-12, 19 +process.env.PIFACTORY_URL.startsWith('pi-factory:') +process.env.PIFACTORY_URL.includes('0.0.0.0') +url: process.env.PIFACTORY_URL +``` +**Proto Services Used:** +- `Catalog.ProtoPackages.ReadPackage` +- `Catalog.ProtoPackages.WritePackage` +- `PlatformInterfaces.ProtoPackages.WritePackage` + +### 1.2 Pipelines V2 Client (`PIFACTORY_URL`) +**File:** `src/modules/pipelinesV2/pipelines-client.ts` +```typescript +// Lines 13-14, 21 +url: process.env.PIFACTORY_URL +``` +**Proto Services Used:** +- `PipelineV2.ProtoPackages.ReadPackage` +- `PipelineV2.ProtoPackages.WritePackage` + +### 1.3 Pipelines Client (Legacy) (`PIFACTORY_URL`) +**File:** `src/modules/pipelines/pipelines-client.ts` +```typescript +// Lines 10-11, 18 +url: process.env.PIFACTORY_URL +``` +**Proto Services Used:** +- `PipelinePackages` from `@dadosfera/protospack` + +--- + +## 2. SERVICE CLASSES (Business Logic to Migrate) + +### 2.1 Catalog Service +**File:** `src/modules/catalog/catalog.service.ts` + +**gRPC Services Initialized:** +- `CatalogReadServices` (from `Catalog.ReadService`) +- `CatalogWriteServices` (from `Catalog.WriteService`) +- `PlatformInterfacesWriteServices` (from `PlatformInterfaces.WriteService`) + +**gRPC Methods Called:** + +| Method | Service | Description | +|--------|---------|-------------| +| `GetAllDataAssets()` | CatalogReadServices | Search/list data assets | +| `GetOneDataAsset()` | CatalogReadServices | Get single data asset by ID | +| `GetOneDataAssetByPipelineAndObject()` | CatalogReadServices | Get asset by pipeline/object | +| `GetDatasetDoc()` | CatalogReadServices | Get dataset documentation | +| `GetDatasetPreview()` | CatalogReadServices | Get data preview | +| `GetDatasetColumnsMetadata()` | CatalogReadServices | Get column metadata | +| `GetCustomerTags()` | CatalogReadServices | Get all tags for customer | +| `GetDatasetCatalogTask()` | CatalogReadServices | Get catalog task status | +| `GetRlsRules()` | CatalogReadServices | Get RLS rules | +| `GetOneRlsRule()` | CatalogReadServices | Get single RLS rule | +| `GetNimbusDashboards()` | CatalogReadServices | Get Nimbus dashboards | +| `CreateDataAsset()` | CatalogWriteServices | Create new data asset | +| `UpdateDataAsset()` | CatalogWriteServices | Update existing data asset | +| `DeleteDataAsset()` | CatalogWriteServices | Delete data asset | +| `ManagePermission()` | CatalogWriteServices | Manage asset permissions | +| `RevokePermission()` | CatalogWriteServices | Revoke asset permissions | +| `MakeAComment()` | CatalogWriteServices | Add comment to asset | +| `UpdateAComment()` | CatalogWriteServices | Update/delete comment | +| `TriggerDatasetCataloging()` | CatalogWriteServices | Trigger catalog process | +| `AddRlsRule()` | CatalogWriteServices | Add RLS rule | +| `RemoveRlsRule()` | CatalogWriteServices | Remove RLS rule | +| `RemoveRlsRulesByRlsId()` | CatalogWriteServices | Batch remove by RLS ID | +| `RemoveRlsRulesByDashboardId()` | CatalogWriteServices | Batch remove by dashboard | +| `GetPiiReporter()` | CatalogWriteServices | Get PII report data | +| `CatalogDataAssets()` | PlatformInterfacesWriteServices | Catalog datasets | + +**Additional HTTP Calls to Nimbus:** +- `POST ${nimbusUrl}/api/catalog/data-docs/` - Create data docs +- `POST ${nimbusUrl}/api/catalog/table-metadata/` - Create table metadata +- `POST ${nimbusUrl}/api/catalog/column-metadata/` - Create column metadata +- `POST ${nimbusUrl}/api/catalog/data-preview/` - Create data preview + +### 2.2 Pipelines V2 Service +**File:** `src/modules/pipelinesV2/pipelines.service.ts` + +**gRPC Services Initialized:** +- `PipelineV2ReadService` (from `PipelineV2.ReadService`) +- `PipelineV2WriteService` (from `PipelineV2.WriteService`) + +**gRPC Methods Called:** + +| Method | Service | Description | +|--------|---------|-------------| +| `PipelineV2Create()` | PipelineV2WriteService | Create new pipeline | +| `PipelineV2Update()` | PipelineV2WriteService | Update pipeline | +| `PipelineV2Remove()` | PipelineV2WriteService | Delete pipeline | +| `PipelineV2UploadFile()` | PipelineV2WriteService | Initialize file upload | +| `PipelineV2CompleteUploadFile()` | PipelineV2WriteService | Complete file upload | +| `PipelineV2FindAll()` | PipelineV2ReadService | List all pipelines | +| `PipelineV2FindOne()` | PipelineV2ReadService | Get single pipeline | +| `PipelineV2FindObjects()` | PipelineV2ReadService | Get pipeline objects | +| `PipelineV2DownloadLogs()` | PipelineV2ReadService | Download pipeline logs | +| `PipelineV2GetDashboardUrl()` | PipelineV2ReadService | Get monitoring dashboard URL | + +### 2.3 Pipelines Service (Legacy) +**File:** `src/modules/pipelines/pipelines.service.ts` + +**Uses:** `PipelinesClientService` + +**Methods:** +- `getPipelineStatus()` - Get pipeline execution status +- `runPipeline()` - Trigger pipeline execution + +### 2.4 Pipelines Client Service (Legacy) +**File:** `src/modules/pipelines/client.service.ts` + +**gRPC Methods Called:** +| Method | Service | Description | +|--------|---------|-------------| +| `getPipelineStatus()` | PipelineService | Get pipeline status | +| `triggerPipeline()` | PipelineService | Trigger pipeline run | + +--- + +## 3. NESTJS MODULES (Module Registration) + +| Module | File | Client Configuration | Exports | +|--------|------|---------------------|---------| +| CatalogModule | `src/modules/catalog/catalog.module.ts` | CatalogClientConfiguration | CatalogService | +| PipelinesV2Module | `src/modules/pipelinesV2/pipelines.module.ts` | PipelinesClientConfiguration | PipelinesService | +| PipelinesModule | `src/modules/pipelines/pipelines.module.ts` | PipelinesClientConfiguration | PipelinesService, PipelinesClientService | + +**App Module Registration:** `src/app.module.ts` (Lines 20, 23, 25, 53-54, 59) + +--- + +## 4. REST CONTROLLERS (API Endpoints to Migrate) + +### 4.1 Catalog Controller +**File:** `src/modules/catalog/catalog.controller.ts` +**Base Path:** `/catalog` + +| HTTP Method | Endpoint | Description | +|-------------|----------|-------------| +| GET | `/catalog` | Search data assets | +| GET | `/catalog/download` | Download assets as CSV | +| GET | `/catalog/data-asset` | Get asset by pipeline/object | +| GET | `/catalog/data-asset/:id` | Get single data asset | +| GET | `/catalog/data-asset/rls/:id` | Get data asset RLS info | +| GET | `/catalog/data-asset/:id/columns-metadata` | Get column metadata | +| GET | `/catalog/data-asset/:id/preview` | Get data preview | +| GET | `/catalog/data-asset/:id/docs` | Get documentation | +| GET | `/catalog/tags` | Get all tags | +| PUT | `/catalog/data-asset/:id` | Update data asset | +| PUT | `/catalog/data-asset/:id/manage-permissions` | Manage permissions | +| PUT | `/catalog/data-asset/:id/revoke-permissions` | Revoke permissions | +| POST | `/catalog` | Create data asset | +| POST | `/catalog/data-asset/:id/docs` | Create documentation | +| POST | `/catalog/data-asset/:id/comment` | Add comment | +| POST | `/catalog/dataset-catalog-task` | Trigger catalog task | +| POST | `/catalog/rls-rule` | Add RLS rule | +| POST | `/catalog/register-dataset` | Register dataset with metadata | +| DELETE | `/catalog/data-asset/:id` | Delete data asset | +| DELETE | `/catalog/data-asset/:id/comment` | Delete comment | +| DELETE | `/catalog/rls-rule/:id` | Remove RLS rule | +| DELETE | `/catalog/rls-rule` | Batch remove RLS rules | +| GET | `/catalog/dataset-catalog-task/:session` | Get catalog task status | +| GET | `/catalog/rls-rule/:id` | Get single RLS rule | +| GET | `/catalog/rls-rule` | Get RLS rules | +| GET | `/catalog/nimbus-dashboards` | Get Nimbus dashboards | +| GET | `/catalog/pii-reporter` | Get PII report | + +**Total: 27 endpoints** + +### 4.2 Pipelines V2 Controller +**File:** `src/modules/pipelinesV2/pipelines.controller.ts` +**Base Path:** `/pipelinesV2` + +| HTTP Method | Endpoint | Description | +|-------------|----------|-------------| +| GET | `/pipelinesV2/monitoring-dashboard` | Get monitoring dashboard URL | +| GET | `/pipelinesV2` | List all pipelines | +| GET | `/pipelinesV2/download-logs` | Download pipeline logs | +| GET | `/pipelinesV2/:id` | Get single pipeline | +| GET | `/pipelinesV2/:id/config` | Get pipeline properties | +| GET | `/pipelinesV2/:id/objects` | Get pipeline objects | +| GET | `/pipelinesV2/:id/status` | Get pipeline status (legacy) | +| POST | `/pipelinesV2` | Create pipeline | +| POST | `/pipelinesV2/init-upload` | Initialize file upload | +| POST | `/pipelinesV2/complete-upload` | Complete file upload | +| POST | `/pipelinesV2/file` | Upload file pipeline | +| POST | `/pipelinesV2/start/:id` | Start pipeline | +| PATCH | `/pipelinesV2/:id` | Update pipeline | +| PUT | `/pipelinesV2/:id` | Update pipeline (deprecated) | +| DELETE | `/pipelinesV2/:id` | Delete pipeline | + +**Total: 15 endpoints** + +### 4.3 Pipelines Controller (Legacy) +**File:** `src/modules/pipelines/pipelines.controller.ts` +**Base Path:** `/pipelines` + +| HTTP Method | Endpoint | Description | +|-------------|----------|-------------| +| POST | `/pipelines/start/:id` | Start pipeline (deprecated) | +| GET | `/pipelines/:id/status` | Get pipeline status (deprecated) | + +**Total: 2 endpoints (deprecated)** + +--- + +## 5. CONFIGURATION FILES + +### 5.1 Helm Chart Values (Production) +**File:** `deploy/helm-chart/values.yaml` +```yaml +# Line 40 +maestro: + pi_factory_url: pi-factory.dadosfera.ai +``` + +### 5.2 Helm Chart Values (Staging) +**File:** `deploy/helm-chart/values-stg.yaml` +```yaml +# Line 4 +maestro: + pi_factory_url: pi-factory.stg.dadosfera.ai +``` + +### 5.3 Helmfiles (Production) +**File:** `deploy/helmfiles/prd.yaml` +```yaml +# Lines 14, 38 +maestro.pi_factory_url: pi-factory.dadosfera.ai +``` + +### 5.4 Deployment Template +**File:** `deploy/helm-chart/templates/deployment.yaml` +```yaml +# Line 85 +- name: PIFACTORY_URL + value: {{ .Values.maestro.pi_factory_url }} +``` + +### 5.5 Environment Type Definition +**File:** `environment.d.ts` +```typescript +// Line 10 +PIFACTORY_URL: string; +``` + +--- + +## 6. PROTO PACKAGE DEPENDENCIES + +### 6.1 Package.json +**File:** `package.json` +```json +{ + "@dadosfera/protospack": "2.5.3", // Legacy pipelines + "@dadosfera/protospack-v2": "3.38.0-beta.14" // PipelineV2, Catalog, PlatformInterfaces +} +``` + +### 6.2 Proto Imports from `@dadosfera/protospack-v2` (PI-Factory related) + +| Import Path | Used In | +|-------------|---------| +| `Catalog` | catalog/catalog-client.ts, catalog/catalog.service.ts | +| `Catalog/interfaces/messages` | catalog/catalog.service.ts, catalog/catalog.controller.ts, catalog/dtos | +| `PlatformInterfaces` | catalog/catalog-client.ts, catalog/catalog.service.ts | +| `PipelineV2` | pipelinesV2/pipelines-client.ts, pipelinesV2/pipelines.service.ts | +| `PipelineV2/interfaces/messages` | pipelinesV2/pipelines.service.ts, pipelinesV2/pipelines.controller.ts | + +### 6.3 Proto Imports from `@dadosfera/protospack` (Legacy) + +| Import Path | Used In | +|-------------|---------| +| `PipelinePackages` | pipelines/pipelines-client.ts | +| `PipelineProtoFilePath` | pipelines/pipelines-client.ts | +| `PipelineServicesNames` | pipelines/client.service.ts | +| `PipelinesServiceInterface` | pipelines/client.service.ts | + +--- + +## 7. FILES TO MODIFY/DELETE (Summary) + +### High Priority - Core Integration Files +``` +src/modules/catalog/catalog-client.ts → DELETE or REPLACE +src/modules/catalog/catalog.service.ts → REPLACE with local implementation +src/modules/pipelinesV2/pipelines-client.ts → DELETE or REPLACE +src/modules/pipelinesV2/pipelines.service.ts → REPLACE with local implementation +src/modules/pipelines/pipelines-client.ts → DELETE or REPLACE +src/modules/pipelines/client.service.ts → REPLACE with local implementation +src/modules/pipelines/pipelines.service.ts → REPLACE with local implementation +``` + +### Medium Priority - Module Registration +``` +src/modules/catalog/catalog.module.ts → UPDATE imports +src/modules/pipelinesV2/pipelines.module.ts → UPDATE imports +src/modules/pipelines/pipelines.module.ts → UPDATE imports +src/app.module.ts → UPDATE if module structure changes +``` + +### Low Priority - Configuration +``` +deploy/helm-chart/values.yaml → REMOVE pi_factory_url +deploy/helm-chart/values-stg.yaml → REMOVE pi_factory_url +deploy/helmfiles/prd.yaml → REMOVE pi_factory_url references +deploy/helm-chart/templates/deployment.yaml → REMOVE PIFACTORY_URL env var +environment.d.ts → REMOVE PIFACTORY_URL type +README.md → UPDATE architecture diagram +``` + +--- + +## 8. DEPENDENCY COUNT BY MODULE + +| Module | Files | gRPC Calls | REST Endpoints | +|--------|-------|------------|----------------| +| Catalog | 3 | 24+ | 27 | +| PipelinesV2 | 3 | 10 | 15 | +| Pipelines (Legacy) | 3 | 2 | 2 | +| **TOTAL** | **9** | **36+** | **44** | + +--- + +## 9. COMPARISON: PI-Factory vs In-Factory + +| Aspect | PI-Factory | In-Factory | +|--------|------------|------------| +| Environment Variable | `PIFACTORY_URL` | `INFACTORY_URL` | +| Modules | 3 | 6 | +| gRPC Calls | 36+ | 27+ | +| REST Endpoints | 44 | 25 | +| Complexity | HIGH | MEDIUM-HIGH | +| Domain | Pipelines, Catalog | Connections, Connectors, Inputs | + +--- + +## 10. DATA MODELS (Proto Messages Used) + +### Catalog Messages +- `CreateDataAssetRequest` / `CreateDataAssetResponse` +- `GetAllDataAssetsRequest` / `GetAllDataAssetsResponse` +- `GetOneDataAssetRequest` / Response +- `UpdateDataAssetRequest` / Response +- `DeleteDataAssetRequest` / Response +- `ManagePermissionRequest` / Response +- `RevokePermissionRequest` / Response +- `MakeACommentRequest` / Response +- `UpdateACommentRequest` / Response +- `TriggerDatasetCatalogingRequest` / Response +- `GetDatasetCatalogTaskRequest` / Response +- `AddRlsRuleRequest` / Response +- `RemoveRlsRuleRequest` / Response +- `GetRlsRulesRequest` / Response +- `GetNimbusDashboardsRequest` / Response +- `PiiMetadata` +- `RegisterDatasetWithMetatadaRequest` + +### PipelineV2 Messages +- `PipelineV2CreateRequest` / `PipelineV2CreateResponse` +- `PipelineV2FindAllRequest` / `PipelineV2FindAllResponse` +- `PipelineV2FindOneRequest` / `PipelineV2FindOneResponse` +- `PipelineV2UpdateRequest` / `PipelineV2UpdateResponse` +- `PipelineV2RemoveRequest` / Response +- `PipelineV2UploadFileRequest` / Response +- `PipelineV2CompleteUploadFileRequest` / Response +- `PipelineV2FindObjectsRequest` / Response +- `PipelineV2DownloadLogsRequest` / Response +- `PipelineV2GetDashboardUrlRequest` / Response + +### Platform Interfaces Messages +- `CatalogDataAssetsRequest` / Response + +--- + +## 11. MIGRATION STRATEGY OPTIONS + +### Option A: Move PI-Factory functionality INTO Maestro +**Pros:** +- Single service to maintain +- No network latency for these operations +- Simpler deployment + +**Cons:** +- Significantly increases Maestro's responsibility +- Requires database access from Maestro +- Large refactoring effort (44 endpoints) + +### Option B: Create REST API wrapper in PI-Factory +**Pros:** +- Minimal changes to Maestro +- Can migrate incrementally + +**Cons:** +- Still maintains dependency +- Additional REST→gRPC translation layer + +### Option C: Direct database access from Maestro +**Pros:** +- Removes runtime dependency +- Better performance + +**Cons:** +- Shared database coupling +- Complex migration + +--- + +## 12. RECOMMENDED MIGRATION ORDER + +Given the complexity, we recommend migrating in this order: + +1. **Legacy Pipelines** (2 endpoints, deprecated) - Lowest risk +2. **PipelinesV2** (15 endpoints) - Core pipeline functionality +3. **Catalog** (27 endpoints) - Most complex, migrate last + +### Phase 1: Legacy Pipelines (Deprecated) +- Remove `/pipelines/start/:id` +- Remove `/pipelines/:id/status` +- Update all clients to use `/pipelinesV2/*` endpoints + +### Phase 2: PipelinesV2 +- Migrate pipeline CRUD operations +- Migrate file upload functionality +- Migrate monitoring dashboard + +### Phase 3: Catalog +- Migrate data asset CRUD +- Migrate permissions management +- Migrate RLS rules +- Migrate dataset registration +- Migrate PII reporting + +--- + +## NEXT STEPS + +1. **Decide on migration strategy** (Option A, B, or C) +2. **Deprecate legacy pipelines module** first +3. **Create database schema** if moving to Option A +4. **Implement local services** one module at a time +5. **Update tests** for each migrated module +6. **Update deployment configs** to remove PIFACTORY_URL +7. **Coordinate with PI-Factory team** for data migration