mirror of
https://github.com/dadosfera/maestro.git
synced 2026-11-17 06:29:09 +00:00
DOCS: Map maestro-pifactory integration points for migration
Add comprehensive documentation mapping all integration points between maestro and pi-factory services to facilitate removing the dependency. Includes: - 3 gRPC client configurations using PIFACTORY_URL - 36+ gRPC method calls across Catalog, PipelineV2, and legacy Pipeline services - 44 REST endpoints that proxy to pi-factory - Configuration files and environment variables - Proto package dependencies (protospack and protospack-v2) - Migration strategy options and recommendations - Comparison with in-factory migration scope
This commit is contained in:
@@ -0,0 +1,486 @@
|
||||
# Maestro ↔ PI-Factory Migration Map
|
||||
|
||||
This document maps all integration points between Maestro and PI-Factory that need to be addressed to remove the dependency.
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
```mermaid
|
||||
graph TD;
|
||||
Frontend<-->Maestro;
|
||||
Maestro<-->DUC;
|
||||
Maestro<-->pi-factory;
|
||||
Maestro<-->in-factory;
|
||||
```
|
||||
|
||||
PI-Factory (`PIFACTORY_URL`) is responsible for:
|
||||
- **Pipeline Management** - Create, read, update, delete pipelines
|
||||
- **Catalog Services** - Data asset management, metadata, previews
|
||||
- **Platform Interfaces** - Dataset cataloging operations
|
||||
|
||||
---
|
||||
|
||||
## Summary Table
|
||||
|
||||
| Category | Count | Impact Level |
|
||||
|----------|-------|--------------|
|
||||
| gRPC Client Configurations | 3 | HIGH |
|
||||
| NestJS Modules | 3 | HIGH |
|
||||
| REST Controllers/Endpoints | 3 (~50 endpoints) | HIGH |
|
||||
| Service Classes | 4 | HIGH |
|
||||
| Configuration Files | 4 | MEDIUM |
|
||||
| Proto Package Dependencies | 2 | HIGH |
|
||||
| Environment Variables | 1 | LOW |
|
||||
|
||||
---
|
||||
|
||||
## 1. gRPC CLIENT CONFIGURATIONS (Files to Migrate)
|
||||
|
||||
These files configure gRPC connections to PI-Factory services:
|
||||
|
||||
### 1.1 Catalog Client (`PIFACTORY_URL`)
|
||||
**File:** `src/modules/catalog/catalog-client.ts`
|
||||
```typescript
|
||||
// Lines 11-12, 19
|
||||
process.env.PIFACTORY_URL.startsWith('pi-factory:')
|
||||
process.env.PIFACTORY_URL.includes('0.0.0.0')
|
||||
url: process.env.PIFACTORY_URL
|
||||
```
|
||||
**Proto Services Used:**
|
||||
- `Catalog.ProtoPackages.ReadPackage`
|
||||
- `Catalog.ProtoPackages.WritePackage`
|
||||
- `PlatformInterfaces.ProtoPackages.WritePackage`
|
||||
|
||||
### 1.2 Pipelines V2 Client (`PIFACTORY_URL`)
|
||||
**File:** `src/modules/pipelinesV2/pipelines-client.ts`
|
||||
```typescript
|
||||
// Lines 13-14, 21
|
||||
url: process.env.PIFACTORY_URL
|
||||
```
|
||||
**Proto Services Used:**
|
||||
- `PipelineV2.ProtoPackages.ReadPackage`
|
||||
- `PipelineV2.ProtoPackages.WritePackage`
|
||||
|
||||
### 1.3 Pipelines Client (Legacy) (`PIFACTORY_URL`)
|
||||
**File:** `src/modules/pipelines/pipelines-client.ts`
|
||||
```typescript
|
||||
// Lines 10-11, 18
|
||||
url: process.env.PIFACTORY_URL
|
||||
```
|
||||
**Proto Services Used:**
|
||||
- `PipelinePackages` from `@dadosfera/protospack`
|
||||
|
||||
---
|
||||
|
||||
## 2. SERVICE CLASSES (Business Logic to Migrate)
|
||||
|
||||
### 2.1 Catalog Service
|
||||
**File:** `src/modules/catalog/catalog.service.ts`
|
||||
|
||||
**gRPC Services Initialized:**
|
||||
- `CatalogReadServices` (from `Catalog.ReadService`)
|
||||
- `CatalogWriteServices` (from `Catalog.WriteService`)
|
||||
- `PlatformInterfacesWriteServices` (from `PlatformInterfaces.WriteService`)
|
||||
|
||||
**gRPC Methods Called:**
|
||||
|
||||
| Method | Service | Description |
|
||||
|--------|---------|-------------|
|
||||
| `GetAllDataAssets()` | CatalogReadServices | Search/list data assets |
|
||||
| `GetOneDataAsset()` | CatalogReadServices | Get single data asset by ID |
|
||||
| `GetOneDataAssetByPipelineAndObject()` | CatalogReadServices | Get asset by pipeline/object |
|
||||
| `GetDatasetDoc()` | CatalogReadServices | Get dataset documentation |
|
||||
| `GetDatasetPreview()` | CatalogReadServices | Get data preview |
|
||||
| `GetDatasetColumnsMetadata()` | CatalogReadServices | Get column metadata |
|
||||
| `GetCustomerTags()` | CatalogReadServices | Get all tags for customer |
|
||||
| `GetDatasetCatalogTask()` | CatalogReadServices | Get catalog task status |
|
||||
| `GetRlsRules()` | CatalogReadServices | Get RLS rules |
|
||||
| `GetOneRlsRule()` | CatalogReadServices | Get single RLS rule |
|
||||
| `GetNimbusDashboards()` | CatalogReadServices | Get Nimbus dashboards |
|
||||
| `CreateDataAsset()` | CatalogWriteServices | Create new data asset |
|
||||
| `UpdateDataAsset()` | CatalogWriteServices | Update existing data asset |
|
||||
| `DeleteDataAsset()` | CatalogWriteServices | Delete data asset |
|
||||
| `ManagePermission()` | CatalogWriteServices | Manage asset permissions |
|
||||
| `RevokePermission()` | CatalogWriteServices | Revoke asset permissions |
|
||||
| `MakeAComment()` | CatalogWriteServices | Add comment to asset |
|
||||
| `UpdateAComment()` | CatalogWriteServices | Update/delete comment |
|
||||
| `TriggerDatasetCataloging()` | CatalogWriteServices | Trigger catalog process |
|
||||
| `AddRlsRule()` | CatalogWriteServices | Add RLS rule |
|
||||
| `RemoveRlsRule()` | CatalogWriteServices | Remove RLS rule |
|
||||
| `RemoveRlsRulesByRlsId()` | CatalogWriteServices | Batch remove by RLS ID |
|
||||
| `RemoveRlsRulesByDashboardId()` | CatalogWriteServices | Batch remove by dashboard |
|
||||
| `GetPiiReporter()` | CatalogWriteServices | Get PII report data |
|
||||
| `CatalogDataAssets()` | PlatformInterfacesWriteServices | Catalog datasets |
|
||||
|
||||
**Additional HTTP Calls to Nimbus:**
|
||||
- `POST ${nimbusUrl}/api/catalog/data-docs/` - Create data docs
|
||||
- `POST ${nimbusUrl}/api/catalog/table-metadata/` - Create table metadata
|
||||
- `POST ${nimbusUrl}/api/catalog/column-metadata/` - Create column metadata
|
||||
- `POST ${nimbusUrl}/api/catalog/data-preview/` - Create data preview
|
||||
|
||||
### 2.2 Pipelines V2 Service
|
||||
**File:** `src/modules/pipelinesV2/pipelines.service.ts`
|
||||
|
||||
**gRPC Services Initialized:**
|
||||
- `PipelineV2ReadService` (from `PipelineV2.ReadService`)
|
||||
- `PipelineV2WriteService` (from `PipelineV2.WriteService`)
|
||||
|
||||
**gRPC Methods Called:**
|
||||
|
||||
| Method | Service | Description |
|
||||
|--------|---------|-------------|
|
||||
| `PipelineV2Create()` | PipelineV2WriteService | Create new pipeline |
|
||||
| `PipelineV2Update()` | PipelineV2WriteService | Update pipeline |
|
||||
| `PipelineV2Remove()` | PipelineV2WriteService | Delete pipeline |
|
||||
| `PipelineV2UploadFile()` | PipelineV2WriteService | Initialize file upload |
|
||||
| `PipelineV2CompleteUploadFile()` | PipelineV2WriteService | Complete file upload |
|
||||
| `PipelineV2FindAll()` | PipelineV2ReadService | List all pipelines |
|
||||
| `PipelineV2FindOne()` | PipelineV2ReadService | Get single pipeline |
|
||||
| `PipelineV2FindObjects()` | PipelineV2ReadService | Get pipeline objects |
|
||||
| `PipelineV2DownloadLogs()` | PipelineV2ReadService | Download pipeline logs |
|
||||
| `PipelineV2GetDashboardUrl()` | PipelineV2ReadService | Get monitoring dashboard URL |
|
||||
|
||||
### 2.3 Pipelines Service (Legacy)
|
||||
**File:** `src/modules/pipelines/pipelines.service.ts`
|
||||
|
||||
**Uses:** `PipelinesClientService`
|
||||
|
||||
**Methods:**
|
||||
- `getPipelineStatus()` - Get pipeline execution status
|
||||
- `runPipeline()` - Trigger pipeline execution
|
||||
|
||||
### 2.4 Pipelines Client Service (Legacy)
|
||||
**File:** `src/modules/pipelines/client.service.ts`
|
||||
|
||||
**gRPC Methods Called:**
|
||||
| Method | Service | Description |
|
||||
|--------|---------|-------------|
|
||||
| `getPipelineStatus()` | PipelineService | Get pipeline status |
|
||||
| `triggerPipeline()` | PipelineService | Trigger pipeline run |
|
||||
|
||||
---
|
||||
|
||||
## 3. NESTJS MODULES (Module Registration)
|
||||
|
||||
| Module | File | Client Configuration | Exports |
|
||||
|--------|------|---------------------|---------|
|
||||
| CatalogModule | `src/modules/catalog/catalog.module.ts` | CatalogClientConfiguration | CatalogService |
|
||||
| PipelinesV2Module | `src/modules/pipelinesV2/pipelines.module.ts` | PipelinesClientConfiguration | PipelinesService |
|
||||
| PipelinesModule | `src/modules/pipelines/pipelines.module.ts` | PipelinesClientConfiguration | PipelinesService, PipelinesClientService |
|
||||
|
||||
**App Module Registration:** `src/app.module.ts` (Lines 20, 23, 25, 53-54, 59)
|
||||
|
||||
---
|
||||
|
||||
## 4. REST CONTROLLERS (API Endpoints to Migrate)
|
||||
|
||||
### 4.1 Catalog Controller
|
||||
**File:** `src/modules/catalog/catalog.controller.ts`
|
||||
**Base Path:** `/catalog`
|
||||
|
||||
| HTTP Method | Endpoint | Description |
|
||||
|-------------|----------|-------------|
|
||||
| GET | `/catalog` | Search data assets |
|
||||
| GET | `/catalog/download` | Download assets as CSV |
|
||||
| GET | `/catalog/data-asset` | Get asset by pipeline/object |
|
||||
| GET | `/catalog/data-asset/:id` | Get single data asset |
|
||||
| GET | `/catalog/data-asset/rls/:id` | Get data asset RLS info |
|
||||
| GET | `/catalog/data-asset/:id/columns-metadata` | Get column metadata |
|
||||
| GET | `/catalog/data-asset/:id/preview` | Get data preview |
|
||||
| GET | `/catalog/data-asset/:id/docs` | Get documentation |
|
||||
| GET | `/catalog/tags` | Get all tags |
|
||||
| PUT | `/catalog/data-asset/:id` | Update data asset |
|
||||
| PUT | `/catalog/data-asset/:id/manage-permissions` | Manage permissions |
|
||||
| PUT | `/catalog/data-asset/:id/revoke-permissions` | Revoke permissions |
|
||||
| POST | `/catalog` | Create data asset |
|
||||
| POST | `/catalog/data-asset/:id/docs` | Create documentation |
|
||||
| POST | `/catalog/data-asset/:id/comment` | Add comment |
|
||||
| POST | `/catalog/dataset-catalog-task` | Trigger catalog task |
|
||||
| POST | `/catalog/rls-rule` | Add RLS rule |
|
||||
| POST | `/catalog/register-dataset` | Register dataset with metadata |
|
||||
| DELETE | `/catalog/data-asset/:id` | Delete data asset |
|
||||
| DELETE | `/catalog/data-asset/:id/comment` | Delete comment |
|
||||
| DELETE | `/catalog/rls-rule/:id` | Remove RLS rule |
|
||||
| DELETE | `/catalog/rls-rule` | Batch remove RLS rules |
|
||||
| GET | `/catalog/dataset-catalog-task/:session` | Get catalog task status |
|
||||
| GET | `/catalog/rls-rule/:id` | Get single RLS rule |
|
||||
| GET | `/catalog/rls-rule` | Get RLS rules |
|
||||
| GET | `/catalog/nimbus-dashboards` | Get Nimbus dashboards |
|
||||
| GET | `/catalog/pii-reporter` | Get PII report |
|
||||
|
||||
**Total: 27 endpoints**
|
||||
|
||||
### 4.2 Pipelines V2 Controller
|
||||
**File:** `src/modules/pipelinesV2/pipelines.controller.ts`
|
||||
**Base Path:** `/pipelinesV2`
|
||||
|
||||
| HTTP Method | Endpoint | Description |
|
||||
|-------------|----------|-------------|
|
||||
| GET | `/pipelinesV2/monitoring-dashboard` | Get monitoring dashboard URL |
|
||||
| GET | `/pipelinesV2` | List all pipelines |
|
||||
| GET | `/pipelinesV2/download-logs` | Download pipeline logs |
|
||||
| GET | `/pipelinesV2/:id` | Get single pipeline |
|
||||
| GET | `/pipelinesV2/:id/config` | Get pipeline properties |
|
||||
| GET | `/pipelinesV2/:id/objects` | Get pipeline objects |
|
||||
| GET | `/pipelinesV2/:id/status` | Get pipeline status (legacy) |
|
||||
| POST | `/pipelinesV2` | Create pipeline |
|
||||
| POST | `/pipelinesV2/init-upload` | Initialize file upload |
|
||||
| POST | `/pipelinesV2/complete-upload` | Complete file upload |
|
||||
| POST | `/pipelinesV2/file` | Upload file pipeline |
|
||||
| POST | `/pipelinesV2/start/:id` | Start pipeline |
|
||||
| PATCH | `/pipelinesV2/:id` | Update pipeline |
|
||||
| PUT | `/pipelinesV2/:id` | Update pipeline (deprecated) |
|
||||
| DELETE | `/pipelinesV2/:id` | Delete pipeline |
|
||||
|
||||
**Total: 15 endpoints**
|
||||
|
||||
### 4.3 Pipelines Controller (Legacy)
|
||||
**File:** `src/modules/pipelines/pipelines.controller.ts`
|
||||
**Base Path:** `/pipelines`
|
||||
|
||||
| HTTP Method | Endpoint | Description |
|
||||
|-------------|----------|-------------|
|
||||
| POST | `/pipelines/start/:id` | Start pipeline (deprecated) |
|
||||
| GET | `/pipelines/:id/status` | Get pipeline status (deprecated) |
|
||||
|
||||
**Total: 2 endpoints (deprecated)**
|
||||
|
||||
---
|
||||
|
||||
## 5. CONFIGURATION FILES
|
||||
|
||||
### 5.1 Helm Chart Values (Production)
|
||||
**File:** `deploy/helm-chart/values.yaml`
|
||||
```yaml
|
||||
# Line 40
|
||||
maestro:
|
||||
pi_factory_url: pi-factory.dadosfera.ai
|
||||
```
|
||||
|
||||
### 5.2 Helm Chart Values (Staging)
|
||||
**File:** `deploy/helm-chart/values-stg.yaml`
|
||||
```yaml
|
||||
# Line 4
|
||||
maestro:
|
||||
pi_factory_url: pi-factory.stg.dadosfera.ai
|
||||
```
|
||||
|
||||
### 5.3 Helmfiles (Production)
|
||||
**File:** `deploy/helmfiles/prd.yaml`
|
||||
```yaml
|
||||
# Lines 14, 38
|
||||
maestro.pi_factory_url: pi-factory.dadosfera.ai
|
||||
```
|
||||
|
||||
### 5.4 Deployment Template
|
||||
**File:** `deploy/helm-chart/templates/deployment.yaml`
|
||||
```yaml
|
||||
# Line 85
|
||||
- name: PIFACTORY_URL
|
||||
value: {{ .Values.maestro.pi_factory_url }}
|
||||
```
|
||||
|
||||
### 5.5 Environment Type Definition
|
||||
**File:** `environment.d.ts`
|
||||
```typescript
|
||||
// Line 10
|
||||
PIFACTORY_URL: string;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. PROTO PACKAGE DEPENDENCIES
|
||||
|
||||
### 6.1 Package.json
|
||||
**File:** `package.json`
|
||||
```json
|
||||
{
|
||||
"@dadosfera/protospack": "2.5.3", // Legacy pipelines
|
||||
"@dadosfera/protospack-v2": "3.38.0-beta.14" // PipelineV2, Catalog, PlatformInterfaces
|
||||
}
|
||||
```
|
||||
|
||||
### 6.2 Proto Imports from `@dadosfera/protospack-v2` (PI-Factory related)
|
||||
|
||||
| Import Path | Used In |
|
||||
|-------------|---------|
|
||||
| `Catalog` | catalog/catalog-client.ts, catalog/catalog.service.ts |
|
||||
| `Catalog/interfaces/messages` | catalog/catalog.service.ts, catalog/catalog.controller.ts, catalog/dtos |
|
||||
| `PlatformInterfaces` | catalog/catalog-client.ts, catalog/catalog.service.ts |
|
||||
| `PipelineV2` | pipelinesV2/pipelines-client.ts, pipelinesV2/pipelines.service.ts |
|
||||
| `PipelineV2/interfaces/messages` | pipelinesV2/pipelines.service.ts, pipelinesV2/pipelines.controller.ts |
|
||||
|
||||
### 6.3 Proto Imports from `@dadosfera/protospack` (Legacy)
|
||||
|
||||
| Import Path | Used In |
|
||||
|-------------|---------|
|
||||
| `PipelinePackages` | pipelines/pipelines-client.ts |
|
||||
| `PipelineProtoFilePath` | pipelines/pipelines-client.ts |
|
||||
| `PipelineServicesNames` | pipelines/client.service.ts |
|
||||
| `PipelinesServiceInterface` | pipelines/client.service.ts |
|
||||
|
||||
---
|
||||
|
||||
## 7. FILES TO MODIFY/DELETE (Summary)
|
||||
|
||||
### High Priority - Core Integration Files
|
||||
```
|
||||
src/modules/catalog/catalog-client.ts → DELETE or REPLACE
|
||||
src/modules/catalog/catalog.service.ts → REPLACE with local implementation
|
||||
src/modules/pipelinesV2/pipelines-client.ts → DELETE or REPLACE
|
||||
src/modules/pipelinesV2/pipelines.service.ts → REPLACE with local implementation
|
||||
src/modules/pipelines/pipelines-client.ts → DELETE or REPLACE
|
||||
src/modules/pipelines/client.service.ts → REPLACE with local implementation
|
||||
src/modules/pipelines/pipelines.service.ts → REPLACE with local implementation
|
||||
```
|
||||
|
||||
### Medium Priority - Module Registration
|
||||
```
|
||||
src/modules/catalog/catalog.module.ts → UPDATE imports
|
||||
src/modules/pipelinesV2/pipelines.module.ts → UPDATE imports
|
||||
src/modules/pipelines/pipelines.module.ts → UPDATE imports
|
||||
src/app.module.ts → UPDATE if module structure changes
|
||||
```
|
||||
|
||||
### Low Priority - Configuration
|
||||
```
|
||||
deploy/helm-chart/values.yaml → REMOVE pi_factory_url
|
||||
deploy/helm-chart/values-stg.yaml → REMOVE pi_factory_url
|
||||
deploy/helmfiles/prd.yaml → REMOVE pi_factory_url references
|
||||
deploy/helm-chart/templates/deployment.yaml → REMOVE PIFACTORY_URL env var
|
||||
environment.d.ts → REMOVE PIFACTORY_URL type
|
||||
README.md → UPDATE architecture diagram
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8. DEPENDENCY COUNT BY MODULE
|
||||
|
||||
| Module | Files | gRPC Calls | REST Endpoints |
|
||||
|--------|-------|------------|----------------|
|
||||
| Catalog | 3 | 24+ | 27 |
|
||||
| PipelinesV2 | 3 | 10 | 15 |
|
||||
| Pipelines (Legacy) | 3 | 2 | 2 |
|
||||
| **TOTAL** | **9** | **36+** | **44** |
|
||||
|
||||
---
|
||||
|
||||
## 9. COMPARISON: PI-Factory vs In-Factory
|
||||
|
||||
| Aspect | PI-Factory | In-Factory |
|
||||
|--------|------------|------------|
|
||||
| Environment Variable | `PIFACTORY_URL` | `INFACTORY_URL` |
|
||||
| Modules | 3 | 6 |
|
||||
| gRPC Calls | 36+ | 27+ |
|
||||
| REST Endpoints | 44 | 25 |
|
||||
| Complexity | HIGH | MEDIUM-HIGH |
|
||||
| Domain | Pipelines, Catalog | Connections, Connectors, Inputs |
|
||||
|
||||
---
|
||||
|
||||
## 10. DATA MODELS (Proto Messages Used)
|
||||
|
||||
### Catalog Messages
|
||||
- `CreateDataAssetRequest` / `CreateDataAssetResponse`
|
||||
- `GetAllDataAssetsRequest` / `GetAllDataAssetsResponse`
|
||||
- `GetOneDataAssetRequest` / Response
|
||||
- `UpdateDataAssetRequest` / Response
|
||||
- `DeleteDataAssetRequest` / Response
|
||||
- `ManagePermissionRequest` / Response
|
||||
- `RevokePermissionRequest` / Response
|
||||
- `MakeACommentRequest` / Response
|
||||
- `UpdateACommentRequest` / Response
|
||||
- `TriggerDatasetCatalogingRequest` / Response
|
||||
- `GetDatasetCatalogTaskRequest` / Response
|
||||
- `AddRlsRuleRequest` / Response
|
||||
- `RemoveRlsRuleRequest` / Response
|
||||
- `GetRlsRulesRequest` / Response
|
||||
- `GetNimbusDashboardsRequest` / Response
|
||||
- `PiiMetadata`
|
||||
- `RegisterDatasetWithMetatadaRequest`
|
||||
|
||||
### PipelineV2 Messages
|
||||
- `PipelineV2CreateRequest` / `PipelineV2CreateResponse`
|
||||
- `PipelineV2FindAllRequest` / `PipelineV2FindAllResponse`
|
||||
- `PipelineV2FindOneRequest` / `PipelineV2FindOneResponse`
|
||||
- `PipelineV2UpdateRequest` / `PipelineV2UpdateResponse`
|
||||
- `PipelineV2RemoveRequest` / Response
|
||||
- `PipelineV2UploadFileRequest` / Response
|
||||
- `PipelineV2CompleteUploadFileRequest` / Response
|
||||
- `PipelineV2FindObjectsRequest` / Response
|
||||
- `PipelineV2DownloadLogsRequest` / Response
|
||||
- `PipelineV2GetDashboardUrlRequest` / Response
|
||||
|
||||
### Platform Interfaces Messages
|
||||
- `CatalogDataAssetsRequest` / Response
|
||||
|
||||
---
|
||||
|
||||
## 11. MIGRATION STRATEGY OPTIONS
|
||||
|
||||
### Option A: Move PI-Factory functionality INTO Maestro
|
||||
**Pros:**
|
||||
- Single service to maintain
|
||||
- No network latency for these operations
|
||||
- Simpler deployment
|
||||
|
||||
**Cons:**
|
||||
- Significantly increases Maestro's responsibility
|
||||
- Requires database access from Maestro
|
||||
- Large refactoring effort (44 endpoints)
|
||||
|
||||
### Option B: Create REST API wrapper in PI-Factory
|
||||
**Pros:**
|
||||
- Minimal changes to Maestro
|
||||
- Can migrate incrementally
|
||||
|
||||
**Cons:**
|
||||
- Still maintains dependency
|
||||
- Additional REST→gRPC translation layer
|
||||
|
||||
### Option C: Direct database access from Maestro
|
||||
**Pros:**
|
||||
- Removes runtime dependency
|
||||
- Better performance
|
||||
|
||||
**Cons:**
|
||||
- Shared database coupling
|
||||
- Complex migration
|
||||
|
||||
---
|
||||
|
||||
## 12. RECOMMENDED MIGRATION ORDER
|
||||
|
||||
Given the complexity, we recommend migrating in this order:
|
||||
|
||||
1. **Legacy Pipelines** (2 endpoints, deprecated) - Lowest risk
|
||||
2. **PipelinesV2** (15 endpoints) - Core pipeline functionality
|
||||
3. **Catalog** (27 endpoints) - Most complex, migrate last
|
||||
|
||||
### Phase 1: Legacy Pipelines (Deprecated)
|
||||
- Remove `/pipelines/start/:id`
|
||||
- Remove `/pipelines/:id/status`
|
||||
- Update all clients to use `/pipelinesV2/*` endpoints
|
||||
|
||||
### Phase 2: PipelinesV2
|
||||
- Migrate pipeline CRUD operations
|
||||
- Migrate file upload functionality
|
||||
- Migrate monitoring dashboard
|
||||
|
||||
### Phase 3: Catalog
|
||||
- Migrate data asset CRUD
|
||||
- Migrate permissions management
|
||||
- Migrate RLS rules
|
||||
- Migrate dataset registration
|
||||
- Migrate PII reporting
|
||||
|
||||
---
|
||||
|
||||
## NEXT STEPS
|
||||
|
||||
1. **Decide on migration strategy** (Option A, B, or C)
|
||||
2. **Deprecate legacy pipelines module** first
|
||||
3. **Create database schema** if moving to Option A
|
||||
4. **Implement local services** one module at a time
|
||||
5. **Update tests** for each migrated module
|
||||
6. **Update deployment configs** to remove PIFACTORY_URL
|
||||
7. **Coordinate with PI-Factory team** for data migration
|
||||
Reference in New Issue
Block a user