Beta's cache-first work added connection-test.service.spec.ts, whose import
graph (connection/client.config.ts) reads process.env.INFACTORY_URL at load
time. The Dockerfile test target only set DUC_URL, so that suite crashed at
import ("Cannot read properties of undefined (reading 'startsWith')") in CI and
in any bare `npm test` run.
Add ENV INFACTORY_URL=0.0.0.0:50052 alongside the existing DUC_URL, matching the
local-connection convention (0.0.0.0 => no SSL). Full suite: 52 passed, 5 skipped.
Co-Authored-By: WOZCODE <contact@withwoz.com>
- platform-api.controller.spec: addCdcTable expectations now include the CDC
fields the controller threads (iceberg_table_name, iceberg_qualify_table_name,
column_exclude_list) which were added by the CDC-iceberg work.
- release_note specs: provide DadosferaLogger mock — ReleaseNoteService gained an
@Inject(DadosferaLogger) dependency (from beta) without its specs being updated,
so they failed DI resolution on merge.
Full suite: 52 passed, 5 skipped, 0 failed.
Co-Authored-By: WOZCODE <contact@withwoz.com>
Beta's refresh-catalog RefreshCatalogReq DTO restricted plugin to
oracle/mysql/postgresql/sqlserver. Under the cache-first catalog model
(adopted for CDC in the beta merge), the CDC schema-fetch flow posts
/connection-test/refresh-catalog with plugin=mysql_cdc, which the @IsIn
rejected ("plugin must be one of: oracle, mysql, postgresql, sqlserver").
Add mysql_cdc/postgresql_cdc/oracle_cdc to the @IsIn and @ApiProperty enum,
matching the platform connection-test SQS plugin set. docsfera.json regenerated.
Co-Authored-By: WOZCODE <contact@withwoz.com>
Resolves conflicts for PR #510 (base: beta):
- package.json: keep protospack ^3.40.0-beta.20 (carries CDC→Iceberg fields);
package-lock.json reconciled (protospack was the only dep delta vs beta).
- connection-test controller/service/dto: keep BOTH feature sets — our CDC
prerequisites validation AND beta's refresh-catalog endpoints.
- Adopt beta's cache-first catalog reads (connections-api proxy) over our gRPC
path. connectionTestListTables now enriches each table with primary_keys
derived from the cached columns endpoint (is_primary_key), preserving the
CDC create flow's need for PKs under the new architecture; spec updated.
- docsfera.json resolved to ours; regenerated on next app bootstrap.
connection-test spec: 6 passed. tsc --noEmit: clean.
Co-Authored-By: WOZCODE <contact@withwoz.com>
Replaces the local file:../protospack-v2/...cdc-iceberg.4.tgz tarball reference
with the published CodeArtifact version ^3.40.0-beta.20 (carries the CDC→Iceberg
qualify_namespace / iceberg_qualify_table_name / column_exclude_list proto fields).
Co-Authored-By: WOZCODE <contact@withwoz.com>
PR #513 merged to beta but no semantic-release ran: its commits used
conventional-commits prefixes (feat(auth):, fix(auth):) which the
.releaserc.json eslint preset does not recognise, so commit-analyzer
found no release-worthy change. This empty FIX: commit matches the
eslint preset's releaseRules (tag FIX -> patch) to cut a beta release
that includes the /auth/me permission-seqids change, so stg can deploy it.
Co-Authored-By: WOZCODE <contact@withwoz.com>
Return payload.permissions verbatim (numeric seqids) instead of
translating them to claim strings. Consumers own the seqid->meaning
mapping. Drops permission-claims.ts entirely; UserDTO.permissions is
now number[].
Co-Authored-By: WOZCODE <contact@withwoz.com>
Keep Maestro a pure identity provider: /auth/me exposes the user's
permission claim strings and nothing consumer-specific. Consumers derive
whatever meaning they need (roles, module access, groups) from the claim
vocabulary — claims are already namespaced group:action.
- UserDTO: drop roles[]/modules[], keep permissions[].
- Helper shrinks to a generic seqid->claim translation
(orchest-identity.ts -> permission-claims.ts, translateSeqidsToClaims).
- api-key branch: permissions: [] only.
The roles/modules derivation moves entirely to the consumer (Orchest's
auth-server adapter).
Co-Authored-By: WOZCODE <contact@withwoz.com>
The JWT `permissions` claim is an array of numeric seqids at runtime
(see authentication.guard.ts / authentication.decorator.ts), not claim
strings. deriveOrchestIdentity previously matched claim strings against
this numeric array, so roles[]/modules[] were always empty for every
real user.
- deriveOrchestIdentity now takes number[] | undefined and matches
seqids sourced from PERMISSIONS_GROUPS (permissions.enum.ts) instead
of hand-copied literals.
- permissions is translated back to claim strings via a full
seqid->claim catalog built once from PERMISSIONS_GROUPS; unknown
seqids are dropped (auth-server ignores permissions[] in v1).
- auth.controller.ts's api-key branch literal is now annotated
`: UserDTO` so tsc enforces the three fields there.
- Both spec files re-fixtured with numeric seqid inputs, including a
mixed admin+module case and an exact claim-string translation
assertion.
Co-Authored-By: WOZCODE <contact@withwoz.com>
Threads the new CDC→Iceberg fields from the REST DTOs to the gRPC calls:
- CdcTableReq.iceberg_qualify_table_name + IcebergDestinationReq.qualify_namespace
in input.model.ts
- inputs.service.ts create map forwards iceberg_qualify_table_name
- platform-api.controller.ts addTable body + cdcTable thread iceberg_qualify_table_name
Bumps protospack to v3.41.0-cdc-iceberg.4; docsfera.json regenerated with the new
/platform/iceberg/{namespaces,tables/validate} routes.
Co-Authored-By: WOZCODE <contact@withwoz.com>
Adds CdcColumnReq {name, type, is_primary_key} and columns? on
CdcTableReq, forwarded through createCdc and the addTable (Edit
Objects add-table) path so the column schema reaches in-factory for
Iceberg deduped-table pre-create. Bumps protospack-v2 to
3.41.0-cdc-iceberg.1, which adds the matching CdcColumn field
(now required on CdcTable) and regenerates docsfera.json.
Co-Authored-By: WOZCODE <contact@withwoz.com>
InputsService.createCdc builds InputCreateCdcRequest by enumerating
fields (not spreading), so a revert of the destination/iceberg_table_name
mapping lines would not be caught by the existing controller spec, which
only mocks InputsService. Add a unit test at the service boundary that
asserts destination and per-table iceberg_table_name reach the gRPC
request, plus a back-compat case with no destination.
Also mark CdcTableReq.iceberg_table_name as advisory/reserved: the
platform derives the Iceberg table name itself today and does not yet
consume this field.
Co-Authored-By: WOZCODE <contact@withwoz.com>
The POST /pipelines/:pipelineId/inputs/:inputId/tables route built its
CdcTable payload field-by-field and silently dropped iceberg_table_name
even though inputsService.addCdcTable/the gRPC AddCdcTable call (and the
protospack CdcTable message) already support it. Widen the inline request
body type and thread the field into the addCdcTable payload; absent for
snowflake, unchanged back-compat.
Co-Authored-By: WOZCODE <contact@withwoz.com>
Unblocks the CdcDestination contract locally on the CDC branch. Swap to a
published registry version before merge (CI guard blocks file: deps).
Co-Authored-By: WOZCODE <contact@withwoz.com>
Removing N tables via Edit Objects previously looped a single-table
DELETE per table (maestro deleteTable hardcodes job_ids:[one]), so the
Debezium source + Snowflake sink connectors were rewritten/restarted once
per table. platform-api's DELETE /pipeline/:id/jobs already batches (2
connector writes total for any N), but nothing above it used the array.
New maestro DELETE /pipelines/:pipelineId/inputs/:inputId/tables takes
{ table_names: [] }: soft-deletes each in DynamoDB (tracking successes),
resolves all table_names -> job_ids from the platform pipeline in one GET,
then makes ONE DELETE /pipeline/:id/jobs with all job_ids. All-or-nothing:
any failure (a later mark, an unmatched table, or the platform delete)
rolls back only the marks made in this call.
The single-table deleteTable route is kept (unchanged) — nothing else
depends on removing it, and that's a separable cleanup.
Tests: N tables -> one platform DELETE with all job_ids and no per-job
call; rollback on platform failure; rollback + no delete when a later mark
fails; 404 for an unmatched table. 8 controller specs pass; maestro builds.
Co-Authored-By: WOZCODE <contact@withwoz.com>
Switch @dadosfera/protospack-v2 from the local file: tarball (removed from
the protospack-v2 repo) to the CI-published CodeArtifact version. Installs
and builds clean; test suites pass.
Co-Authored-By: WOZCODE <contact@withwoz.com>
The guard read `status[status.length-1].last_status` on the pipeline_run
history. For a pipeline that has never run — which is the permanent state
of CDC pipelines, since they replicate continuously and never record
batch runs — pipeline_run is empty, so the last element is undefined and
`.last_status` threw, surfacing as "Error checking pipeline status:
Cannot read properties of undefined (reading 'last_status')". This
blocked Edit Objects (and delete-table) for every CDC pipeline.
Treat an empty/statusless run history as "not running" and allow the
edit (undefined never satisfied 'running' anyway). Also stop the catch
from double-wrapping the deliberate is-running BadRequestException, so
that rejection keeps its clear message; genuine status-check failures
still fail closed with the wrapped message (safe default for a
destructive-op gate).
Adds a guard spec: empty history -> allow, statusless -> allow, not
running -> allow, running -> is-running message (not wrapped), platform
error -> wrapped message.
Co-Authored-By: WOZCODE <contact@withwoz.com>
Claude-Session: https://claude.ai/code/session_01145m1zZMfx8RSJBxAhySdg
The pipeline create body carries `config.tables[].destinations`, which
the CDC path in pi-factory needs to honor a user-supplied raw Snowflake
table name. The gRPC PipelineV2CreateRequest previously had no `config`
field, so `...body` dropped it on the wire.
Bump protospack to cdc.3 (adds optional `config` string). Serialize
`body.config` into the create request the same way `properties` is
handled, and add `config?` to the ICreatePipelineV2Req DTO so it's
typed. Add a spec asserting the gRPC request carries a stringified
config with the destination intact.
Co-Authored-By: WOZCODE <contact@withwoz.com>
Claude-Session: https://claude.ai/code/session_01145m1zZMfx8RSJBxAhySdg
updatePlatformJobs pushes batch sync_mode/memory to jobs by positional
index — meaningless for CDC and corrupting. CDC add/remove use dedicated
endpoints, so skip updatePlatformJobs for CDC inputs. The input record
update still runs.
Co-Authored-By: WOZCODE <contact@withwoz.com>
deleteTable called the DB-row-only DELETE /jobs/{id}, leaving the CDC
connector still replicating a removed table. For CDC jobs, call
DELETE /pipeline/{id}/jobs (RemoveJobsUsecase) with delete_snowflake_tables
false so replication stops but landed data is kept. Batch path unchanged.
Co-Authored-By: WOZCODE <contact@withwoz.com>
- Fix proxy path: /jobs/jdbc/:jobId/rename-tables → /jobs/:jobId/rename-tables
- Remove pipeline_id from ES data asset lookup, search by table_name + table_schema only
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add POST /platform/jobs/:jobId/rename-tables that renames Snowflake
tables via platform-api and syncs the rename to Elasticsearch and
Nimbus (table-metadata, column-metadata, data-preview). If catalog
sync fails, all completed catalog steps are rolled back in reverse
order and the Snowflake rename is reverted.
- Support any connector type (jdbc, singer, s3) via getJobByAnyConnectorType
- Resolve old table names from output_config (raw/qualify)
- Skip qualify sync when output_config.qualify has no table_name
- Add findDataAssetByPipelineAndTable and updateDataAsset to ElasticsearchService
- Add renameTableOnNimbus, renameColumnMetadataOnNimbus, renameDataPreviewOnNimbus to CatalogService
- Add upstream error logging to PlatformApiService
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This fix addresses a breaking change introduced in class-validator 0.14.0
where the default for forbidUnknownValues changed from false to true.
Issue:
- POST /connections was returning 400 "an unknown value was passed to the
validate function" errors
- This occurred because CreateConnectionDto and UpdateConnectionDto have no
validation decorators, causing class-validator 0.14.0+ to treat them as
"unknown values"
- Extra fields (like connector_version) in request payloads would fail
validation
Solution:
- Set forbidUnknownValues: false to allow DTOs without validation decorators
- Set whitelist: true to automatically strip extra properties not defined
in DTOs
- This maintains backward compatibility while adding security by removing
unexpected fields
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
This fix addresses a breaking change introduced in class-validator 0.14.0
where the default for forbidUnknownValues changed from false to true.
Issue:
- POST /connections was returning 400 "an unknown value was passed to the
validate function" errors
- This occurred because CreateConnectionDto and UpdateConnectionDto have no
validation decorators, causing class-validator 0.14.0+ to treat them as
"unknown values"
- Extra fields (like connector_version) in request payloads would fail
validation
Solution:
- Set forbidUnknownValues: false to allow DTOs without validation decorators
- Set whitelist: true to automatically strip extra properties not defined
in DTOs
- This maintains backward compatibility while adding security by removing
unexpected fields
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Add auth_provider to values.yaml (cognito/keycloak)
- Add AUTH_PROVIDER env var to deployment template
- Note: maestro only needs to know which provider is used, duc handles connection
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add auth_provider config to values.yaml (cognito/keycloak)
- Add AUTH_PROVIDER env var to deployment template
- Add conditional Keycloak env vars (URL, realm, client ID, secret)
- Enables parallel deployments with different auth providers
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add authProvider to user entity DTO
- Update user service to include authProvider
- Update auth controller response
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Update multer override from 1.4.5-lts.1 to 2.0.2 to fix HIGH vulnerabilities:
- CVE-2025-47935 (DoS via memory leaks)
- CVE-2025-47944
- CVE-2025-48997
- CVE-2025-7338
Reduces HIGH vulnerabilities from 4 to 0
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Pass TOTP code to DUC for Keycloak users with MFA
- Add Dockerfile.local for local protospack builds
- Add .dockerignore to exclude node_modules from Docker context
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Users no longer need to provide customer_id in the request body for
execute, pause, and unpause pipeline operations - it's now automatically
injected from the authenticated user's session.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The protobuf definition expects reference_column to be an object with
name and type fields, but it was being stored as just a string (column
name). This caused pipeline fetching to fail with the error:
".NewTable.reference_column: object expected"
Changes:
- Update ReferenceColumn interface in DynamoDB service
- Update extractTablesFromJobs to create reference_column object
- Update syncJobInputToDynamoDB to handle reference_column object
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The protobuf definition expects reference_column to be an object with
name and type fields, but it was being stored as just a string (column
name). This caused pipeline fetching to fail with the error:
".NewTable.reference_column: object expected"
Changes:
- Update ReferenceColumn interface in DynamoDB service
- Update extractTablesFromJobs to create reference_column object
- Update syncJobInputToDynamoDB to handle reference_column object
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The type field was missing from the createPipeline call,
causing ES documents to not have the type field set.
Maps: jdbc->database, singer->application, s3->file
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The type field was missing from the createPipeline call,
causing ES documents to not have the type field set.
Maps: jdbc->database, singer->application, s3->file
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add normalization for pipelineId and runId in getPipelineRuns, getPipelineRun, and getPipelineRunLogs
- Remove unused createPipelineRun route
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add ELASTICSEARCH_URL and ELASTICSEARCH_API_KEY env vars
- Pull credentials from {env}/microservices/elasticsearch secret
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add schedule limit validation against customer's scheduleLimit from DUC
- Improve ES update to only update provided fields
- Mark GET /platform/pipeline/:pipelineId/pipeline_run as READY
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add cron-validation utility with Airflow presets support (@once, @daily, etc.)
- Validate cron expressions against customer scheduleLimit from DUC
- Fetch scheduleLimit via gRPC to ensure up-to-date configuration
- Improve ES updatePipeline to fetch current doc and merge changes
- Only update fields explicitly provided in PATCH request body
- Preserve config.tables when updating config.cron
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add Platform API module to proxy requests to Platform API service
- Add Elasticsearch service for pipeline catalog sync (CRUD operations)
- Add DynamoDB service for storing pipeline tables/inputs configuration
- Sync pipeline creation/update/delete to Elasticsearch index
- Extract tables from jobs with connector-specific mappings:
- JDBC: table_name, load_type, column_include_list, incremental_column_name
- Singer: replication_method -> full_load/incremental
- S3: same as Singer
- Map connector types to DynamoDB types (jdbc->database, singer->application, s3->file)
- Validate connector type is provided in job input
- Normalize pipeline IDs for Platform API (replace - with _), keep UUIDs for ES
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Change TLS check from ENV !== 'local' to REDIS_TLS === 'true'
- Add REDIS_TLS env var to Helm deployment template
- Default to true for OCI Redis, can override for self-hosted K8s
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Resolve ingress.yaml timeout annotation conflicts by keeping both configurations
- Resolve stg.yaml conflicts by merging affinity/resources config with restricted_ip
- Fix chart references in unimed-maestro and private-maestro sections
- Update all chart paths from ../maestro to ../helm-chart for consistency
This merge brings in the latest beta changes while preserving our Oracle migration
and affinity standardization configuration.
- Change trigger branches from main/stg to main/beta
- Update environment extraction to map beta → stg environment
- Align with standard branch naming convention (beta for staging)
- Consistent with deploy-manually.yml workflow configuration
Oracle Migration Changes:
- Create deploy/helm-chart/ directory structure (moved from maestro/)
- Create deploy/helmfiles/ directory structure (moved from helmfiles/)
- Update chart references in helmfiles from ../maestro to ../helm-chart
- Update deploy-manually.yml from Azure AKS to Oracle OKE
- Update validate-k8s.yml from Azure AKS to Oracle OKE
- Replace Azure CLI with OCI CLI installation and configuration
- Replace Azure authentication with OCI authentication using secrets
- Replace az aks get-credentials with oci ce cluster create-kubeconfig
- Update helmfile paths from helmfiles/ to deploy/helmfiles/
- Remove DockerHub integration (push_to_dockerhub input and related steps)
- Change runner from [self-hosted, prd-azure] to [self-hosted, prd-oracle]
- Add environment field for proper Oracle deployment environment handling
- Add HOME environment variable for OCI CLI
Affinity/Resources Standardization:
- Add affinity configuration with name=general node selector in values.yaml
- Update deployment template to use conditional affinity and resources blocks
- Configure staging to disable both affinity and resources (null values)
- Enable production to use affinity targeting name=general nodes
- Enable production to use standard resource limits (100m-2000m CPU, 1500Mi-2Gi memory)
This brings maestro into full compliance with Oracle migration requirements and
implements the standardized affinity/resources pattern consistent with in-factory and duc.
Rafael Santana