Review (maestro #510): batch and CDC tables are both removed through
DELETE /pipeline/{id}/jobs (the platform dispatches by type: Airflow refresh
vs Kafka Connect reconfigure), so the connector 'if' in the controller is
gone. deleteTable/deleteTables/addTable are now thin controller methods over
PipelineTablesService (mark -> resolve jobs -> platform -> rollback), with
typed request bodies.
Behavior change for batch: the platform refuses to remove the LAST table of
a pipeline (400 'Cannot remove all jobs'), where DELETE /jobs/{id} allowed it.
Co-Authored-By: WOZCODE <contact@withwoz.com>
Review (maestro #510): drop every 'as any' on the Input write client
(AddCdcTable/RemoveCdcTable/UnmarkTableDeleted take the generated request
types), type IPipelineV2.config, and build the CdcTable payload in one mapper
so 'name' mirrors table_name in a single place. isCdc via helper, no '!!'.
Co-Authored-By: WOZCODE <contact@withwoz.com>
- platform-api.controller.spec: addCdcTable expectations now include the CDC
fields the controller threads (iceberg_table_name, iceberg_qualify_table_name,
column_exclude_list) which were added by the CDC-iceberg work.
- release_note specs: provide DadosferaLogger mock — ReleaseNoteService gained an
@Inject(DadosferaLogger) dependency (from beta) without its specs being updated,
so they failed DI resolution on merge.
Full suite: 52 passed, 5 skipped, 0 failed.
Co-Authored-By: WOZCODE <contact@withwoz.com>
Adds CdcColumnReq {name, type, is_primary_key} and columns? on
CdcTableReq, forwarded through createCdc and the addTable (Edit
Objects add-table) path so the column schema reaches in-factory for
Iceberg deduped-table pre-create. Bumps protospack-v2 to
3.41.0-cdc-iceberg.1, which adds the matching CdcColumn field
(now required on CdcTable) and regenerates docsfera.json.
Co-Authored-By: WOZCODE <contact@withwoz.com>
The POST /pipelines/:pipelineId/inputs/:inputId/tables route built its
CdcTable payload field-by-field and silently dropped iceberg_table_name
even though inputsService.addCdcTable/the gRPC AddCdcTable call (and the
protospack CdcTable message) already support it. Widen the inline request
body type and thread the field into the addCdcTable payload; absent for
snowflake, unchanged back-compat.
Co-Authored-By: WOZCODE <contact@withwoz.com>
Removing N tables via Edit Objects previously looped a single-table
DELETE per table (maestro deleteTable hardcodes job_ids:[one]), so the
Debezium source + Snowflake sink connectors were rewritten/restarted once
per table. platform-api's DELETE /pipeline/:id/jobs already batches (2
connector writes total for any N), but nothing above it used the array.
New maestro DELETE /pipelines/:pipelineId/inputs/:inputId/tables takes
{ table_names: [] }: soft-deletes each in DynamoDB (tracking successes),
resolves all table_names -> job_ids from the platform pipeline in one GET,
then makes ONE DELETE /pipeline/:id/jobs with all job_ids. All-or-nothing:
any failure (a later mark, an unmatched table, or the platform delete)
rolls back only the marks made in this call.
The single-table deleteTable route is kept (unchanged) — nothing else
depends on removing it, and that's a separable cleanup.
Tests: N tables -> one platform DELETE with all job_ids and no per-job
call; rollback on platform failure; rollback + no delete when a later mark
fails; 404 for an unmatched table. 8 controller specs pass; maestro builds.
Co-Authored-By: WOZCODE <contact@withwoz.com>
deleteTable called the DB-row-only DELETE /jobs/{id}, leaving the CDC
connector still replicating a removed table. For CDC jobs, call
DELETE /pipeline/{id}/jobs (RemoveJobsUsecase) with delete_snowflake_tables
false so replication stops but landed data is kept. Batch path unchanged.
Co-Authored-By: WOZCODE <contact@withwoz.com>
