Add POST /platform/jobs/:jobId/rename-tables that renames Snowflake
tables via platform-api and syncs the rename to Elasticsearch and
Nimbus (table-metadata, column-metadata, data-preview). If catalog
sync fails, all completed catalog steps are rolled back in reverse
order and the Snowflake rename is reverted.
- Support any connector type (jdbc, singer, s3) via getJobByAnyConnectorType
- Resolve old table names from output_config (raw/qualify)
- Skip qualify sync when output_config.qualify has no table_name
- Add findDataAssetByPipelineAndTable and updateDataAsset to ElasticsearchService
- Add renameTableOnNimbus, renameColumnMetadataOnNimbus, renameDataPreviewOnNimbus to CatalogService
- Add upstream error logging to PlatformApiService
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Enhanced logs to include more detailed request body information.
- Standardized log calls to include extra metadata, such as service name, environment, customer name, user ID, HTTP method, and request path.
- Improved error logging to provide better debugging insights.
Example log format:
EXTRA[{"service":{"name":"maestro","environment":"stg"},"customer_name":"dadosfera","user_id":"047824d8-0021-709a-bf15-39b8dd069aaf","method":"POST","path":"/catalog/register-dataset"}]
Related tickets or issues: [TECN-9441]
As part of our hybrid approach, we need to allow dataset cataloging from outside our cloud environment.
Instead of relying on our GRPC APIs, we leveraged an existing REST API to handle this process.
- Updated Maestro to call Nimbus directly for dataset registration.
- Modified protospack to generate the necessary TypeScript interfaces.
- Added the `/catalog/register-dataset` endpoint to the API.
This change enables seamless dataset registration from external sources while maintaining compatibility with our existing infrastructure.
Related tickets or issues: [TECN-9441]
marcos-silva-rodrigues