FEAT: add Platform API proxy with Elasticsearch and DynamoDB sync

- Add Platform API module to proxy requests to Platform API service
- Add Elasticsearch service for pipeline catalog sync (CRUD operations)
- Add DynamoDB service for storing pipeline tables/inputs configuration
- Sync pipeline creation/update/delete to Elasticsearch index
- Extract tables from jobs with connector-specific mappings:
  - JDBC: table_name, load_type, column_include_list, incremental_column_name
  - Singer: replication_method -> full_load/incremental
  - S3: same as Singer
- Map connector types to DynamoDB types (jdbc->database, singer->application, s3->file)
- Validate connector type is provided in job input
- Normalize pipeline IDs for Platform API (replace - with _), keep UUIDs for ES

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
Rafael
2025-12-10 12:42:18 -03:00
co-authored by Claude Opus 4.5
parent e789076ed4
commit df674dd441
15 changed files with 3277 additions and 295 deletions
@@ -0,0 +1,11 @@
export const PLATFORM_API_CONFIG = {
getUrl: (): string => {
const url = process.env.PLATFORM_API_URL;
if (!url) {
throw new Error('PLATFORM_API_URL environment variable is not set');
}
return url;
},
region: process.env.AWS_REGION || 'us-east-1',
timeout: parseInt(process.env.PLATFORM_API_TIMEOUT || '30000', 10),
};