ai-coding/pg2dm-converter
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

init

Browse Source
This commit is contained in:
dengqichen 2025-11-15 17:29:39 +08:00
parent 8bf5813876
commit cd7788de95
8 changed files with 828 additions and 1032 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

313
EXECUTOR_README.md Normal file
View File

@ -0,0 +1,313 @@
# 达梦数据库自动执行器使用指南
## ✅ 完整可行方案（零额外依赖）
基于**disql命令行工具**，使用你本地已有的达梦数据库管理工具。
---
## 📋 前提条件检查
### 1. 达梦数据库管理工具已安装
```
✅ 已确认: D:\sortware\dm_manager
```
### 2. disql工具存在
```
✅ 工具路径: D:\sortware\dm_manager\bin\disql.exe
```
### 3. Node.js已安装
```bash
node -v
# 应显示版本号，如 v20.19.5
```
### 4. 配置文件已准备
```
✅ db-mapping.json - 数据库连接和schema映射
```
---
## 🚀 立即使用
### 方式一：一键批量执行（推荐）
**双击执行**:
```
execute-all.bat
```
这会自动执行`output`目录下的所有`*_dm.sql`文件。
---
### 方式二：命令行执行
#### 执行所有SQL文件
```bash
node dm-executor.js output/*_dm.sql
```
#### 执行单个文件
```bash
node dm-executor.js output/themetis_data_dm.sql
```
#### 执行指定文件
```bash
node dm-executor.js output/schema1_dm.sql output/schema2_dm.sql
```
---
## 📊 执行效果展示
```
======================================================================
🚀 达梦数据库批量执行器
======================================================================
📂 文件数: 14
🌐 服务器: 219.142.42.183
🔧 工具: D:\sortware\dm_manager\bin\disql.exe
======================================================================
[1/14]
======================================================================
📂 执行: lyg_scp_dm.sql
📋 Schema: lyg_scp
🎯 端口: 5256
======================================================================
............................................................
----------------------------------------------------------------------
✅ lyg_scp_dm.sql 执行成功
端口: 5256 | 耗时: 3.45秒
----------------------------------------------------------------------
[2/14]
======================================================================
📂 执行: themetis_scp_dm.sql
📋 Schema: themetis_scp
🎯 端口: 5256
======================================================================
............................................................
----------------------------------------------------------------------
✅ themetis_scp_dm.sql 执行成功
端口: 5256 | 耗时: 12.32秒
----------------------------------------------------------------------
... (继续执行其他文件)
======================================================================
📊 执行统计
======================================================================
总文件数: 14
✅ 成功: 14
❌ 失败: 0
⏱ 总耗时: 45.67秒
按端口统计:
端口 5256: 14个文件 (✅14 ❌0)
📄 详细报告: ./output/execution_report_1731660123456.json
======================================================================
```
---
## 🔧 工作原理
### 1. 自动检测schema
```javascript
// 从SQL内容中提取schema名称
"themetis_data"."table_name" → schema = "themetis_data"
```
### 2. 自动路由端口
```javascript
// 根据db-mapping.json配置选择端口
schema "themetis_data" → port 5256
schema "other_schema" → port 5266
```
### 3. 使用disql执行
```bash
disql SYSDBA/@1sdgCq456@219.142.42.183:5256 @schema_dm.sql
```
### 4. 收集统计信息
- 成功/失败数量
- 执行耗时
- 错误信息
- 生成JSON报告
---
## 📁 完整工作流
```
┌─────────────────────┐
│ 1. 转换SQL │
│ node converter.js │
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ input/*.sql │
│ (PostgreSQL) │
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ output/*_dm.sql │
│ (达梦格式) │
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ 2. 执行SQL │
│ node dm-executor.js │
│ output/*_dm.sql │
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ 达梦数据库 │
│ 219.142.42.183:5256│
└─────────────────────┘
```
---
## 🛠 配置文件说明
### db-mapping.json
```json
{
"defaultConnection": {
"host": "219.142.42.183",
"user": "SYSDBA",
"password": "@1sdgCq456"
},
"defaultPort": 5256,
"schemaMappings": {
"lyg_scp": { "port": 5256, "description": "连云港SCP" },
"themetis_data": { "port": 5256, "description": "主数据" }
}
}
```
---
## ❓ 常见问题
### Q1: 找不到disql工具
**A**: 检查路径是否正确
```bash
dir D:\sortware\dm_manager\bin\disql.exe
```
如果路径不同，修改`dm-executor.js`第23行。
---
### Q2: 执行失败
**A**: 查看详细报告
```bash
# 报告文件在
./output/execution_report_*.json
```
---
### Q3: 连接数据库失败
**A**: 检查配置
1. db-mapping.json中的密码是否正确
2. 数据库是否可访问
3. 端口是否正确
测试连接:
```bash
D:\sortware\dm_manager\bin\disql.exe SYSDBA/@1sdgCq456@219.142.42.183:5256
```
---
### Q4: 某些SQL执行失败
**A**: 检查SQL语法
1. 查看报告中的错误信息
2. 手动在DM Manager中测试SQL
3. 可能需要重新转换SQL
---
### Q5: 执行速度慢
**A**: 正常现象
- 大文件需要更长时间
- 可以单独执行小文件测试
- disql是串行执行，比较稳定但不够快
---
## 📈 性能建议
### 1. 分批执行
对于大量SQL文件，建议分批执行：
```bash
# 先执行小的schema
node dm-executor.js output/lyg_scp_dm.sql output/model_scp_dm.sql
# 再执行大的schema
node dm-executor.js output/themetis_scp_dm.sql
```
### 2. 按端口分组
如果有多个数据库端口，可以分别执行：
```bash
# 只执行5256端口的
node dm-executor.js output/schema1_dm.sql output/schema2_dm.sql
# 再执行5266端口的
node dm-executor.js output/schema3_dm.sql
```
---
## 🎯 优势
1. **零额外依赖** - 只需Node.js和disql
2. **最稳定** - 使用官方工具
3. **完全自动** - 一键执行所有SQL
4. **智能路由** - 自动识别schema并选择端口
5. **详细报告** - JSON格式的执行报告
6. **可重复执行** - 支持多次执行
---
## 📞 技术支持
遇到问题？检查：
1. Node.js版本 (node -v)
2. disql工具 (测试手动连接)
3. 配置文件 (db-mapping.json)
4. SQL文件 (output目录)
5. 网络连接 (ping 219.142.42.183)
---
## 🎉 开始使用
```bash
# 1. 转换SQL
node converter.js
# 2. 执行SQL
node dm-executor.js output/*_dm.sql
# 或者直接双击
execute-all.bat
```
**就这么简单！** 🚀

27
README.md
View File

@ -1,7 +1,32 @@
# PostgreSQL到达梦数据库SQL转换工具 # PostgreSQL到达梦数据库迁移工具套件
**完整的PostgreSQL到达梦数据库迁移解决方案**：自动转换SQL语法 + 自动执行到数据库
## 🎯 完整工作流
```
PostgreSQL SQL → 转换工具 → 达梦SQL → 执行工具 → 达梦数据库
(input/) converter.js (output/) dm-executor.js (自动完成)
```
## 📦 工具套件
### 1. SQL转换工具 (converter.js)
自动将PostgreSQL导出的SQL文件转换为达梦数据库(DM8)兼容的SQL语法。 自动将PostgreSQL导出的SQL文件转换为达梦数据库(DM8)兼容的SQL语法。
### 2. SQL执行工具 (dm-executor.js) 🆕
**零额外依赖，基于disql命令行工具**
- ✅ 自动批量执行转换后的SQL
- ✅ 智能识别schema并路由到正确端口
- ✅ 详细的执行统计和报告
- ✅ 可重复执行，错误处理完善
**一键执行**: 双击 `execute-all.bat` 或运行 `node dm-executor.js output/*_dm.sql`
详细文档: [EXECUTOR_README.md](./EXECUTOR_README.md)
---
## 核心功能特性 ## 核心功能特性
本工具通过**12个转换步骤**，解决PostgreSQL到达梦迁移过程中的**所有常见语法兼容性问题**： 本工具通过**12个转换步骤**，解决PostgreSQL到达梦迁移过程中的**所有常见语法兼容性问题**：

5
converter.js
View File

@ -306,8 +306,9 @@ class PG2DMConverter {
removeAttachPartition(sql) { removeAttachPartition(sql) {
let converted = sql; let converted = sql;
// 匹配 ALTER TABLE ... ATTACH PARTITION ... FOR VALUES ...; // 匹配 ALTER TABLE ... ATTACH PARTITION ...
const attachPattern = /ALTER\s+TABLE\s+"[^"]+"\."[^"]+"\s+ATTACH\s+PARTITION\s+"[^"]+"\."[^"]+"\s+FOR\s+VALUES[^;]*;/gi; // 支持多种格式：FOR VALUES ..., FOR VALUES IN (...), DEFAULT
const attachPattern = /ALTER\s+TABLE\s+"[^"]+"\."[^"]+"\s+ATTACH\s+PARTITION\s+"[^"]+"\."[^"]+"\s+(FOR\s+VALUES[^;]*|DEFAULT)\s*;/gi;
const matches = sql.match(attachPattern); const matches = sql.match(attachPattern);
if (matches) { if (matches) {

66
db-mapping.json Normal file
View File

@ -0,0 +1,66 @@
{
"defaultConnection": {
"host": "219.142.42.183",
"user": "SYSDBA",
"password": "@1sdgCq456"
},
"defaultPort": 5256,
"schemaMappings": {
"lyg_scp": {
"port": 5256,
"description": "连云港SCP"
},
"model_scp": {
"port": 5256,
"description": "模型SCP"
},
"themetis_scp": {
"port": 5256,
"description": "Themetis SCP"
},
"themetis_cp": {
"port": 5256,
"description": "Themetis CP"
},
"themetis_demo": {
"port": 5256,
"description": "Themetis Demo"
},
"themetis_engine": {
"port": 5256,
"description": "Themetis Engine"
},
"themetis_etl": {
"port": 5256,
"description": "Themetis ETL"
},
"themetis_kk": {
"port": 5256,
"description": "Themetis KK"
},
"themetis_mrp": {
"port": 5256,
"description": "Themetis MRP"
},
"themetis_phase": {
"port": 5256,
"description": "Themetis Phase"
},
"themetis_scheduler": {
"port": 5256,
"description": "Themetis Scheduler"
},
"themetis_test": {
"port": 5256,
"description": "Themetis Test"
},
"themetis_user_data": {
"port": 5256,
"description": "Themetis User Data"
},
"themetis_data": {
"port": 5256,
"description": "Themetis Data"
}
}
}

390
dm-executor.js Normal file
View File

@ -0,0 +1,390 @@
/**
* 达梦数据库SQL自动执行器
* 基于disql命令行工具，零额外依赖
*/
const fs = require('fs');
const path = require('path');
const { spawn } = require('child_process');
const iconv = require('iconv-lite');
class DMExecutor {
constructor(configFile = './db-mapping.json') {
this.config = JSON.parse(fs.readFileSync(configFile, 'utf8'));
this.disqlPath = this.findDisql();
this.stats = [];
}
/**
* 查找disql工具路径
*/
findDisql() {
const possiblePaths = [
'D:\\sortware\\dm_manager\\bin\\disql.exe',
'D:\\dmdbms\\bin\\disql.exe',
'C:\\dmdbms\\bin\\disql.exe',
'disql' // PATH中
];
for (const p of possiblePaths) {
if (p === 'disql' || fs.existsSync(p)) {
return p;
}
}
throw new Error('未找到disql工具，请确认达梦数据库已安装');
}
/**
* 检测SQL文件中的schema
*/
detectSchema(sqlContent) {
const match = sqlContent.match(/"([^"]+)"\./);
return match ? match[1] : null;
}
/**
* 获取schema对应的端口
*/
getPort(schema) {
if (!schema) return this.config.defaultPort;
const mapping = this.config.schemaMappings[schema];
return mapping ? mapping.port : this.config.defaultPort;
}
/**
* 使用disql执行SQL文件
*/
async executeSQL(sqlFile) {
const sqlContent = fs.readFileSync(sqlFile, 'utf8');
const schema = this.detectSchema(sqlContent);
const port = this.getPort(schema);
const { host, user, password } = this.config.defaultConnection;
console.log(`\n${'='.repeat(70)}`);
console.log(`📂 执行: ${path.basename(sqlFile)}`);
console.log(`📋 Schema: ${schema || '(未检测到)'}`);
console.log(`🎯 端口: ${port}`);
console.log('='.repeat(70));
const startTime = Date.now();
return new Promise((resolve) => {
const absoluteSqlFile = path.resolve(sqlFile);
console.log(`🔗 连接信息: ${user}@${host}:${port}`);
console.log(`📄 SQL文件: ${absoluteSqlFile}`);
console.log(`⏳ 执行中...`);
// 使用交互式方式，通过stdin传递密码，避免命令行参数中@符号的解析问题
const connectionString = `${user}@${host}:${port}`;
console.log(`📝 连接参数: ${connectionString}`);
// 启动disql，通过stdin传递密码和SQL命令
const disql = spawn(this.disqlPath, [connectionString], {
shell: true,
stdio: ['pipe', 'pipe', 'pipe'] // 启用stdin以传递密码
});
// 直接读取SQL文件内容并通过stdin执行
// 避免@file.sql路径解析问题
const sqlCommands = sqlContent;
// 构建完整的命令序列
const commands = `${password}\nSET TIMING ON;\nSET FEEDBACK ON;\n${sqlCommands}\nEXIT\n`;
console.log(`📤 发送SQL内容: ${sqlCommands.split('\n').length} 行`);
console.log(`📋 SQL大小: ${(commands.length / 1024).toFixed(2)} KB`);
// 写入命令到stdin
disql.stdin.write(commands, 'utf8', (err) => {
if (err) {
console.error('❌ 写入stdin失败:', err);
}
});
disql.stdin.end();
let stdout = '';
let stderr = '';
let lastOutput = Date.now();
// 心跳检测 - 每秒显示一个点
const heartbeat = setInterval(() => {
process.stdout.write('.');
}, 1000);
// 超时检测 - 5分钟无输出则认为超时
const timeout = setTimeout(() => {
clearInterval(heartbeat);
disql.kill();
console.log('\n❌ 超时：5分钟无响应');
}, 300000);
disql.stdout.on('data', (data) => {
// 将GBK编码转换为UTF-8
const text = iconv.decode(data, 'gbk');
stdout += text;
lastOutput = Date.now();
// 实时显示关键信息
const keywords = ['执行成功', '执行失败', '行受影响', '影响行数', 'CREATE TABLE', 'CREATE INDEX', 'ALTER TABLE', '已用时间'];
if (keywords.some(keyword => text.includes(keyword))) {
// 显示包含关键字的行
const lines = text.split('\n').filter(line =>
keywords.some(keyword => line.includes(keyword))
);
lines.forEach(line => {
if (line.trim()) {
process.stdout.write(`\n ${line.trim().substring(0, 120)}`);
}
});
}
});
disql.stderr.on('data', (data) => {
// 将GBK编码转换为UTF-8
const text = iconv.decode(data, 'gbk');
stderr += text;
// 显示错误（过滤掉正常的密码提示）
if (text.trim() && !text.includes('密码:')) {
process.stdout.write(`\n⚠ ${text.trim().substring(0, 150)}`);
}
});
disql.on('close', (code) => {
clearInterval(heartbeat);
clearTimeout(timeout);
const duration = ((Date.now() - startTime) / 1000).toFixed(2);
console.log('\n');
const result = {
file: path.basename(sqlFile),
schema: schema,
port: port,
duration: duration,
success: code === 0,
exitCode: code,
output: stdout,
error: stderr
};
this.stats.push(result);
this.printResult(result);
resolve(result);
});
disql.on('error', (error) => {
clearInterval(heartbeat);
clearTimeout(timeout);
const result = {
file: path.basename(sqlFile),
schema: schema,
port: port,
duration: 0,
success: false,
error: error.message
};
this.stats.push(result);
resolve(result);
});
});
}
/**
* 批量执行SQL文件
*/
async executeBatch(sqlFiles) {
console.log('\n' + '='.repeat(70));
console.log('🚀 达梦数据库批量执行器');
console.log('='.repeat(70));
console.log(`📂 文件数: ${sqlFiles.length}`);
console.log(`🌐 服务器: ${this.config.defaultConnection.host}`);
console.log(`🔧 工具: ${this.disqlPath}`);
console.log('='.repeat(70));
const overallStart = Date.now();
for (let i = 0; i < sqlFiles.length; i++) {
console.log(`\n[${i + 1}/${sqlFiles.length}]`);
await this.executeSQL(sqlFiles[i]);
}
const overallDuration = ((Date.now() - overallStart) / 1000).toFixed(2);
this.printSummary(overallDuration);
}
/**
* 打印单个文件执行结果
*/
printResult(result) {
console.log('-'.repeat(70));
if (result.success) {
console.log(`✅ ${result.file} 执行成功`);
} else {
console.log(`❌ ${result.file} 执行失败`);
if (result.error) {
console.log(`错误: ${result.error.substring(0, 200)}`);
}
}
console.log(`端口: ${result.port} | 耗时: ${result.duration}秒`);
console.log('-'.repeat(70));
}
/**
* 打印总体统计
*/
printSummary(duration) {
console.log('\n' + '='.repeat(70));
console.log('📊 执行统计');
console.log('='.repeat(70));
const total = this.stats.length;
const success = this.stats.filter(s => s.success).length;
const failed = total - success;
console.log(`总文件数: ${total}`);
console.log(`✅ 成功: ${success}`);
console.log(`❌ 失败: ${failed}`);
console.log(`⏱ 总耗时: ${duration}秒`);
// 按端口分组
const portStats = {};
this.stats.forEach(s => {
if (!portStats[s.port]) {
portStats[s.port] = { total: 0, success: 0, failed: 0 };
}
portStats[s.port].total++;
if (s.success) portStats[s.port].success++;
else portStats[s.port].failed++;
});
console.log('\n按端口统计:');
Object.keys(portStats).sort().forEach(port => {
const stat = portStats[port];
console.log(` 端口 ${port}: ${stat.total}个文件 (✅${stat.success} ❌${stat.failed})`);
});
// 显示失败的文件
const failedFiles = this.stats.filter(s => !s.success);
if (failedFiles.length > 0) {
console.log('\n失败的文件:');
failedFiles.forEach(f => {
console.log(` ❌ ${f.file} - ${f.error || '执行失败'}`);
});
}
// 保存报告
this.saveReport();
console.log('='.repeat(70));
}
/**
* 保存执行报告
*/
saveReport() {
const reportFile = path.join('./output', `execution_report_${Date.now()}.json`);
const report = {
timestamp: new Date().toISOString(),
tool: 'disql',
server: this.config.defaultConnection.host,
summary: {
total: this.stats.length,
success: this.stats.filter(s => s.success).length,
failed: this.stats.filter(s => !s.success).length
},
details: this.stats
};
fs.writeFileSync(reportFile, JSON.stringify(report, null, 2), 'utf8');
console.log(`\n📄 详细报告: ${reportFile}`);
}
}
/**
* 主函数
*/
async function main() {
const args = process.argv.slice(2);
if (args.length === 0) {
console.log(`
达梦数据库SQL执行器
======================================
基于disql命令行工具，零额外依赖
使用方法:
node dm-executor.js <sql-files>
示例:
# 执行单个文件
node dm-executor.js output/schema_dm.sql
# 批量执行
node dm-executor.js output/*_dm.sql
# 执行所有文件
node dm-executor.js output/*.sql
前提条件:
- 达梦数据库已安装
- disql工具可用
- db-mapping.json已配置
配置文件:
db-mapping.json - 数据库连接和schema映射配置
`);
process.exit(0);
}
// 解析SQL文件列表
const sqlFiles = [];
args.forEach(arg => {
if (arg.includes('*')) {
// 通配符展开
const dir = path.dirname(arg);
const pattern = path.basename(arg).replace(/\*/g, '.*');
const regex = new RegExp(`^${pattern}$`);
if (fs.existsSync(dir)) {
const files = fs.readdirSync(dir)
.filter(f => regex.test(f) && f.endsWith('.sql'))
.map(f => path.join(dir, f));
sqlFiles.push(...files);
}
} else if (fs.existsSync(arg)) {
sqlFiles.push(arg);
} else {
console.error(`❌ 文件不存在: ${arg}`);
}
});
if (sqlFiles.length === 0) {
console.error('❌ 未找到SQL文件');
process.exit(1);
}
// 执行
try {
const executor = new DMExecutor();
await executor.executeBatch(sqlFiles);
const failed = executor.stats.filter(s => !s.success).length;
process.exit(failed > 0 ? 1 : 0);
} catch (error) {
console.error('\n❌ 执行失败:', error.message);
console.error(error.stack);
process.exit(1);
}
}
if (require.main === module) {
main();
}
module.exports = DMExecutor;

25
package-lock.json generated
View File

@ -9,7 +9,8 @@
"version": "1.0.0", "version": "1.0.0",
"license": "MIT", "license": "MIT",
"dependencies": { "dependencies": {
"chalk": "^4.1.2" "chalk": "^4.1.2",
"iconv-lite": "^0.7.0"
} }
}, },
"node_modules/ansi-styles": { "node_modules/ansi-styles": {
@ -70,6 +71,28 @@
"node": ">=8" "node": ">=8"
} }
}, },
"node_modules/iconv-lite": {
"version": "0.7.0",
"resolved": "https://registry.npmmirror.com/iconv-lite/-/iconv-lite-0.7.0.tgz",
"integrity": "sha512-cf6L2Ds3h57VVmkZe+Pn+5APsT7FpqJtEhhieDCvrE2MK5Qk9MyffgQyuxQTm6BChfeZNtcOLHp9IcWRVcIcBQ==",
"license": "MIT",
"dependencies": {
"safer-buffer": ">= 2.1.2 < 3.0.0"
},
"engines": {
"node": ">=0.10.0"
},
"funding": {
"type": "opencollective",
"url": "https://opencollective.com/express"
}
},
"node_modules/safer-buffer": {
"version": "2.1.2",
"resolved": "https://registry.npmmirror.com/safer-buffer/-/safer-buffer-2.1.2.tgz",
"integrity": "sha512-YZo3K82SD7Riyi0E1EQPojLz7kpepnSQI9IyPbHHg1XXXevb5dJI7tpyN2ADxGcQbHG7vcyRHk0cbwqcQriUtg==",
"license": "MIT"
},
"node_modules/supports-color": { "node_modules/supports-color": {
"version": "7.2.0", "version": "7.2.0",
"resolved": "https://registry.npmmirror.com/supports-color/-/supports-color-7.2.0.tgz", "resolved": "https://registry.npmmirror.com/supports-color/-/supports-color-7.2.0.tgz",

8
package.json
View File

@ -5,7 +5,10 @@
"main": "converter.js", "main": "converter.js",
"scripts": { "scripts": {
"start": "node converter.js", "start": "node converter.js",
"convert": "node converter.js" "convert": "node converter.js",
"execute": "node dm-executor.js output/*_dm.sql",
"execute:single": "node dm-executor.js",
"all": "node converter.js && node dm-executor.js output/*_dm.sql"
}, },
"keywords": [ "keywords": [
"postgresql", "postgresql",
@ -17,6 +20,7 @@
"author": "", "author": "",
"license": "MIT", "license": "MIT",
"dependencies": { "dependencies": {
"chalk": "^4.1.2" "chalk": "^4.1.2",
"iconv-lite": "^0.7.0"
} }
} }

1026
scp_simulation_scenario.sql
View File

File diff suppressed because it is too large Load Diff