有許多情況下，您可能需要從 XML 導出數據到 MongoDB。
儘管在 MongoDB 中使用的 XML 和 JSON(B) 格式有許多共同之處，但它們也有一些不同之處，這使它們不可互換。
因此，在面對從 XML 導出數據到 MongoDB 的任務之前，您將需要：
- 編寫自己的 XML 解析腳本；
- 使用 ETL 工具。
儘管現代語言模型可以很好地編寫解析腳本，比如 Python，這些腳本將存在一個嚴重問題 — 它們不會統一。對於每種文件類型，現代語言模型將生成一個單獨的腳本。如果您有多種類型的 XML，這已經在維護多個解析腳本上造成了重大問題。
上述問題通常使用專門的 ETL 工具來解決。在本文中，我們將介紹一個名為 SmartXML 的 ETL 工具。雖然 SmartXML 也支持將 XML 轉換為 關系表示形式，但我們將只關注將 XML 上傳到 MongoDB 的過程。
實際的 XML 可能非常龐大和複雜。本文是一篇入門文章，因此我們將分析一種情況，即：
- 所有 XML 具有相同的結構；
- XML的邏輯模型與MongoDB中的存儲模型相同；
- 提取的字段不需要進行復雜處理；
稍後我們將涵蓋這些情況，但首先，讓我們看一個簡單的例子：
XML
<marketingData>
<customer>
<name>John Smith</name>
<email>[email protected]</email>
<purchases>
<purchase>
<product>Smartphone</product>
<category>Electronics</category>
<price>700</price>
<store>TechWorld</store>
<location>New York</location>
<purchaseDate>2025-01-10</purchaseDate>
</purchase>
<purchase>
<product>Wireless Earbuds</product>
<category>Audio</category>
<price>150</price>
<store>GadgetStore</store>
<location>New York</location>
<purchaseDate>2025-01-11</purchaseDate>
</purchase>
</purchases>
<importantInfo>
<loyaltyStatus>Gold</loyaltyStatus>
<age>34</age>
<gender>Male</gender>
<membershipID>123456</membershipID>
</importantInfo>
<lessImportantInfo>
<browser>Chrome</browser>
<deviceType>Mobile</deviceType>
<newsletterSubscribed>true</newsletterSubscribed>
</lessImportantInfo>
</customer>
<customer>
<name>Jane Doe</name>
<email>[email protected]</email>
<purchases>
<purchase>
<product>Laptop</product>
<category>Electronics</category>
<price>1200</price>
<store>GadgetStore</store>
<location>San Francisco</location>
<purchaseDate>2025-01-12</purchaseDate>
</purchase>
<purchase>
<product>USB-C Adapter</product>
<category>Accessories</category>
<price>30</price>
<store>TechWorld</store>
<location>San Francisco</location>
<purchaseDate>2025-01-13</purchaseDate>
</purchase>
<purchase>
<product>Keyboard</product>
<category>Accessories</category>
<price>80</price>
<store>OfficeMart</store>
<location>San Francisco</location>
<purchaseDate>2025-01-14</purchaseDate>
</purchase>
</purchases>
<importantInfo>
<loyaltyStatus>Silver</loyaltyStatus>
<age>28</age>
<gender>Female</gender>
<membershipID>654321</membershipID>
</importantInfo>
<lessImportantInfo>
<browser>Safari</browser>
<deviceType>Desktop</deviceType>
<newsletterSubscribed>false</newsletterSubscribed>
</lessImportantInfo>
</customer>
<customer>
<name>Michael Johnson</name>
<email>[email protected]</email>
<purchases>
<purchase>
<product>Headphones</product>
<category>Audio</category>
<price>150</price>
<store>AudioZone</store>
<location>Chicago</location>
<purchaseDate>2025-01-05</purchaseDate>
</purchase>
</purchases>
<importantInfo>
<loyaltyStatus>Bronze</loyaltyStatus>
<age>40</age>
<gender>Male</gender>
<membershipID>789012</membershipID>
</importantInfo>
<lessImportantInfo>
<browser>Firefox</browser>
<deviceType>Tablet</deviceType>
<newsletterSubscribed>true</newsletterSubscribed>
</lessImportantInfo>
</customer>
<customer>
<name>Emily Davis</name>
<email>[email protected]</email>
<purchases>
<purchase>
<product>Running Shoes</product>
<category>Sportswear</category>
<price>120</price>
<store>FitShop</store>
<location>Los Angeles</location>
<purchaseDate>2025-01-08</purchaseDate>
</purchase>
<purchase>
<product>Yoga Mat</product>
<category>Sportswear</category>
<price>40</price>
<store>FitShop</store>
<location>Los Angeles</location>
<purchaseDate>2025-01-09</purchaseDate>
</purchase>
</purchases>
<importantInfo>
<loyaltyStatus>Gold</loyaltyStatus>
<age>25</age>
<gender>Female</gender>
<membershipID>234567</membershipID>
</importantInfo>
<lessImportantInfo>
<browser>Edge</browser>
<deviceType>Mobile</deviceType>
<newsletterSubscribed>false</newsletterSubscribed>
</lessImportantInfo>
</customer>
<customer>
<name>Robert Brown</name>
<email>[email protected]</email>
<purchases>
<purchase>
<product>Smartwatch</product>
<category>Wearable</category>
<price>250</price>
<store>GadgetPlanet</store>
<location>Boston</location>
<purchaseDate>2025-01-07</purchaseDate>
</purchase>
<purchase>
<product>Fitness Band</product>
<category>Wearable</category>
<price>100</price>
<store>HealthMart</store>
<location>Boston</location>
<purchaseDate>2025-01-08</purchaseDate>
</purchase>
</purchases>
<importantInfo>
<loyaltyStatus>Silver</loyaltyStatus>
<age>37</age>
<gender>Male</gender>
<membershipID>345678</membershipID>
</importantInfo>
<lessImportantInfo>
<browser>Chrome</browser>
<deviceType>Mobile</deviceType>
<newsletterSubscribed>true</newsletterSubscribed>
</lessImportantInfo>
</customer>
</marketingData>
在這個例子中，我們將只在MongoDB中上傳那些具有實際目的的字段，而不是整個XML文件。
創建一個新項目
建議從GUI中創建一個新項目。這將自動創建必要的文件夾結構和解析規則。項目結構的詳細描述可在官方文檔中找到。
本文中描述的所有參數都可以在圖形模式下配置，但為了清晰起見，我們將專注於文本表示。
除了包含項目設置的config.txt文件和用於批處理的job.txt外，項目本身還包括：
- 位於項目文件夾
templates/data-templates.red中的中間內部SmartDOM視圖模板。 SmartDOM本身的處理和轉換規則，位於rules文件夾中。
讓我們看一下data-templates.red的結構：
Plain Text
#[
sample: #[
marketing_data: #[
customers: [
customer: [
name: none
email: none
purchases: [
purchase: [
product: none
category: none
price: none
store: none
location: none
purchase_date: none
]
]
]
]
]
]
]
注意
- 名稱
sample是類別的名稱，並且並不重要。 marketing_data是子類別的名稱。我們至少需要一個代碼子類別（子類型）。- 中間視圖名稱不需要與 XML 標記名稱完全匹配。在這個例子中，我們故意使用了
snake_case樣式。
提取規則
這些規則位於項目文件夾中的 rules 目錄中。
在使用 MongoDB 時，我們只對兩個規則感興趣：
tags-matching-rules.red— 設置 XML 標記樹與 SmartDOM 之間的匹配grow-rules.red— 描述 SmartDOM 节点与真实 XML 节点之间的关系
Plain Text
sample: [
purchase: ["purchase"]
customer: ["customer"]
]
密钥將是 SmartDOM 中节点的名称；值將是包含真实 XML 文件中节点拼写变体的数组。在我们的示例中，这些名称是相同的。
忽略的標記
为了避免在上述示例中將次要數據加載到 MongoDB 中，我們在 ignores 文件夾中創建文件 — 每個部分一個文件，以每個部分命名。這些文件包含在提取過程中要跳過的標記列表。在我们的示例中，我們將有一個包含以下內容的 sample.txt 文件：
Plain Text
["marketingData" "customer" "lessImportantInfo" "browser"]
["marketingData" "customer" "lessImportantInfo" "deviceType"]
["marketingData" "customer" "lessImportantInfo" "newsletterSubscribed"]
因此，在分析形態時，中間表示將採用以下形式：
Plain Text
customers: [
customer: [
name: "John Smith"
email: "[email protected]"
loyalty_status: "Gold"
age: "34"
gender: "Male"
membership_id: "123456"
purchases: [
purchase: [
product: "Smartphone"
category: "Electronics"
price: "700"
store: "TechWorld"
location: "New York"
purchase_date: "2025-01-10"
]
]
]
]
請注意，在形態分析後，僅顯示包含來自第一個找到節點的數據的最小表示。
這是將生成的 JSON 文件：
JSON
{
"customers": [
{
"name": "John Smith",
"email": "[email protected]",
"loyalty_status": "Gold",
"age": "34",
"gender": "Male",
"membership_id": "123456",
"purchases": [
{
"product": "Smartphone",
"category": "Electronics",
"price": "700",
"store": "TechWorld",
"location": "New York",
"purchase_date": "2025-01-10"
},
{
"product": "Wireless Earbuds",
"category": "Audio",
"price": "150",
"store": "GadgetStore",
"location": "New York",
"purchase_date": "2025-01-11"
}
]
},
{
"name": "Jane Doe",
"email": "[email protected]",
"loyalty_status": "Silver",
"age": "28",
"gender": "Female",
"membership_id": "654321",
"purchases": [
{
"product": "Laptop",
"category": "Electronics",
"price": "1200",
"store": "GadgetStore",
"location": "San Francisco",
"purchase_date": "2025-01-12"
},
{
"product": "USB-C Adapter",
"category": "Accessories",
"price": "30",
"store": "TechWorld",
"location": "San Francisco",
"purchase_date": "2025-01-13"
},
{
"product": "Keyboard",
"category": "Accessories",
"price": "80",
"store": "OfficeMart",
"location": "San Francisco",
"purchase_date": "2025-01-14"
}
]
},
{
"name": "Michael Johnson",
"email": "[email protected]",
"loyalty_status": "Bronze",
"age": "40",
"gender": "Male",
"membership_id": "789012",
"purchases": [
{
"product": "Headphones",
"category": "Audio",
"price": "150",
"store": "AudioZone",
"location": "Chicago",
"purchase_date": "2025-01-05"
}
]
},
{
"name": "Emily Davis",
"email": "[email protected]",
"loyalty_status": "Gold",
"age": "25",
"gender": "Female",
"membership_id": "234567",
"purchases": [
{
"product": "Running Shoes",
"category": "Sportswear",
"price": "120",
"store": "FitShop",
"location": "Los Angeles",
"purchase_date": "2025-01-08"
},
{
"product": "Yoga Mat",
"category": "Sportswear",
"price": "40",
"store": "FitShop",
"location": "Los Angeles",
"purchase_date": "2025-01-09"
}
]
},
{
"name": "Robert Brown",
"email": "[email protected]",
"loyalty_status": "Silver",
"age": "37",
"gender": "Male",
"membership_id": "345678",
"purchases": [
{
"product": "Smartwatch",
"category": "Wearable",
"price": "250",
"store": "GadgetPlanet",
"location": "Boston",
"purchase_date": "2025-01-07"
},
{
"product": "Fitness Band",
"category": "Wearable",
"price": "100",
"store": "HealthMart",
"location": "Boston",
"purchase_date": "2025-01-08"
}
]
}
]
}
配置到 MongoDB 的連接
由於 MongoDB 不支持直接的 HTTP 數據插入，將需要一個中介服務。
讓我們安裝相依項目：pip install flask pymongo。
服務本身：
Python
from flask import Flask, request, jsonify
from pymongo import MongoClient
import json
app = Flask(__name__)
# Connection to MongoDB
client = MongoClient('mongodb://localhost:27017')
db = client['testDB']
collection = db['testCollection']
.route('/insert', methods=['POST'])
def insert_document():
try:
# Flask will automatically parse JSON if Content-Type: application/json
data = request.get_json()
if not data:
return jsonify({"error": "Empty JSON payload"}), 400
result = collection.insert_one(data)
return jsonify({"insertedId": str(result.inserted_id)}), 200
except Exception as e:
import traceback
print(traceback.format_exc())
return jsonify({"error": str(e)}), 500
if __name__ == '__main__':
app.run(port=3000)
我們將在config.txt文件中設置 MongoDB 連接設定（參見nosql-url）：
Plain Text
job-number: 1
root-xml-folder: "D:/data/data-samples"
xml-filling-stat: false ; table: filling_percent_stat should exists
ignore-namespaces: false
ignore-tag-attributes: false
use-same-morphology-for-same-file-name-pattern: false
skip-schema-version-tag: true
use-same-morphology-for-all-files-in-folder: false
delete-data-before-insert: none
connect-to-db-at-project-opening: true
source-database: "SQLite" ; available values: PostgreSQL/SQLite
target-database: "SQLite" ; available values: PostgreSQL/SQLite/NoSQL
bot-chatID: ""
bot-token: ""
telegram-notifications: true
db-driver: ""
db-server: "127.0.0.1"
db-port: ""
db-name: ""
db-user: ""
db-pass: ""
sqlite-driver-name: "SQLite3 ODBC Driver"
sqlite-db-path: ""
nosql-url: "http://127.0.0.1:3000/insert"
append-subsection-name-to-nosql-url: false
no-sql-login: "" ; login and pass are empty
no-sql-pass: ""
請記住，如果資料庫和集合不存在，MongoDB 將自動創建具有相同名稱的資料庫和集合。但是，這種行為可能會導致錯誤，建議默認禁用。
讓我們運行服務本身：
Python
python .\app.py
接著，點擊解析，然後發送 JSON 至 NoSQL。
現在以任何方便的方式連接到 MongoDB 控制台並執行以下命令：
Plain Text
show databases
admin 40.00 KiB
config 72.00 KiB
local 72.00 KiB
testDB 72.00 KiB
use testDB
switched to db testDB
db.testCollection.find().pretty()
結果應該如下所示：
JSON
{
_id: ObjectId('278e1b2c7c1823d4fde120ef'),
customers: [
{
name: 'John Smith',
email: '[email protected]',
loyalty_status: 'Gold',
age: '34',
gender: 'Male',
membership_id: '123456',
purchases: [
{
product: 'Smartphone',
category: 'Electronics',
price: '700',
store: 'TechWorld',
location: 'New York',
purchase_date: '2025-01-10'
},
{
product: 'Wireless Earbuds',
category: 'Audio',
price: '150',
store: 'GadgetStore',
location: 'New York',
purchase_date: '2025-01-11'
}
]
},
{
name: 'Jane Doe',
email: '[email protected]',
loyalty_status: 'Silver',
age: '28',
gender: 'Female',
membership_id: '654321',
purchases: [
{
product: 'Laptop',
category: 'Electronics',
price: '1200',
store: 'GadgetStore',
location: 'San Francisco',
purchase_date: '2025-01-12'
},
{
product: 'USB-C Adapter',
category: 'Accessories',
price: '30',
store: 'TechWorld',
location: 'San Francisco',
purchase_date: '2025-01-13'
},
{
product: 'Keyboard',
category: 'Accessories',
price: '80',
store: 'OfficeMart',
location: 'San Francisco',
purchase_date: '2025-01-14'
}
]
},
{
name: 'Michael Johnson',
email: '[email protected]',
loyalty_status: 'Bronze',
age: '40',
gender: 'Male',
membership_id: '789012',
purchases: [
{
product: 'Headphones',
category: 'Audio',
price: '150',
store: 'AudioZone',
location: 'Chicago',
purchase_date: '2025-01-05'
}
]
},
{
name: 'Emily Davis',
email: '[email protected]',
loyalty_status: 'Gold',
age: '25',
gender: 'Female',
membership_id: '234567',
purchases: [
{
product: 'Running Shoes',
category: 'Sportswear',
price: '120',
store: 'FitShop',
location: 'Los Angeles',
purchase_date: '2025-01-08'
},
{
product: 'Yoga Mat',
category: 'Sportswear',
price: '40',
store: 'FitShop',
location: 'Los Angeles',
purchase_date: '2025-01-09'
}
]
},
{
name: 'Robert Brown',
email: '[email protected]',
loyalty_status: 'Silver',
age: '37',
gender: 'Male',
membership_id: '345678',
purchases: [
{
product: 'Smartwatch',
category: 'Wearable',
price: '250',
store: 'GadgetPlanet',
location: 'Boston',
purchase_date: '2025-01-07'
},
{
product: 'Fitness Band',
category: 'Wearable',
price: '100',
store: 'HealthMart',
location: 'Boston',
purchase_date: '2025-01-08'
}
]
}
]
}
結論
在這個例子中，我們看到了如何在不需要編寫任何代碼的情況下自動將 XML 檔案上傳到 MongoDB。 儘管這個例子只考慮了一個檔案，在一個項目的框架內，可以處理大量類型和子類型的具有不同結構的檔案，並進行相當複雜的操作，例如 類型轉換 和使用外部服務即時處理字段值。 這不僅允許從 XML 卸載數據，還可以通過外部 API 處理一些值，包括使用大型語言模型。