将CSV文件上传到MySQL服务器

Posted 2023-01-17

我正在使用PowerShell脚本将CSV文件上传到我的数据库。 最终目标是将最新的CSV文件下载到我们的FTP（已完成）中，然后将该CSV文件上传到我们的数据库中。 这些CSV文件始终使用相同的格式，并且我创建了数据库以匹配此格式。

我使用在网上找到的脚本来帮助我，但它似乎仍然无法正常工作。 下面是脚本，我希望有人可以帮助我找出实现此目标的更好方法，或者我做错了什么。

# Database variables
$sqlserver = "****"
$database = "****"
$table = "****"
$user = "****"
$pass = "****!"

# CSV variables
$csvfile = "C:\Users\Lucy\Documents\FTPFiles\vc_report_20171211.csv"
$csvdelimiter = ","
$FirstRowColumnNames = $true

################### No need to modify anything below ###################
Write-Host "Script started..."
$elapsed = [System.Diagnostics.Stopwatch]::StartNew()
[void][Reflection.Assembly]::LoadWithPartialName("System.Data")
[void][Reflection.Assembly]::LoadWithPartialName("System.Data.SqlClient")

# 50k worked fastest and kept memory usage to a minimum
$batchsize = 50000

# Build the sqlbulkcopy connection, and set the timeout to infinite
$connectionstring = "Data Source=$sqlserver;User Id=$user;Password=$pass;Initial Catalog=$database;"
$bulkcopy = New-Object Data.SqlClient.SqlBulkCopy($connectionstring, [System.Data.SqlClient.SqlBulkCopyOptions]::TableLock)
$bulkcopy.DestinationTableName = $table
$bulkcopy.bulkcopyTimeout = 0
$bulkcopy.batchsize = $batchsize

# Create the datatable, and autogenerate the columns.
$datatable = New-Object System.Data.DataTable

# Open the text file from disk
$reader = New-Object System.IO.StreamReader($csvfile)
$columns = (Get-Content $csvfile -First 1).Split($csvdelimiter)
if ($FirstRowColumnNames -eq $true)  $null = $reader.readLine() 

foreach ($column in $columns) 
    $null = $datatable.Columns.Add()


# Read in the data, line by line, not column by column
while (($line = $reader.ReadLine()) -ne $null) 
    $null = $datatable.Rows.Add($line.Split($csvdelimiter))

    # Import and empty the datatable before it starts taking up too much RAM, but
    # after it has enough rows to make the import efficient.
    $i++;
    if (($i % $batchsize) -eq 0) 
        $bulkcopy.WriteToServer($datatable)
        Write-Host "$i rows have been inserted in $($elapsed.Elapsed.ToString())."
        $datatable.Clear()
    


# Add in all the remaining rows since the last clear
if ($datatable.Rows.Count -gt 0) 
    $bulkcopy.WriteToServer($datatable)
    $datatable.Clear()


# Clean Up
$reader.Close(); $reader.Dispose()
$bulkcopy.Close(); $bulkcopy.Dispose()
$datatable.Dispose()

Write-Host "Script complete. $i rows have been inserted into the database."
Write-Host "Total Elapsed Time: $($elapsed.Elapsed.ToString())"
$i = 0;
# Sometimes the Garbage Collector takes too long to clear the huge datatable.
[System.GC]::Collect()

我们的数据库由SiteGround托管，因此我需要连接到数据库。 起初，我得到以下错误：

Exception calling "WriteToServer" with "1" argument(s): "A network-related or
instance-specific error occurred while establishing a connection to SQL Server.
The server was not found or was not accessible. Verify that the instance name is
correct and that SQL Server is configured to allow remote connections. (provider:
Named Pipes Provider, error: 40 - Could not open a connection to SQL Server)"
At C:\Users\Lucy\Documents\FTPFiles\upload.ps1:61 char:5
+     $bulkcopy.WriteToServer($datatable)
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
    + FullyQualifiedErrorId : SqlException

然后，将端口3306添加到$sqlserver = "servername,3306"的末尾。 当我做的时候，我新得到了错误：

Exception calling "WriteToServer" with "1" argument(s): "A network-related or
instance-specific error occurred while establishing a connection to SQL Server.
The server was not found or was not accessible. Verify that the instance name is
correct and that SQL Server is configured to allow remote connections. (provider:
Named Pipes Provider, error: 40 - Could not open a connection to SQL Server)"
At C:\Users\Lucy\Documents\FTPFiles\upload.ps1:63 char:5
+     $bulkcopy.WriteToServer($datatable)
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
    + FullyQualifiedErrorId : SqlException