使用Powershell将大输出从Oracle导出到CSV

Question

我每周需要从Oracle导出一个很大的CSV文件。

我尝试了两种方法。

Adapter.fill（数据集）
循环浏览列和行以一次一行保存到CSV文件中。

第一个在运行时内存不足（服务器计算机只有4 GB RAM），第二个大约要花一个小时，因为要导出的行超过400万。

这里是代码1：

#Your query. It cannot contain any double quotes otherwise it will break.
$query = "SELECT manycolumns FROM somequery"

#Oracle login credentials and other variables
$username = "username"
$password = "password"
$datasource = "database address"
$output = "\\NetworkLocation\Sales.csv"

#creates a blank CSV file and make sure it's in ASCI
Out-File $output -Force ascii

#This here will look for "Oracle.ManagedDataAccess.dll" file inside "C:\Oracle" folder. We usually have two versions of Oracle installed so the adaptor can be in different locations. Needs changing if the Oracle is installed elsewhere.
$location = Get-ChildItem -Path C:\Oracle -Filter Oracle.ManagedDataAccess.dll -Recurse -ErrorAction SilentlyContinue -Force

#Establishes connection to Oracle using the DLL file
Add-Type -Path $location.FullName
$connectionString = 'User Id=' + $username + ';Password=' + $password + ';Data Source=' + $datasource
$connection = New-Object Oracle.ManagedDataAccess.Client.OracleConnection($connectionString)
$connection.open()
$command=$connection.CreateCommand()
$command.CommandText=$query

#Creates a table in memory and fills it with results from the query. Then, export the virtual table into CSV.
$DataSet = New-Object System.Data.DataSet
$Adapter = New-Object Oracle.ManagedDataAccess.Client.OracleDataAdapter($command)
$Adapter.Fill($DataSet)
$DataSet.Tables[0] | Export-Csv $output -NoTypeInformation

$connection.Close()

这里是＃2

#Your query. It cannot contain any double quotes otherwise it will break.
$query = "SELECT manycolumns FROM somequery"

#Oracle login credentials and other variables
$username = "username"
$password = "password"
$datasource = "database address"
$output = "\\NetworkLocation\Sales.csv"
$tempfile = $env:TEMP + "\Temp.csv"

#creates a blank CSV file and make sure it's in ASCI
Out-File $tempfile -Force ascii

#This here will look for "Oracle.ManagedDataAccess.dll" file inside "C:\Oracle" folder. Needs changing if the Oracle is installed elsewhere.
$location = Get-ChildItem -Path C:\Oracle -Filter Oracle.ManagedDataAccess.dll -Recurse -ErrorAction SilentlyContinue -Force

#Establishes connection to Oracle using the DLL file
Add-Type -Path $location.FullName
$connectionString = 'User Id=' + $username + ';Password=' + $password + ';Data Source=' + $datasource
$connection = New-Object Oracle.ManagedDataAccess.Client.OracleConnection($connectionString)
$connection.open()
$command=$connection.CreateCommand()
$command.CommandText=$query

#Reads results column by column. This way you don't have to specify how many columns it has.
$reader = $command.ExecuteReader()
  while($reader.Read()) 
       $props = @
       for($i = 0; $i -lt $reader.FieldCount; $i+=1) 
           $name = $reader.GetName($i)
           $value = $reader.item($i)
           $props.Add($name, $value)   
       
       #Exports each line to CSV file. Works best when the file is on local drive as it saves it after each line.
       new-object PSObject -Property $props | Export-Csv $tempfile -NoTypeInformation -Append
  

Move-Item $tempfile $output -Force

$connection.Close()

理想情况下，我想使用第一个代码，因为它比第二个要快，但要避免内存不足。

你们是否知道有某种方法可以“填充”前1百万条记录，将它们附加到CSV，清理“ DataSet”表，再清除1百万条，等等？代码运行完CSV权重后，大约需要1.3 GB，但是在运行时，即使8 GB的内存也不够用（我的笔记本电脑只有8 GB，但是服务器只有4 GB，这确实让它很难受）。

任何技巧都将不胜感激。

Answer 1

另一答案