需要帮助上传 csv

Posted 2023-02-24

【中文标题】需要帮助上传 csv

【英文标题】：Need a help for upload csv

【发布时间】：2013-07-06 23:07:07

【问题描述】：

我有一个 csv 文件，其中 9 个缺少数据（csv 文件的大小为 229mb）。我将 csv 文件拆分为更小的，即一个 csv 包含 4000 或 5000 个数据。我上传 csv 文件并从中读取数据并插入它进入数据库。1st我将数据插入到一个虚拟表中并列出它。然后提交列表并插入到主表中。但这需要1个多小时。我的代码是，

$path = "csv1.csv";
if (($handle = fopen("$path", "r")) !== FALSE) 
$cntcnt=1;
   while (($info = fgetcsv($handle, 10000, ",")) !== FALSE) 
      foreach($info as $key=>$value) 
      $datas[$key]=$value;
                             if(isset($datas[3]))       $bname      = $datas[3];        else $bname     = "";
                            if(isset($datas[4]))        $address    = $datas[4];        else $address       = "";
                            if(isset($datas[5]))        $city       = $datas[5];        else $city          = "";
                            if(isset($datas[7]))        $zipcode    = $datas[7];        else $zipcode       = "";
                            if(isset($datas[8]))        $cname      = $datas[8];        else $cname         = "";
                            if(isset($datas[13]))       $phone      = $datas[13];       else $phone             = "";
                            if(isset($datas[14]))       $fax        = $datas[14];       else $fax           = "";
                            if(isset($datas[15]))       $email      = $datas[15];       else $email         = "";
                            if(isset($datas[16]))       $website    = $datas[16];       else $website           = "";
                            $bname= mysql_real_escape_string($bname); 
                            $cname= mysql_real_escape_string($cname);
                            $address =mysql_real_escape_string($address);
                            $city =mysql_real_escape_string($city);
$data=array('vendorid'=>$vendorid,'cid'=>$cid,'scid'=>$scid,'sscid'=>$sscid,'rid'=>$rid,'pid'=>$pid,'rcid'=>$cityid,'lid'=>$lid,'bname'=>$bname,'address'=>$address,'street'=>$street,'zipcode'=>$zipcode,'phone'=>$phone,'cell'=>$cell,'fax'=>$fax,'email'=>$email,'website'=>$website,'description'=>$description,'mapaddress'=>$mapaddress,'price'=>$price,'date'=>$date,'ref_code'=>$ref_code,'realestate_category'=>$realestate_category,'template'=>$template,'busname'=>$busname,'code'=>$code,'b_type'=>$b_type,'edit_date'=>$edit_date,'phone2'=>$phone2,'cname'=>$cname,'ssscid'=>$ssscid);
$this->db->insert('c_tbldummy',$data);
        
        

有什么方法可以加快这个过程吗？因为我的客户要上传50000到75000的数据。请大家帮帮我。

【参考方案1】：

使用 LOAD DATA LOCAL INFILE 可以更快地上传并为您的表添加索引

这里是 LOAD DATA INFILE 示例

代码：

$query = "LOAD DATA LOCAL INFILE '$path_of_csv_file' INTO TABLE $table_name FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n' IGNORE 1 LINES ($column_of_table);

您可以为不在 csv 中的列添加“设置”

代码：

$query = "LOAD DATA LOCAL INFILE '$path_of_csv_file' INTO TABLE $table_name FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n' IGNORE 1 LINES ($column_of_table) set column1 = '$value1', column2= '$value2' ";

【参考方案2】：

使用 batch_insert 应该会减少你的时间...

$this->db->insert_batch();

在这里，您正在将每一行的数据插入表中...因此，由于您有 75000 + 数据..这意味着 75000+ 插入查询..我想这需要时间....这可能不会明显加快但我相信这会减少插入所需的时间..

批量插入会将查询减少到一个..并将所有数据插入一个查询中..

你可以试试看……

 while (($info = fgetcsv($handle, 10000, ",")) !== FALSE) 
  foreach($info as $key=>$value) 
  $datas[$key]=$value;
                         if(isset($datas[3]))       $bname      = $datas[3];        else $bname     = "";
                        if(isset($datas[4]))        $address    = $datas[4];        else $address       = "";
                        if(isset($datas[5]))        $city       = $datas[5];        else $city          = "";
                        if(isset($datas[7]))        $zipcode    = $datas[7];        else $zipcode       = "";
                        if(isset($datas[8]))        $cname      = $datas[8];        else $cname         = "";
                        if(isset($datas[13]))       $phone      = $datas[13];       else $phone             = "";
                        if(isset($datas[14]))       $fax        = $datas[14];       else $fax           = "";
                        if(isset($datas[15]))       $email      = $datas[15];       else $email         = "";
                        if(isset($datas[16]))       $website    = $datas[16];       else $website           = "";
                        $bname= mysql_real_escape_string($bname); 
                        $cname= mysql_real_escape_string($cname);
                        $address =mysql_real_escape_string($address);
                        $city =mysql_real_escape_string($city);
               $data[]=array('vendorid'=>$vendorid,'cid'=>$cid,'scid'=>$scid,'sscid'=>$sscid,'rid'=>$rid,'pid'=>$pid,'rcid'=>$cityid,'lid'=>$lid,'bname'=>$bname,'address'=>$address,'street'=>$street,'zipcode'=>$zipcode,'phone'=>$phone,'cell'=>$cell,'fax'=>$fax,'email'=>$email,'website'=>$website,'description'=>$description,'mapaddress'=>$mapaddress,'price'=>$price,'date'=>$date,'ref_code'=>$ref_code,'realestate_category'=>$realestate_category,'template'=>$template,'busname'=>$busname,'code'=>$code,'b_type'=>$b_type,'edit_date'=>$edit_date,'phone2'=>$phone2,'cname'=>$cname,'ssscid'=>$ssscid);

    
    

$this->db->insert_batch('c_tbldummy',$data);

CI doc 用于 insert_batch()