php/sql ...我应该如何使用 50mb 文件中的数据？

Posted 2023-02-24

【中文标题】php/sql ...我应该如何使用 50mb 文件中的数据？

【英文标题】：php/sql ...how am i supposed to use data from a 50mb file?

【发布时间】：2013-12-27 10:50:22

【问题描述】：

到目前为止，我的代码如下。我在正确的轨道上吗？我能够读取 1mb 的文件，但 50mb 变得疯狂而缓慢。也许我不应该尝试输出整个数据？但我至少读对了吗？以下是适用于测试 1mb 文件但不适用于 pubs.txt 50mb 文件的指南和我的代码。

Write a php program that reads the posted pubs.txt file, parses it, and inserts the 
data into mysql tables. 

pubs.txt contains many publications. Each <pub> … </pub> pair specifies a 
publication, with ID, title, year, journal (<booktitle>), pages, and authors 
information. Some information may be missing. It is your own choice to use a 
default value or NULL for missing fields. Some information looks incorrect but 
you do not have to worry about it. The data in pubs.txt was automatically 
extracted from web resources by computer. 

You have the freedom to design the MySql database, as long as you can answer 
the queries correctly and hopefully efficiently. 
It is your own choice to execute this program from command line or web browser. 
3. Use PHP to write a web interface, which should provide intuitive forms allowing 
users to:

• Insert a publication into the database
• Query all publications by a particular author
• Query all publications in a particular year 

下面是我的 php 代码。感谢您的指导。

<?php



$mysqli = new mysqli('localhost', 'root', '', 'db1');

if (mysqli_connect_errno())
printf("connect failed\n", mysqli_connect_error());
exit();    



error_reporting(E_ALL);
$header = '<?xml version="1.0" encoding="UTF-8"?>'."\n<datalist>";
$content = $header."\n".file_get_contents("pubs.txt")."\n</datalist>";
$ob = simplexml_load_string($content);
$json = json_encode($ob);    
$array = json_decode($json, true);
$alldata = $array["pub"];




foreach ($alldata as $key => $value)  //access all data in loop
$id = $value["ID"];
$title = $value["title"];
$year = $value["year"];
$booktitle = $value["booktitle"];
$pages = $value["pages"];
$authors = implode(",", $value["authors"]["author"]);


$stmt = $mysqli->prepare("INSERT INTO pubs VALUES (?, ?, ?, ?, ?, ?)");
$stmt->bind_param('ssssss',
               $value["ID"],
               $value["title"],
               $value["year"],
               $value["booktitle"],
               $value["pages"], implode(",", $value["authors"]["author"]));
$stmt->execute();
printf("%d row insrt\n", $stmt->affected_rows);

echo "<table>
<tr>
<th>ID</th>
<th>title</th>
<th>year</th>
<th>booktitle</th>
<th>pages</th>
<th>authors</th>
</tr>";
echo "<tr>";
echo "<td>" . $value['ID'] . "</td>";
echo "<td>" . $value['title'] .  "</td>";
echo "<td>" . $value['year'] .  "</td>";
echo "<td>" . $value['booktitle'] .  "</td>";
echo "<td>" . $value['pages'] . "</td>";
echo "<td>" . $value['authors'] . "</td>";
echo "</tr>";





echo "</table>";
?>

【问题讨论】：

将 50MB 加载到 XML dom 会很慢并且会消耗内存。但是作业中没有任何内容表明您需要快速或有效地加载。所以不要出汗，集中精力让数据库和查询正确。如果您真的担心它，请使用基于“XML 解析器”事件的解析器，但请注意，它的工作方式与简单的 XML 解析器完全不同。

您认为在标准戴尔笔记本电脑上加载此文件需要多长时间？

50M不算大，可以轻松放入内存

【参考方案1】：

对于这么大的文档，您应该使用a progressive XML parser that doesn't depend on loading and parsing it all at once。

【讨论】：

值得注意的是，在上面的链接中，用户在页面底部提供的示例 xml2array(...) 将无法正确处理多维对象。

值得注意的是，大多数用户在 php.net 上的贡献都很糟糕，但我主要指的是左侧边栏中的 3 个示例。

我完全同意，尼尔斯。只是想让 OP 意识到： xml2array 听起来好得令人难以置信。同样的解决方案也出现在：***.com/questions/6167279/…

就像他说的那样，它不起作用，因此它好得令人难以置信。如果你真的想的话，也不能阻止你使用有缺陷的代码。