空格在 fread 中未被识别为 NA

Posted 2023-02-18

【中文标题】空格在 fread 中未被识别为 NA

【英文标题】：Blank space not recognised as NA in fread

【发布时间】：2018-12-03 18:25:17

【问题描述】：

我有一个必须在 R 中导入的大文件。为此我使用了fread。 fread 将数字字段中的空格识别为 NA，但未将字符和 integer64 字段中的空格识别为 NA。

fread 将空格识别为字符字段的空单元格，并将空格识别为整数 64 字段的 0。

当我使用 read.table 导入相同的数据时，它会将所有空格识别为 NA。

请找一个可重现的例子，

library(data.table)
x1 <- c("","","")
x2 <- c("1006678566","","1011160152")
x3 <- c("","ac","")
x4 <- c("","2","3")
df <- cbind.data.frame(x1,x2,x3,x4)
write.csv(df,"tr.csv")

tr1 <- fread("tr.csv", header=T, fill = T,
             sep= ",", na.strings = c("",NA), data.table = F,
             stringsAsFactors = FALSE)

tr2 <- read.table("tr.csv", fill = TRUE, header=T, 
                  sep= ",", na.strings = c(""," ", NA), 
                  stringsAsFactors = FALSE)

详细输出：

Input contains no \n. Taking this to be a filename to open
[01] Check arguments
  Using 4 threads (omp_get_max_threads()=4, nth=4)
  NAstrings = [<<>>, <<NA>>]
  None of the NAstrings look like numbers.
  show progress = 1
  0/1 column will be read as integer
[02] Opening the file
  Opening file tr.csv
  File opened, size = 409 bytes.
  Memory mapped ok
[03] Detect and skip BOM
[04] Arrange mmap to be \0 terminated
  \n has been found in the input and different lines can end with different line endings (e.g. mixed \n and \r\n in one file). This is common and ideal.
[05] Skipping initial rows if needed
  Positioned on line 1 starting: <<"","x1","x2","x3","x4","x5","x>>
[06] Detect separator, quoting rule, and ncolumns
  Using supplied sep ','
  sep=','  with 7 fields using quote rule 0
  Detected 7 columns on line 1. This line is either column names or first data row. Line starts as: <<"","x1","x2","x3","x4","x5","x>>
  Quote rule picked = 0
  fill=true and the most number of columns found is 7
[07] Detect column types, good nrow estimate and whether first row is column names
  'header' changed by user from 'auto' to true
  Number of sampling jump points = 1 because (407 bytes from row 1 to eof) / (2 * 407 jump0size) == 0
  Type codes (jump 000)    : 56A255A  Quote rule 0
  All rows were sampled since file is small so we know nrow=16 exactly
[08] Assign column names
[09] Apply user overrides on column types
  After 0 type and 0 drop user overrides : 56A255A
[10] Allocate memory for the datatable
  Allocating 7 column slots (7 - 0 dropped) with 16 rows
[11] Read the data
  jumps=[0..1), chunk_size=1048576, total_size=373
Read 16 rows x 7 columns from 409 bytes file in 00:00.042 wall clock time
[12] Finalizing the datatable
  Type counts:
         1 : bool8     '2'
         3 : int32     '5'
         1 : int64     '6'
         2 : string    'A'
=============================
   0.009s ( 22%) Memory map 0.000GB file
   0.029s ( 68%) sep=',' ncol=7 and header detection
   0.002s (  5%) Column type detection using 16 sample rows
   0.001s (  2%) Allocation of 16 rows x 7 cols (0.000GB) of which 16 (100%) rows used
   0.001s (  2%) Reading 1 chunks (0 swept) of 1.000MB (each chunk 16 rows) using 1 threads
   +    0.000s (  0%) Parse to row-major thread buffers (grown 0 times)
   +    0.000s (  0%) Transpose
   +    0.001s (  2%) Waiting
   0.000s (  0%) Rereading 0 columns due to out-of-sample type exceptions
   0.042s        Total

请帮我解决这个问题。

谢谢！

【问题讨论】：

您在运行时根据需要在数据表中给出结果的示例

我猜这是在 data.table 版本 1.11.0 中引入的（请参阅github.com/Rdatatable/data.table/blob/master/NEWS.md，第 1 节）引用：“我们希望 ,,=>NA 与数字类型保持一致，并且” ",=>空字符串作为 fwrite/fread 字符列的标准默认值，因此 fread(fwrite(DT))==DT 不需要对任何参数进行任何更改。"

@docendodiscimus 是的，还有其他一些与此相关的未决问题，例如github.com/Rdatatable/data.table/issues/2857 和 github.com/Rdatatable/data.table/issues/2714

我遇到了同样的问题。作为一种解决方法，我使用 fread 来提高其导入速度，然后使用以下代码将空白字段替换为 NA： lapply(tr1, function(x) ifelse(x == "", NA, x))

不幸的是，目前 fread 似乎还没有解决此问题的方法。在 1 行中我能想到的最快方法是 sapply(fread("tr.csv", quote = ""), function(x) gsub("\"", NA, x))。唱歌时超过 1 行：dt

【参考方案1】：

如果你想在读取文件后避免额外的操作，你可以尝试使用

quote = FALSE

写入 csv 时。这可以防止在值周围使用引号" "，并且所有缺失值现在应读取为NAs。它应该看起来像这样 -

# also turned off row names to prevent an additional column when reading the file.
write.csv(df, "tr.csv", quote = FALSE, row.names = FALSE) 

输出 -

tr1 <- fread("tr.csv", header=T, fill = T,
             sep= ",", na.strings = c("",NA), data.table = F,
             stringsAsFactors = FALSE)
tr1

 x1         x2   x3 x4
1 NA 1006678566 <NA> NA
2 NA         NA   ac  2
3 NA 1011160152 <NA>  3

tr2 <- read.table("tr.csv", fill = TRUE, header=T, 
                  sep= ",", na.strings = c(""," ", NA), 
                  stringsAsFactors = FALSE)
tr2

  x1         x2   x3 x4
1 NA 1006678566 <NA> NA
2 NA         NA   ac  2
3 NA 1011160152 <NA>  3

【讨论】：

效果很好，应该是正确的答案。作为一个附带问题，使用fill=TRUE 是否有任何不利（风险）？

@hunter_git 不是真的。它用 NA 填充行（如果有）的缺失列，而不是引发Stopped early 警告。因此，除非您确定行的长度，否则应将fill 保留为FALSE，以便在行长度存在任何差异时发出警告。

【参考方案2】：

我发现的一件事是我们执行 write.csv() 时保存数据的方式。

打开 csv 文件并点击删除 X4 中的空白单元格并保存。如果你现在导入它，NA 会出现在 R 中。

检查：

apply(tr1, 2, function(x) length(which(is.na(x))))

V1 x1 x2 x3 x4

0 3 1 2 1

如果有一个带有空格的 csv 文件，我们会使用 fread

na.strings("", NA)

对于空格，字符数据类型也显示为“NA”。

【讨论】：

@ Deb: na.strings = c("", NA) 适用于 fread() 并且如果 colSums(is.na(DT)) 计算了 s，这将是正确的解决方案作为NA。但事实并非如此。

【参考方案3】：

@SJB 在fread() 中使用na.strings = c(NA_character_, "") 作为参数，空格/单元格将被读取为NA。

对于各种数据类型，存在多种形式的 NA。见help(NA)： NA_character_ NA_real_ NA_integer_等