CRAN R 包的 conda 构建失败

Posted

技术标签:

【中文标题】CRAN R 包的 conda 构建失败【英文标题】:conda build fails for CRAN R packages 【发布时间】:2018-03-30 10:49:37 【问题描述】:

我正在尝试使用此处显示的方法从 CRAN 包构建 conda-forge 上不存在的 R conda 包:https://www.anaconda.com/blog/developer-blog/conda-data-science/

我以 R 包 rann 为例 - 这个包在 conda-forge 上已经作为 r-rann 存在,但我用它来测试构建过程是否正常工作。

在构建过程中,执行包含$R CMD INSTALL --build . 行的脚本“conda_build.sh”。我的问题是构建过程失败并出现错误Fatal error: cannot open file 'CMD': No such file or directory,与我尝试构建的包无关。我不知道环境变量(?)“CMD”应该来自哪里。

我采取了以下步骤:

    conda skeleton cran rann conda build r-rann

我还测试了从 pip 构建 Python 包,这些包没有任何问题,因此这似乎不是我的 Miniconda 安装的普遍问题。

我的系统:

Ubuntu 17.10 Miniconda 3,版本 4.4.10

这是构建过程的完整日志:

eega@eega-nb:~/workspace$ conda build r-rann/
Adding in variants from internal_defaults
INFO:conda_build.variants:Adding in variants from internal_defaults
Attempting to finalize metadata for r-rann
INFO:conda_build.metadata:Attempting to finalize metadata for r-rann
Solving environment: ...working... done


==> WARNING: A newer version of conda exists. <==
  current version: 4.4.10
  latest version: 4.5.0

Please update conda by running

    $ conda update -n base conda


Solving environment: ...working... done


==> WARNING: A newer version of conda exists. <==
  current version: 4.4.10
  latest version: 4.5.0

Please update conda by running

    $ conda update -n base conda


Solving environment: ...working... done


==> WARNING: A newer version of conda exists. <==
  current version: 4.4.10
  latest version: 4.5.0

Please update conda by running

    $ conda update -n base conda


BUILD START: ['r-rann-2.5.1-r341h599a50d_0.tar.bz2']
Solving environment: ...working... done


==> WARNING: A newer version of conda exists. <==
  current version: 4.4.10
  latest version: 4.5.0

Please update conda by running

    $ conda update -n base conda



## Package Plan ##

  environment location: /tmp/conda-builds/r-rann_1522400915206/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold


The following NEW packages will be INSTALLED:

    bzip2:           1.0.6-1          conda-forge
    ca-certificates: 2018.1.18-0      conda-forge
    cairo:           1.14.10-0        conda-forge
    curl:            7.59.0-0         conda-forge
    fontconfig:      2.12.6-0         conda-forge
    freetype:        2.8.1-0          conda-forge
    gettext:         0.19.8.1-0       conda-forge
    glib:            2.55.0-0         conda-forge
    graphite2:       1.3.11-0         conda-forge
    gsl:             2.1-2            conda-forge
    harfbuzz:        1.7.6-0          conda-forge
    icu:             58.2-0           conda-forge
    jpeg:            9b-2             conda-forge
    krb5:            1.14.2-0         conda-forge
    libffi:          3.2.1-3          conda-forge
    libgcc:          7.2.0-h69d50b8_2            
    libgcc-ng:       7.2.0-hdf63c60_3            
    libiconv:        1.15-0           conda-forge
    libpng:          1.6.34-0         conda-forge
    libssh2:         1.8.0-2          conda-forge
    libstdcxx-ng:    7.2.0-hdf63c60_3            
    libtiff:         4.0.9-0          conda-forge
    libxml2:         2.9.8-0          conda-forge
    ncurses:         5.9-10           conda-forge
    openssl:         1.0.2n-0         conda-forge
    pango:           1.40.14-0        conda-forge
    pcre:            8.39-0           conda-forge
    pixman:          0.34.0-1         conda-forge
    r-base:          3.4.1-3          conda-forge
    readline:        7.0-0            conda-forge
    tk:              8.6.7-0          conda-forge
    xz:              5.2.3-0          conda-forge
    zlib:            1.2.11-0         conda-forge

Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
Solving environment: ...working... done


==> WARNING: A newer version of conda exists. <==
  current version: 4.4.10
  latest version: 4.5.0

Please update conda by running

    $ conda update -n base conda


Solving environment: ...working... done


==> WARNING: A newer version of conda exists. <==
  current version: 4.4.10
  latest version: 4.5.0

Please update conda by running

    $ conda update -n base conda



## Package Plan ##

  environment location: /tmp/conda-builds/r-rann_1522400915206/_build_env


The following NEW packages will be INSTALLED:

    binutils_impl_linux-64: 2.28.1-had2808c_3            
    binutils_linux-64:      7.2.0-26                     
    gcc_impl_linux-64:      7.2.0-habb00fd_3             
    gcc_linux-64:           7.2.0-26                     
    gxx_impl_linux-64:      7.2.0-hdf63c60_3             
    gxx_linux-64:           7.2.0-26                     
    libstdcxx-ng:           7.2.0-hdf63c60_3             
    make:                   4.2.1-0           conda-forge

Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
Source cache directory is: /tmp/conda-builds/src_cache
Found source in cache: RANN_2.5.1_75277e5d8a.tar.gz
Extracting download
source tree in: /tmp/conda-builds/r-rann_1522400915206/work
INFO: activate-binutils_linux-64.sh made the following environmental changes:
+ADDR2LINE=/tmp/conda-builds/r-rann_1522400915206/_build_env/bin/x86_64-conda_cos6-linux-gnu-addr2line
+AR=/tmp/conda-builds/r-rann_1522400915206/_build_env/bin/x86_64-conda_cos6-linux-gnu-ar
+AS=/tmp/conda-builds/r-rann_1522400915206/_build_env/bin/x86_64-conda_cos6-linux-gnu-as
+CXXFILT=/tmp/conda-builds/r-rann_1522400915206/_build_env/bin/x86_64-conda_cos6-linux-gnu-c++filt
+ELFEDIT=/tmp/conda-builds/r-rann_1522400915206/_build_env/bin/x86_64-conda_cos6-linux-gnu-elfedit
+GPROF=/tmp/conda-builds/r-rann_1522400915206/_build_env/bin/x86_64-conda_cos6-linux-gnu-gprof
+HOST=x86_64-conda_cos6-linux-gnu
+LD_GOLD=/tmp/conda-builds/r-rann_1522400915206/_build_env/bin/x86_64-conda_cos6-linux-gnu-ld.gold
+LD=/tmp/conda-builds/r-rann_1522400915206/_build_env/bin/x86_64-conda_cos6-linux-gnu-ld
+NM=/tmp/conda-builds/r-rann_1522400915206/_build_env/bin/x86_64-conda_cos6-linux-gnu-nm
+OBJCOPY=/tmp/conda-builds/r-rann_1522400915206/_build_env/bin/x86_64-conda_cos6-linux-gnu-objcopy
+OBJDUMP=/tmp/conda-builds/r-rann_1522400915206/_build_env/bin/x86_64-conda_cos6-linux-gnu-objdump
+RANLIB=/tmp/conda-builds/r-rann_1522400915206/_build_env/bin/x86_64-conda_cos6-linux-gnu-ranlib
+READELF=/tmp/conda-builds/r-rann_1522400915206/_build_env/bin/x86_64-conda_cos6-linux-gnu-readelf
+SIZE=/tmp/conda-builds/r-rann_1522400915206/_build_env/bin/x86_64-conda_cos6-linux-gnu-size
+STRINGS=/tmp/conda-builds/r-rann_1522400915206/_build_env/bin/x86_64-conda_cos6-linux-gnu-strings
+STRIP=/tmp/conda-builds/r-rann_1522400915206/_build_env/bin/x86_64-conda_cos6-linux-gnu-strip
INFO: activate-gcc_linux-64.sh made the following environmental changes:
+CC=/tmp/conda-builds/r-rann_1522400915206/_build_env/bin/x86_64-conda_cos6-linux-gnu-cc
+CFLAGS=-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -pipe -I/tmp/conda-builds/r-rann_1522400915206/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/include -fdebug-prefix-map=$SRC_DIR=/usr/local/src/conda/$PKG_NAME-$PKG_VERSION -fdebug-prefix-map=$PREFIX=/usr/local/src/conda-prefix
+_CONDA_PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_x86_64_conda_cos6_linux_gnu
+CPPFLAGS=-DNDEBUG -D_FORTIFY_SOURCE=2 -O2
+CPP=/tmp/conda-builds/r-rann_1522400915206/_build_env/bin/x86_64-conda_cos6-linux-gnu-cpp
+DEBUG_CFLAGS=-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-all -fno-plt -Og -g -Wall -Wextra -fvar-tracking-assignments -pipe -I/tmp/conda-builds/r-rann_1522400915206/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/include -fdebug-prefix-map=$SRC_DIR=/usr/local/src/conda/$PKG_NAME-$PKG_VERSION -fdebug-prefix-map=$PREFIX=/usr/local/src/conda-prefix
+DEBUG_CPPFLAGS=-D_DEBUG -D_FORTIFY_SOURCE=2 -Og
+GCC_AR=/tmp/conda-builds/r-rann_1522400915206/_build_env/bin/x86_64-conda_cos6-linux-gnu-gcc-ar
+GCC_NM=/tmp/conda-builds/r-rann_1522400915206/_build_env/bin/x86_64-conda_cos6-linux-gnu-gcc-nm
+GCC_RANLIB=/tmp/conda-builds/r-rann_1522400915206/_build_env/bin/x86_64-conda_cos6-linux-gnu-gcc-ranlib
+GCC=/tmp/conda-builds/r-rann_1522400915206/_build_env/bin/x86_64-conda_cos6-linux-gnu-gcc
+LDFLAGS=-Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,-rpath,/tmp/conda-builds/r-rann_1522400915206/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/lib -L/tmp/conda-builds/r-rann_1522400915206/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/lib
INFO: activate-gxx_linux-64.sh made the following environmental changes:
+CXXFLAGS=-fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -pipe -I/tmp/conda-builds/r-rann_1522400915206/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/include -fdebug-prefix-map=$SRC_DIR=/usr/local/src/conda/$PKG_NAME-$PKG_VERSION -fdebug-prefix-map=$PREFIX=/usr/local/src/conda-prefix
+CXX=/tmp/conda-builds/r-rann_1522400915206/_build_env/bin/x86_64-conda_cos6-linux-gnu-c++
+DEBUG_CXXFLAGS=-fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-all -fno-plt -Og -g -Wall -Wextra -fvar-tracking-assignments -pipe -I/tmp/conda-builds/r-rann_1522400915206/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/include -fdebug-prefix-map=$SRC_DIR=/usr/local/src/conda/$PKG_NAME-$PKG_VERSION -fdebug-prefix-map=$PREFIX=/usr/local/src/conda-prefix
+GXX=/tmp/conda-builds/r-rann_1522400915206/_build_env/bin/x86_64-conda_cos6-linux-gnu-g++
Fatal error: cannot open file 'CMD': No such file or directory
Traceback (most recent call last):
  File "/home/eega/miniconda3/bin/conda-build", line 6, in <module>
    sys.exit(conda_build.cli.main_build.main())
  File "/home/eega/miniconda3/lib/python3.6/site-packages/conda_build/cli/main_build.py", line 420, in main
    execute(sys.argv[1:])
  File "/home/eega/miniconda3/lib/python3.6/site-packages/conda_build/cli/main_build.py", line 411, in execute
    verify=args.verify)
  File "/home/eega/miniconda3/lib/python3.6/site-packages/conda_build/api.py", line 199, in build
    notest=notest, need_source_download=need_source_download, variants=variants)
  File "/home/eega/miniconda3/lib/python3.6/site-packages/conda_build/build.py", line 2097, in build_tree
    notest=notest,
  File "/home/eega/miniconda3/lib/python3.6/site-packages/conda_build/build.py", line 1370, in build
    utils.check_call_env(cmd, env=env, cwd=src_dir, stats=build_stats)
  File "/home/eega/miniconda3/lib/python3.6/site-packages/conda_build/utils.py", line 301, in check_call_env
    return _func_defaulting_env_to_os_environ('call', *popenargs, **kwargs)
  File "/home/eega/miniconda3/lib/python3.6/site-packages/conda_build/utils.py", line 280, in _func_defaulting_env_to_os_environ
    raise subprocess.CalledProcessError(proc.returncode, _args)
subprocess.CalledProcessError: Command '['/bin/bash', '-e', '/tmp/conda-builds/r-rann_1522400915206/work/conda_build.sh']' returned non-zero exit status 2.

conda 创建的 build.sh 内容:

#!/bin/bash

if [[ $target_platform =~ linux.* ]] || [[ $target_platform == win-32 ]] || [[ $target_platform == win-64 ]] || [[ $target_platform == osx-64 ]]; then
  export DISABLE_AUTOBREW=1
  mv DESCRIPTION DESCRIPTION.old
  grep -v '^Priority: ' DESCRIPTION.old > DESCRIPTION
  $R CMD INSTALL --build .
else
  mkdir -p $PREFIX/lib/R/library/RANN
  mv * $PREFIX/lib/R/library/RANN
fi

conda 创建的 meta.yaml 的内容:

% set version = '2.5.1' %

% set posix = 'm2-' if win else '' %
% set native = 'm2w64-' if win else '' %

package:
  name: r-rann
  version:  version|replace("-", "_") 

source:
  fn: RANN_2.5.1.tar.gz
  url:
    -  cran_mirror /src/contrib/RANN_2.5.1.tar.gz
    -  cran_mirror /src/contrib/Archive/RANN/RANN_2.5.1.tar.gz
  sha256: 75277e5d8a13ca01ff387f99d403268a8077862d4e95b076b74fb1b5538a8546

build:
  merge_build_host: True  # [win]
  # If this is a new build for the same version, increment the build number.
  number: 0

  # This is required to make R link correctly on Linux.
  rpaths:
    - lib/R/lib/
    - lib/

# Suggests: testthat
requirements:
  build:
    -  compiler('c')           # [not win]
    -  compiler('cxx')         # [not win]
    - nativetoolchain          # [win]
    - posixfilesystem          # [win]
    - posixmake

  host:
    - r-base

  run:
    - r-base
    - nativegcc-libs           # [win]

test:
  commands:
    # You can put additional test commands to be run here.
    - $R -e "library('RANN')"           # [not win]
    - "\"%R%\" -e \"library('RANN')\""  # [win]

  # You can also put a file called run_test.py, run_test.sh, or run_test.bat
  # in the recipe that will be run at test time.

  # requires:
    # Put any additional test requirements here.

about:
  home: https://github.com/jefferis/RANN
  license: GPL (>= 3)
  summary: Finds the k nearest neighbours for every point in a given dataset in O(N log N) time
    using Arya and Mount's ANN library (v1.1.3). There is support for approximate as
    well as exact searches, fixed radius searches and 'bd' as well as 'kd' trees. The
    distance is computed using the L2 (Euclidean) metric. Please see package 'RANN.L1'
    for the same functionality using the L1 (Manhattan, taxicab) metric.
  license_family: GPL3

# The original CRAN metadata for this package was:

# Package: RANN
# Title: Fast Nearest Neighbour Search (Wraps ANN Library) Using L2 Metric
# Author: Sunil Arya and David Mount (for ANN), Samuel E. Kemp, Gregory Jefferis
# Maintainer: Gregory Jefferis <jefferis@gmail.com>
# Copyright: ANN library is copyright University of Maryland and Sunil Arya and David Mount. See file COPYRIGHT for details.
# Description: Finds the k nearest neighbours for every point in a given dataset in O(N log N) time using Arya and Mount's ANN library (v1.1.3). There is support for approximate as well as exact searches, fixed radius searches and 'bd' as well as 'kd' trees. The distance is computed using the L2 (Euclidean) metric. Please see package 'RANN.L1' for the same functionality using the L1 (Manhattan, taxicab) metric.
# URL: https://github.com/jefferis/RANN
# BugReports: https://github.com/jefferis/RANN/issues
# Encoding: UTF-8
# License: GPL (>= 3)
# Suggests: testthat
# Version: 2.5.1
# RoxygenNote: 6.0.1
# NeedsCompilation: yes
# Packaged: 2017-05-19 13:52:30 UTC; jefferis
# Repository: CRAN
# Date/Publication: 2017-05-21 07:56:50 UTC

# See
# http://docs.continuum.io/conda/build.html for
# more information about meta.yaml

任何帮助将不胜感激!

【问题讨论】:

你能像它建议的那样做conda update -n base conda 日志有点不同,而且肯定更长,但我得到了同样的错误。不过,我无法替换原始帖子中的日志,因为它太长了。 我无法在 macOS 上重现这个...稍后将在 Ubuntu 上尝试 【参考方案1】:

“CMD”是 R 命令行指令。在 conda-build 期间,您下载 RANN 的源代码,然后在临时环境中使用 R 从中构建一个包,然后(临时)安装它以检查它是否有效。

那行只是告诉 R 构建包rann。在开发 R 包时,您可以使用 R CMD build &lt;my_package&gt;R CMD INSTALL --build &lt;my_package&gt; 制作该包的可安装存档。

看起来您的 conda-build 找不到 R 来构建包,尽管这通常是在运行 conda-build 时设置的临时环境中完成的;并且构建过程几乎肯定会安装r-base,因此没有真正的理由不找到 R。

我已经使用 conda-build 3.7.2 和 conda 4.3.34 以及 R v3.4.1(带有详细输出)在 windows-subsystem for linux(实际上是 Ubuntu 16.04)下运行了代码

conda skeleton cran rann
conda build r-rann --R=3.4.1 --debug

它试图在我的系统上运行Rscript CMD INSTALL --build,而不是R CMD build。前者不是通常在 R 下构建包的方式,正是在这一步引发了“找不到文件 CMD”错误,所以我更详细地查看了它。

首先,我将 meta.yaml 与 conda-forge 上的进行了比较。运行要求中没有 libgcc,并且 conda 骨架设置的 yaml 中的构建要求中没有 r-base - 我添加了这些并重建(尽管我怀疑这些会导致问题)。这没有解决任何问题,“找不到 CMD”错误仍然抛出。

我查看了conda-build 的更新日志。从 3.7.2 版本开始,Rscript 用于运行测试,而不是 R。我恢复为 conda-build=3.7.1:

conda install conda-build=3.7.1 -vv
conda build r-rann --R=3.4.1 --debug

现在,我通过了对 R CMD INSTALL --build ... 的 ccall,但在编译过程中仍然出现错误;所以我还没有解决这个问题。

尽管如此,您最初发布的错误似乎是由于对 conda-build 的更改(使用 Rscript 而不是 R 测试包时),这导致构建 R 包时出现问题。您能否将问题交叉引用到 conda-build github,请。

经过一番搜索,我通过 a) 安装 gfortran 和 b) 使用 conda-build 3.6.0 创建骨架和构建包来实现此功能。 conda-build 的 3.7.0/1 抛出了与版本号相关的错误(没有 --version XXX 参数,conda-skeleton 抛出错误 re Namespace not have a version;使用 --version XXX 参数,conda-skeleton 抛出没有为 R 包实现的错误版本号)。

上面的错误已经在conda-build 3.8.0修复了,但是我还没用过

【讨论】:

非常感谢你,拉斯!我设法使用 conda-build 3.6 构建了 r-rann。您是否相信所有阻止构建 R 包的错误都将在 conda-build 3.8.0 中修复?如果没有,我将创建您建议的错误报告。 检查更新日志:这是提到的第一个问题 (#2782) - 所以如果它没有修复,你应该向他们报告 好的,已在 3.8.0 版中修复。再次感谢拉斯!

以上是关于CRAN R 包的 conda 构建失败的主要内容,如果未能解决你的问题,请参考以下文章

R CMD CHECK --as-cran在OS X Catalina上失败

使用并行或 doParallel 为多个操作系统作为 CRAN 包的自定义包

更改 conda 包的构建字符串

无法 conda 构建 R 包

R 包的 CRAN 提交中的 DOI?

如何在Conda中安装R库