opencv：wasm 中的匹配错误
Posted
技术标签:
【中文标题】opencv：wasm 中的匹配错误【英文标题】：opencv : bad matches in wasm 【发布时间】：2020-07-28 23:19:13 【问题描述】：我的目的是编写一个网站（html/js/wasm），其中用户提供两个jpg，它们在wasm 中使用OpenCV 进行“比较”。通过“比较”，我的意思是：计算关键点、描述符、进行 BF 匹配并打印距离。
我能做的是一个纯 c++ 程序读取磁盘上的两个文件并比较它们。没关系。 使用我的两张测试图像，最佳匹配的距离约为 100，而最佳匹配的距离在 200 以下。
事实证明，在尝试使用 wasm 时，关键点计算正确，但匹配错误。最好的距离在 200 左右。
这是我的（最小的？）示例。
html 部分创建了两个存储图像的画布。
<!doctype html>
<html>
<body>
<head>
<meta charset="utf-8">
<title>c++ generated</title>
</head>
<canvas id="viewport1"></canvas>
<input type='file' id='inputimage1' accept='image/*'>
<canvas id="viewport2"></canvas>
<input type='file' id='inputimage2' accept='image/*'>
<script>
var Module =
onRuntimeInitialized: function()
createQuery();
;
</script>
<script src="josef.js"></script>
<script src="main.js"></script>
</body>
</html>
JS 部分包含画布操作和对 c++ 的调用。两个电话：
对于第一张图片，我们调用函数recordImage 将cv::Mat 存储在全局变量中
对于第二张图片，我们调用函数compare 比较两张图片。
/* global Module */
function putOnHeap(imgData, wasmModule)
const uint8ArrData = new Uint8Array(imgData.data);
const numBytes = uint8ArrData.length * uint8ArrData.BYTES_PER_ELEMENT;
const dataPtr = wasmModule._malloc(numBytes);
const dataOnHeap = new Uint8Array(wasmModule.HEAPU8.buffer, dataPtr, numBytes);
dataOnHeap.set(uint8ArrData);
answer = "byteOffset": dataOnHeap.byteOffset,
"length":uint8ArrData.length,
"dataPtr": dataPtr,
"dataOnHeap": dataOnHeap,
"wasmModule":wasmModule,
;
return answer;
/*
Call the c++ function on the given image.
*/
async function toCpp(canvas, wasmModule, functionName)
const context = canvas.getContext('2d');
const imgData = context.getImageData(0, 0,
canvas.width, canvas.height);
const heapInfo = putOnHeap(imgData, wasmModule);
const func = wasmModule[functionName];
func(heapInfo.byteOffset, heapInfo.length,
canvas.width, canvas.height);
wasmModule._free(heapInfo.dataPtr);
return answer;
/*
Record the image on the wasm side.
*/
async function recordImage(canvas, wasmModule)
await toCpp(canvas, wasmModule, "recordImage");
/*
Compare the image with the recorded one.
*/
async function compare(canvas, wasmModule)
await toCpp(canvas, wasmModule, "compare");
/* When the user provides the first image, wasm stores it
* in a cv::Mat.
*/
async function doFirstCanvas()
const canvas = document.getElementById('viewport1');
const width = this.width;
const height = this.height;
canvas.width = width;
canvas.height = height;
const context = canvas.getContext('2d');
let imageData = context.getImageData(0, 0, width, height);
context.putImageData(imageData, 0, 0);
context.drawImage(this, 0, 0);
await recordImage(canvas, Module);
/*
* When the user provides the second image, wasm compares
* that image with the first one.
*
*/
async function doSecondCanvas()
const canvas = document.getElementById('viewport2');
const width = this.width;
const height = this.height;
canvas.width = width;
canvas.height = height;
const context = canvas.getContext('2d');
let imageData = context.getImageData(0, 0, width, height);
context.putImageData(imageData, 0, 0);
context.drawImage(this, 0, 0);
compare(canvas, Module);
function failed()
function createQuery()
document.getElementById('inputimage1').onchange = function ()
console.log('First image received');
const img = new Image();
img.onload = doFirstCanvas;
img.onerror = failed;
img.src = URL.createObjectURL(this.files[0]);
;
document.getElementById('inputimage2').onchange = function ()
console.log('Second image received');
const img = new Image();
img.onload = doSecondCanvas;
img.onerror = failed;
img.src = URL.createObjectURL(this.files[0]);
;
c++ 部分特别包含函数get_image，它读取堆并创建相应的cv::Mat。
#include <emscripten/bind.h>
#include <emscripten/val.h>
#include <opencv2/core/core.hpp>
#include <opencv2/imgcodecs.hpp>
#include <opencv2/opencv.hpp>
// The first image is recorded in `recordedImage`
cv::Mat recordedImage;
/*
* `get_image` reads the part of the heap indicated by JS.
* Then it interprets it as a cv::Mat.
* */
cv::Mat get_image(int offset, size_t size, int width, int height)
uint8_t* pos;
pos = reinterpret_cast<uint8_t*>(offset);
auto gotMatrix = cv::Mat(width, height, CV_8UC4, pos);
return gotMatrix.clone();
/*The first image is recorded in `recordedImage`.*/
void recordImage(int offset, size_t size, int width, int height)
recordedImage = get_image(offset, size, width, height);
/*The second image is compared to the first one.*/
void compare(int offset, size_t size, int width, int height)
std::cout << "===== c++ comparing the images ======" << std::endl;
auto image1 = recordedImage;
auto image2 = get_image(offset, size, width, height);
auto orbDetector = cv::ORB::create(500);
cv::BFMatcher matcher(cv::NORM_L2);
std::vector<cv::KeyPoint> kpts1;
std::vector<cv::KeyPoint> kpts2;
cv::Mat descr1;
cv::Mat descr2;
orbDetector -> detectAndCompute(image1, cv::noArray(), kpts1, descr1);
orbDetector -> detectAndCompute(image2, cv::noArray(), kpts2, descr2);
std::vector<cv::DMatch> matches;
matcher.match(descr1, descr2, matches);
for (auto & match : matches)
std::cout << match.distance << std::endl;
int main()
std::cout << "Run main in c++." << std::endl;
return 0;
// Export the functions.
EMSCRIPTEN_BINDINGS(my_module)
emscripten::function("main", &main);
emscripten::function("compare", &compare);
emscripten::function("recordImage", &recordImage);
我已经尝试打印cv::Mat，而且，事实上，我在这个版本中的图像与纯 c++ 中的图像不同。不同之处在于画布添加了“alpha”通道，而 OpenCV 置换了 R 和 B 通道。我已经在 JS 中进行了一些预先操作来修复这些差异，并检查了 html/js/wasm 版本中的矩阵与 c++ 版本中的矩阵是否完全相同（据我所知）。
我没有在此处的示例中包含这些操作。
我的问题：match.distance 的值远大于使用纯 c++ 程序读取磁盘 (cv::imread) 获得的相同图像的值。
为什么？
【问题讨论】：
【参考方案1】：函数“get_image”被破坏有几个原因。这是我的解决方案
cv::Mat get_image(int offset, size_t size, int width, int height)
// Some comments about the implementation
// - People in the doc do not need the cast trick
// for the position. Why ???
// - The inversion height <--> width is crucial
// - I'm not sure how useful the line `cvtColor` is.
// - The `clone` trick is crucial. If not, the returned matrix is
// filled with references to the heap. In that case, if one "send"
// a second matrix from js to wasm, the second matrix "erases" the
// first one.
uint8_t* position = reinterpret_cast<uint8_t*>(offset);
auto gotMatrix = cv::Mat(height, width, CV_8UC4, position);
cv::Mat newImage;
cv::cvtColor(gotMatrix, newImage, cv::COLOR_RGBA2BGR );
return newImage.clone();
【讨论】：
以上是关于opencv：wasm 中的匹配错误的主要内容，如果未能解决你的问题，请参考以下文章