移除接近相同的行 numpy数组。

Posted 2023-01-16

如果我有以下numpy数组。

import numpy as np

arr = np.array([[285, 849],
                [399, 715],
                [399, 716],
                [400, 715],
                [400, 716]])

我将如何删除接近相同的行？我并不介意最后是否会有这样的行 [399, 715], [399, 716], [400, 715] 或 [400, 716]. 例如，我想得到的最终结果是。

out = remove_near_identical(arr)
print(out)

[[285 849]
 [399 715]]

答案

下面假设你有一个二维数据集，并且它保留了顺序。如果元素有一个 average difference = ((array_1 - array_2)/#of_element) < threshold

import numpy as np


def remove_near_indentical_rows(arr, threshold):

    row, column = arr.shape
    arg = arr.argsort(axis=0)[:, 0]
    arr=arr[arg]

    arr_mask = np.zeros(row, dtype=bool)
    cur_row = arr[0]
    arr_mask[0] = True
    for i in range(1, row):
        if np.sum(np.abs(arr[i] - cur_row))/column > threshold:
            arr_mask[i] = True
            cur_row = arr[i]

    arg = arg[arr_mask]
    return arr[arg]

arr = np.array([[399, 715],
                [285, 849],
                [399, 716],
                [400, 715],
                [400, 716]])

arr = remove_near_indentical_rows(arr, 10)
print(arr)

产出

[[399 715]
 [285 849]]

另一答案

办法1

好吧，如果你不知道决定的是什么。近似 标准，我想一个著名的标准就是基于它们之间的距离。考虑到这一点，某种基于距离的聚类解决方案可能很适合这里。所以，这里有一个与 sklearn.cluster.AgglomerativeClustering -

from sklearn.cluster import AgglomerativeClustering

def cluster_based_on_distance(a, dist_thresh=10):
    kmeans= AgglomerativeClustering(n_clusters=None, distance_threshold=dist_thresh).fit(a)
    return a[np.sort(np.unique(kmeans.labels_, return_index=True)[1])]

抽样调查----

In [16]: a
Out[16]: 
array([[285, 849],
       [450, 717],
       [399, 715],
       [399, 716],
       [400, 715],
       [450, 716],
       [150, 716]])

In [17]: cluster_based_on_distance(a, dist_thresh=10)
Out[17]: 
array([[285, 849],
       [450, 717],
       [399, 715],
       [150, 716]])

In [18]: cluster_based_on_distance(a, dist_thresh=100)
Out[18]: 
array([[285, 849],
       [450, 717],
       [150, 716]])

In [19]: cluster_based_on_distance(a, dist_thresh=1000)
Out[19]: array([[285, 849]])

办法2

另一个是基于 euclidean-distance 基于阈值，具有 KDTree -

from scipy.spatial import cKDTree

def cluster_based_on_eucl_distance(a, dist_thresh=10):
    d,idx = cKDTree(a).query(a, k=2)
    min_idx = idx.min(1)
    mask = d[:,1]>dist_thresh
    mask[min_idx[~mask]] = True
    return a[mask]

办法3

另一种是基于任一列之间的绝对差异------------。

def cluster_based_on_either_xydist(a, dist_thresh=10):
    c0 = np.abs(a[:,0,None]-a[:,0])<dist_thresh
    c1 = np.abs(a[:,1,None]-a[:,1])<dist_thresh
    c01 = c0 & c1
    return a[~np.triu(c01,1).any(0)]

另一答案

一种仅基于距离的方法。

import numpy as np
from scipy.spatial.distance import deist

arr = np.array([[285, 849],
                [399, 715],
                [399, 716],
                [400, 715],
                [400, 716]])

# get distances between every set of points
dists = cdist(arr, arr)
dists[np.isclose(dists, 0)] = np.inf # set 0 (self) distances to be large, ie. ignore

# get indices of points less than some threshold value (too close)
i, j = np.where(dists <= 1)
# get the unique indices from either i or j
# and delete all but one of these points from the original array
np.delete(arr, np.unique(i)[1:], axis=0)
>>> array([[285, 849],
           [399, 715]])