GraphFrames:合并具有相似列值的边缘节点

Posted

技术标签:

【中文标题】GraphFrames:合并具有相似列值的边缘节点【英文标题】:GraphFrames: Merge edge nodes with similar column values 【发布时间】:2020-08-28 04:48:42 【问题描述】:

tl;dr:如何简化图,删除具有相同 name 值的边缘节点?

我有一个定义如下的图表:

import graphframes
from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()
vertices = spark.createDataFrame([
    ('1', 'foo', '1'),
    ('2', 'bar', '2'),
    ('3', 'bar', '3'),
    ('4', 'bar', '5'),
    ('5', 'baz', '9'),
    ('6', 'blah', '1'),
    ('7', 'blah', '2'),
    ('8', 'blah', '3')
], ['id', 'name', 'value'])

edges = spark.createDataFrame([
    ('1', '2'),
    ('1', '3'),
    ('1', '4'),
    ('1', '5'),
    ('5', '6'),
    ('5', '7'),
    ('5', '8')
], ['src', 'dst'])

f = graphframes.GraphFrame(vertices, edges)

生成如下图(其中数字表示顶点 ID):

从等于1 的顶点ID 开始,我想简化图形。这样具有相似name 值的节点将合并为单个节点。结果图看起来有点像 像这样:

注意我们只有一个foo (ID 1)、一个bar (ID 2)、一个baz (ID 5) 和一个blah (ID 6)。顶点的value 无关紧要,只是为了表明每个顶点都是唯一的。

我试图实现一个解决方案,但它很老套,效率极低,我确信有更好的方法(我也不认为它有效):

f = graphframes.GraphFrame(vertices, edges)

# Get the out degrees for our nodes. Nodes that do not appear in
# this dataframe have zero out degrees.
outs = f.outDegrees

# Merge this with our nodes.
vertices = f.vertices
vertices = f.vertices.join(outs, outs.id == vertices.id, 'left').select(vertices.id, 'name', 'value', 'outDegree')
vertices.show()

# Create a new graph with our out degree nodes.
f = graphframes.GraphFrame(vertices, edges)

# Find paths to all edge vertices from our vertex ID = 1
# Can we make this one operation instead of two??? What if we have more than two hops?
one_hop = f.find('(a)-[e]->(b)').filter('b.outDegree is null').filter('a.id == "1"')
one_hop.show()

two_hop = f.find('(a)-[e1]->(b); (b)-[e2]->(c)').filter('c.outDegree is null').filter('a.id == "1"')
two_hop.show()

# Super ugly, but union the vertices from the `one_hop` and `two_hop` above, and unique
# on the name.
vertices = one_hop.select('a.*').union(one_hop.select('b.*'))
vertices = vertices.union(two_hop.select('a.*').union(two_hop.select('b.*').union(two_hop.select('c.*'))))
vertices = vertices.dropDuplicates(['name'])
vertices.show()

# Do the same for the edges
edges = two_hop.select('e1.*').union(two_hop.select('e2.*')).union(one_hop.select('e.*')).distinct()

# We need to ensure that we have the respective nodes from our edges. We do this  by
# Ensuring the referenced vertex ID is in our `vertices` in both the `src` and the `dst`
# columns - This does NOT seem to work as I'd expect!
edges = edges.join(vertices, vertices.id == edges.src, "left").select("src", "dst")
edges = edges.join(vertices, vertices.id == edges.dst, "left").select("src", "dst")
edges.show()

是否有更简单的方法来删除节点(及其相应的边),以便边节点在其name 上是唯一的?

【问题讨论】:

从集合中选择一个 vx 的任何规则,即为什么是 2、3、4 -> 2; 6, 7, 8 -> 6 ? 没有规则,这是随机选择。重要的是节点name 是唯一的。 idvalue 在很大程度上无关紧要。 【参考方案1】:

您为什么不简单地将name 列视为新的id

import graphframes

vertices = spark.createDataFrame([
    ('1', 'foo', '1'),
    ('2', 'bar', '2'),
    ('3', 'bar', '3'),
    ('4', 'bar', '5'),
    ('5', 'baz', '9'),
    ('6', 'blah', '1'),
    ('7', 'blah', '2'),
    ('8', 'blah', '3')
], ['id', 'name', 'value'])

edges = spark.createDataFrame([
    ('1', '2'),
    ('1', '3'),
    ('1', '4'),
    ('1', '5'),
    ('5', '6'),
    ('5', '7'),
    ('5', '8')
], ['src', 'dst'])

#create a dataframe with only one column
new_vertices = vertices.select(vertices.name.alias('id')).distinct()

#replace the src ids with the name column
new_edges = edges.join(vertices, edges.src == vertices.id, 'left')
new_edges = new_edges.select(new_edges.dst, new_edges.name.alias('src'))

#replace the dst ids with the name column
new_edges = new_edges.join(vertices, new_edges.dst == vertices.id, 'left')
new_edges = new_edges.select(new_edges.src, new_edges.name.alias('dst'))

#drop duplicate edges
new_edges = new_edges.dropDuplicates(['src', 'dst'])

new_edges.show()
new_vertices.show()

f = graphframes.GraphFrame(new_vertices, new_edges)

输出:

+---+----+
|src| dst|
+---+----+
|foo| baz|
|foo| bar|
|baz|blah|
+---+----+

+----+
|  id|
+----+
|blah|
| bar|
| foo|
| baz|
+----+

【讨论】:

以上是关于GraphFrames:合并具有相似列值的边缘节点的主要内容,如果未能解决你的问题,请参考以下文章

合并两个具有聚合列值的数据框作为结果

合并具有多索引和列值的数据框

是否可以将 SELECT 语句中具有重复列值的记录与 SQL 中的另一条记录合并?

根据 Pandas 中的公共列值合并两个数据框

合并两个共享相同列值的数据框

QTableView 获取具有相似列值/数据的行列表