NASA Metadata: tf-idf of Description Texts and Keywords
Posted
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了NASA Metadata: tf-idf of Description Texts and Keywords相关的知识，希望对你有一定的参考价值。
鏍囩锛?a href='http://www.mamicode.com/so/1/aaa' title='aaa'>aaa rbo space standard page EOS oca fda aging
NASA鏈?2,000澶氫釜鏁版嵁闆嗭紝骞朵笖NASA鏈夊叴瓒ｄ簡瑙ｈ繖浜涙暟鎹泦涔嬮棿鐨勮仈绯伙紝浠ュ強涓嶯ASA浠ュ鍏朵粬鏀垮簻缁勭粐涓叾浠栭噸瑕佹暟鎹泦鐨勮仈绯汇€傛湁鍏砃ASA鏁版嵁闆嗙殑鍏冩暟鎹?nbsp; 鍙互JSON鏍煎紡鍦ㄧ嚎鑾峰緱銆傝鎴戜滑浣跨敤tf-idf鍦ㄦ弿杩板瓧娈典腑鎵惧埌閲嶈鐨勫崟璇嶏紝骞跺皢鍏朵笌鍏抽敭瀛楄仈绯昏捣鏉ャ€?/p>
鑾峰彇鍜屾暣鐞哊ASA鍏冩暟鎹?/strong>
璁╂垜浠笅杞?2,000澶氫釜NASA鏁版嵁闆嗙殑鍏冩暟鎹€?/p>
library(jsonlite)
library(dplyr)
library(tidyr)
metadata <- fromJSON("https://data.nasa.gov/data.json")
names(metadata$dataset)
## [1] "_id" "@type" "accessLevel" "accrualPeriodicity"
## [5] "bureauCode" "contactPoint" "description" "distribution"
## [9] "identifier" "issued" "keyword" "landingPage"
## [13] "language" "modified" "programCode" "publisher"
## [17] "spatial" "temporal" "theme" "title"
## [21] "license" "isPartOf" "references" "rights"
## [25] "describedBy"
nasadesc <- data_frame(id = metadata$dataset$`_id`$`$oid`, desc = metadata$dataset$description)
nasadesc
## # A tibble: 32,089 x 2
## id
## <chr>
## 1 55942a57c63a7fe59b495a77
## 2 55942a57c63a7fe59b495a78
## 3 55942a58c63a7fe59b495a79
## 4 55942a58c63a7fe59b495a7a
## 5 55942a58c63a7fe59b495a7b
## 6 55942a58c63a7fe59b495a7c
## 7 55942a58c63a7fe59b495a7d
## 8 55942a58c63a7fe59b495a7e
## 9 55942a58c63a7fe59b495a7f
## 10 55942a58c63a7fe59b495a80
## # ... with 32,079 more rows, and 1 more variables: desc <chr>
## # A tibble: 32,089 x 2
## id
## <chr>
## 1 55942a57c63a7fe59b495a77
## 2 55942a57c63a7fe59b495a78
## 3 55942a58c63a7fe59b495a79
## 4 55942a58c63a7fe59b495a7a
## 5 55942a58c63a7fe59b495a7b
## 6 55942a58c63a7fe59b495a7c
## 7 55942a58c63a7fe59b495a7d
## 8 55942a58c63a7fe59b495a7e
## 9 55942a58c63a7fe59b495a7f
## 10 55942a58c63a7fe59b495a80
## # ... with 32,079 more rows, and 1 more variables: desc <chr>璁╂垜浠墦鍗板嚭鍏朵腑鐨勪竴閮ㄥ垎銆?/p>
nasadesc %>% select(desc) %>% sample_n(5)
## # A tibble: 5 x 1
## desc
## <chr>
## 1 A Group for High Resolution Sea Surface Temperature (GHRSST) Level 4 sea surface temperature analysis produced as a retrospective dataset at the JPL P
## 2 ML2CO is the EOS Aura Microwave Limb Sounder (MLS) standard product for carbon monoxide derived from radiances measured by the 640 GHz radiometer. The
## 3 Crew lock bag. Polygons: 405 Vertices: 514
## 4 JEM Engineering proved the technical feasibility of the FlexScan array?a very low-cost, highly-efficient, wideband phased array antenna?in Phase I, an
## 5 MODIS (or Moderate Resolution Imaging Spectroradiometer) is a key instrument aboard the
Terra (EOS AM) and Aqua (EOS PM) satellites. Terra鈥榮 orbit aro杩欐槸鍏抽敭璇嶃€?/p>
nasakeyword <- data_frame(id = metadata$dataset$`_id`$`$oid`,
keyword = metadata$dataset$keyword) %>%
unnest(keyword)
nasakeyword## # A tibble: 126,814 x 2
## id keyword
## <chr> <chr>
## 1 55942a57c63a7fe59b495a77 EARTH SCIENCE
## 2 55942a57c63a7fe59b495a77 HYDROSPHERE
## 3 55942a57c63a7fe59b495a77 SURFACE WATER
## 4 55942a57c63a7fe59b495a78 EARTH SCIENCE
## 5 55942a57c63a7fe59b495a78 HYDROSPHERE
## 6 55942a57c63a7fe59b495a78 SURFACE WATER
## 7 55942a58c63a7fe59b495a79 EARTH SCIENCE
## 8 55942a58c63a7fe59b495a79 HYDROSPHERE
## 9 55942a58c63a7fe59b495a79 SURFACE WATER
## 10 55942a58c63a7fe59b495a7a EARTH SCIENCE
## # ... with 126,804 more rows鏈€甯歌鐨勫叧閿瓧鏄粈涔堬紵
nasakeyword %>% group_by(keyword) %>% count(sort = TRUE)
## # A tibble: 1,774 x 2
## keyword n
## <chr> <int>
## 1 EARTH SCIENCE 14362
## 2 Project 7452
## 3 ATMOSPHERE 7321
## 4 Ocean Color 7268
## 5 Ocean Optics 7268
## 6 Oceans 7268
## 7 completed 6452
## 8 ATMOSPHERIC WATER VAPOR 3142
## 9 OCEANS 2765
## 10 LAND SURFACE 2720
## # ... with 1,764 more rows鐪嬭捣鏉?ldquo;宸插畬鎴愰」鐩?rdquo;瀵逛簬鏌愪簺鐩殑鏉ヨ鍙兘涓嶆槸鏈夌敤鐨勫叧閿瓧锛屾垜浠彲鑳藉笇鏈涘皢鎵€鏈夎繖浜涢兘鏇存敼涓哄皬鍐欐垨澶у啓锛屼互娑堥櫎璇稿“ OCEANS”鍜?ldquo; Oceans”涔嬬被鐨勯噸澶嶉」銆?/p>
nasakeyword <- nasakeyword %>% mutate(keyword = toupper(keyword))
璁＄畻鏂囧瓧鐨則f-idf
浠€涔堟槸tf-idf锛熻瘎浼版枃妗ｄ腑鍗曡瘝鐨勯噸瑕佹€х殑涓€绉嶆柟娉曞彲鑳芥槸鍏?nbsp; 鏈棰戠巼 锛坱f锛夛紝鍗冲崟璇嶅湪鏂囨。涓嚭鐜扮殑棰戠巼銆備絾鏄紝涓€浜涚粡甯稿嚭鐜扮殑鍗曡瘝骞朵笉閲嶈銆傚湪鑻辫涓紝杩欎簺璇嶅彲鑳芥槸“ the”锛?ldquo; is”锛?ldquo; of”绛夎瘝銆傚彟涓€绉嶆柟娉曟槸鏌ョ湅鏈鐨?nbsp; 閫嗘枃妗ｉ鐜?/em> 锛坕df锛夛紝杩欎細闄嶄綆甯哥敤鍗曡瘝鐨勬潈閲嶏紝鑰屽鍔犲湪鏂囨。闆嗕腑寰堝皯浣跨敤鐨勫崟璇嶇殑鏉冮噸銆?/p>
杩欎簺鏄疦ASA璇存槑瀛楁涓渶甯歌鐨?ldquo;鍗曡瘝”锛屾槸璇嶉鏈€楂樼殑鍗曡瘝銆傝鎴戜滑鐪嬩竴涓嬬涓€涓暟鎹泦锛屼緥濡傦細 tf-idf绠楁硶搴旇鍑忓皯鎵€鏈夎繖浜涚殑鏉冮噸锛屽洜涓哄畠浠緢甯歌锛屼絾鏄垜浠彲浠ユ牴鎹渶瑕侀€氳繃鍋滅敤璇嶅皢鍏跺垹闄ゃ€傜幇鍦紝璁╂垜浠负鎻忚堪瀛楁涓殑鎵€鏈夊崟璇嶈绠梩f-idf銆?/p>
娣诲姞鐨勫垪鏄痶f锛宨df锛岃繖涓や釜鏁伴噺鐩镐箻鍦ㄤ竴璧锋槸tf-idf锛岃繖鏄垜浠劅鍏磋叮鐨勪笢瑗裤€侼ASA鎻忚堪瀛楁涓渶楂樼殑tf-idf璇嶆槸浠€涔堬紵 鍥犳锛岃繖浜涙槸鐢╰f-idf琛￠噺鐨勬弿杩板瓧娈典腑鏈€“閲嶈”鐨勮瘝锛岃繖鎰忓懗鐫€瀹冧滑寰堝父瑙侊紝浣嗕笉澶父鐢ㄣ€?/p>
tf-idf绠楁硶灏嗚涓鸿繖鏄竴涓潪甯搁噸瑕佺殑璇嶃€?/p>
鍥犳锛岀幇鍦ㄦ垜浠煡閬撴弿杩颁腑鐨勫摢涓瘝鍏锋湁杈冮珮鐨則f-idf锛屽苟涓斿湪鍏抽敭瀛椾腑涔熸湁杩欎簺鎻忚堪鐨勬爣绛俱€?/p>
璁╂垜浠潵鐪嬪嚑涓ず渚嬪叧閿瓧涓渶閲嶈鐨勫崟璇嶃€?/p>
tf-idf绠楁硶鍦ㄤ粎2涓瓧闀跨殑鎻忚堪涓棤娉曞緢濂藉湴宸ヤ綔锛屾垨鑰呰嚦灏戝畠灏嗗杩欎簺瀛楄繘琛岄潪甯搁噸鐨勫姞鏉冦€傚疄闄呬笂锛屼篃璁歌繖鏄笉鍚堥€傜殑銆?/p>
library(tidytext)
descwords <- nasadesc %>% unnest_tokens(word, desc) %>%
count(id, word, sort = TRUE) %>%
ungroup()
descwords## # A tibble: 2,728,224 x 3
## id word n
## <chr> <chr> <int>
## 1 55942a88c63a7fe59b498280 amp 679
## 2 55942a88c63a7fe59b498280 nbsp 655
## 3 55942a8ec63a7fe59b4986ef gt 330
## 4 55942a8ec63a7fe59b4986ef lt 330
## 5 55942a8ec63a7fe59b4986ef p 327
## 6 55942a8ec63a7fe59b4986ef the 231
## 7 55942a86c63a7fe59b49803b amp 208
## 8 55942a86c63a7fe59b49803b nbsp 204
## 9 56cf5b00a759fdadc44e564a the 201
## 10 55942a86c63a7fe59b4980a2 gt 191
## # ... with 2,728,214 more rowsnasadesc %>% filter(id == "55942a88c63a7fe59b498280") %>% select(desc)
## # A tibble: 1 x 1
## desc
## <chr>
## 1 <p>The objective of the Variable Oxygen Regulator Element is to develop an oxygen-rated, contaminant-tolerant oxygen regulator to control suit pdescwords <- descwords %>% bind_tf_idf(word, id, n)
descwords## # A tibble: 2,728,224 x 6
## id word n tf idf tf_idf
## <chr> <chr> <int> <dbl> <dbl> <dbl>
## 1 55942a88c63a7fe59b498280 amp 679 0.35661765 3.1810813 1.134429711
## 2 55942a88c63a7fe59b498280 nbsp 655 0.34401261 4.2066578 1.447143322
## 3 55942a8ec63a7fe59b4986ef gt 330 0.05722213 3.2263517 0.184618705
## 4 55942a8ec63a7fe59b4986ef lt 330 0.05722213 3.2903671 0.188281801
## 5 55942a8ec63a7fe59b4986ef p 327 0.05670192 3.3741126 0.191318680
## 6 55942a8ec63a7fe59b4986ef the 231 0.04005549 0.1485621 0.005950728
## 7 55942a86c63a7fe59b49803b amp 208 0.32911392 3.1810813 1.046938133
## 8 55942a86c63a7fe59b49803b nbsp 204 0.32278481 4.2066578 1.357845252
## 9 56cf5b00a759fdadc44e564a the 201 0.06962245 0.1485621 0.010343258
## 10 55942a86c63a7fe59b4980a2 gt 191 0.12290862 3.2263517 0.396546449
## # ... with 2,728,214 more rowsdescwords %>% arrange(-tf_idf)
## # A tibble: 2,728,224 x 6
## id word n tf idf
## <chr> <chr> <int> <dbl> <dbl>
## 1 55942a7cc63a7fe59b49774a rdr 1 1 10.376269
## 2 55942ac9c63a7fe59b49b688 palsar_radiometric_terrain_corrected_high_res 1 1 10.376269
## 3 55942ac9c63a7fe59b49b689 palsar_radiometric_terrain_corrected_low_res 1 1 10.376269
## 4 55942a7bc63a7fe59b4976ca lgrs 1 1 8.766831
## 5 55942a7bc63a7fe59b4976d2 lgrs 1 1 8.766831
## 6 55942a7bc63a7fe59b4976e3 lgrs 1 1 8.766831
## 7 55942ad8c63a7fe59b49cf6c template_proddescription 1 1 8.296827
## 8 55942ad8c63a7fe59b49cf6d template_proddescription 1 1 8.296827
## 9 55942ad8c63a7fe59b49cf6e template_proddescription 1 1 8.296827
## 10 55942ad8c63a7fe59b49cf6f template_proddescription 1 1 8.296827
## tf_idf
## <dbl>
## 1 10.376269
## 2 10.376269
## 3 10.376269
## 4 8.766831
## 5 8.766831
## 6 8.766831
## 7 8.296827
## 8 8.296827
## 9 8.296827
## 10 8.296827
## # ... with 2,728,214 more rowsnasadesc %>% filter(id == "55942a7cc63a7fe59b49774a") %>% select(desc)
## # A tibble: 1 x 1
## desc
## <chr>
## 1 RDR杩炴帴鍏抽敭瀛楀拰鎻忚堪
descwords <- full_join(descwords, nasakeyword, by = "id")
descwords## # A tibble: 11,013,838 x 7
## id word n tf idf tf_idf keyword
## <chr> <chr> <int> <dbl> <dbl> <dbl> <chr>
## 1 55942a88c63a7fe59b498280 amp 679 0.35661765 3.181081 1.1344297 ELEMENT
## 2 55942a88c63a7fe59b498280 amp 679 0.35661765 3.181081 1.1344297 JOHNSON SPACE CENTER
## 3 55942a88c63a7fe59b498280 amp 679 0.35661765 3.181081 1.1344297 VOR
## 4 55942a88c63a7fe59b498280 amp 679 0.35661765 3.181081 1.1344297 ACTIVE
## 5 55942a88c63a7fe59b498280 nbsp 655 0.34401261 4.206658 1.4471433 ELEMENT
## 6 55942a88c63a7fe59b498280 nbsp 655 0.34401261 4.206658 1.4471433 JOHNSON SPACE CENTER
## 7 55942a88c63a7fe59b498280 nbsp 655 0.34401261 4.206658 1.4471433 VOR
## 8 55942a88c63a7fe59b498280 nbsp 655 0.34401261 4.206658 1.4471433 ACTIVE
## 9 55942a8ec63a7fe59b4986ef gt 330 0.05722213 3.226352 0.1846187 JOHNSON SPACE CENTER
## 10 55942a8ec63a7fe59b4986ef gt 330 0.05722213 3.226352 0.1846187 PROJECT
## # ... with 11,013,828 more rows鍙鍖栫粨鏋?/strong>
plot_words <- descwords %>% filter(!near(tf, 1)) %>%
filter(keyword %in% c("SOLAR ACTIVITY", "CLOUDS",
"VEGETATION", "ASTROPHYSICS",
"HUMAN HEALTH", "BUDGET")) %>%
arrange(desc(tf_idf)) %>%
group_by(keyword) %>%
distinct(word, keyword, .keep_all = TRUE) %>%
top_n(20, tf_idf) %>% ungroup() %>%
mutate(word = factor(word, levels = rev(unique(word))))
plot_words## # A tibble: 122 x 7
## id word n tf idf tf_idf keyword
## <chr> <fctr> <int> <dbl> <dbl> <dbl> <chr>
## 1 55942a60c63a7fe59b49612f estimates 1 0.5000000 3.172863 1.586432 CLOUDS
## 2 55942a76c63a7fe59b49728d ncdc 1 0.1666667 7.603680 1.267280 CLOUDS
## 3 55942a60c63a7fe59b49612f cloud 1 0.5000000 2.464212 1.232106 CLOUDS
## 4 55942a5ac63a7fe59b495bd8 fife 1 0.2000000 5.910360 1.182072 CLOUDS
## 5 55942a5cc63a7fe59b495deb allometry 1 0.1428571 7.891362 1.127337 VEGETATION
## 6 55942a5dc63a7fe59b495ede tgb 3 0.1875000 5.945452 1.114772 VEGETATION
## 7 55942a5ac63a7fe59b495bd8 tovs 1 0.2000000 5.524238 1.104848 CLOUDS
## 8 55942a5ac63a7fe59b495bd8 received 1 0.2000000 5.332843 1.066569 CLOUDS
## 9 55942a5cc63a7fe59b495dfd sap 1 0.1250000 8.430358 1.053795 VEGETATION
## 10 55942a60c63a7fe59b496131 abstract 1 0.3333333 3.118561 1.039520 CLOUDS
## # ... with 112 more rowsnasadesc %>% filter(id == "55942a60c63a7fe59b49612f") %>% select(desc)
## # A tibble: 1 x 1
## desc
## <chr>
## 1 Cloud estimateslibrary(ggplot2)
library(ggstance)
library(ggthemes)
ggplot(plot_words, aes(tf_idf, word, fill = keyword, alpha = tf_idf)) +
geom_barh(stat = "identity", show.legend = FALSE) +
labs(title = "Highest tf-idf words in NASA Metadata Description Fields",
subtitle = "Distribution of tf-idf for words from datasets labeled with various keywords",
caption = "NASA metadata from https://data.nasa.gov/data.json",
y = NULL, x = "tf-idf") +
facet_wrap(~keyword, ncol = 3, scales = "free") +
theme_tufte(base_family = "Arial", base_size = 13, ticks = FALSE) +
scale_alpha_continuous(range = c(0.2, 1)) +
scale_x_continuous(expand=c(0,0)) +
theme(strip.text=element_text(hjust=0)) +
theme(plot.caption=element_text(size=9))
?
以上是关于NASA Metadata: tf-idf of Description Texts and Keywords的主要内容，如果未能解决你的问题，请参考以下文章
用于聚类（和分类）短句的 NLP bag-of-words/TF-IDF
Android Studio报错：The binary version of its metadata is 1.5.1, expected version is 1.1.15.
Android Studio报错：The binary version of its metadata is 1.5.1, expected version is 1.1.15.
detectron2报AttributeError: Attribute ‘evaluator_type‘ does not exist in the metadata of dataset(代码片段