异常处理RSelenium switchToFrame()错误:ElementNotVisible
Posted
tags:
篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了异常处理RSelenium switchToFrame()错误:ElementNotVisible相关的知识,希望对你有一定的参考价值。
我正在尝试在RSelenium
中实现异常处理,并需要帮助。请注意,我已使用robotstxt
包检查了抓取此页面的权限。
library(RSelenium)
library(XML)
library(janitor)
library(lubridate)
library(magrittr)
library(dplyr)
remDr <- remoteDriver(
remoteServerAddr = "192.168.99.100",
port = 4445L
)
remDr$open()
# Open TightVNC to follow along as RSelenium drives the browser
# navigate to the main page
remDr$navigate("https://docs.google.com/spreadsheets/d/1o1PlLIQS8v-XSuEz1eqZB80kcJk9xg5lsbueB7mTg1U/pub?output=html&widget=true#gid=690408156")
# look for table element
tableElem <- remDr$findElement(using = "id", "pageswitcher-content")
# switch to table
remDr$switchToFrame(tableElem)
# parse html for first table
doc <- htmlParse(remDr$getPageSource()[[1]])
table_tmp <- readHTMLTable(doc)
table_tmp <- table_tmp[[1]][-2, -1]
table_tmp <- table_tmp[-1, ]
colnames(table_tmp) <- c("team_name", "team_size", "start_time", "end_time", "total_time", "puzzels_solved")
table_tmp$city <- rep("montreal", nrow(table_tmp))
table_tmp$date <- rep(Sys.Date() - 5, nrow(table_tmp))
# switch back to the main/outer frame
remDr$switchToFrame(NULL)
# I found the elements I want to manipulate with Inspector mode in a browser
webElems <- remDr$findElements(using = "css", ".switcherItem") # Month/Year tabs at the bottom
arrowElems <- remDr$findElements(using = "css", ".switcherArrows") # Arrows to scroll left and right at the bottom
# Create NULL object to be used in for loop
big_df <- NULL
for (i in seq(length(webElems))) {
# choose the i'th Month/Year tab
webElem <- webElems[[i]]
webElem$clickElement()
tableElem <- remDr$findElement(using = "id", "pageswitcher-content") # The inner table frame
# switch to table frame
remDr$switchToFrame(tableElem)
Sys.sleep(3)
# parse html with XML package
doc <- htmlParse(remDr$getPageSource()[[1]])
Sys.sleep(3)
# Extract data from HTML table in HTML document
table_tmp <- readHTMLTable(doc)
Sys.sleep(3)
# put this into a format you can use
table <- table_tmp[[1]][-2, -1]
table <- table[-1, ]
# rename the columns
colnames(table) <- c("team_name", "team_size", "start_time", "end_time", "total_time", "puzzels_solved")
# add city name to a column
table$city <- rep("Montreal", nrow(table))
# add the Month/Year this table was extracted from
today <- Sys.Date() %m-% months(i + 1)
table$date <- today
# concatenate each table together
big_df <- dplyr::bind_rows(big_df, table)
# Switch back to main frame
remDr$switchToFrame(NULL)
################################################
### I should use exception handling here ###
################################################
}
当浏览器到达January 2018
表时,它无法再找到下一个webElems
元素并抛出错误:
Selenium消息:元素当前不可见,因此可能无法与构建信息进行交互:版本:'2.53.1',修订版:'a36b8b1',时间:'2016-06-30 17:37:03'系统信息:主机:'617e51cbea11',ip:'172.17.0.2',os.name:'Linux',os.arch:'amd64',os.version:'4.14.79-boot2docker',java.version:'1.8.0_91'驱动程序信息:driver.version:未知
错误:摘要:ElementNotVisible详细信息:无法完成元素命令,因为该元素在页面上不可见。 class:org.openqa.selenium.ElementNotVisibleException进一步的细节:运行errorDetails方法另外:有50个或更多警告(使用warnings()查看前50个)
通过在for循环结束时包含这些代码,我一直在天真地处理它。这不是一个好主意,原因有两个:1)滚动速度很难找出并且会在其他(更长的)谷歌页面上失败,2)for循环最终在尝试单击右箭头时失败但最终失败它已经在最后 - 因此它不会下载最后几张表。
# click the right arrow to scroll right
arrowElem <- arrowElems[[1]]
# once you "click"" the element it is "held down" - no way to " unclick" to prevent it from scrolling too far
# I currently make sure it only scrolls a short distance - via Sys.sleep() before switching to outer frame
arrowElem$clickElement()
# give it "just enough time" to scroll right
Sys.sleep(0.3)
# switch back to outer frame to re-start the loop
remDr$switchToFrame(NULL)
我希望发生的是当弹出此错误时执行arrowElem$clickElement()
来处理此异常。我想通常会使用tryCatch()
;然而,这也是我第一次学习异常处理。我想我可以将它包含在for循环的remDr$switchToFrame(tableElem)
部分中,但它不起作用:
tryCatch({
suppressMessages({
remDr$switchToFrame(tableElem)
})
},
error = function(e) {
arrowElem <- arrowElems[[1]]
arrowElem$clickElement()
Sys.sleep(0.3)
remDr$switchToFrame(NULL)
}
)
我试了一下。当异常处理时,我喜欢使用表单中的某些东西
check <- try(expression, silent = TRUE) # or suppressMessages(try(expression, silent = TRUE))
if (any(class(check) == "try-error")) {
# do stuff
}
我发现它使用方便,通常工作正常,包括使用硒时。然而,这里遇到的问题是点击箭头一次总会把我带到最后一张可见的纸张 - 跳过中间的所有东西。
Alternative Solution
所以这里有一个替代方案,它将解决报废表的任务,而不是上述意义上的异常处理任务。
代码
# Alernative: -------------------------------------------------------------
remDr <- RSelenium::remoteDriver(
remoteServerAddr = "192.168.99.100",
port = 4445L
)
remDr$open(silent = TRUE)
# navigate to the main page
# needs no be done once before looping, else content is not available
remDr$navigate("https://docs.google.com/spreadsheets/d/1o1PlLIQS8v-XSuEz1eqZB80kcJk9xg5lsbueB7mTg1U/pub?output=html&widget=true#gid=690408156")
# I. Preliminaries:
#
# 1. build the links to all spreadsheets
# 2. define the function create_table
#
# 1.
# get page source
html <- remDr$getPageSource()[[1]]
# split it line by line
html <- unlist(strsplit(html, '
'))
# restrict to script section
script <- grep('^s*vars+gidMatch', html, value = TRUE)
# split the script by semi-colon
script <- unlist(strsplit(script, ';'))
# retrieve information
sheet_months <- gsub('.*name:.{2}(.*?).{1},.*', '1',
grep('{names*:', script, value = TRUE), perl = TRUE)
sheet_gid <- gsub('.*gid:.{2}(.*?).{1},.*', '1',
grep('gids*:', script, value = TRUE), perl = TRUE)
sheet_url <- paste0('https://docs.google.com/spreadsheets/d/1o1PlLIQS8v-XSuEz1eqZB80kcJk9xg5lsbueB7mTg1U/pubhtml/sheet?headers%5Cx3dfalse&gid=',
sheet_gid)
#
# 2.
# table yielding function
# just for readability in the loop
create_table <- function (remDr) {
# parse html with XML package
doc <- XML::htmlParse(remDr$getPageSource()[[1]])
Sys.sleep(3)
# Extract data from HTML table in HTML document
table_tmp <- XML::readHTMLTable(doc)
Sys.sleep(3)
# put this into a format you can use
table <- table_tmp[[1]][-2, -1]
# add a check-up for size mismatch
table_fields <- as.character(t(table[1,]))
if (! any(grepl("size", tolower(table_fields)))) {
table <- table[-1, ]
# rename the columns
colnames(table) <- c("team_name", "start_time", "end_time", "total_time", "puzzels_solved")
table$team_size <- NA_integer_
table <- table[,c("team_name", "team_size", "start_time", "end_time", "total_time", "puzzels_solved")]
} else {
table <- table[-1, ]
# rename the columns
colnames(table) <- c("team_name", "team_size", "start_time", "end_time", "total_time", "puzzels_solved")
}
# add city name to a column
table$city <- rep("Montreal", nrow(table))
# add the Month/Year this table was extracted from
today <- Sys.Date()
lubridate::month(today) <- lubridate::month(today)+1
table$date <- today
# returns the table
table
}
# II. Scrapping the content
#
# 1. selenium to generate the pages
# 2. use create_table to extract the table
#
big_df <- NULL
for (k in seq_along(sheet_url)) {
# 1. navigate to the page
remDr$navigate(sheet_url[k])
# remDr$screenshot(display = TRUE) maybe one wants to see progress
table <- create_table(remDr)
# 2. concatenate each table together
big_df <- dplyr::bind_rows(big_df, table)
# inform progress
cat(paste0('
Gathered table for: ', sheet_months[k]))
}
# close session
remDr$close()
结果
在这里你可以看到head
和tail
的big_df
head(big_df)
# team_name team_size start_time end_time total_time puzzels_solved city date
# 1 Tortoise Tortes 5 19:00 20:05 1:05 5 Montreal 2019-02-20
# 2 Mulholland Drives Over A Smelly Cat 4 7:25 8:48 1:23 5 Montreal 2019-02-20
# 3 B.R.O.O.K. 2 7:23 9:05 1:42 5 Montreal 2019-02-20
# 4 Motivate 4 18:53 20:37 1:44 5 Montreal 2019-02-20
# 5 Fighting Mongooses 3 6:31 8:20 1:49 5 Montreal 2019-02-20
# 6 B Lovers 3 6:40 8:30 1:50 5 Montreal 2019-02-20
tail(big_df)
# team_name team_size start_time end_time total_time puzzels_solved city date
# 545 Ale Mary <NA> 6:05 7:53 1:48 5 Montreal 2019-02-20
# 546 B.R.O.O.K. <NA> 18:45 20:37 1:52 5 Montreal 2019-02-20
# 547 Ridler Co. <NA> 6:30 8:45 2:15 5 Montreal 2019-02-20
# 548 B.R.O.O.K. <NA> 18:46 21:51 3:05 5 Montreal 2019-02-20
# 549 Rotating Puzzle Collective <NA> 18:45 21:51 3:06 5 Montreal 2019-02-20
# 550 Fire Team <NA> 19:00 22:11 3:11 5 Montreal 2019-02-20
简短说明
- 为了执行任务,我所做的是首先生成文档中所有电子表格的链接。去做这个:
导航一次到文档
提取源代码
使用
gid
提取工作表月份和URL(通过regex
数字) - 完成此操作后,循环遍历Urls,收集并绑定表
另外,为了便于阅读,我创建了一个名为create_table
的小函数,它以适当的格式返回表格。它主要是循环中包含的代码。我只添加了列数的安全措施(一些电子表格没有team_size
字段 - 在这些情况下我将其设置为NA_integer
)。
以上是关于异常处理RSelenium switchToFrame()错误:ElementNotVisible的主要内容,如果未能解决你的问题,请参考以下文章