如何从 html 中只获取没有垃圾的文本？

Posted 2023-02-24

【中文标题】如何从 html 中只获取没有垃圾的文本？

【英文标题】：How to get from html only the text without garbage?

【发布时间】：2016-07-02 03:27:56

【问题描述】：

我正在使用 jsoup 试图仅获取用于我的研究的文本，并且我得到了很多文本“垃圾”，我不知道如何摆脱它。 我的代码：

Document doc = Jsoup.connect(link).userAgent("Mozilla").ignoreHttpErrors(true).timeout(0).get();

String plainText = doc.body().text();

对于这个网站的url（保存在上面代码中的变量链接中）例如： http://www.besthealthmag.ca/best-eats/healthy-eating/5-reasons-to-eat-more-avocados/

输出将是：

加拿大最佳健康杂志生活得更好。感觉很棒 最好的外观 美容 头发 指甲 皮肤风格 最好的你 感冒和流感 糖尿病 健康 心脏健康 口腔健康 关系 睡眠 健康 最好吃 烹饪 饮食 消化 健康饮食 营养 食谱 冰沙 交换和掉落比赛和游戏 比赛 游戏 优惠券 订阅 送礼物 你在这里： Home / Best Eats / 健康饮食 / 多吃鳄梨的 5 个理由 多吃鳄梨的 5 个理由 忘记脂肪鳄梨是一种超级健康的方式，可以在饮食中添加有价值的营养和纤维（是的，还有健康的脂肪！）。这就是为什么 By Best Health 有些人为了保持健康意识，避免食用鳄梨，因为这些水果的脂肪和卡路里含量相对较高（半个中等大小的鳄梨中含有 138 卡路里和 14.1 克脂肪）。然而，鳄梨是您可以吃的最好的食物之一，富含营养和有益心脏健康的化合物。以下是定期食用它们的五个重要理由。 1. 鳄梨富含类胡萝卜素 鳄梨是叶黄素的重要来源，叶黄素是一种类胡萝卜素，可作为抗氧化剂，有助于预防眼部疾病。它们还含有相关的类胡萝卜素玉米黄质、α-胡萝卜素和β-胡萝卜素，以及生育酚（维生素 E）。但是鳄梨本身不仅是类胡萝卜素的丰富来源，它们还可以帮助您从其他食物中获取更多这些营养素。类胡萝卜素是亲脂性的（溶于脂肪，而不是水），所以吃富含类胡萝卜素的食物，如水果和蔬菜以及富含单不饱和脂肪的鳄梨，有助于身体吸收类胡萝卜素。一个简单的方法是在混合沙拉中加入鳄梨片。 2.牛油果让你有饱腹感半个牛油果含有3.4克纤维，包括可溶性和不溶性纤维，这两种纤维都是你的身体保持消化系统顺畅运转所必需的。此外，可溶性纤维可减缓体内碳水化合物的分解，帮助您长时间保持饱腹感。鳄梨还含有油酸，这种脂肪可以激活大脑中让你感到饱腹的部分。与加工食品中不太健康的饱和脂肪和反式脂肪相比，含有油酸的更健康的不饱和脂肪已被证明能产生更大的饱腹感。 3. 牛油果可以保护你未出生的婴儿和你的心脏 一杯牛油果可以提供你每日推荐摄入量叶酸的近四分之一，叶酸是一种降低出生缺陷风险的维生素。如果您怀孕或计划成为，鳄梨将有助于保护您未出生的婴儿。高叶酸摄入量也与心脏病发作和心脏病的风险降低有关。您的家人是否有心脏病史，或者您是否有心脏病的危险因素（例如超重或吸烟）？鳄梨有助于保持心脏健康。 4. 鳄梨可以帮助降低胆固醇 除了增加饱腹感外，鳄梨中的油酸还可以帮助降低胆固醇水平。在一项研究中，食用富含鳄梨的饮食的人总胆固醇水平显着降低，包括低密度脂蛋白胆固醇的降低。他们的高密度脂蛋白胆固醇（健康型）水平增加了 11%。高胆固醇是心脏病的主要危险因素之一。鳄梨的降胆固醇特性及其叶酸含量有助于保持心脏健康。 5. 鳄梨味道很好 最后一个原因是简单的'鳄梨是一种健康的方式来提升你的饭菜的味道和质地。将切碎的鳄梨撒在沙拉或一碗汤上，将鳄梨酱作为开胃菜或调味品，或者尝试其中一种健康的鳄梨食谱，让更健康的鳄梨进入你的饮食。 ‘鳄梨鸡肉俱乐部三明治’鳄梨和虾杯‘鳄梨生牛肉片配野蓝莓干酪混合’烤三文鱼配鳄梨芒果沙司‘经典鳄梨酱不要错过！订阅我们的免费每周通讯，获取营养食谱、健康减肥秘诀、保持身材的简单方法以及您需要的所有健康新闻，这些信息会直接发送到您的收件箱。提交如下：健康饮食 标记为：保持健康和快乐脊柱伸展的饮食与营养秘诀 来源：最佳健康杂志，2011 年夏季； Kagan McLeod 的插图谁不想要更好的姿势？这个快速的动作可以帮助您轻松站（和坐）高，因为​​它可以对齐脊柱。它可以缓解臀部和肩部的紧张，玛戈特说…… [阅读更多...] 花园蔬菜烤牛肉 来源：为健康的心脏烹饪聪明，加拿大读者文摘成分 2 个蒜瓣，切碎的 1?2 茶匙胡椒 1 带骨牛肉沙朗牛排 2 厘米厚，约 1.2 公斤 500 克土豆，四分之一 500 克青豆 1 个大红洋葱，非常…… [阅读更多...] 热带水果冰沙配料 1/4 杯（60 毫升）新鲜或罐装菠萝块 1/4 杯（60毫升）新鲜芒果块 2 新鲜或冷冻草莓，去壳 1 杯（250 毫升）纯低脂豆浆 1 汤匙（15 毫升）酸橙汁 搅拌至光滑。提供一个。每份营养…… [阅读更多...] 美味迷迭香“Squaffles” " 最佳健康杂志，2012 年 3 月/4 月；照片由 Michael Alberstat 拍摄成分1 杯（250 毫升）全麦面粉1 杯（250 毫升）通用面粉1 1/2 茶匙（7 毫升）干迷迭香（或 1 汤匙/15 毫升切碎的新鲜）1 茶匙（5 毫升）发酵粉1茶匙（5 毫升）磨碎…… [阅读更多...] 菜肉馅煎蛋饼配芦笋和大葱成分 1 磅新鲜的细芦笋 4 盎司意大利熏火腿或培根，厚片，富含脂肪（约 4 片）1/2 磅大葱 3 汤匙特级初榨橄榄油 1/2 茶匙粗海盐或犹太盐，或更多以品尝8 大... [阅读更多...] 我们还认为您会喜欢：凯特琳·布里斯托（Kaitlyn Bristowe）谈论大头发，碧昂斯？和单身汉肉毒杆菌毒素、填充剂和一般皮肤治疗：在你预订臃肿之前要知道什么？如何知道它是否是肠易激综合征如果没有人相信你生病了怎么办？ #PhysiotherapyHelpsLives：中风康复 在社交网络上关注我们？ ? ? ? ? ?特色内容 5 种缓解背痛的方法 背痛？我们深入研究了最新的研究，以找出您可以做些什么来帮助缓解背痛 柳树：缓解肌肉和关节疼痛的止痛食品和补充剂的自然疗法 影响加拿大人的 5 种自身免疫性疾病竞赛 最适合最后进入赢得三项之一（ 3) 奖品包！赢得牙买加假期 输入有机会赢得牙买加之旅！加载厚箱... 关于最佳健康 Best Health 是著名出版商读者文摘的健康与保健杂志，为当今的当代加拿大女性带来鼓舞人心的声音。单击此处了解广告机会。您的隐私权 |客户服务 |订阅 |网站地图 最近的帖子 凯特琳布里斯托谈论大头发，碧昂斯？和单身汉肉毒杆菌毒素、填充剂和一般皮肤治疗：在你预订臃肿之前要知道什么？如何知道它是否是肠易激综合征如果没有人相信你生病了怎么办？ #PhysiotherapyHelpsLives：中风康复 不要错过最佳健康的一天 注册我们的时事通讯 在社交上与我们联系？ ? ? ? ? ?显示完整网站 版权所有 © 2016 The Reader's Digest Association, Inc.

有很多垃圾：？问号，以及 [阅读更多...] 等等..

我该怎么办？

【参考方案1】：

您应该缩小范围以仅提取您想要的文本。

对于您示例中的网站，请尝试以下操作：

String plainText = doc.select("article[itemProp=blogPost]").text();

DEMO

由于您要定位 500 多个网站，因此上述 CSS 查询可能并非在所有地方都有效。您需要适应每个网站:-\。

【讨论】：

这个网站很好用，但问题是是否有办法自动适应其他网站？导致它根本无法与其他网站一起使用..

要么找到某些站点共有的模式，要么手动查看每个站点以确定从给定站点中提取的内容。由于网站是为人类创建的，因此（目前）很难要求程序“查看”页面并仅提取有意义的文本。

【参考方案2】：

您可以使用 Scanner 并跳过您不喜欢的令牌。像这样迭代令牌

Scanner s = new Scanner(": Best Health Magazine Canada Live Better. Feel Great Best Looks Beauty Hair Nails Skin Style Best You Cold and Flu Diabetes Health Heart Health Oral Health Relationships Sleep Wellness Best Eats Cooking Diet Digestion Healthy Eating Nutrition Recipes Smoothies Swap and Drop Contests and Games Contests Games Coupons Subscribe Give a gift You are here : Home / Best Eats / Healthy Eating / 5 reasons to eat more avocados 5 reasons to eat more avocados Forget the fat'avocados are a super-healthy way to add valuable nutrients and fibre (yes, and healthy fat!) to your diet. Here's why By Best Health Some people, in their attempts to be health-conscious, avoid avocados due to the relatively high fat and calorie content of these fruits (138 calories and 14.1g fat in half a medium-sized avocado). Yet avocados are one of the best foods you can eat, packed with nutrients and heart-healthy compounds. Here are five great reasons to eat them regularly. 1. Avocados are packed with carotenoids Avocados are a great source of lutein, a carotenoid that works as an antioxidant and helps protect against eye disease. They also contain the related carotenoids zeaxanthin, alpha-carotene and beta-carotene, as well as tocopherol (vitamin E). But avocados aren’t just a rich source of carotenoids by themselves’they also help you get more of these nutrients from other foods. Carotenoids are lipophilic (soluble in fat, not water), so eating carotenoid-packed foods like fruits and vegetables along with monounsaturated-fat-rich avocados helps your body absorb the carotenoids. An easy way to do this is to add sliced avocado to a mixed salad. 2. Avocados make you feel full Half an avocado contains 3.4 grams of fibre, including soluble and insoluble, both of which your body needs to keep the digestive system running smoothly. Plus, soluble fibre slows the breakdown of carbohydrates in your body, helping you feel full for longer. Avocados also contain oleic acid, a fat that activates the part of your brain that makes you feel full. Healthier unsaturated fats containing oleic acid have been shown to produce a greater feeling of satiety than less-healthy saturated fats and trans fats found in processed foods. 3. Avocados can protect your unborn baby’and your heart One cup of avocado provides almost a quarter of your recommended daily intake of folate, a vitamin which cuts the risk of birth defects. If you’re pregnant’or planning to be’avocados will help protect your unborn baby. A high folate intake is also associated with a lower risk of heart attacks and heart disease. Does your family have a history of heart problems, or do you have risk factors (such as being overweight or smoking) for heart disease? Avocados could help keep your heart healthy. 4. Avocados can help lower your cholesterol As well as increasing feelings of fullness, the oleic acid in avocados can help reduce cholesterol levels. In one study, individuals eating an avocado-rich diet had a significant decrease in total cholesterol levels, including a decrease in LDL cholesterol. Their levels of HDL cholesterol (the healthy type) increased by 11%. High cholesterol is one of the main risk factors for heart disease. The cholesterol-lowering properties of avocado, along with its folate content, help keep your heart healthy. 5. Avocados taste great The last reason is simple’avocados are a healthy way to boost the flavour and texture of your meals. Toss chopped avocado on a salad or bowl of soup, serve guacamole as an appetizer or condiment, or try one of these healthy avocado recipes to get more healthy avocado into your diet. ‘ Avocado and Chicken Club Sandwich ‘ Avocado and Shrimp Cups ‘ Avocado Carpaccio with Wild Blueberry Cottage Cheese Mix ‘ Broiled Salmon with Avocado-Mango Salsa ‘ Classic Guacamole Don’t miss out! Sign up for our free weekly newsletters and get nutritious recipes, healthy weight-loss tips, easy ways to stay in shape and all the health news you need, delivered straight to your inbox. Filed Under: Healthy Eating Tagged With: Diet & Nutrition Secrets to Staying Healthy & Happy Spine stretch Source: Best Health Magazine, Summer 2011; Illustration by Kagan McLeodWho doesn't want better posture? This quick move can help you stand (and sit) tall with ease, as it aligns the spine. It can relieve tension in the hips and shoulders, says Margot … [Read More...] Seared Beef with Garden Vegetables Source: Cook Smart for a Healthy Heart, Reader's Digest CanadaIngredients2 garlic cloves, finely chopped1?2 teaspoon pepper1 bone-in beef sirloin steak 2 cm thick, about 1.2 kg500 g potatoes, quartered500 g green beans1 large red onion, very … [Read More...] Tropical Fruit Smoothie Ingredients1/4 cup (60 mL) fresh or canned pineapple chunks1/4 cup (60 mL) fresh mango chunks2 fresh or frozen strawberries, hulled1 cup (250 mL) plain low-fat soy milk1 Tbsp (15 mL) lime juiceDirections Blend until smooth.Serves one.Nutritions Per … [Read More...] Savoury Rosemary Squaffles Best Health Magazine, March/April 2012; Photo by Michael AlberstatIngredients1 cup (250 mL) whole-wheat flour1 cup (250 mL) all-purpose flour1 1/2 tsp (7 mL) dried rosemary (or 1 Tbsp/15 mL chopped fresh)1 tsp (5 mL) baking powder1 tsp (5 mL) grated … [Read More...] Frittata with Asparagus and Scallions Ingredients1 pound fresh, thin asparagus spears4 ounces prosciutto or bacon, thick slices with ample fat (about 4 slices)1/2 pound scallions3 tablespoons extra-virgin olive oil1/2 teaspoon coarse sea salt or kosher salt, or more to taste8 large … [Read More...] We Also Think You’ll Like: Kaitlyn Bristowe Talks Big Hair, Beyonc? and The Bachelor Botox, Fillers and General Skin Treatments: What to Know Before You Book Bloated? How to Know if it’s Irritable Bowel Syndrome What If No One Believed You Were Sick? #PhysiotherapyHelpsLives: Stroke Recovery Follow us on Social Networks ? ? ? ? ? ? Featured Content 5 ways to relieve back pain Suffering from an aching back? We dug into the latest research to find out what you can do to help relieve back pain Willow: A Natural Remedy for Pain Foods and Supplements to Ease Muscle and Joint Pain 5 autoimmune diseases affecting Canadians CONTESTS Best for Last Enter to win one of three (3) Prize Packs! Win A Holiday to Jamaica Enter for a chance to win a trip to Jamaica! Load thickbox... ABOUT BEST HEALTH Best Health is a health & wellness magazine from renowned publisher Reader’s Digest that brings an inspiring voice to today’s contemporary Canadian woman. Click here to learn about advertising opportunities. Your Privacy Rights | Customer Care | Subscribe | Sitemap Recent Posts Kaitlyn Bristowe Talks Big Hair, Beyonc? and The Bachelor Botox, Fillers and General Skin Treatments: What to Know Before You Book Bloated? How to Know if it’s Irritable Bowel Syndrome What If No One Believed You Were Sick? #PhysiotherapyHelpsLives: Stroke Recovery Don't miss a day of Best Health Sign up for our newsletters CONNECT WITH US ON SOCIAL ? ? ? ? ? ? Show Full Site Copyright © 2016 The Reader's Digest Association, Inc.");
StringBuilder nogarbage = new StringBuilder();
StringBuilder trashbin  = new StringBuilder();
StringBuilder receiver  = null;
while (s.hasNext())

    String token = s.next();

    if (isGood(token))        //implement isGood somewhere
    
        receiver = nogarbage;
    
    else
    
        receiver = trash;
    
    receiver.append(token);

System.out.println(nogarbage);
System.out.println("Trashbin contents "+trashbin);

【讨论】：

但是我正在为很多网站（500+）做这个过程，并且每个特定网站的垃圾词的标记都是不同的。如果有一个通用的解决方案或不同的打开方式，我会感到很痛苦可以更好地处理我的请求的来源？

您可以为不同的网站定义isGood以使用不同的垃圾词恕我直言。 idk 关于一个比我发布的更通用的通用解决方案。 ://

我不能为每个不同的网站制作 isGood(token)，因为每次都会在新网站中出现一些新的垃圾词。也许已经存在一个涵盖所有垃圾词的列表，我不知道不知道吗？

afaik 你可以找到停用词 ***.com/a/1218341/1737819 。在wikipedia article about stop words 上，它说所有自然语言处理工具都使用一个通用的停用词列表，实际上并非所有工具都使用这样的列表。您还可以尝试使用频率来检测标记是否是停用词nlp.stanford.edu/IR-book/html/htmledition/…，必须对每个文本进行至少两次扫描。首先扫描计算频率，然后收集好的令牌。

问题是是否有任何类型的列表，例如用于解析 html 文件的停用词？例如包含