2021年北京积分落户名单公布了，爬了两个多小时得到了所有数据，竟然有惊人的发现（附源码）

Posted 2021-08-10 万猫学社

2021年北京积分落户名单公布了，手痒痒就写了一段Java代码，运行了两个多小时，终于到了所有数据，如下截图：

本着“Talk is cheap, Show me the code.”的原则，先看一下源码。

源码

落户实体类

先写一个落户实体类，便于储存和分析。

    @Setter
    @Getter
    static class Person {

        private int id;
        private String number;
        private String name;
        private int year;
        private int month;
        private String company;
        private double totalScore;
        private double[] detailScore;
    }

获取落户名单

获取落户名单的Ajax请求返回的居然是html，想法比较惊奇。直接写个正则表达式，提取想要的数据。

    private final static Pattern LIST_PATTERN = Pattern.compile(
            "<tr>[^<]*?<td[^>]*?>(\\\\S*?)</td>[^<]*?<td[^>]*?>(\\\\S*?)</td>[^<]*?<td[^>]*?>(\\\\d+)\\\\-(\\\\d+)</td>[^<]*?<td[^>]*?>(\\\\S*?)</td>[^<]*?<td[^>]*?>(\\\\S*?)</td>[^<]*?<td[^>]*?>[^<]*?<a[\\\\s\\\\S]*?οnclick=\\"showDetails\\\\('(\\\\d+)'\\\\)\\">查看</a>[^<]*?</td>[^<]*?</tr>");

    private static List<Person> findPersonList() throws InterruptedException {
        String url = "http://fuwu.rsj.beijing.gov.cn/jf2021integralpublic/settlePerson/tablePage";
        List<Person> personList = new ArrayList<>();
        for (int page = 0; page <= 6040; page += 10) {
            Map<String, String> params = new HashMap<>();
            params.put("name", "");
            params.put("rows", "10");
            params.put("page", Integer.toString(page));
            String result = HttpUtils.doPost(url, params);
            Matcher matcher = LIST_PATTERN.matcher(result);
            while (matcher.find()) {
                Person person = new Person();
                person.setNumber(matcher.group(1));
                person.setName(matcher.group(2));
                person.setYear(Integer.parseInt(matcher.group(3)));
                person.setMonth(Integer.parseInt(matcher.group(4)));
                person.setCompany(matcher.group(5));
                person.setTotalScore(Double.parseDouble(matcher.group(6)));
                person.setId(Integer.parseInt(matcher.group(7)));
                personList.add(person);
            }
            log.info("page: {} ", page);
            Thread.sleep(1000);
        }
        return personList;
    }

获取积分详情

积分详情的Ajax请求返回也是HTML，直接写10个正则表达式，提取想要的数据。

    private final static Pattern[] DETAIL_PATTERN_ARRAY = {
            Pattern.compile("合法稳定就业</td>[^<]*?<td[^>]*?>([\\\\d\\\\.\\\\-]+)"),
            Pattern.compile("合法稳定住所</td>[^<]*?<td[^>]*?>([\\\\d\\\\.\\\\-]+)"),
            Pattern.compile("教育背景</td>[^<]*?<td[^>]*?>([\\\\d\\\\.\\\\-]+)"),
            Pattern.compile("扣除取得学历（学位）期间累计的居住及就业分值</td>[^<]*?<td[^>]*?>([\\\\d\\\\.\\\\-]+)"),
            Pattern.compile("创新创业</td>[^<]*?<td[^>]*?>([\\\\d\\\\.\\\\-]+)"),
            Pattern.compile("职住区域</td>[^<]*?<td[^>]*?>([\\\\d\\\\.\\\\-]+)"),
            Pattern.compile("纳税</td>[^<]*?<td[^>]*?>([\\\\d\\\\.\\\\-]+)"),
            Pattern.compile("年龄</td>[^<]*?<td[^>]*?>([\\\\d\\\\.\\\\-]+)"),
            Pattern.compile("荣誉表彰</td>[^<]*?<td[^>]*?>([\\\\d\\\\.\\\\-]+)"),
            Pattern.compile("守法记录</td>[^<]*?<td[^>]*?>([\\\\d\\\\.\\\\-]+)"),
    };

    private static void enrichPersonList(List<Person> personList) throws InterruptedException {
        String url = "http://fuwu.rsj.beijing.gov.cn/jf2021integralpublic/settlePerson/settlePersonDetails";
        for (int i = 0; i < personList.size(); i++) {
            Person person = personList.get(i);
            Map<String, String> params = new HashMap<>();
            params.put("id", Integer.toString(person.getId()));
            String result = HttpUtils.doPost(url, params);
            double[] detailScore = new double[DETAIL_PATTERN_ARRAY.length];
            for (int j = 0; j < DETAIL_PATTERN_ARRAY.length; j++) {
                Matcher matcher = DETAIL_PATTERN_ARRAY[j].matcher(result);
                if (matcher.find()) {
                    detailScore[j] = Double.parseDouble(matcher.group(1));
                } else {
                    log.error("index: {}\\n{}", j, result);
                }
            }
            person.setDetailScore(detailScore);
            log.info("person count: {} / {}", i, personList.size());
            Thread.sleep(1000);
        }
    }

数据分析

现在已经有很多统计和分析，比如：年龄分布、公司排名，都已经烂大街了，一搜就能搜到，我们来看看不一样的。

有163人没上过大学，其中有19人年薪超过65万，占比11.65%；有5882人上了大学，其中有1476人年薪超过65万，占比25.09%。所以，要想获得更好的生活条件和境遇，需要更高的学历。