Python请求与curl

Posted 2021-03-30

我正在尝试使用具有特定标头和正文有效负载的POST从URL中提取数据。使用curl或像Advanced Rest Client这样的工具，我能够获得包含我想要的数据的JSON和漂亮的代码200.但是，如果我尝试使用python请求执行相同的请求，它将无法工作。我得到一个代码405和坏json。

这些是我正在使用的配置：

curl -H "Content-Type: application/json" --data @payload.json -X POST https://www.walmart.com/terra-firma/fetch?rgs=REVIEWS_MAP

json文件payload.json，如下所示：

{"productId":"10VJLBWPW82T","paginationContext":{"sort":"relevancy","page":3}}

这是python3代码：

import requests

url = "http://www.walmart.com/terra-firma/fetch?rgs=REVIEWS_MAP"

head = {'Content-type':'application/json'}
payload = {"productId":"10VJLBWPW82T","paginationContext":{"sort":"relevancy","page":3}}

rep = requests.post(url, headers=head, data=payload)

有人知道如何解释这种差异吗？

答案

两件事，首先使用请求中的'json'参数而不是'data'来发布json。其次对这个特定的网页使用https而不是http协议。

import requests

url = "https://www.walmart.com/terra-firma/fetch?rgs=REVIEWS_MAP"
head = {'Content-type':'application/json'}
payload = {"productId":"10VJLBWPW82T","paginationContext":{"sort":"relevancy","page":3}}
rep = requests.post(url, headers=head, json=payload)
print (rep.status_code)
print (rep.json())

输出：

200
{'status': 'OK', 'header': {'headerAttributes': {'timing': {'total': {'threads': 3, 'orchestrate': 1, 'delay': 0, 'elapsed': 30, 'futures': 37}}, 'tenant': {'WM_BU_ID': '0', 'WM_VERTICAL_ID': '0', 'WM_LOCALE_ID': 'en_US', 'WM_MART_ID': '0', 'WM_CHANNEL_ID': '0'}, 'server': {'ver': '84.0.2', 'env': 'prod-a', 'dc': 'prod-dfw8', 'name': 'app-273848643-9-280204056', 'sha': 'bd15b234fd7d120a2aa489becec41385c5e34719'}, 'context': {'browser': 'CRAWLER', 'device': 'UNSPECIFIED', 'topTxId': 'b0886fbe-1e8-161e8486db3000', 'access': True}}}, 'errors': [], 'payload': {'products': {}, 'reviews': {'10VJLBWPW82T': {'percentageFourCount': 2, 'percentageFiveCount': 91, 'ratingValueTwoCount': 1, 'customerReviews': [{'showRecommended': True, 'photos': [], 'userNickname': 'murrie56', 'positiveFeedback': 5, 'negativeFeedback': 0, 'userAttributes': {}, 'rating': 5.0, 'authorId': '6b61e60fc8b3f358b84858dbe0fd50f7', 'reviewTitle': 'LOVE IT! LOVE IT! LOVE IT!', 'reviewText': "I've been using this ever since it was introduced back in the late 70's, I was in High School. I still get compliments on it. I'll never wear anything else as long as it's available. I buy in bulk.", 'videos': [], 'reviewSubmissionTime': '11/30/2014', 'reviewId': '40087713', 'recommended': True}, {'showRecommended': False, 'photos': [], 'userNickname': 'Tony', 'positiveFeedback': 0, 'negativeFeedback': 0, 'userAttributes': {}, 'rating': 5.0, 'authorId': 'ce2dbcdd5602bd2fde14f2d40bfc3217', 'reviewText': 'my girl friend loved it', 'videos': [], 'reviewSubmissionTime': '12/18/2017', 'reviewId': '186608534'}, {'showRecommended': True, 'photos': [], 'userNickname': 'SandyC', 'positiveFeedback': 5, 'badges': [{'id': 'VerifiedPurchaser', 'badgeType': 'Custom', 'contentType': 'REVIEW'}], 'userAge': '45 - 54', 'negativeFeedback': 0, 'userAttributes': {'Usage': 'Every day', 'Age': '45 - 54', 'Ownership': 'Longer than one year', 'Gender': 'Female'}, 'rating': 5.0, 'authorId': '6a0c52dc6fc5c63ccff7bde8ad1f3e9a', 'reviewTitle': 'Have been using this perfume for years', 'reviewText': 'Love it!', 'userLocation': 'Barnwell, SC', 'userGender': 'Female', 'reviewSubmissionTime': '12/21/2013', 'reviewId': '19488031', 'videos': [], 'recommended': True}, {'showRecommended': True, 'photos': [], 'userNickname': 'corkmiester', 'positiveFeedback': 6, 'negativeFeedback': 1, 'userAttributes': {}, 'rating': 5.0, 'authorId': '95edbfc402035dd0237b48ef92a5401f', 'reviewTitle': 'Lifetime Favorite', 'reviewText': "I've used this product since my teenage years, and I'm happy to say that the scent has never varied, never changed. It smells the same as it did then and is my all-time favorite perfume. I love the staying power of this product. An application in the morning lasts all day. I've had many a compliment over the years when wearing this and the affordable price makes it that much better. I hope they never stop making this stuff. I'll be a happy camper as long as I can find it. :)", 'videos': [], 'reviewSubmissionTime': '12/10/2014', 'reviewId': '41086799', 'recommended': True}, {'showRecommended': True, 'photos': [], 'userNickname': 'Clydesdales', 'positiveFeedback': 4, 'negativeFeedback': 0, 'userAttributes': {}, 'rating': 5.0, 'authorId': '0b1608e40c638ce49215bfba6624e27d', 'reviewTitle': 'GL McC', 'reviewText': 'Have used this cologne for years and I can always get it at Walmart. The other places I have gotten it before no longer carry the product. Thanks Walmart for carrying this item!!', 'videos': [], 'reviewSubmissionTime': '3/30/2016', 'reviewId': '84475295', 'recommended': True}], 'overallRatingRange': 5.0, 'totalReviewCount': 45, 'activeFilters': {}, 'percentageTwoCount': 2, 'roundedAverageOverallRating': 4.8, 'recommendedPercentage': 95, 'ratingValueFourCount': 1, 'pagination': {'total': 43, 'currentSpan': '11-15', 'next': {'num': 0, 'url': 'sort=relevancy&page=4', 'gap': False, 'active': False}, 'pages': [{'num': 1, 'url': 'sort=relevancy&page=1', 'gap': False, 'active': False}, {'num': 2, 'url': 'sort=relevancy&page=2', 'gap': False, 'active': False}, {'num': 3, 'url': 'sort=relevancy&page=3', 'gap': False, 'active': True}, {'num': 4, 'url': 'sort=relevancy&page=4', 'gap': False, 'active': False}, {'num': 5, 'url': 'sort=relevancy&page=5', 'gap': False, 'active': False}, {'num': 6, 'url': 'sort=relevancy&page=6', 'gap': False, 'active': False}, {'num': 0, 'gap': True, 'active': False}, {'num': 9, 'url': 'sort=relevancy&page=9', 'gap': False, 'active': False}], 'previous': {'num': 0, 'url': 'sort=relevancy&page=2', 'gap': False, 'active': False}}, 'percentageOneCount': 2, 'ratingValueFiveCount': 41, 'ratingValueThreeCount': 1, 'ratingValueOneCount': 1, 'activeSort': 'relevancy', 'percentageThreeCount': 2, 'averageOverallRating': 4.7778}}}}