Post Request Giving Empty Results

I've written some code in python using POST request to fetch specific data from a webpage. However, when I run it, I get nothing as result except for a blank console. I've tried to

Solution 1:

You are not sending the values in a POST body, params sets the URL query parameters. Use data instead:

response = requests.post(
    "https://www.optigura.com/product/ajax/details.php",
    data=payload, 
    headers=headers)

You may need to set a referrer header (add 'Referer': 'https://www.optigura.com/uk/product/gold-standard-100-whey/' to your headers dictionary), and use a session object to capture and manage cookies (issue a GET request to https://www.optigura.com/uk/product/gold-standard-100-whey/ first).

With a little experimentation I noticed that the site also demands that the X-Requested-With header is set before it'll respond with the contents.

The following works:

with requests.session():
    session.get('https://www.optigura.com/uk/product/gold-standard-100-whey/')
    headers = {
        'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.81 Safari/537.36',
        'Referer': 'https://www.optigura.com/uk/product/gold-standard-100-whey/',
        'X-Requested-With': 'XMLHttpRequest'
    }
    response = session.post(
        "https://www.optigura.com/product/ajax/details.php",
        data=payload, headers=headers)

The response comes as JSON data:

data = response.json()

Solution 2:

You should try below request structure:

• Data to send:

  data = {'opt': 'flavor', 'opt1': '207', 'opt2': '47', 'ip': 105}
  

• Headers:

  headers = {'X-Requested-With': 'XMLHttpRequest'}
  

• URL:

  url = 'https://www.optigura.com/product/ajax/details.php'

• Also you need to get cookies, so requests.session() is required:

  s = requests.session()
  r = s.get('https://www.optigura.com/uk/product/gold-standard-100-whey/')
  cookies = r.cookies
  

Complete request:

response = s.post(url, cookies=cookies, headers=headers, data=data)

Now you can get required piece of HTML as

print(response.json()['info2'])

Output:

'<ulclass="opt2"><liclass="active"><label><inputtype="radio"name="ipr"value="1360"data-opt-sel="47"checked="checked" /> Delicious Strawberry - <spanclass="green">In Stock</span></label></li><li><label><inputtype="radio"name="ipr"value="1356"data-opt-sel="15"  /> Double Rich Chocolate - <spanclass="green">In Stock</span></label></li><li><label><inputtype="radio"name="ipr"value="1169"data-opt-sel="16"  /> Vanilla Ice Cream - <spanclass="green">In Stock</span></label></li></ul>'

Then you can use lxml to scrape flavor values:

from lxml import html

flavors = response.json()['info2']
source = html.fromstring(flavors)

[print(element.replace(' - ', '').strip()) for element in source.xpath('//label/text()[2]')]

Output:

Delicious Strawberry
Double Rich Chocolate
Vanilla Ice Cream