Unable to parse the value of an element from response in python?

Question

import requests
from bs4 import BeautifulSoup
s = requests.Session()
content = s.get('https://nucleus.niituniversity.in/Default.aspx').content
soup = BeautifulSoup(content,"html5lib")
print("viewState = " + str(soup.select_one("#__VIEWSTATE")["value"]))
print("UserDet = " + str(soup.select_one("#SchSel_hidCoreUserDet")["value"]))

Result :
ViewState = qdwrf3rf

KeyError : Value whoch means no value exists for userdet

But see the response when I manually login.The value for Userdet exists.

Why is the error occuring during requests with python while no problem with manual login.?

looks like you have an extra single quote at the end of your string #SchSel_hidCoreUserDet' — glls
– glls, Commented Aug 11, 2016 at 3:07

alecxe · Accepted Answer · 2016-08-11 03:24:42Z

0

No, the element with id="SchSel_hidCoreUserDet" does not have a value. Here is what I see if I open this page in the browser:

<input type="hidden" name="SchSel$hidCoreUserDet" id="SchSel_hidCoreUserDet" value="">

The value is actually empty - it is the browser who adds the value attribute with an empty value.

And, here is what I see if printing the element found by BeautifulSoup:

<input id="SchSel_hidCoreUserDet" name="SchSel$hidCoreUserDet" type="hidden"/>

In both cases, the element does not have a value.

You might be previously logged in to this site and it might remember you through cookies or the local storage. Try inspecting the source of the page in an incognito window, or a browser which you have not used to log in to this site.

answered Aug 11, 2016 at 3:24

alecxe

476k127 gold badges1.1k silver badges1.2k bronze badges

Sign up to request clarification or add additional context in comments.

10 Comments

anderson Over a year ago

See 440 line [here](view-source:nucleus.niituniversity.in/).It is assigning the value

alecxe Over a year ago

@anderson I see hidCoreUserDet.value = ""; on line 440.

anderson Over a year ago

sorry see 453 line

alecxe Over a year ago

@anderson yeah, and it would be executed only if there is the "last login data" available, see line 408. That's exactly my point. You've been logged in to this site before and that's why you see the value in the browser.

anderson Over a year ago

exactly..I have done the same.here

|

Collectives™ on Stack Overflow

Unable to parse the value of an element from response in python?

1 Answer 1

10 Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

10 Comments

Related