2
$\begingroup$

I am considering the following link:

https://myimmitracker.com/en/ca/trackers/consolidated-e-apr-tracker-express-entry-permanent-residency-application

It produces the following page with a table in browser.

enter image description here

If I use chrome inspector tools to see the URL request this page makes to ask for data, I see it uses the following:

https://myimmitracker.com/ca/trackers/consolidated-e-apr-tracker-express-entry-permanent-residency-application/cases?start=100&filter=%7B%22state%22%3A%5B%22Active%22%5D%7D&sort=%5B%5B%22xuset-kavav-casez-nypek-sybet-synyg-nocan-tyzef-tyxux%22%2C%22desc%22%5D%2C%5B%22updated%22%2C%22desc%22%5D%5D

The response "JSON" from the chrome tools is as follows (I have truncated the full response since it exceeds the character limit allowed by stackexchange):

{"values":[{"comments_count":"0 comments","username":["Sibi Abraham","case-68891"],"watched":false,"dashboard":"/en/dashboard","c_at":"Mar  4, 2019","updated":"about 24 hours","state":"Active","xodos-nymub-sovog-zolyr-kafit-finit-hyluk-cymek-bixax":"India","xidar-kiboc-feruv-vazum-dazul-noror-kemip-pagit-fixix":"FSW-Outland","xuset-kavav-casez-nypek-sybet-synyg-nocan-tyzef-tyxux":"Feb 15, 2019","xukor-mimap-cacat-dymip-dopun-kycup-tisyn-sycun-daxex":18,"xopos-kybed-picys-supot-gukab-tetyl-luzyd-lekez-gixex":"e-APR AOR","xobik-zodog-fabit-gakyk-mopip-cirof-kacav-bykaz-ryxox":"2","xecat-gatas-lekyf-vymuz-hyfan-kuban-nipyc-bicos-hixux":"United Arab Emirates"},{"comments_count":"0 comments","username":["kmb","case-68758"],"watched":false,"dashboard":"/en/dashboard","c_at":"Mar  3, 2019","updated":"3 days","state":"Active","xodos-nymub-sovog-zolyr-kafit-finit-hyluk-cymek-bixax":"India","xidar-kiboc-feruv-vazum-dazul-noror-kemip-pagit-fixix":"FSW-Outland","xuset-kavav-casez-nypek-sybet-synyg-nocan-tyzef-tyxux":"Feb 15, 2019","xukor-mimap-cacat-dymip-dopun-kycup-tisyn-sycun-daxex":18,"xopos-kybed-picys-supot-gukab-tetyl-luzyd-lekez-gixex":"e-APR AOR","xobik-zodog-fabit-gakyk-mopip-cirof-kacav-bykaz-ryxox":"3","xocoz-vubum-bifub-mitak-varab-habog-genyl-ginat-kaxyx":"Feb 15, 2019","xufol-codoz-vaban-rahuk-sobuc-tumyn-kesev-tikus-naxex":"In Progress (1)","xecat-gatas-lekyf-vymuz-hyfan-kuban-nipyc-bicos-hixux":"New Zealand"},{"comments_count":"0 comments","username":["Othmane","case-67203"],"watched":false,"dashboard":"/en/dashboard","c_at":"Feb  9, 2019","updated":"20 days","state":"Active","xodos-nymub-sovog-zolyr-kafit-finit-hyluk-cymek-bixax":"Morocco","xidar-kiboc-feruv-vazum-dazul-noror-kemip-pagit-fixix":"FSW-Outland","xuset-kavav-casez-nypek-sybet-synyg-nocan-tyzef-tyxux":"Feb  8, 2019","xukor-mimap-cacat-dymip-dopun-kycup-tisyn-sycun-daxex":25,"xopos-kybed-picys-supot-gukab-tetyl-luzyd-lekez-gixex":"e-APR AOR","xobik-zodog-fabit-gakyk-mopip-cirof-kacav-bykaz-ryxox":"3","xufol-codoz-vaban-rahuk-sobuc-tumyn-kesev-tikus-naxex":"In Progress (1)","xecat-gatas-lekyf-vymuz-hyfan-kuban-nipyc-bicos-hixux":"Morocco"},{"comments_count":"0 comments","username":["Aele","case-67548"],"watched":false,"dashboard":"/en/dashboard","c_at":"Feb 13, 2019","updated":"20 days","state":"Active","xodos-nymub-sovog-zolyr-kafit-finit-hyluk-cymek-bixax":"Nigeria","xidar-kiboc-feruv-vazum-dazul-noror-kemip-pagit-fixix":"FSW-Outland","xuset-kavav-casez-nypek-sybet-synyg-nocan-tyzef-tyxux":"Feb  8, 2019","xukor-mimap-cacat-dymip-dopun-kycup-tisyn-sycun-daxex":25,"xopos-kybed-picys-supot-gukab-tetyl-luzyd-lekez-gixex":"e-APR AOR","xobik-zodog-fabit-gakyk-mopip-cirof-kacav-bykaz-ryxox":"1","xubot-hatel-matec-vycid-fycel-gugyp-tesag-ludyd-lexux":"Copy of the biographical data page of a passport or travel document, Digital Photo, Educational Credential Assessment (ECA) report, Language test results, Medical examination confirmation, Police certificates, Proof of funds, Proof of studies, Proof of work experience","xecat-gatas-lekyf-vymuz-hyfan-kuban-nipyc-bicos-hixux":"Nigeria"},{"comments_count":"0 comments","username":["Prashanth Gopenathan","case-67499"],"watched":false,"dashboard":"/en/dashboard","c_at":"Feb 13, 2019","updated":"20 days","state":"Active","xodos-nymub-sovog-zolyr-kafit-finit-hyluk-cymek-bixax":"India","xidar-kiboc-feruv-vazum-dazul-noror-kemip-pagit-fixix":"CEC","xuset-kavav-casez-nypek-sybet-synyg-nocan-tyzef-tyxux":"Feb  8, 2019","xukor-mimap-cacat-dymip-dopun-kycup-tisyn-sycun-daxex":25,"xopos-kybed-picys-supot-gukab-tetyl-luzyd-lekez-gixex":"e-APR AOR","xecat-gatas-lekyf-vymuz-hyfan-kuban-nipyc-bicos-hixux":"Canada"},{"comments_count":"2 comments","username":["Belly_BearS","case-65222"],"watched":false,"dashboard":"/en/dashboard","c_at":"Jan 15, 2019","updated":"22 days","state":"Active","xodos-nymub-sovog-zolyr-kafit-finit-hyluk-cymek-bixax":"Pakistan","xidar-kiboc-feruv-vazum-dazul-noror-kemip-pagit-fixix":"FSW-Outland","xuset-kavav-casez-nypek-sybet-synyg-nocan-tyzef-tyxux":"Feb  8, 2019","xukor-mimap-cacat-dymip-dopun-kycup-tisyn-sycun-daxex":25,"xopos-kybed-picys-supot-gukab-tetyl-luzyd-lekez-gixex":"e-APR AOR","xecat-gatas-lekyf-vymuz-hyfan-kuban-nipyc-bicos-hixux":"Pakistan"},{"comments_count":"1 comments","username":["adityaljoshi","case-67128"],"watched":false,"dashboard":"/en/dashboard","c_at":"Feb  8, 2019","updated":"25 days","state":"Active","xodos-nymub-sovog-zolyr-kafit-finit-hyluk-cymek-bixax":"India","xidar-kiboc-feruv-vazum-dazul-noror-kemip-pagit-fixix":"FSW-Outland","xuset-kavav-casez-nypek-sybet-synyg-nocan-tyzef-tyxux":"Feb  8, 2019","xukor-mimap-cacat-dymip-dopun-kycup-tisyn-sycun-daxex":25,"xopos-kybed-picys-supot-gukab-tetyl-luzyd-lekez-gixex":"e-APR AOR","xobik-zodog-fabit-gakyk-mopip-cirof-kacav-bykaz-ryxox":"4","xecat-gatas-lekyf-vymuz-hyfan-kuban-nipyc-bicos-hixux":"India"},{"comments_count":"0 comments","username":["maestro","case-67155"],"watched":false,"dashboard":"/en/dashboard","c_at":"Feb  9, 2019","updated":"25 days","state":"Active","xodos-nymub-sovog-zolyr-kafit-finit-hyluk-cymek-bixax":"Cameroon","xidar-kiboc-feruv-vazum-dazul-noror-kemip-pagit-fixix":"FSW-Outland","xuset-kavav-casez-nypek-sybet-synyg-nocan-tyzef-tyxux":"Feb  8, 2019","xukor-mimap-cacat-dymip-dopun-kycup-tisyn-sycun-daxex":25,"xopos-kybed-picys-supot-gukab-tetyl-luzyd-lekez-gixex":"e-APR AOR","xobik-zodog-fabit-gakyk-mopip-cirof-kacav-bykaz-ryxox":"1","xecat-gatas-lekyf-vymuz-hyfan-kuban-nipyc-bicos-hixux":"United States of America"},{"comments_count":"0 comments","username":["Navraj Singh Dhindsa","case-67135"],"watched":false,"dashboard":"/en/dashboard","c_at":"Feb  8, 2019","updated":"25 days","state":"Active","xodos-nymub-sovog-zolyr-kafit-finit-hyluk-cymek-bixax":"India","xidar-kiboc-feruv-vazum-dazul-noror-kemip-pagit-fixix":"CEC","xuset-kavav-casez-nypek-sybet-synyg-nocan-tyzef-tyxux":"Feb  8, 2019","xukor-mimap-cacat-dymip-dopun-kycup-tisyn-sycun-daxex":25,"xopos-kybed-picys-supot-gukab-tetyl-luzyd-lekez-gixex":"e-APR AOR","xobik-zodog-fabit-gakyk-mopip-cirof-kacav-bykaz-ryxox":"1","xufol-codoz-vaban-rahuk-sobuc-tumyn-kesev-tikus-naxex":"Not Applicable","xubot-hatel-matec-vycid-fycel-gugyp-tesag-ludyd-lexux":"Copy of the biographical data page of a passport or travel document, Digital Photo, Medical examination confirmation, Police certificates, Proof of funds, Proof of work experience","xecat-gatas-lekyf-vymuz-hyfan-kuban-nipyc-bicos-hixux":"Canada","xiden-fadin-lasof-dulaz-mavuk-fyzal-myhuv-humyc-tyxix":"Feb  8, 2019"},{"comments_count":"0 comments","username":["chemenger","case-67130"],"watched":false,"dashboard":"/en/dashboard","c_at":"Feb  8, 2019","updated":"25 days","state":"Active","xodos-nymub-sovog-zolyr-kafit-finit-hyluk-cymek-bixax":"India","xidar-kiboc-feruv-vazum-dazul-noror-kemip-pagit-fixix":"FSW-Outland","xuset-kavav-casez-nypek-sybet-synyg-nocan-tyzef-tyxux":"Feb  8, 2019","xukor-mimap-cacat-dymip-dopun-kycup-tisyn-sycun-daxex":25,"xopos-kybed-picys-supot-gukab-tetyl-luzyd-lekez-gixex":"e-APR AOR","xecat-gatas-lekyf-vymuz-hyfan-kuban-nipyc-bicos-hixux":"United States of America"}],"cases_count":11534,"total_cases_count":16666}

How can I use Wolfram Language's HTTP Requests & Responses to get the response from the URL request the page makes when loading new data on scrolling? Basically I want to get the above JSON response using the Mathematica's HTTP functionality?

Improve this question
$\endgroup$
1
  • $\begingroup$ Have you tried simply Importing the url? Otherwise, look at URLRead. $\endgroup$ Commented Mar 6, 2019 at 8:01

1 Answer 1

Reset to default
7
$\begingroup$

It looks like this particular request requires you to set a header or two before it will respond the same way it does in the browser. In particular, it looks like it needs you to set the x-requested-by, Accept, and Referer headers. (I discovered this by simply whack-a-moling the headers to see which will give the correct response).

I initially took the URL and used URLParse in order to get a URL association, to which I then added the Headers.

Import[URLRead[<|"Scheme" -> "https", "User" -> None, 
   "Domain" -> "myimmitracker.com", "Port" -> None, 
   "Path" -> {"", "ca", "trackers", 
     "consolidated-e-apr-tracker-express-entry-permanent-residency-\
application", "cases"}, 
   "Query" -> {"start" -> "100", 
     "filter" -> "{\"state\":[\"Active\"]}", 
     "sort" -> 
      "[[\"xuset-kavav-casez-nypek-sybet-synyg-nocan-tyzef-tyxux\",\"\
desc\"],[\"updated\",\"desc\"]]"}, "Fragment" -> None,
   "Headers" -> <|
     "Accept" -> "application/json;charset=UTF-8",
     "referer" -> 
      "https://myimmitracker.com/en/ca/trackers/consolidated-e-apr-\
tracker-express-entry-permanent-residency-application",
     "x-requested-with" -> "XMLHttpRequest"
     |>|>], "RawJSON"]

This returns an association in the form of the JSON that you requested.

Improve this answer
$\endgroup$
7
  • $\begingroup$ Can you please explain if I build the url using your association, URLBuild[<|...|>] then the resulting url is the same as that from the browser. But now if I do URLRead from the build url, URLRead[URLBuild[<|...|>]] I get a different HTTPResponse. Why can't it understand from the build url, the headers and other things? $\endgroup$ Commented Mar 6, 2019 at 9:11
  • $\begingroup$ @user13892 The headers won't stick around if you do URLBuild on the full association - you need to pass the Headers association as a second argument to URLRead. $\endgroup$ Commented Mar 6, 2019 at 9:15
  • $\begingroup$ Also what happens if I set "start" to "0" in the query. Why can't it import as RAWJSON in this case? Clearly body of the resulting HTTPResponse show that is its indeed JSON string, URLRead[<|...|>]["Body"]. $\endgroup$ Commented Mar 6, 2019 at 9:28
  • $\begingroup$ Is there an error message? $\endgroup$ Commented Mar 6, 2019 at 10:04
  • $\begingroup$ Yes two error messages from Import: Import::utf8expcontinuation: Input is not a valid UTF8 byte sequence. 116 is expected to be a continuation byte. and Import::jsonhintposandchar: An error occurred near character 't', at line 1:57844 $\endgroup$ Commented Mar 6, 2019 at 10:09

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.