UPDATE
I discovered that the entire domain "https://www.mots.go.th" appears to give DNS error when using urlfetchapp, not just this specific page. Perhaps this is some sort of protection against scraping?
I've been trying to use the Google App script UrlFetchApp command to extract an excel file from a uRL link on a website (which you can normally click on to download to your PC). This is the URL:
https://www.mots.go.th/images/v2022_1741001884711RmViLTIwMjVQLnhsc3g=.xlsx
However, it always give me the following error:
Error
Exception: DNS error: https://www.mots.go.th/images/v2022_1741001884711RmViLTIwMjVQLnhsc3g=.xlsx
(without using encodeURIComponent)
or
Error
Exception: DNS error: https://www.mots.go.th/images/v2022_1741001884711RmViLTIwMjVQLnhsc3g%3D.xlsx
(when using encodeURIComponent)
I am not sure why I am having this error with this particular URL. Normally it's not a problem. Here is the code I used to extract the blob in Google App Script:
function apiFetch() {
// URL part
var encodeComponent = "v2022_1741001884711RmViLTIwMjVQLnhsc3g=.xlsx"
var baseurl = "https://www.mots.go.th/images/"
var url = baseurl + encodeURIComponent(encodeComponent)
// Fetch contents from API endpoint.
const apiResponse = UrlFetchApp.fetch(url).getBlob();
}
Note that I have no idea how this URL gets the excel file or if there is some sort of automatic redirect to some place the UrlFetchApp doesn't like (like a private server) which is causing the DNS problem for GAS but not for a browser. Some people in the community have suggested that this may be the case, but there is no way of telling.
In any case, I tried the following to deal with redirect and it still gives me the same DNS error:
const apiResponse = UrlFetchApp.fetch(url, {'followRedirects': false, 'muteHttpExceptions': false});
And in my effort to figure out where the redirect URL is also failed as I still get the same DNS error:
var followedPost = UrlFetchApp.fetch(url, {'followRedirects': false, 'muteHttpExceptions': false});
Logger.log(followedPost.getHeaders()['Location']);
I even tried the code on this website: https://script.gs/get-the-redirect-location-of-a-url-using-google-apps-script/ and I still get the same DNS error
curl
works. The website isn't redirecting. It's a direct access(as can be seen withcurl
) So I think it's a bug on DNS servers used by Google apps script.