Return redirect http status for bots/crawlers
G
Gonen Dukas
*** PROBLEM:
Upgraded to "hobby" plan to have referrer hidden. Once turned on, sharing a short link on WhatsApp stopped scraping the image and text from the target website (in facebook it does work ok).
*** CAUSE OF PROBLEM:
Once referrer is hidden, instead of returning http status 302, a page with a script does the redirect.
That's ok for humans using browsers. WhatsApp scraper does not run any scripts. It only scans the static response returned. Target website is not scanned by the scraper.
*** POSSIBLE SOLUTION:
Upgrade to "pro" plan and manually set in each link created og:image and og:title.
Extra money and a lot of extra work just to have referrer hidden.
*** SUGGESTED SOLUTION:
In case of bots/scrapers, even if referrer hidden is enabled, return redirect status (302, 301 or whatever is the setting) - just like the setting is disabled.
This super easy solution will allow to hide source of human traffic while not disturbing the work of scrapers like WhatsApp and Facebook have.
VID20251113210737