Bingbot in terminal logspout

Technical Help
#1

Hello,

I am getting bingbot logs like this in logspout. whats wierd is that it was not happening in a previous install.

quantrocket-houston-1|40.77.167.51 - - [07/Nov/2024:17:28:55 +0000] "GET /~i/text/upload?PPT=bogglesworldesl+cloze+verb+answers.pdf HTTP/1.1" 404 0 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36"
quantrocket-houston-1|40.77.167.51 - - [07/Nov/2024:17:28:56 +0000] "GET /@w/chap/slug?RTF=solutions+manual+rizzoni+electrical+5th+edition.pdf HTTP/1.1" 404 0 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36"
quantrocket-houston-1|40.77.167.51 - - [07/Nov/2024:17:28:57 +0000] "GET /=a/text/niche?MD=lister+petter+lpa+lpw+lpwt+lpws+lpwg+alpha+series+workshop+s.pdf HTTP/1.1" 404 0 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36"

Is there anyway to stop this? I installed on digital ocean and did use username and password in env.

Thanks

Geo

#2

The approach used in the cloud installation tutorial creates a public IP for your droplet, with username/password protection. Because the IP is public, bots and scanners can try to access your droplet, but they will fail without the username and password. In your output, Houston (which uses Nginx as its web server) has responded with 404 Not Found; if the bot happened to guess a real URL, Houston would respond with 401 Not Authorized.

Your options are:

  1. Ignore the bots. They clutter the logs a bit but can't access anything because they don't know your credentials. The bots will probably move on after a bit. As long as the volume of requests doesn't become so high that the requests themselves bog down your system (called a Denial of Service attack, and typically targeted at major websites, not random DigitalOcean droplets), you're fine.
  2. Redeploy QuantRocket behind a VPN as mentioned near the top of the cloud installation tutorial. This is more complicated and so is not listed as the standard cloud installation approach, but some QuantRocket users go this route.

Using a public IP with username/password can be compared to having a door that faces a public street but is locked. Random people can knock on the door but can't get in. Using a VPN is more like having a secret entrance so random people can't even knock.

#3

Thank You. This was really helpful. My logs were cluttered and scrolling too fast to read!

Just FYI, I relinquished the reserved public IP from digital ocean and reassigned the standard random issued one to domain. That actually stopped it (for now) . I guess to use your analogy , I am not in a public street as much but in a side alley now :slight_smile: