Pretty sure I set mine to not allow crawling, as I generally do anytime I can. I like to pretend that I have some modicum of privacy on the web, even though, having worked in computer security for years, I know that there is no such thing as web privacy...LOL