Allow more bots on robots.txt

Feb

2009

Id very much like the robots.txt page on all of our forums to be altered. as we dont have ftp access, perhaps admin of forumotion could change the text,

currently, we are only allowing google, msn and yahoo to crawl and index our forums, and blocking dozens of other bots including alexa and ask.

would you please agree to this suggestion so we can perhaps get crawled by more search engines?

Thanks

1 oleander, on 03/04/2009 at 16h59

has this been done yet?

2 Jaffa909, on 28/08/2010 at 16h25

I votes yes becuase our forums get more visits and more people that join our forums they might join forumotion! But if they allowed all search engine bots and you submitted to like one of those websites where they send like 500 search bots your forum might not be able to handle 500+ bots indexing andn scanning all the posts at one might overload it! but very nice idea

3 StoneRaven, on 28/08/2010 at 16h37

I vote yes. The more we can get crawled the more we show up on more search engines which ='s more real visitors. I don't see 500+ bots at one time though maybe 3 or 4 at a time.

4 Walshy95, on 28/08/2010 at 20h58

Voted yes. Great idea. Would be a great way of publicity.

5 Cadence, on 11/09/2011 at 14h55

good!! I have bots on all the time (Yahoo) and i wanna make my user a bot so when i log in chatbox it will chat to anyone there... how do i add a bot user to chat box?????

6 LoveStudio, on 11/09/2011 at 15h33

I like this héhé

7 Cadence, on 13/09/2011 at 02h44

Well, bots are from "official" websites. Usually they make bots so they're website may come back with Info from your forum, brought to theres so they may add to it...You can dis bots, most likely, I wouldn't. More searches CLOSE or FAR OFF searches feature your weblink, so its a far trade Wink

bots are useful, but if you seem to not like them I guess not...Oh and the reason you cannot edit that is because, the CSS is the theme, look, style and sorts of functions, it does not add bots. If it were a bot to be added, most likely to be added in the GUEST section and not do any data modifiers left behind, so I recommend not to make/add a bot, you must know your FULL WAY throw coding, plus, it wouldn't be are your website!! 0.0 it go through others, and you cannot see the data...I hope this helped...

8 !_NICK_!, on 13/09/2011 at 04h02

You could edit a template and make it look like that's how many members are on. I don it once, but did not keep it. I just previewed it though, and didn't publish it.

9 brandfiesta, on 25/09/2011 at 09h33

I totally agree, I understand the issue about overload but come on. Im not sure if it is the culprit but this summer I was placing high in search "paid to post forum" now I cannot even find myself on the first page anymore.

Please do not put this in robots.txt

Disallow: /

High Appreciated,
Team BrandFiesta
Voted #1 PTP Forum

10 hanzboyz, on 25/09/2011 at 10h08

Until now my forum havent show yahoo! bots. only showing bing & google.. T_T

11 True Blue, on 05/10/2011 at 16h39

Hello thealbertross

I'm yes on this one, as you've already stated placing a robot notpad in the root directory would work, my forum already has the "noodp" robots written into it by Forumotion.com so I will receive bots from Yahoo!, and MSN as well as the first Google one I've spotted today.

Sometime problems do arise with different crawlers that why its better to change "noodp" to "noodp,noydir" thumright

Liz