Indexing by search engines - detrimental?

[d*l*b]

On several occasions I have searched for supplies (legit, but also possibly questionable products such as solvents) and have found that the Nexus and the supplier are the only results.

I wonder if SE indexing some areas of the site may be counterproductive and may lead to stricter checks by suppliers and the creation of restrictions on supply in the long run with many products.

 

[Jorkest]

very interesting indeed..

 

[obliguhl]

I think the supplier subforum should be made private.

 

[spacedinvader]

I think if the admin of the site was on the ball he would have implemented a proper robots.txt file already and this would not be a problem.

It's okay though, he had to earlier be courteously reminded to stop giving away dmt-nexus.me in the referrer. Ever notice how now when you link to a site, it first takes you to, "http://www.anoniem.org/?"!

See:

http://www.dmt-nexus.me/robots.txt

Then see:

The Web Robots Pages

Literally to 'fix' this 'problem' a single slash needs to be added to the second line:

current configuration:
User-agent: *
Disallow:

fixed configuration:
User-agent: *
Disallow: /


If the admin does not want to block every directory from being indexed, then explicitly add which directly needs to be blocked. (he likes it when the google stops by).

 

[balaganist]

very good point... nexus needs to stay underground!!

 

[The Traveler]

I think it's important that people can find the DMT-Nexus in search engines, in the end many members came by those means. So changing the robots.txt is not an option.

As for the suppliers section, I made it hidden for non-members (including search engines), you now have to be logged in to see that section. It will take some time though before the cache of the search engines are purged.


Kind regards,

The Traveler

 

[obliguhl]

Thanks!!! very nice to see!