How To View Hidden Directories In A Website Using Robots.txt

Many a times in hacking you need to know the web directories.
a major hole in this security is the Robots.txt
google-bot

now you will aks
What is Robots.txt ?

Robot.txt is the file in the [ wwwroot ]of a server to define the Bots the Functionality on the website.

There are many Bots on the internet like the most famous are Google Search Engine BOT aka Google Spider , Yahoo Search Engine Bot and many others.
What robots.txt do is that it gives orders to the Bot on how to spider on the website

Now you may ask what’s the use of Robots.txt file.

see it is used by webmasters to add functionality to the upcoming bots on there website and also to hide the directories in the website and where the Bot should not go and spider.

Analyzing Robots.txt For Hacking Stuff

Well its really simple, the first question you would ask is Where is robots.txt Located ?
The answer is its in the [ WWWROOT ].

its in the main Directory.

Lets take the example

http://www.masti-cafe.blogspot.com/robots.txt

Go Ahead and type it in the Address bar of your Browser then what do you see,

Do you see that , this is the Robots.txt for the masti-cafe website Now lets first Analyze this Robots.txt

First Line :-
User-agent: Mediapartners-Google

This means that the above statements are given for the Google Search Engine Bot i.e. Google Spider.

Second Line :-
Disallow:

This mean that nothing is disallowed to the Google Bot, Remember these Orders are given to the Google bot only not other bots.

Third Line :-
User-agent: *

This means that now all the bots coming to the blog will follow these rules.note that previous rules were for only Google Bot.

Fourth Line :-
Disallow: /search

This means that all the bots will not spider the files under the directory /search in the following Blog.

Fifth Line :-
Sitemap: http://masti-cafe.blogspot.com/feeds/posts/default?orderby=updated

This is basically my blogs sitemap. Not very important.