you have to put robots.txt file in the root folder of the wordpress installation. you can use ftp software like filezilla to upload this file.
read http://www.robotstxt.org/ for details.
adding robots.txt is not mandatory. You may also use robots meta plugin which will help do much of the function of robots.txt without needing to upload this file.
http://ww.wp.xz.cn/extend/plugins/robots-meta/
Google has no problem indexing http://psdstuff.vndv.com/wordpress/?p=222
However, having url like http://psdstuff.vndv.com/wordpress/bla-bla-bla/ is more search engine and human friendly. no needto have .html at the end of the page
one more question,,, google says my robots.txt file should be in my root that is http://www.psdstuff.vndv.com/ but my wordpress installation is at http://psdstuff.vndv.com/wordpress/ so where do i put it??? google has no options to change the url of the file,, it only wants it in my root….though it doesnt detect it even when i put it in root 🙁
also kindly tell a plugin that would resolve this issue
http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449
The robots.txt file must reside in the root of the domain and must be named “robots.txt”.
You need a robots.txt file only if your site includes content that you don’t want search engines to index. If you want search engines to index everything in your site, you don’t need a robots.txt file (not even an empty one).
ok i got it… one more thing…. when i post something, i give it some tags, do these tags serve as meta tags??? that google can detect?
