Article Read. You Can Find All Kind of Articles

Home | Search Article

Search

-- All Categories --Auto and TrucksEducationFamilyFood and DrinkGadgets and GizmosHealthHobbiesHome ImprovementKids and TeensLegal MattersParentingPets and AnimalsRecreation and SportsSite PromotionTravel and LeisureWeb DevelopmentWomenWriting
Advanced Search

• Auto and Trucks
• Education
• Family
• Food and Drink
• Gadgets and Gizmos
• Health
• Hobbies
• Home Improvement
• Kids and Teens
• Legal Matters
• Parenting
• Pets and Animals
• Recreation and Sports
• Site Promotion
• Travel and Leisure
• Web Development
• Women
• Writing

---

Are you familiar with the Robot Exclusion Standard? This is a way whereby you can inform search engines and other automated spiders (programs which scan your web sites for information) what pages and directories are not to be examined. The standard is precisely defined and a little obscure, and while not overly complicated it can be difficult to understand.There is help available, believe it or not. That obscure file called "robots.txt" no longer needs to be hand-edited. No longer do you need to type in server path names, remember robot and spider names and enter strange symbols which dont seem to make any sense.Okay, heres a quick review. The "robots.txt" file simply lists the spiders that you want to prevent from listing all or part of your site. Along with each spider is the list of directories and files that it is not allowed to examine. Wildcards are allowed.Its not very difficult but it is time consuming if you have lots of robots and files to specify. In addition, its easy to misspell something and fali to achieve your goal.Now, someone has created a nice little piece of shareware called RoboGen, and its not very expensive considering the benefits that you will receive.I stumbled across this entirely by accident, since it never occurred to me that someone would write a program such as this. I downloaded the 30 day demonstration and quickly discovered how wonderful it is. Robogen has already saved me hours and hours of work.Okay, heres how it works. You enter your FTP login information and log into your site. You then select robots by name in the left pane, and indicate which directories and/or files are to be excluded by the robot in the right pane.You can create a new file or edit an existing one. Even better, Robogen will work directly off the "robots.txt" file on your web site. It will log in, read the existing file, allow you to edit to your hearts content, then update the file directly.I like little specialized tool such as this which truly make a webmasters life easy. The product is inexpensive (there is a limited free version as well), well documented and complete. Highly recommended.

About the Author

Richard Lowe Jr. is the webmaster of Internet Tips And Secretsat http://www.internet-tips.net - Visit our website any time toread over 1,000 complete FREE articles about how to improve yourinternet profits, enjoyment and knowledge.