Creating a robots.txt file by GNC Web Creations



Company <http://www.gnc-web-creations.com/>
Web Design <http://www.gnc-web-creations.com/web-design-services.htm>
Graphics <http://www.gnc-web-creations.com/graphics-design.htm>
Web Services <http://www.gnc-web-creations.com/services.htm>
Articles <http://www.gnc-web-creations.com/published-articles.htm>
Training Center <http://www.gnc-web-creations.com/web-training.htm>
Resources <http://www.gnc-web-creations.com/resource_links_directory.htm>
Home <http://www.gnc-web-creations.com/> About Us
<http://www.gnc-web-creations.com/introducing_us.htm> Award Pages
<http://www.gnc-web-creations.com/gnc_web_creations_award_pages.htm>
Website Portfolio <http://www.gnc-web-creations.com/portfolio.htm>
Contact Information <http://www.gnc-web-creations.com/contact.htm>
Payment Information <http://www.gnc-web-creations.com/payment_info.htm>
Web Design <http://www.gnc-web-creations.com/web-design-services.htm>
Website Cost <http://www.gnc-web-creations.com/pricing.htm> Small
Business
<http://www.gnc-web-creations.com/small-business-web-design.htm>
Brochure Websites
<http://www.gnc-web-creations.com/brochure-website.htm> Motorcycle
Websites <http://www.gnc-web-creations.com/motorcycle_website_design.htm>
Biker Graphics
<http://www.gnc-web-creations.com/custom_biker_graphics.htm> Custom
Buttons <http://www.gnc-web-creations.com/buttons-bars.htm> Custom
Banners <http://www.gnc-web-creations.com/banners-custom.htm> Custom
Graphics <http://www.gnc-web-creations.com/custom_graphic.htm> Graphics
Directory <http://www.gnc-web-creations.com/graphics-design.htm>
SEO Consulting <http://www.gnc-web-creations.com/seo-consulting.htm>
Website Review <http://www.gnc-web-creations.com/website-review.htm>
Internet Marketing
<http://www.gnc-web-creations.com/internet-marketing-consultant.htm>
Stock Photography
<http://www.gnc-web-creations.com/stock-photography.htm> Business
Planning <http://www.gnc-web-creations.com/business-planning.htm>
Computer <http://www.gnc-web-creations.com/computer-articles.htm>
Marketing <http://www.gnc-web-creations.com/marketing-tutorials.htm>
Motivational <http://www.gnc-web-creations.com/articles.htm> SEO
Articles
<http://www.gnc-web-creations.com/search-engine-optimization.htm> Web
Design <http://www.gnc-web-creations.com/web_design_articles.htm>
SEO Training <http://www.gnc-web-creations.com/seo-optimization.htm> SEO
Techniques <http://www.gnc-web-creations.com/seo-techniques.htm> Website
Marketing
<http://www.gnc-web-creations.com/successful-website-marketing.htm> Web
Development
<http://www.gnc-web-creations.com/website-development-training.htm>
Class Registration <http://www.gnc-web-creations.com/class-registration.htm>
Site Map <http://www.gnc-web-creations.com/sitemap.htm> Link To Us
<http://www.gnc-web-creations.com/link_to_gnc_web_creations.htm> Submit
Site <http://www.gnc-web-creations.com/submit_your_site.htm> Website
Tools <http://www.gnc-web-creations.com/xara_web_tools.htm> Resource
Directory <http://www.gnc-web-creations.com/resource_links_directory.htm>

	


	

Click Here To Visit Our Stock Photo Gallery
<http://www.gnc-web-creations.com/stock-photography.htm>


Photo Gallery
Discover the difference using professional quality stock photos can make
in the visual appeal of your custom business website!

View Photo Gallery <http://www.gnc-web-creations.com/stock-photography.htm>


Click Here To Visit SEO Techniques
<http://www.gnc-web-creations.com/seo-techniques.htm>


SEO Techniques
20 chapters with step by step tutorials to optimize your site for high
search engine placement!

Each method is based on solid SEO techniques that produce consistent
results that you can count on in the search engines!

More Information <http://www.gnc-web-creations.com/seo-techniques.htm>
Sunday 20. June 2010

	
	


  *Create Robots.txt File - Robots Text File*

Creating a robots.txt file



Frequently Asked Questions

    * * Why do I need a robots.txt file? <#Why do I need a robots.txt file>*
    * * What is a simple robots.txt file? <#What is a simple robots.txt
      file>*
    * * What if I don't want a particular file crawled? <#What if I dont
      want a particular file crawled>*
    * * What if I want to disallow a particular robot? <#What if I want
      to disallow a particular robot>*
    * * How do I create a robots.txt file? <#How do I create a
      robots.txt file>*
    * * How do I know if I have done everything correctly? <#How do I
      know if I have done everything correctly>*
    * * What if I need more information about robots.txt files? <#What
      if I need more information about robots.txt files>*


    *Why do I need a robots.txt file?*

Creating a robots.txt file will not improve your search engine
positioning, but it does provide robots with information concerning
which files you will *not* allow to be crawled and indexed in the search
engines.


When a robot crawls your site it looks for the  robots.txt file. If it
doesn't find one it assumes automatically that it may crawl and index
the entire site. Not having a robots.txt file can also create
unnecessary 404 errors in your server logs, making it more difficult to
track "real" 404 errors.


*Assuming you want your entire site indexed and only want to stop the
unnecessary 404 errors from occurring you have a couple of options.*

    *

      Upload a blank robots.txt file to the root directory of your domain.

    *

      Upload a simple robots.txt file to the root directory of your domain.

back to top <#back to top>



    *What is a simple robots.txt file?*

Please note: This will allow all robots to crawl and index all files.


*This allows all robots to crawl all files.*

User-agent: *
Disallow:


back to top <#back to top>



    *What if I don't want a particular file crawled?*

Please note: Disallowing a specific file to be crawled will keep it from
being indexed. The file disallowed will not show up in the search
engines. HOWEVER, this is only effective for friendly robots. Robots can
choose to ignore your instructions.


*This allows all robots to crawl all files except the images file.*

User-agent: *
Disallow: /images/


*This allows all robots to crawl all files except the images file and
the stats file.*

User-agent: *
Disallow: /images/
Disallow: /stats/


back to top <#back to top>



    *What if I want to disallow a particular robot?*

Occasionally you may find that you would like to disallow specific
robots from crawling your site or limit which files they may have access to.


*This denies access to Googlebot-image to any files in your domain*

User-agent: Googlebot-Image
Disallow: /


*This specifically denies Googlebot-image to your images file*

User-agent: Googlebot-Image
Disallow: /images/


For a current data base of robot names and information, visit:

* http://www.robotstxt.org/wc/active/html/index.html*

 

back to top <#back to top>



    *How do I create a robots.txt file?*

Simply create a text document and save the new document as robots.txt
*Do not use a html editor* to create the file unless is has the ability
to create a plain text document (ASCII). Most computers will allow you
to create a text document using notepad.

    *

      Right click on your desktop

    *

      Choose new

    *

      Choose text document

    *

      Open the document you just created

    *

      Insert instructions to robots

    *

      Click on save as

    *

      Save document as robots.txt

back to top <#back to top>



    *How do I know if I have done everything correctly?*

Once you have uploaded the file to the root directory of your domain
it's good idea to use a robots.txt validator to confirm that everything
is correct. You can search Google for * free robots.txt validator
<http://www.google.com/search?sourceid=navclient&ie=UTF-8&rls=GGLJ,GGLJ:2006-29,GGLJ:en&q=free+robots.txt+validator>
*or try the one listed below.

* http://tool.motoricerca.info/robots-checker.phtml*


back to top <#back to top>



    * What if I need more information about robots.txt files?*

This page is intended to cover creating a very simple robots.txt file.
If you require a more detailed robots.txt file for your website there
are many help resources available on the net. Google Information for
Webmasters recommends visiting:


 *http://www.robotstxt.org/wc/norobots.html *




J Walker (a.k.a. "Cricket")


J. Walker of *GNC Web Creations*

Contact Me With Your Comments or Questions
<http://www.gnc-web-creations.com/contact.htm>


Articles are Copyright © J. Walker of GNC Web Creations Unless Otherwise
Noted


We welcome your comments on creating a robots.txt file.

* Comments and Suggestions <http://www.gnc-web-creations.com/contact.htm>*



    Additional Search Engine Optimization Article

* Search Engine Optimization Articles
<http://www.gnc-web-creations.com/search-engine-optimization.htm>*

    *

      * Increase Website Traffic Free
      <http://www.gnc-web-creations.com/secrets.htm>*

    *

      * Create Site Map - Anchor Text
      <http://www.gnc-web-creations.com/site_map__using_anchor_text.htm>*

    *

      * SEO Strategy - Importance of Incoming Links to SEO Strategy
      <http://www.gnc-web-creations.com/seo-strategy.htm>*

    *

      * Ethical Search Engine Optimization - Ethical SEO Techniques
      <http://www.gnc-web-creations.com/ethical-search-engine-optimization.htm>*



    Ethical Search Engine Optimization Class

SEO Techniques - Search Engine Optimization


You've ended up in the right place to learn the basics of SEO, along
with many valuable internet marketing strategies, designed to increase
your online business profitability and visibility in the search engines.
We provide structured online classes through our Yahoo group * SEO
Techniques <http://groups.yahoo.com/group/SEO_Techniques/>*. The online
classes are 100% free and are email based so you are able to work at a
pace that best fits your needs.


* Ethical Search Engine Optimization Class
<http://www.gnc-web-creations.com/seo-optimization.htm>*

Information on Ethical Search Engine Optimization Class



    SEO Techniques Step by Step eBook

Search Engine Optimization Techniques


When the website challenge you are facing is poor search engine
placement, consider SEO Techniques Step by Step as a low-cost
alternative to SEO consultant fees.


SEO Techniques Step by Step
<http://www.gnc-web-creations.com/seo-techniques.htm> 	

Many members have found class much easier to follow, and have
consistently reported higher success rates in the search engines by
owning the eBook, * SEO Techniques Step By Step*
<http://www.gnc-web-creations.com/seo-techniques.htm>. It is not a
requirement of attending class. You will be entirely welcome either way!


We're committed to doing our very best to exceed your expectations! If
you have questions or comments, please don't hesitate to * contact
<http://www.gnc-web-creations.com/contact.htm>* us.

 


* SEO Techniques Step By Step
<http://www.gnc-web-creations.com/seo-techniques.htm>*

The eBook With All The Lessons Compiled Here



Click Here For Free Xara Webstyle 4 Trial Download!
<http://stats.xaraonline.com/ID.1055835/product.Webstyle4CD/redirectid.WebstyleHome/affiliate.html>


Sometimes a program is so incredible that we have to share it with
others! We are extremely impressed with this all new version of * Xara
Webstyle 4
<http://stats.xaraonline.com/ID.1055835/product.Webstyle4CD/redirectid.WebstyleHome/affiliate.html>
*and are currently using it as a tool in designing our sites! This is a
"must have" tool for web designers! Take a look at our complete * Xara
Webstyle 4 Review <http://www.gnc-web-creations.com/xara-webstyle.htm>*
today!

    * * Create Complete Web Photo Albums
      <http://www.gnc-web-creations.com/web-photo-album.htm> *
    * * Fastest Photo Editors Available Today
      <http://www.gnc-web-creations.com/photo-editor-software.htm> *
    * * Design Banners, Logos, Buttons and Bars
      <http://www.gnc-web-creations.com/banner-ads.htm>*
    * * Design Navigation Bars and Dynamic Menus
      <http://www.gnc-web-creations.com/dynamic-menus.htm> *
    * * Create Complete Themed Web Page Layouts
      <http://www.gnc-web-creations.com/web-page-layouts.htm> *

Try Xara Webstyle 4 FREE Today!
<http://stats.xaraonline.com/ID.1055835/product.Webstyle4CD/redirectid.WebstyleHome/affiliate.html>

 


 

GNC Web Creations - 2125 Universal City BLVD - Universal City, TX 78148
- (210) 564-9153

Copyright © 2003 - 2010 J. Walker of GNC Web Creations All Rights Reserved

 


	


Company <http://www.gnc-web-creations.com/>

	
Web Design <http://www.gnc-web-creations.com/web-design-services.htm> 	
Graphics <http://www.gnc-web-creations.com/custom_graphic.htm> 	*
* Web Services <http://www.gnc-web-creations.com/services.htm> 	
Articles <http://www.gnc-web-creations.com/published-articles.htm> 	
Training Center <http://www.gnc-web-creations.com/web-training.htm> 	
Resources <http://www.gnc-web-creations.com/resource_links_directory.htm>

About Us <http://www.gnc-web-creations.com/introducing_us.htm>
Contact Info <http://www.gnc-web-creations.com/contact.htm>
Award Pages
<http://www.gnc-web-creations.com/gnc_web_creations_award_pages.htm>
Web Portfolio <http://www.gnc-web-creations.com/portfolio.htm>
Payment Info <http://www.gnc-web-creations.com/payment_info.htm>

	

Web Design <http://www.gnc-web-creations.com/web-design-services.htm>
Website Cost <http://www.gnc-web-creations.com/pricing.htm>
Small Business
<http://www.gnc-web-creations.com/small-business-web-design.htm>
Motorcycle Sites
<http://www.gnc-web-creations.com/motorcycle_website_design.htm>
Brochure Website <http://www.gnc-web-creations.com/brochure-website.htm>

	

Biker Graphics <http://www.gnc-web-creations.com/custom_biker_graphics.htm>
Graphic Design <http://www.gnc-web-creations.com/custom_graphic.htm>
Custom Buttons <http://www.gnc-web-creations.com/buttons-bars.htm>
Website Banners <http://www.gnc-web-creations.com/banners-custom.htm>
Custom Graphics <http://www.gnc-web-creations.com/graphics-design.htm>

	

SEO Consulting <http://www.gnc-web-creations.com/seo-consulting.htm>
Website Review <http://www.gnc-web-creations.com/website-review.htm>
Business Planning <http://www.gnc-web-creations.com/business-planning.htm>
Internet Marketing
<http://www.gnc-web-creations.com/internet-marketing-consultant.htm>
Stock Photography <http://www.gnc-web-creations.com/stock-photography.htm>

	

Computer <http://www.gnc-web-creations.com/computer-articles.htm>
Marketing <http://www.gnc-web-creations.com/marketing-tutorials.htm>
Motivational <http://www.gnc-web-creations.com/articles.htm>
SEO Articles
<http://www.gnc-web-creations.com/search-engine-optimization.htm>
Web Design <http://www.gnc-web-creations.com/web_design_articles.htm>

	

SEO Training <http://www.gnc-web-creations.com/seo-optimization.htm>
SEO Techniques <http://www.gnc-web-creations.com/seo-techniques.htm>
Website Marketing
<http://www.gnc-web-creations.com/successful-website-marketing.htm>
Web Development
<http://www.gnc-web-creations.com/website-development-training.htm>
Class Registration <http://www.gnc-web-creations.com/class-registration.htm>

	

Site Map <http://www.gnc-web-creations.com/sitemap.htm>
Link To Us <http://www.gnc-web-creations.com/link_to_gnc_web_creations.htm>
Submit Site <http://www.gnc-web-creations.com/submit_your_site.htm>
Website Tools <http://www.gnc-web-creations.com/xara_web_tools.htm>
1and1 Hosting <http://www.gnc-web-creations.com/1and1.htm>

 

Custom Website Creations <http://www.gnc-web-creations.com> | 
Mississippi Photo Gallery <http://www.mississippi-photo-gallery.com/> | 
Website Development Training
<http://www.website-development-training.com/>  |  Ask Cricket
<http://www.cricket.gnc-web-creations.com/>  | Legal Notices
<http://www.gnc-web-creations.com/privacy.htm>