When a search engine spider crawls your site, it does so two ways:

1. it looks in all public directories for web page files (.html, .htm, .asp, .php, etc.), as well as image files (.jpg, .gif)
2. it automatically follows the links on your site in order to index individual pages.

If you want control over which directories on your web site get indexed, you can the ROBOTS meta tag in the header of your root index page as well as other web pages. Not all of the search engine robots use this tag, but it's a good idea to include it for the ones that do.

The correct syntax for the tag is:

meta name="robots" content="noindex,nofollow"

[more]

Yalia Technology Design was formed to fill a specific gap in the marketplace, as a company that offers clients something more than a traditional web design company. Pulling from expertise in the areas of technology, project management, marketing communications and branding, Yalia provides a multi-disciplinary approach to web design and custom application development. More than just a technology company - we also offer a full range of marketing services including traditional collateral design, brand consulting, online and email marketing services, search engine optimization and search engine submissions.