What is Crawl and Index

In the SEO world, crawling means following your links and “crawling” around your website. When bots come to your website (any page), they follow other linked pages also on your website. This is one reason why we create site maps, as they contain all of the links in our blog and bots can use them to look deeply into a website.

What is Indexing? In terms, indexing is the process of adding webpages into a Web Search Engine. Depending upon which meta tag you used (index or NO-index), Website Search Engine bots will crawl and index your pages. A no-index tag means that that page will not be added into the web search’s index.

There are millions of websites on this earth, and Most people are left constantly wondering why their articles aren’t getting indexed.

Let’s take a look at some major factors which play some important roles at the backend of crawling and indexing.

The following factors can negatively affect site’s crawl ability:

Slow server responses or a significant number of errors
A significant number of low value–add pages. These can include:
- Many versions of a page with URL parameters that offer useless filtering, false navigation, or session identifiers
- Duplicate content
- Low-quality pages
- Dirty pages
Long redirect chains
Long page-load times that may timeout
Nonstrategic use of noindex and nofollow tags
Pages served up through AJAX without links in the page source
Blocking bots from crawling JavaScript and CSS files
“Dirt” in your sitemap

What is Crawl and Index

1 comment: