Okuzeka
Software Engineering vocabulary · Web, Mobile and Interface Development

Crawl

Meaning in software engineering, with examples

/krɔːl/verbCommon term

In software engineering, crawl means: For a search engine bot to visit pages automatically, follow their links and download their content so that they can be indexed.

In Turkish: taramak, dolaşmak

How it is used

The bot is called a crawler or spider; Google's is Googlebot. Developers control crawling with robots.txt and talk about crawl errors and, on very large sites, the 'crawl budget'.

Example sentences

  1. Googlebot crawled the staging site because nobody had blocked it.

  2. Pages that can only be reached through a search form are hard for bots to crawl.

Common collocations

  • crawl a site
  • search engine crawler
  • crawl budget
  • crawl errors
  • block crawlers
  • indexThe position number of an item in an array or list. In most languages counting starts at zero, so the first item has index 0.
  • sitemapA file, usually sitemap.xml, that lists the pages of a website so that search engines can find and crawl them more easily.
  • SEOSearch Engine Optimization: work that helps a site's pages to be found, understood and shown higher in search engines such as Google.
  • URLUniform Resource Locator: the full address of a page or file on the web, such as https://example.com/blog?page=2.

This word in other fields

The same English word can have a different meaning in another field:

More terms: Web, Mobile and Interface Development

Practise with graded reading

Reading at the right level is the fastest way to make new words stick. Pick free English books by level: