Understanding how SEO functions can be split up into 3 main processes.
Web Crawling is the process in which a search engine will use robots known as web crawlers or spiders to look through and monitor all the files, links, text, HTML tags, headings, meta tags, and various other forms of information contained on your page and then proceed to index them. Crawlers begin crawling a website by downloading its robots.txt file (wiki link to robots.txt).
These robot files contain rules about which pages search engines should or should not crawl on the website. The robots.txt file may also contain information about sitemaps (wiki link to site maps), which are comprised of lists of URLs that the site needs crawled.
These crawlers are automated algorithms and will jump from site to site, indexing information, and learning how to make your page stand out to them is key in increasing traffic to your site. The more links and pages you put on your site, the more indexing the spiders will do on your site, ultimately, increasing the ranking of your content.
Indexing is the process where the “spider” scans through websites and collects information that will later be compiled to form an index that is continuously expanding with keywords, websites, and other site information. For example, once a new page is discovered by the spider, Google will analyze the content and begin to catalog information such as images and video files embedded within the page to better understand the page and position it to be available to answer potential search queries. As a whole, this process is known specifically as search engine indexing.
Continuing in how search engines work, the spider moves websites that it has crawled and indexed to the search engine database. Those databases use the indexed data to categorize further the characteristics of the collected data within the cache even further to optimize its capability to rank it relative to other websites that best match the search query.
When a user enters a search, every page that Google thinks will best answer your question is brought up in a ranked order. This is also known as the SERP (Search Engine Results Page). Google will list the pages within its Search Engine Database in a specific order using what's referred to as PageRanking.
Now you may be asking, “what exactly IS PageRank?”. PageRank is a Google-specific algorithm that calculates each web page's unique value by counting the total number of links that point to said page (backlinks, internal links, etc.). This value determines the rank of the page in comparison to every other page within the search engine database.
The links within your page pass ‘PageRank’ through to other pages just as they might pass it back to yours. It's easier to understand the concept if you think of these links as people voting. Each of these links pointing towards your page is a “vote” that your page is beneficial or relevant to those searching. The same, of course, applies to the links your page points. These votes can also have larger values if the pages pointing to yours have high authority. For example, a link to your page on a Forbes article will have a more substantial vote than a link to your page on a deep web blog. Google will note that the Forbes article has high traffic volumes and judge that ‘vote’ as quality and relevant. This also works in reverse, as having links on your page to pages with high authority will help increase your own page’s rank.
It is important to note that although PageRank is a Google-specific algorithm, other search engines have their way of calculating “PageRank” using their algorithms to measure page value based on their values regarding linking within websites. Yahoo’s “Web Rank” functions similarly to Google's “PageRank.”
This loop, from site indexes to search databases to PageRanking, runs periodically, nonstop. In Google’s case, which has millions of spiders crawling with their vast bandwidth and massive index of pages, they can easily find your new content. Even more so if your webpage already has a high PageRank. For this reason, among many others increasing your PageRanking is key to successful SEO application. With a higher page rank your page will appear closer to the top of the SERP (Search Engine Results Pages) along side other pages that the engine feels answer the query made by the user.