Bandwidth costs money, so there’s a limit to how much and how often website operators will let their websites be crawled. This limit means that website operators are picky about who they let crawl their websites. Only a select few crawlers are allowed access to the entire web, and Google is given extra special privileges on top of that. This isn’t illegal and it isn’t Google’s fault, but this monopoly on web crawling that has naturally emerged prevents any other company from being able to effectively compete with Google in the search engine market.