Menu
Looking for Google APIs and Tools? Google Developers is the place to find all Google developer documentation, resources, events, and products.
Do You Provide Data from the Google Search API?The Google Search API was deprecated in April 2017 and was replaced by the Custom Research API which returns results from user-built Google custom search engines. These custom search engines are commonly embedded in websites and blogs and poll a list of URLs defined by the user. While Google is inarguably the best-known search engine in the world, it no longer provides its search results in machine-readable format. By comparison, the Webhose API scans and extracts data from hundreds of thousands of global data sources, providing organizations with access to a wide range of data from blogs, news sources, and other content sources.
Only Webhose’s APIs are suitable for providing enterprise-class applications with the volume of data they require from open web search results.
It is no news that Google is the largest search engine in the world. Lots of people will go the extra mile to have their content rank highly on Google before any other search engine. As a result of this, Google has lots of quality results for every search and with great ranking algorithms you can expect to get the best of search results on Google.This has an implication. Its implication is that there exists lots of useful data on Google and that calls for a need to scrape this golden data. The scraped data can be used for quality data analysis and discovery of wonderful insights.
It can also be important in getting great research information in one attempt.Talking about scraping, this can be done with third party tools. It can also be done with a Python library known as Scrapy. Scrapy is rated to be one of the best scraping tools, and can be used to scrape almost any web page.
You can find out more on the.However, regardless of the strengths of this wonderful library. Scraping data on Google could be one difficult task. Google comes down hard on any web scraping attempts, ensuring that scraping scripts do not even make as many 10 scrape requests in an hour before having the IP address banned. This renders third party and personal web scraping scripts useless.Google does give the opportunity to scrape information.
However, whatever scraping that would be done has to be through an Application Programming Interface (API).Just incase you do not already know what an Application Programming Interface is, there’s nothing to worry about as I’ll provide a brief explanation. By definition, an API is a set of functions and procedures that allow the creation of applications which access the features or data of an operating system, application, or other service. Basically, an API allows you gain access to the end result of processes without having to be involved in those processes. For example, a temperature API would provide you with the Celsius/Fahrenheit values of a place without you having to go there with a thermometer to make the measurements yourself.Bringing this into the scope of scraping information from Google, the API we would be using allows us access to the needed information without having to write any script to scrape the results page of a Google search.
Through the API, we can simply have access to the end result (after Google does the “scraping” at their end) without writing any code to scrape web pages.While Google has for different purposes, we are going to be using the Custom Search JSON API for the purpose of this article. More information on this API can be found.This API allows us make 100 search queries per day for free, with pricing plans available for making more queries if necessary.
Creating A Custom Search EngineIn order to be able to use the Custom Search JSON API, we would be needing a Custom Search Engine ID. From googleapiclient.discovery import buildmyapikey = 'Your API Key”mycseid = 'Your CSE ID 'def googlesearch(searchterm, apikey, cseid,.kwargs):service = build('customsearch ', 'v1 ', developerKey=apikey)res = service.cse.list(q=searchterm, cx=cseid,.kwargs).executereturn resIn the function above, the myapikey and mycseid variables should be replaced by the API Key and the Search Engine ID respectively as string values.All that needs to be done now is to call the function passing in the search term, the api key and the cse id.
Result = googlesearch ( 'Coffee', myapikey, mycseid )print (result )The function call above would search for the keyword “Coffee” and assign the returned value to the result variable, which is then printed. A JSON object is returned by the Custom Search API, therefore any further parsing of the resulting object would require a little knowledge of JSON.This can be seen from a sample of the result as seen below:The JSON object returned above is very similar to the result from the Google search:SummaryScraping Google for information isn’t really worth the stress. The Custom Search API makes life easy for everyone, as the only difficulty is in parsing the JSON object for the needed information. As a reminder, always remember to keep your Custom Search Engine ID and API Key values private.