In this section, we will see how to send actual requests and fetch the responses. For that purpose, we are taking the help of a website that is as follows:
As we know the get requests are used to fetch the data so let’s see the code:
The above code gets the data from the URL. After getting the data, we convert it into JSON format and then print the request. Following will be the output generated as a result:
As we can see that we have given the full URL in which the parameters are also attached. In the above case page is a parameter whose value is 2. We can convert the above code as following which makes it easier to understand, and the code will also become easy to know when there are lots of parameters:
Now we will what is the status code is generated as a result of a request so if we print r.status_code. The following output will be given:
200 status code indicates that a request is successful.
Error in Response:
Sometimes there could be an error which in return gives the negative response. Following is the example of an error response. The requests that we are going to make is not present that is why the response that is generated is empty, and the status code will be of 404
Following is the code:
The output of the above-written code is as follows:
This error is generated because no such user is presented in the database which results in the error in the query.
This error can be avoided by using the try-except statement. However, this can only be done during production so that no one can see the error by which anyone can manipulate with the code, only during debugging so that it is easy to debug our code.
Now we will see how to handle post requests. Post requests are used when we have to send the data confidentially or to add the data. The difference between post data and get data is, in getting data anyone can see the information that is being sent, but in the post, requests data is hidden and can’t be seen. Following is the example of post requests:
In the above example, we set a payload in which name and job are assigned in the dictionary. Then we post data to create a new user, and along with this, we have sent the payload that we assigned earlier. The output of the above code is as follows:
The output shows the status code that is 201 which indicates successfully created new user and response shows the id at which new user is created and the time of creation.
The delete request as the name indicates to delete the request and If we print the response, it will generate an error because there is no data left to show as a response. Following is the example of delete request in python:
When we print the status, it will be 204 which indicates the data has been deleted, but when we see the response, it will show an error because no data to display. The output is as follows:
The JSON was expecting a value, but none was found.
Put request is used to edit the data. To verify this, we try to update the data and in response, the time of updating will be set. The code is as follows:
The output will show the updated time that indicated it is updated:
Getting HTML response:
As we studied earlier that getting the HTML response is used when we try to scrape the data. The response gives the HTML, and then we parse the data and extract the information we needed. Following is the code to fetch the HTML from the website:
If we print the r.text it will generate the HTML as follows:
The above image is the print statement. It contains all the page source that is needed to scrape we can get the data from this.
It is all the request module. In the next section, we will learn web scraping by using this requests module to scrape the python.org website and what are the uses of scraping the data and in which department it is used extensively. We will also learn about the beautiful soup in the next section.