sh BASH - HTTP和INTERNET UTILS

Posted 2021-05-06

https://curl.haxx.se/docs/manpage.html#-d


#######################################
# output : File where the url is to be saved
# With this we can download the HTML of any page
curl http://some.url --output some.file

#######################################
# Silent use - Good for scripts
# 
curl -s http://some.url --output some.file
curl --silent http://some.url --output some.file

#######################################
# Curl and piping
#
curl -s http://example.com | wc -l




https://unix.stackexchange.com/questions/115150/use-curl-to-download-from-a-url

# CA CERTIFICATES
curl --include --request GET --cacert ~/cacert.pem --user ppcusr:\!ppcusr. --header "X-ID-TENANT-NAME:usexample" "https://apps.deddie.gr/ords/ordstest/expfiles/expf/?p_filename=DXREOS.zip&p_dt=20171031" --output FILECURL4.ZIP

https://curl.haxx.se/docs/caextract.html


###################################################
# MAKE POST REQUEST WITH CURL


2278
down vote
+50
With fields:

curl --data "param1=value1&param2=value2" https://example.com/resource.cgi
Multipart:

curl --form "fileupload=@my-file.txt" https://example.com/resource.cgi
Multipart with fields and a filename:

curl --form "fileupload=@my-file.txt;filename=desired-filename.txt" --form param1=value1 --form param2=value2 https://example.com/resource.cgi
Without data:

curl --data '' https://example.com/resource.cgi

curl -X POST https://example.com/resource.cgi

curl --request POST https://example.com/resource.cgi
For more information see the cURL manual. The cURL tutorial on emulating a web browser is helpful.

With libcurl, use the curl_formadd() function to build your form before submitting it in the usual way. See the libcurl documentation for more information.

For large files, consider adding parameters to show upload progress:

curl --tr-encoding -X POST -v -# -o output -T filename.dat \
  http://example.com/resource.cgi
The -o output is required, otherwise no progress bar will appear.

For a RESTful HTTP POST containing XML:

curl -X POST -d @filename.txt http://example.com/path/to/resource --header "Content-Type:text/xml"
or for JSON, use this:

curl -X POST -d @filename.txt http://example.com/path/to/resource --header "Content-Type:application/json"


113
down vote
Data from stdin: use -d @-. Example:

echo '{"text": "Hello **world**!"}' | curl -d @- https://api.github.com/markdown
Output:

<p>Hello <strong>world</strong>!</p>


55
down vote
If you want to login to a site, do the following:

curl -d "username=admin&password=admin&submit=Login" --dump-header headers http://localhost/Login
curl -L -b headers http://localhost/

################################################################
# View the complete request and response
curl -v http://example.com 

################################################################
# Curl and following redirects

#If a site has WordPress® installed for example and they are using 301 redirects you will by default download the redirect response only. To ensure you follow the redirects and get the final file you will need to use the -L option. If you try
curl google.com
#you will just get the redirect page, if you now try
curl -L google.com
#you will get the page you were after

################################################################
# View only response headers
curl -I google.com

################################################################
#Skipping SSL checks

When connecting to a remote server that has a self signed certificate you will want to skip the ssl checks. To do this use the
-k
option. For example
curl -k https://google.com


Setting the user agent

The
-A
option allows you to set the user agent. For example
curl -A "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0" -L google.com
.

Rate Limiting

To avoid hitting the remote server hard you can limit the download rate you will use. The command to do this is
--limit-rate
and use like this
--limit-rate 100k
.

FTP Login

To set the username and password you can use the –user username:password option.

FTP upload and download

To download you just need to use the basic curl command but add your username and password like this
curl --user username:password -o filename.tar.gz ftp://domain.com/directory/filename.tar.gz
.

To upload you need to use both the –user option and the -T option as follows.
curl --user username:password -T filename.tar.g ftp://domain.com/directory/

To delete a file from the remote server.
curl --user username:password -X 'DELE filename.tar.gz' ftp://domain.com/

For details on the
-X
option please see the next section.

Sending POST requests and different FTP Commands

The -X command allows you to send custom commands to the receiving server. For HTTP this could be a POST request or WebDAV’s copy or move. For FTP you can use the
-X
option to send other commands instead of the default file LIST, like in the previous section’s example of using
-X
to send the DELE command to an FTP server. However this can also be used to send full POST data to an HTTP server.

If the page you wanted to POST to had a form like this:

<form action="test.php" method="POST">
  <input name="Name" type="text" /> 
  <input name="button1" type="submit" value="OK" />
</form>
you could submit the form request using
curl -X POST --data "name=barrym&button1=OK" http://www.example.com/test.php
.

Using Cookies

If the server you are connecting to requires a cookie then you can send it using
curl -b cookiefile.txt http://example.com

Sending custom headers

If you need to send a custom header to the server you would use the
-H
option like this
curl -H "Accept: text/html" http://domain.com
. Which would set the content type to text/html.

Verifying an SSL Certificate

If you want to verify that your SSL cert is valid without using your browser and run into potential caching issues then use
curl --cacert mycert.crt https://domain.com
. This is also useful if you need to validate the connection to ensure that you are connecting to the right server.

The certificate must be in PEM format. If the optional password is not specified, it will be queried for on the terminal. Note that this option assumes a “certificate” file that is the private key and the private certificate concatenated.

Limiting the connection timeout time

Sometimes you will want the connection to time out quickly if it can’t make the connection within a certain time frame. The option to use to do this is –connect-timeout. This only affects the connection time so once you are connected it no longer applies.
curl --connect-timeout 5 http://domain.com