COMP9321/19T1/resources/22490

Posted 2021-02-13 whltay

Resources / Assignments (/COMP9321/19T1/resources/22490)
/ Assignment 2 (/COMP9321/19T1/resources/24962)
/ Data Service for World Bank Economic Indicators
Data Service for World Bank Economic
Indicators
Data Service for World Bank Economic
Indicators
Change Log: Add some tips about the submission, and fix the spec inconsistent problem.
Assignment template : https://www.cse.unsw.edu.au/~cs9321/19T1/assn/assn2.tar.gz
(https://www.cse.unsw.edu.au/~cs9321/19T1/assn/assn2.tar.gz)
You shall develop it with Flask (1.0.2), Flask-SQLAlchemy (2.3.2) and flask-restplus (0.12.1)[optional] .
As for you are safe to install the three packages with pip3 on option "--user". You are only allowed to
install those three packages if one of them not available.
You can use any package already installed in CSE machine.
In this assignment, you are asked to develop a Flask-Restplus data service that allows a client to read
some publicly available economic indicator data for countries around the world, and allow the consumers
to access the data through a REST API.
IMPORTANT : For this assignment , you will be asked to access the given Web content programmatically.
Some Web hosts do not allow their web pages to be accessed programmatically, some hosts may block
your IP if you access their pages too many times. During the implementation, download a few test pages
and access the content locally - try not to perform too many programmatically.
The source URL: http://api.worldbank.org/v2/ (http://api.worldbank.org/v2/)
(http://api.worldbank.org/v2/) Documentations on API Call Structure:
(https://datahelpdesk.worldbank.org/knowledgebase/articles/898581-api-basic-call-structure)
https://datahelpdesk.worldbank.org/knowledgebase/articles/898581-api-basic-call-structure
(https://datahelpdesk.worldbank.org/knowledgebase/articles/898581-api-basic-call-structure)
(https://datahelpdesk.worldbank.org/knowledgebase/articles/898581-api-basic-call-structure)
The World Bank indicators API provides 50 year of data over more than 1500 indicators for countries
around the world.
List of indicators can be found at: http://api.w o rldbank.org/v2/indicators
(http://api.worldbank.org/v2/indicators)
List of countries can be found at: (http://api.worldbank.org/v2/countries)
http://api.worldbank.org/v2/countries (http://api.worldbank.org/v2/countries)
As the documentation shows , you can construct URLs specifying different parameters to acquire
necessary data. Use a custom URL to get information about a specific indicator. For example, the data on
the GDP of all countries from 2013 to 2018 can be acquired via this URL:
(http://api.worldbank.org/v2/countries/all/indicators/NY.GDP.MKTP.CD?date=2012:2017)
http://api.worldbank.org/v2/countries/all/indicators/NY.GDP.MKTP.CD?date=2013:2018&format=json
(http://api.worldbank.org/v2/countries/all/indicators/NY.GDP.MKTP.CD?date=2012:2017&format=json)
For this assignment we are interested in annual values of a user specified indicator from 2013 to 2018 for
one or all countries. The content of the file is quite straightforward. The data service specification will ask
you to ‘import‘ this data into a ‘data storage‘. You should inspect the content of the file carefully and
decide on a data model and storage method. You will find that a JSON format is suitable to accessing data
and publishing it.
The database we used in this assignment is SQLite . You need to determine the way about how to store
the data. We don‘t care about your output format as long as your data is correct and reasonable
arranged. Please download the python code template a2.py. Don‘t change the function signature
"create_db" which you need to use it to create the database. All the other function, it‘s flexible. Also,
no auto-test will be provided.
Note: You may need some additional key in the record for storage purpose.
Don‘t send too much requests to the source API as it will ban your IP address, be careful.
Please make sure your have a swaggerUI page on you API.
How to create database in SQLite: http://www.sqlitetutorial.net/sqlite-python/creating-database/
(http://www.sqlitetutorial.net/sqlite-python/creating-database/)
Assignment Specification
The data service should use JSON format to publish its data and implement following operations.
1- Import a collection from the data service
This operation can be considered as an on-demand ‘import‘ operation. The service will download the
JSON data for all countries (http://api.worldbank.org/v2/countries/all/indicators/NY.GDP.MKTP.CD?
date=2012:2017&format=json) respective to the year 2013 to 2018 and identified by the indicator id given
by the user and process the content into an internal data format.
Parameters should be given to the endpoint (in the payload) by the user:
indicator_id : an indicator (http://api.worldbank.org/v2/indicators)
http://api.worldbank.org/v2/indicators (http://api.worldbank.org/v2/indicators)
After importing the collection, the service should return a response containing at least the following
information:
location : the URL with which the imported collection can be retrieved
collection_id : a unique identifier automatically generated
Specification Make Submission Check Submission Collect Submission
creation_time : the time the collection stored in the database
Example:
HTTP operation: POST /<collections>
Input Payload:
{ "indicator_id" : "NY.GDP.MKTP.CD" }
Returns: 201 Created
{
"location" : "/<collections>/<collection_id>",
"collection_id" : "<collection_id>",
"creation_time": "2019-03-09T12:06:11Z",
"indicator" : "<indicator>"
}
The return response contains the location of the data entry created as the result of the processing
the import.
You should return appropriate responses in case of invalid indicators or any invalid attempts to use
the endpoint ( e.g. If the input indicator id doesn‘t exist in the data source, return error 404 )
If an input contains a n indicator that already has been imported before, you should still return the
location of the data entry - but with status code 200 OK (instead of 20 1 Created).
A POINT TO PONDER ABOUT: An `asynchronous POST‘?? If a POST takes too long, you may not
want the client to wait. What you would do? You do not need to address this in the assignment.
The source API has pagination; in order to get all of data you need to send many request to import a
single collection; however, you are required to get only first two pages instead of all:
http://api.worldbank.org/v2/countries/all/indicators/NY.GDP.MKTP.CD?
date=2013:2018&format=json&page=2
(http://api.worldbank.org/v2/countries/all/indicators/NY.GDP.MKTP.CD?
date=2012:2017&format=json&page=2)
The data entries inside the collection must be converted as described below:
Data entry conversion:
Here is an example of source data entry as it is in the source API
(http://api.worldbank.org/v2/countries/all/indicators/NY.GDP.MKTP.CD?date=2013:2018&format=json) :
{
"indicator": {
"id": "NY.GDP.MKTP.CD",
"value": "GDP (current US$)"
},
"country": {
"id": "1A",
"value": "Arab World"
},
"countryiso3code": "",
"date": "2016",
"value": 2513935702899.65,
"unit": "",
"obs_status": "",
"decimal": 0
}
However, you do not need to store all of its attributes; instead convert it to a JSON format as below:
{
"country": "Arab World",
"date": "2016",
"value": 2513935702899.65
}
And as a result a collection should be formatted and stored in the database as follow:
{
"collection_id" : "<collection_id>",
"indicator": "NY.GDP.MKTP.CD",
"indicator_value": "GDP (current US$)",
"creation_time" : "<creation_time>"
"entries" : [
{ "country": "Arab World", "date": "2016", "value": 2513935702899
.65 },
...
{ "country": "Caribbean small states", "date": "2017", "value":
68823642409.779 },
...
]
}
2- Deleting a collection with the data service
This operation deletes an existing collection from the database. The interface should look like as below:
HTTP operation: DELETE /<collections>/{collection_id}
Returns: 200 OK
{
"message" :"Collection = <collection_id> is removed from the database!"
}
3 - Retrieve the list of available collections
This operation retrieves all available collections. The interface should look like as like below:
HTTP operation: GET /<collections>
Returns: 200 OK
[
{
"location" : "/<collections>/<collection_id_1>",
"collection_id" : "collection_id_1",
"creation_time": "<time>",
"indicator" : "<indicator>"
},
{
"location" : "/<collections>/<collection_id_2>",
"collection_id" : "collection_id_2",
"creation_time": "<time>",
"indicator" : "<indicator>"
},
...
]
4 - Retrieve a collection
This operation retrieves a collection by its ID . The response of this operation will show the imported
content from world bank API for all 6 years. That is, the data model that you have designed is visible here
inside a JSON entry‘s content part.
The interface should look like as like below:
HTTP operation: GET /<collections>/{collection_id}
Returns: 200 OK
{
"collection_id" : "<collection_id>",
"indicator": "NY.GDP.MKTP.CD",
"indicator_value": "GDP (current US$)",
"creation_time" : "<creation_time>"
"entries" : [
{"country": "Arab World", "date": "2016", "value": 2513935702899
.65 },
{"country": "Caribbean small states", "date": "2017", "value": 6
8823642409.779 },
...
]
}
5 - Retrieve economic indicator value for given country and
a year
The interface should look like as like below:
HTTP operation: GET /<collections>/{collection_id}/{year}/{country}
Returns: 200 OK
{
"collection_id": <collection_id>,
"indicator" : "<indicator_id>",
"country": "<country>,
"year": "<year">,
"value": <indicator_value_for_the_country>
}
6 - Retrieve top/bottom economic indicator values for a
given year
The interface should look like as like below:
HTTP operation: GET /<collections>/{collection_id}/{year}?q=<query>
Returns: 200 OK
{
"indicator": "NY.GDP.MKTP.CD",
"indicator_value": "GDP (current US$)",
"entries" : [
{
"country": "Arab World",
"date": "2016",
"value": 2513935702899.65
},
...
]
}
The <query> is an optional parameter which can be either of following:
top<N> : Return top N countries sorted by indicator value
bottom<N> : Return bottom N countries sorted by indicator value
where N can be an integer value between 1 and 100. For example, a request like " GET /<collections>/<
id>/2015?q=top10 " should returns top 10 countries according to the collection_id.
Notes:
REASON : the source API will ban your (and tutors‘) IP addresses if you send too many requests
The source API has pagination; in order to get all of data you need to send many request to import a
single collection; however, you are required to get only first two pages instead of all:
http://api.worldbank.org/v2/countries/all/indicators/NY.GDP.MKTP.CD?
date=2013:2018&format=json&page=1
(http://api.worldbank.org/v2/countries/all/indicators/NY.GDP.MKTP.CD?
date=2012:2017&format=json&page=1)
http://api.worldbank.org/v2/countries/all/indicators/NY.GDP.MKTP.CD?
date=2013:2018&format=json&page=2
(http://api.worldbank.org/v2/countries/all/indicators/NY.GDP.MKTP.CD?
date=2012:2017&format=json&page=2) or to get all data use :
http://api.worldbank.org/v2/countries/all/indicators/NY.GDP.MKTP.CD?
date=2013:2018&format=json&per_page=2000
(http://api.worldbank.org/v2/countries/all/indicators/NY.GDP.MKTP.CD?
date=2012:2017&format=json&per_page=2000)
In Q6: top1 should return the highest numeric value
Your code must implemented in flask-restplus and provide swagger doc for testing the endpoints.
Late Penalties:
2 marks per day.
Resource created 7 days ago (7 days ago), last modified about 23 hours ago (about 23 hours ago).
Submission:
When you make a submission, please make sure your create_db function can make a db for you so that
we can test it. DB‘s names doesn‘t matter, just make sure it‘s same as what you used on your API.
give cs9321 assn2 a2.py
Plagiarism:
this will result in a final 0 mark for COMP9321 2019 Term 1.
Comments
! " (/COMP9321/19T1/forums/search?forum_choice=resource/24864)
# (/COMP9321/19T1/forums/resource/24864)
$ Add a comment
Meng Sun (/users/z5149213) about 3 hours ago (Thu Mar 14 2019 06:31:53 GMT+1100 (AEDT))
Hi professor,
For the source data of the 2 page json, could my program save them locally as files. And
the read these files to operate? Or I have to visit the website every time when I need
them.
Reply
Xiaocong Chen (/users/z5027195) about 2 hours ago (Thu Mar 14 2019 06:51:12 GMT+1100
(AEDT))
When testing you can do that as it may ban your ip due to many requests. But for
submission version, you have to using the API version instead file
Reply
Meng Sun (/users/z5149213) about 2 hours ago (Thu Mar 14 2019 07:33:25 GMT+1100
(AEDT))
Thanks.
As for the db file, could we save it locally?
Reply
Elliott Vercoe (/users/z3411256) about 4 hours ago (Thu Mar 14 2019 05:15:17 GMT+1100 (AEDT))
For q6, if the query is malformed, do we throw a 400 error or do we just ignore the
query?
So if I sent GET /<collections>/< id>/2015?q= top6a4 should I send a 400, or just ignore
the query and send all entries for that year, as though this query wasnt there?
Reply
Xiaocong Chen (/users/z5027195) about 4 hours ago (Thu Mar 14 2019 05:31:25 GMT+1100
(AEDT))
If the thing you request is not exist just raise the error
Reply
Yingjie Zheng (/users/z5213329) about 8 hours ago (Thu Mar 14 2019 01:00:42 GMT+1100
(AEDT)), last modified about 8 hours ago (Thu Mar 14 2019 01:02:47 GMT+1100 (AEDT))
Hi.
It‘s hard to tell which part has been update. And it‘s a long article.
Can you provide some "diff" when you update the spec? Say, if you use git for this
article and share the repo, we can simply view the git history and easily notice any
change.
That will help.
Reply
Xiaocong Chen (/users/z5027195) about 4 hours ago (Thu Mar 14 2019 05:31:57 GMT+1100
(AEDT))
check below
Reply
Yingjie Zheng (/users/z5213329) about 3 hours ago (Thu Mar 14 2019 05:58:17
GMT+1100 (AEDT))
I don’t understand. What are you referring by “below”?
The question Meng Sun ask? That has nothing to do with my question.
Reply
Meng Sun (/users/z5149213) about 10 hours ago (Wed Mar 13 2019 23:10:13 GMT+1100 (AEDT))
Hi everyone, I have a question, how do we send the http operation? do we need to
design a UI for the user to input and make opration?
Reply
Elliott Vercoe (/users/z3411256) about 8 hours ago (Thu Mar 14 2019 01:24:32 GMT+1100
(AEDT))
you can use curl. if you‘re on windows, go to command prompt and type
curl -i -H "Content-Type: application/json" -X POST -d "{"indicator_id" :
"NY.GDP.MKTP.CD"}" http://localhost:5000/collections
Reply
Kaylen Payer (/users/z5076219) about 11 hours ago (Wed Mar 13 2019 22:17:25 GMT+1100
(AEDT)), last modified about 11 hours ago (Wed Mar 13 2019 22:17:57 GMT+1100 (AEDT))
Hi Everyone,
I‘ve got a basic hello world flask app running fine on my local machine. However, on
when I ssh into cse i have issues:
When i run the app and use `curl http://0.0.0.0:4444/` i get:
"Failed to connect to 0.0.0.0 port 4444: Connection refused"
I‘m using the following commands to start the flask app (since the default port was
giving me "flask OSError: [Errno 98] Address already in use")
```
a pp = Flask(__name__)
if __name__ == ‘__main__‘:
app.run(debug=True, host=os.getenv(‘IP‘, ‘0.0.0.0‘), port=int(os.getenv(‘PORT‘,
4444)))
```
If anyone could suggest how i am able to test this on the uni server without physically
logging in to a physical machine, I would be very appreciative for any guidance.
Thank you :)
Reply
Xiaocong Chen (/users/z5027195) about 11 hours ago (Wed Mar 13 2019 22:43:52
GMT+1100 (AEDT))
don’t try to hold services at 0.0.0.0. Try 127.0.0.1
Reply
Kaylen Payer (/users/z5076219) about 10 hours ago (Wed Mar 13 2019 22:59:35
GMT+1100 (AEDT))
Thanks for getting back to me Xiaocong. I did originally try using 127.0.0.1, and
have just tested this again, but it still gives the same connection refused error :(
Reply
Xiaocong Chen (/users/z5027195) about 10 hours ago (Wed Mar 13 2019 23:10:50
GMT+1100 (AEDT))
hmm, you can go to one of the labs and ask for help.
Reply
Elliott Vercoe (/users/z3411256) about 24 hours ago (Wed Mar 13 2019 09:40:44 GMT+1100
(AEDT))
Has anyone had any luck with getting cascade deletes working on SQLAlchemy? I‘ve
used similar packages, but sqlalchemy seems to hate deleting children when the parents
are deleted.
Reply
Michael Tran (/users/z3461919) about 23 hours ago (Wed Mar 13 2019 10:03:14 GMT+1100
(AEDT))
it’s a limitation with SQLite I believe. When you start a connection, you need to
redefine the relationship.
Something something pragma something.
Would give a more concrete explanation, but I’m sleep-typing
Reply
Chaoran Huang (/users/z5022460) about 3 hours ago (Thu Mar 14 2019 06:26:24
GMT+1100 (AEDT))
I removed your post on testing. As you know this is an individual assignment, and
sharing such testing can be an at least lv2 academic misconduct.
Reply
Michael Tran (/users/z3461919) about 3 hours ago (Thu Mar 14 2019 06:47:21
GMT+1100 (AEDT))
Understood.
Reply
Elliott Vercoe (/users/z3411256) about 8 hours ago (Thu Mar 14 2019 01:25:53 GMT+1100
(AEDT))
thanks Michael. I gave up on trying to set up ondelete cascade and am just
manually deleting the required instances. this is a lot faster than troubleshooting
it.
Reply
Yingjie Zheng (/users/z5213329) a day ago (Wed Mar 13 2019 09:03:58 GMT+1100 (AEDT)), last
modified about 24 hours ago (Wed Mar 13 2019 09:37:11 GMT+1100 (AEDT))
Dear teaching stuff:
I notice there might be an inconsistency between the following statements:
"After importing the collection, the service should return a response containing at least
the following information:" (location, collection_id, creation_time).
"You do not have to return the actual imported content with POST. Just return the
location . "
Please confirm it (and fix it, if applicable).
BTW: I also counter same question Christopher Chaaya
(https://webcms3.cse.unsw.edu.au/users/z5016114) asked 3 days ago . Would you please
take a look at it?
Reply
Xiaocong Chen (/users/z5027195) about 23 hours ago (Wed Mar 13 2019 10:07:10
GMT+1100 (AEDT))
Yeah, fixed, please check now
Reply
Chaoran Huang (/users/z5022460) a day ago (Wed Mar 13 2019 08:17:26 GMT+1100 (AEDT))
For <collections> please refer to please read https://restful-apidesign.readthedocs.io/en/lates...
(https://restful-apidesign.readthedocs.io/en/latest/resources.html)
Reply
Christopher Chaaya (/users/z5016114) about 23 hours ago (Wed Mar 13 2019 10:31:18
GMT+1100 (AEDT))
So am I understanding this correctly - you want us to create a separate resource for
every inputted <collections> in the URL?
For example, say we fetch the indicator "NY.GDP.MKTP.CD" from the remote by
POSTing to an endpoint: "/hello" and it results in a collection with ID 1.
We can then retrieve this collection with a GET /hello/1. BUT, we cannot retrieve it
with a GET /byebye/1.
HOWEVER - If we then POST the same indicator "NY.GDP.MKTP.CD" to the endpoint
"/byebye" - we should now be able to access it using a GET /byebye/1?
Is this correct?
Reply
Load More Comments
Yipu Ding (/users/z5180553) a day ago (Wed Mar 13 2019 07:04:49 GMT+1100 (AEDT))
Sorry, I still don‘t get the meaning of <collections>. If it is an arbitrary string, then that
would mean we seperate different collections, but isn‘t that what the collection_id in the
layer below is doing?
So shouldn‘t <collections> just be some default string?
Reply
Chaoran Huang (/users/z5022460) a day ago (Wed Mar 13 2019 08:16:20 GMT+1100 (AEDT))
please read https://restful-api-design.readthedocs.io/en/lates... (https://restful-apidesign.readthedocs.io/en/latest/resources.html)
Reply
Chaoran Huang (/users/z5022460) a day ago (Wed Mar 13 2019 08:14:41 GMT+1100 (AEDT))
Collection means a collections of web resources here, the resources are essentially
returned required data.
Reply
Fengting Yang (/users/z5089358) a day ago (Wed Mar 13 2019 05:16:41 GMT+1100 (AEDT))
What does <collections> means? A collection name defined by user?
Reply
Chaoran Huang (/users/z5022460) a day ago (Wed Mar 13 2019 08:16:30 GMT+1100 (AEDT))
please read https://restful-api-design.readthedocs.io/en/lates... (https://restful-apidesign.readthedocs.io/en/latest/resources.html)
Reply
Chaoran Huang (/users/z5022460) a day ago (Wed Mar 13 2019 08:12:15 GMT+1100 (AEDT))
Collection means a collections of web resources here, the resources are essentially
returned required data
Reply
Fengting Yang (/users/z5089358) about 22 hours ago (Wed Mar 13 2019 10:57:34
GMT+1100 (AEDT))
can you please give some possible values on ＜collections>
Reply

因为专业，所以值得信赖。如有需要，请加QQ：99515681 或邮箱：[email protected] 

微信：codinghelp