Responding JSON, XML and CSV

Good question. In PythonOnWheels these three formats are the default formats (besides rendering HTML views) that any model can be serialized to by default. These are implemented because they are the most commonly these days. There will be many cases where you can access data as csv (think of all the Excel files out there) or retrive data from other services and APIs (mostly) as JSON these days. XML is very common in enterprise environments ... 

But anyway. If you know how to respond one of those, you can easily write your own encoder to send anything you like.

The scenario for this tutorial

• We will create a small app with a tinyDB database and a test model. 
• We will generate a handler and a REST API
• We will see how to respond with data from our Database as JSON, XML and CSV 

Prerequisites: (~2 Minutes)

• Just generate our default todo application. If you don't know how, just follow the short getting started info.  
• I also assume that you have worked with models and handlers before. If not, just read these short intros (5 minutes) Working with models (NoSQL) Adding a handler to your todo app
  

Let's generate our testmodel: 

python generate_model.py -n todo -t tinydb

As you know this generates a todo.py model file in your models\tinydb directory, which looks like this.

class Todo(TinyModel): 
    # 
    # Use the cerberus schema style  
    # which offer you immediate validation with cerberus 
    # http://docs.python-cerberus.org/en/stable/validation-rules.html 
    # types: http://docs.python-cerberus.org/en/stable/validation-rules.html#type 
    # 
    schema = { 
        'title' :   { 'type' : 'string', 'maxlength' : 35 }, 
        'text'  :   { 'type' : 'string' }, 
        'tags'  :   { 'type' : 'list', "default" : [] }, 
        "votes" :   { "type" : "integer", "default" : 0 }    
        } 

We will leave the schema definition unchanged in this case since the focus is to respond JSON, XML and CSV.

Next: generate a handler

python generate_handler.py -n todo -t tinydb --rest

This will create a handler for us with the following nice features:

• it is named todo (-n todo) [ok, no too spectecular, but wait for the other two ;) ]
• it will be automatically linked to the todo model (by convention over configuration. Since we named it todo as well and gave it the type -t tinydb)
• it will create a REST API for us automatically to create, read, update, delete (CRUD) todo models.  (check the generating REST handlers tutorial, if you want to know more about the generated routes and methods.)

That's pretty nice for a one-liner !

Now let's add some data to our Database

We generated our model and access to the embedded tinyDB is preconfiguired for us in the todo/config.py file (section database.tinydb)

so open a python shell in your virtualenv and execute the following to produce 10 todos.

>>> from models.tinydb.todo import Todo

Create 10 todos in a loop:

>>> for x in range(0,10): 
...     t=Todo() 
...     t.title="todo #" +str(x) 
...     t.upsert() 
... 
insert, new eid: 1 
insert, new eid: 2 
insert, new eid: 3 
insert, new eid: 4 
insert, new eid: 5 
insert, new eid: 6 
insert, new eid: 7 
insert, new eid: 8 
insert, new eid: 9 
insert, new eid: 10 
>>> 

Now we have everything we need.

• A model
• A handler 
• A REST API 
• And 10 todos saved in our Database

But how do we serve JSON, XML or CSV now ?

This is a two step process:

• First: We need to find out which format the client wants

In PythonOnwheels there are two ways where the server checks which format a clients request wants:

• The Accept HTTP-Header
• The URI Format: /host:port/ressource.format known from RAILS.
• Example: http:/localhost:8080/todo.json => will call the todo.list method but encode the response to JSON

Both are absolutely valid. The .format is actually checked first. 

• Second: model instances need to be able to serialize to the given format.

Nothing easier than this. Every PythonOnWheels model instance can encode (or serialize) itself to JSON, CSV or XML. The flow is always the same. The model converts it's data to JSON and calls and encoder to transform into the format you want. Because PythonOnWheels altready includes encoders for XML and CSV you can use those out of the box (batteries included ;)

Before we go into the details of where the encoders are, where the definition is which encoder to choose for which format and how you can extend that with your own encoders let's first check that this really works.

Start the server

python server.py

You will see that the server will wait for requests on localhost:8080

Now Let's Test our API using the HTTP Accept header

First we go for JSON

I will use curl to give you a more raw view and the superb Insomnia REST Client to show you the call and results in much a nicer formatted way.

• Curl Request using Accept Header:

curl -H "Accept: application/json" -X GET http://localhost:8080/todo

• Response on the command line: 

{"message": "todo, index", "http_status": 200, "prev": null, "next": null, "data": [{"title": "todo #0", "text": "", "tags": [], "votes": 0, "id": "0044e328-b544-4d5d-baa9-1d2343b6fe1e", "_uuid": "0044e328-b544-4d5d-baa9-1d2343b6fe1e", "created_at": "2019-08-15 14:37:31", "last_updated": "2019-08-15 14:37:31"}, {"title": "todo #1", "text": "", "tags": [], "votes": 0, "id": "9f9769d6-ee50-4938-8633-c0b8452ebddf", "_uuid": "9f9769d6-ee50-4938-8633-c0b8452ebddf", "created_at": "2019-08-15 14:37:31", "last_updated": "2019-08-15 14:37:31"}, {"title": "todo #2", "text": "", "tags": [], "votes": 0, "id": "aa057a26-5c87-4acf-9811-b8c8d514edee", "_uuid": "aa057a26-5c87-4acf-9811-b8c8d514edee", "created_at": "2019-08-15 14:37:31", "last_updated": "2019-08-15 14:37:31"}, {"title": "todo #3", "text": "", "tags": [], "votes": 0, "id": "e9549346-50d4-4b28-bce9-5ce2064175aa", "_uuid": "e9549346-50d4-4b28-bce9-5ce2064175aa", "created_at": "2019-08-15 14:37:31", "last_updated": "2019-08-15 14:37:31"}, {"title": "todo #4", "text": "", "tags": [], "votes": 0, "id": "7d976057-f125-4c6d-8336-3e6a8dcc557a", "_uuid": "7d976057-f125-4c6d-8336-3e6a8dcc557a", "created_at": "2019-08-15 14:37:31", "last_updated": "2019-08-15 14:37:31"}, {"title": "todo #5", "text": "", "tags": [], "votes": 0, "id": "e5121361-cded-40c4-a34f-96d128884e7a", "_uuid": "e5121361-cded-40c4-a34f-96d128884e7a", "created_at": "2019-08-15 14:37:31", "last_updated": "2019-08-15 14:37:31"}, {"title": "todo #6", "text": "", "tags": [], "votes": 0, "id": "ec796ee5-97d7-4f73-83a4-b70fe39337c1", "_uuid": "ec796ee5-97d7-4f73-83a4-b70fe39337c1", "created_at": "2019-08-15 14:37:31", "last_updated": "2019-08-15 14:37:31"}, {"title": "todo #7", "text": "", "tags": [], "votes": 0, "id": "a329337e-e7a2-4720-8866-027ed71bd1db", "_uuid": "a329337e-e7a2-4720-8866-027ed71bd1db", "created_at": "2019-08-15 14:37:31", "last_updated": "2019-08-15 14:37:31"}, {"title": "todo #8", "text": "", "tags": [], "votes": 0, "id": "2e8f3699-bd50-4681-9952-8bf2c32ca04c", "_uuid": "2e8f3699-bd50-4681-9952-8bf2c32ca04c", "created_at": "2019-08-15 14:37:31", "last_updated": "2019-08-15 14:37:31"}, {"title": "todo #9", "text": "", "tags": [], "votes": 0, "id": "3c3a96f3-8358-4064-a46d-d9c339f4b9f9", "_uuid": "3c3a96f3-8358-4064-a46d-d9c339f4b9f9", "created_at": "2019-08-15 14:37:31", "last_updated": "2019-08-15 14:37:31"}]}

Not very beutifully formatted but clearly a JSON response of our 10 todos. 

Lets use Insomnia to make this a little more readable:

• URL is the same as in the curl call
• I also set the Accept Header to application/json.
•  You can see the result on the right.

Let's request XML

Just change the Accept Header from application/json to 

application/xml

And finally CSV

Again: just change the Accept header to: 

application/csv

Nice and simple. Just works.

For JSON and XML you can see that the actual format is not only responding with the pure data but with a schema like this:

Message format for JSON:

 
{ 
  "message": "todo, index", 
  "http_status": 200, 
  "prev": null, 
  "next": null, 
  "data": [...] 
} 

Message format for XML:

 
<root> 
    <message type="str">todo, index</message> 
    <http_status type="int">200</http_status> 
    <prev type="null"></prev> 
    <next type="null"></next> 
    <data type="list"> 
        <item type="dict"> 
        </item> 
        ... 
    </data> 
</root> 

Message format for CSV:

This makes no sense for CSV so you just get the data here.