Search API for movies [Tutorial]

Demo dashboard for movies API is here (angular.js)
Example has only 8000 movies but it can work with millions of movies with the same response time.
Note: This tutorial is from November 2015. We have improved many things and some information may be not up-to-date. Sorry for that.

Do you think is it possible to create RESTful API for movies search engine with full text search and powerful filters from scratch in less than 1 hour ?

Yes, with ItemsAPI it is possible! I will show you how.

Why you should finish tutorial ?

It will be probably your first search engine for movies with generated REST endpoints:

GET /api/v1/movies?query&per_page&page&sort
GET /api/v1/movies/:id
PUT /api/v1/movies/:id
DELETE /api/v1/movies/:id
POST /api/v1/movies

You will learn how ItemsAPI is working. It allows you for building other search engines extremely fast, i.e. for:

  • songs
  • clubs
  • cars
  • places
  • restaurants
  • libraries
  • recipes, etc..

1. ItemsAPI installation

I assume that you have node.js, npm and elasticsearch installed. If not, below is short instruction (tested on Ubuntu 14.04)

Node.js installation:

$ sudo apt-get install curl
$ curl -sL https://deb.nodesource.com/setup_4.x | sudo -E bash -
$ sudo apt-get install -y nodejs
$ sudo npm update npm -g

Elasticsearch installation and running:

$ sudo apt-get install openjdk-7-jre
$ wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.2.deb
$ sudo dpkg -i elasticsearch-1.7.2.deb

Clone ItemsAPI repository from github:

$ git clone https://github.com/itemsapi/itemsapi.git

Install all node.js packages:

$ npm install

2. Adding configuration

Add ./config/local.json file:

{
  "collections": {
    "db": "json",
    "filename":  "./config/collections.json"
  }
}

It overrides fields from ./config/root.json and defines where is stored our collections json database. Create now collections file ./config/collections.json:

[
  {
    "name": "movie",
    "schema": {
      "name": {"type": "string", "store": true },
      "image": {"type": "string", "display": "image" },
      "year": {"type": "integer", "store": true },
      "votes": {"type": "integer", "store": true },
      "director": {"type": "string", "index": "not_analyzed", "store": true },
      "actors": {"type": "string", "index": "not_analyzed", "display": "array", "store": true },
      "tags": {"type": "string", "display": "array", "store": true },
      "rating": {"type": "float", "store": true }
    }
  }
]

This is our simple configuration - currently only for searching. We will improve it in the last part of tutorial with filters and sorting.

3. Importing data

Create file ./movies.json and copy content from https://github.com/itemsapi/itemsapi-example-data/blob/master/movies.json

$ wget https://raw.githubusercontent.com/itemsapi/itemsapi-example-data/master/movies.json

Import it to the system using console. It will save data to the elasticsearch automatically.

$ node cli.js --import --collection=movie --file ./movies.json

4. Testing your API

Firstly run your app manually (you can use pm2 process manager too):

$ node app.js

For testing API you can use great Postman plugin for google chrome or test with cURL like:

$ curl http://localhost:3000/api/v1/movie

5. More advanced configuration

To generate filters and sortings in our API, we need to change ./config/collections.json:

[
  {
    "name": "movie",
    "schema": {
      "name": {"type": "string", "store": true },
      "image": {"type": "string", "display": "image" },
      "year": {"type": "integer", "store": true },
      "votes": {"type": "integer", "store": true },
      "director": {"type": "string", "index": "not_analyzed", "store": true },
      "actors": {"type": "string", "index": "not_analyzed", "display": "array", "store": true },
      "tags": {"type": "string", "display": "array", "store": true },
      "rating": {"type": "float", "store": true }
    },
    "aggregations": {
      "actors_terms": {"type": "terms", "field": "actors", "size": 10, "title": "Actors"},
      "years_range": {
        "type": "range",
        "field": "year",
        "title": "Years range",
        "ranges": [
          {"lte": 1950, "name": "1950 - 1960"},
          {"gte": 1950, "lte": 1960, "name": "1950 - 1960"},
          {"gte": 1960, "lte": 1970, "name": "1960 - 1970"},
          {"gte": 1970, "lte": 1980, "name": "1970 - 1980"},
          {"gte": 1980, "lte": 1990, "name": "1980 - 1990"},
          {"gte": 1990, "lte": 2000, "name": "1990 - 2000"},
          {"gte": 2000, "lte": 2010, "name": "2000 - 2010"},
          {"gte": 2010, "name": "> 2010"}
        ]
      }
    },
    "sortings": {
      "rating": {
        "title": "Best rating",
        "type": "normal",
        "order": "desc",
        "field": "rating"
      },
      "most_votes": {
        "title": "Most votes",
        "type": "normal",
        "order": "desc",
        "field": "votes"
      }
    }
  }
]

It will generate filters and allows for easy filtering in GET query

You can test now with additional parameters like sort

$ curl http://localhost:3000/api/v1/movie?sort=most_votes

Below is example of bigger query (you should use urlencode for it)

GET http://localhost/api/v1/movie/find
?page=1
&per_page=12
&sort=most_votes
&aggs={"actors_terms":["Robert De Niro"],"years_range":["1980 - 1990","1990 - 2000"]}

6. Summary

Congratulations! You’ve created a search engine for movies! Add comment if you have some issue, idea or you want to say hello.