Current status of ItemsAPI and related tools

Introduction

I started working on ItemsAPI in 2015 when I was living in Germany (Berlin). I always liked to organize information for specific topics. Imagine various atlas books like atlas of birds, atlas of herbs, atlas of excercises etc in a digital way with nice UX.

That's why I created ItemsAPI. I wanted to have a tool with simple interface for creating, updating, deleting and listing items. The generic API with search features repeatable accross booking systems, meta search engines, eCommerce systems, etc.

After two years I've created quite much tools related to my original goal - organizing information in a easy way. Many of them were failure and waste of time but from today perspective they were also invaluable lessons about learning new technologies or various processes like lean or kaizen.

All in all after so many mistakes the next tools I do are better and simplier. This knowledge is also useful for working with customers.

Below I will give you information about current status of my open source projects.

ItemsAPI

https://github.com/itemsapi/itemsapi - it's an API over Elasticsearch written in Express.js. It's focused on simplicity for creating full text search, facets, sorting and items recommendation only based on your configuration in JSON. I've used this technology for a few websites i.e. http://devteams.co/ and also I've used it for the first iterations of https://shoprank.co/ which now needs more sophisticated solutions.

The biggest problem here for me is complicated codebase. It's not intuitive and not easy to work with. I am also not happy of many different database dependencies like Redis or MongoDB besides Elasticsearch. From today perspective I also find it as a mistake to start API rather than higher level client for Elasticsearch in Node.js. The second solution would be much simplier and it could also solve the problem. I started API because I was a little bit hyped of the frontend technologies like Angular or React and Dockerization trend, like you can put all API technologies into docker and it works.

The plan for the ItemsAPI - I will keep it now as it is and if there will more needs for such a API I will let it evolve. Maybe also creating next API version around ElasticItems which is much simplier.

ElasticItems

https://github.com/itemsapi/elasticitems - it's higher level client for Elasticsearch written in Node.js. It's actually the same as ItemsAPI above but it's just a client in Node.js. It's simplier for me because it doesn't need many abstractions layer like ES client -> API -> API client, it is just Elasticsearch client. It doesn't also require running additional server like ItemsAPI does. It is just connecting with Elasticsearch directly. One drawback it works only with Node.js.

I am quite happy of this solution. It has simple interface and it is easy to start with. I am using it with http://shoprank.co/ and also for a few commercial enterprise projects. Mostly as internal research and business intelligence tools.

ItemsJS

https://github.com/itemsapi/itemsjs - this is full text, faceted search engine in JavaScript. It has somehow similar interface to ItemsAPI and also similar JSON responses. Before creating it I have read / scanned some theory about search engine. https://www.amazon.com/Search-Engines-Information-Retrieval-Practice/dp/0136072240. It was interesting to see how algorithms works internally. I had lot of algorithms classes at my University but those kinds of algorithms (i.e. inverted search) were quite new for me.

The reason for creating it was to make something cheaper than Elasticsearch. Also something which works perfect for small requirements (up to 1000 items) and cost nothing as it is backend-free solution. The next reasons was a curiosity if it is possible to create fast solution in pure JavaScript and if I am able to do it at all. It was a big challenge for me. I've also learnt lot. When I was testing it from Node.js I've naturally understood the needs for sharding, multi-threading. I am also more aware now about Node.js limits.

The next time if I decide to make something similar as an experiment but for bigger dataset I will try Rust (https://www.rust-lang.org/rn-US/) which is relatively easy to work with and has performance similar to C++. It's sometimes even a few times faster than Java which is the language of Elasticsearch. For the performance stat about Rust vs Java you can look here https://benchmarksgame.alioth.debian.org/u64q/compare.php?lang=rust&lang2=java

In terms of the next features I will consider making items similarity which is based on collaborative filtering (https://en.wikipedia.org/wiki/Collaborative_filtering) algorithm, also maybe some caching and pre-computed optimization to make even faster. If you want to see a real demo of ItemsJS in action you can read this article Search engine with facets in JavaScript

Elasticbulk

https://github.com/itemsapi/elasticbulk - the Node.js convenient tool for adding data in bulk to Elasticsearch directly using data in JSON or by streams from PostgreSQL, MySQL or MongoDB. The interesting thing I got a few feedback from people using it and I didn't make any marketing about it. Probably the power of github network effect.

It was created as a needs to index a few millions of records to Elasticsearch in an easy and fast way. Indexing around million of records to Elasticsearch takes right now about 5-10 minutes (depends on network speed and hardware). The plan now is to make it faster by being more asynchronous while still using streams and also keep supporting all Elasticsearch versions 1.x, 2.x and 5.x. The mocha test framework is doing a good job here.

Other

There are also another tools which I didn't mention in the article like https://github.com/itemsapi/starter or https://github.com/itemsapi/dashboard which I'll probably not develop any more. I've created them experimentally and I'll invest time in the most useful and promising tools. You can find some part of the base code useful though.

Future

In the beginning I was focused mostly on ItemsAPI but now and in the future I will focus on different tools related to search and organizing information. Developing open source tools, products and maybe also providing use cases and good patterns as a form of blog posts.