kirby.directory

Bildnachweis: Photo by Oleg Laptev on Unsplash - https://unsplash.com/photos/QRKJwE6yfJo

This is my first post in English. Since it is not my mother tongue, there will be some mistakes. Forgive me.

I have been a big friend of Kirby CMS for some years now. A lightweight, file-based Content Management System maintained by Bastian Allgeier.

Kirby's greatest strength is the extensibility through plugins. Unlike popular CMS System like Wordpress, Kirby itself has no build-in Plugin Directory. If you need a plugin, you have to search for it on the web, the Kirby Support Forum or had to use this site which existed at that time, but was only rarely maintained.

the Kirby Plugins Github Repository

In 2016 Jens Tornell started a Github Repository which main purpose was the collection of all available Kirby Plugins. The plugins can be filtered by category, license or other attributes using labels. A very time-consuming job, which probably paid off quickly. The repo became the place to go for Kirby plugins. The only drawback was that Github Issues are kind of hard to navigate if you're looking for something special or just lack the knowledge of Github's Search API. In fact - one of the early repo readme.md mentioned that Jens was planning to something:

Future Ideas: A website built with Github API to make the plugins more visually appealing.

Since I have been planning to do a project with Vue.js for a long time, this was the opportunity for me. And I did it. Twice.

First Steps

After learning the basics of Vue.js with help of the documentation and the great tutorial series by Jeffrey Way on laracasts I started with a vue-cli scaffolding with webpack-simple.

I also added vue-router, even though it wasn't necessary. The page consist of only two parts - a list and a detail view.

  • The list view is a list of all items. With pagination and filtered by selected labels.
  • The detail view displays the content of an issue. Including comments and a direct view of the readme of the respective plugin - if it was hosted on Github.

The item-fetching was done with help of axios and the well-documented Github Issue API. For the markdown parsing I included vue-markdown.

Vue-router, axios and vue-markdown saved me a lot of time - Results came in pretty fast 1. In a suprisingly short time I had a working prototype. And a problem.

the first version, purley with the github api and a lot of requests

Rewrite all the things

Well, several problems:

  • The Github Api is without authentification very limited. The Search API only allows 10 request per minute, the issue api is limited by 60 requests per hour. This has the consequence that already 2-3 simultaneous users can shut down the whole site. Try it yourself with the link above - just navigate around and use the search function 2. So if I really want to upload this project and not only use it for myself, this problem had to be solved.
  • I had a lot of redundant functions and props to pass through to various components. I have also often used stuff like $parent to call methods in parent components. Overall it was not really DRY.

The solution for the first problem was easy. I need to drastically reduce the amount of request to the Github API. The most obvious way to me was to use localstorage and save the data from the Github API in the browser without triggering a request with every interaction.

The second problem could not be solved so easily. That's why I wrote to Nils to do a code review, but also to discuss the problem. I have known Nils for several years and have worked on several projects with him. He recommend that I should use Vuex, a solution similar to Redux for React. With vuex I could create a global state with global actions, without passing props or states to components 3.
The best part: I could save the state (with all the plugin data) to localStorage, which has the consequence that I could reduce the request per user significantly.

Nils also told me that it would probably be easier to restart the site directly with vuex in mind instead of integrating it into an existing project. And that's what I did. In the end I was able to take over Markup and SCSS without major difficulties.

Thanks to this article, implementing localStorage with Vuex was a breeze. But then I had a second look and decided to ditch my own implementation and just use vuex-persistedstate as it just worked out better than my solution. Here is a snapshot, hosted on now.sh 4.

this is an early version of the rewrite. If you visit it for the first time, you will have an approximatly loading time of 6~8 seconds.

Finally fixed the request problem. For good.

A last thing to was to contact Jens, the maintainer of the Github Repository. I contacted him and added a preview link to the current status and asked if he would be okay with it if I built on his data. And he was!

That is so cool!! :+1: Yes, release it when you are done. Looks awesome.

Well, that was the best part. Up to this point I could completely live with the fact that I would possibly use this small project only for myself. But these two sentences were the turning point. Thanks Jens! 😎

He also included some ideas and suggestions for improvement, which I liked to implement. After I told him about my localStorage implementation, however, he was right to say that it is still possible to break the site.
The problem is that Github only allows request with 100 items at max. Therefore, a user has not only one request, but a total of 5, since the repo has about 470 entries at this time.
Since I only used the Search API and my 10 requests are reset every 30-60 seconds I considered this only a small problem. But it still bothered me.

I wrote a shell script that gets all items from the server and stores them in a JSON file using jq. This script is executed once an hour using a cronjob on my uberspace server 5, which reduces the effective requests for the user to 0. I was happy.

Btw. this the shell script. As I am quiet a noob in shell scripting and jq, there is propably a lot of space for improvement. Obviously the hardcoded counter (to 6) will be a problem if there are ever more than 600 entries.

#! /bin/bash

# loop over all items. save them to separate json files
for (( counter=1; counter<=6; counter+=1 )); do
    curl -vs "https://api.github.com/search/issues?q=is:open+repo:jenstornell/kirby-plugins&sort=updated&order=desc&per_page=100&page=$counter"| jq -r '.items' > tmp/page-$counter.json
done

# merge all generated json files
jq -s '[.[][]]' tmp/page-*.json > items.json

rm tmp/page-*

That's it

And now - here it is - the Kirby Directory. Matching domain included.

Feedback is very welcome. As long as I don't have a comment function here yet 6, preferably by e-mail or Twitter.
Or - since the project is on Github- with pleasure also directly as an issue.

Please but be easy on me - I'm still learning :)

The "final" version of kirby.directory.

Whats next

The biggest initial work is done. But what happens next? Will I continue to work on the project or will it lie dormant now?
At the moment I have two ideas on the roadmap. For one thing, I would do the same thing again with the Kirby Theme Repo.
On the other hand I have a website here that deals with Kirby, but is not based on Kirby. And that's a little weird. Since the upcoming Version 3 of Kirby will get a rest API, I will probably use Kirby as backend.

  1. Ok, of course it was fast. I used so many well-done third party components. Not sure if I could had done all that by myself.

  2. I know that I could probably have solved the problem with a Github authentication by the user. However, since I only work on the frontend side and my stack is very limited, there is no way to store the secret key safely. And I don't think it's a good idea to let the user register first to use this site. In addition, the rate limit would only increase from 10 to 30 requests. And even this limit can be reached very quickly.

  3. I thought of vuex before, but was convinced its better to focus on just vue.js for my first project.

  4. btw. now.sh is a great service. I had a build up there in seconds! Go try them out!

  5. Sadly, at this moment I had to ditch now.sh.

  6. btw. there are some! Look Mum, I can hard-link! -> https://kirby.directory/#/search/comment/1