Image viewing

The way the images are loaded is relatively unique. As this is an application to display files from the same local machine, they are loaded using file:/// URLs. Normally this results in the images not loading, due to security policies. And rightfully so! However, these policies can be circumvented in Firefox by some tinkering.

The reason I built it this way is because it is hugely more performant than streaming it through the Node.js backend. In fact, with my media collection on my SSD, I haven't had the need yet to create thumbnails. The images on the search page are loaded full resolution, scaled down by the browser. This clearly isn't ideal but works well enough for me as I'm developing this for myself. Testing it with media on a HDD, I definitely felt the need for thumbnails and do intend to support it at some point.

I also would like to have an option to stream the images through the backend anyway, so you could potentially navigate the application using a mobile device as well. But we'll see if I can find the time for that!

AI suggested tags

As it turned out to be pretty tedious to manually search for the right tags, I figured I could let AI do some of the heavy lifting. But, because AI is imperfect, I didn't want it to automatically add all results to all images. Hence, the current solution is to show the AI tags in the "add tag" dialog whenever it is shown. There you can simply click all the tags you think apply, and save only those.

This is still a lot of work, so in the future I will probably store AI generated tags for every image anyway. However, I will keep them separated from the "humanly confirmed" tags, allowing for searches with and without AI tags.

In terms of tech, this system is incredibly simple. When a request is made to retrieve AI tags for a certain image, the request is processed by our Node.js (Fastify) backend. It then forks a new Python process, which is a very barebones script that does the following:

1. Loads the (pretrained) model & related vocabulary lists.
2. Opens the image formats it to be processed by the model (change resolution to 512x512, remove alpha color channel).
3. Pass the image & model to Tensorflow and wait for output.
4. Take resulting values and find the corresponding tags.
5. Format the corresponding values as JSON and return it in the stdout.

Node.js then reads the stdout, and returns the result to the client. It surprised me how fast the model is, as I usually have the tags within a second of the inital request! (on an RTX3070 8GB)

File discovery & Database structure

When it comes to crawling the target folder for media, I have made use of a library called fdir. It's very performant and allows me to quickly scan for any changes since the last time it was crawled. Any changes (additions, removals, moves) are detected and processed accordingly in the database. Each file has a row in the database, uniquely identified based on path, inode and modification timestamp.
As a baseline, I prevent opening/reading the actual files as this would slow down the process immensely. I have some code in place to do this to retrieve metadata and md5 hash, but generally prevent running this code unless I have to. In the future I will probably run this in a background thread, controllable by the user.

The database is structured in a very simple way. Tags have their own table, each with an ID and label. Then, file - tag relations, each tag variant has a junction table with two columns: The tag ID and the file ID. When querying, a JOIN clause is added to get the file info for the queried tags.