Fifteen Years Ago, I Became an Evernote User
Fifteen years ago, I became an Evernote user. A happy time, when my thoughts were all about startups, Silicon Valley, and technology. And although today's Obsidian evangelists are absolutely certain that they are the first to create a second brain, allow me to say: I was building my second brain in the 2010s using Evernote and Moleskine. And Evernote's OCR search through images and documents was absolutely magical. It's no coincidence, apparently, that before that, engineers had been working on OCR solutions for the US government.
This article is about how you can build a local search for images and photos that will work without large corporate services, without LLM/SaaS, and without sending personal data outside.
What Evernote Meant to Me
Of course, as is the case now, behind a fairly simple but good product there must be something more. An idea or distribution. After all, it's a note-taking service. For me, it was the idea that overseas, a bearded Russian guy, Stepan Pachikov, a tech guy, took it, built it, and made some crazy money from it. I used it, recommended it to friends, participated in his hackathons, and even won prizes there.
Of all of Evernote's features, only three were the most valuable in my opinion. But it was precisely these that kept me as a user until 2022 — the year I abandoned SaaS services from large companies. More than ten years of subscription...
So, what were these anchor features:
Search through documents, images, and photos
An incredibly useful thing. It's clear that with proper tag and folder organization, you can find any, even the oldest, note or document just by navigating through some system. But if you add very accurate search to the system, the process becomes much more pleasant. I had many medical documents that were very easy to find by keywords present in the image or scan. And I didn't have to manually perform preliminary indexing and organization just for the sake of organization. Just adding one tag — and the OCR search did its job. Yes, at that time the search was indeed just OCR. But we'll come back to this.
Alarm on notes
I had several successful use cases: the alarms would go off a couple of times a year and helped me not miss the right moment, bringing back the context of what I needed to do. Now I use email for this.
Skitch (screenshot annotator)
In my profession, I need to communicate with other people, showing certain elements of a product and leaving comments on them. Similar services existed and still exist. The Linux alternative still can't draw normal arrows.
Recently, while browsing through screenshots on my PC in search of one specific image, I thought how cool it would be to have search like in Evernote. And yes, I know that there are and have been many similar solutions. Not only commercial products but also open-source ones — just search on GitHub. But all of them, as always, are "not the same," "the old one was better." Although, of course, it wasn't actually better; the search in Evernote worked on the OCR principle. Obviously, since the company still exists, I dare assume that it has been improved to hybrid search. That is, a combination of OCR (search by text recognized from an image) and semantic search, i.e., search by embeddings obtained from the image using some model. We'll be talking about small models that can be run on ordinary user hardware and have a very limited focus on what they do.
What Do I Want from Image Search?
If an image shows a cat or a metal grate, I would like to be able to write that query and get similar images. I would really like to search for images by color palette. You might ask why, but it's incredibly convenient. In my use cases, when I remember the design of a service clearly, searching for "profile form" might not help, but "purple color + profile form" will find the result. I want to search using not only English text but also Russian in queries. So that I can write "metal grate" or "металлический люк" and get similar results. Of course, the search should also account for text on the image if there's a lot of it, as well as the meaning of that text. For example, I took a screenshot of a YouTube recipe for pizza dough, which was written with a marker on a board. The text in the image contains the words water, flour, and salt. Ideally, I would want to find it by the query "pizza recipe" or just "recipe." Also, the search should be fast. Okay, I'm willing to wait for long indexing and preliminary analysis of the image database, but the search itself should be near-instantaneous. So, is it possible to do this using modern models without using external services and APIs?
Step 1. Semantic Search.
Or search by image embeddings and the embedding obtained from the user's text query. To solve this, we need a multimodal embedding model, preferably with an MIT/Apache license. And there is one: Nomic Embed Vision. Like any trendy transformer-based model with a BERT architecture. The beauty of this model for our engineering task is that text and images as input produce vectors from the same space. We can search for text by image, image by image, or image by text. A downside is that Russian language queries don't work optimally, but we'll fix that too, later.
Let's figure out how I'm going to test the quality of the models adopted in the search pipeline. I'll do it using a qualitative method — whether I like it or not — on my local folder with screenshots. There are about 500 of them in total. The global goal is to feed my entire knowledge base, notes, screenshots, and references, but for now, we'll limit ourselves to searching through screenshots. I had a list of images that I wanted to find in my screenshot folder. And the search should find them. No complex benchmarks or methodology, I just need to enjoy the search.
So, we take Rust, take the ONNX Nomic Vision models, and see what happens.
The very first result worked perfectly. Of course, it was a coincidence, but a fortunate one. I had a screenshot from a YouTube video on how to weld a manhole cover in a parking lot so that the management company could bury a pump there. Finding this image was instantaneous, having only Semantic search with a single model at hand. The embeddings themselves can be stored in any database with vector support and search capabilities. SQLite has a plugin, and there's also LanceDB. They are ideal because they don't have excessive complexity in support and interaction. Remember, SQLite is installed on practically every smartphone in the world and is an absolutely bulletproof database.
The only problem was with queries in Russian. I tackled this head-on and selected a small model for translating between Russian and English. https://huggingface.co/Teradata/opus-mt_tiny_rus-eng Queries started working significantly better. Of course, we lose some quality with each such step. If the model were already multilingual, it would be better, but we're taking the simple path. Training a similar model would require not only a stack of 16+ H100/H200 GPUs but also extensive work with datasets, which is unacceptable for my task. However, with modest effort (by lengthening the model pipeline), the target result improved significantly, and it became possible to search for a metallic object using the word "метал" (metal) as well as "metal."
Step 2. Search by Colors.
This is my long-standing dream of how I would like to search through my library of image references. And colors fit perfectly into a vector form. To implement this search, we need to do two things. First, extract dominant colors or a kind of palette from the image. Second, encode this color palette as a vector. Then we can simply search for similar vectors, exactly as I did before with embeddings from the Nomic Vision model. To extract the palette, you can use different algorithms, but as I like to approach such tasks, I start with the simplest solutions that can yield a significantly good result in ML. There's k-means clustering, and that's what I used. There are several good libraries for Rust; I used https://crates.io/crates/kmeans_colors. For each image, k-means extracts 5 dominant colors in the Lab color space, concatenates them into a single flat vector, and indexes them in LanceDB. When a user searches simultaneously by text and color, the system first performs a semantic search (vector or hybrid), and then re-ranks the results based on a combined score. In this score, semantic relevance is primary, and color proximity is a refining factor.
Let's test the color search.
Bam! It works, of course, in combination with semantic search. For example, I want to search for login UX screenshots. And I remember that the site I'm looking for is orange or red.
Works like a charm. Exactly what I wanted. In the my next youtube video, I'll talk about what else can be added to this pipeline to make it even better. Let me remind you that the search already works absolutely magically, and we haven't even done OCR yet — we haven't recognized the text on the images and added it to so-called hybrid search.
More than 20 years have passed since work on Evernote began. If at the beginning of this SaaS journey, image search technology was indeed a formidable weapon against competitors, over time the technology has become very cheap to develop and implement. Yes, of course, Tesseract OCR became open-source decades ago too. But it's like heaven and earth compared to what we can now run locally on a user's machine and with what level of quality.
We are supposedly supposed to be entering an era of thousands and thousands of new applications and services that can solve old problems much more simply, quickly, and better. LLMs seem to have solved the software problem and left only the problem of the idea. But where are these thousands of new startups?
P.S.
You can try the result by downloading the application via the link. The application is currently only available for macOS, Linux and Windows. Battle tested only on Linux yet.. I am continuing to work on it right now and don't yet see what will become of it. For now, a working idea has emerged: to create my own Skitch. But with entirely local models for search and image processing. If you're interested, you can subscribe at https://zatsepin.dev/subscribe or on YouTube at https://www.youtube.com/@yurizatsepin