Different search backends lead to differing search experiences. This often requires unique methods of search interaction. While vector search offers an alternative to the traditional keyword search, merely using it as a direct replacement doesn't tap into its full potential.
In this article, we'll embark on a journey to transform a basic search experience, introducing more advanced features tailored for vector search. The code used in this article is part of our getting started guide.
The elements discussed in this article can be used in concert to create unique search journeys through novel yet intuitive patterns. In the example below we combine semantic filtering with search negations:
The design of the search bar has largely remained consistent over time. On most websites, the search apparatus consists of a straightforward text input and a button, often accompanied by options to filter and sort results.
While it's straight forward to integrate a vector search backend into this traditional search bar design and achieve high-quality results, the capabilities of vector search extend far beyond simply converting a single piece of text into a vector.
To illustrate the concepts introduced in this article, we'll progressively enhance our demo UI. This demo is derived from the Marqo cloud's getting started guide.
We'll begin with a basic search bar powered by a vector search backend.
While functional, this setup merely allows users to input queries and receive related results. However, this isn't particularly exciting.
Semantic filtering works by modifying the semantic representation of a query. Importantly, this can occur behind the scenes, abstracting complexity away from the user. These filters can be adjusted any time as they are purely semantic and do not rely on any additional metadata like traditional filtering does.
This technique draws inspiration from the way zero-shot classification tasks are executed using CLIP models. An example of this can be seen on OpenAI’s CLIP repo.
At its core, the approach is about enriching the user's query with additional descriptive language. This is reminiscent of prompt engineering techniques employed with generative image models. We achieve this by defining a dictionary that associates filters with specific prompt templates.
In this setup, <QUERY> is seamlessly replaced with the user's input query. This method allows us to filter search results based on image characteristics without relying on metadata.
The example prompts above were crafted with the help of ChatGPT, proving its efficacy for such tasks.
This concept can be extended in many directions such as style based filtering for fashion trends or interior design. It can also be used to control the type and quality of results that are surfaced. Prompting can be used to modify or expand the query without the users knowledge, for example if your dataset has many images of dubious quality then the query can be prefixed with a string like "high quality professional photo of" to improve the quality of the results.
Why confine ourselves to just one search bar? Vectors encapsulate the semantic essence of the text they represent, and their true power emerges when they're combined using elementary vector operations. By taking a weighted average of vectors with positive weights, we can produce a new vector that seamlessly blends the semantics of its contributors. The introduction of negative weights further enriches this dynamic, enabling us to effectively subtract the semantics of one vector from another, offering a nuanced and multi-layered search experience.
We've showcased this approach on our demo for quite some time. Beyond the conventional search bar, we've incorporated two supplementary text input fields. These allow users to amplify or diminish specific aspects of their query. To implement this element the search terms from the three bars are combined in the backend via a weighted average. The negative term is given a large negative weight in order to steer the results away from that part of the vector space. The positive search term is modified by prepending the main search term to it, this helps keep the search aligned with the main query and avoids having results that might match the main query and positive term independently.
Marqo handles most of this for you, for the example below the query appears as follows:
While the ability to add or negate terms might be absent in traditional search engines, for vector search, it's an intrinsic feature, amplifying its potency and adaptability.
Vector search paves the way for content recommendations without the need for external systems like collaborative filtering or heuristic-based recommenders. Additional query terms or vectors of existing documents in the index can be harnessed as query expansion terms, steering search results towards analogous items.
Recommendations can be broadly categorised into two types:
The documents used for recommendations can be specified by the user as is done in our example or they can be included implicitly by looking at a current shopping cart, purchase history, or browsing history. In the examples below we add additional images into the query which are embedded by CLIP models alongside the query term and then combined with a weighted average. Another effective way to implement this is the use the vectors of the original documents which may contain additional information (in the case of this example data it would include brand, name, and image in one vector per product).
A design pattern that emerged with GPT is the introduction of custom instructions. Here, the system prompts users to share specific details, facilitating more tailored results. These instructions can come explicitly from the user as is done here however they can also be derived from existing information at hand such as a user profile of interactions history.
In the realm of search, these can manifest as persistent query expansions linked to individual users. The custom instructions are combined with the query with a weighted average.
Now, users can guide the search engine based on their unique preferences. This guidance can be explicit, such as "I'm partial to the color orange," or inferred from personal tidbits like "I love skateboarding" or "Hiking is my favourite hobby!".
While not all techniques discussed may be applicable to every use case or dataset, they hold the potential to significantly enhance the search experience for users.
Mathematical operations on vectors can be adeptly abstracted, paving the way for intuitive interactions through natural language. Such methods can be harnessed to create novel search avenues or to underpin recommendation systems without revealing their intricate workings to the end-users.
As organisations make the shift from traditional keyword search systems to more advanced vector search systems, giving due consideration to the user experience becomes paramount. In essence, a holistic approach, where UX and search go hand in hand, can prove to be a game-changer.