UI Concepts for Vector Search

Rethinking UI/UX for Vector Search

Different search backends lead to differing search experiences. This often requires unique methods of search interaction, especially for context-aware, semantic search functionality (like similarity search). While vector search, which leverages a vector database and vector embeddings, offers an alternative to the traditional keyword search, merely using it as a direct replacement doesn't tap into its full potential.

In this article, we'll embark on a journey to transform a basic search experience, introducing more advanced machine learning features tailored for vector search (in other words, we will build an AI application for search). The open source code used in this article is part of our getting started tutorial on github.

The elements discussed in this article can be used in concert to create unique search journeys through novel yet intuitive patterns (especially when working with unstructured data). In the example below we combine semantic filtering with search negations:

The Search Bar

The design of the search bar has largely remained consistent over time. On most websites, the search apparatus consists of a straightforward text input and a button, often accompanied by options to filter and sort results.

While it's straight forward to integrate a vector search backend into this traditional search bar design and achieve high-quality results, the capabilities of vector search extend far beyond simply converting a single piece of text into a vector and computing results.

Running Example

To illustrate the concepts introduced in this article, we'll progressively enhance our demo UI. This demo is derived from the Marqo cloud's getting started guide.

We'll begin with a basic search bar powered by a vector search backend.

While functional, this setup merely allows users to input queries and receive related results. However, this isn't particularly exciting.

Semantic Filtering

Semantic filtering works by modifying the semantic representation of a query. Importantly, this can occur behind the scenes, abstracting complexity away from the user. These filters can be adjusted any time as they are purely semantic and do not rely on any additional metadata like traditional filtering does.

This technique draws inspiration from the way zero-shot classification tasks are executed using CLIP models. An example of this can be seen on OpenAI’s CLIP repo.

At its core, the approach is about enriching the user's query with additional descriptive language (part of the embedding model). This is reminiscent of prompt engineering techniques employed with generative image models. We achieve this by defining a dictionary that associates filters with specific prompt templates.

STYLE_MAPPING = {
    "stylized": "An image of a <QUERY> rendered in a unique, stylized manner",
    "cartoon": "An image in the style of Saturday morning cartoons featuring a <QUERY>",
    "abstract": "An image inspired by modern art, abstractly representing a <QUERY>",
    "pixelart": "An 8-bit or 16-bit pixel art representation of a <QUERY>",
    "grayscale": "A black and white, monochromatic image of a <QUERY>",
    "cyberpunk": "An image of a <QUERY> set in a neon-lit, futuristic cyberpunk world",
    "steampunk": "An image blending Victorian and industrial elements, depicting a <QUERY>",
    "realism": "A hyper-realistic, detailed image of a <QUERY>",
    "surreal": "A surreal, dream-like image of a <QUERY>",
    "popart": "An image in the style of pop art, featuring a <QUERY>",
    "watercolor": "A watercolor painting-like image of a <QUERY>",
    "oilpaint": "An oil painting-like image, rich in texture, of a <QUERY>",
    "glitch": "A glitch-art styled, distorted image of a <QUERY>",
    "neon": "An image of a <QUERY> with vibrant, neon colors",
    "fantasy": "A fantastical, otherworldly image of a <QUERY>",
    "comic": "An image resembling a comic book panel, featuring a <QUERY>",
    "outline": "An outline line art depiction of a <QUERY>",
}

In this setup, <QUERY> is seamlessly replaced with the user's input query. This method allows us to filter search results based on image characteristics without relying on metadata.

The example prompts above were crafted with the help of Open AI’s ChatGPT (based on the gpt generative ai large language model), proving its efficacy for such tasks.

This concept can be extended in many directions such as style based filtering for fashion trends or interior design. It can also be used to control the type and quality of results that are surfaced from search queries. Prompting can be used to modify or expand the query without the users knowledge, for example if your dataset has many images of dubious quality then the query can be prefixed with a string like "high quality professional photo of" to improve the quality of the results.

The Case for Multiple Search Bars

Why confine ourselves to just one search bar? Vectors encapsulate the semantic essence of the text they represent, and their true power emerges when they're combined using elementary vector operations. By taking a weighted average of vectors with positive weights, we can produce a new vector that seamlessly blends the semantics of its contributors. The introduction of negative weights further enriches this dynamic, enabling us to effectively subtract the semantics of one vector from another, offering a nuanced and multi-layered search experience. These variables allow us to unlock the power of vector data.

More of This? Less of That? Search Refinement Mechanisms

We've showcased this approach on our demo for quite some time. Beyond the conventional search bar, we've incorporated two supplementary text input fields. These allow users to amplify or diminish specific aspects of their query. To implement this element the search terms from the three bars are combined in the backend via a weighted average. The negative term is given a large negative weight in order to steer the results away from that part of the vector space. The positive search term is modified by prepending the main search term to it, this helps keep the search aligned with the main query and avoids having results that might match the main query and positive term independently.

Marqo handles most of this for you, for the example below the query appears as follows:

query = {
    "shoes": 1.0,
    "shoes, mens business": 0.6,
    "lacers": -1.1
}

While the ability to add or negate terms might be absent in traditional search engines, for vector search, it's an intrinsic feature, amplifying its potency and adaptability.

Personalisation through Vector Search

Search as a Recommender

Vector search paves the way for content recommendations without the need for external systems like collaborative filtering or heuristic-based recommenders (or the pain of productizing “traditional” machine learning models). Additional query terms or vectors of existing documents in the index can be harnessed as query expansion terms, steering search results towards analogous items.

Recommendations can be broadly categorised into two types:

• Intra-category Recommendations: These suggest items from the same category. For instance, recommending another watch based on a user's preference for a specific watch model.
• Inter-category Recommendations: Here, suggestions span different categories. An example might be recommending a couch influenced by a user's affinity for a certain rug and coffee table.

The documents used for recommendations can be specified by the user as is done in our example or they can be included implicitly by looking at a current shopping cart, purchase history, or browsing history. In the examples below we add additional images into the query which are embedded by CLIP models alongside the query term and then combined with a weighted average. Another effective way to implement this is the use the vectors of the original documents which may contain additional information (in the case of this example data it would include brand, name, and image in one vector per product).

Inter-category recommendation:

Intra-category recommendation:

‍

Custom Instructions

A design pattern that emerged with GPT is the introduction of custom instructions. Here, the system prompts users to share specific details, facilitating more tailored results. These instructions can come explicitly from the user as is done here however they can also be derived from existing information at hand such as a user profile of interactions history.

In the realm of search, these can manifest as persistent query expansions linked to individual users. The custom instructions are combined with the query with a weighted average.

Now, users can guide the search engine based on their unique preferences. This guidance can be explicit, such as "I'm partial to the color orange," or inferred from personal tidbits like "I love skateboarding" or "Hiking is my favourite hobby!".

Tying Vector Search into Your Applications

While not all techniques discussed may be applicable to every use case or dataset, they hold the potential to significantly enhance the real-time search experience for users.

Mathematical operations on vectors can be adeptly abstracted, paving the way for intuitive interactions through natural language. Such methods can be harnessed to create novel search avenues or to underpin recommendation systems without revealing their intricate workings to the end-users.

As organisations make the shift from traditional keyword search systems to more advanced vector search systems, giving due consideration to the user experience becomes paramount. In essence, a holistic approach, where UX and search go hand in hand, can prove to be a game-changer.

To learn more about how Marqo can support this use case and other search use cases, request a demo or check out our docs.