博客

使用 ML 生成过滤器和切面

探索在搜索体验中使用 ML 模型自动创建过滤器和切面与传统硬编码方法的利弊。

Andre Luiz
相关性ML 研究

过滤器和切面是用于完善搜索结果的机制，可帮助用户更快地找到相关内容或产品。在传统方法中，规则是人工定义的。例如，在电影目录中，流派等属性是预定义的，可用于筛选器和切面。另一方面，通过人工智能模型，可以自动从电影特征中提取新的属性，使整个过程更加动态和个性化。在本博客中，我们将探讨每种方法的优缺点，重点介绍它们的应用和挑战。

筛选器与分面

在开始之前，我们先来定义一下什么是过滤器和切面。过滤器是用于限制结果集的预定义属性。例如，在市场中，甚至在进行搜索之前就可以使用筛选器。用户可以先选择一个类别，如"Video games" ，然后再搜索"PS5" ，将搜索范围缩小到更具体的子集，而不是整个数据库。这大大增加了获得更多相关结果的机会。

筛选

面板的工作原理与筛选器类似，但只有在执行搜索后才可用。换句话说，搜索会返回结果，并根据这些结果生成新的细化选项列表。例如，在搜索 PS5 游戏机时，可以显示存储容量、运输成本和颜色等信息，帮助用户选择理想的产品。

面面观

既然我们已经定义了过滤器和切面，下面我们就来讨论经典方法和基于机器学习 (ML) 的方法对其实施和使用的影响。每种方法都有影响搜索效率的优势和挑战。

过滤器和分面的经典方法

在这种方法中，过滤器和切面是根据预定义规则手动定义的。这意味着，考虑到目录结构和用户需求，可用于细化搜索的属性是固定的，并事先进行了规划。

例如，在市场中，"Electronics" 或"Fashion" 等类别可能有特定的筛选条件，如品牌、格式和价格范围。这些规则是静态创建的，可确保搜索体验的一致性，但每当出现新的产品或类别时，就需要进行手动调整。

虽然这种方法提供了对所显示过滤器和面的可预测性和控制，但当出现需要动态改进的新趋势时，这种方法就会受到限制。

优点

  • 可预测性和控制：由于筛选器和切面是手动定义的，因此管理变得更加容易。

  • 低复杂性：无需训练模型。

  • 易于维护：由于规则是预定义的，因此可以快速进行调整和修正。

缺点

  • 新过滤器需要重新索引：每当需要使用新属性作为筛选器时，就必须对整个数据集重新索引，以确保文档包含该信息。

  • 缺乏动态适应性：过滤器是静态的，不能根据用户行为的变化自动调整。

滤波器/滤面的实现 - 经典方法

在开发工具 Kibana 中，我们将使用经典方法创建过滤器/面板演示。

首先，我们定义映射来构建索引：

PUT videogames
{
  "mappings": {
    "properties": {
      "name": { "type": "text" },
      "brand": { "type": "keyword" },
      "storage": { "type": "keyword" },
      "price": { "type": "float" },
      "description": { "type": "text" }
    }
  }
}

品牌和存储 字段被设置为关键字，可直接用于聚合（面）。价格字段为浮动类型，可以创建价格范围。

下一步，将对产品数据编制索引：

POST videogames/_bulk
{ "index": { "_id": 1 } }
{ "name": "Play Station 5", "brand": "Sony", "storage": "1TB", "price": 499.99, "description": "Stunning Gaming: Marvel at stunning graphics and experience the features of the new PS5. Breathtaking Immersion: Discover a deeper gaming experience with support for haptic feedback, adaptive triggers, and 3D Audio technology. Slim Design: With the PS5 Digital Edition, gamers get powerful gaming technology in a sleek, compact design. 1TB of Storage: Have your favorite games ready and waiting for you to play with 1TB of built-in SSD storage. Backward Compatibility and Game Boost: The PS5 console can play over 4,000 PS4 games. With Game Boost, you can even enjoy faster, smoother frame rates in some of the best PS4 console games." }
{ "index": { "_id": 2 } }
{ "name": "Xbox Series X", "brand": "Microsoft", "storage": "1TB", "price": 499.99, "description": "Fastest, most powerful Xbox console ever. Play thousands of titles: Every game looks and plays better on Xbox Series X. At the heart of Series X is the Xbox Velocity. Architecture, which combines a custom SSD and built-in software to significantly reduce load times in and out of game. Switch between multiple games in an instant with Quick Resume. Explore new worlds and experience the action like never before with an unparalleled 12 teraflops of graphics processing power. Enjoy 4K gaming at up to 120 frames per second, premium advanced 3D sound, and more. 4K at 120 FPS: requires compatible content and display X version - with disc drive" }
{ "index": { "_id": 3 } }
{ "name": "Nintendo Switch", "brand": "Nintendo", "storage": "512GB", "price": 299.99, "description": "SHARPER, VIBRANT VISUALS. The new 7-inch screen on the Nintendo Switch OLED takes your gaming to the next level: vibrant colors with sharp contrasts for every moment. INTEGRATED GAMEPLAY. Enjoy the console's many multiplayer modes and connect with other players. Online or locally, the fun on the Nintendo Switch is guaranteed. ENJOY IMMERSION FOR LONGER. In addition to delivering an unparalleled experience, thanks to its improved audio, the Nintendo Switch has a rechargeable battery while you play. From 4.5 hours to 9 hours of battery life. INCLUDES SUPER MARIO BROS. WONDER. Transform your world with the phenomenal flowers in this new Mario game, full of amazing adventures, power-ups and new abilities. NINTENDO SWITCH ONLINE SUBSCRIPTION. Access online games, play with friends and enjoy the exclusive benefits of the Nintendo Switch Online subscription." }
{ "index": { "_id": 4 } }
{ "name": "Steam Deck", "brand": "Valve", "storage": "512GB", "price": 399.99, "description": "You can save games, apps, photos and videos without worrying about space. High-Level Performance: The 4-core processor and graphics ensure a dynamic experience and fast responses. High-Definition Images: Smooth transitions and sharp images provide complete immersion in the game. Wireless Connectivity: Wi-Fi technology allows you to play wherever you want, without wires or cables limiting your fun" }
{ "index": { "_id": 5 } }
{ "name": "Nintendo Switch Lite", "brand": "Nintendo", "storage": "512GB", "price": 299.99, "description": "MADE TO BE PORTABLE. Nintendo Switch Lite is designed specifically for portable gaming. The console lets you jump into your favorite games wherever you are. COMPACT AND LIGHTWEIGHT. With its sleek, lightweight design, this console is ready to hit the road wherever you are. COMPATIBLE GAMES. The Nintendo Switch Lite system plays the library of Nintendo Switch games that work in handheld mode. A WORLD OF COLOR TO CHOOSE FROM. Available in a variety of vibrant and unique colors, Nintendo Switch Lite lets you bring even more personality wherever you go." }

现在，让我们按照品牌、存储空间和价格范围对结果进行分组，从而检索出经典的面孔。在查询中，定义了 size:0。在这种情况下，目标是只检索聚合结果，而不包括与查询相对应的文档。

POST videogames/_search
{
  "size": 0,
  "aggs": {
    "brands": {
      "terms": { "field": "brand" }
    },
    "storage_sizes": {
      "terms": { "field": "storage" }
    },
    "price_ranges": {
      "range": {
        "field": "price",
        "ranges": [
          { "to": 300 },   
          { "from": 300, "to": 500 },  
          { "from": 500 }  
        ]
      }
    }
  }
}

回复将包括品牌、存储和价格的计数，有助于创建筛选器和面。

"aggregations": {
   "brands": {
     "doc_count_error_upper_bound": 0,
     "sum_other_doc_count": 0,
     "buckets": [
       {
         "key": "Microsoft",
         "doc_count": 1
       },
       {
         "key": "Nintendo",
         "doc_count": 1
       },
       {
         "key": "Sony",
         "doc_count": 1
       },
       {
         "key": "Valve",
         "doc_count": 1
       }
     ]
   },
   "storage_sizes": {
     "doc_count_error_upper_bound": 0,
     "sum_other_doc_count": 0,
     "buckets": [
       {
         "key": "1TB",
         "doc_count": 2
       },
       {
         "key": "512GB",
         "doc_count": 2
       }
     ]
   },
   "price_ranges": {
     "buckets": [
       {
         "key": "*-300.0",
         "to": 300,
         "doc_count": 1
       },
       {
         "key": "300.0-500.0",
         "from": 300,
         "to": 500,
         "doc_count": 3
       },
       {
         "key": "500.0-*",
         "from": 500,
         "doc_count": 0
       }
     ]
   }
 }

基于机器学习/人工智能的筛选器和分面方法

在这种方法中，机器学习（ML）模型（包括人工智能（AI）技术）分析数据属性，生成相关的过滤器和面。ML/AI 不依赖预定义规则，而是利用索引数据特征。这样就能动态发现新的切面和过滤器。

优点

  • 自动更新：自动生成新的过滤器和切面，无需手动调整。

  • 发现新属性：它可以将以前未考虑过的 数据特征识别为过滤器，从而丰富搜索体验。

  • 减少人工操作：当人工智能从可用数据中学习时，团队无需不断定义和更新过滤规则。

缺点

  • 维护复杂性：使用模型可能需要预先验证，以确保生成的过滤器的一致性。

  • 需要 ML 和 AI 专业知识：该解决方案需要合格的专业人员来微调和监控模型性能。

  • 无关过滤器的风险：如果模型没有得到很好的校准，可能会生成对用户无用的切面。

  • 成本：使用 ML 和 AI 可能需要第三方服务，从而增加运营成本。

值得注意的是，即使有了校准良好的模型和精心制作的提示，生成的切面仍应经过审查步骤。这种验证可以是手动的，也可以基于审核规则，以确保内容的适当性和安全性。虽然这不一定是一个缺点，但这是一个重要的考虑因素，以确保在提供给用户之前，面的质量和适用性。

实施过滤器/面板--人工智能方法

在本演示中，我们将使用一个人工智能模型来自动分析产品特性并提出相关属性建议。有了结构良好的提示，我们就能从目录中提取信息，并将其转化为过滤器和切面。下面，我们将介绍这一过程的每个步骤。

最初，我们将使用推理 API注册一个端点，以便与 ML 服务集成。以下是与OpenAI 服务集成的示例。

PUT _inference/completion/generate_filter_ia
{
   "service": "openai",
   "service_settings": {
       "api_key": "your-key",
       "model_id": "gpt-4o-mini"
   }
}

现在，我们定义一个管道来执行提示并获取模型生成的新过滤器。

PUT /_ingest/pipeline/generate_filter_ai
{
   "processors": [
     {
       "script": {
         "source": """ctx.prompt = "You are an expert in data organization for search and product categorization. Your task is to analyze the following product and identify the best dynamic facets that can be used in an e-commerce search experience. Product: " + ctx.name + "description: " + ctx.description + "Instructions: - Analyze the product name and description. - Extract only the dynamic facets (technological features or product characteristics that can be inferred from the description, try to create max 3 facets by characteristics found). Put the values into an array. Using key and value, e.g. dynamic_facets: [{ \"name\": \"Gaming Experience\", \"value\": \"Haptic Feedback\" },{ \"name\": \"Gaming Experience\", \"value\": \"Adaptive Triggers\" } - Return only a JSON."
         """
       }
     },
     {
       "inference": {
         "model_id": "generate_filter_ia",
         "input_output": {
           "input_field": "prompt",
           "output_field": "result"
         }
       }
     },
     {
       "gsub": {
         "field": "result",
         "pattern": "```json",
         "replacement": ""
       }
     },
     {
       "json" : {
         "field" : "result",
         "strict_json_parsing": false,
         "add_to_root" : true
       }
     },
     {
       "remove": {
         "field": "result"
       }
     },
     {
       "remove": {
         "field": "prompt"
       }
     }
   ]
}

为"PlayStation 5" 产品运行该流水线的模拟，说明如下：

令人惊叹的游戏：惊叹于令人惊叹的画面，体验全新 PS5 的功能。

令人惊叹的沉浸感：支持触觉反馈、自适应触发器和 3D 音频技术，探索更深层次的游戏体验。

超薄设计：通过 PS5 数字版，玩家可以在时尚、紧凑的设计中获得强大的游戏技术。

1TB 存储空间：内置 1TB SSD 存储空间，让您随时随地畅玩最喜爱的游戏。

向后兼容和游戏提升：PS5 游戏机可播放 4,000 多款 PS4 游戏。有了 Game Boost，您甚至可以在一些最好的 PS4 游戏机游戏中享受更快、更流畅的帧率。

让我们观察一下这次模拟产生的提示输出。

{
 "docs": [
   {
     "doc": {
       "_index": "index",
       "_version": "-3",
       "_id": "1",
       "_source": {
         "name": "Play Station 5",
         "result": """```json
{
 "dynamic_facets": [
   { "name": "Storage Capacity", "value": "1TB SSD" },
   { "name": "Graphics Technology", "value": "Stunning Graphics" },
   { "name": "Audio Technology", "value": "3D Audio" }
 ]
}
```""",
         "description": "Stunning Gaming: Marvel at stunning graphics and experience the features of the new PS5. Breathtaking Immersion: Discover a deeper gaming experience with support for haptic feedback, adaptive triggers, and 3D Audio technology. Slim Design: With the PS5 Digital Edition, gamers get powerful gaming technology in a sleek, compact design. 1TB of Storage: Have your favorite games ready and waiting for you to play with 1TB of built-in SSD storage. Backward Compatibility and Game Boost: The PS5 console can play over 4,000 PS4 games. With Game Boost, you can even enjoy faster, smoother frame rates in some of the best PS4 console games.",
         "model_id": "generate_filter_ia",
         "prompt": """You are an expert in data organization for search and product categorization. Your task is to analyze the following product and identify the best dynamic facets that can be used in an e-commerce search experience. Product: Play Station 5description: Stunning Gaming: Marvel at stunning graphics and experience the features of the new PS5. Breathtaking Immersion: Discover a deeper gaming experience with support for haptic feedback, adaptive triggers, and 3D Audio technology. Slim Design: With the PS5 Digital Edition, gamers get powerful gaming technology in a sleek, compact design. 1TB of Storage: Have your favorite games ready and waiting for you to play with 1TB of built-in SSD storage. Backward Compatibility and Game Boost: The PS5 console can play over 4,000 PS4 games. With Game Boost, you can even enjoy faster, smoother frame rates in some of the best PS4 console games.Instructions: - Analyze the product name and description. - Extract only the dynamic facets (technological features or product characteristics that can be inferred from the description, try create max 3 facets by characteristics found). Put the values like arrays. Using key and value, e.g. dynamic_facets: [{ "name": "Gaming Experience", "value": "Haptic Feedback" },{ "name": "Gaming Experience", "value": "Adaptive Triggers" } - Return only a JSON."""
       },
       "_ingest": {
         "timestamp": "2025-03-19T22:14:32.0161803Z"
       }
     }
   }
 ]
}

现在，新索引中将添加一个新字段dynamic_facets，用于存储人工智能生成的面。

PUT videogames_1
{
 "mappings": {
   "properties": {
     "name": { "type": "text" },
     "brand": { "type": "keyword" },
     "storage": { "type": "keyword" },
     "price": { "type": "float" },
     "description": { "type": "text" },
     "dynamic_facets": { "type": "nested",
     "properties": { "name": { "type": "keyword" },
                     "value": { "type": "keyword" } } }
   }
 }
}

我们将使用Reindex API 将videogames索引重新编入videogames_1，并在此过程中应用generate_filter_ai管道。该管道将在索引编制过程中自动生成动态切面。

POST _reindex?wait_for_completion=false
{
 "source": {
   "index": "videogames"
 },
 "dest": {
   "index": "videogames_1",
   "pipeline": "generate_filter_ai"
 }
}

现在，我们将运行搜索并获得新的筛选器：

GET videogames_1/_search
{
 "size": 0,
 "query": {
   "match": {
     "name": "nintendo"
   }
 },
 "aggs": {
   "dynamic_facets": {
     "nested": {
       "path": "dynamic_facets"
     },
     "aggs": {
       "facets": {
         "terms": {
           "field": "dynamic_facets.name"
         },
         "aggs": {
           "facets": {
             "terms": {
               "field": "dynamic_facets.value"
             }
           }
         }
       }
     }
   }
 }
}

结果

"aggregations": {
   "dynamic_facets": {
     "doc_count": 3,
     "facets": {
       "doc_count_error_upper_bound": 0,
       "sum_other_doc_count": 0,
       "buckets": [
         {
           "key": "Frame Rate",
           "doc_count": 1,
           "facets": {
             "doc_count_error_upper_bound": 0,
             "sum_other_doc_count": 0,
             "buckets": [
               {
                 "key": "120 FPS",
                 "doc_count": 1
               }
             ]
           }
         },
         {
           "key": "Gaming Resolution",
           "doc_count": 1,
           "facets": {
             "doc_count_error_upper_bound": 0,
             "sum_other_doc_count": 0,
             "buckets": [
               {
                 "key": "4K",
                 "doc_count": 1
               }
             ]
           }
         },
         {
           "key": "Graphics Processing Power",
           "doc_count": 1,
           "facets": {
             "doc_count_error_upper_bound": 0,
             "sum_other_doc_count": 0,
             "buckets": [
               {
                 "key": "12 Teraflops",
                 "doc_count": 1
               }
             ]
           }
         }
       ]
     }
   }
 }

下面是一个简单的前端，以表示面的实现：

执行方面

这里提供了用户界面代码。

结论

这两种创建过滤器和切面的方法各有利弊。基于手动规则的传统方法可提供控制并降低成本，但需要不断更新，且无法动态适应新产品或新功能。

另一方面，基于人工智能和机器学习的方法可以自动提取切面，使搜索更加灵活，并且无需人工干预即可发现新的属性。不过，这种方法的实施和维护可能更为复杂，需要进行校准以确保结果的一致性。

在传统方法和基于人工智能的方法之间做出选择，取决于企业的需求和复杂程度。对于数据属性稳定且可预测的简单场景，传统方法可以更高效、更易于维护，从而避免基础设施和人工智能模型的不必要成本。另一方面，使用 ML/AI 提取切面可以大大增加价值，改善搜索体验，使过滤更加智能。

重要的是要评估自动化是否值得投资，或者更传统的解决方案是否已能有效满足业务需求。

常见问题

什么是过滤器？

筛选器是预定义的属性，用于限制结果集。

什么是分面？

分面是在执行搜索后生成的一组全新的筛选细化选项列表。

针对筛选器与分面，传统的处理方式是怎样的？

在针对筛选器与分面的传统处理方式中，可用于细化搜索的属性是固定且预先规划好的。 这种方法在可预测性方面表现良好，但缺乏动态适应性，并且需要创建新的过滤器以进行重新索引。

针对筛选器与分面，机器学习的处理方式是怎样的？

针对筛选器与分面，机器学习 (ML) 的处理方式主要基于 AI。ML 模型通过分析数据属性来生成相关的筛选器和分面。相较于依赖预先设定的规则，ML/AI 会利用已索引的数据特征来开展工作。

相关内容

相关性在 Elastic 内部

使用判断列表评估搜索查询的相关性

Jhon Guzmán
混合搜索相关性

你懂的，语境--第一部分：混合搜索和语境工程的演变

Woody Walton
相关性基础功能

如何使用我们的同义词 API 自动生成和上传同义词

Andre Luiz
相关性向量数据库

扩展 Elasticsearch 中的后期交互模型 — 第 2 部分

Peter Straßer
混合搜索相关性

混合搜索不头疼：用检索器简化混合搜索

Mike Pellegrini

准备好打造最先进的搜索体验了吗？

足够先进的搜索不是一个人的努力就能实现的。Elasticsearch 由数据科学家、ML 操作员、工程师以及更多和您一样对搜索充满热情的人提供支持。让我们联系起来，共同打造神奇的搜索体验，让您获得想要的结果。

亲自试用