Suggester: Phrase suggest option to limit suggestions to exising phrases

[radev]

When using phrase suggest API to provide "Did you mean ?" corrections it would be nice to include only suggestions that would return results.

So returned phrase must exist at least in one document in the index.

[s1monw]

Thanks for opening this. do you feel like attaching a pullrequest? I'd be happy to help you sketching out the functionality here!

[nik9000]

So I had a similar problem that came from filters: I was getting phrases that exist in the index but were filtered out on subsequent searches with the same filter set. I've since mostly worked around the problem by splitting my index along the most common filter. So if it isn't that much more work to get it to include filters that'd be great. If OTOH, you do something like only return suggestions that match an n-gram then by all means just do that and ignore me. I'd still use it and it'd probably be faster in the end.

[s1monw]

folks, I don't think we can filter the process of drawing candidates etc. since performance will suffer badly. What I can imagine is to execute a match query with each suggestoin that is returned to make a decision to drop them or not. Yet, this will allow for filtering for sure but it will only be a helper to prune the result list. if this is ok for you guys I think we can certainly do that!

[s1monw]

@nik9000 do you wanna take a look at this. I won't be able to do in the near future.

[nik9000]

I can take a look sometime in the next few days, yeah. We just deployed our elasticsearch software to a much larger group of users yesterday so I'm getting a bunch of high priority bugs that aren't (yet) this. What I'll do is file this issue as a medium priority one on my side and pick it up when I've cleared everything higher.

So what'd be most useful for me would be to have an api like this:

curl -XPOST 'localhost:9200/_search' -d {
  "suggest" : {
    "text" : "Xor the Got-Jewel",
    "simple_phrase" : {
      "phrase" : {
        "field" : "body",
        "size" : 5,
        "shard_size": 10,
        "confidence": 2.0,
        "filter_replace_string": "{}",
        "filter": {
          "bool" : {
                "must" : {
                    "query": { "match_phrase" : { "body" : "{}", "slop": 3 } }
                },
                "must_not" : {
                    "range" : {
                        "age" : { "from" : 10, "to" : 20 }
                    }
                },
                "should" : [
                    {
                        "term" : { "tag" : "sometag" }
                    },
                    {
                        "term" : { "tag" : "sometagtag" }
                    }
                ]
            }
        }
      }
    }
  }
}

I can see how this could be a performance problem but caching should kick in and help with most of the filters. Also, if you set your confidence nice and high you might not have to do this too many time.

And another thing, you'd have to crank up the shard_size value or you might end up filtering out all the suggestions that come from the shards.

[nik9000]

Now that I'm digging into this I like this API better

curl -XPOST 'localhost:9200/_search' -d {
  "suggest" : {
    "text" : "Xor the Got-Jewel",
    "simple_phrase" : {
      "phrase" : {
        "field" : "body",
        "size" : 5,
        "shard_size": 10,
        "confidence": 2.0,
        "filter": {
          "bool" : {
                "must_not" : {
                    "range" : {
                        "age" : { "from" : 10, "to" : 20 }
                    }
                },
                "should" : [
                    {
                        "term" : { "tag" : "sometag" }
                    },
                    {
                        "term" : { "tag" : "sometagtag" }
                    }
                ]
            }
        }
      }
    }
  }
}```
Internally I'd build a bool filter containing a phrase_match against the field against which we generate suggestions and the filter you passes in.  This is less fiddly to code and allows some simple syntax shortcuts:
```bash
curl -XPOST 'localhost:9200/_search' -d {
  "suggest" : {
    "text" : "Xor the Got-Jewel",
    "simple_phrase" : {
      "phrase" : {
        "field" : "body",
        "size" : 5,
        "shard_size": 10,
        "confidence": 2.0,
        "filter": "yes"
      }
    }
  }
}```
which would add the phrase_match and
```bash
curl -XPOST 'localhost:9200/twitter/_search?pretty=true' -d '
{
    "query" : {
        "term" : { "message" : "something" }
    },
    "filter" : {
        "term" : { "tag" : "green" }
    }
    "suggest" : {
        "text" : "Xor the Got-Jewel",
        "simple_phrase" : {
            "phrase" : {
                "field" : "body",
                "size" : 5,
                "shard_size": 10,
                "confidence": 2.0,
                "filter": "query"
            }
        }
    }
}'

which would go and get the filters from the top level query.

I'm still concerned about how slow this might be.

[nik9000]

So I finally have something for this that kinda works. It doesn't fully work and I'd like some guidance. I can't post the code right now because I'm traveling. Such is life. Any way, I'd like giluidance on two things:

1. Since there is no good way to parse the request during the reduce phase I'm parsing what I need on the shards and sending it back. Is that right/normal?
2. I can make queries during the reduce phase and block, waiting for them to return pretty easy. That seems to work on when I start the server but badly in tests, presumably because they have a lower bound on tr thread pool. Is it OK to make blocking calls in the reduce step? Anything else seems like a pretty invasive change for this feature which doesn't have a ton of traction.

[s1monw]

moved over to 0.90.8 I will look at this soon as well

[timbunce]

Any news on this, or the similar #2842?