Lucene Query
MongoCamp Mongo Driver support the usage of Lucene Query to search in the MongoDb.
Usage
Explicit Usage
The LuceneConverter has the methods to parse a String to and Query and a other to the document conversion.
val search = PersonDAO.find(LuceneQueryConverter.toDocument(luceneQuery), sortByBalance).resultList()Implicit Usage
Like the Map to Bson conversion there is also an implicit method to convert Query to find Bson.
val search = PersonDAO.find(luceneQuery, sortByBalance).resultList()Parse String to Query
We have an individual parser to parse an string to Lucene Query, because the default Lucene Analyser is case-insensitive and convert all search data into lower case. So the best way to seach in MongoDb with Lucene Query is to use this code.
val luceneQuery = LuceneQueryConverter.parse("(favoriteFruit:\"apple\" AND age:\"25\") OR name:*Cecile* AND -active:false AND 123", "id")Analyzer and Tokenizer
LuceneQueryConverter.parse uses the MongoCampLuceneAnalyzer with its default settings. The analyzer gets a factory for the tokenizer as parameter, so the way a search value is split into terms is never chosen silently.
| Parameter | Default | Description |
|---|---|---|
stopWords | CharArraySet.EMPTY_SET | Terms that are removed from the query. |
tokenizerFactory | MongoCampLuceneAnalyzer.defaultTokenizerFactory | Creates the Lucene Tokenizer that splits the values to terms. |
The default factory creates a MongoCampTokenizer with a maxTokenLength of 255. It splits values only at whitespace, tokens longer than maxTokenLength are split into chunks of maxTokenLength characters.
To use another tokenizer, create the analyzer with a factory for it, parse the query with the MongoCampLuceneQueryParser and convert the result with LuceneQueryConverter.toDocument. The MongoCampLuceneQueryParser is a Lucene QueryParser that allows leading wildcards and searches quoted values as exact value.
val analyzer = new MongoCampLuceneAnalyzer(tokenizerFactory = () => new MongoCampTokenizer(maxTokenLength = 255))
val queryParser = new MongoCampLuceneQueryParser("name", analyzer)
val luceneQuery = queryParser.parse("email:latashamcmillan@ultrimax.com")
analyzer.close()
val search = PersonDAO.find(LuceneQueryConverter.toDocument(luceneQuery), sortByBalance).resultList()The analyzer is thread safe. Lucene creates the token stream components for each thread, so the analyzer calls the factory for each thread and every thread uses its own tokenizer.
WARNING
The factory must create a new Tokenizer on each call. A Tokenizer instance can not be shared, if the factory returns an instance a second time the analyzer throws an IllegalStateException.
Exact Values
A quoted value is searched as exact value, name:"Latasha Mcmillan" finds only documents with exactly this name. The quoted value is not split into terms, so the search is case-sensitive and whitespace is kept as it is. Quotes inside the value are escaped with a backslash (name:"Hallo \"Welt\"").
val luceneQuery = LuceneQueryConverter.parse("name:\"Latasha Mcmillan\"", "unbekannt")A quoted value with * is searched as wildcard query, name:"Latasha *millan" finds Latasha Mcmillan. A ? in a quoted value is no wildcard, so name:"Wie geht's?" is searched as exact value. A * in a quoted value is always a wildcard, also if it is escaped (see Escaped Characters).
If the query is parsed by another parser, like the Lucene QueryParser, quoted values are phrase queries. They are searched as exact value too, but the terms of the phrase are joined by a single space.
Values
With the default tokenizer values are split only at whitespace, so values like email addresses (email:john.doe@example.com) or dates with time zone offset (registered:20140419T224427000\+0200) are searched as one value. Leading and trailing single quotes are removed ('value' matches value).
Wildcard and prefix queries are converted to regular expressions. * and ? are used as wildcards, all other regular expression characters like . or + are escaped.
Like in Lucene the wildcard value has to match the whole value. The search is case-insensitive and * matches line breaks too.
| Query | Search |
|---|---|
name:John* | name starts with John |
name:*Dowe | name ends with Dowe |
name:"*John Dowe*" | name contains John Dowe |
name:J?hn | name is J + one character + hn |
Escaped Characters
Characters with a special meaning in the Lucene syntax (\ + - ! ( ) { } [ ] ^ " ~ * ? : / & |) are escaped with a backslash. An escaped character is always the literal character, so name:*Mary\-Jane* finds Mary-Jane Smith and name:*100\** finds 100* but not 1000. A backslash is escaped with a second backslash (path:*C\\temp* finds C\temp).
| Query | Search |
|---|---|
name:*Mary\-Jane* | name contains Mary-Jane |
mail:*a\+b@example.com* | mail contains a+b@example.com |
name:Mary\-Ja* | name starts with Mary-Ja |
name:*a\?b* | name contains a?b, the ? is no wildcard |
name:100\* | name is 100*, the * is no wildcard |
The Lucene QueryParser keeps the escape characters only in wildcard values like *Mary\-Jane*, the LuceneQueryConverter interprets them like Lucene does. In prefix values (Mary\-Ja*), values without wildcard and quoted values the parser removes the escape characters before the value is converted.
WARNING
In a quoted value the escape characters are removed by the Lucene QueryParser, so \* and * can not be distinguished. A * in a quoted value is always a wildcard, name:"100\**" finds 100* and 1000. To search a literal * use an unquoted value with escaped characters, like name:*100\**.
The following search cases are tests in LuceneEscapeSearchSuite.
private val allLabels = documents.map(_.getStringValue("label"))
private def allExcept(labels: List[String]): List[String] = allLabels.diff(labels)
private val searchCases = List(
SearchCase("""name:*Mary\-Jane*""", List("Mary-Jane Smith", "Mary-Jane"), "escaped - in a contains search"),
SearchCase("""mail:*a\+b@example.com*""", List("a+b@example.com"), "escaped + is a literal +"),
SearchCase("""mail:*@my\-company.org""", List("x@my-company.org"), "escaped - in an ends with search, . is literal"),
SearchCase("""phone:*\+1555*""", List("+15551234"), "escaped + at the beginning"),
SearchCase("""name:*100\**""", List("100*"), "escaped * is a literal *"),
SearchCase("""name:*a\?b*""", List("a?b"), "escaped ? is a literal ?"),
SearchCase("""name:*a?b*""", List("a?b", "axb", "a.b"), "unescaped ? matches one character"),
SearchCase("""name:*a.b*""", List("a.b"), "unescaped . is a literal ."),
SearchCase("""path:*C\\temp*""", List("C\\temp"), "escaped backslash is a literal backslash"),
SearchCase("""NOT name:*Mary\-Jane*""", allExcept(List("Mary-Jane Smith", "Mary-Jane")), "negated wildcard with escaped -"),
SearchCase("""-name:*Mary\-Jane*""", allExcept(List("Mary-Jane Smith", "Mary-Jane")), "negated wildcard with escaped - and minus"),
SearchCase("""name:*mary\-jane*""", List("Mary-Jane Smith", "Mary-Jane"), "wildcard with escaped - is case-insensitive"),
SearchCase("""name:*MARY\-JANE*""", List("Mary-Jane Smith", "Mary-Jane"), "upper case wildcard with escaped -"),
SearchCase("""name:(*Mary\-Jane* OR Peter)""", List("Mary-Jane Smith", "Mary-Jane", "Peter"), "escaped - in a group"),
SearchCase("""name:Mary\-Jane* AND NOT name:*Smith""", List("Mary-Jane"), "escaped - combined with a negated wildcard"),
SearchCase("""name:Mary\-Ja*""", List("Mary-Jane Smith", "Mary-Jane"), "escaped - in a starts with search"),
SearchCase("""name:100\**""", List("100*"), "escaped * in a starts with search"),
SearchCase("""path:C\\te*""", List("C\\temp"), "escaped backslash in a starts with search"),
SearchCase("""-name:Mary\-Ja*""", allExcept(List("Mary-Jane Smith", "Mary-Jane")), "negated starts with search with escaped -"),
SearchCase("""name:Mary\-Jane""", List("Mary-Jane"), "escaped - in an exact value"),
SearchCase("""name:100\*""", List("100*"), "escaped * in a value without wildcard is no wildcard"),
SearchCase("""path:C\\temp""", List("C\\temp"), "escaped backslash in an exact value"),
SearchCase("""name:"100*"""", List("100*", "1000"), "* in a quoted value is a wildcard"),
SearchCase("""name:"100\**"""", List("100*", "1000"), "\\* in a quoted value is a wildcard too"),
SearchCase("""name:"*a?b*"""", List("a?b"), "? in a quoted value is a literal ?"),
SearchCase("""path:"*C\\temp*"""", List("C\\temp"), "escaped backslash in a quoted wildcard value")
)TIP
A wildcard at the beginning of the value can not use an index of the field, so the search is slow on large collections.
Date values are parsed as ISO date (2014-04-19T22:44:27+02:00) or in the basic format (20140419T224427000+0200). Date values without time zone offset are interpreted in the default time zone of the JVM (TimeZone.getDefault), date values with Z as UTC.
Search Cases
The following search cases show the behavior of the lucene queries in MongoDB. Every search case is a test in LuceneSearchCasesSuite, so the cases are always up to date.
The search cases search in these documents, the default field of the queries is nickname.
private val documents = List(
Document("name" -> "John W. Dowe", "email" -> "john.w.dowe@example.com", "registered" -> date("2014-04-19T22:44:27Z")),
Document("name" -> "JohnDowe", "email" -> "john+dowe@example.com"),
Document("name" -> "John Dowe", "email" -> "john.dowe@example.com", "registered" -> date("2014-04-20T10:00:00Z")),
Document("name" -> "Big John W. Dowe"),
Document("name" -> "John W. Dowe Jr."),
Document("name" -> "Jane Dowe", "registered" -> date("2014-04-22T23:59:59Z")),
Document("name" -> "john dowe"),
Document("name" -> "Hallo Welt"),
Document("name" -> "Hallo Welt"),
Document("name" -> "Hallo \"Welt\""),
Document("name" -> "Wie geht's?"),
Document("name" -> "Hallo"),
Document("name" -> "Nick Welt", "nickname" -> "Welt"),
Document("name" -> "O'Brien", "email" -> "o'brien@example.com"),
Document("name" -> "Example X", "email" -> "x@exampleXcom"),
Document("name" -> "Line Breaks", "text" -> "first line\nsecond line")
)Every search case has the query, the names of the found documents and a description.
private val allNames = documents.map(_.getStringValue("name"))
private def allExcept(names: List[String]): List[String] = allNames.diff(names)
private val startsWithJohn = List("John W. Dowe", "JohnDowe", "John Dowe", "John W. Dowe Jr.", "john dowe")
private val searchCases = List(
// email addresses are one value
SearchCase("""email:john.w.dowe@example.com""", List("John W. Dowe"), "email address is searched as one value"),
SearchCase("""email:"john.w.dowe@example.com"""", List("John W. Dowe"), "quoted email address"),
SearchCase("""email:'john.w.dowe@example.com'""", List("John W. Dowe"), "single quotes are removed"),
SearchCase("""email:john+dowe@example.com""", List("JohnDowe"), "email address with plus"),
SearchCase("""email:o'brien@example.com""", List("O'Brien"), "single quote inside the value is kept"),
SearchCase("""email:*@example.com""", List("John W. Dowe", "JohnDowe", "John Dowe", "O'Brien"), "regex characters like . are escaped"),
SearchCase("""-email:john.w.dowe@example.com""", allExcept(List("John W. Dowe")), "negated email address"),
// quoted values are exact values
SearchCase("""name:"John Dowe"""", List("John Dowe"), "quoted value is searched as exact value"),
SearchCase("""name:"john dowe"""", List("john dowe"), "exact value is case-sensitive"),
SearchCase("""name:"John"""", List(), "exact value is no contains search"),
SearchCase("""name:"Hallo Welt"""", List("Hallo Welt"), "exact value with whitespace"),
SearchCase("""name:"Hallo Welt"""", List("Hallo Welt"), "whitespace of the exact value is kept"),
SearchCase("""name:"Hallo \"Welt\""""", List("Hallo \"Welt\""), "quotes are escaped with a backslash"),
SearchCase("""name:"Wie geht's?"""", List("Wie geht's?"), "? is no wildcard in a quoted value"),
SearchCase("""-name:"John Dowe"""", allExcept(List("John Dowe")), "negated exact value"),
// unquoted values with whitespace are multiple queries
SearchCase("""name: Hallo Welt""", List("Hallo", "Nick Welt"), "name is Hallo OR the default field is Welt"),
SearchCase("""name:John* Dowe""", startsWithJohn, "name starts with John OR the default field is Dowe"),
// wildcards have to match the whole value and are case-insensitive
SearchCase("""name:John*""", startsWithJohn, "starts with"),
SearchCase("""name:*Dowe""", List("John W. Dowe", "JohnDowe", "John Dowe", "Big John W. Dowe", "Jane Dowe", "john dowe"), "ends with"),
SearchCase("""name:"*W. Dowe*"""", List("John W. Dowe", "Big John W. Dowe", "John W. Dowe Jr."), "contains"),
SearchCase("""name:"John*Dowe"""", List("John W. Dowe", "JohnDowe", "John Dowe", "john dowe"), "* matches any text, also no text"),
SearchCase("""name:"John * Dowe"""", List("John W. Dowe"), "* between whitespace needs text between the words"),
SearchCase(
"""name:"*John*Dowe*"""",
List("John W. Dowe", "JohnDowe", "John Dowe", "Big John W. Dowe", "John W. Dowe Jr.", "john dowe"),
"contains John and later Dowe"
),
SearchCase("""name:J?hn*""", startsWithJohn, "? matches one character"),
SearchCase("""-name:John*""", allExcept(startsWithJohn), "negated wildcard"),
SearchCase("""text:*second*""", List("Line Breaks"), "contains in a value with line breaks"),
SearchCase("""text:first*""", List("Line Breaks"), "starts with in a value with line breaks"),
SearchCase("""text:*line?second*""", List("Line Breaks"), "? matches a line break"),
SearchCase("""text:"*line second*"""", List(), "a line break is no whitespace"),
// dates keep the time zone offset, dates without offset use the default time zone of the JVM
SearchCase("""registered:20140420T004427000\+0200""", List("John W. Dowe"), "date with time zone offset"),
SearchCase("""registered:"2014-04-20T00:44:27+02:00"""", List("John W. Dowe"), "ISO date with time zone offset"),
SearchCase("""registered:20140419T224427000Z""", List("John W. Dowe"), "date in UTC"),
SearchCase("""registered:2014-04-19T22\:44\:27Z""", List("John W. Dowe"), "ISO date in UTC"),
SearchCase("""registered:[2014-04-20T00:00:00Z TO 2014-04-22T23:59:59Z]""", List("John Dowe", "Jane Dowe"), "date range"),
SearchCase(
"""registered:[2014-04-20T00\:00\:00+02\:00 TO 2014-04-21T00\:00\:00+02\:00]""",
List("John W. Dowe", "John Dowe"),
"date range with time zone offset"
)
)