[MarkLogic Dev General] Custom search grammar

Will Thompson wthompson at oconnors.com
Wed Feb 1 12:34:13 PST 2017


Hi Oleksi,

If it seems odd to you that MarkLogic continues to pressure you *not* to use the API they built, you are not alone. 

Erik,

I don't think it's reasonable to suggest to customers only needing to extend or replace a small portion of Search API that they rewrite significant parts of its functionality using 3rd-party tools (some of which haven't been updated for years) instead of using the clearly documented extensibility hooks of the ML-provided API.

If MarkLogic wants to push customers with more complex Search API needs in a different direction, that's fine, but it would be a lot more palatable if ML actually did some of the legwork upfront - provide code examples, blogs, documentation, etc. - to demonstrate how that should be done correctly. At a minimum it's confusing to be told not to use the the provided tools and frustrating that the alternatives suggested require a lot more work and uncertainty. Ideally, if ML doesn't want customers using parts of the Search API, they should just build a replacement that they are willing to endorse (maybe even using one of the JS parsers you recommend).

-Will


> On Feb 1, 2017, at 11:58 AM, Erik Hennum <Erik.Hennum at marklogic.com> wrote:
> 
> Hi, Oleksii:
> 
> To be clear, we discourage use of custom grammars.
> 
> Besides the JavaScript parser generators that I mentioned previously, you might also consider the XQuery approach demonstrated in:
> 
>    https://github.com/mblakele/xqysp
> 
> These approaches will support more flexible and performant parsers than the dynamic grammar of the Search API.
> 
> If you have a requirement that can be addressed only by a custom grammar and not by one of these approaches, please open a support ticket.
> 
> 
> Erik Hennum
> 
> ________________________________________
> From: general-bounces at developer.marklogic.com [general-bounces at developer.marklogic.com] on behalf of Oleksii Segeda [osegeda at worldbankgroup.org]
> Sent: Wednesday, February 01, 2017 7:28 AM
> To: MarkLogic Developer Discussion
> Subject: Re: [MarkLogic Dev General] Custom search grammar
> 
> Hi Erik,
> 
> Did you figure out how to extend the grammar?
> 
> Regards,
> Oleksii Segeda
> IT Analyst
> Information and Technology Solutions
> 
> 
> -----Original Message-----
> From: general-bounces at developer.marklogic.com [mailto:general-bounces at developer.marklogic.com] On Behalf Of Oleksii Segeda
> Sent: Monday, January 30, 2017 3:09 PM
> To: MarkLogic Developer Discussion <general at developer.marklogic.com>
> Subject: Re: [MarkLogic Dev General] Custom search grammar
> 
> Hi Erik,
> 
> Yes, that's is desired behavior.
> 
> Ideally, I would like to avoid custom constraints, simply because search grammar looks cleaner in the search box. In addition, some of our users are already familiar with simple search operators like AND, OR, so BOOST won't look like an alien to them.
> 
> I guess a postprocessing can be used as you suggested, however I'm interested in custom search grammar, because I may need to extend it more in the future.
> 
> Thank you,
> Oleksii Segeda
> IT Analyst
> Information and Technology Solutions
> 
> 
> -----Original Message-----
> From: general-bounces at developer.marklogic.com [mailto:general-bounces at developer.marklogic.com] On Behalf Of Erik Hennum
> Sent: Monday, January 30, 2017 2:42 PM
> To: MarkLogic Developer Discussion <general at developer.marklogic.com>
> Subject: Re: [MarkLogic Dev General] Custom search grammar
> 
> Hi, Oleksii:
> 
> Thanks for providing more detail.
> 
> Just to confirm, is it clear that, in a boost query, the right-hand
> term is optional?  Documents with only the left-hand term will still
> appear in the results though with less relevance than documents
> that have both terms.
> 
> By contrast, AND-related terms are both required and both
> contribute to relevance.
> 
> Anyway, to increase weight, one approach would be to define a tag
> for a quoted phrase and pass the phrase to a Search API custom
> constraint or to cts:parse() with a binding to a query generator function:
> 
>    http://docs.marklogic.com/guide/search-dev/cts_query#id_13456
> 
> The custom code could then tokenize the phrase and combine the
> terms with a boost-query or and-query, adding appropriate weight.
> 
> Another approach would be to do postprocessing of the query tree
> returned by cts:parse() or search:parse() to replace the default
> boost-query or and-query with a query that has more weight.
> 
> In either approach, you would then search on the query.
> 
> I mention cts:parse() because it parses query text more quickly
> than search:parse()
> 
> 
> Hoping that helps,
> 
> Erik Hennum
> 
> ________________________________________
> From: general-bounces at developer.marklogic.com [general-bounces at developer.marklogic.com] on behalf of Oleksii Segeda [osegeda at worldbankgroup.org]
> Sent: Monday, January 30, 2017 10:55 AM
> To: general at developer.marklogic.com
> Subject: Re: [MarkLogic Dev General] Custom search grammar
> 
> Hi Erik,
> 
> I'm trying to boost some parts of search query. For example, if user types `trade BOOST water`, I want documents with the word "water" to be higher in the results.
> cts:boost-query seems to be a perfect fit, but the default BOOST doesn't let you specify weights.
> 
> My ultimate goal is to convert `trade BOOST water` to something like this:
> 
>    cts:boost-query(cts:word-query("trade"), cts:word-query("water", (), 10.0) )
> 
> Regards,
> Oleksii Segeda
> IT Analyst
> Information and Technology Solutions
> 
> -----Original Message-----
> From: general-bounces at developer.marklogic.com [mailto:general-bounces at developer.marklogic.com] On Behalf Of general-request at developer.marklogic.com
> Sent: Monday, January 30, 2017 1:08 PM
> To: general at developer.marklogic.com
> Subject: General Digest, Vol 151, Issue 42
> 
> Send General mailing list submissions to
>        general at developer.marklogic.com
> 
> To subscribe or unsubscribe via the World Wide Web, visit
>        http://developer.marklogic.com/mailman/listinfo/general
> or, via email, send a message with subject or body 'help' to
>        general-request at developer.marklogic.com
> 
> You can reach the person managing the list at
>        general-owner at developer.marklogic.com
> 
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of General digest..."
> 
> 
> Today's Topics:
> 
>   1. Custom search grammar (Oleksii Segeda)
>   2. Re: Custom search grammar (Erik Hennum)
> 
> 
> ----------------------------------------------------------------------
> 
> Message: 1
> Date: Mon, 30 Jan 2017 16:51:26 +0000
> From: Oleksii Segeda <osegeda at worldbankgroup.org>
> Subject: [MarkLogic Dev General] Custom search grammar
> To: "general at developer.marklogic.com"
>        <general at developer.marklogic.com>
> Message-ID:
>        <BN1PR0101MB0769B9CDCD5E7697ACE8381BCB4B0 at BN1PR0101MB0769.prod.exchangelabs.com>
> 
> Content-Type: text/plain; charset="us-ascii"
> 
> Hi there,
> 
> I'm trying to declare a custom search grammar. I declared a custom function via search options, which supposed to parse "BOOST" keyword:
> 
> <joiner strength="2" apply="custom-boost" ns="http://worldbankgroup.org/search/grammar" at="/lib/grammar-boost.xqy" tokenize="word">BOOST</joiner>
> 
> I declared this function and just copied existing implementation from impl:joiner-boost function in /MarkLogic/appservices/search/search-impl.xqy :
> 
> declare function grammar:custom-boost($ps as map:map, $left as element()?, $opts as element()?) as schema-element(cts:query) {
>    let $symbol := impl:symbol-lookup($ps)
>    let $_ := tdop:advance($ps)
>    let $expr1 := tdop:expression($ps, $symbol/@strength)
>    return
>        if (empty($left))
>        then ($left, impl:msg($ps, <cts:annotation warning="SEARCH-IGNOREDQTEXT:[{string($symbol)} {$expr1}]: expected two arguments"/>))
>        else
>            element { xs:QName($symbol/@element) } {
>                attribute qtextjoin {concat($symbol/string())},
>                attribute strength {$symbol/@strength},
>                attribute qtextgroup { impl:opts($ps)/opt:grammar/opt:starter[@apply eq "grouping"]/(string(), @delimiter/string()) },
>                for $opt in $symbol/@options/tokenize(normalize-space(.)<mailto:$symbol/@options/tokenize(normalize-space(.)>, "\s") return <cts:option>{$opt}</cts:option>,
>                element cts:matching-query {
>                    attribute qtextref { "schema-element(cts:query)" },
>                    $left },
>                element cts:boosting-query {
>                    attribute qtextref { "schema-element(cts:query)" },
>                    $expr1 }
>            }
> };
> 
> Unfortunately this doesn't work, because for some reason impl:symbol-lookup returns an empty sequence.
> Any ideas what went wrong here?
> 
> 
> Oleksii Segeda
> 
> IT Analyst
> 
> Information and Technology Solutions
> 
> [http://siteresources.worldbank.org/NEWS/Images/spacer.png]
> 
> [http://siteresources.worldbank.org/NEWS/Images/WBG_Information_and_Technology_Solutions.png]
> 
> 
> 
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: http://developer.marklogic.com/pipermail/general/attachments/20170130/1958bd77/attachment-0001.html
> -------------- next part --------------
> A non-text attachment was scrubbed...
> Name: image003.png
> Type: image/png
> Size: 6577 bytes
> Desc: image003.png
> Url : http://developer.marklogic.com/pipermail/general/attachments/20170130/1958bd77/attachment-0002.png
> -------------- next part --------------
> A non-text attachment was scrubbed...
> Name: image004.png
> Type: image/png
> Size: 170 bytes
> Desc: image004.png
> Url : http://developer.marklogic.com/pipermail/general/attachments/20170130/1958bd77/attachment-0003.png
> 
> ------------------------------
> 
> Message: 2
> Date: Mon, 30 Jan 2017 18:07:41 +0000
> From: Erik Hennum <Erik.Hennum at marklogic.com>
> Subject: Re: [MarkLogic Dev General] Custom search grammar
> To: MarkLogic Developer Discussion <general at developer.marklogic.com>
> Message-ID:
>        <DFDF2FD50BF5AA42ADAF93FF2E3CA1850BD7D93C at EXCHG10-BE01.marklogic.com>
> Content-Type: text/plain; charset="windows-1252"
> 
> Hi, Oleksii:
> 
> Can you explain what you are trying to accomplish?
> 
> There may be better ways of doing the same thing than creating a custom grammar, which is really a tool of last resort.
> 
> For instance, a custom constraint can map a term to a custom query.
> 
> For other cases, it's often useful to do postprocessing on the generated query.
> 
> If a custom grammar really is unavoidable, in many cases a special-purpose third-party parsing tool may provide a faster and more flexible alternative to the limited custom grammar in the Search API.
> 
> For instance, the Jison.js and Peg.js parsers work with server-side JavaScript. (A nearly.js parser is also available, though I've heard no reports about it yet.)
> 
> 
> Hoping that helps,
> 
> 
> Erik Hennum
> 
> 
> ________________________________
> From: general-bounces at developer.marklogic.com [general-bounces at developer.marklogic.com] on behalf of Oleksii Segeda [osegeda at worldbankgroup.org]
> Sent: Monday, January 30, 2017 8:51 AM
> To: general at developer.marklogic.com
> Subject: [MarkLogic Dev General] Custom search grammar
> 
> Hi there,
> 
> I?m trying to declare a custom search grammar. I declared a custom function via search options, which supposed to parse ?BOOST? keyword:
> 
> <joiner strength="2" apply="custom-boost" ns="http://worldbankgroup.org/search/grammar" at="/lib/grammar-boost.xqy" tokenize="word">BOOST</joiner>
> 
> I declared this function and just copied existing implementation from impl:joiner-boost function in /MarkLogic/appservices/search/search-impl.xqy :
> 
> declare function grammar:custom-boost($ps as map:map, $left as element()?, $opts as element()?) as schema-element(cts:query) {
>    let $symbol := impl:symbol-lookup($ps)
>    let $_ := tdop:advance($ps)
>    let $expr1 := tdop:expression($ps, $symbol/@strength)
>    return
>        if (empty($left))
>        then ($left, impl:msg($ps, <cts:annotation warning="SEARCH-IGNOREDQTEXT:[{string($symbol)} {$expr1}]: expected two arguments"/>))
>        else
>            element { xs:QName($symbol/@element) } {
>                attribute qtextjoin {concat($symbol/string())},
>                attribute strength {$symbol/@strength},
>                attribute qtextgroup { impl:opts($ps)/opt:grammar/opt:starter[@apply eq "grouping"]/(string(), @delimiter/string()) },
>                for $opt in $symbol/@options/tokenize(normalize-space(.)<mailto:$symbol/@options/tokenize(normalize-space(.)>, "\s") return <cts:option>{$opt}</cts:option>,
>                element cts:matching-query {
>                    attribute qtextref { "schema-element(cts:query)" },
>                    $left },
>                element cts:boosting-query {
>                    attribute qtextref { "schema-element(cts:query)" },
>                    $expr1 }
>            }
> };
> 
> Unfortunately this doesn?t work, because for some reason impl:symbol-lookup returns an empty sequence.
> Any ideas what went wrong here?
> 
> 
> Oleksii Segeda
> 
> IT Analyst
> 
> Information and Technology Solutions
> 
> [http://siteresources.worldbank.org/NEWS/Images/spacer.png]
> 
> [http://siteresources.worldbank.org/NEWS/Images/WBG_Information_and_Technology_Solutions.png]
> 
> 
> 
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: http://developer.marklogic.com/pipermail/general/attachments/20170130/33effe92/attachment.html
> -------------- next part --------------
> A non-text attachment was scrubbed...
> Name: image003.png
> Type: image/png
> Size: 6577 bytes
> Desc: image003.png
> Url : http://developer.marklogic.com/pipermail/general/attachments/20170130/33effe92/attachment.png
> -------------- next part --------------
> A non-text attachment was scrubbed...
> Name: image004.png
> Type: image/png
> Size: 170 bytes
> Desc: image004.png
> Url : http://developer.marklogic.com/pipermail/general/attachments/20170130/33effe92/attachment-0001.png
> 
> ------------------------------
> 
> _______________________________________________
> General mailing list
> General at developer.marklogic.com
> Manage your subscription at:
> http://developer.marklogic.com/mailman/listinfo/general
> 
> 
> End of General Digest, Vol 151, Issue 42
> ****************************************
> _______________________________________________
> General mailing list
> General at developer.marklogic.com
> Manage your subscription at:
> http://developer.marklogic.com/mailman/listinfo/general
> _______________________________________________
> General mailing list
> General at developer.marklogic.com
> Manage your subscription at:
> http://developer.marklogic.com/mailman/listinfo/general
> _______________________________________________
> General mailing list
> General at developer.marklogic.com
> Manage your subscription at:
> http://developer.marklogic.com/mailman/listinfo/general
> 
> _______________________________________________
> General mailing list
> General at developer.marklogic.com
> Manage your subscription at:
> http://developer.marklogic.com/mailman/listinfo/general
> _______________________________________________
> General mailing list
> General at developer.marklogic.com
> Manage your subscription at: 
> http://developer.marklogic.com/mailman/listinfo/general



More information about the General mailing list