[Corona] Corona explore endpoint and web-interface?

Tue Dec 13 03:00:14 PST 2011

Hi,

I think there is something subtle going on here..

Looking at the REST library you have this code for finding matching
request elements (rest-impl:matches). It can either raise errors or not.
It seems to be intended as a two-pass process:

- First is the rewriting phase, which uses rest(-impl):rewrite and
rest-impl:matching-requests. This doesn't raise any errors, by design I
suppose.
- After that, the endpoint seems to be supposed to use
rest(impl):process-request (not always called?) to retrieve validated
parameters. This does raise errors, but effectively only those about
parameters (I think). Not those about incorrect uri, unsupported methods,
unacceptated mimes, etc. It couldn't, since a request element has already
been picked from the endpoints config, which (supposedly) already passed
the rewrite/matching-request phase. I am not even sure the process-request
could even raise parameter errors, since these are checked in
rewrite/matching-request phase too..

So, that is why the endpoints config is such tolerant. Not doing so would
just give a 404 kind of behavior (which is the corona/lib/rewriter.xqy
fallback) for any kind of request that doesn't match in the rewrite
phase..

Could this be a (minor) design flaw in the REST library?

Kind regards,
Geert

-----Oorspronkelijk bericht-----
Van: corona-bounces at developer.marklogic.com
[mailto:corona-bounces at developer.marklogic.com] Namens Colleen Whitney
Verzonden: maandag 12 december 2011 20:28
Aan: Ryan Grimm
CC: Norman Walsh; Corona Email List
Onderwerp: Re: [Corona] Corona explore endpoint and web-interface?

We're talking about two different things:

One is usability and documentation, which is the point raised here.

The second is around error handling; the range of things that can fail and
the ways they can fail is different for batch operations than for
single-document operations.  If there can be partial success/failure, the
end user may need different handling.

Ryan the REST library has some subtlety around where to mark parameters as
required;  if you do it in the rewriter, you get the no-match behavior,
whereas it you push that to the endpoint you will get a more specific
message about the missing parameter.

--Colleen

________________________________________
From: Ryan Grimm [grimm at xqdev.com]
Sent: Monday, December 12, 2011 9:26 AM
To: Colleen Whitney
Cc: Corona Email List; Norman Walsh
Subject: Re: [Corona] Corona explore endpoint and web-interface?

I'm happy to answer that question (and by answer I mean share my opinion
and experience :).

Splitting the bulk deletion out into it's own endpoint won't help with the
error handling.  The difficulty doesn't arise from having the uri
parameter being required in some cases and not in others.  The central
issue is that when a parameter is marked as required and isn't specified
by the user, responding with a helpful error is hard because none of the
defined requests will match.  This results in generic "I couldn't figure
out how to handle this request" code being called.

If being able to delete multiple documents from the document store in one
query is a good idea or not is a different topic, reporting errors will
remain the same regardless.

I should note that I don't view the current state of error reporting to be
troublesome at all.  A while ago I switched to viewing the endpoints.xqy
file as a configuration instead of a definition, a subtle but important
distinction.  I forgot about this distinction when suggesting that Geert
could use the required attributes to help with his WADL generation.

--Ryan

On Dec 12, 2011, at 9:02 AM, Colleen Whitney wrote:

> Yes. There's also a question in my mind as to whether the design makes
it more difficult to deal with errors; perhaps document CRUD and
bulk/database-level operations should not share an endpoint.
>
> Sent from my iPhone
>
> On Dec 12, 2011, at 8:45 AM, "Eric Bloch" <Eric.Bloch at marklogic.com>
wrote:
>
>> Seems a familiar lesson here, true with almost all API/library design,
regardless of language, protocol, rest, etc.
>>
>> If you implement your API using another library, you may not want to
expose that lower-level library's error reporting directly to your users.
At the very least, you want control over the error reporting.  Same is
true for logging mechanisms and other interaction points (e.g. config).
>>
>> -Eric
>>
>> On Dec 12, 2011, at 8:14 AM, Ryan Grimm wrote:
>>
>>> Hi Geert,
>>>
>>> Yep, DELETE requests can take in a query :)
(https://github.com/marklogic/Corona/wiki/Deleting-Documents).
>>>
>>> On the surface it might seem a bit unusual, but in practice it's
pretty handy.  For example, many times I don't know what the URI of a
document is.  If I'd like to delete a document where the id attribute
equals "123456", that's very easy to satisfy with a query.  It also makes
it easy to delete multiple documents in one request (e.g.: delete all of
the emails with a given thread id).
>>>
>>> I couldn't really speak to how feasible it would be to get good error
messages from ml-rest-lib.  But I'm going to include Norm Walsh (author of
the library) on this thread and see if he has any thoughts on what options
we have as users and if any changes to the library would be reasonable.
>>>
>>> --Ryan
>>>
>>>
>>> On Dec 12, 2011, at 8:02 AM, Geert Josten wrote:
>>>
>>>> Hi Ryan,
>>>>
>>>> The delete can take a query? Isn't that a bit unusual for /store? ;-)
>>>>
>>>> Yeah, I noticed that lacking a param declaration in endpoints
>>>> configuration makes a request not match when you do pass it in. I
agree
>>>> that it makes it difficult to give more clear error messages back. I
guess
>>>> same is valid for non-supported http methods. If you don't specify
them,
>>>> you won't get the opportunity to warn the user about his mistake, but
have
>>>> to rely on the response of ml-rest-lib to be clear enough.
>>>>
>>>> Two thoughts about that:
>>>> 1. Improve the error responses from ml-rest-lib (feasible?)
>>>> 2. Allow ml-rest-lib to relax a bit, passing more 'wrong' requests
>>>> through, allowing the endpoint scripts to give feedback themselves
>>>> (problem with store.xqy vs. store-get.xqy?)
>>>>
>>>> Kind regards,
>>>> Geert
>>>>
>>>> -----Oorspronkelijk bericht-----
>>>> Van: Ryan Grimm [mailto:grimm at xqdev.com]
>>>> Verzonden: maandag 12 december 2011 16:30
>>>> Aan: Geert Josten
>>>> CC: corona at developer.marklogic.com; Jason Hunter
>>>> Onderwerp: Re: [Corona] Corona explore endpoint and web-interface?
>>>>
>>>> Hi Geert,
>>>>
>>>> I noticed yesterday that earlier you found some inconsistencies in
the
>>>> endpoints.xqy config (e.g.: missing the optional / at the end of the
>>>> kvquery endpoint).  I've incorporated those changes, thanks for
spotting
>>>> them.
>>>>
>>>> I also took a look at your latest changes and made some comments on
the
>>>> commit
>>>>
(https://github.com/grtjn/Corona/commit/5684c065103d22caa8a93bac38354d166e
>>>> 724d3b#diff-0).
>>>>
>>>> I didn't see where you changed the URI to be required for the /store
>>>> endpoint which is good as it isn't actually required.  One of the
reasons
>>>> it isn't required is because DELETE requests can take in a query.
>>>> However, it is required for PUT, POST and GET requests.  But sadly
marking
>>>> it as so makes it very hard to give the user a good error message.
If
>>>> it's marked as required and the request doesn't have it, the rule
won't
>>>> match and the request will end up falling through to the generic code
that
>>>> simply states that it doesn't know how to handle the request.
>>>>
>>>> --Ryan
>>>>
>>>>
>>>> On Dec 12, 2011, at 3:05 AM, Geert Josten wrote:
>>>>
>>>>> FYI,
>>>>>
>>>>> I just committed some more changes to my personal fork of Corona. It
now
>>>>> contains a first attempt to a reasonable WADL (still rough though),
a
>>>>> simple index page for Corona with links to all browse-able
endpoints, a
>>>>> primitive uri explore feature (linking to store to view the
contents),
>>>> and
>>>>> XML output for all manage endpoints (could use some polishing)..
>>>>>
>>>>> Note: I didn't spend much effort on clean coding. Just wanted to
first
>>>>> show where this could go. If you think it is interesting to pull
into
>>>> the
>>>>> master, I'll clean up some more.
>>>>>
>>>>> I did notice a few slight 'issues' with the endpoints configuration.
>>>> Some
>>>>> params are required, but not marked as such (uri at store for
instance).
>>>>> Some resources contain multiple http methods, but only support a
part of
>>>>> them, depending on the exact uri. I did a few changes to improve
index
>>>> and
>>>>> wadl. Generating a soapui test project for the interface can help to
>>>> test
>>>>> for more of such small inconsistencies..
>>>>>
>>>>> Kind regards,
>>>>> Geert
>>>>>
>>>>> -----Oorspronkelijk bericht-----
>>>>> Van: Ryan Grimm [mailto:grimm at xqdev.com]
>>>>> Verzonden: zaterdag 10 december 2011 17:38
>>>>> Aan: Geert Josten
>>>>> CC: Jason Hunter; corona at developer.marklogic.com
>>>>> Onderwerp: Re: [Corona] Corona explore endpoint and web-interface?
>>>>>
>>>>> You are correct, the setup page is a start to a web admin without a
>>>> doubt.
>>>>> I did create a template (corona/htools/template.xqy) to ease the
>>>> creation
>>>>> of other pages.  I suspect that with a day or two of work I could
get a
>>>>> very crude interface going for the key parts of /manage.  If I got
>>>> started
>>>>> on these pages, would that make it easier for you to contribute
whenever
>>>>> desire and some free moments intersect?
>>>>>
>>>>> Also, I think that's a great suggestion for an enhancement to the
setup
>>>>> page.  We could easily grab a list of range indexes that exist but
>>>> aren't
>>>>> configured under Corona and ask the user to assign names to them and
>>>>> configure their buckets if they so desire.
>>>>>
>>>>> --Ryan
>>>>>
>>>>>
>>>>> On Dec 10, 2011, at 5:50 AM, Geert Josten wrote:
>>>>>
>>>>>> Ryan, Jason,
>>>>>>
>>>>>> I was actually talking about the database uris. Something like the
>>>>> explore
>>>>>> in CQ. And what I meant was a (crude) web-interface provided by
Corona
>>>>>> itself. Maybe I'll find a spare hour to look into it myself, and
think
>>>>>> about this 'corona-admin web interface' as well.. ;-)
>>>>>>
>>>>>> Oh by the way, there already is a bit of admin interface. I think
you
>>>>>> could get more out of this setup wizard. Why doesn't it auto-detect
>>>>>> present indexes, and suggest to add them as facets? (Thinking out
>>>>> loud..)
>>>>>>
>>>>>> Kind regards,
>>>>>> Geert
>>>>>>
>>>>>> -----Oorspronkelijk bericht-----
>>>>>> Van: Jason Hunter [mailto:jhunter at marklogic.com]
>>>>>> Verzonden: zaterdag 10 december 2011 5:11
>>>>>> Aan: Geert Josten
>>>>>> CC: corona at developer.marklogic.com
>>>>>> Onderwerp: Re: [Corona] Corona explore endpoint and web-interface?
>>>>>>
>>>>>> Hey Geert,
>>>>>>
>>>>>> I think you should try writing a sample web-interface app using
>>>> whatever
>>>>>> non-XQuery language you like.  Test out the Corona APIs.  Take 'em
out
>>>>> for
>>>>>> a spin.  :)
>>>>>>
>>>>>> -jh-
>>>>>>
>>>>>> On Dec 9, 2011, at 2:56 PM, Geert Josten wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> There doesn't seem to be an endpoint that simply lists all
existing
>>>>> uris
>>>>>>> (alphabetically ordered, optionally paged) does it? Could be the
>>>>> default
>>>>>>> reply if someone accesses /store, but without uri param. Or under
a
>>>>>>> different uri.
>>>>>>>
>>>>>>> Not sure about this either: Corona doesn't provide a (primitive)
>>>>>>> web-interface if someone accesses it with a browser, does it?
Would be
>>>>>> fun
>>>>>>> to have a simple search box page. Would be fun for explore too..
>>>>>>>
>>>>>>> Just some thoughts..
>>>>>>>
>>>>>>> Kind regards,
>>>>>>> Geert
>>>>>>>
>>>>>>> drs. G.P.H. (Geert) Josten
>>>>>>> Senior Developer
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Dayon B.V.
>>>>>>> Delftechpark 37b
>>>>>>> 2628 XJ Delft
>>>>>>>
>>>>>>> T +31 (0)88 26 82 570
>>>>>>>
>>>>>>> geert.josten at dayon.nl
>>>>>>> www.dayon.nl
>>>>>>>
>>>>>>> De informatie - verzonden in of met dit e-mailbericht - is
afkomstig
>>>>> van
>>>>>>> Dayon BV en is uitsluitend bestemd voor de geadresseerde. Indien u
dit
>>>>>>> bericht onbedoeld hebt ontvangen, verzoeken wij u het te
verwijderen.
>>>>>> Aan
>>>>>>> dit bericht kunnen geen rechten worden ontleend.
>>>>>>> _______________________________________________
>>>>>>> Corona mailing list
>>>>>>> Corona at developer.marklogic.com
>>>>>>> http://developer.marklogic.com/mailman/listinfo/corona
>>>
>>> _______________________________________________
>>> Corona mailing list
>>> Corona at developer.marklogic.com
>>> http://developer.marklogic.com/mailman/listinfo/corona
>>
>> _______________________________________________
>> Corona mailing list
>> Corona at developer.marklogic.com
>> http://developer.marklogic.com/mailman/listinfo/corona

_______________________________________________
Corona mailing list
Corona at developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/corona