Written Tutorial

Problem

You need to cancel requests that have queued on the Task Server without clearing the entire queue or restarting the host.

Solution

Applies to MarkLogic versions 7+

xquery version "1.0-ml";
  
declare namespace hs = "https://marklogic.com/xdmp/status/host";
declare namespace ss = "https://marklogic.com/xdmp/status/server";
 
(: set debug to fn:false() to actually cancel tasks :)
let $debug := fn:true()
let $seconds-to-run as xs:integer := 3600
let $seconds-to-pause as xs:integer := 3
(: Cancel tasks running longer than: :)
let $max-task-duration := xs:dayTimeDuration("PT7M")
let $max-iterations := xs:integer($seconds-to-run div $seconds-to-pause)
let $_ := xdmp:set-request-time-limit($seconds-to-run)
for $i in (1 to $max-iterations)
return (
  if ($debug) then (text{"Iteration", $i, "of", $max-iterations}) else (),
  xdmp:invoke-function(
    function () {
      for $host as xs:unsignedLong in xdmp:hosts()
      let $task-server-id := xdmp:host-status($host)//hs:task-server-id
      let $compare-time := fn:current-dateTime() - $max-task-duration
      for $request in xdmp:server-status($host, $task-server-id)//ss:request-status[
        not(xs:boolean(ss:canceled))
        (: and ss:user = 7071164300007443533 :)
        (: and ss:retry-count >= 2 :)
        (: and ss:request-text = "/some/module/uri.xqy" :)
        and ss:start-time < $compare-time
      ]
      let $request-id as xs:unsignedLong := $request/ss:request-id
      return (
        if ($debug) then (
          text{"Cancelling request:", $request-id, "with duration:", fn:current-dateTime() - xs:dateTime($request/ss:start-time) }
        ) else (),
        try { xdmp:request-cancel($host, $task-server-id, $request-id) } catch ($e) { }
      )
    },
    <options xmlns="xdmp:eval">
      <isolation>different-transaction</isolation>
    </options>
  ),
  if ($debug) then (text{"pausing for", $seconds-to-pause, "seconds"}) else (),
  xdmp:sleep($seconds-to-pause * 1000)
)

Required Privileges:

• https://marklogic.com/xdmp/privileges/xdmp-set-request-time-limit-any or https://marklogic.com/xdmp/privileges/xdmp-set-request-time-limit-my
• https://marklogic.com/xdmp/privileges/xdmp-invoke
• https://marklogic.com/xdmp/privileges/status
• https://marklogic.com/xdmp/privileges/cancel-any-request or https://marklogic.com/xdmp/privileges/cancel-my-requests

Discussion

When we ask for status information about the task server, we can find out how many tasks are queued, along with the task IDs of the tasks that are currently running, but there’s no way to inspect the queued tasks. We can clear the queue by restarting MarkLogic, but often that’s not desirable. Also, we might only want to remove some tasks from the queue. This recipe allows us to specify which tasks to remove.

Because we can only see the running tasks, we can only cancel tasks that have started running. This also means that if there are as many good tasks running (ones that we want to continue running) as there are Task Server threads (typically 16), we won’t be able to cancel any tasks until some of the good ones finish. For example, suppose we set some selection criteria and among the currently running tasks 12 of them do not match those criteria. That leaves just four that we can look at and try to cancel. During the next iteration of the loop, those four will have been replaced by four queued tasks (if there are some queued). If none of the 12 have finished, then we can only cancel four more.