Scrolling

By default, the LifeOmic FHIR Service Advanced Search will only return the first 10,000 results. In order to obtain more than 10,000 results, you must enabling scrolling.

Scrolling is supported by the LifeOmic FHIR Service DSL and LifeOmic FHIR Service SQL APIs.

Scrolling encapsulates the Elasticsearch Scroll API.

Quick Start

To enable scrolling for any LifeOmic FHIR Service Advanced Search request, set the HTTP parameter scroll=true:

POST /v1/fhir-search/projects/${project}?scroll=true HTTP/1.1
Host: api.us.lifeomic.com
LifeOmic-Account: ${account}
LifeOmic-User: ${user}
Authorization: Bearer ${token}
Content-Type: application/json

{
    "type": "select",
    "columns": "*",
    "from": [
        {
            "table": "patient"
        }
    ]
}

The response will contain an additional value called _scroll_id:

{
    "_scroll_id": "DnF1ZXJ5VGhlbkZldGNoBQAAAAAAAc2VFjNsTmJ6TmpYUnc2VmZ6Q2s2N1doS3cAAAAAAAHNmBYzbE5iek5qWFJ3NlZmekNrNjdXaEt3AAAAAAABzZYWM2xOYnpOalhSdzZWZnpDazY3V2hLdwAAAAAAAc2ZFjNsTmJ6TmpYUnc2VmZ6Q2s2N1doS3cAAAAAAAHNlxYzbE5iek5qWFJ3NlZmekNrNjdXaEt3",
    "took": 1,
    "timed_out": false,
    ...
}

To obtain the next page of results, set the HTTP parameter scroll to the _scroll_id from the response.

POST /v1/fhir-search/projects/${project}?scroll=DnF1ZXJ5VGhlbkZldGNoBQAAAAAAAc2VFjNsTmJ6TmpYUnc2VmZ6Q2s2N1doS3cAAAAAAAHNmBYzbE5iek5qWFJ3NlZmekNrNjdXaEt3AAAAAAABzZYWM2xOYnpOalhSdzZWZnpDazY3V2hLdwAAAAAAAc2ZFjNsTmJ6TmpYUnc2VmZ6Q2s2N1doS3cAAAAAAAHNlxYzbE5iek5qWFJ3NlZmekNrNjdXaEt3 HTTP/1.1
Host: api.us.lifeomic.com
LifeOmic-Account: ${account}
LifeOmic-User: ${user}
Authorization: Bearer ${token}
Content-Type: application/json

{
    "type": "select",
    "columns": "*",
    "from": [
        {
            "table": "patient"
        }
    ]
}

Repeat this process as necessary.

Once all results have been returned, subsequent requests will return an empty result set.

Python Example

#!/usr/bin/env python

import os
import requests

account = os.getenv('LO_ACCOUNT')
user = os.getenv('LO_USER')
project = os.getenv('LO_PROJECT')
token = os.getenv('LO_API_KEY')

url = f"https://api.us.lifeomic.com/v1/fhir-search/projects/{project}"

payload = {
    'type': 'select',
    'columns': '*',
    'from': [
        {
            'table': 'patient'
        }
    ],
    'limit': [
        {
            'type': 'number',
            'value': 0
        },
        {
            'type': 'number',
            'value': 1000
        }
    ]
}

headers = {
    'LifeOmic-Account': account,
    'LifeOmic-User': user,
    'Authorization': f"Bearer {token}"
}

scroll = 'true'
results = []
while True:
    params = {'scroll': scroll}
    r = requests.post(url, headers=headers, params=params, json=payload)
    body = r.json()
    scroll = body['_scroll_id']
    hits = body['hits']['hits']
    results = [*results, *hits]
    if len(hits) is 0:
        break


print(f"Collected {len(results)} results")

Scrolling Context

The scroll context TTL is 60 seconds. Once this time period has elapsed the _scroll_id will no longer be valid. Every subsequent request resets the TTL.

To be clear, you have 60 seconds to request the next page of results. You do not have to page over all results in 60 seconds.

The TTL cannot be changed.