Table of Contents

This guide describes how to tune LinuxForHealth FHIR Server and its database to get the best performance. It also describes different FHIR query strategies which may help to work around specific performance issues.

Note: all logical-ids and resources in this guide are examples and do not refer to actual patient data.

The sizing table below should be considered a starting point. Actual requirements may vary greatly based on the specific scenarios for a given deployment. For example, search-heavy workloads will require more database CPU and IOPS capacity than a system servicing simple reads.

CPU consumption of the LinuxForHealth FHIR Server is closely correlated with the number of resources being processed, particularly during ingestion where processing involves:

The following sizes are guidelines only. You should test and measure for your specific use-cases.

Data Volume and Load Examples

Example System Sizing

Note and Assumptions:

Terminology:

This section describes how to configure the LinuxForHealth FHIR Server and its database for concurrency.

Liberty Profile uses an executor service to handle incoming HTTP/S requests. By default, the executor service automatically adjusts its thread pool size to most efficiently handle the request load. Although the executor service can be configured, we recommend using the default configuration. The best solution for supporting greater concurrency is to scale-out additional instances of the LinuxForHealth FHIR Server.

PostgreSQL limits the maximum number of open connections. It is important to configure the database in conjunction with the Liberty Profile datasource connection pools to avoid connection failures which will result in HTTP 500 errors being returned from the LinuxForHealth FHIR Server.

Assuming there are N instances of the LinuxForHealth FHIR Server, the recommended connection limits should be configured as follows:

See Managing PostgreSQL Connections in the IBM Cloud documentation for more information.

The recommended approach for tenant datatstore configuration is to use individual JTA datasources, each with their own connection manager (connection pool):

Because each datasource gets its own connection manager, you can tune each independently. If multiple datasources point to the same database (for example using different schemas to support multi-tenancy) be sure to configure the database max_connections accordingly. Also, remember to sum the maxPoolSize for all datasources across all LinuxForHealth FHIR Server nodes in your deployment. The numConnectionsPerThreadLocal value should be set to 1. This improves concurrency and reduces the amount of time it takes to acquire a connection, especially on systems with large core counts.

When numConnectionsPerThreadLocal=1, be aware that connection-related errors will cause the client request to fail with a status 500 Server Error. In addition, this failure will not purge all bad connections and so clients may see multiple errors. This can occur when there is a database failover event, for example. When numConnectionsPerThreadLocal=0 and validationTimeout is configured (see above), connections are validated each time before they are used. In this case, the chance of a client request failing due to a bad database connection is much lower.

If clients need to be protected from this type of error in addition to achieving high concurrency, the recommendation is to set numConnectionsPerThreadLocal=0 and scale out multiple instances of the LinuxForHealth FHIR Server over a larger number of smaller nodes instead of a smaller number of larger nodes. This helps to reduce contention on the mutex locks protecting the application server connection pool.

Each JTA datasource should be configured in its own .xml server configuration file and placed into {fhir-server-home}/configDropins/overrides where it will be picked up automatically by Liberty Profile on startup.

Long transactions consume significant resources so to protect the system, Liberty will time-out a transaction after 2 minutes (120s) by default. When a transaction times out, Liberty will forcibly close any database connection currently executing a statement and the LinuxForHealth FHIR Server will return an HTTP 500 response to the caller. The maximum transaction time can be modified using the <transaction> element in the Liberty server configuration and, by default, the LinuxForHealth FHIR Server will set this from the FHIR_TRANSACTION_MANAGER_TIMEOUT variable as described in Section 3.3.1.3 Database Access TransactionManager Timeout of the LinuxForHealth FHIR Server User’s Guide.

The following table summarizes how the transaction timeout is used for different request types:

Because some requests use multiple transactions under the covers, the overall request response time can sometimes be greater than the transaction timeout. There is no server-side tuneable property for the overall request processing time. Tuning of the client read timeout and/or network configuration may be required when extending the maximum transaction time to more than 2 minutes, or supporting multi-transaction requests which also exceed 2 minutes.

Firewalls or other components in the flow between a client and the LinuxForHealth FHIR Server may forcibly close (reset) a connection which is considered idle. This is usually because no packets associated with the TCP connection will flow between the client and the LinuxForHealth FHIR Server until the response is returned. There are two ways to address this:

TLS connection setup is a costly CPU operation. It is therefore important to ensure that routing components are configured for session affinity to avoid unnecessary connection setup costs. Clients should be written to reuse connections when making multiple requests.

The LinuxForHealth FHIR Server uses internal memory caches for resource type names, parameter names, references, codes and systems. These caches use a least-recently-used (LRU) strategy to avoid unbound growth which would result in an out-of-memory (OOM) condition.

Currently the LinuxForHealth FHIR Servers do not use sharding(*) to distribute client requests and so any client request may hit any server in a given cluster. This means that the memory used for caching value-id lookups is not distributed, but each servers’ cache is complete and may contain data also cached in another server. This may be revised in future releases if it becomes a scaling issue.

(*) - sharding is really the responsibility of the component used to route requests to the LinuxForHealth FHIR Servers, and if such a component were to support sharding, it would help to reduce the cache pressure by distributing values among the available cache instances.

The following datasource properties in fhir-server-config.json are used to tune the size of the value-id caches:

The caches are isolated by tenant and specific to each datasource defined for that tenant:

Currently no cache-hit metrics are exposed related to the caches. Tuning relies on Application Performance Management (APM) profiling and monitoring database activity looking for frequent value-id lookups against the following tables:

The values for PARAMETER_NAMES and RESOURCE_TYPES are supposed to be fully cached. Any substantial reads (selects) from these tables after initial startup/first request should be considered a defect.

Resources are assigned to various compartments using expressions with multiple terms. In the LinuxForHealth FHIR Server JDBC persistence layer, these expressions are translated to SQL predicates with multiple OR statements. These ORs make it more difficult for the query optimizer to compute the most efficient execution plan resulting in a slow query. To address this, the LinuxForHealth FHIR Server evaluates the compartment membership expression during ingestion and stores the results. The SQL query can then be written using a single value predicate resulting in faster query.

the LinuxForHealth FHIR Server has a dynamic registry of conformance resources. The built-in “ServerRegistryResourceProvider” can be used to bridge conformance resources from the tenant data store (uploaded through the REST API) to the registry. When enabled, this means that each call to the registry (e.g. for extension StructureDefinition lookups during resource creation) can result in a round trip to the database.

For optimal performance, the LinuxForHealth FHIR Server team recommends to disable this resource provider via the following setting:

This configuration setting avoids an extra Search during ingestion.

The LinuxForHealth FHIR Server supports multi-tenant SearchParameter extensions described in the extension-search-parameters.json file. When the extension-search-parameters.json is missing, the SearchParameter value extraction tries to open the file for every resource. This is a file-system operation which results in a context switch and impacts performance.

The LinuxForHealth FHIR Server team recommends each tenant include an extension-search-parameters.json file, even if it is empty.

An example of the empty search parameters file is:

For PostgreSQL, we recommend tuning the following properties:

The recommended values should be considered a starting point. Monitor database metrics and tune appropriately for your given workload. See the PostgreSQL wiki for additional guidance.

FHIR search queries are translated into SQL expressions. When several search parameters are included in a request the resulting join spans many tables, several of which may be wrapped in views. In order to give the PostgreSQL query optimizer sufficient freedom to optimize the query, include the searchOptimizerOptions parameter map to the datasource configuration as shown below to increase the values for from_collapse_limit and join_collapse_limit. The default for both of these is 8. When there are more than 8 tables involved in the search query, the optimizer may not generate an efficient execution plan unless the limits are increased:

See the PostgreSQL Query Planning guide for more information.

In PostgreSQL, the default fillfactor for each table is 100 - no room is reserved for updates. This maximizes storage utilization, but impacts performance for updates which occur when new versions of a resource are ingested. Update statements are also used frequently during the reindex process, if index IDs are not specified.

To provide space for updates, all the <resourceType>_logical_resources should be configured with a fillfactor of 80 as a starting point. DBAs may specify their own fillfactor values based on their own knowledge and understanding of the system.

The fillfactor for the logical_resources table may benefit from an even lower value to support the heavy update load during a reindex operation, if index IDs are not specified. This is a special case due to the fact that every row in the table is updated once.

To change the fillfactor for existing data, a VACUUM FULL operation is required:

This should only be performed during a maintenance window when there is no load on the system.

When running reindex operations (after a search parameter configuration change, for example) without specifying index IDs, the logical_resources table undergoes frequent updates to an indexed column. Due to the nature of how PostgreSQL handles updates, this results in a significant amount of old index blocks which slows progress. The table storage parameters may need to be tuned to vacuum the logical_resources table more aggressively. To address this, tune the storage parameters for this table as follows:

The default value for autovacuum_vacuum_cost_limit is likely too restrictive for a system with good IO performance. Increasing the value to 2000 increases the throttling threshold 10x, significantly improving throughput and helping the logical_resources vacuuming to be completed before it negatively impacts performance of the reindex operation (when index IDs are not specified on the reindex operation).

See the PostgreSQL VACUUM documentation for more details.

In addition, administrators may also choose to run a manual vacuum as shown in the following example:

Be ware of multixact wraparound issues, as highlighted by the following warning when running a manual vacuum:

This indicates that the automatic vacuum process needs to be more aggressive. See here for details.

Use the following query to see the impact of updates and deletes on the LinuxForHealth FHIR Server tables (assuming the tenant is configured to use the fhirdata schema):

The values reported are since the database was last restarted. To check uptime, run the following query:

The following query can be used to see how many auto-vacuum jobs are currently in progress and for which tables:

By default, only 3 vacuum jobs can run concurrently.

To drop an LinuxForHealth FHIR Server schema in PostgreSQL, set the following configuration in postgresql.conf:

This change requires a database restart.

Derby is not recommended for production use and therefore tuning Derby will not be addressed in this guide.

Using random values for resource identifiers can cause performance issues in large databases. This is a particular issue when using PostgreSQL with the LinuxForHealth FHIR Server due to an issue known as write amplification from full page writes. For details, see this blog post: https://www.2ndquadrant.com/en/blog/on-the-impact-of-full-page-writes.

For best performance, ids generated by clients should not be purely random but instead be structured to include a prefix which increments over time. This causes index entries for new values to share pages (right-hand inserts), greatly reducing the write amplification overhead.

One example of a suitable id generation strategy can be found in the LinuxForHealth FHIR Server fhir-persistence-jdbc project.

This strategy provides both the desirable trait of global uniqueness as well as a low write amplification overhead thanks to the time-based prefix.

The LinuxForHealth FHIR Server also uses normalization to avoid storing (and indexing) long identifier strings in multiple places. This saves space, and the database-generated identity values are based on sequences which naturally produce the desired right-hand-insert behavior.

In scenarios where the server is not the source of truth, clients may want to reload/refresh the server with all of their data on some periodic basis.

One technique for this is to use client-assigned resource ids and perform an HTTP PUT (update or create-on-update) with the content on each ingestion run. However, this can lead to unnecessarily updating each FHIR resource on each ingestion run.

Avoiding these unnecessary updates is important for two reasons:

The HL7 FHIR specification defines experimental support for both conditional create and conditional update and the LinuxForHealth FHIR Server implements each of these. However, this approach suffers multiple issues:

Beginning with LinuxForHealth FHIR Server version 5.0.0, the server implementation performs optimization to avoid unnecessary updates. The server will only create a new resource version if the resource contents from the update differ from the contents of the resource in the database.

Two resources will be considered equivalent based on the following criteria:

When the update is skipped, the response will contain a Location header that points to the existing resource version (e.g. [base]/Patient/1234/_history/1) instead of a newly created instance of this resource ([base]/Patient/1234/_history/2) and the response body will be sent according to the client’s return preference. If the client indicates a return preference of OperationOutcome and the update is skipped on the server, the response will contain an informational issue to indicate this case.

To opt out of this optimization, users can force the server to perform an update by setting an HTTP header named X-FHIR-FORCE-UPDATE to “true”.

The LinuxForHealth FHIR Server translates a FHIR search request into a SQL query. The database performs query optimization to generate what it thinks is the most efficient execution plan before running the query. This optimization depends on the database having good statistics (and a clever algorithm) to make the right choice. When this goes wrong, the result is a slow response which can also end up consuming significant resources which impact the capacity of the system as a whole.

The FHIR search specification includes a rich set of capabilities designed to make it easier for clients to find data. If a particular search performs poorly, there are likely other ways the same data can be fetched. One solution is to use multiple requests, using FHIR bundle requests to request multiple resources in one server request.

There are many ways to retrieve data:

There may also be some subtle semantic differences among searches which might appear to be equivalent. This is particularly true for compartment-based queries due to the complex definition of compartment membership defined in the FHIR specification.

Logical id-based read requests are the fastest way to access a resource, for example:

This translates into a single query which utilizes indexes to quickly locate the required record:

When the data is currently cached by the database, this query takes less than a millisecond to execute, as shown in the following execution plan analysis from PostgreSQL:

The FHIR specification supports reading a specific version of a resource:

The resulting query is similar, except in this case, the specified version is requested from the PATIENT_RESOURCES table. Performance is similar to the plain read, depending on what data is currently cached, of course:

The history query returns all versions of a resource. Because there is no limit to the number of versions for a given resource, the results are ordered by the version_id (resource version number) and paginated using OFFSET and FETCH NEXT ROWS clauses:

In order to include an overall count of the number of resource versions, the LinuxForHealth FHIR Server must execute an additional SQL query:

In most cases the history queries will execute very quickly. Performance will be slower for cases where a single resource has thousands of versions. To avoid this, ingestion pipelines must ensure they only update a version when necessary.

Omitting the count

For search queries with low specificity, the response time is dominated by the “count query” that is used to determine how many total results match the query. The LinuxForHealth FHIR Server supports skipping this step when clients set a query parameter named _total to the value of none as described at https://hl7.org/fhir/R4B/search.html#total.

Resource subsetting

For search queries that return lots of data (e.g. ones that return large resources such as Patient resources that embed a profile picture), the response time can be dominated by the network between the client and the server. By default, a FHIR search response bundle will contain the entire contents of each resource being returned. However, it is possible to request the server to return a subset of each resource through either the _summary or _elements parameters.

Predicate Order

The LinuxForHealth FHIR Server translates FHIR search queries into SQL statements which may require many tables to be joined. The database attempts to optimize the query execution plan by analyzing join conditions, filter predicates, available indexes and column statistics. The optimizer also attempts to order the joins in order to reduce the amount of work it must do. This usually involves computing the most selective clauses first. When there are many tables involved, the database optimizer may not always find the most efficient execution plan which can result in higher response times or 500 server errors if the total time exceeds the transaction timeout limit. For example, on a large database the following query may perform poorly if there are many ExplanationOfBenefit records with a Claim matching one of the given priorities:

The above query requires a join of around 13 tables which is too many for the database to try all possible orders and so the most efficient plan is never tried. The LinuxForHealth FHIR Server builds the SQL based on the order of filter predicates in the search request. This can be used along with knowledge of the data to place the most selective filter first which, in this case, is the patient birthdate range. Rewriting the query as follows can significantly improve the response time:

The section contains search examples and performance considerations for various types of search parameters.

STRING SEARCH Because the default behavior for string search has case-insensitive “begins-with” semantics, we encourage users to use the :exact modifier when possible for the best performance.

TOKEN SEARCH HL7 FHIR supports a few variants of token search:

Token-based searches should include a code-system when possible. The same code value might exist in multiple code-systems and so, unless the code-system is included in the search query, the database join may need to consider multiple matches in order to find all the associated resources. This multiplies the amount of work the database must do to execute the query. This also impacts cardinality estimation by the optimizer. If both the code-system and code value are provided, this matches a unique index in the schema allowing the optimizer to infer the SQL fragment will produce a single row.

For optimal performance, users should prefer the [system]|[code] variant. Explicitly providing the code is always preferred. If no system is provided, in some cases the LinuxForHealth FHIR Server can determine the correct code-system to use automatically, which helps query performance.

Don’t do this: Patient/175517d8bea-32d33eec-d98f-4c99-a3cf-06a113ddcf08/CareTeam?status=active

Instead, do this: Patient/175517d8bea-32d33eec-d98f-4c99-a3cf-06a113ddcf08/CareTeam?status=http://hl7.org/fhir/care-team-status|active

This is especially important for code values that are common across systems (e.g short strings like “active”). However, the LinuxForHealth FHIR Server supports a SearchParameter extension which allows the server to add an implicit [system]| prefix for certain token parameter searches that come in with just a [code].

For example, when a search parameter targets an element of type Code that has a required binding, there is typically a single implicit system for the code. In such cases, the LinuxForHealth FHIR Server models these Code subtypes as Enums and the values are indexed with their implicit system. When the corresponding SearchParameter definition is decorated with the same implicit system (via this extension), it provides maximum specificity for the query to efficiently retrieve the token value.

For all such search parameters in the base spec, and first-class implementation guides that we package, we have taken the liberty to add these extensions a priori. For example, for SearchParameter-Account-status, the following extension tells the server to process a query like Account?status=active as if it were specified like Account?status=http://hl7.org/fhir/audit-event-outcome|active:

COMPARTMENT SEARCHES

Search for all ExplanationOfBenefit records which reference the patient according to its logical id 17546b5cd4a-cae29bb0-d6d7-4a1a-a8df-505e2e3a975e.

Search the patient compartment 17546b5cd4a-cae29bb0-d6d7-4a1a-a8df-505e2e3a975e for all matching ExplanationOfBenefit resources.

COMPARTMENT CHAINED SEARCH

Requests any ExplanationOfBenefit resources belonging to the compartment for patient 17546b5cd4a-cae29bb0-d6d7-4a1a-a8df-505e2e3a975e with a Claim created on a given day: 2015-10-16.

CLIENT SEARCH BUNDLES

If the response time is not acceptable, an alternative strategy is to first fetch all the ExplanationOfBenefit resources associated with the given patient, then compose a second bundle request for the Claim records associated with the ExplanationOfBenefit resources returned by the first search. The initial request can fetch the Patient resource along with the ExplanationOfBenefit resources for that patient.

Note the bundle type is specified as transaction. Because all the entries in the bundle are reads, there’s no modification to the database and so no semantic difference between using transaction or batch. However, for transaction, the LinuxForHealth FHIR Server initiates a single transaction for the entire request. In high load/concurrency conditions, this improves throughput.

The number of ExplanationOfBenefit resources for a given patient is likely to be relatively small. By iterating over the response to the previous request, the client can package a number of resource reads into one or more bundle requests. Read requests are the most efficient type of request for accessing data from the LinuxForHealth FHIR Server.

Although this approach involves multiple requests to the LinuxForHealth FHIR Server, the overall response time may be quicker due to the simplicity of the requests.

The client can filter the required Claim resources, or alternatively convert the reads to search requests:

INCLUDE

Another alternative which tends to perform well uses the _include function. The _include function is useful because it avoids the need for additional round-trips (the server performs the iteration internally).

This example retrieves ExplanationOfBenefit resources for patient 17478598887-c1279929-f2ca-48e5-815a-b812bfc2e756 which were created between 2010 and 2020. The result bundle also includes the Patient, Provider, CareTeam and Coverage resources referenced by each matching ExplanationOfBenefit resource.

REVINCLUDE

Used when a client wants a particular resource and other resources pointing to it. For example, to retrieve a Patient resource and any Observations referring to that patient as the subject:

Find all Observations matching the composite code-value-quantity value and for each such Observation, also return any DiagnosticReport referring to that Observation as the result:

The above query is likely to be expensive if the number of matching Observations is high. For better performance, consider constraining the search with additional predicates like the patient id.

LAST UPDATED

Searching for all resources updated on a single day is expensive:

If possible, limit the search to a specific resource type. For example:

If multiple resource types are of interest, consider packaging multiple search requests into a bundle:

Note that the tenant header must match the tenant header defined in the fhir-server-config.json. The default tenant header is X-FHIR-TENANT-ID. If this header is not provided, the tenant value will be default.

The curl command can also be used to make POST calls to the LinuxForHealth FHIR Server. This example creates a new Observation resource for the patient subject abc123:

Examples of valid resources can be found in the fhir-examples project.

See FHIR client API in the LinuxForHealth FHIR Server User’s Guide.