Data Sources: connecting clinical systems to Patient Journey Intelligence
A data source in Patient Journey Intelligence is a registered, validated connection to a system that holds clinical data. The Data Sources module is where those connections are created, configured, credentialed, monitored, and retired. Nothing enters the ingestion pipeline except through a registered source or a Local Upload, so everything downstream depends on what is defined here.
Data Sources: registering and managing clinical data connections
This video demonstrates the interface visually without audio narration.
This video demonstrates the interface visually without audio narration.
Supported connectors by data type
A connector is the typed integration between Patient Journey Intelligence and one external system. Thirteen connector types are available today. Files on your own machine can also be imported directly with Local Upload, and two Databricks connectors are in development.
All 13 supported connectors, grouped by data type. Register an external source to use its connector; choose Local Upload during ingestion to import files from your machine.
The cards below follow the same groups. Each one names the connectors in that group and the kind of data they bring in. Where a step-by-step tutorial exists, the card links to it, so you can go from choosing a source to a finished ingestion without searching the docs.
EHR and clinical APIs
Epic (FHIR R4, backend OAuth 2.0), EHR, AWS HealthLake, and HTTP Plugin for RESTful endpoints. Brings in patient records, FHIR resources, and clinical notes.
File and object storage
Amazon S3, Google Cloud Storage, and SFTP. Brings in clinical documents such as notes, reports, and scanned PDFs, from a whole bucket or a single folder.
Tutorial: Ingest documents from S3 →
Tutorial: S3 to curated data →
OMOP CDM and data warehouses
OMOP CDM for a direct connection to an existing OMOP database, with optional table selection, and Snowflake for data warehouse tables.
Files from your machine
Local Upload, chosen when you start an ingestion rather than registered here. Takes FHIR R4 bundles, FHIR and HL7 CDA documents, and document batches with structured metadata.
Tutorial: FHIR R4 into OMOP CDM →
Tutorial: FHIR documents →
Tutorial: HL7 CDA documents →
Tutorial: Batches with OMOP metadata →
Medical imaging
DICOMweb for any DICOMweb-compatible imaging server, with four authentication options, and AWS HealthImaging for DICOM objects in an AWS datastore.
Databases
PostgreSQL and Elasticsearch. Beyond ingestion, both can be explored read-only, and PostgreSQL sources support saved row-level filters.
What each connector needs
Each connector type renders its own configuration fields in the source creation wizard and applies its own validation rules.
| Connector | What it brings in | Configuration |
|---|---|---|
| Amazon S3 | Clinical documents from an S3 bucket | The S3 bucket to read |
| Google Cloud Storage | Files from a Google Cloud Storage bucket | GCP cloud credential, bucket name, and project ID. An optional prefix restricts ingestion to a single path, so one connector can target a department folder such as radiology/2026/ instead of the whole bucket |
| SFTP | Files from an SFTP endpoint | Host details and secure paths |
| EHR | Data from an electronic health record source | Endpoint and authentication settings specific to the connector |
| Epic | Patient records and clinical notes over Epic's FHIR R4 API | Backend OAuth 2.0 with a private key held as an encrypted Epic credential. Patients are selected by search when you set up the ingestion |
| AWS HealthLake | FHIR-native structured data | An AWS HealthLake datastore |
| HTTP Plugin | Data from RESTful endpoints | HTTP methods, headers, and payload schemas |
| AWS HealthImaging | DICOM image objects | An AWS HealthImaging datastore |
| DICOMweb | Images from any DICOMweb-compatible imaging server | Server base URL and one of four authentication methods: none, basic, bearer token, or OAuth 2.0 |
| Snowflake | Data warehouse tables | Account, warehouse, database, schema, username, and password. Role, stage name, and filename are optional |
| OMOP CDM | Records from an existing OMOP Common Data Model database, with no file export step in between | Host, port, database name, schema, username, and password. Optionally select tables such as person, visit_occurrence, and condition_occurrence; an empty list imports every table |
| PostgreSQL | Tables from a PostgreSQL database | Host, port, database name, and credentials stored under the separate Database credential type, never on the source. TLS is optional: paste a certificate authority, or turn certificate checking off for test environments |
| Elasticsearch | Indices from an Elasticsearch cluster | Host, port, and authentication. TLS works as for PostgreSQL |
Local Upload is not a registered connector. You choose it instead of a data source when you start an ingestion job in the Data Ingestion module, and it is the usual route for FHIR and CDA content you already hold as files.
Connectors in development
Two connector types appear in the source type list so you can plan integration work against them. They are not yet available for operational use and will not complete a connection test until backend support ships.
| Connector | Planned capability |
|---|---|
| Databricks Unity Catalog | Access to files stored in Unity Catalog volumes, configured with a Databricks credential and a Unity Catalog volume path |
| Databricks Delta CDM | Access to OMOP CDM tables held in Databricks Delta Lake through a Databricks SQL warehouse, configured with a Databricks credential, SQL warehouse HTTP path, catalog, schema, and optional table selection |
Register a source in three steps
Selecting Add Source opens a guided three-step wizard.
Step 1: Name the source and choose its type
Enter a unique name and pick the connector type. Each type in the list carries a short description to guide the choice, and the two Databricks types are listed but not yet usable.
Step 2: Configure the connection
The wizard shows the fields for the selected type, as listed in What each connector needs. Typical parameters are bucket or datastore identifiers, endpoint URLs, the geographic region for cloud systems, credentials and authentication tokens, HTTP methods, headers, and payload schemas, and SFTP host details and paths. Advanced network options include private VPC routing and static IP allowlisting.
All credential inputs are encrypted, masked, and stored in compliance with enterprise security and healthcare regulatory requirements.
Step 3: Review and create
The final screen summarizes the source metadata, the connection parameters, and the selected ingestion schedule. Selecting Create Source makes the connector available to the Data Ingestion module immediately.
Manage sources from the connector registry
The registry is a searchable, sortable table of every connector, active and retired. Its audit metadata supports governance and compliance reviews.
| Field | Description |
|---|---|
| Source identifier | Descriptive label for the connection |
| Connector type | One of the connector types listed above |
| Created by / timestamp | Audit trail for the origin of the configuration |
| Last modified | Most recent update to parameters or schedule |
| Last successful connection | Confirms ingestion recency |
| Status | Active, Completed, or Failed |
| Ingestion schedule | None, Daily, Weekly, or Monthly |
| Actions | Edit, Delete |
Both actions are restricted by role through the platform's access control layer:
- Edit reopens the full configuration. Name, connection parameters, credentials, and schedule can change at any time without affecting previously ingested data. The connector type is locked after creation to preserve schema compatibility with that data.
- Delete removes the connector from active use and stops future ingestion jobs. All previously ingested data is retained.
Credential rotation
When you edit a source, existing credentials stay hidden. Leave the credential fields blank to keep the current keys, or enter new ones to rotate them. Keys stay out of the interface after setup, and a rotation never requires deleting and recreating the source or interrupting active ingestion jobs.
Schedule ingestion per source
Set the schedule during source creation and change it at any time:
| Schedule | Behavior |
|---|---|
| None | Ingestion is triggered manually from the UI or via API |
| Daily | Executes once per 24-hour cycle |
| Weekly | Executes on specific days of the week |
| Monthly | Executes on designated calendar dates |
Scheduled jobs are queued in the ingestion pipeline automatically, so the Data Ingestion module needs no further configuration.
Monitor connection health
Every registered source reports four signals in real time:
- Last visit timestamp: the most recent successful connection
- Status indicators: Active, Completed, or Failed
- Historical execution status and latency diagnostics: past run outcomes and connection latency, which identify degraded or intermittent sources
- Error logs: diagnostic output for resolving connection failures
When a job fails, its status changes to Failed and its error log appears here, so you can diagnose the problem and retrigger the job manually before a gap in source data reaches the downstream OMOP datasets or the NLP enrichment pipeline.
Database sources: explore and saved filters
PostgreSQL and Elasticsearch sources feed the ingestion pipeline like any other connector. They also let you explore the source structure and, for PostgreSQL, build reusable filters directly against it.
Exploring a source
Select a registered database source and click Explore to open a read-only interface.
- For PostgreSQL, browse tables and columns, see column types and sample values, and run read-only SQL queries. Queries return a preview and import nothing into the platform.
- For Elasticsearch, browse the available indices and run index queries.
Exploration is the fastest way to verify a connection, read the schema before building a filter, and spot-check content after a change at the source.
Saved filters on PostgreSQL sources
A saved filter defines a subset of rows in a PostgreSQL source. Build one with the point-and-click condition builder by picking columns, operators, and values, or write the query directly. Before saving, the platform reports how many rows the filter matches, so you validate the scope while it is still a configuration question.
Saved filters can be edited, run again, and deleted. They are private to the person who created them and cannot be shared.
Elasticsearch and saved filters
Saved filters are supported on PostgreSQL sources only. You can register and explore an Elasticsearch source, and filter saving is not yet available for Elasticsearch.
Example: a reusable lab population
A data engineer registers the institution's external lab database as a PostgreSQL source and builds a saved filter for HbA1c results above 9.0 in the past 12 months. The builder reports 4,212 matching rows, which matches the count the lab team expected, so the filter is saved. Running it again next quarter returns the current population without anyone rewriting the query.
How data flows from source to pipeline
After registration, data is scanned, downloaded, organized into batches, and staged in local processing storage, where the ingestion pipeline picks it up.
From source registration to processing-ready files. External clinical systems, including object storage, EHR and FHIR services, imaging servers, and data warehouses, are registered, scanned, downloaded, organized into batches, and staged in local processing storage for downstream extraction and modeling.
Every ingestion job then runs the full pipeline: NLP extraction from unstructured notes, OMOP conversion of structured and semi-structured data, terminology normalization across 40+ standard vocabularies, patient-level deduplication, and clinical reasoning and enrichment.
Registered sources cover all three clinical data modalities: structured data (FHIR resources, EHR extracts, Snowflake tables, PostgreSQL tables, and records imported directly from an existing OMOP CDM database), unstructured data (clinical notes, scanned documents), and imaging metadata (DICOM and DICOMweb). With encrypted credentials, role-based access control, and an audit trail for every connection, the module is the governance anchor for the data supply chain feeding the platform.
FAQ
Thirteen connector types are available today: Amazon S3, Google Cloud Storage, and SFTP for files; EHR, Epic, AWS HealthLake, and HTTP Plugin for health records and clinical APIs; AWS HealthImaging and DICOMweb for imaging; Snowflake and OMOP CDM for warehouses and OMOP data; and PostgreSQL and Elasticsearch for databases. Databricks Unity Catalog and Databricks Delta CDM appear in the list but are still in development and will not complete a connection test.
All credential inputs are encrypted, masked, and stored in compliance with enterprise security and healthcare regulatory requirements. When you edit a source, existing credentials stay hidden: leave the fields blank to keep them or enter new values to rotate them. PostgreSQL and Elasticsearch credentials are held under the separate Database credential type and are never saved on the source.
Yes. Deleting a connector removes it from the active registry and stops future ingestion jobs. All previously ingested data is retained in the platform.
No. The connector type is locked after creation to preserve schema compatibility with ingested data. Name, connection parameters, credentials, and schedule can all be changed at any time without affecting previously ingested records.
Each source has a schedule of none, daily, weekly, or monthly, and scheduled jobs are queued in the ingestion pipeline automatically. If a job fails, the source status changes to Failed and the error log appears in the monitoring view, so you can diagnose the problem and retrigger the job manually.
Both can be explored read-only: browse PostgreSQL tables, columns, types, and sample values and run read-only SQL, or browse Elasticsearch indices and run index queries. PostgreSQL sources also support private saved filters that report a match count before saving. Saved filters are not yet available for Elasticsearch.
Use the OMOP CDM connector, which connects directly to the database with no file export step. It needs the host, port, database name, schema, username, and password. Select tables such as person, visit_occurrence, and condition_occurrence, or leave the list empty to import every table. The OMOP CDM tutorial also covers patient identifier remapping.
The Snowflake account, warehouse, database, schema, username, and password. Role, stage name, and filename are optional.
No authentication, basic authentication, bearer token, and OAuth 2.0. The connector works with any DICOMweb-compatible imaging server and also needs the server base URL.
Yes. The Google Cloud Storage connector accepts an optional prefix that restricts ingestion to a single path inside the bucket, such as a department folder.
It is the entry point for all external data. No ingestion job runs without a registered, active data source, except a Local Upload. Once a source is registered, the Data Ingestion module runs NLP extraction, OMOP conversion, terminology normalization, deduplication, and enrichment.
Related tutorials
Walkthroughs that start from a data source or a Local Upload, in the Patient Journey Intelligence cookbook:
| Data source | Tutorial | What it covers |
|---|---|---|
| Epic | Ingest Epic EHR patients and clinical notes over FHIR R4 | Epic app, key pair, and credential setup, then patient search and ingestion |
| Amazon S3 | Ingest clinical documents from an S3 source | Registering an S3 source and ingesting its documents |
| Amazon S3 | Turn clinical documents into curated, reviewable data | S3 ingestion through cohort, dataset, and data curation |
| OMOP CDM | Connect an existing OMOP CDM database | Direct connection, patient identifier remapping, and import |
| Local Upload: FHIR R4 | Ingest FHIR R4 into OMOP CDM, with or without NLP | A FHIR bundle or archive converted to OMOP CDM records |
| Local Upload: FHIR documents | Ingest external FHIR documents without patient linkage | Document-first ingestion with no OMOP person record |
| Local Upload: HL7 CDA | Ingest external HL7 CDA documents without patient linkage | Document-first ingestion with no OMOP person record |
| Local Upload: document batches | Link document batches to patients with structured OMOP metadata | JSON metadata files that tie each document to a patient, visit, and provider |