Sources
Keep a Google Drive or SharePoint folder in sync
Connect a folder once and it stays current. Importing a folder copies what is in it today. Keeping it in sync brings over every later change on a schedule, so the answers your team gets track the documents your team actually edits, with nobody remembering to re-import.
There is nothing to set up beyond the account you already linked. No key to paste, no policy to write, no IAM step. Engram reads the folder through the connection you authorized, and only fetches what the provider says has changed since the last check, so a nightly sync over a large library costs very little.
Two places to turn it on
- In the setup wizard, on the Review step after you import a folder. Pick how often to check and whether deletions should follow, and press Keep this folder in sync.
- On the Documents tab of any document base, in the Sources section at the top. Choose Keep a folder in sync, pick the account and the folder, and save. Google Drive opens Google's own picker; SharePoint opens a browser of your sites, libraries and folders.
Choose a folder rather than loose files. A schedule watches a folder, so that is what Engram needs to know about. For Google Drive the folder you pick in Google's own picker is also what grants Engram access to it, which is why only folders you have picked ever appear.
An Amazon S3 bucket is set up from the CLI instead, because its last step is a role in your own AWS account. See Amazon S3 buckets.
What a sync does
Two settings decide what a sync does:
| Setting | Options | What it means |
|---|---|---|
| Check for changes | Every 15 minutes, Hourly, Every 6 hours, Daily | How often Engram looks. Fifteen minutes is the floor. |
| What a sync does | Add and update | New and changed files come over. Documents already here stay, even when the file is deleted there. |
| Mirror the folder | The document base matches the folder. A file deleted there is removed here, and only documents this folder delivered are ever removed. |
A sync stays inside its own lane, which is what makes a mirror safe on a busy base. A document
belongs to the folder that delivered it, identified by the document base, the source and the file's
path inside the folder, and a sync only ever touches its own folder's documents. A mirror never
removes a website upload, an engram push, or another source's documents, and anything
uploaded or pushed belongs to no source at all.
A filename belongs to whoever delivered it first. If two connected folders on one base both hold a
reports/keep.txt, the document that is already there keeps the name and the other sync
counts the file as path_taken, so the clash is reported rather than settled silently
behind you. Give one of the two its own name, folder or base and both come through. The same rules for
every kind of source are on Additive or mirror.
Watching it, changing it, stopping it
Everything lives on one tab. The folder has its own card in the Sources section at the top of the Documents tab, counting the documents it has delivered, with Sync now to pull straight away and Remove to stop it. To go through those documents one by one, press Download manifest on the card: the CSV lists every file with its status, any failure reason and when it last changed. See Download a manifest. Every sync shows up in Sync history at the bottom of the tab, with what it added, updated, skipped and removed. A skipped file is counted by reason rather than dropped quietly, and the reasons are on What gets skipped.
Changing the schedule or the mode is one command, and nothing already ingested is disturbed:
engram sources update --corpus Sales --source <source id> --every 6h --mode mirrorOver the API that is PUT /v1/corpora/{id}/sources/{source_id}. Removing a source ends
future syncs and leaves the documents it already delivered in place, moved under the Website
uploads card.
What the source on a run means
Every sync run carries a source, and so does the sync_run.completed
webhook payload. It says which route the documents came in by, and two of the six names are not the
word you would guess, so here is the whole set:
source | The run came from |
|---|---|
upload | A single document written through the API with POST /v1/corpora/{id}/documents/upsert, which is also what engram docs upsert calls. One call, one document, no manifest. |
push | A manifest commit: engram push from the CLI, or any client that diffs a folder and commits it. This is the one to filter on for a CLI push. Files dropped on the Documents tab open no run at all; they appear in the document list, not the run history. |
s3 | A registered Amazon S3 bucket, on its schedule or a Sync now. |
google_drive | A connected Google Drive folder. |
sharepoint | A connected SharePoint library. |
backfill | A platform-side reprocessing job, not something you started. |
The two to watch are upload and push. A CLI
engram push reports upload, because it commits a manifest like the
Documents tab does; the API's documents/upsert reports push. Switch on the
value, not on the English word.
Which plan turns this on
Keeping a source in sync comes with the Pro plan and every tier above. SharePoint appears once your workspace has it enabled; Google Drive is available now.