How to always force delta refresh a specific external OT

What is your use case or the issue you are facing with the full refresh?

Syncing large amount of data (years worth of emails) majority of which will be unchanged anyway. So we want to sync only recently modified items.

If the database table has a column for the modification timestamp or you can add such a column, have a look at the section Updating Recently Modified Data Only in the guide. This configuration option has been designed for these kinds of situations.

Will this work for custom object type connectors as well? and what format should the query be in this case?

This configuration is added in Advanced Vault Settings, refer to the guide for details. It may work for custom object type connectors as well but I can't say for sure. If the timestamp column is already there it should be pretty easy and quick to test.

I did try that, but as far as I can see, these settings have no effect for my custom connector (during nightly maintanance, it is retrieving all items)...
There is also this note in the guide, which I think explains this behavior? Although it is mentioning the setting from legacy database connection, and the connector in question here is using the new Application connection model

It is worth mentioning that this is not data stored in database, rather it is fetched from the web service and range retrieval is supported in the connector (SetTimestampRangeToRetrieve method is overriden appropriately).

Hi Antonia,

For the delta approach you need to make sure you implement the other interface - IDataSourceConnection2, from recollection.

Have you done this?  If so then the delta approach used by the first-party connectors should also be available to you (as there is nothing special about the first-party ones; they implement the same interfaces).

As you said there is some logic around which of these statements is used in different situations, but it should operate according to the documentation (and, again, be consistent between first- and third-party connectors).

Yeah, we did that awhile back, the advice back then was we had to override SetTimestampRangeToRetrieve and IsPartialResultSet.

Just double checked that email and realized my terminology is wrong yet again, so let me rephrase in correct terms now - we want to always use delta refresh and never the regular full/quick refresh where all the data is loaded into mfiles and then synced.

Delta implementation is working because I can see that when I refresh manually using the Admin, the request includes range parameter to limit the number of results coming into m-files.

What is not working is nightly maintanance job -  this one is not sending delta parameter no matter how I configure Advanced Vault Settings.

Yeah, we did that awhile back, the advice back then was we had to override SetTimestampRangeToRetrieve and IsPartialResultSet.

Just double checked that email and realized my terminology is wrong yet again, so let me rephrase in correct terms now - we want to always use delta refresh and never the regular full/quick refresh where all the data is loaded into mfiles and then synced.

Delta implementation is working because I can see that when I refresh manually using the Admin, the request includes range parameter to limit the number of results coming into m-files.

What is not working is nightly maintanance job -  this one is not sending delta parameter no matter how I configure Advanced Vault Settings.

Let me check with the team. I think there are always situations where the system could choose to do a full refresh, but let me check about nightly maintenance. 

Much appreciated.

I understand from the guide that Refresh Time Filtering setting should do what I am asking, but I am not sure how to make it work with custom connector.

As I said: the first-party connectors (OLEDB/Salesforce/CRM) use the same interfaces as you do; the server doesn't differentiate.  If the settings in the vault are correct then the behaviour should be consistent.

Understood, but this specific setting (SELECT statement) looks to me like it is from the legacy connections, so it might not work with new application connections where we no longer thave these SELECT/UPDATE/DELETE statements

I've not forgotten about this.  I have just taken a quick look (I've had quite a few calls today!) and I can see why you're questioning this.

I might have time to create a quick ExtOTDS tomorrow to check this out.  Of course if you have one that I can use for testing then that would save me some time.

I am using our REST OT Connector for this, if that will be of any help, I can send you vault backup, or whatever other format might be helpful (configs, endpoints, whatever?)

I probably need the source to be able to check what's happening.  I'll try to create something later if I get some time.

I can send you that as well, since we did send this for validation anyhow