When trying to retrieve all file objects, getting only part of the items in the vault/class

I am trying to get all the objects in a vault/class into my allFiles variable.  For some reason, I get odd results.  I should be getting back about 2.4 million records but I am only getting back about 2.2 when I try to just pull everything.  When Trying to pull back as in the code below in batches of 500, or 5000 I also get weird results.  When using a batch size of 500, I get back ~1 million files.  When I try to pull back with the exact same code in batches of 5000, I get back on 750K records.  Not sure if I am running into some issue with my pagination or some other limition but doesnt make sense to me.  

Here is my code : 

public static List<ObjectVersion> GetFileBatchWithConditionsEx(
Vault vault,
IEnumerable<SearchCondition> baseConditions
)
{
const int batchSize = 500;
var allResults = new List<ObjectVersion>();
int lastSeenID = 0;

while (true)
{

var conditions = new SearchConditions();
foreach (var c in baseConditions)
conditions.Add(-1, c);

var idFilter = new SearchCondition
{
ConditionType = MFConditionType.MFConditionTypeGreaterThanOrEqual,
Expression = new Expression
{
DataStatusValueType = MFStatusType.MFStatusTypeObjectID
},
TypedValue = new TypedValue()
};
idFilter.TypedValue.SetValue(MFDataType.MFDatatypeInteger, lastSeenID);
conditions.Add(-1, idFilter);

var results = vault.ObjectSearchOperations
.SearchForObjectsByConditionsEx(
conditions,
MFSearchFlags.MFSearchFlagReturnLatestVisibleVersion,
true, // sort
batchSize, // <-- maxResults
3600 // timeout
);

if (batch.Count == 0)
break; // we’re done

allResults.AddRange(batch);

WaiterService.UpdateStatus($"Fetched {allResults.Count} records so far...");

lastSeenID = batch.Max(o => o.ObjVer.ID) + 1;
}

return allResults;
}

Assume it could be something to do with sorting or something similar...

Ideas ?

Thanks in advance,

Pat

Parents Reply
  • So the segmented search seems to be the secret sauce for how to do this.  Appreciate that!  One other follow on question is that we do have a few files that dont seem to be showing up.  Out of 2.24 million, we are missing about 150 files.  Only filters I have set are for deletion and latest version only.  Ran my process twice and still missing the same files.  Any other gotchas that might account for this? I am using the reporting db to compare the numbers as a way to validate. 

Children