When trying to retrieve all file objects, getting only part of the items in the vault/class

I am trying to get all the objects in a vault/class into my allFiles variable.  For some reason, I get odd results.  I should be getting back about 2.4 million records but I am only getting back about 2.2 when I try to just pull everything.  When Trying to pull back as in the code below in batches of 500, or 5000 I also get weird results.  When using a batch size of 500, I get back ~1 million files.  When I try to pull back with the exact same code in batches of 5000, I get back on 750K records.  Not sure if I am running into some issue with my pagination or some other limition but doesnt make sense to me.  

Here is my code : 

public static List<ObjectVersion> GetFileBatchWithConditionsEx(
Vault vault,
IEnumerable<SearchCondition> baseConditions
)
{
const int batchSize = 500;
var allResults = new List<ObjectVersion>();
int lastSeenID = 0;

while (true)
{

var conditions = new SearchConditions();
foreach (var c in baseConditions)
conditions.Add(-1, c);

var idFilter = new SearchCondition
{
ConditionType = MFConditionType.MFConditionTypeGreaterThanOrEqual,
Expression = new Expression
{
DataStatusValueType = MFStatusType.MFStatusTypeObjectID
},
TypedValue = new TypedValue()
};
idFilter.TypedValue.SetValue(MFDataType.MFDatatypeInteger, lastSeenID);
conditions.Add(-1, idFilter);

var results = vault.ObjectSearchOperations
.SearchForObjectsByConditionsEx(
conditions,
MFSearchFlags.MFSearchFlagReturnLatestVisibleVersion,
true, // sort
batchSize, // <-- maxResults
3600 // timeout
);

if (batch.Count == 0)
break; // we’re done

allResults.AddRange(batch);

WaiterService.UpdateStatus($"Fetched {allResults.Count} records so far...");

lastSeenID = batch.Max(o => o.ObjVer.ID) + 1;
}

return allResults;
}

Assume it could be something to do with sorting or something similar...

Ideas ?

Thanks in advance,

Pat

Parents Reply
  • Hmm, odd.  Probably start by taking a look at a couple of those 150 files that are missing and see whether you can see something odd about them - could it be that the user doing the export doesn't have rights to see those objects?

    Alternatively it could be some odd bug in your code perhaps relating to the pagination, but I'd maybe start by seeing if there's anything obviously different about those items.

Children