Options
  • State Verified Answer
  • Replies 2 replies
  • Subscribers 10 subscribers
  • Views 882 views
  • Users 0 members are here
Related

Paging through items in a vault

darren.oakey

I have a need to iterate over every item in one of our vaults, and I can't find a reliable way of doing it.

The first thing I thought of was just to read by id - for instance read id's 1-1000, then 1000-2000 and so on - so I tried /objects/0?q=id>=1000&id<=1099&limit=100.     And I got back 100 items - but they just had random ids

I read a lot more and found the "p" syntax - and looked for the id of the id property - but there didn't seem to be one - so I thought - I'll use created date time.  The created field in /structure/properties was 20 - so I tried this:

/objects/0?q=p20>=2000-01-01T00:00:00Z&limit = 100

now - I know _everything in the database_ is later than 2000 - but I got back nothing.

What am I doing wrong?   And what is the best way to achieve what I want?   Basically I want to loop through every single item (there may be about 1,000,000 of them) - in a repeatable way such that I can stop at any point and then start again from where I left off with no risk of missing anything.   And then run it later and pick up any new values.  

How can I do that?

Thanks,

Darren

Parents
  • +1

    Pagination of search results is currently painful, although there are ongoing discussions around how we remove some of this pain.  I hope to have some news on this in the coming months.  With the COM API you can definitely query by ID (we have the concept of a "segmented search"); with the REST API it is also possible but uses an undocumented parameter so is not something I would personally recommend for a production environment.

    It looks like your query for the datetime search was incorrect.  I think you should not have the "q=" at the start of the querystring; I think it's trying to search by that datetime string and that's why nothing was returned.

Reply
  • +1

    Pagination of search results is currently painful, although there are ongoing discussions around how we remove some of this pain.  I hope to have some news on this in the coming months.  With the COM API you can definitely query by ID (we have the concept of a "segmented search"); with the REST API it is also possible but uses an undocumented parameter so is not something I would personally recommend for a production environment.

    It looks like your query for the datetime search was incorrect.  I think you should not have the "q=" at the start of the querystring; I think it's trying to search by that datetime string and that's why nothing was returned.

Children
  • 0 in reply to Craig Hawker

    thanks about the q - and I've been able to get datetime working (well - date) - but id would be so much better - please can you share with me the undocumented parameter.   What I'm doing is referencing all the m-files documents in another system... right now it's all about the bulk history because ongoing references will be created as we create documents - so I only need it to work short term - and being able to go by id will be so much cleaner than date!  (thx)

M-Files
M-Files Support


© 2024 M-Files, All Rights Reserved.