Need to speed up metadata extraction

A couple of things:

• I definitely agree with Radu Moca : convert this to retrieve data in batches.  For reading data batches in the hundreds should work.  It won't make it hundreds of times faster, but it'll make it faster.
• Your current code seems to populate that propertyDefsCache for each object, which means tens of millions (at least - possibly hundreds of millions) of additional vault queries to get this data.  My suggestion would instead be to retrieve all of the properties once, first, populate a static dictionary instead, then use that cache inside your GetMetadataAsJson method.

A couple of things:

• I definitely agree with Radu Moca : convert this to retrieve data in batches.  For reading data batches in the hundreds should work.  It won't make it hundreds of times faster, but it'll make it faster.
• Your current code seems to populate that propertyDefsCache for each object, which means tens of millions (at least - possibly hundreds of millions) of additional vault queries to get this data.  My suggestion would instead be to retrieve all of the properties once, first, populate a static dictionary instead, then use that cache inside your GetMetadataAsJson method.

As a newbie, appreciate all the comments and insight.  I also am downloading each file in the same manner.  Is there a batch download that would correspond to the metadata batch?  I download all the file names in batches as a part of my out loop wherein this code is contained and iterate through that list but perhaps I should just do everything in batches.  

The actual file download will need to be one-by-one, unfortunately. You may be able to build something which creates a queue of items to download, though, and download multiple in parallel...? 

How about this, Make a view that has your files. Remove the view limit, remove the timeout from preferably server or any client and right click and download everything in said view ? Shouldn't take longer than 16 days and you would have the files and the metadata

or you can go for segmentation for data like, make multiple views based on creation date of the files. And do the export operation

I did try downloading in parallel but both threads were actually slower than just downloading sequentially so I thought that perhaps this put some kind of overhead on the server or something.  Might revisit this though.  Is the server throttled or was this just likely some inefficiency I created in my code?

It's been a while since I tried to do this. You could be right. I'd be interested in your results, although it is possible that the other changes detailed above will make a significant impact on performance regardless (as that will remove potentially millions of requests ongoing).