New Features

[jonnyboy]

I have just added a script that backfills 1 group completely. It gets 20k headers per thread per loop until it has completed the group or reached your backfill days. The group is selected by the oldest post, so the furthest backfilled group gets completed first. This gets the easy groups out of the way first.

In my tests, I backfilled a.b.inner-sanctum completely in ~90 min, ~48 mil headers, using 28 threads. I was not running releases at the same time and it took ~5 hours to create all of the releases from the group. Pay attention, this can overwhelm your db, but is better that backfilling 20k per thread on different groups.

You can run the script from:

Code: [Select]

python -OO misc/update_scripts/threaded_scripts/backfill_safe_threaded.py
or select Safe Backfill in tmux.

[jonnyboy]

This has changed slightly since it was first posted and I wanted to explain a little more in how it works. It seems there is some confusion.

This downloads 20k headers per group and that is hard-coded for a reason and should not be changed. Since each thread can finish out of sequence, the db is only updated to the new 'first' header that you have after every thread has completed.

After the threads have completed and the db is updated, then 1 thread attempts to download 1k headers to see if the group has been completed. If it is, then it is deactivated.

And, every backfill in tmux is killed every 80 min to run backfill quick, this is to touch every group so that collections that are not complete are not marked as complete.

[jonnyboy]

Another new feature, Grab NZBs. You can select to grab nzbs during binaries/backfill. For now, it only grabs single part nzbs. This does add some time per loop while it downloads and imports the nzbs. In most cases, you would have downloaded the part and discarded it during postprocessing.
This should eliminate some nzbs to download an nzb to download a release.

[jonnyboy]

Both of these features are complete, now. At least as far as I can see. Backfill safe uses all user settings from admin and tmux sections to control the number of headers to download per thread and per loop. So take care, if you have 500k set in tmux and 20 threads, that's 10mil per loop. It only updates the db after the last thread to know what header is the oldest you have.

Grab NZBs now checks if the number of articles available match what should be available. If it matches, then it downloads and inserts directly as a release, no more temp storage on disk. It then removes all collections/binaries/parts related to it.

They are still in dev until I get some feedback that they are not broken.

[jonnyboy]

If you are not using GrabNZBs, you may need to truncate the table. At first, it was populating the table for everyone, then the option to run or not was added.

Code: [Select]

truncate table nzbs;

[Saner]

If you use GrabNZB's are you supposed to disable 'Update Releases' ?

I am presuming you are not, as 'update releases' does other things too, but I get a lot of duplicates.

[jonnyboy]

If you use GrabNZB's are you supposed to disable 'Update Releases' ?

I am presuming you are not, as 'update releases' does other things too, but I get a lot of duplicates.

The duplicates were already there. It's not because of of GrabNZBs. It's the way they are posted makes the release and the nzb separate posts and therefor separate releases.
GrabNZBs only saves update_releases from having to create that release, if the nzb is grabbed first, then it deletes the associated collection and it is never created.

[Saner]

Ok, thank you for clearing that up.