backfill limit?

[phirestalker]

I activated a new group and I set the backfill days to 500 and it backfilled 30 days last night so I let it go and this morning it isn't past 30 days. What else do I have to do to make it backfill 500 days?

I have the tmux settings in the web interface set for safe, oldest, days per group.

[phirestalker]

ok I don't see any errors in tmux output, but during the backfill it says

group a.b.warez's oldest article is 129,843, newest is 2,616,570,025.
Our target article is 2,553,947,973. Our oldest article is article 2,553,948,973.

however it is still only 30 days backfilled and set to 500. What could be wrong? Do I have to clear out something that is stuck or a cache of some sort that has the old backfill setting for that group?

[donchaos]

Try using "Safe Backfill day", i think this is more reliable than the "Days per Group" setting. Remember, for "Safe Backfill day" you have to set the number of days in the site-settings.

[kilgore]

What is your release retention set at in the Site Settings?

Under your groups.. like a.b.teevee did you set backfill days here as well?

I am not 100% sure but if the Site Settings is at 30 days and you backfill for 500.. it may only keep the last 30?  I always set the site at a much higher retention than my backfill..

The programmers may have a better answer to that though.  It's always something I have wondered if it is overridden by group settings.

[phirestalker]

I have retention set at 2700.

I didn't want to use safe backfill date because I only wanted to go back for this one group. The reason I wanted to go back is gone now, but now I just want to make sure it works for when I do need it.

Oh and my other groups have older posts than 30 days, so no problem there. Now this groups has 31.2 but that is only time passed since I reset the group to see if that would fix anything.

[phirestalker]

resetting the group didn't help. What else should I check?

[kilgore]

What group is it?  Are there a lot of Hashed titles? 

Is it possible your Newgroup service no longer has the parts?

If it's hashed, you may be locked up processing them..

If you let me know what group it is, I'll run it with a backfill and see what happens for me.

[phirestalker]

A.b.warez

[kilgore]

I'm running a 90 day backfill on it now.. I won't be back for about 8 hours to give you a response on it though.

[phirestalker]

ok just checked on the stats section it says I only have 871 hashed releases altogether.

Also here are some stats. Tell me if they look ok?

Monitor Running v0.6.0 [430]:  10 hrs 27 mins 58 secs           
USP Connections:   0 active (0 total) - secure.usenetserver.com:563
Newest Release:    Addliss_-_In_Between_(Frisky_Radio)-SBD-12-25-20
Release Added:     3 mins 2 secs  ago                             
Predb Updated:     5 secs  ago                                     
Collection Age[2]: 10 hrs 28 mins 3 secs  ago                     
Parts in Repair:   0                                               
Postprocess:       stale for 10 secs                               

Collections                       Binaries                     Parts
================ ========================= =========================
2,984                               14,912                 2,647,775

PPA Lists                        Unmatched                   Matched
================ ========================= =========================
NFO's                               0(-10)               30,731(42%)
predb                     6,960,838(1,351)               33,674(46%)
requestID                             0(0)                 5,516(7%)

Category                        In Process               In Database
================ ========================= =========================
Audio                               0(-13)                9,890(13%)
Books                                 0(0)                 1,081(1%)
Console                               0(0)                 4,713(6%)
Misc                         4,884(-2,405)                 3,575(4%)
Movie                                 0(0)                 4,798(6%)
PC                                   0(-2)               11,376(15%)
TV                                   0(-1)               35,116(48%)
XXX                                   0(0)                 1,844(2%)
================ ========================= =========================
Total                        4,884(-2,431)            72,393(-1,971)

Groups                              Active                  Backfill
================ ========================= =========================
Activated                          21(203)                    0(203)

[phirestalker]

alright I'm not sure if it's related or not, but I have a "bad" release that is clogging up the system and I don't know how to find it. When I run post_process.php manually in an attempt to clear the large number of in process releases it gives me the error:

PHP Fatal error:  Allowed memory size of 1073741824 bytes exhausted

so I went into php.ini under the cli directory and changed the memory limit to -1 (unlimited). Now the system is just eating all the 4G of RAM and 4G of swap.
and now I get [476124][292.6MB](cB)z(cB)z(cB)z(cB)Killed.
I deleted the release, but how to be sure it isn't re-downloaded during backfill?

Is there some feature I forgot to turn on that can check for these memory eating releases?

some notes. This release is zip and not rar, and I am using rarinfo instead of zip and rar, as it said it was faster. This problem still happened not using rarinfo
Oh and stupid question the unrar directory, is it also the temp dir for zip files?

Thanks for reading this long mess and for the help

[kilgore]

An update on my end..  I only backfill 1000 articles at a time.. i'm up to 22 days...    I'll see where it's at tomorrow. 

I unrar into a Ramdisk..  so not sure exactly where it's doing it without..

Do you have your regex stuff setup?  blacklisting, etc? 

I do see one thing that may be because it's turned off.. on the back fill at the bottom of your stats.. it should be 1(203) to show that you are back filling a.b.warez... if it was a 2(203) you would be back filling two groups. you may have turned it off though when you deleted the release.

I'll update tomorrow.

[phirestalker]

I turned off backfill to give it time to go through the in process releases. It is down to 2671. I ran misc/update/postprocess.php all true and it completed successfully leaving still over 2000 in process releases. How do I get them to be processed?

[kilgore]

I changed the tmux backfill settings to Group instead of Safe...  It's doing a lot of processing.. I purged the group last night... so I am back at 29 days so far..  I'll update in a few hours...

[kilgore]

Update..

I'm at 34 days backfill so far.. so setting it to Group backfill instead of Safe seems to have either gotten it over the problem or made it process faster...

Yesterday, when it was on Safe - it sat at 29.5 days forever...

What is your TMUX back fill settings at?