nZEDb

nZEDb => Support => Topic started by: phirestalker on 2016-05-22, 09:36:09 am

Title: backfill limit?
Post by: phirestalker on 2016-05-22, 09:36:09 am
I activated a new group and I set the backfill days to 500 and it backfilled 30 days last night so I let it go and this morning it isn't past 30 days. What else do I have to do to make it backfill 500 days?

I have the tmux settings in the web interface set for safe, oldest, days per group.
Title: Re: backfill limit?
Post by: phirestalker on 2016-05-22, 06:59:34 pm
ok I don't see any errors in tmux output, but during the backfill it says

group a.b.warez's oldest article is 129,843, newest is 2,616,570,025.
Our target article is 2,553,947,973. Our oldest article is article 2,553,948,973.

however it is still only 30 days backfilled and set to 500. What could be wrong? Do I have to clear out something that is stuck or a cache of some sort that has the old backfill setting for that group?
Title: Re: backfill limit?
Post by: donchaos on 2016-05-23, 05:37:29 am
Try using "Safe Backfill day", i think this is more reliable than the "Days per Group" setting. Remember, for "Safe Backfill day" you have to set the number of days in the site-settings.
Title: Re: backfill limit?
Post by: kilgore on 2016-05-23, 05:45:53 am
What is your release retention set at in the Site Settings?

Under your groups.. like a.b.teevee did you set backfill days here as well?

I am not 100% sure but if the Site Settings is at 30 days and you backfill for 500.. it may only keep the last 30?  I always set the site at a much higher retention than my backfill..

The programmers may have a better answer to that though.  It's always something I have wondered if it is overridden by group settings.
Title: Re: backfill limit?
Post by: phirestalker on 2016-05-23, 08:47:00 am
I have retention set at 2700.

I didn't want to use safe backfill date because I only wanted to go back for this one group. The reason I wanted to go back is gone now, but now I just want to make sure it works for when I do need it.

Oh and my other groups have older posts than 30 days, so no problem there. Now this groups has 31.2 but that is only time passed since I reset the group to see if that would fix anything.
Title: Re: backfill limit?
Post by: phirestalker on 2016-05-23, 01:53:28 pm
resetting the group didn't help. What else should I check?
Title: Re: backfill limit?
Post by: kilgore on 2016-05-23, 08:32:20 pm
What group is it?  Are there a lot of Hashed titles? 

Is it possible your Newgroup service no longer has the parts?

If it's hashed, you may be locked up processing them..

If you let me know what group it is, I'll run it with a backfill and see what happens for me.



Title: Re: backfill limit?
Post by: phirestalker on 2016-05-23, 08:39:55 pm
A.b.warez
Title: Re: backfill limit?
Post by: kilgore on 2016-05-24, 05:35:25 am
I'm running a 90 day backfill on it now.. I won't be back for about 8 hours to give you a response on it though.

Title: Re: backfill limit?
Post by: phirestalker on 2016-05-24, 12:10:04 pm
ok just checked on the stats section it says I only have 871 hashed releases altogether.

Also here are some stats. Tell me if they look ok?

Monitor Running v0.6.0 [430]:  10 hrs 27 mins 58 secs           
USP Connections:   0 active (0 total) - secure.usenetserver.com:563
Newest Release:    Addliss_-_In_Between_(Frisky_Radio)-SBD-12-25-20
Release Added:     3 mins 2 secs  ago                             
Predb Updated:     5 secs  ago                                     
Collection Age[2]: 10 hrs 28 mins 3 secs  ago                     
Parts in Repair:   0                                               
Postprocess:       stale for 10 secs                               

Collections                       Binaries                     Parts
================ ========================= =========================
2,984                               14,912                 2,647,775

PPA Lists                        Unmatched                   Matched
================ ========================= =========================
NFO's                               0(-10)               30,731(42%)
predb                     6,960,838(1,351)               33,674(46%)
requestID                             0(0)                 5,516(7%)

Category                        In Process               In Database
================ ========================= =========================
Audio                               0(-13)                9,890(13%)
Books                                 0(0)                 1,081(1%)
Console                               0(0)                 4,713(6%)
Misc                         4,884(-2,405)                 3,575(4%)
Movie                                 0(0)                 4,798(6%)
PC                                   0(-2)               11,376(15%)
TV                                   0(-1)               35,116(48%)
XXX                                   0(0)                 1,844(2%)
================ ========================= =========================
Total                        4,884(-2,431)            72,393(-1,971)

Groups                              Active                  Backfill
================ ========================= =========================
Activated                          21(203)                    0(203)
Title: Re: backfill limit?
Post by: phirestalker on 2016-05-24, 05:54:55 pm
alright I'm not sure if it's related or not, but I have a "bad" release that is clogging up the system and I don't know how to find it. When I run post_process.php manually in an attempt to clear the large number of in process releases it gives me the error:

PHP Fatal error:  Allowed memory size of 1073741824 bytes exhausted

so I went into php.ini under the cli directory and changed the memory limit to -1 (unlimited). Now the system is just eating all the 4G of RAM and 4G of swap.
and now I get [476124][292.6MB](cB)z(cB)z(cB)z(cB)Killed.
I deleted the release, but how to be sure it isn't re-downloaded during backfill?

Is there some feature I forgot to turn on that can check for these memory eating releases?

some notes. This release is zip and not rar, and I am using rarinfo instead of zip and rar, as it said it was faster. This problem still happened not using rarinfo
Oh and stupid question the unrar directory, is it also the temp dir for zip files?

Thanks for reading this long mess and for the help
Title: Re: backfill limit?
Post by: kilgore on 2016-05-24, 08:05:45 pm
An update on my end..  I only backfill 1000 articles at a time.. i'm up to 22 days...    I'll see where it's at tomorrow. 

I unrar into a Ramdisk..  so not sure exactly where it's doing it without..

Do you have your regex stuff setup?  blacklisting, etc? 

I do see one thing that may be because it's turned off.. on the back fill at the bottom of your stats.. it should be 1(203) to show that you are back filling a.b.warez... if it was a 2(203) you would be back filling two groups. you may have turned it off though when you deleted the release.

I'll update tomorrow.

Title: Re: backfill limit?
Post by: phirestalker on 2016-05-24, 10:38:49 pm
I turned off backfill to give it time to go through the in process releases. It is down to 2671. I ran misc/update/postprocess.php all true and it completed successfully leaving still over 2000 in process releases. How do I get them to be processed?
Title: Re: backfill limit?
Post by: kilgore on 2016-05-25, 01:04:44 pm
I changed the tmux backfill settings to Group instead of Safe...  It's doing a lot of processing.. I purged the group last night... so I am back at 29 days so far..  I'll update in a few hours...

Title: Re: backfill limit?
Post by: kilgore on 2016-05-25, 02:16:19 pm
Update..

I'm at 34 days backfill so far.. so setting it to Group backfill instead of Safe seems to have either gotten it over the problem or made it process faster...

Yesterday, when it was on Safe - it sat at 29.5 days forever...

What is your TMUX back fill settings at?

Title: Re: backfill limit?
Post by: phirestalker on 2016-05-25, 02:30:41 pm
ok I set it to group, it was on safe. I will have to test later though as my in process is filling up, I must have broke something trying to get rid of the 2000+ in process misc releases.
Title: Re: backfill limit?
Post by: kilgore on 2016-05-25, 07:57:03 pm
I'm at 50 days now.. so I am not sure if it's just faster using group because it collects more articles at a time or whatever glitch caused it to freeze at 30 days it bypasses.

Title: Re: backfill limit?
Post by: phirestalker on 2016-05-25, 08:10:26 pm
changing it to group didn't fix it for me. It is at 34 days now and only progressing backward with each passing day. There must be something else wrong on my end
Title: Re: backfill limit?
Post by: kilgore on 2016-05-26, 06:32:22 am
Just seeing if your site settings are the same on these:

Release Retention:   
180
!!THIS IS NOT HEADER RETENTION!! The number of days releases will be retained for use throughout site. Set to 0 to disable.
Misc->Other Retention Hours:   
18
The number of hours releases categorized as Misc->Other will be retained. Set to 0 to disable.
Misc->Hashed Retention Hours:   
18
The number of hours releases categorized as Misc->Hashed will be retained. Set to 0 to disable.
Release Completion:   
98
The minimum completion % to keep a release. Set to 0 to disable.
Title: Re: backfill limit?
Post by: phirestalker on 2016-05-26, 09:11:18 am
2700
72
72
95
Title: Re: backfill limit?
Post by: phirestalker on 2016-05-26, 12:42:15 pm
still not working. I found out what the in process ones are still there for. They cannot be categorized, and it is unrelated to it not backfilling a.b.warez

I'm not sure what else to try, would a screenshot or something of my settings in the web interface help?
Title: Re: backfill limit?
Post by: kilgore on 2016-05-26, 01:36:01 pm
Weird.. I have seen a database get wonky before and even though it's telling you the right thing, it's not..

I'd just do a fresh install.. New Database..  I set mine up as innodb.. not sure what yours is, but it helped prevent table locks..  which did slow me down a lot during backfills... I'd backfill one table at a time.. Using Group instead of Safe mode.. see how it goes.. 

I don't know if you have browsed that newsgroup with a reader to see how far back your service has that newsgroup.  It may be missing on their end??  I have seen newgroups do that before.





Title: Re: backfill limit?
Post by: phirestalker on 2016-05-26, 02:10:42 pm
This was hell, there is no way I can start over. But isn't there any way to get debug info out of the backfill script?

well I ran backfill.php alt.binaries.warez and it was chugging along just fine (got to 40 days before I killed it). I wonder what is keeping it from backfilling with tmux?
Title: Re: backfill limit?
Post by: kilgore on 2016-05-26, 10:26:20 pm
All you have to do is run the resetdb.php script...  There is also a resetposprocessing.php script..  Never used it, but seems self explanatory.

I'm not saying to delete the nzedb program and database and start over..

Just clear out the Database and add one group and see how it goes..

I have noticed that I am still backfilling 24 hours after I stopped..  So it takes time...

How fast is your processor and how much ram do you have?   

Run HTOP and see how much your CPU is getting used..  if all the cores are at their limits, and your RAM is all used up, it's going to really slow it down a lot.

have you run mysqltuner to see if your database needs tweaking?



Title: Re: backfill limit?
Post by: phirestalker on 2016-05-26, 10:38:14 pm
yes I've run mysqltuner and taken it's recomendations.

I have a Pentium(R) Dual-Core  CPU      E5500  @ 2.80GHz and 4GB of RAM
uptime:  22:36:53 up  7:12,  6 users,  load average: 2.08, 2.25, 2.35
free -h
              total        used        free      shared  buff/cache   available
Mem:           3.8G        2.4G        188M         24M        1.2G        1.3G
Swap:          4.0G        284M        3.7G

also I've noticed that it will "ignore" some groups for hours and low traffic groups for days
Title: Re: backfill limit?
Post by: Darius on 2016-05-26, 11:38:56 pm
Your free memory is low. 4 GB RAM is not enough for nZEDb.
Title: Re: backfill limit?
Post by: kilgore on 2016-05-27, 08:50:54 am
Darius is right.. That load average is very high.  boost your ram to 8gig or 16gig.. I run 8 gig

Here's my load avg:
10:42:46 up 19 days, 14:53,  1 user,  load average: 0.82, 0.64, 0.55
Ram used: 4956  Ram Avail: 2984 Swap Used: 553MB

If I had a bigger database I would go to 16gig..  The idea is to keep the database doing everything in RAM and not swap space.. 
The load average being high like that.. I was getting a lot of that from Table Locks.. are you MySaim or Innodb...  That's why I switched over to Innodb and it really sped up the database.. There is a FAQ about all of this that explains how to do the conversion.

What is running at 200% when you use TOP?  If it's MYSQL.. there is your answer..

On low traffic groups, if there are no new articles.. it will look like they are being ignored.  A lot of ebook newsgroups get posts every day or two..


Title: Re: backfill limit?
Post by: phirestalker on 2016-05-27, 09:59:19 am
I am using compressed innodb since my 5G data won't fit in RAM

top says mysqld is using between 75 and 95% CPU.
I had a problem with account-daemon. It was hogging CPU because I stay logged in through ssh all the time. I'm going to set up logrotate to fix it.

it's more like:
 10:39:27 up 19:14,  8 users,  load average: 1.97, 1.79, 2.08
when php and mysql are busy
and an hour later
 12:29:07 up 21:04,  7 users,  load average: 0.83, 1.34, 1.49


I realize some groups post infrequently. I was going by the "Last Updated" column. Or do I misunderstand what that represents?
Title: Re: backfill limit?
Post by: phirestalker on 2016-05-27, 02:31:15 pm
ok so, back on topic.

I have had logging on for the past few days and in /var/www/nzedb/resources/logs/ I see logs for all the scripts postprocessing, fixReleaseNames, decryptHashes, etc. What I don't see is backfill. So it seems that tmux is just refusing to run backfill script at all. What could cause that. I have turned on every thing that I have found that has to do with backfilling.

If you tell me where nzedb stores the actual config files I will post them.
Title: Re: backfill limit?
Post by: kilgore on 2016-05-27, 02:54:13 pm
Is the backfill pane active on your TMUX screen?

See attached?

If it is and you see it processing things there.. Then it is functioning.

(http://i1323.photobucket.com/albums/u594/brian70809/Screenshot%20from%202016-05-27%2016-47-46_zpslw2tynjq.jpg) (http://s1323.photobucket.com/user/brian70809/media/Screenshot%20from%202016-05-27%2016-47-46_zpslw2tynjq.jpg.html)
Title: Re: backfill limit?
Post by: phirestalker on 2016-05-27, 02:57:13 pm
if I understand correctly mine won't look like that because I'm on sequential.
Title: Re: backfill limit?
Post by: kilgore on 2016-05-27, 03:01:56 pm
Yep.. Change update binaries to simple threaded..  Disable the sequential...

It only does 1 group at a time with 1 thread..  it's a lot easier to see whats going on...

You will have to Stop TMUX and restart it though..

Title: Re: backfill limit?
Post by: phirestalker on 2016-05-27, 03:07:22 pm
I'm on table per group and my only options for update binaries are
simple threaded update and complete threaded update.

As for run sequential, I can disable that, but how much more load will there be running everything at the same time?
Title: Re: backfill limit?
Post by: kilgore on 2016-05-27, 03:09:04 pm
Yes.. run Simple Threaded.. as far as load.  I didn't notice a difference, but my system is a 6 core with 8 gig of RAM.. so it can take more abuse.

Just run it for a little while and watch the TOP or HTOP to see how it goes..

Title: Re: backfill limit?
Post by: phirestalker on 2016-05-27, 03:28:36 pm
ok that FINALLY got me able to see the BLOODY error message. I needed to install cymysql

I can't understand why it should be so difficult to find and error message?????

Thanks for all the help guys.

Is there any way to mark this thread as solved?