Author Topic: backfill limit?  (Read 144776 times)

Offline phirestalker

  • Junior Indexer
  • **
  • Posts: 49
  • Helpful: +0/-0
Re: backfill limit?
« Reply #15 on: 2016-05-25, 02:30:41 pm »
ok I set it to group, it was on safe. I will have to test later though as my in process is filling up, I must have broke something trying to get rid of the 2000+ in process misc releases.

Offline kilgore

  • Junior Indexer
  • **
  • Posts: 22
  • Helpful: +0/-0
Re: backfill limit?
« Reply #16 on: 2016-05-25, 07:57:03 pm »
I'm at 50 days now.. so I am not sure if it's just faster using group because it collects more articles at a time or whatever glitch caused it to freeze at 30 days it bypasses.


Offline phirestalker

  • Junior Indexer
  • **
  • Posts: 49
  • Helpful: +0/-0
Re: backfill limit?
« Reply #17 on: 2016-05-25, 08:10:26 pm »
changing it to group didn't fix it for me. It is at 34 days now and only progressing backward with each passing day. There must be something else wrong on my end

Offline kilgore

  • Junior Indexer
  • **
  • Posts: 22
  • Helpful: +0/-0
Re: backfill limit?
« Reply #18 on: 2016-05-26, 06:32:22 am »
Just seeing if your site settings are the same on these:

Release Retention:   
180
!!THIS IS NOT HEADER RETENTION!! The number of days releases will be retained for use throughout site. Set to 0 to disable.
Misc->Other Retention Hours:   
18
The number of hours releases categorized as Misc->Other will be retained. Set to 0 to disable.
Misc->Hashed Retention Hours:   
18
The number of hours releases categorized as Misc->Hashed will be retained. Set to 0 to disable.
Release Completion:   
98
The minimum completion % to keep a release. Set to 0 to disable.

Offline phirestalker

  • Junior Indexer
  • **
  • Posts: 49
  • Helpful: +0/-0
Re: backfill limit?
« Reply #19 on: 2016-05-26, 09:11:18 am »
2700
72
72
95

Offline phirestalker

  • Junior Indexer
  • **
  • Posts: 49
  • Helpful: +0/-0
Re: backfill limit?
« Reply #20 on: 2016-05-26, 12:42:15 pm »
still not working. I found out what the in process ones are still there for. They cannot be categorized, and it is unrelated to it not backfilling a.b.warez

I'm not sure what else to try, would a screenshot or something of my settings in the web interface help?

Offline kilgore

  • Junior Indexer
  • **
  • Posts: 22
  • Helpful: +0/-0
Re: backfill limit?
« Reply #21 on: 2016-05-26, 01:36:01 pm »
Weird.. I have seen a database get wonky before and even though it's telling you the right thing, it's not..

I'd just do a fresh install.. New Database..  I set mine up as innodb.. not sure what yours is, but it helped prevent table locks..  which did slow me down a lot during backfills... I'd backfill one table at a time.. Using Group instead of Safe mode.. see how it goes.. 

I don't know if you have browsed that newsgroup with a reader to see how far back your service has that newsgroup.  It may be missing on their end??  I have seen newgroups do that before.






Offline phirestalker

  • Junior Indexer
  • **
  • Posts: 49
  • Helpful: +0/-0
Re: backfill limit?
« Reply #22 on: 2016-05-26, 02:10:42 pm »
This was hell, there is no way I can start over. But isn't there any way to get debug info out of the backfill script?

well I ran backfill.php alt.binaries.warez and it was chugging along just fine (got to 40 days before I killed it). I wonder what is keeping it from backfilling with tmux?
« Last Edit: 2016-05-26, 03:43:34 pm by phirestalker »

Offline kilgore

  • Junior Indexer
  • **
  • Posts: 22
  • Helpful: +0/-0
Re: backfill limit?
« Reply #23 on: 2016-05-26, 10:26:20 pm »
All you have to do is run the resetdb.php script...  There is also a resetposprocessing.php script..  Never used it, but seems self explanatory.

I'm not saying to delete the nzedb program and database and start over..

Just clear out the Database and add one group and see how it goes..

I have noticed that I am still backfilling 24 hours after I stopped..  So it takes time...

How fast is your processor and how much ram do you have?   

Run HTOP and see how much your CPU is getting used..  if all the cores are at their limits, and your RAM is all used up, it's going to really slow it down a lot.

have you run mysqltuner to see if your database needs tweaking?




Offline phirestalker

  • Junior Indexer
  • **
  • Posts: 49
  • Helpful: +0/-0
Re: backfill limit?
« Reply #24 on: 2016-05-26, 10:38:14 pm »
yes I've run mysqltuner and taken it's recomendations.

I have a Pentium(R) Dual-Core  CPU      E5500  @ 2.80GHz and 4GB of RAM
uptime:  22:36:53 up  7:12,  6 users,  load average: 2.08, 2.25, 2.35
free -h
              total        used        free      shared  buff/cache   available
Mem:           3.8G        2.4G        188M         24M        1.2G        1.3G
Swap:          4.0G        284M        3.7G

also I've noticed that it will "ignore" some groups for hours and low traffic groups for days

Offline Darius

  • Enforcer
  • *****
  • Posts: 121
  • Helpful: +8/-1
Re: backfill limit?
« Reply #25 on: 2016-05-26, 11:38:56 pm »
Your free memory is low. 4 GB RAM is not enough for nZEDb.

Offline kilgore

  • Junior Indexer
  • **
  • Posts: 22
  • Helpful: +0/-0
Re: backfill limit?
« Reply #26 on: 2016-05-27, 08:50:54 am »
Darius is right.. That load average is very high.  boost your ram to 8gig or 16gig.. I run 8 gig

Here's my load avg:
10:42:46 up 19 days, 14:53,  1 user,  load average: 0.82, 0.64, 0.55
Ram used: 4956  Ram Avail: 2984 Swap Used: 553MB

If I had a bigger database I would go to 16gig..  The idea is to keep the database doing everything in RAM and not swap space.. 
The load average being high like that.. I was getting a lot of that from Table Locks.. are you MySaim or Innodb...  That's why I switched over to Innodb and it really sped up the database.. There is a FAQ about all of this that explains how to do the conversion.

What is running at 200% when you use TOP?  If it's MYSQL.. there is your answer..

On low traffic groups, if there are no new articles.. it will look like they are being ignored.  A lot of ebook newsgroups get posts every day or two..



Offline phirestalker

  • Junior Indexer
  • **
  • Posts: 49
  • Helpful: +0/-0
Re: backfill limit?
« Reply #27 on: 2016-05-27, 09:59:19 am »
I am using compressed innodb since my 5G data won't fit in RAM

top says mysqld is using between 75 and 95% CPU.
I had a problem with account-daemon. It was hogging CPU because I stay logged in through ssh all the time. I'm going to set up logrotate to fix it.

it's more like:
 10:39:27 up 19:14,  8 users,  load average: 1.97, 1.79, 2.08
when php and mysql are busy
and an hour later
 12:29:07 up 21:04,  7 users,  load average: 0.83, 1.34, 1.49


I realize some groups post infrequently. I was going by the "Last Updated" column. Or do I misunderstand what that represents?
« Last Edit: 2016-05-27, 12:29:55 pm by phirestalker »

Offline phirestalker

  • Junior Indexer
  • **
  • Posts: 49
  • Helpful: +0/-0
Re: backfill limit?
« Reply #28 on: 2016-05-27, 02:31:15 pm »
ok so, back on topic.

I have had logging on for the past few days and in /var/www/nzedb/resources/logs/ I see logs for all the scripts postprocessing, fixReleaseNames, decryptHashes, etc. What I don't see is backfill. So it seems that tmux is just refusing to run backfill script at all. What could cause that. I have turned on every thing that I have found that has to do with backfilling.

If you tell me where nzedb stores the actual config files I will post them.

Offline kilgore

  • Junior Indexer
  • **
  • Posts: 22
  • Helpful: +0/-0
Re: backfill limit?
« Reply #29 on: 2016-05-27, 02:54:13 pm »
Is the backfill pane active on your TMUX screen?

See attached?

If it is and you see it processing things there.. Then it is functioning.