Author Topic: Need advice on tuning  (Read 238836 times)

Offline xeddog

  • Prolific Indexer
  • ****
  • Posts: 240
  • Helpful: +9/-2
Need advice on tuning
« on: 2013-05-10, 08:48:58 AM »
Hi all.  I hope this board starts getting more attention.  

I have an old machine that should be callled quartenary since it is the fourth machine down my line after two machines with I-7 processors, and one with an I-5.  It's an old 1.82ghz Core2 (I don't remember for certain, but it just might be an I-5 processor too) with only 1GB ram, an old nvidia graphics card, and would you believe a 17" crt monitor that probably dates back to the 90's?  About the only thing that is any where near recent (with the possible exception of the I-5) is the 500GB sata disk drive.  Onto this machine I have installed Ubuntu-GNOME 13.04, SABnzbd+, SickBeard, CouchPotato, and of course nZEDb.  The problem I am having is that performance sucks badly.

When I boot the machine, the first thing I do is launch the system monitor.  The second thing I do is open a terminal and launch the tmux scripts for nZEDb.  As soon as I see tmux coming up I have to quckly switch back to the system monitor.  What i see there is the two CPU's both top 97% busy almost immediately.  Then the memory starts stepping up from a static 35-ish % to about 45%, then 60%.  The swap is stagnate at almost zero.  Network history shows that I start receiving at about 4Mbps.  After that, the system becomes completely unresponsive and the system monitor stops updating.  The only thing I know is that I can hear that something is really beating the crap outta the disk drive, so I assume it is swapping it's butt off.  If this was just a temporary thing I could deal with it, but it seems to be permanent.  I have left it in this state for several hours and it just doesn't come around.  The only thing I can do to get out of this situation is hit the power button.  Even Ctl-alt-F1 doesn't respond.

The only settings I can remember are the two php.ini files where I can set the memory_limit, but changing those values doesn't seem to help.  I started with 1024M, and then started dropping them to 768M, 512M, 256M, 128M, 64M, and presently to 24M.  I will have to look at those files again to look for additional items to change, but until I get there . . .

So.  Any ideas what I can do about this?  

Thanks,

X
« Last Edit: 1969-12-31, 04:00:00 PM by Guest »

Offline nzeLover

  • Decent Indexer
  • ***
  • Posts: 58
  • Helpful: +6/-0
Re: Need advice on tuning
« Reply #1 on: 2013-05-10, 09:44:06 AM »
Well - You ONLY  have 1 GB ram and you have alot of services installed incl. a desktop enviroment. I have to say - forget it on that pc. You can use it for nze alone as a clean server without any desktop and other services installed - and that is perhaps not enough.

Sorry - Sure not the answer you wish to hear but that the only answer I can give.
« Last Edit: 1969-12-31, 04:00:00 PM by Guest »
// nzeLover
---
Using nZEDb with Ubuntu 14.04 LTS on 2 x dedicated servers.

Offline xeddog

  • Prolific Indexer
  • ****
  • Posts: 240
  • Helpful: +9/-2
Re: Need advice on tuning
« Reply #2 on: 2013-05-10, 09:55:34 AM »
Yeah, not the answer I was hoping for, but the one I expected.  I still think that there must be something that can be done because I also ran this on a virtual machine on my "big" computer using Virtualbox.  Allocating only one processor and 1GB ram it was not as bad as this machine, although still not acceptable.  Changing the allocation to 2 cpu's and keeping the 1GB ram was even worse, but adding another GB of ram made this config (2 cpu + 2 GB ram) at least workable.  I guess I may need to go memory hunting.  

Thanks,

X
« Last Edit: 1969-12-31, 04:00:00 PM by Guest »

Offline jonnyboy

  • Epic Indexer
  • *****
  • Posts: 1046
  • Helpful: +93/-1
  • Lazzy Trucker
    • nZEDb
Re: Need advice on tuning
« Reply #3 on: 2013-05-10, 12:59:12 PM »
ust don't run the threaded scripts. Run everything in sequence. It is still much faster than the alternative. With the limited ram, just take it easy. Remember, it doesn't have to get done today.
« Last Edit: 1969-12-31, 04:00:00 PM by Guest »

ubjeckle

  • Guest
Re: Need advice on tuning
« Reply #4 on: 2013-05-11, 09:24:58 AM »
i can see where he is coming from , sorry if i am hijacking this thread, but since the git pull on friday, i have to say even using 1 thread for post processing spikes my cpu usage + memory usage high , and the server i am running this off of is a quad core i3 with 8 gig of ram , when i tested nzedb this morning as soon as i had post processing enabled while backfill + update binaries was on my memory usage went from 436mb being used straigh to 7.49 gb being used. Which it hasnt been like that until now.


** Update **

I Will retract my statement just did another git pull and put post processing on and at the moment with 1 thread it is using minimal memory.
« Last Edit: 1969-12-31, 04:00:00 PM by Guest »

Offline xeddog

  • Prolific Indexer
  • ****
  • Posts: 240
  • Helpful: +9/-2
Re: Need advice on tuning
« Reply #5 on: 2013-05-11, 12:31:48 PM »
OK, I started running sequential and the machine is at least responsive.  I also sprang the $40 to get a 4GB RAM upgrade so hopefully next week I can actually start using it.   :-)

Thanks for the info,

X
« Last Edit: 1969-12-31, 04:00:00 PM by Guest »

Offline xeddog

  • Prolific Indexer
  • ****
  • Posts: 240
  • Helpful: +9/-2
Re: Need advice on tuning
« Reply #6 on: 2013-05-11, 01:47:12 PM »
ok, I take that back.  It just took longer to manifest itself.  Before running sequential it only took about 15-20 seconds to become unresponsive, but after it took about 15-20 minutes.  But it is still unresponsive.  i'll just have to wait until I get the memory upgrade.

X
« Last Edit: 1969-12-31, 04:00:00 PM by Guest »

Offline jonnyboy

  • Epic Indexer
  • *****
  • Posts: 1046
  • Helpful: +93/-1
  • Lazzy Trucker
    • nZEDb
Re: Need advice on tuning
« Reply #7 on: 2013-05-12, 04:58:04 PM »
Well, Kevin123 runs his script and it never exceeds 500MB and he indexes every group. So, it is possible.
« Last Edit: 1969-12-31, 04:00:00 PM by Guest »

Offline xeddog

  • Prolific Indexer
  • ****
  • Posts: 240
  • Helpful: +9/-2
Re: Need advice on tuning
« Reply #8 on: 2013-05-13, 06:04:41 PM »
There is definitely something going on here.  I have increased ram to 4GB and run the tmux scripts sequential.  It STILL goes ape (meaning the system goes unresponsive).  Maybe some incompatibility with Ubuntu 13.04, or maybe an incompatibility with gnome3, or both????  Anyway, I think I might try installing plane ol' Ubuntu 13.04 and see what happens.  

Just in case anyone didn't get my software setup - I installed the official Gnome Ubuntu-GNOME distro from the gnome3 website.  This installs Ubuntu 13.04 with the gnome3 DE and NO Unity.  Other than that, I followed the installation instructions for nZEDb in the document that outlines how to install nZEDb on Ubuntu 12.10, and It all seemed to work without any errors.  It just won't run for long without the machine going unresponsive.

X
« Last Edit: 1969-12-31, 04:00:00 PM by Guest »

Offline zombiehoffa

  • Junior Indexer
  • **
  • Posts: 17
  • Helpful: +5/-0
Re: Need advice on tuning
« Reply #9 on: 2013-05-13, 07:28:07 PM »
With such an underpowered machine, have you thought about not using gnome? I'm using LXDE (via lubuntu) and am very happy with its responsiveness on old machines with low ram. It's not annoying for people used to kde/gnome like fluxbox is, but it still uses way less resources.
« Last Edit: 1969-12-31, 04:00:00 PM by Guest »

Offline xeddog

  • Prolific Indexer
  • ****
  • Posts: 240
  • Helpful: +9/-2
Re: Need advice on tuning
« Reply #10 on: 2013-05-14, 09:27:31 AM »
Yeah, I have thought about it.  Not Lubuntu maybe, but Xubuntu.  But it just seems that even though this machine isn't the most powerful by any stretch, it should be up to running this.  I am not trying to be argumentative with you, but if the machine was that underpowered, I would think that the machine would be hosed from the git-go.  But it will run and seem to be perfectly normal for a few minutes and then ka-bloowey.

I started it up about 10-15 minutes ago and it still seems to be ok.  I have been playing around with the settings so MAYbe I just hit the right combo, but time will tell.   Looking at the system monitor I see the graph for the two cpu's going all over the place, but there does seem to be a cycle to the movements that corresponds roughly to the memory/swap usage, and the network usage.  As for the memory/swap graphs, the swap is zero activity.  The memory usage is regular in that it will start at about 20% utilization and be there for about 15 seconds or so, then over the next 8-10 seconds or so it will stairstep up to about 80%, remain there for about 6-7 seconds, and then drop back to 20%.  Then the cylce will start over again.  The network utilization graphs for the most part look more like a sine wave running from near 0 to around 4Mbps (BITS per second, not bytes).  At times it will jump to about 20Mbps for 10-15 seconds two or three times over about 30 seconds, and then go back to sinewave.  

My primary machine is a I-7 processor with 6GB ram ,1TB esata disk drive and runing Ubuntu 12.04.  I can't stand Unity so I am using gnome-session-fallback for the desktop.  On this machine I have installed Virtualbox and created a Ubuntu 13.04 guest (meaning Unity desktop) and I have installed nZEDb there too.  To that machine I have allocated as little as one processor and 1GB ram, and although it was awfully slow, it ran.  I could run nZEDb, SABnzbd+, SickBeard, and CouchPotato, and still use the machine.  It was painfully slow, like it would take 10-15 seconds just to minimize a window, but I was always able to at least manipulate the vm.  I increased the cpu count to 2, and the ram to 2GB and while still somewhat slow it is acceptable for what it is.  I would think that a dedicated machine, even though it is an older Core2 but has more RAM should at least be able to do as well.  The only software differences between the two is really Unity vs Gnome.  

So you may be asking why I want to run it on this old machine in the first place since I have a running copy in a vm.  Well, . . . why not?  It is a machine that has just been sitting unused for a while, and it would allow me to "unburden" my primary machine so I could burden it with something else.  

Anyway,  Any suggestions would be appreciated as long as you aren't telling me to go out and buy a big new machine.  :-)  


Thanks,

X


EDIT:  Well, it ran for around an hour this time before going kablooey.  Unfortunately, the monitor had gone into standby mode and I could not get it to come alive again even after 30 minutes of trying so I don't know what the system monitor would have told me.  When I pressed the ctl key, the monitor did come back online (meaning that the power led changed from yellow back to green), but remained black except for the cursor.


EDIT2:  Well the screen finally came back on after another half hour.  The system monitor showed that the network graphs had both gone to zero (send and receive), the cpu utilizations dropped to around 10% or less, but the memory usage went to 100% and the swap rose to around 75% and looks like it was slowly climbing.  Since the system monitor stopped updating at that point, and the system is just unresponsive, I can't switch over to see what the processes were doing,
« Last Edit: 1969-12-31, 04:00:00 PM by Guest »

Offline xeddog

  • Prolific Indexer
  • ****
  • Posts: 240
  • Helpful: +9/-2
Re: Need advice on tuning
« Reply #11 on: 2013-05-15, 12:51:06 PM »
Well nZEDb ran for several hours last night.  I don't know what changed from the last several invocations, but things are still not what they should be.  I noticed that on the "utils" screen in the lower left quadrant that postprocessing.php was having some issues.  I kept seeing a recurring message that all memory allocation was exceeded and postprocessing.php was terminating.  I'm sorry I don't have the exact text of the message, but it is pretty hard to replicate.  

Anyway, in the two php.ini files I had set memory_limit at 128M (that is the number that postprocessing.php was complaining about), so I raised them to 256M and restarted the tmux scripts.  Same thing happened with postprocessing.php but now it was complaining that 256M wasn't eough.  So I raised the two memory_limits to 512M and again restarted the tmux scripts.  This time my system became unresponsive almost immediately after postprocessing.php starting.  &^%$#%^*&!!!  I also noticed that every so often, postprocessing.php will abort because it is apparently hung.  During the time that it is closing, and the time that it restarts (about one or two seconds), the system is responsive.  I was able to swiwtch to the system monitor during one of these cycles and saw that postprocessing.php was using 2.5GB of memory and 50% CPU.

So if anyone is still reading this and has any suggestions as to fixing this problem . . . . .

Thanks,

X
« Last Edit: 1969-12-31, 04:00:00 PM by Guest »

Offline jonnyboy

  • Epic Indexer
  • *****
  • Posts: 1046
  • Helpful: +93/-1
  • Lazzy Trucker
    • nZEDb
Re: Need advice on tuning
« Reply #12 on: 2013-05-15, 05:30:20 PM »
At a minimum each php script uses 178MB on my box. Maybe that can guide you.
Also, I have already suggested that you should not be running the tmux scripts and should probably just use the screen scripts.
« Last Edit: 1969-12-31, 04:00:00 PM by Guest »

Offline zombu2

  • Enforcer
  • *****
  • Posts: 13
  • Helpful: +2/-0
Re: Need advice on tuning
« Reply #13 on: 2013-05-16, 01:19:29 PM »
set scheduler to CFQ on boot that should give you some improovement on responsiveness
« Last Edit: 1969-12-31, 04:00:00 PM by Guest »

Offline xeddog

  • Prolific Indexer
  • ****
  • Posts: 240
  • Helpful: +9/-2
Re: Need advice on tuning
« Reply #14 on: 2013-05-17, 09:17:18 AM »
Well, I guess you can say that I "fixed" it.  I started getting a couple of other errors, two  in update_binaries and one in update_releases,  so I decided just to start over from scratch starting with re-installing Ubuntu-GNOME.  Just installing Ubuntu-GNOME turned out to be an ordeal itself.  To make a long story short, I don't think Ubuntu-GNOME liked having a sata disk as primary and a IDE as secondary.

When I did my initial installation of Ubuntu-GNOME I had disabled the IDE drive in the bios just to make sure I didn't accidentally write to it because it had some backup data on it that I wanted to preserve.  Even at that it took a couple of tries to get a clean install due to an assortment of errors, but once installed I thought it was good.  Then I installed all of the requisite software and nZEDb and saw no errors there either.  At some point when I deemed it safe after nZEDb appeared to be running ok, I re-enabled the IDE drive in the bios.

However, when RE-installing Ubuntu-GNOME, It would flat NOT install with the IDE drive enabled.  This is the very same configuration that I had Windows 7 on,   and I also installed the regular Ubuntu 13.04 (the one with Unity) without issue, but Ubuntu-GNOME flat ass refused to go on.  After removing the IDE drive all together, Ubuntu-GNOME installed just fine.  So did all of the requisite software again, and so did nZEDb. 

And without the IDE drive, nZEDb is now running perfectly.  I have it running threaded, I have set the two php.ini files with memory_limit set to 1024M, backfills running, more groups activated than I had before, SABnzbd+ running, SickBeard running, and Couch Potato running.  and the machine is handling all of it beautifully.   No hangs, no errors, and still have crisp response.

I'm happy again (for a little while at least),

X

P.S.  Just in case you happen to be wondering, I installed the IDE drive into an external USB enclosure and hooked it up to another computer.  It seems to work fine.