working systemd service files and scripts

[gravitymaster]

I set up a new ubuntu 15.04 server and wanted to get nZEDb working under systemd.  I'm using screens with two sequential scripts and IRCScraper. 

Edit 1/3/2016: Updated using the latest version of sequential scripts from dev.

/lib/systemd/system/nzedb-main.service

Code: [Select]

[Unit]
Description=nZEDb Main
After=mysql.service
Requires=mysql.service
[Service]
Type=forking
User=root
ExecStart=/usr/bin/screen -d -m -S nzedb-main /var/www/nZEDb/misc/update/nix/screen/sequential/user_threaded.sh
ExecStop=/usr/bin/screen -S nzedb-main -X quit
WorkingDirectory=/var/www/nZEDb/
[Install]
WantedBy=multi-user.target

/lib/systemd/system/nzedb-irc.service

Code: [Select]

[Unit]
Description=nZEDb IRC Scrapper
After=mysql.service
Requires=mysql.service
[Service]
Type=forking
User=root
ExecStart=/usr/bin/screen -d -m -S nzedb-irc /usr/bin/php /var/www/nZEDb/misc/IRCScraper/scrape.php true
ExecStop=/usr/bin/screen -S nzedb-irc -X quit
WorkingDirectory=/var/www/nZEDb/
[Install]
WantedBy=multi-user.target

/lib/systemd/system/nzedb-post.service

Code: [Select]

[Unit]
Description=nZEDb Post
After=mysql.service
Requires=mysql.service
[Service]
Type=forking
User=root
ExecStart=/usr/bin/screen -d -m -S nzedb-post /var/www/nZEDb/misc/update/nix/screen/sequential/postprocess.sh
ExecStop=/usr/bin/screen -S nzedb-post -X quit
WorkingDirectory=/var/www/nZEDb/
[Install]
WantedBy=multi-user.target

The first script does everything but posted processing

Code: [Select]

#!/usr/bin/env bash
export NZEDB_ROOT="/var/www/nZEDb"
export niceness=15
export VARIOUS_PATH="${NZEDB_ROOT}/misc/testing/Various"
export START_PATH="${NZEDB_ROOT}"
export NZEDB_PATH="${NZEDB_ROOT}/misc/update"
export TEST_PATH="${NZEDB_ROOT}/misc/testing/Release"
export DEV_PATH="${NZEDB_ROOT}/misc/testing/Developers"
export DB_PATH="${NZEDB_ROOT}/misc/testing/DB"
export MULTI_PATH="${NZEDB_ROOT}/misc/update/nix/multiprocessing"
export NZEDB_SLEEP_TIME="10" # in seconds
#
command -v php5 >/dev/null 2>&1 && export PHP=`command -v php5` || { export PHP=`command -v php`; }
#
export PHP="nice -n$niceness $PHP"
if [[ $1 != "true" ]]
then
$PHP ${NZEDB_PATH}/nix/tmux/bin/resetdelaytime.php
fi
#
loop=1
while [ $loop -ge 1 ]
do
date1=`date +%s`
clear
echo
echo
if [[ $# -eq 1 && $1 == "true" ]]
then
loop=0
fi
#
## Binaries options
echo "--- Header Fetch ---"
        $PHP ${MULTI_PATH}/binaries.php 0                     
## Backfill options
echo "--- Backfill ---"
        $PHP ${MULTI_PATH}/backfill.php                       
## NZB Import
# echo "--- Import ---"       
# $PHP ${MULTI_PATH}/import.php                           
## Release Processing
echo "--- Create Releases ---"
$PHP ${MULTI_PATH}/releases.php
echo "--- Decrypt Hashes ---"
$PHP ${NZEDB_PATH}/decrypt_hashes.php 10240
echo "--- PreDB ---"
$PHP ${NZEDB_PATH}/nix/tmux/bin/postprocess_pre.php
echo "--- PREs ---"
$PHP ${NZEDB_PATH}/match_prefiles.php 10240 show
## Fix Release names options
echo "--- FixRelNames ---"
$PHP ${MULTI_PATH}/fixrelnames.php nfo
$PHP ${MULTI_PATH}/fixrelnames.php filename
$PHP ${MULTI_PATH}/fixrelnames.php predbft
$PHP ${MULTI_PATH}/fixrelnames.php miscsorter
$PHP ${MULTI_PATH}/fixrelnames.php md5
$PHP ${MULTI_PATH}/fixrelnames.php par2
#
echo "--- Addtional ---"
$PHP ${VARIOUS_PATH}/renametopre.php 4
$PHP ${NZEDB_PATH}/requestid.php all true
$PHP ${DB_PATH}/populate_nzb_guid.php true
$PHP ${TEST_PATH}/delete_disabled_category_releases.php true
#
echo "--- removeCrapReleases --"
        $PHP ${TEST_PATH}/removeCrapReleases.php true full wmv_all
        $PHP ${TEST_PATH}/removeCrapReleases.php true full sample
        $PHP ${TEST_PATH}/removeCrapReleases.php true full size
        $PHP ${TEST_PATH}/removeCrapReleases.php true full scr
        $PHP ${TEST_PATH}/removeCrapReleases.php true full passwordurl
        $PHP ${TEST_PATH}/removeCrapReleases.php true full passworded
        $PHP ${TEST_PATH}/removeCrapReleases.php true full installbin
        $PHP ${TEST_PATH}/removeCrapReleases.php true full executable
        $PHP ${TEST_PATH}/removeCrapReleases.php true full short
#
date2=`date +%s`
diff=$(($date2-$date1))
echo "=== Total Running Time: $(($diff / 60)) minutes and $(($diff % 60)) seconds ==="
$PHP ${NZEDB_PATH}/nix/tmux/bin/showsleep.php $NZEDB_SLEEP_TIME
done

The Second script does the post processing:

Code: [Select]

#!/usr/bin/env bash
#
export NZEDB_ROOT="/var/www/nZEDb"
export niceness=10
export NZEDB_PATH="${NZEDB_ROOT}/misc/update"
export MULTI_PATH="${NZEDB_ROOT}/misc/update/nix/multiprocessing"
export NZEDB_SLEEP_TIME="120" # in seconds
command -v php5 >/dev/null 2>&1 && export PHP=`command -v php5` || { export PHP=`command -v php`; }
export PHP="nice -n$niceness $PHP"
#
if [[ $1 != "true" ]]
then
$PHP ${NZEDB_PATH}/nix/tmux/bin/resetdelaytime.php
fi
#
loop=1
while [ $loop -ge 1 ]
do
date1=`date +%s`
clear
echo
echo
if [[ $# -eq 1 && $1 == "true" ]]
then
loop=0
fi
#
## Post-Processing options
echo "--- Addtional ---"
$PHP ${MULTI_PATH}/postprocess.php add
echo "--- NFO ---"
$PHP ${MULTI_PATH}/postprocess.php nfo
        echo "--- Movie ---"
$PHP ${MULTI_PATH}/postprocess.php mov
        echo "--- TV ---"
$PHP ${MULTI_PATH}/postprocess.php tv
echo "--- Amazon ---"
$PHP ${MULTI_PATH}/postprocess.php ama
#
date2=`date +%s`
diff=$(($date2-$date1))
echo "=== Total Running Time: $(($diff / 60)) minutes and $(($diff % 60)) seconds ==="
$PHP ${NZEDB_PATH}/nix/tmux/bin/showsleep.php $NZEDB_SLEEP_TIME
done

To start the scripts:

Code: [Select]

systemctl start nzedb-main.service
systemctl start nzedb-irc.service
systemctl start nzedb-post.service

To attach each screen, CTRL-A D to detach from screen:

Code: [Select]

screen -R nzedb-main
screen -R nzedb-post
screen -R nzedb-irc

To enable the scripts on boot

Code: [Select]

systemctl enable nzedb-irc.service
systemctl enable nzedb-post.service
systemctl enable nzedb-main.service

Also created a scipt under cron.daily for basic maintenance set as needed:

Code: [Select]

systemctl stop nzedb-post
systemctl stop nzedb-main
systemctl stop nzedb-irc
php /var/www/nZEDb/cli/data/predb_import_daily_batch.php progress local false #Need to stop services before running
systemctl start nzedb-irc
systemctl start nzedb-post
systemctl start nzedb-main
php /var/www/nZEDb/misc/update/update_theaters.php


[Wally73]

put the code stuff between code tags then its a lot more easily to read

[josh4trunks]

do either of you mind helping me modernize my user_threaded.sh setup, or at least giving me some pointers? I noticed you were calling scripts I wasn't. My setup is basically still like this.
http://forums.nzedb.com/index.php?topic=1379.msg10646#msg10646

BTW, I am using tmux so I have everything in a single script. or is it better to break it out the way you did?

[josh4trunks]

Are you guys actually using "renametopre.php all"? In the past I have been always using "renametopre.php 4"
When I run "all" it tries to process 7 million records and usually complains about not allocating enough RAM to PHP (I tried 8GB and it still crashed). If I get through it once will it not process all 7 million records again?
Thanks,

[kevin123]

I'd recommend adding After/Requires mysqld.service to those systemd configs.

[gravitymaster]

do either of you mind helping me modernize my user_threaded.sh setup, or at least giving me some pointers? I noticed you were calling scripts I wasn't. My setup is basically still like this.
http://forums.nzedb.com/index.php?topic=1379.msg10646#msg10646

BTW, I am using tmux so I have everything in a single script. or is it better to break it out the way you did?

There are ways to setup TMUX via a systemd but the way the devs have it setup could not figure out an easy way to break it out. Plus they are making updates which could break any changes I needed to make   This was a very easy way to have a startup on system reboot and run in the order and options I wanted.

As far as the scripts being called just a matter of what you want the system to do, run each one manually look at the options and add as needed.

[gravitymaster]

Are you guys actually using "renametopre.php all"? In the past I have been always using "renametopre.php 4"
When I run "all" it tries to process 7 million records and usually complains about not allocating enough RAM to PHP (I tried 8GB and it still crashed). If I get through it once will it not process all 7 million records again?
Thanks,

Thanks for the catch, this should have been full or could user preid

[gravitymaster]

I'd recommend adding After/Requires mysqld.service to those systemd configs.

Go call updated the scripts

[egandt]

Here is my start/stop script for systemd using tmux:

Code: [Select]

[Unit]
Description=nZEDb Main
After=mysql.service
Requires=mysql.service network.target
[Service]
Type=forking
User=nzedb
group=www-data
# ensure that when nZEDb looks for tmux it finds /usr/local/bin version first
Environment='PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin'
ExecStart=/usr/bin/php /websites/nZEDb16/misc/update/nix/tmux/start.php
# default name for a tmux session is 0 then 1 ...
ExecStop=/usr/local/bin/tmux kill-session -t nZEDb16
WorkingDirectory=/websites/nZEDb16
[Install]
WantedBy=multi-user.target

First I set the environment to /usr/local/bin first so I pick up custom binaries as opposed to those installed by Ubuntu (such as tmux 1.9a).  Second I want mysql and networking running before I start nZEDb.  Basically it is similar to those for screen, but the kill commaond is customized for tmux.  Note "nZEDb16" is the name of the session from the tmux properties, if need to change this to the name you set there.

Note: Tested and functional on Ubuntu 16.04.

ERIC