Setting up Predb Guide

[blindpet]

This is an expanded guide to importing the large predb, importing the daily dumps since the large dump and setting up ZNC and IRC Scraper to automaticallly update the predb. It is a supplement to my expanded nZEDb Ubuntu Guide based heavily on your official guide.

Create the predb
Create the predb and update it
This is a large predb, it can take hours to import on a low end machine.

If you are on a low end machine you should stop your indexer updating scripts.
cd /tmp

wget https://www.dropbox.com/s/qkmgbvmdv9a5w8q/predb_dump_08172015.tar.gz

gunzip predb_dump_08172015.tar.gz

cd /var/www/nZEDb/./misc/testing/Dev

php dump_predb.php local /tmp/predb_dump_08172015.tar.gz

Import updated databases, these are smaller daily updates since July 2014

cd /var/www/nZEDb/cli

sudo chmod 777 /var/www/nZEDb/resources

sudo chown -R username:www-data /var/www/nZEDb/cli

php /var/www/nZEDb/cli/data/predb_import_daily_batch.php 0 local true

When this is done enable IRC scraping

It consists of ZNC to connect to the predb channel and IRC Scraper to scrape the data from the channel

Install the required app for adding PPA's:

sudo apt-get install python-software-properties

Add the PPA:

sudo add-apt-repository ppa:teward/znc

Update your sources:

sudo apt-get update

Install ZNC

sudo apt-get install znc znc-dbg znc-dev znc-perl znc-python znc-tcl -y

znc --makeconf

znc server config
port 6665
ipv4 yes
ipv6 up to you
ssl false
load partyline
load webadmin

make yourself an admin
make nick whatever you want
alt nick different from regular nick
ident can be same as nick
realname same as nick
bindhost 0.0.0.0
buffer 100 lines (can increase)
chanmodes +stn

load modules chansaver, control panel, perform and webadmin

choose network synirc
load modules chansaver, keepnick, kickrejoin, nickserv, perform, simple_away

server contego.ny.us.synirc.net
password leave blank
port 6697
SSL yes

Add another srever, yes
server toronto.on.ca.synirc.net
port 6697

add channels
automatically join yes
channel name #nZEDbPRE
add another channel no
add another network no
add another user no
launch znc now yes

Make ZNC run on boot http://wiki.znc.in/Running_ZNC_as_a_system_daemon

Configure IRC Scraper

Irc scraper uses perl to scrape the data from the IRC channel that ZNC is connected to

sudo cp /var/www/nZEDb/misc/testing/IRCScraper/settings_example.php /var/www/nZEDb/misc/testing/IRCScraper/settings.php

sudo nano /var/www/nZEDb/misc/testing/IRCScraper/settings.php

change $username to what you chose in ZNC

$username = 'htpcguides';

SCRAPE_IRC_SERVER to 0.0.0.0

SCRAPE_IRC_PORT to 6665

SCRAPE_IRC_TLS to false

define('SCRAPE_IRC_PASSWORD', 'htpcguides');

[Wally73]

.....
This is a large predb, it can take hours to import on a low end machine.

If you are on a low end machine you should stop your indexer updating scripts.

You should ALWAYS stop the update scripts

znc server config
port 6665

You can pick a port yourself
....

server contego.ny.us.synirc.net
password leave blank
port 6697
SSL yes

Add another srever, yes
server toronto.on.ca.synirc.net
port 6697

SSL yes

i would suggest people register a nice on sync first to use with ircscraper.

Code: [Select]

/nickserv register nickname password (email)request a vhost

Code: [Select]

/hs request name.of.host.you.want

put the nickserv identify password in the right field in znc (nickserv)

[drkhelmt]

Thanks, blindpet and Wally.

[david_ritterhous]

what am i doing wrong here

Code: [Select]

// WARNING
//
// Do NOT edit this file while ZNC is running!
// Use webadmin or *controlpanel instead.
//
// Altering this file by hand will forfeit all support.
//
// But if you feel risky, you might want to read help on /znc saveconfig and /znc rehash.
// Also check http://en.znc.in/wiki/Configuration

Version = 1.6.0
<Listener l>
        Port = 6665
        IPv4 = true
        IPv6 = false
        SSL = false
</Listener>
LoadModule = webadmin
LoadModule = partyline

<User davidritt23>
        Pass       = sha2RAMBLEi$
        Admin      = true
        Nick       = nick
        AltNick    = nick2_
        Ident      = nick
        RealName   = nick
        BindHost   = 0.0.0.0
        Buffer = 100
        LoadModule = chansaver
        LoadModule = controlpanel
LoadModule = perform
LoadModule = webadmin

        <Network synirc>
                LoadModule = simple_away
       LoadModule = chansaver
        LoadModule = controlpanel
LoadModule = keepnick
LoadModule = kickrejoin
LoadModule = nickserv
LoadModule = perform

                Server     = contego.ny.us.synirc.net +6697

                <Chan #nZEDbPRE>
                </Chan>
        </Network>
</User>


[redalert]

wget https://www.dropbox.com/s/ko53hpgjhh0mlg5/predb_dump-062714.csv.gz

Where can i get this file ? The link is dead.

Thx

[Nox71]

wget https://www.dropbox.com/s/ko53hpgjhh0mlg5/predb_dump-062714.csv.gz

Where can i get this file ? The link is dead.

Thx

You can find the correct link here: http://forums.nzedb.com/index.php?topic=1614.0

[mysonsdad]

Hi all,

as far as i understood the whole scraping...

ZNC is connecting to the IRC-Servers in background to scrape information from them.

Local script "scrape.php" connects to local ZNC and collects Information to put them in the DB.

So, why do I need a password here and where is it defined in the ZNC-configuration?

SCRAPE_IRC_TLS to false

define('SCRAPE_IRC_PASSWORD', 'htpcguides');

Thanks

[Wally73]

Hi all,

as far as i understood the whole scraping...

ZNC is connecting to the IRC-Servers in background to scrape information from them.

Local script "scrape.php" connects to local ZNC and collects Information to put them in the DB.

So, why do I need a password here and where is it defined in the ZNC-configuration?

when you set up a user in znc (which will connect to synirc) i'm pretty sure the znc user has a password

define('SCRAPE_IRC_PASSWORD', 'htpcguides');

password is the form of 'zncuser/network:password' in ircscrapper settings.php

[mysonsdad]

HI Wally,

yes, you are right. There is a Password when Setting up ZNC.
It's mentioned in another guide (https://github.com/nZEDb/nZEDb_Misc/blob/master/Guides/Various/ZNC/Guide.md)

Bets regards

[Zoing]

The full predb dump link seems to be dead.

https://www.dropbox.com/s/qkmgbvmdv9a5w8q/predb_dump_08172015.tar.gz

Can anyone upload a newer version or update the download link.

Thanks

[WyldWolf]

Not only is the link dead but even though the daily dumps can be found via GH can't be used because the assembly php is also a 404.

[Wally73]

Not only is the link dead but even though the daily dumps can be found via GH can't be used because the assembly php is also a 404.

thx for noticing the wrong link in that file on dumps (fixed that)

daily dumps have nothing to do with that old predb dump


and for the rest this is off topic so open a post in support