Author Topic: Unicode error  (Read 177937 times)

activ

  • Guest
Unicode error
« on: 2013-11-10, 01:19:26 AM »
Hey guys,

I have a problem with postprocessing, this is the error:

PostProcess tv Threaded Started at 07:55:25
We will be using a max of 1 threads, a queue of 75 tv releases.
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/local/lib/python3.3/threading.py", line 637, in _bootstrap_inner
    self.run()
  File "/var/www/html/nZEDb/www/../misc/update_scripts/python_scripts/postprocess_threaded.py", line 136, in run
    subprocess.call(["php", pathname+"/../nix_scripts/tmux/bin/postprocess_new.php", ""+my_id])
  File "/usr/local/lib/python3.3/subprocess.py", line 520, in call
    with Popen(*popenargs, **kwargs) as p:
  File "/usr/local/lib/python3.3/subprocess.py", line 820, in __init__
    restore_signals, start_new_session)
  File "/usr/local/lib/python3.3/subprocess.py", line 1380, in _execute_child
    restore_signals, start_new_session, preexec_fn)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 39-40: ordinal not in range(128)

This halts the entire postprocessing, please help.

Oh, and my linux lang variable is set to UTF8 in case that matters.


Offline jonnyboy

  • Epic Indexer
  • *****
  • Posts: 1046
  • Helpful: +93/-1
  • Lazzy Trucker
    • nZEDb
Re: Unicode error
« Reply #1 on: 2013-11-10, 06:30:38 AM »
Sorry. I tried. I failed. Don't know how to fix it.

activ

  • Guest
Re: Unicode error
« Reply #2 on: 2013-11-10, 06:38:57 AM »
Thanks for the attempt at least, I have been trying to figure this out too.
I even tried to process all the releases using the unthreaded php script, that works fine, but when I go back to the threaded python one it will give the error again.
Anything I can do to figure out how this is caused? Or if there is no way to fix it for now, maybe you can provide a setting allowing me to have the tmux script use the php version of the script instead of the python one?

Offline jonnyboy

  • Epic Indexer
  • *****
  • Posts: 1046
  • Helpful: +93/-1
  • Lazzy Trucker
    • nZEDb
Re: Unicode error
« Reply #3 on: 2013-11-10, 06:43:12 AM »
Its a non utf-8 (unicode) character, not being encoded correctly.

Offline junkman690

  • Decent Indexer
  • ***
  • Posts: 60
  • Helpful: +4/-0
    • ubernzb
Re: Unicode error
« Reply #4 on: 2013-11-11, 01:19:05 AM »
It is possible to force the php output to UTF-8 rather than trying to set it in the python scripts, or am I speaking crazy talk.... (honestly dont know)

Offline jonnyboy

  • Epic Indexer
  • *****
  • Posts: 1046
  • Helpful: +93/-1
  • Lazzy Trucker
    • nZEDb
Re: Unicode error
« Reply #5 on: 2013-11-11, 04:15:28 AM »
It is possible to force the php output to UTF-8 rather than trying to set it in the python scripts, or am I speaking crazy talk.... (honestly dont know)
No.  The error is when python tries to pass a non utf-8 character.

activ

  • Guest
Re: Unicode error
« Reply #6 on: 2013-11-12, 11:58:55 PM »
What about having a setting that allows you to use the php script in tmux instead? The php script works fine.

activ

  • Guest
Re: Unicode error
« Reply #7 on: 2013-11-15, 10:53:48 PM »
Did another git pull and haven't seen it since.

activ

  • Guest
Re: Unicode error
« Reply #8 on: 2013-11-15, 11:26:53 PM »
Shouldn't have said anything, it's back.

Offline jonnyboy

  • Epic Indexer
  • *****
  • Posts: 1046
  • Helpful: +93/-1
  • Lazzy Trucker
    • nZEDb
Re: Unicode error
« Reply #9 on: 2013-11-16, 05:45:57 AM »
Have you set the default character set, for both server and client in my.cnf?
Code: [Select]
[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock
default-character-set = utf8

[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0

[mysqld]
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
#tmpdir = /dev/shm
tmpdir = /tmp
character-set-server = utf8
performance_schema = on

transaction-isolation = READ-COMMITTED

lc-messages-dir = /usr/share/mysql
skip-external-locking

#inndb recovery mode
innodb_force_recovery = 0

max-connections = 100
max_allowed_packet = 32M
thread_stack = 192K
thread_cache_size        = 20
group_concat_max_len    = 8192

tmp_table_size = 2G
max_heap_table_size = 2G
table_open_cache = 20000
low_priority_updates = 1
concurrent_insert = ALWAYS
join_buffer_size = 64M
key_buffer_size = 8M
read_buffer_size = 8M

explicit_defaults_for_timestamp = TRUE

myisam-recover-options  = BACKUP
#max_connections        = 100
#table_cache            = 64
#thread_concurrency       = 10

query_cache_limit = 64M
query_cache_size = 1G
#query_cache_size = 0

#general_log_file = /var/log/mysql/mysql.log
#general_log = 1
log_error = /var/log/mysql/error.log

slow-query-log = 1
slow_query_log_file = /var/log/mysql/mysql-slow.log
long_query_time = 5
log-queries-not-using-indexes


innodb_ft_min_token_size = 1

max_user_connections = 200
#table_cache = 4000

#InnoDB specifics

#Engine options
innodb = ON
innodb_file_format = Barracuda
innodb_file_per_table = 1

#misc uncategorized
innodb_table_locks = false
innodb_lock_wait_timeout = 240
innodb_use_sys_malloc = 1

#threading
innodb_thread_concurrency = 0 #or set to 0 and let mysql decide what it needs
innodb_read_io_threads = 64
innodb_write_io_threads = 64
innodb_purge_threads = 1

#buffer pool related
innodb_buffer_pool_size = 16G
innodb_buffer_pool_instances = 16 # each buffer pool instance should be at least 1 gigabyte
innodb_change_buffering = all
innodb_max_dirty_pages_pct = 90

expand_fast_index_creation = 1
#innodb_merge_sort_block_size = 1G

#flush related
innodb_flush_method = O_DIRECT
innodb_flush_log_at_trx_commit = 0

#transaction logs
innodb_log_buffer_size = 1G
innodb_log_files_in_group = 2
innodb_log_file_size = 4G #this should be 25% of innodb_buffer_pool_size

[mysqldump]
quick
quote-names
max_allowed_packet = 32M

[mysql]
#no-auto-rehash # faster start of mysql but no tab completition

[isamchk]
key_buffer_size = 16M

#
# * IMPORTANT: Additional settings that can override those from this file!
#   The files must end with '.cnf', otherwise they'll be ignored.
#
!includedir /etc/mysql/conf.d/

activ

  • Guest
Re: Unicode error
« Reply #10 on: 2013-11-16, 10:33:40 PM »
Added it now, and restarted mysqld to be sure. No difference.

PostProcess movie Threaded Started at 05:33:09
We will be using a max of 1 threads, a queue of 100 movie releases.
.Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/local/lib/python3.3/threading.py", line 637, in _bootstrap_inner
    self.run()
  File "/var/www/html/nZEDb/misc/update_scripts/python_scripts/postprocess_threaded.py", line 136, in run
    subprocess.call(["php", pathname+"/../nix_scripts/tmux/bin/postprocess_new.php", ""+my_id])
  File "/usr/local/lib/python3.3/subprocess.py", line 520, in call
    with Popen(*popenargs, **kwargs) as p:
  File "/usr/local/lib/python3.3/subprocess.py", line 820, in __init__
    restore_signals, start_new_session)
  File "/usr/local/lib/python3.3/subprocess.py", line 1380, in _execute_child
    restore_signals, start_new_session, preexec_fn)
UnicodeEncodeError: 'ascii' codec can't encode character '\xe4' in position 25: ordinal not in range(128)

Offline jonnyboy

  • Epic Indexer
  • *****
  • Posts: 1046
  • Helpful: +93/-1
  • Lazzy Trucker
    • nZEDb
Re: Unicode error
« Reply #11 on: 2013-11-16, 10:36:51 PM »
Are you current?

activ

  • Guest
Re: Unicode error
« Reply #12 on: 2013-11-16, 11:02:54 PM »
Yes, did a pull an hour ago.

Offline jonnyboy

  • Epic Indexer
  • *****
  • Posts: 1046
  • Helpful: +93/-1
  • Lazzy Trucker
    • nZEDb
Re: Unicode error
« Reply #13 on: 2013-11-16, 11:04:03 PM »
I do not know how to fix it then.

Offline mtwitty

  • Junior Indexer
  • **
  • Posts: 28
  • Helpful: +0/-0
Re: Unicode error
« Reply #14 on: 2013-11-16, 11:20:58 PM »
Jonny,

I was having this problem months ago when I upgraded to python3 and couldn't find a solution at the time. So I downgraded to python2. I did a pull from master a couple of hours ago, so I'm current, and now I'm getting the error with python2.

But even with python2 no post processing is being done right now.