Unicode error

[activ]

Hey guys,

I have a problem with postprocessing, this is the error:

PostProcess tv Threaded Started at 07:55:25
We will be using a max of 1 threads, a queue of 75 tv releases.
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/local/lib/python3.3/threading.py", line 637, in _bootstrap_inner
    self.run()
  File "/var/www/html/nZEDb/www/../misc/update_scripts/python_scripts/postprocess_threaded.py", line 136, in run
    subprocess.call(["php", pathname+"/../nix_scripts/tmux/bin/postprocess_new.php", ""+my_id])
  File "/usr/local/lib/python3.3/subprocess.py", line 520, in call
    with Popen(*popenargs, **kwargs) as p:
  File "/usr/local/lib/python3.3/subprocess.py", line 820, in __init__
    restore_signals, start_new_session)
  File "/usr/local/lib/python3.3/subprocess.py", line 1380, in _execute_child
    restore_signals, start_new_session, preexec_fn)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 39-40: ordinal not in range(128)

This halts the entire postprocessing, please help.

Oh, and my linux lang variable is set to UTF8 in case that matters.

[jonnyboy]

Sorry. I tried. I failed. Don't know how to fix it.

[activ]

Thanks for the attempt at least, I have been trying to figure this out too.
I even tried to process all the releases using the unthreaded php script, that works fine, but when I go back to the threaded python one it will give the error again.
Anything I can do to figure out how this is caused? Or if there is no way to fix it for now, maybe you can provide a setting allowing me to have the tmux script use the php version of the script instead of the python one?

[jonnyboy]

Its a non utf-8 (unicode) character, not being encoded correctly.

[junkman690]

It is possible to force the php output to UTF-8 rather than trying to set it in the python scripts, or am I speaking crazy talk.... (honestly dont know)

[jonnyboy]

It is possible to force the php output to UTF-8 rather than trying to set it in the python scripts, or am I speaking crazy talk.... (honestly dont know)

No.  The error is when python tries to pass a non utf-8 character.

[activ]

What about having a setting that allows you to use the php script in tmux instead? The php script works fine.

[activ]

Did another git pull and haven't seen it since.

[activ]

Shouldn't have said anything, it's back.

[jonnyboy]

Have you set the default character set, for both server and client in my.cnf?

Code: [Select]

[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock
default-character-set = utf8

[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0

[mysqld]
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
#tmpdir = /dev/shm
tmpdir = /tmp
character-set-server = utf8
performance_schema = on

transaction-isolation = READ-COMMITTED

lc-messages-dir = /usr/share/mysql
skip-external-locking

#inndb recovery mode
innodb_force_recovery = 0

max-connections = 100
max_allowed_packet = 32M
thread_stack = 192K
thread_cache_size        = 20
group_concat_max_len    = 8192

tmp_table_size = 2G
max_heap_table_size = 2G
table_open_cache = 20000
low_priority_updates = 1
concurrent_insert = ALWAYS
join_buffer_size = 64M
key_buffer_size = 8M
read_buffer_size = 8M

explicit_defaults_for_timestamp = TRUE

myisam-recover-options  = BACKUP
#max_connections        = 100
#table_cache            = 64
#thread_concurrency       = 10

query_cache_limit = 64M
query_cache_size = 1G
#query_cache_size = 0

#general_log_file = /var/log/mysql/mysql.log
#general_log = 1
log_error = /var/log/mysql/error.log

slow-query-log = 1
slow_query_log_file = /var/log/mysql/mysql-slow.log
long_query_time = 5
log-queries-not-using-indexes


innodb_ft_min_token_size = 1

max_user_connections = 200
#table_cache = 4000

#InnoDB specifics

#Engine options
innodb = ON
innodb_file_format = Barracuda
innodb_file_per_table = 1

#misc uncategorized
innodb_table_locks = false
innodb_lock_wait_timeout = 240
innodb_use_sys_malloc = 1

#threading
innodb_thread_concurrency = 0 #or set to 0 and let mysql decide what it needs
innodb_read_io_threads = 64
innodb_write_io_threads = 64
innodb_purge_threads = 1

#buffer pool related
innodb_buffer_pool_size = 16G
innodb_buffer_pool_instances = 16 # each buffer pool instance should be at least 1 gigabyte
innodb_change_buffering = all
innodb_max_dirty_pages_pct = 90

expand_fast_index_creation = 1
#innodb_merge_sort_block_size = 1G

#flush related
innodb_flush_method = O_DIRECT
innodb_flush_log_at_trx_commit = 0

#transaction logs
innodb_log_buffer_size = 1G
innodb_log_files_in_group = 2
innodb_log_file_size = 4G #this should be 25% of innodb_buffer_pool_size

[mysqldump]
quick
quote-names
max_allowed_packet = 32M

[mysql]
#no-auto-rehash # faster start of mysql but no tab completition

[isamchk]
key_buffer_size = 16M

#
# * IMPORTANT: Additional settings that can override those from this file!
#   The files must end with '.cnf', otherwise they'll be ignored.
#
!includedir /etc/mysql/conf.d/

[activ]

Added it now, and restarted mysqld to be sure. No difference.

PostProcess movie Threaded Started at 05:33:09
We will be using a max of 1 threads, a queue of 100 movie releases.
.Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/local/lib/python3.3/threading.py", line 637, in _bootstrap_inner
    self.run()
  File "/var/www/html/nZEDb/misc/update_scripts/python_scripts/postprocess_threaded.py", line 136, in run
    subprocess.call(["php", pathname+"/../nix_scripts/tmux/bin/postprocess_new.php", ""+my_id])
  File "/usr/local/lib/python3.3/subprocess.py", line 520, in call
    with Popen(*popenargs, **kwargs) as p:
  File "/usr/local/lib/python3.3/subprocess.py", line 820, in __init__
    restore_signals, start_new_session)
  File "/usr/local/lib/python3.3/subprocess.py", line 1380, in _execute_child
    restore_signals, start_new_session, preexec_fn)
UnicodeEncodeError: 'ascii' codec can't encode character '\xe4' in position 25: ordinal not in range(128)

[jonnyboy]

Are you current?

[activ]

Yes, did a pull an hour ago.

[jonnyboy]

I do not know how to fix it then.

[mtwitty]

Jonny,

I was having this problem months ago when I upgraded to python3 and couldn't find a solution at the time. So I downgraded to python2. I did a pull from master a couple of hours ago, so I'm current, and now I'm getting the error with python2.

But even with python2 no post processing is being done right now.