UnicodeError: source returned bytes, but no encoding specified

[adrya.stembridge@…]

How to Reproduce

While doing a GET operation on /newticket, Trac issued an internal error.

I upgraded trac from .12 to 1.0, then ran the trac-admin upgrade command. I moved the existing ckeditor folder to ckeditor_suspect, and downloaded the latest version using instructions on your install page. Restarted apache, still get same error as in subject. This error appears anywhere the ckEditor plugin would appear (viewing a ticket, creating a ticket). The error does not appear when ckEditor is not involved.'

Request parameters:

{}

User agent: Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:16.0) Gecko/20100101 Firefox/16.0

System Information

Trac	1.0
Trac	1.0
Genshi	0.7 (without speedups)
mod_python	3.3.1
pysqlite	2.4.1
Python	2.6.6 (r266:84292, Jul 10 2013, 22:48:45) [GCC 4.4.7 20120313 (Red Hat 4.4.7-3)]
Python	2.6.6 (r266:84292, Jul 10 2013, 22:48:45) [GCC 4.4.7 20120313 (Red Hat 4.4.7-3)]
setuptools	0.6
setuptools	0.6
SQLite	3.6.20
Subversion	1.6.11 (r934486)
jQuery	1.7.2

Enabled Plugins

CKIntegration	1.1dev
TracAccountManager	0.5dev-r13381
TracAnnouncer	1.0dev-r12503

Python Traceback

Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/Trac-1.0-py2.6.egg/trac/web/main.py", line 497, in _dispatch_request
    dispatcher.dispatch(req)
  File "/usr/lib/python2.6/site-packages/Trac-1.0-py2.6.egg/trac/web/main.py", line 233, in dispatch
    content_type)
  File "/usr/lib/python2.6/site-packages/Trac-1.0-py2.6.egg/trac/web/chrome.py", line 968, in render_template
    stream |= self._filter_stream(req, method, filename, stream, data)
  File "/usr/lib/python2.6/site-packages/Genshi-0.7-py2.6-linux-x86_64.egg/genshi/core.py", line 133, in __or__
    return Stream(_ensure(function(self)), serializer=self.serializer)
  File "/usr/lib/python2.6/site-packages/Trac-1.0-py2.6.egg/trac/web/chrome.py", line 1147, in inner
    data)
  File "/usr/lib/python2.6/site-packages/CKIntegration-1.1dev-py2.6.egg/ckintegration/__init__.py", line 241, in filter_stream
    stream |= Transformer('.//body').prepend(HTML(ck_plugin_init))
  File "/usr/lib/python2.6/site-packages/Genshi-0.7-py2.6-linux-x86_64.egg/genshi/input.py", line 442, in HTML
    return Stream(list(HTMLParser(BytesIO(text), encoding=encoding)))
  File "/usr/lib/python2.6/site-packages/Genshi-0.7-py2.6-linux-x86_64.egg/genshi/core.py", line 273, in _ensure
    event = stream.next()
  File "/usr/lib/python2.6/site-packages/Genshi-0.7-py2.6-linux-x86_64.egg/genshi/input.py", line 449, in _coalesce
    for kind, data, pos in chain(stream, [(None, None, None)]):
  File "/usr/lib/python2.6/site-packages/Genshi-0.7-py2.6-linux-x86_64.egg/genshi/input.py", line 337, in _generate
    raise UnicodeError("source returned bytes, but no encoding specified")
UnicodeError: source returned bytes, but no encoding specified

[anonymous]

I am using CKEditor 3.6.6.1

The web server has full ownership of the htdocs/js/ckeditor folder.

[anonymous]

Here is the trac.ini

# -*- coding: utf-8 -*-

[account-manager]
db_htdigest_realm = 
force_passwd_change = true
hash_method = HtDigestHashMethod
htpasswd_file = /www/virtualhosts/trac/conf/trac.htpasswd
htpasswd_hash_type = md5
password_store = HtPasswdStore
persistent_sessions = true
refresh_passwd = False
user_lock_max_time = 0
verify_email = true

[announcer]
email_enabled = true

[attachment]
max_size = -1
render_unsafe_content = false

[browser]
color_scale = True
downloadable_paths = /trunk, /branches/*, /tags/*
hide_properties = svk:merge
intermediate_color = 
intermediate_point = 
newest_color = (255, 136, 136)
oldest_color = (136, 136, 255)
oneliner_properties = trac:summary
render_unsafe_content = false
wiki_properties = trac:description

[changeset]
max_diff_bytes = 10000000
max_diff_files = 0
wiki_format_messages = true

[ckeditor]
code_styles = php, js, html, sql, default, xml
editor_source = site/js/ckeditor/ckeditor.js
editor_type = full_integration

[components]
acct_mgr.admin.* = enabled
acct_mgr.api.* = enabled
acct_mgr.db.sessionstore = enabled
acct_mgr.htfile.htdigeststore = disabled
acct_mgr.htfile.htpasswdstore = enabled
acct_mgr.http.httpauthstore = disabled
acct_mgr.notification.* = enabled
acct_mgr.pwhash.htdigesthashmethod = enabled
acct_mgr.pwhash.htpasswdhashmethod = disabled
acct_mgr.register.* = enabled
acct_mgr.svnserve.* = enabled
acct_mgr.svnserve.svnservepasswordstore = disabled
acct_mgr.web_ui.* = enabled
announcer.api.announcementsystem = enabled
announcer.distributors.mail.emaildistributor = enabled
announcer.distributors.mail.sendmailemailsender = enabled
announcer.formatters.ticketformatter = enabled
announcer.formatters.wikiformatter = enabled
announcer.opt.acct_mgr.announce.accountmanagerannouncement = enabled
announcer.opt.subscribers.allticketsubscriber = enabled
announcer.opt.subscribers.userchangesubscriber = enabled
announcer.opt.subscribers.watchsubscriber = enabled
announcer.pref.announcerpreferences = enabled
announcer.pref.subscriptionmanagementpanel = enabled
announcer.producers.attachmentchangeproducer = enabled
announcer.producers.ticketchangeproducer = enabled
announcer.producers.wikichangeproducer = enabled
announcer.subscribers.carboncopysubscriber = enabled
announcer.subscribers.ticketownersubscriber = enabled
announcer.subscribers.ticketreportersubscriber = enabled
ckintegration.* = enabled
permredirect.* = enabled
trac.web.auth.loginmodule = disabled
tracopt.versioncontrol.svn.svn_fs.subversionconnector = enabled
tracopt.versioncontrol.svn.svn_prop.subversionmergepropertydiffrenderer = enabled
tracopt.versioncontrol.svn.svn_prop.subversionmergepropertyrenderer = enabled
tracopt.versioncontrol.svn.svn_prop.subversionpropertyrenderer = enabled

[inherit]
plugins_dir = 
templates_dir = 

[logging]
log_type = file

[milestone]
stats_provider = DefaultTicketGroupStatsProvider

[mimeviewer]
max_preview_size = 262144
mime_map = text/x-dylan:dylan,text/x-idl:ice,text/x-ada:ads:adb
tab_width = 8
treat_as_binary = application/octet-stream,application/pdf,application/postscript,application/rtf

[notification]
admit_domains = 
always_notify_owner = true
always_notify_reporter = true
always_notify_updater = true
default_email_format = text/html
email_sender = SendmailEmailSender
ignore_domains = 
mime_encoding = none
sendmail_path = /usr/sbin/sendmail
smtp_always_bcc = astembr@emory.edu
smtp_always_cc = 
smtp_default_domain = 
smtp_enabled = true
smtp_from = trac@localhost
smtp_from_name = 
smtp_password = 
smtp_port = 25
smtp_replyto = trac@localhost
smtp_server = localhost
smtp_subject_prefix = __default__
smtp_user = 
ticket_email_header_fields = reporter, priority, severity
ticket_subject_template = $prefix #$ticket.id: $summary
use_public_cc = false
use_short_addr = false
use_tls = false

[project]
admin = 
admin_trac_url = .
descr = MyApp Description
footer = Visit the Trac open source project at<br /><a href="http://trac.edgewall.org/">http://trac.edgewall.org/</a>
icon = common/trac.ico
name = MyApp
url = 

[query]
default_anonymous_query = status!=closed&cc~=$USER
default_query = status!=closed&owner=$USER
items_per_page = 100
ticketlink_query = ?status=!closed

[report]
items_per_page = 100
items_per_page_rss = 0

[revisionlog]
default_log_limit = 100

[roadmap]
stats_provider = DefaultTicketGroupStatsProvider

[search]
min_query_length = 3

[svn]
branches = trunk,branches/*
tags = tags/*

[ticket]
default_cc = 
default_component = MyApp
default_description = 
default_keywords = 
default_milestone = 
default_owner = 
default_priority = major
default_resolution = fixed
default_severity = Normal
default_summary = 
default_type = defect
default_version = 1.2.3.4
max_comment_size = 262144
max_description_size = 262144
preserve_newlines = default
restrict_owner = false
workflow = ConfigurableTicketWorkflow
[ticket-workflow]
accept = new,assigned,accepted,reopened -> accepted
accept.operations = set_owner_to_self
accept.permissions = TICKET_MODIFY
leave = * -> *
leave.default = 1
leave.operations = leave_status
reassign = new,assigned,accepted,reopened -> assigned
reassign.operations = set_owner
reassign.permissions = TICKET_MODIFY
reopen = closed -> reopened
reopen.operations = del_resolution
reopen.permissions = TICKET_CREATE
resolve = new,assigned,accepted,reopened -> closed
resolve.operations = set_resolution
resolve.permissions = TICKET_MODIFY

[timeline]
abbreviated_messages = True
changeset_collapse_events = false
changeset_long_messages = false
changeset_show_files = 0
default_daysback = 30
max_daysback = 90
newticket_formatter = oneliner
ticket_show_details = false
[trac]
auth_cookie_lifetime = 0
auth_cookie_path = 
authz_file = 
authz_module_name = 
auto_preview_timeout = 2.0
auto_reload = False
backup_dir = db
base_url = 
check_auth_ip = false
database = sqlite:db/trac.db
debug_sql = False
default_charset = iso-8859-15
genshi_cache_size = 128
htdocs_location = 
ignore_auth_case = false
mainnav = wiki,timeline,roadmap,browser,tickets,newticket,search
metanav = login,logout,prefs,help,about
mysqldump_path = mysqldump
never_obfuscate_mailto = false
permission_policies = DefaultPermissionPolicy, LegacyAttachmentPolicy
permission_store = DefaultPermissionStore
pg_dump_path = pg_dump
repository_dir = 
repository_sync_per_request = (default)
repository_type = svn
resizable_textareas = true
secure_cookies = False
show_email_addresses = false
show_ip_addresses = false
timeout = 20
use_base_url_for_redirect = False

[wiki]
ignore_missing_pages = false
max_size = 262144
render_unsafe_content = false
split_page_names = false

[adrya.stembridge@…]

Do the maintainers of this plugin need additional information to look into the reported issue?

[adrya.stembridge@…]

I just deleted my entire trac installation, including the ckeditor plugin. Cleared out related egg files, tracd, trac-admin, and restarted the httpd service.

Installed a new copy of trac with easy_install Trac. Deployed environment. Restarted httpd. Trac works and I see the standard Markdown text editor.

Next, I installed CkEditorPlugin with easy_installed per the instructions. Checked out a copy of ckEditor with the command listed on the CkEditorPlugin instructions. Added this line to my trac.ini:

[components] ckintegration.* = enabled

Next, restarted httpd service. Tried to create a ticket and get the same error as reported in this ticket:

UnicodeError: source returned bytes, but no encoding specified

[anonymous]

Disabling ckintegration and restarting httpd suppresses the error and returns to the standard text entry box.

#[components] #ckintegration.* = enabled

[Jun Omae]

The issue is caused by changes of encoding argument of genshi.input.HTML() API. Therefore, the workaround is downgrading to Genshi 0.6.x from 0.7.

I just created the patch to avoid the issue with Genshi 0.7 and 0.6.x, t11330-r13403.diff​.

[adrya.stembridge@…]

Applied t11330-r13403.diff​, restarted httpd and CkEditorPlugin is now functioning as expected.

Grateful for the reply & patch.

[Jun Omae]

Oh, the patch have yet to be applied.

[anonymous]

I applied locally fwiw.

[Franz]

In 14145:

refs #11330: avoid error with Genshi 0.7:

UnicodeError: source returned bytes, but no encoding specified

[Franz]

Thanks to jun66j5 - the patch works fine. Fixed with commit [14145].