TypeError: ('expected string in list', u'displayName') after Trac[auth] INFO: get users

[brian@…]

In a clean trac installation only with Accountmanager and DirectoryAuthPlugin, all is fine (fetching the user and the user groups is working correctly). But if than the

[account-manager]
password_store = DirAuthStore

is selected in trac.ini (this is only possible within the trac.ini file because the Accountmanager will not unlock the option) I always get the following error:

2016-01-19 03:51:23,424 Trac[auth] INFO: get users
2016-01-19 03:51:23,437 Trac[main] ERROR: Internal Server Error: <RequestWithSession "POST '/login'">, referrer 'https://[...]'
Traceback (most recent call last):
  File "/opt/lib/python2.7/site-packages/Trac-1.0.9-py2.7.egg/trac/web/main.py", line 554, in _dispatch_request
    dispatcher.dispatch(req)
  File "/opt/lib/python2.7/site-packages/Trac-1.0.9-py2.7.egg/trac/web/main.py", line 212, in dispatch
    self._pre_process_request(req, chosen_handler)
  File "/opt/lib/python2.7/site-packages/Trac-1.0.9-py2.7.egg/trac/web/main.py", line 375, in _pre_process_request
    chosen_handler = filter_.pre_process_request(req, chosen_handler)
  File "/opt/lib/python2.7/site-packages/Trac-1.0.9-py2.7.egg/trac/versioncontrol/api.py", line 372, in pre_process_request
    if is_default(reponame):
  File "/opt/lib/python2.7/site-packages/Trac-1.0.9-py2.7.egg/trac/versioncontrol/api.py", line 34, in is_default
    return not reponame or reponame in ('(default)', _('(default)'))
  File "/opt/lib/python2.7/site-packages/Trac-1.0.9-py2.7.egg/trac/util/translation.py", line 202, in gettext
    if not self.isactive:
  File "/opt/lib/python2.7/site-packages/Trac-1.0.9-py2.7.egg/trac/util/translation.py", line 182, in isactive
    self.activate(get_locale(), env_path)
  File "/opt/lib/python2.7/site-packages/Trac-1.0.9-py2.7.egg/trac/web/main.py", line 520, in <lambda>
    translation.make_activable(lambda: req.locale, env.path if env else None)
  File "/opt/lib/python2.7/site-packages/Trac-1.0.9-py2.7.egg/trac/web/api.py", line 355, in __getattr__
    value = self.callbacks[name](self)
  File "/opt/lib/python2.7/site-packages/Trac-1.0.9-py2.7.egg/trac/web/main.py", line 310, in _get_locale
    preferred = req.session.get('language')
  File "/opt/lib/python2.7/site-packages/Trac-1.0.9-py2.7.egg/trac/web/api.py", line 355, in __getattr__
    value = self.callbacks[name](self)
  File "/opt/lib/python2.7/site-packages/Trac-1.0.9-py2.7.egg/trac/web/main.py", line 302, in _get_session
    return Session(self.env, req)
  File "/opt/lib/python2.7/site-packages/Trac-1.0.9-py2.7.egg/trac/web/session.py", line 205, in __init__
    if req.authname == 'anonymous':
  File "/opt/lib/python2.7/site-packages/Trac-1.0.9-py2.7.egg/trac/web/api.py", line 355, in __getattr__
    value = self.callbacks[name](self)
  File "/opt/lib/python2.7/site-packages/Trac-1.0.9-py2.7.egg/trac/web/main.py", line 161, in authenticate
    authname = authenticator.authenticate(req)
  File "/opt/lib/python2.7/site-packages/TracAccountManager-0.5dev-py2.7.egg/acct_mgr/util.py", line 81, in wrap
    return func(self, *args, **kwds)
  File "/opt/lib/python2.7/site-packages/TracAccountManager-0.5dev-py2.7.egg/acct_mgr/web_ui.py", line 390, in authenticate
    username = self._remote_user(req)
  File "/opt/lib/python2.7/site-packages/TracAccountManager-0.5dev-py2.7.egg/acct_mgr/web_ui.py", line 741, in _remote_user
    if acctmgr.check_password(username, password) == True:
  File "/opt/lib/python2.7/site-packages/TracAccountManager-0.5dev-py2.7.egg/acct_mgr/api.py", line 300, in check_password
    valid = store.check_password(user, password)
  File "/opt/lib/python2.7/site-packages/TracDirectoryAuth-2.0.1dev_r14887-py2.7.egg/tracext/dirauth/auth.py", line 228, in check_password
    self.get_users()
  File "/opt/lib/python2.7/site-packages/TracDirectoryAuth-2.0.1dev_r14887-py2.7.egg/tracext/dirauth/auth.py", line 131, in get_users
    self.proxy_attr, self.name_attr])
  File "/opt/lib/python2.7/site-packages/python_ldap-2.4.25-py2.7-linux-x86_64.egg/ldap/ldapobject.py", line 597, in search_s
    return self.search_ext_s(base,scope,filterstr,attrlist,attrsonly,None,None,timeout=self.timeout)
  File "/opt/lib/python2.7/site-packages/python_ldap-2.4.25-py2.7-linux-x86_64.egg/ldap/ldapobject.py", line 963, in search_ext_s
    return self._apply_method_s(SimpleLDAPObject.search_ext_s,*args,**kwargs)
  File "/opt/lib/python2.7/site-packages/python_ldap-2.4.25-py2.7-linux-x86_64.egg/ldap/ldapobject.py", line 900, in _apply_method_s
    return func(self,*args,**kwargs)
  File "/opt/lib/python2.7/site-packages/python_ldap-2.4.25-py2.7-linux-x86_64.egg/ldap/ldapobject.py", line 590, in search_ext_s
    msgid = self.search_ext(base,scope,filterstr,attrlist,attrsonly,serverctrls,clientctrls,timeout,sizelimit)
  File "/opt/lib/python2.7/site-packages/python_ldap-2.4.25-py2.7-linux-x86_64.egg/ldap/ldapobject.py", line 586, in search_ext
    timeout,sizelimit,
  File "/opt/lib/python2.7/site-packages/python_ldap-2.4.25-py2.7-linux-x86_64.egg/ldap/ldapobject.py", line 106, in _ldap_call
    result = func(*args,**kwargs)
TypeError: ('expected string in list', u'displayName')

I have updated all dependencies and searched the internet. It seems that this TypeError is a known Unicode problem but I don't know how to fix this!

My current setting for the Accountmanager (after adding the last value "password_store = DirAuthStore" the above mentioned error occurs):

[account-manager]
account_changes_notify_addresses =
allow_delete_account = enabled
authentication_url =
cookie_refresh_pct = 10
db_htdigest_realm =
db_htpasswd_hash_type = crypt
dir_basedn = cn=users,dc=XXX,dc=XXX
dir_binddn = uid=XXX,cn=users,dc=XXX,dc=XXX
dir_bindpw = XXX
user_attr = displayName
email_regexp = (?i)^[A-Z0-9._%+-]+@(?:[A-Z0-9-]+\.)+[A-Z]{2,6}$
environ_auth_overwrite = enabled
force_passwd_change = enabled
generated_password_length = 8
hash_method = HtDigestHashMethod
htdigest_file =
htdigest_realm =
htpasswd_file =
htpasswd_hash_type = crypt
login_attempt_max_count = 0
login_opt_list = False
notify_actions =
password_file =
password_format =
password_store =
persistent_sessions = False
refresh_passwd = disabled
register_basic_token =
register_check = BasicCheck, EmailCheck, BotTrapCheck, RegExpCheck,
 UsernamePermCheck
reset_password = enabled
user_lock_max_time = 86400
user_lock_time = 0
user_lock_time_progression = 1
username_char_blacklist = :[]
username_regexp = (?i)^[A-Z0-9.\-_]{5,}$
verify_email = enabled
group_class_attr = posixGroup
group_tracadmin = @XXX
group_basedn = cn=groups,dc=XXX,dc=XXX
password_store = DirAuthStore

[brian@…]

I found the issue in auth.py starting at line 131:

131                 users = lcnx.search_s(self.dir_basedn, ldap.SCOPE_SUBTREE,
132                                       "objectClass=person",
133                                       [self.user_attr, self.email_attr,
134                                        self.proxy_attr, self.name_attr])

self.* are unicode variables but string variables are expected, so this is working fine in my configuration:

131                 users = lcnx.search_s(self.dir_basedn, ldap.SCOPE_SUBTREE,
132                                       "objectClass=person",
133                                       [str(self.user_attr), str(self.email_attr),
134                                        str(self.proxy_attr), str(self.name_attr)])

[Ryan J Ollos]

It looks like the attributes should be utf-8 encoded. Could you please try the following patch?

directoryauthplugin/trunk/tracext/dirauth/auth.py

@@ -17 +17 @@
 from trac.config import IntOption, Option
 from trac.core import Component, TracError, implements
 from trac.perm import IPermissionGroupProvider
-from trac.util.text import to_unicode
+from trac.util.text import to_unicode, to_utf8
 from trac.util.translation import _
 
 from acct_mgr.api import IPasswordStore
@@ -127 +127 @@
             else:
                 users = lcnx.search_s(self.dir_basedn, ldap.SCOPE_SUBTREE,
                                       "objectClass=person",
-                                      [self.user_attr, self.email_attr,
-                                       self.proxy_attr, self.name_attr])
+                                      [to_utf8(self.user_attr),
+                                       to_utf8(self.email_attr),
+                                       to_utf8(self.proxy_attr),
+                                       to_utf8(self.name_attr)])
                 userinfo = [self._get_userinfo(u[1]) for u in users]
         else:
             raise TracError('Unable to bind to Active Directory')
@@ -141 +143 @@
             group = group[1:]
             self.log.debug("search groups cn=%s,%s"
                            % (group, self.group_basedn))
-        g = cnx.search_s("cn=%s,%s" % (group, self.group_basedn), ldap.SCOPE_BASE, attrlist=[str(self.member_attr)])
+        g = cnx.search_s("cn=%s,%s" % (group, self.group_basedn), ldap.SCOPE_BASE, attrlist=[to_utf8(self.member_attr)])
         self.log.debug(g)
         if g and self.member_attr in g[0][1]:
             users = []

There are other places where the string should probably be utf-8 encoded, but I won't worry about those issues for now if you aren't seeing errors. If the changes work for you, I'll commit them.

[Ryan J Ollos]

In 15167:

2.0.1dev: Encode LDAP parameters to utf-8

Untested patch.

Fixes #12634.

[mddeff]

When trying to implement the DirAuth plugin (using 2.0.1dev), I ran into a similar issue above.

Environment

• Trac Host
  • CentOS 7.2.1511
  • Apache 2.4.6
  • Trac 1.0
  • AccountManager 0.4.4
  • DirectoryAuth 2.0.1dev

• LDAP Server
  • Microsoft Active Directory on Windows Server 2012 R2

Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/trac/web/api.py", line 502, in send_error
    data, 'text/html')
  File "build/bdist.linux-x86_64/egg/trac/web/chrome.py", line 955, in render_template
    message = req.session.pop('chrome.%s.%d' % (type_, i))
  File "build/bdist.linux-x86_64/egg/trac/web/api.py", line 304, in __getattr__
    value = self.callbacks[name](self)
  File "build/bdist.linux-x86_64/egg/trac/web/main.py", line 268, in _get_session
    return Session(self.env, req)
  File "build/bdist.linux-x86_64/egg/trac/web/session.py", line 200, in __init__
    if req.authname == 'anonymous':
  File "build/bdist.linux-x86_64/egg/trac/web/api.py", line 304, in __getattr__
    value = self.callbacks[name](self)
  File "build/bdist.linux-x86_64/egg/trac/web/main.py", line 135, in authenticate
    authname = authenticator.authenticate(req)
  File "build/bdist.linux-x86_64/egg/acct_mgr/util.py", line 82, in wrap
    return func(self, *args, **kwds)
  File "build/bdist.linux-x86_64/egg/acct_mgr/web_ui.py", line 338, in authenticate
    user = self._remote_user(req)
  File "build/bdist.linux-x86_64/egg/acct_mgr/web_ui.py", line 684, in _remote_user
    if acctmgr.check_password(user, password) == True:
  File "build/bdist.linux-x86_64/egg/acct_mgr/api.py", line 259, in check_password
    valid = store.check_password(user, password)
  File "build/bdist.linux-x86_64/egg/tracext/dirauth/auth.py", line 232, in check_password
    self.get_users()
  File "build/bdist.linux-x86_64/egg/tracext/dirauth/auth.py", line 126, in get_users
    userinfo = self.expand_group_users(lcnx, self.group_validusers)
  File "build/bdist.linux-x86_64/egg/tracext/dirauth/auth.py", line 148, in expand_group_users
    attrlist=[to_utf8(self.member_attr)])
  File "build/bdist.linux-x86_64/egg/trac/util/text.py", line 216, in to_utf8
    u = unicode(text, 'utf-8')
TypeError: decoding Unicode is not supported

I tracked the issue down to this ticket. I manually made the changes suggested in comment 1 (making them str)and it worked successfully.

Theory

I think this might be caused by the fact that MS AD in 2012R2 uses LDAPv3 which may ​default to UTF-8 for all of its content. When the information is queried, and subsequently passed through to_utf8, it fails as it already is utf8.

Solution

I'm not sure the best way to solve this problem; potentially check to see if

1. The ldap server is returning utf8 already
2. check to see what is configured for dir_charset

relevant trac config for good measure

[account-manager]
authentication_url =
cache_memprune = 5
cache_memsize = 400
cache_memsize_warn = 300
cache_timeout = 60
cache_ttl = 90
db_htdigest_realm =
dir_basedn = DC=corp,DC=example,DC=com
dir_binddn = xxx
dir_bindpw = xxx
#dir_charset = utf-8
dir_scope = 2
dir_timeout = 5
dir_uri = ldap://domaincontroller1.corp.example.com:3268
email_attr = mail
force_passwd_change = true
group_basedn = OU=Groups,OU=example,DC=corp,DC=example,DC=com
group_class_attr = group
group_expand = 1
group_tracadmin = @tracadmins
group_validusers = @tracusers
hash_method = HtDigestHashMethod
htdigest_file =
htdigest_realm =
htpasswd_file =
htpasswd_hash_type = crypt
member_attr = member
name_attr = displayName
password_file =
password_store = DirAuthStore
persistent_sessions = False
proxy_attr = proxyAddress
refresh_passwd = False
user_attr = sAMAccountName
verify_email = true

Thoughts?

If wanted, I can open up a new ticket.

[Ryan J Ollos]

Replying to mddeff:

I think this might be caused by the fact that MS AD in 2012R2 uses LDAPv3 which may ​default to UTF-8 for all of its content. When the information is queried, and subsequently passed through to_utf8, it fails as it already is utf8.

If the content is already utf-8, then why do we see the error TypeError: decoding Unicode is not supported? It appears the content is unicode.

I was unsure of the type of the content, which is why to_utf8 seemed like the safest bet. However, the behavior of to_utf8 in Trac 1.0 is poor, in that it doesn't handle unicode input gracefully.

Here's the behavior in ​Trac 1.0:

>>> from trac.util.text import to_utf8
>>> unicode_text = u'śśś'
>>> utf8_text = unicode_text.encode('utf-8')
>>> type(unicode_text)
<type 'unicode'>
>>> type(utf8_text)
<type 'str'>
>>> unicode_text
u'\u015b\u015b\u015b'
>>> utf8_text
'\xc5\x9b\xc5\x9b\xc5\x9b'
>>> to_utf8(unicode_text)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "trac/util/text.py", line 216, in to_utf8
    u = unicode(text, 'utf-8')
TypeError: decoding Unicode is not supported
>>> to_utf8(utf8_text)
'\xc5\x9b\xc5\x9b\xc5\x9b'

Here's the behavior in ​Trac 1.0.9:

>>> from trac.util.text import to_utf8
>>> unicode_text = u'śśś'
>>> utf8_text = unicode_text.encode('utf-8')
>>> type(unicode_text)
<type 'unicode'>
>>> type(utf8_text)
<type 'str'>
>>> unicode_text
u'\u015b\u015b\u015b'
>>> utf8_text
'\xc5\x9b\xc5\x9b\xc5\x9b'
>>> to_utf8(unicode_text)
'\xc5\x9b\xc5\x9b\xc5\x9b'
>>> to_utf8(utf8_text)
'\xc5\x9b\xc5\x9b\xc5\x9b'

Are you comfortable with applying and testing a patch? I will prepare a patch.

[Ryan J Ollos]

Behavior of to_utf8 was improved in ​[trac 11752].

Would you mind testing the patch t12634.diff​? Thanks.

[mddeff]

Success; that patch fixed the issue. Thanks again!

[Ryan J Ollos]

Thanks for feedback. I'll commit the change.

[Ryan J Ollos]

In 15257:

2.0.1dev: Revise [15167]

Account for poor behavior of to_utf8 in Trac < 1.0.2

Fixes #12634.