Submit form ended to 'expected string or buffer'

[ntmlod]

I have edit my form several times because in the beginning I wasn't not sure if I had followed the good method to implement it (more than one {{{#!WikiForms }}} in the page, issue with void value at submission).

I have simplified my initial form for testing.

{{{#!WikiForms
<tf>submit:label='Record your changes'</tf>
<tf>lastmodified</tf>
<tf>dump:.*</tf>
{{{#!td
1
}}}
{{{#!td
<tf>input:1_revision</tf>
}}}
{{{#!td
<tf>select:1_arch:'' 'select 1' 'select 2'</tf>
}}}
{{{#!td
<tf>input:1_compile</tf>
}}}
{{{#!td
<tf>textarea:1_report</tf>
}}}
{{{#!td
<tf>textarea:1_comments</tf>
}}}
{{{#!td
''<tf>when:1_revision</tf>'' \\by __<tf>who:1_revision</tf>__
}}}
|-
{{{#!td
2
}}}
{{{#!td
<tf>input:2_revision</tf>
}}}
{{{#!td
<tf>select:2_arch:'' 'select 1' 'select 2'</tf>
}}}
{{{#!td
<tf>input:2_compile</tf>
}}}
{{{#!td
<tf>textarea:2_report</tf>
}}}
{{{#!td
<tf>textarea:2_comments</tf>
}}}
{{{#!td
''<tf>when:2_revision</tf>'' \\by __<tf>who:2_revision</tf>__
}}}
}}}

When I submit some fake values, I get the error message on /wikiforms/update but regarding the database the process seems to have crashed in the line 2 with missing records.

I have nothing in the Trac log and I assume the message comes from the Apache server because the pattern is unusual.

[Ryan J Ollos]

I'm unsure of the cause of the error, but one problem I see is that the plugin interpolates SQL query strings, which will likely cause problems with PostgreSQL due to incorrect quoting of arguments. I'll fix that issue in the next commit. Please test those changes and let me know if you still encounter the issue.

[Ryan J Ollos]

In 16872:

WikiForms 0.3dev: Fix string interpolation in DB queries

Refs #13300.

[Ryan J Ollos]

In 16873:

WikiForms 0.3dev: Fix regression in r16871

The variables need to be poped from the request args dict
so they aren't stored in the database.

Refs #13300.

[Ryan J Ollos]

I noticed that deleting a wiki page does not delete the associated form data from the database. Do you think it should?

[ntmlod]

Unfortunately I still got the same error.

Previously I had made a test uncommentting the line for the trac log to get the args list and finally I have an 'auto' entry that really comes from nowhere. In the next extract, I have removed its content because you have the entire bibliography in BibTex format that comes from the reference manual of our code.
Perhaps this can be related to the use of DoxygenPlugin to render an HTML version of our manual converted from LaTeX.

2017-10-14 00:32:50,013 Trac[main] DEBUG: Dispatching <RequestWithSession "POST '/wikiforms/update'">
2017-10-14 00:32:50,020 Trac[doxygentrac] DEBUG: match_request /wikiforms/update
2017-10-14 00:32:50,021 Trac[MyPagePlugin] DEBUG: match_request: req = /wikiforms/update
2017-10-14 00:32:50,365 Trac[wikiforms] DEBUG: {'/wiki/Admin/PageTemplates/test_form/1_comments': u'c', 'auto': [.........], '/wiki/Admin/PageTemplates/test_form/1_arch': u'select 1', '__FORM_TOKEN': u'fde4e525a0c775870054e591', '/wiki/Admin/PageTemplates/test_form/1_report': u'b', '/wiki/Admin/PageTemplates/test_form/1_revision': u'1', '/wiki/Admin/PageTemplates/test_form/1_compile': u'a', '__BACKPATH': u'/nemo/wiki/Admin/PageTemplates/test_form', '__PAGE': u'/wiki/Admin/PageTemplates/test_form', '__SUBMIT': u'Record your changes'}

Replying to Ryan J Ollos:

I noticed that deleting a wiki page does not delete the associated form data from the database. Do you think it should?

I would say yes.

For me and regarding how we use this plugin, the drawback is when you rename the wikipage. This plugin and the other version TracFormsPlugin lost the link to their entries.
Now that I'm getting used to manipulate the DB, I know how to fix that but, from a user perspective, it is a bit unintelligible.

[Ryan J Ollos]

Replying to ntmlod:

Previously I had made a test uncommentting the line for the trac log to get the args list and finally I have an 'auto' entry that really comes from nowhere. In the next extract, I have removed its content because you have the entire bibliography in BibTex format that comes from the reference manual of our code.
Perhaps this can be related to the use of DoxygenPlugin to render an HTML version of our manual converted from LaTeX.

Assuming you have nothing on your page except the wikiform process content shown in comment:description, most likely the req.args['auto'] is added by a plugin implementing IRequestFilter. Are you using TracBibPlugin or similar?

[Ryan J Ollos]

Looks like req.args['auto'] is added by tracbib.tracbib​. I haven't looked closely at what that plugin is trying to do, but adding content to req.args is probably not the best way.

[Ryan J Ollos]

In 16875:

WikiForms 0.3dev: Show traceback on exception

Refs #13300.

[Ryan J Ollos]

In 16876:

WikiForms 0.3dev: Remove unneeded code

Refs #13300.

[Ryan J Ollos]

Please test again with r16876. You should get a traceback in the browser and the log after r16875.

[ntmlod]

You're right, I use also TracBibPlugin for other stuff related to bibliography. I didn't mention it before it was not in the log (generated from 'wikiforms' search) so I thought it was rather linked to the doxygen integration plugin.

Thanks to your patch, i've got the traceback:

2017-10-14 20:01:22,822 Trac[wikiforms] WARNING: Failed processing request for /wikiforms/update:
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/wikiforms/wikiforms.py", line 202, in process_request
    value = re.sub('(\r\n|\r|\n)', '\\\\n', value)
  File "/usr/lib64/python2.7/re.py", line 151, in sub
    return _compile(pattern, flags).sub(repl, string, count)
TypeError: expected string or buffer

So finally is it a WikiFormsPlugin or TracBibPlugin issue ?

[Ryan J Ollos]

Looks like a WikiFormsPlugin issue. I haven't been able to reproduce.

Could you log the values in fields_to_be_stored​? Add above line 201:

self.log.debug(fields_to_be_stored)

[ntmlod]

Pretty much the same as before, I add the pattern of the value in the auto arg:

2017-10-15 17:16:14,232 Trac[wikiforms] DEBUG: {'/wiki/Admin/PageTemplates/test_form/1_comments': u'c', 'auto': [{u'bib_ref': {u'bib_field': u'value', u'bib_field': u'value', ...}, {u'bib_ref': {u'bib_field': u'value', u'bib_field': u'value', ...}, ... }], '/wiki/Admin/PageTemplates/test_form/1_compile': u'a', '/wiki/Admin/PageTemplates/test_form/1_report': u'b', '/wiki/Admin/PageTemplates/test_form/1_revision': u'1', '/wiki/Admin/PageTemplates/test_form/1_arch': u'select 1'}

[Ryan J Ollos]

Oh, I overlooked the obvious and suspect you were ahead of me on this one. The problem is due to auto containing a list. I'll look at a possible solutions.

[Ryan J Ollos]

In 16879:

WikiForms 0.4dev: Ignore arguments that are not string type

Fixes #13300.

[Ryan J Ollos]

Two things to consider in the future:

• Make WikiForms string arguments uniquely identifiable so that it doesn't conflict with other plugins.
• Look at TracBibPlugin to see if there is a better solution than inserting to req.args.

[Ryan J Ollos]

Replying to ntmlod:

Replying to Ryan J Ollos:

I noticed that deleting a wiki page does not delete the associated form data from the database. Do you think it should?

I would say yes.

For me and regarding how we use this plugin, the drawback is when you rename the wikipage. This plugin and the other version TracFormsPlugin lost the link to their entries.
Now that I'm getting used to manipulate the DB, I know how to fix that but, from a user perspective, it is a bit unintelligible.

=> #13303.