[FE-discuss] FormEncode wishlist

[Mike Orr]

At PyCon Ian mentioned he's looking for a maintainer for FormEncode.
I don't have time to do this right now but I may in a few months. In
the meantime I'd like to start a discussion on what direction
FormEncode should go.

- The manual is incomplete, both in terms of how to use FormEncode
under a web framework, and many validators are undocumented. Ian and
I made a list of things to include:
http://wiki.pylonshq.com/display/pylonscookbook/The+other+FormEncode+manual+(UNFINISHED)

- Are there any features missing in the validators?

- Are there any features missing in htmlfill?

- Are there unused dead ends we can delete? Some of the anemic
validators like Constant, MaxLength, MinLength, Empty, and NotEmpty
aren't really useful alone because the user would typically want to do
more than just these. not_empty is now an attribute of
FancyValidator, for instance. MinLength, MaxLength, and Regex could
likewise be attributes of FancyValidator (with None to skip those
tests).

- OneOf, DictConverter, ListConverter, and Set are confusing because
they use Python data types differently than expected. For instance,
OneOf is what you normally do with a list (make sure the entered value
is in the list). And Set you would think is for working with sets,
but it isn't, it's a multiple-value control (akin to a MultiDict
value). I would somehow group the validators between the universal
ones everybody should know vs the specialized ones.

- Are there any API changes that would make it more useful or convenient?

- What about wrapper functions, along the lines of Pylons' @validate.
Are there any that would make FormEncode more useful in web
applications?

- FormEncode and htmlfill could be merged into WebHelpers since they
are often used together. That would also solve the problem of "Why is
FormEncode called FormEncode when it does validation rather than
encoding?" and "Why is htmlfill part of FormEncode when it doesn't do
validation?" I'm not sure if the ToscaWidgets developers would like
this though since I think they use the validators alone.

- The validators are also useful outside forms, for instance to
validate configuration files. But the error messages are not quite
ideal for that since they don't show the filename and line number of
the error. Maybe this could be added somehow.

[Ian Bicking]

These are all just ideas... I don't have the, I dunno... willpower(?) to
make these happen. But if a maintainer were to emerge, these are
suggestions about things I think would be useful.

Post by Mike Orr
At PyCon Ian mentioned he's looking for a maintainer for FormEncode.
I don't have time to do this right now but I may in a few months. In
the meantime I'd like to start a discussion on what direction
FormEncode should go.
- The manual is incomplete, both in terms of how to use FormEncode
under a web framework, and many validators are undocumented. Ian and
http://wiki.pylonshq.com/display/pylonscookbook/The+other+FormEncode+manual+(UNFINISHED)<http://wiki.pylonshq.com/display/pylonscookbook/The+other+FormEncode+manual+%28UNFINISHED%29>
- Are there any features missing in the validators?

I'd like if the state object was better documented and exposed, and didn't
default to None quite like it does now. There's some more complicated
validations that would be easier if people had a useful state object and
knew how to use it (e.g., you can validate off another field).

Also maybe some decorators to create validators from simpler functions.

Post by Mike Orr
- Are there any features missing in htmlfill?
- Are there unused dead ends we can delete? Some of the anemic
validators like Constant, MaxLength, MinLength, Empty, and NotEmpty
aren't really useful alone because the user would typically want to do
more than just these. not_empty is now an attribute of
FancyValidator, for instance. MinLength, MaxLength, and Regex could
likewise be attributes of FancyValidator (with None to skip those
tests).

I don't think Regex can be eliminated.

Post by Mike Orr
- OneOf, DictConverter, ListConverter, and Set are confusing because
they use Python data types differently than expected. For instance,
OneOf is what you normally do with a list (make sure the entered value
is in the list). And Set you would think is for working with sets,
but it isn't, it's a multiple-value control (akin to a MultiDict
value). I would somehow group the validators between the universal
ones everybody should know vs the specialized ones.
- Are there any API changes that would make it more useful or convenient?

Another general change I've thought about is if the to_python method were
__call__, and from_python was an optional method. Then functions could act
as validators. Also a simpler way to raise exceptions (not as easy to apply
i18n to or otherwise customize, but mostly for custom validators). E.g.,
just allow ValueError. Adding a to_python function would help make this
possible. Or a decorator.

Post by Mike Orr
Are there any that would make FormEncode more useful in web
applications?

Maybe a simpler way to handle the logic. Maybe just as simple as:

values, errors = convert(schema, values)

Something so people can avoid @validate if they don't need it would be
helpful, as it's a lot easier to think in terms of simple logic and if
statements than the decorator.

Post by Mike Orr
- FormEncode and htmlfill could be merged into WebHelpers since they
are often used together. That would also solve the problem of "Why is
FormEncode called FormEncode when it does validation rather than
encoding?"

It actually does translation, hence the bidirectional nature of all
validators.

Post by Mike Orr
and "Why is htmlfill part of FormEncode when it doesn't do
validation?" I'm not sure if the ToscaWidgets developers would like
this though since I think they use the validators alone.

htmlfill isn't particularly large, so simply the fact both pieces of source
are distributed together doesn't seem concerning. It's pretty clear they
aren't bound to each other.

Post by Mike Orr
- The validators are also useful outside forms, for instance to
validate configuration files. But the error messages are not quite
ideal for that since they don't show the filename and line number of
the error. Maybe this could be added somehow.

After going about this a few ways, I think the best way for a configuration
file to work is to produce simple values but give a way to get the line
number. E.g., parsed_config_file.position({'section': 'key'}) would return
('config.ini', 21). So wrapping it together would involve a way of taking
the Invalid exception (which has the necessary information) and formatting
that using knowledge of the source config file.

INITools has a ConfigParser-compatible interface that also has methods to
get source location, so some glue between the two could make this possible.

[Mike Orr]

Post by Ian Bicking
These are all just ideas... I don't have the, I dunno... willpower(?) to
make these happen.  But if a maintainer were to emerge, these are
suggestions about things I think would be useful.

The important thing is to document what we want, and to have an
explicit consensus among the motivated FormEncode users (i.e., the
subscribers of this list).

Post by Ian Bicking
I'd like if the state object was better documented and exposed, and didn't
default to None quite like it does now.  There's some more complicated
validations that would be easier if people had a useful state object and
knew how to use it (e.g., you can validate off another field).

What should the state object be? I thought that was completely up to
the application.

The problem with @validate is it doesn't have access to
request-specific information such as a database record. That's what
you normally want in a state object. However, it's a chicken and egg
thing because you have to validate the ID and convert it to an integer
in order to do the lookup, in order to complete the validation. Some
Pylons users cheat by using 'c' variables from the request path in
their validators. Although the counterargument is that @validate is
just wrong-headed in itself, because you want to do validation in the
middle of the action rather than before.

What kind of default state object would be useful?

Post by Ian Bicking

Post by Mike Orr
- Are there any API changes that would make it more useful or convenient?

Another general change I've thought about is if the to_python method were
__call__, and from_python was an optional method.  Then functions could act
as validators.  Also a simpler way to raise exceptions (not as easy to apply
i18n to or otherwise customize, but mostly for custom validators).  E.g.,
just allow ValueError.  Adding a to_python function would help make this
possible.  Or a decorator.

Most of my usage is one-way, I think.

Are there other existing validation packages/APIs we should consider?

[Ian Bicking]

Post by Ian Bicking

Post by Ian Bicking
I'd like if the state object was better documented and exposed, and

didn't

Post by Ian Bicking
default to None quite like it does now. There's some more complicated
validations that would be easier if people had a useful state object and
knew how to use it (e.g., you can validate off another field).

What should the state object be? I thought that was completely up to
the application.
request-specific information such as a database record. That's what
you normally want in a state object. However, it's a chicken and egg
thing because you have to validate the ID and convert it to an integer
in order to do the lookup, in order to complete the validation. Some
Pylons users cheat by using 'c' variables from the request path in
just wrong-headed in itself, because you want to do validation in the
middle of the action rather than before.
What kind of default state object would be useful?

Anything but None would be more useful. object() would be okay. Things
like Schema set attributes on the state object if they are able to.

A slightly fancier state might be appropriate. Maybe one that carries a _
method (for i18n) or some other simple stuff. But it can be very simple.

Post by Ian Bicking

Post by Ian Bicking
- Are there any API changes that would make it more useful or

convenient?

Post by Ian Bicking
Another general change I've thought about is if the to_python method were
__call__, and from_python was an optional method. Then functions could

act

Post by Ian Bicking
as validators. Also a simpler way to raise exceptions (not as easy to

apply

Post by Ian Bicking
i18n to or otherwise customize, but mostly for custom validators). E.g.,
just allow ValueError. Adding a to_python function would help make this
possible. Or a decorator.

Most of my usage is one-way, I think.
Are there other existing validation packages/APIs we should consider?

The __call__ idea comes from, I think, ConfigObj, which uses something like
that for validation. I haven't read other systems very closely, so I don't
know if there's good ideas to be taken from them.

[Mike Orr]

Post by Mike Orr
What kind of default state object would be useful?

Anything but None would be more useful.  object() would be okay.  Things
like Schema set attributes on the state object if they are able to.

``object`` can't take arbitrary attributes so it would have to be
something else. We could just define an empty class with
generally-useful things like '._'.

However I do like ``None`` because then it's obvious there's no state.
Otherwise you get into, "What kind of state object is this? Is it
compatible with me?"

[Marius Gedminas]

Post by Mike Orr
At PyCon Ian mentioned he's looking for a maintainer for FormEncode.
I don't have time to do this right now but I may in a few months. In
the meantime I'd like to start a discussion on what direction
FormEncode should go.

* Fix the bugs reported in the sourceforge bug tracker.

* Improve the documentation. Some of the things with FormEncode are
confusing:

e.g. Bool() versus StringBool() --- one is for submit buttons, the
other is for checkboxes/etc. Maybe. I'm not sure :)

e.g. String() versus UnicodeString() [sp?] --- you might end up
doing str <-> unicode conversion twice if both your web framework
(Pylons) and FormEncode try to do that.

* Move to a saner bug tracker. (Sourceforge makes me go yuck,
especially when it forgets to HTML-escape my bug titles correctly in
some of its "recent changes" boxes, or when it randomly inserts
backslashes in those titles---trying to blindly avoid SQL injection
attacks?)

Marius Gedminas

[Ian Bicking]

Post by Marius Gedminas
* Move to a saner bug tracker. (Sourceforge makes me go yuck,
especially when it forgets to HTML-escape my bug titles correctly in
some of its "recent changes" boxes, or when it randomly inserts
backslashes in those titles---trying to blindly avoid SQL injection
attacks?)

I plan to move FormEncode to bitbucket soon, along several other projects of
mine. I might move to the bitbucket issue tracker, or perhaps another issue
tracker (bitbucket's being the easy/obvious choice).

[Iain Duncan]

Post by Mike Orr
- FormEncode and htmlfill could be merged into WebHelpers since they
are often used together. That would also solve the problem of "Why is
FormEncode called FormEncode when it does validation rather than
encoding?" and "Why is htmlfill part of FormEncode when it doesn't do
validation?" I'm not sure if the ToscaWidgets developers would like
this though since I think they use the validators alone.

FWIW, I've used formencode to success in wxpython apps too, so I would
like to see it standalone from any web-specific packages.

Great news that we will see new activity in formencode!

Iain