Guido wrote:
>
> Brian,
>
> (Also in response to your bug report.) I'm a little worried that
> upping the limit to 1024 would cause some performance problems if
> you're making a lot of select() calls. The select allocates three
> arrays of length FD_SETSIZE+3; each array item is 12 bytes. This is a
> total allocation of more than 36K for a meager select() call!
> And all
> that memory also has to be cleared by the FD_ZERO() call.
>
> If you actually have that many sockets, that's worth paying for (the
> socket objects themselves use up just as much memory, and your Python
> data structures for the sockets, no matter how small, are probably
> several times bigger), but for a more typical program, I see
> this as a
> lot of overhead.
>
> Is there a way that this can be done more dynamically, e.g. by making
> the set size as big as needed on windows but no bigger?
>
> (Before you suggest allocating that memory statically, remember it's
> possible to call select from multiple threads. Allocating 36K of
> thread-local space for each thread also doesn't sound too pleasant.)
>
> --Guido van Rossum (home page: http://www.python.org/~guido/)
Hmm - after going through all of the Win32 sdks, it doesn't
appear to be possible to do it any other way than as a -D
option at compile time, so optimizing for the common case
(folks who _don't_ need large numbers of fds) is reasonable.
Since we distribute a python15.dll with Zope on windows, this
isn't that big a deal for us - we can just compile in a higher
limit in our distributed dll. I was mostly thinking of the win32
users who don't have the ability to rebuild their dll, but
maybe this isn't that much of a problem; I suspect that the
people who write significant socket apps that would run into
this problem probably have access to a compiler if they need it.
Brian Lloyd brian(a)digicool.com
Software Engineer 540.371.6909
Digital Creations http://www.digicool.com
> Is there some low limit on maximum number of sockets you can
> have in the
> Python-NT's select call? A program that happens to work
> perfectly on Linux
> seems to die on NT around 64(?) sockets to the 'too many file
> descriptors
> in call' error.
>
> Any portable ways to bypass it?
>
> -Markus
Hi Markus,
It turns out that NT has a default 64 fd limit on arguments to
select(). The good news is that you can actually bump the limit up
to whatever number you want by specifying a define when compiling
python15.dll.
If you have the ability to rebuild your python15.dll, you can add
the define:
FD_SETSIZE=1024
to the preprocessor options for the python15 project to raise the
limit to 1024 fds.
The default 64 fd limit is too low for anyone trying to run
an async server that handles even a modest load, so I've
submitted a bug report to python.org asking that the define
above find its way into the next python release...
Brian Lloyd brian(a)digicool.com
Software Engineer 540.371.6909
Digital Creations http://www.digicool.com
> [Jack seems to like an asynch IO model]
>
> > def foo():
> > obj = stdin.delayed_read()
> > obj2 = stdout.delayed_write("data")
> > do_lengthy_computation()
> > data = obj.get() # Here we wait for the read to complete
> > del obj2 # Here we wait for the write to
> > complete.
> >
> > This gives a fairly nice programming model.
>
> Indeed. Taking this a little further, I come up with something like:
>
> inlock = threading.Lock()
> buffer = stdin.delayed_read(inlock)
>
> outlock = threading.Lock()
> stdout.delayed_write(outlock, "The data")
>
> fired = threading.Wait(inlock, outlock) # new fn :-)
>
> if fired is inlock: # etc.
I think this is exactly what I _didn't_ want:-)
I'd like the delayed read to return an object that will automatically wait
when I try to get the data from it, and the delayed write object to
automatically wait when I garbage-collect it.
Of course, there's no reason why you couldn't also wait on these objects (or,
on unix, pass them to select(), or whatever).
On second thought the method of the delayed read should be called read() in
stead of get(), of course.
--
Jack Jansen | ++++ stop the execution of Mumia Abu-Jamal ++++
Jack.Jansen(a)oratrix.com | ++++ if you agree copy these lines to your sig ++++
www.oratrix.nl/~jack | see http://www.xs4all.nl/~tank/spg-l/sigaction.htm
8.2b1 is released:
Some surprising news: they now use cygwin tools to do the windows build.
Not surprising news: they still haven't incorporated some bug fixes I
submitted eons ago =)
http://www.scriptics.com/software/relnotes/tcl8.2b1
--david
> What Im getting at is that a Python IO model should maybe go a little
> further than "tradtional" IO - asynchronous IO and synchronisation
> capabilities should also be specified. Of course, these would be optional,
> but it would be excellent if a platform could easily slot into pre-defined
> Python semantics if possible.
What Python could do with reasonable ease is a sort of "promise" model, where
an I/O operation returns an object that waits for the I/O to complete upon
access or destruction.
Something like
def foo():
obj = stdin.delayed_read()
obj2 = stdout.delayed_write("data")
do_lengthy_computation()
data = obj.get() # Here we wait for the read to complete
del obj2 # Here we wait for the write to complete.
This gives a fairly nice programming model.
--
Jack Jansen | ++++ stop the execution of Mumia Abu-Jamal ++++
Jack.Jansen(a)oratrix.com | ++++ if you agree copy these lines to your sig ++++
www.oratrix.nl/~jack | see http://www.xs4all.nl/~tank/spg-l/sigaction.htm
Howdy,
my modules are nearly ready. I will be out of my office
for two weeks, but had no time to finalize and publish yet.
Stackless Python has reached what I wanted it to reach:
A continuation can be saved at every opcode.
The continuationmodule has been shrunk heavily.
Some extension is still needed, continuations
are still frames, but they can be picked like
Sam wanted it originally.
Sam, I'm pretty sure this is more than enough for
coroutines. Just have a look at getpcc(), this is
now very easy. All involved frames are armed so that
they *can* save themselves, but will do so just if
necessary. The cheapest solution I could think of,
no other optimization is necessary. If your coroutine
functions like to swap two frames, and if they manage to
do so that the refcount of the target stays at one,
no extra frame will be generated.
That's it, really.
If someone wants to play, get the stackless module, replace
ceval.c, and build continuationmodule.c as a dll or whatever.
testct.py contains a lot of crap. The first implementation
of class coroutine is working right.
The second one is wrong by concept.
later - chris
ftp://ftp.pns.cc/pub/veryfar.zip
--
Christian Tismer :^) <mailto:tismer@appliedbiometrics.com>
Applied Biometrics GmbH : Have a break! Take a ride on Python's
Kaiserin-Augusta-Allee 101 : *Starship* http://starship.python.net
10553 Berlin : PGP key -> http://wwwkeys.pgp.net
PGP Fingerprint E182 71C7 1A9D 66E9 9D15 D3CC D4D7 93E2 1FAE F6DF
we're tired of banana software - shipped green, ripens at home
I like the list pop method because it provides a way to use
lists as thread safe queues and stacks (since append and pop
are protected by the global interpreter lock).
With pop, you can essentially test whether the list is
empty and get a value if it isn't in one atomic operation:
try:
foo=queue.pop(0)
except IndexError:
... empty queue case
else:
... non-empty case, do something with foo
Unfortunately, this incurs exception overhead. I'd rather do
something like:
foo=queue.pop(0,marker)
if foo is marker:
... empty queue case
else:
... non-empty case, do something with foo
I'd be happy to provide a patch.
Jim
--
Jim Fulton mailto:jim@digicool.com Python Powered!
Technical Director (888) 344-4332 http://www.python.org
Digital Creations http://www.digicool.comhttp://www.zope.org
Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email
address may not be added to any commercial mail list with out my
permission. Violation of my privacy with advertising or SPAM will
result in a suit for a MINIMUM of $500 damages/incident, $1500 for
repeats.
> [Tim, notes that Perl line-at-a-time text mode input runs 3x
> faster than
> Python's on his platform]
>
> And much to my surprise, it turns out Perl reads lines a
> character at a time
> too! And they do not reimplement stdio. But they do cheat.
>
> [some notes on the cheating and PerlIO api snipped]
>
> The usual *implementation* of these guys is as straight macro
> substitution
> to the corresponding C stdio call. It's possible to
> implement them some
> other way, but I don't see anything in the source that
> suggests anyone has
> done so, except possibly to build it all on AT&T's SFIO lib.
Hmm - speed bonuses not withstanding, an implementation of
such a beast in the Python sources would've helped a lot to
reduce the ugly hairy gymnastics required to get Python going
on Win CE, where (until very recently) there was no concept
of most of the things you expect to find in stdio...
Brian Lloyd brian(a)digicool.com
Software Engineer 540.371.6909
Digital Creations http://www.digicool.com
Howdy,
please find attached my latest running version of
continuationmodule.c which is really able to do continuations.
You need stackless Python 0.3 for it, which I just submitted.
This module is by no means ready.
The central functions are getpcc() and putcc.
Call/cc is at the moment to be done like:
def callcc(fun, *args, **kw):
cont = getpcc()
return apply(fun, (cont,)+args, kw)
getpcc(level=1) gets a parent's current continuation.
putcc(cont, val) throws a continuation.
At the moment, these are still frames (albeit special ones)
which I will change. They should be turned into objects which
have a link to the actual frame, which can be unlinked after
a shot or by hand. This makes it easier to clean up circular
references.
I have a rough implementation of this in Python, also a couple of
generators and coroutines, but all not pleasing me yet.
Due to the fact that my son is ill, my energy has dropped
a little for the moment, so I thought I'd better release
something now. I will make the module public when things
have been settled a little more.
ciao - chris
--
Christian Tismer :^) <mailto:tismer@appliedbiometrics.com>
Applied Biometrics GmbH : Have a break! Take a ride on Python's
Kaiserin-Augusta-Allee 101 : *Starship* http://starship.python.net
10553 Berlin : PGP key -> http://wwwkeys.pgp.net
PGP Fingerprint E182 71C7 1A9D 66E9 9D15 D3CC D4D7 93E2 1FAE F6DF
we're tired of banana software - shipped green, ripens at home
I'm currently testing a pure Python version of mxDateTime
(my date/time package), which uses a contributed Python version
of the C extension.
Now, to avoid problems with pickled DateTime objects (they include
the complete module name), I would like to name *both* the
Python and the C extension version mxDateTime. With the current
lookup scheme (shared mods are searched before Python modules)
this is no problem since the shared mod is found before the
Python version and used instead, so getting this working is
rather simple.
The question is: will this setup remain a feature in future
versions of Python ? (Does it work this way on all platforms ?)
Cheers,
--
Marc-Andre Lemburg
______________________________________________________________________
Y2000: 162 days left
Business: http://www.lemburg.com/
Python Pages: http://www.lemburg.com/python/
