python3 pycurl

The magic port was meant as a warmup, the idea was porting pycurl.
It is done, and works for me.

You can grab the diff here or from pycurl patch tracker.

UPDATE New version, which workarounds the bug mentioned.

To use:

cvs -d:pserver:anonymous@pycurl.cvs.sourceforge.net:/cvsroot/pycurl login
cvs -z3 -d:pserver:anonymous@pycurl.cvs.sourceforge.net:/cvsroot/pycurl co -P pycurl
wget -O /dev/stdout http://p.carnivore.it/wtnYp3?download | patch -p0
/opt/dionaea/bin/python3 setup.py install

I ported the binding, the setup.py and 2 examples, the share interface is untested.

bugs

WRITEDATA is io

Due to missing PyFile_AsFile in python3, I decided to use

        int fd = PyObject_AsFileDescriptor(obj);
        fp = fdopen(fd, "w");

instead, unfortunately this breaks something, files are downloaded completely, but not written to disk completely. Possible workaround, specify WRITEFUNCTION in the binding, and write yourself.

example

/opt/dionaea/bin/python3 /tmp/pycurl/examples/retriever-multi.py /tmp/urls 5
PycURL libcurl/7.19.6 OpenSSL/0.9.8g zlib/1.2.3.3 libidn/1.15 (compiled against 0x71305)
----- Getting 12 URLs using 5 connections -----
Success: doc_005.dat libemu.carnivore.it http://libemu.carnivore.it
Success: doc_002.dat nepenthes.carnivore.it http://nepenthes.carnivore.it
Success: doc_007.dat nebula.carnivore.it http://nebula.carnivore.it
Success: doc_006.dat liblcfg.carnivore.it http://liblcfg.carnivore.it
Success: doc_008.dat honeytrap.carnivore.it http://honeytrap.carnivore.it
Success: doc_009.dat dionaea.carnivore.it http://dionaea.carnivore.it
Success: doc_010.dat p.carnivore.it http://p.carnivore.it
Success: doc_012.dat ftp.carnivore.it ftp://ftp.carnivore.it
Success: doc_011.dat regoogle.carnivore.it http://regoogle.carnivore.it
Success: doc_004.dat svn.carnivore.it http://svn.carnivore.it
Success: doc_001.dat carnivore.it http://carnivore.it
Success: doc_003.dat trac.carnivore.it http://carnivore.it/

The files are incomplete:

-rw-r--r-- 1 common common 98304 2009-12-19 21:13 doc_001.dat
-rw-r--r-- 1 common common  8192 2009-12-19 21:13 doc_002.dat
-rw-r--r-- 1 common common 98304 2009-12-19 21:13 doc_003.dat
-rw-r--r-- 1 common common  8192 2009-12-19 21:13 doc_004.dat
-rw-r--r-- 1 common common     0 2009-12-19 21:13 doc_005.dat
-rw-r--r-- 1 common common  8192 2009-12-19 21:13 doc_006.dat
-rw-r--r-- 1 common common  4096 2009-12-19 21:13 doc_007.dat
-rw-r--r-- 1 common common  4096 2009-12-19 21:13 doc_008.dat
-rw-r--r-- 1 common common 16384 2009-12-19 21:13 doc_009.dat
-rw-r--r-- 1 common common  4096 2009-12-19 21:13 doc_010.dat
-rw-r--r-- 1 common common  8192 2009-12-19 21:13 doc_011.dat
-rw-r--r-- 1 common common     0 2009-12-19 21:13 doc_012.dat

caveats

ob_type

access with Py_TYPE

PyInt

#if PY_MAJOR_VERSION >= 3
  #define PyInt_Type                   PyLong_Type
  #define PyInt_Check(op)              PyLong_Check(op)
  #define PyInt_FromLong               PyLong_FromLong
  #define PyInt_AsLong                 PyLong_AsLong
#endif

PyFile

PyFile_Check

    if (PyFile_Check(obj)) {

replace with

    extern PyTypeObject PyIOBase_Type;
    if(PyObject_IsInstance(obj, (PyObject *)&PyIOBase_Type) ) {

PyFile_AsFile

        fp = PyFile_AsFile(obj);

maybe?

        int fd = PyObject_AsFileDescriptor(obj);
        fp = fdopen(fd, "w");

Py_FindMethod

If used in get/setattr use get/setattro and PyObject_GenericSetAttr instead.

Comments

1

Hi there, I adapted your patch to the last pycurl sources. The correction is in the pycurl tracker: http://sourceforge.net/projects/pycurl/develop ID: 3188495 (Your blog does not allow comments with complex URLs…)

Cheers, Emmanuel

decitre
2011/02/21 20:54
2

@decitre: Hi, complex urls like: corrected python3 support?

Giving Christopher Warner credits is plain wrong, he did not write the patch, did not apply it, and he did not review it, basically he is the reason why you still have to patch pycurl yourself for python3, not really sure if thats worth credits.

Nevertheless, in the end you found the patch and can could make use of it.

Markus
2011/02/24 15:50
3

Hallo Markus. Sorry for the misunderstanding. Pretty embarrassing. I added an erratum to my comment at pycurl. zanee claimed there were no contact attached to your patch.

decitre
2011/02/26 10:29
4

Hi,

he could have used sourceforges internal message system to contact me, but it does not matter. Let's just wait and see if it gets merged this time.

Markus
2011/02/27 18:58


2009/12/19/python3_pycurl.txt · Last modified: 2010/06/15 11:46 by common
chimeric.de = chi`s home Creative Commons License Valid CSS Driven by DokuWiki do yourself a favour and use a real browser - get firefox!! Recent changes RSS feed Valid XHTML 1.0