python3 pycurl

The magic port was meant as a warmup, the idea was porting pycurl.
It is done, and works for me.

You can grab the diff here or from pycurl patch tracker.

UPDATE New version, which workarounds the bug mentioned.

To use:

cvs -d:pserver:anonymous@pycurl.cvs.sourceforge.net:/cvsroot/pycurl login
cvs -z3 -d:pserver:anonymous@pycurl.cvs.sourceforge.net:/cvsroot/pycurl co -P pycurl
wget -O /dev/stdout http://p.carnivore.it/wtnYp3?download | patch -p0
/opt/dionaea/bin/python3 setup.py install

I ported the binding, the setup.py and 2 examples, the share interface is untested.

bugs

WRITEDATA is io

Due to missing PyFile_AsFile in python3, I decided to use

        int fd = PyObject_AsFileDescriptor(obj);
        fp = fdopen(fd, "w");

instead, unfortunately this breaks something, files are downloaded completely, but not written to disk completely. Possible workaround, specify WRITEFUNCTION in the binding, and write yourself.

example

/opt/dionaea/bin/python3 /tmp/pycurl/examples/retriever-multi.py /tmp/urls 5
PycURL libcurl/7.19.6 OpenSSL/0.9.8g zlib/1.2.3.3 libidn/1.15 (compiled against 0x71305)
----- Getting 12 URLs using 5 connections -----
Success: doc_005.dat libemu.carnivore.it http://libemu.carnivore.it
Success: doc_002.dat nepenthes.carnivore.it http://nepenthes.carnivore.it
Success: doc_007.dat nebula.carnivore.it http://nebula.carnivore.it
Success: doc_006.dat liblcfg.carnivore.it http://liblcfg.carnivore.it
Success: doc_008.dat honeytrap.carnivore.it http://honeytrap.carnivore.it
Success: doc_009.dat dionaea.carnivore.it http://dionaea.carnivore.it
Success: doc_010.dat p.carnivore.it http://p.carnivore.it
Success: doc_012.dat ftp.carnivore.it ftp://ftp.carnivore.it
Success: doc_011.dat regoogle.carnivore.it http://regoogle.carnivore.it
Success: doc_004.dat svn.carnivore.it http://svn.carnivore.it
Success: doc_001.dat carnivore.it http://carnivore.it
Success: doc_003.dat trac.carnivore.it http://carnivore.it/

The files are incomplete:

-rw-r--r-- 1 common common 98304 2009-12-19 21:13 doc_001.dat
-rw-r--r-- 1 common common  8192 2009-12-19 21:13 doc_002.dat
-rw-r--r-- 1 common common 98304 2009-12-19 21:13 doc_003.dat
-rw-r--r-- 1 common common  8192 2009-12-19 21:13 doc_004.dat
-rw-r--r-- 1 common common     0 2009-12-19 21:13 doc_005.dat
-rw-r--r-- 1 common common  8192 2009-12-19 21:13 doc_006.dat
-rw-r--r-- 1 common common  4096 2009-12-19 21:13 doc_007.dat
-rw-r--r-- 1 common common  4096 2009-12-19 21:13 doc_008.dat
-rw-r--r-- 1 common common 16384 2009-12-19 21:13 doc_009.dat
-rw-r--r-- 1 common common  4096 2009-12-19 21:13 doc_010.dat
-rw-r--r-- 1 common common  8192 2009-12-19 21:13 doc_011.dat
-rw-r--r-- 1 common common     0 2009-12-19 21:13 doc_012.dat

caveats

ob_type

access with Py_TYPE

PyInt

#if PY_MAJOR_VERSION >= 3
  #define PyInt_Type                   PyLong_Type
  #define PyInt_Check(op)              PyLong_Check(op)
  #define PyInt_FromLong               PyLong_FromLong
  #define PyInt_AsLong                 PyLong_AsLong
#endif

PyFile

PyFile_Check

    if (PyFile_Check(obj)) {

replace with

    extern PyTypeObject PyIOBase_Type;
    if(PyObject_IsInstance(obj, (PyObject *)&PyIOBase_Type) ) {

PyFile_AsFile

        fp = PyFile_AsFile(obj);

maybe?

        int fd = PyObject_AsFileDescriptor(obj);
        fp = fdopen(fd, "w");

Py_FindMethod

If used in get/setattr use get/setattro and PyObject_GenericSetAttr instead.

Comments



2009/12/19/python3_pycurl.txt · Last modified: 2010/06/15 11:46 by common
chimeric.de = chi`s home Creative Commons License Valid CSS Driven by DokuWiki do yourself a favour and use a real browser - get firefox!! Recent changes RSS feed Valid XHTML 1.0