We hit the Speed limit of python (Peer closed the socket connection)

Python is a beautiful language but we know the limitations of the python compared with compiled language. So here I'm explaining a scenario where python was not the ideal choice. We are using python for network programming, specifically a TCP client/server program which read packets from another C based server. We picked python because on the requirements the traffic volume wasn't huge enough to consider compiled languages like C/C++ or Java, and we are already using python so more comfortable with it. 

The Network Topology 

The data flowing from Server A to Server B via TCP. The Server B is implemented in python for now. This setup was running for more than a year without any problems or crashing. Then it started showing problems. The Server B crashing sometime because of the socket connection between the Server A and Server B getting closed by the Server A, We don't know why. Bellow listed are the errors that we are getting, one interesting thing is; 
these two errors won't happen together, some time Errno 104, some time Errno 107. 

1. Errno: 104 - Peer closed the socket connection. 
2. Errno: 107 - Transport endpoint closed. 

I initially thought this was happening because of python's long running process exiting due to memory overflow or some other reasons. We thought this because python's reputation as a long running daemon is not great. Then we figure out that long running daemon wasn't the reason for this issue, since the main python process wasn't consuming more RAM and it's very stable in memory usage. Our python server using very simple data structures, so python VM very efficiently handling the Memory. Then I started to look into other areas of program, by changing the settings and tuning parameters to see any of it fixes the issue. All of the options seems to work for a while but it throws the same error after few days. I was stuck and no idea about the issue and spend around a week to go through the program's in and out to see any possible causes. But no clue from it. Finally Stack Overflow came into help, IRC wasn't helpful for these type of very specific issues. Programmers can't live without Google and Stack Overflow. I put the question on the SO, people helped me to make the question better by adding the TCP traces picked up using tcpdump. With in hours I got the answer I was looking for. It's mainly due the python socket reading related. In the Python side we are not clearing the TCP packets from the receive buffer of TCP socket completely, only reading few bytes (300B to 10KB) at a time. So which causing the Python server to send receive window size flag of TCP protocol as ZERO for multiple times In the heavy traffic period. This might triggering this issue. That was the idea I got from the stack Overflow. The problem on the python side is that, it's a single threaded server which reading the data from the socket, one thing we are right now not doing is reading until socket is empty due to some implementation restrictions. From the SO answer I got some quick solutions to try, Remove unwanted codes from the main loop, so the total time taken to process one socket read would be as small as possible. Tweak with the Kernel parameters for the TCP sending and receiving buffer size. Improve the Code base or re-implement it in C completely or do the main part as C extension. The method 1 and 2 was easy go ahead with, So I did it right away. It gives considerable difference and the crashes reduced considerably. But still it happened once or twice. So I know that it's not enough and need reimplementation of the parts which causing this problem. But we are holding on the reimplementation since it will take lot of time to change the existing stuff. Any way the current changes give enough time to try more things to improve the existing code and investigate the problem more. Mean while I did some more investigation locally to understand the issue more. From that what I understood was, when the network is heavily loaded the Python server side not clearing the traffic quick enough and which forcing the remote server to close the TCP socket (Not yet found why it's closing it). Right after the socket is closed on the remote end, Python side of the code might be trying socket.send or socket.recv and it will fail since no active socket. So in the beginning I mentioned that I'm getting two errors, this is because some time right after the remote server closed the socket, the next socket interaction in the python side may be socket.send which was triggering the error Errno 104. If it was socket.recv it would have been Errno 107. One other thing I learned after this was that, the TCP server usually handles the slow remote servers by blocking the new massage write into the sending buffer. In my environment I understood the remote server A is not blocking the write(Non-blocking socket) instead closing the socket( This could be due to timeout).If the TCP server has option to cache the extra packets into disk then it could have handled this peak traffic period. 

A TCP Server has to provide atleast these features 

Way to handle the buffer overflow if the client is very slow. Write the buffer overflowed data into durable queue or file system. When the Client comes back, the server has to automatically deliver the pending messages in FIFO order. 

Final Thoughts 

If these options are there with the TCP server that I was handling, this problem couldn't have happened. So finally now I can say that Python could have handled this situation if the traffic spikes some times only - In our case that was the situation. Python couldn't have done it if the rate of server is more than the rate at which Python can clear the socket backlog. 
Reference by: http://haridas.in/what-happens-when-we-hit-the-speed-limit-of-python.html

Read more...

Cython - Making Python as Fast as C

by Mosky

Read more...

Compile Python Code

hohoho,.......

This should have been done a long time to execute if the python in the compiled file, to make the file more quickly if the execution. Python source code is automatically compiled into Python byte code by the CPython interpreter. Compiled code is usually stored in PYC (or PYO) files, and is regenerated when the source is updated, or when otherwise necessary.

Python automatically compiles Python source code when you import a module, so the easiest way to create a PYC file is to import it. If you have a module sampleModule.py, just do:
>>> import sampleModule

to create a sampleModule.pyc file in the same directory.

To do this programmatically, and without executing the code, you can use the py_compile module:
>>>import py_compile

>>>py_compile.compile("fileModule.py")

There’s also a compileall module which can be used to compile all modules in an entire directory tree.
>>>import compileall

>>>compileall.compile_dir("folderDestination", force=1)

Good luck, :)

Read more...

Overview of PyPy

This may be foreign to you, but this is a fact. PyPy is a Python programming language interpreter written in Python and is equipped with a JIT compiler (just in time). PyPy development focuses on the speed of program performance, efficiency, and maintain compatibility with CPython interpreter. Using Python in PyPy development, the developers PyPy allows hacking of its implementation and identify any areas that require improvement. PyPy fact currently implemented as a high level language makes it more flexible and easier in the experiment when compared to CPython, and also allows developers to experiment on some form of implementation of certain specific features.

PyPy translation model also provides a general framework for developing and supporting forms to create a dynamic programming language, providing a clear separation between the specification of a programming language with aspects and forms of implementation. In addition, the PyPy provides a programming language Python that is compatible with CPython with the flexibility and level of performance is better.

whether it can be concluded that the programming language compilers much faster?

Read more...

Entity on Programming

Entity is an object whose existence can be distinguished against other objects. Entities can be people, objects, places, events, concepts. Example :

• People: STUDENTS, LECTURERS, SUPPLIER, VENDOR
• Objects: CAR, ENGINE, ROOM
• Venue: THE STATE, THE VILLAGE, KAMPONG
• Genesis: SALES, REGISTER
• Concept: ACCOUNT, COURSE

An entity has a number of attributes

Example: The student has the name and address

Entity set is aa set of entities that share the same attributes

In programming OOP (Object Oriented Programming) or object-oriented Programming is a new way of thinking and logic in dealing with the problems that will try to overcome with the help of computers. OOP, unlike its predecessor (Structured Programming), trying to see problems through observation of the real world where every object is a single entity that has a combination of data structures and specific functions. This contrasts with structured programming where data structures and functions are defined separately and are not closely related.
it can be explained in the following code fragment:

in Python

in Java




hopefully useful for those who love OOP

Read more...

Connection Python to SQLite

SQLite is a in-process library that implements a self-contained, serverless, zero-configuration, transactional SQL database engine. The code for SQLite is in the public domain and is thus free for use for any purpose, commercial or private. SQLite is currently found in more applications than we can count, including several high-profile projects.

SQLite is an embedded SQL database engine. Unlike most other SQL databases, SQLite does not have a separate server process. SQLite reads and writes directly to ordinary disk files. A complete SQL database with multiple tables, indices, triggers, and views, is contained in a single disk file. The database file format is cross-platform - you can freely copy a database between 32-bit and 64-bit systems or between big-endian and little-endian architectures.

The SQLite library is a light-weight embedded SQL engine, with a nice DB-API compliant Python binding, originally developed by Michael Owens.

A newer version, called sqlite3, was added to Python’s standard library in Python 2.5 to up.

Read more...

Pythagorean Theorem

If you are on school and you are taking algebra like me or just want to find the missing length of one side of the triangle than this program should help you out. Pythagorean Theorem is great and extremely essay to use.
We can find the side of a triangle using the Pythagorean algorithm. This uses the Python programming language.

def menu():
      #print the options you have
      print " "
      print "Welcome to Pythagorean Theorem"
      print "Please keep in mind that for this program to work you need the lenght of two side of the triangle"
      print " "
      print "Your Options Are:"
      print " "
      print "1) If you have the lenght of A and B"
      print " "
      print "2) If you have the lenght of C and A"
      print " "
      print "3) If you have the lenght of C and B"
      print " "
      print "4) Quit Pythagorean Theorem"
      print " "
      return input ("Choose your option: ")

from math import *
      #On this one we have the lenght of A and B and we are trying to find the lenght of C
def AnB(a,b):
      print a**2, "+", b**2, "=", a**2 + b**2
      print sqrt(a**2 + b**2)
      #On this one we have the lenght of A and C and we are trying to find the lenght of B
def CnA(c,a):
      print c**2, "-", a**2, "=", c**2 - a**2
      print sqrt(c**2 - a**2)
      #On this one we have the lenght of B and C and we are trying to find the lenght of A
def CnB(c,b):
      print c**2, "-", b**2, "=", c**2 - b**2
      print sqrt(c**2 - b**2)
      #Code is Run
loop = 1
choice = 0
while loop == 1:
      choice = menu()
      if choice == 1:
            AnB(input("A: "),input("B: "))
      elif choice == 2:
            CnA(input("C: "),input("A: "))
      elif choice == 3:
            CnB(input("C: "),input("B: "))
      elif choice == 4:
            loop = 0
print "Thankyou fo using Pythagorean Theorem"

Honest Face

Read more...

Class Inheritance

Use __class__, __bases__ and __dict__ for sub and super class

inheritance learn python programming language, Python is very simple. Now using __class__, __bases__ and __dict__ for sub and super class. Try making a super class and child class. Examples like this:

class super:
         def hello(self):
               self.data1 = 'spam'

class sub(super):
         def hola(self):
                self.data2 = 'eggs'

X = sub()
X.__dict__
{}

print X.__class__

print sub.__bases__

print super.__bases__


Y = sub()

X.hello()
print X.__dict__

X.hola()
print X.__dict__

print sub.__dict__

print super.__dict__

print sub.__dict__.keys(), super.__dict__.keys()

print Y.__dict__

print X.data1, X.__dict__['data1']

X.data3 = 'toast'
print X.__dict__

X.__dict__['data3'] = 'ham'
print X.data3

print X.__dict__
print X.__dict__.keys()

print dir(X)
print dir(sub)
print dir(super)

Result

Read more...

A simple Login with Dictionary (Python)

Just a Simple login that uses Dictionary. Don't know if it work on a console. Plus i can't figure out a way to add something to the dictionary while the program is running.

database={'username': '1234', 'username2': '5678', 'username3': '9012'}
name = raw_input('Enter username: ')
ask = raw_input('Enter pin: ')
if ask in database[name]:
  print 'Welcome', name
else:
  print 'Invalid code'

you can try it's
Neoriz

Read more...

Using Connection cx_Oracle to Oracle in Python

For Connection python to Oracle use cx_Oracle. cx Oracle is a Python extension module thats allows access to Oracle Databases and conforms to the Python Database API specification. This module is currently built against Oracle 9.2.0, 10.2.0 and 11.1.0.

The interface specification consists of several sections:

• Module Interface
• Connection Objects
• Cursor Object
• Type Object and Constructors
• Implementattion Hints for Module Authors
• Optional DB API Extensions
• Optional Error Handling Extensions
• Optional Two-Phase Commit Extensions
• Frequently Asked Questions
• Major Changes from version 1.0 to Version 2.0
• Open Issues
• Footnotes
• Unknowledgements

This information can look here.
for Excample use cx_Oracle :

import sys, cx_Oracle
connect=cx_Oracle.Connection("system/password@Server_database")
cursor=connect.cursor()
cursor.execute("PL/SQL or Query language")
data=cursor.fetchall()
data in method list, but can for print such as looping
for (list fields) in data:
print list fields

good luck,.......
Read more...

Python is powerful, simple and fast

You can exercise python,...
first you can text mode in window/Python Interactive Shell or PythonW in Edior, you write :

print "Hello word Neoriz"
result >> Hello word Neoriz

Using the Python Interpreter

Invoking the Interpreter

The Python interpreter is usually installed as /usr/local/bin/python on those machines where it is available; putting /usr/local/bin in your Unix shell's search path makes it possible to start it by typing the command

python

to the shell. Since the choice of the directory where the interpreter lives is an installation option, other places are possible; check with your local Python guru or system administrator. (E.g., /usr/local/python is a popular alternative location.)

On Windows machines, the Python installation is usually placed in C:\Python25, though you can change this when you're running the installer. To add this directory to your path, you can type the following command into the command prompt in a DOS box:

set path=%path%;C:\python25

Typing an end-of-file character (Control-D on Unix, Control-Z on Windows) at the primary prompt causes the interpreter to exit with a zero exit status. If that doesn't work, you can exit the interpreter by typing the following commands: "import sys; sys.exit()".

The interpreter's line-editing features usually aren't very sophisticated. On Unix, whoever installed the interpreter may have enabled support for the GNU readline library, which adds more elaborate interactive editing and history features. Perhaps the quickest check to see whether command line editing is supported is typing Control-P to the first Python prompt you get. If it beeps, you have command line editing; see Appendix A for an introduction to the keys. If nothing appears to happen, or if P is echoed, command line editing isn't available; you'll only be able to use backspace to remove characters from the current line.

The interpreter operates somewhat like the Unix shell: when called with standard input connected to a tty device, it reads and executes commands interactively; when called with a file name argument or with a file as standard input, it reads and executes a script from that file.

A second way of starting the interpreter is "python -c command [arg] ...", which executes the statement(s) in command, analogous to the shell's -c option. Since Python statements often contain spaces or other characters that are special to the shell, it is best to quote command in its entirety with double quotes.

Some Python modules are also useful as scripts. These can be invoked using "python -m module [arg] ...", which executes the source file for module as if you had spelled out its full name on the command line.

Note that there is a difference between "python file" and "python ". In the latter case, input requests from the program, such as calls to input() and raw_input(), are satisfied from file. Since this file has already been read until the end by the parser before the program starts executing, the program will encounter end-of-file immediately. In the former case (which is usually what you want) they are satisfied from whatever file or device is connected to standard input of the Python interpreter.

When a script file is used, it is sometimes useful to be able to run the script and enter interactive mode afterwards. This can be done by passing -i before the script. (This does not work if the script is read from standard input, for the same reason as explained in the previous paragraph.)


Argument Passing

When known to the interpreter, the script name and additional arguments thereafter are passed to the script in the variable sys.argv, which is a list of strings. Its length is at least one; when no script and no arguments are given, sys.argv[0] is an empty string. When the script name is given as '-' (meaning standard input), sys.argv[0] is set to '-'. When -c command is used, sys.argv[0] is set to '-c'. When -m module is used, sys.argv[0] is set to the full name of the located module. Options found after -c command or -m module are not consumed by the Python interpreter's option processing but left in sys.argv for the command or module to handle.


Interactive Mode

When commands are read from a tty, the interpreter is said to be in interactive mode. In this mode it prompts for the next command with the primary prompt, usually three greater-than signs (">>> "); for continuation lines it prompts with the secondary prompt, by default three dots ("... "). The interpreter prints a welcome message stating its version number and a copyright notice before printing the first prompt:

python Python 1.5.2b2 (#1, Feb 28 1999, 00:02:06) [GCC 2.8.1] on sunos5 Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam >>>

Continuation lines are needed when entering a multi-line construct. As an example, take a look at this if statement:

>>> the_world_is_flat = 1
>>> if the_world_is_flat:
... print "Be careful not to fall off!" ... Be careful not to fall off!

The Interpreter and Its Environment

Error Handling

When an error occurs, the interpreter prints an error message and a stack trace. In interactive mode, it then returns to the primary prompt; when input came from a file, it exits with a nonzero exit status after printing the stack trace. (Exceptions handled by an except clause in a try statement are not errors in this context.) Some errors are unconditionally fatal and cause an exit with a nonzero exit; this applies to internal inconsistencies and some cases of running out of memory. All error messages are written to the standard error stream; normal output from executed commands is written to standard output.

Typing the interrupt character (usually Control-C or DEL) to the primary or secondary prompt cancels the input and returns to the primary prompt.2.1Typing an interrupt while a command is executing raises the KeyboardInterrupt exception, which may be handled by a try statement.


Executable Python Scripts

On BSD'ish Unix systems, Python scripts can be made directly executable, like shell scripts, by putting the line

#! /usr/bin/env python

(assuming that the interpreter is on the user's PATH) at the beginning of the script and giving the file an executable mode. The "#!" must be the first two characters of the file. On some platforms, this first line must end with a Unix-style line ending ("\n"), not a Mac OS ("\r") or Windows ("\r\n") line ending. Note that the hash, or pound, character, "#", is used to start a comment in Python.

The script can be given an executable mode, or permission, using the chmod command:

$ chmod +x myscript.py

Source Code Encoding

It is possible to use encodings different than ASCII in Python source files. The best way to do it is to put one more special comment line right after the #! line to define the source file encoding:

# -*- coding: encoding -*-

With that declaration, all characters in the source file will be treated as having the encoding encoding, and it will be possible to directly write Unicode string literals in the selected encoding. The list of possible encodings can be found in the Python Library Reference, in the section on codecs.

For example, to write Unicode literals including the Euro currency symbol, the ISO-8859-15 encoding can be used, with the Euro symbol having the ordinal value 164. This script will print the value 8364 (the Unicode codepoint corresponding to the Euro symbol) and then exit:

# -*- coding: iso-8859-15 -*-

currency = u"€"
print ord(currency)

If your editor supports saving files as UTF-8 with a UTF-8 byte order mark (aka BOM), you can use that instead of an encoding declaration. IDLE supports this capability if Options/General/Default Source Encoding/UTF-8 is set. Notice that this signature is not understood in older Python releases (2.2 and earlier), and also not understood by the operating system for script files with #! lines (only used on Unix systems).

By using UTF-8 (either through the signature or an encoding declaration), characters of most languages in the world can be used simultaneously in string literals and comments. Using non-ASCII characters in identifiers is not supported. To display all these characters properly, your editor must recognize that the file is UTF-8, and it must use a font that supports all the characters in the file.


The Interactive Startup File

When you use Python interactively, it is frequently handy to have some standard commands executed every time the interpreter is started. You can do this by setting an environment variable named PYTHONSTARTUP to the name of a file containing your start-up commands. This is similar to the .profile feature of the Unix shells.

This file is only read in interactive sessions, not when Python reads commands from a script, and not when /dev/tty is given as the explicit source of commands (which otherwise behaves like an interactive session). It is executed in the same namespace where interactive commands are executed, so that objects that it defines or imports can be used without qualification in the interactive session. You can also change the prompts sys.ps1 and sys.ps2 in this file.

If you want to read an additional start-up file from the current directory, you can program this in the global start-up file using code like "if os.path.isfile('.pythonrc.py'): execfile('.pythonrc.py')". If you want to use the startup file in a script, you must do this explicitly in the script:

import os
filename = os.environ.get('PYTHONSTARTUP')
if filename and os.path.isfile(filename):
execfile(filename)

Read more...

Python

Python is a dynamic object-oriented programming language that can be used for many kinds of software development. It offers strong support for integration with other languages and tools, comes with extensive standard libraries, and can be learned in a few days. Many Python programmers report substantial productivity gains and feel the language encourages the development of higher quality, more maintainable code.

Python runs on Windows, Linux/Unix, Mac OS X, OS/2, Amiga, Palm Handhelds, and Nokia mobile phones. Python has also been ported to the Java and .NET virtual machines.

Python is distributed under an OSI-approved open source license that makes it free to use, even for commercial products.

The Python Software Foundation (PSF) holds and protects the intellectual property rights behind Python, underwrites the PyCon conference, and funds grants and other projects in the Python community.

You can download ActivePython here, it's connect with database to use ODBC or cx_Oracle if connected with Oracle can run on window difference with Python.org, If you use python.org download here.

Python is OSI Certified Open Source

About Python

Python is a remarkably powerful dynamic programming language that is used in a wide variety of application domains. Python is often compared to Tcl, Perl, Ruby, Scheme or Java. Some of its key distinguishing features include:

• very clear, readable syntax
• strong introspection capabilities
• intuitive object orientation
• natural expression of procedural code
• full modularity, supporting hierarchical packages
• exception-based error handling
• very high level dynamic data types
• extensive standard libraries and third party modules for virtually every task
• extensions and modules easily written in C, C++ (or Java for Jython, or .NET languages for IronPython)
• embeddable within applications as a scripting interface

Python is powerful... and fast

Fans of Python use the phrase "batteries included" to describe the standard library, which covers everything from asynchronous processing to zip files. The language itself is a flexible powerhouse that can handle practically any problem domain. Build your own web server in three lines of code. Build flexible data-driven code using Python's powerful and dynamic introspection capabilities and advanced language features such as meta-classes, duck typing and decorators.

Python lets you write the code you need, quickly. And, thanks to a highly optimized byte compiler and support libraries, Python code runs more than fast enough for most applications.

Python plays well with others

Python can integrate with COM, .NET, and CORBA objects.

For Java libraries, use Jython, an implementation of Python for the Java Virtual Machine.

For .NET, try IronPython , Microsoft's new implementation of Python for .NET, or Python for .NET.

Python is also supported for the Internet Communications Engine (ICE) and many other integration technologies.

If you find something that Python cannot do, or if you need the performance advantage of low-level code, you can write extension modules in C or C++, or wrap existing code with SWIG or Boost.Python. Wrapped modules appear to your program exactly like native Python code. That's language integration made easy. You can also go the opposite route and embed Python in your own application, providing your users with a language they'll enjoy using.

Python runs everywhere

Python is available for all major operating systems: Windows, Linux/Unix, OS/2, Mac, Amiga, among others. There are even versions that run on .NET, the Java virtual machine, and Nokia Series 60 cell phones. You'll be pleased to know that the same source code will run unchanged across all implementations.

Your favorite system isn't listed here? It may still support Python if there's a C compiler for it. Ask around on news:comp.lang.python - or just try compiling Python yourself.

Python is friendly... and easy to learn

The Python newsgroup is known as one of the friendliest around. The avid developer and user community maintains a wiki, hosts international and local conferences, runs development sprints, and contributes to online code repositories.

Python also comes with complete documentation, both integrated into the language and as separate web pages. Online tutorials target both the seasoned programmer and the newcomer. All are designed to make you productive quickly. The availability of first-rate books completes the learning package.

Python is Open

The Python implementation is under an open source license that makes it freely usable and distributable, even for commercial use. The Python license is administered by the Python Software Foundation.

Read more...