Topic Thread

Significance of python compared to C#

  • 1.  Significance of python compared to C#

    Posted 01-07-2018 15:26
    Hi,
    Though Python is one of the top  programming skills in the analytics world with its numerous capabilities, wondering why C# is not an equally competing programming language, especially when .Net based programming languages also support machine learning? (curious, being from .Net background). I have been surfing internet but didn't find a convincing answer. Appreciate any help here !

    ------------------------------
    Varalakshmi T(Laks)
    University of Washington-Tacoma
    Tacoma WA
    ------------------------------


  • 2.  RE: Significance of python compared to C#

    Posted 01-08-2018 01:48

    I come from a C++/Matlab background myself, and have used C# extensively as well. I do prefer those languages but have been working with R more recently.

    The short list of reasons for why Python is more popular than C#:

    1) Not tied to the Microsoft eco-system (although C# is open source), and cross-platform
    2) Python is easy to learn for people who are new to writing software, also people with a software background can pick it up fast
    3) Very active communities with tutorials, documentation, etc.
    4) Many libraries have been built for Python, which makes it more desirable to use
    5) Much of the Python libraries are written in C++ under the hood so you can get decent speed

    I'm sure there are other reasons but this is what I've found in my experience this far.

    Erik



    ------------------------------
    Erik Burd
    University of San Francisco
    San Francisco CA
    ------------------------------



  • 3.  RE: Significance of python compared to C#

    Posted 01-09-2018 01:15
    I concur with Erik Burd's list. I have been using programming languages for scientific assessment since the mid-to-late 1970s, when I started with FORTRAN & Assembler (yes...that makes me old). I was raised through "formal" training in Pascal, C, C++, C#, Java... etc.

    Nevertheless, I like the relative ease of use of the various Python libraries (NumPy, SciPy, Plotly, ...) and the ability to cobble together analysis very quickly.

    Am also a fan of R, of Matlab, and (quite frankly) of visual basic programming in Excel, which I still find far and away to be a very nice tool for translating numbers and series into visual displays as well as being quite capable of supporting rapid "what-if" assessments, particularly of raw measurements.

    I work principally with measured data from patients and I have found no tool to enable me to visualize and target specific areas of interest better than Excel. I then write targeted programming routines for detailed analysis in Python or R.

    ------------------------------
    John Zaleski, Ph.D., CAP, CPHIMS
    Chief Analytics Officer
    Bernoulli Enterprise, Inc.
    jzaleski@bernoullihealth.com
    C1: (484) 319-7345
    C2: (203) 343-9225
    Elkton, MD & Milford, CT
    United States
    ------------------------------



  • 4.  RE: Significance of python compared to C#

    Posted 01-11-2018 02:55
    Hi,
    Thank you everyone for your valuable inputs in this discussion. I agree that Python being light weight and open source helps the Data Scientists to get work done quickly with readily available packages. The trend could change with many programming languages going forward.
    I miss the Intellisense part though ! Glad that Visual studio supports Python.

    ------------------------------
    Varalakshmi(Laks) Tyaggarajan
    University of Washington-Tacoma
    Tacoma WA
    ------------------------------



  • 5.  RE: Significance of python compared to C#

    Posted 01-08-2018 21:45
    Hi!

    I believe that Python has a repl - read, evaluate, process, loop capability, which makes it easier to test code.

    Also, in general, languages that do not require compilation as a separate step, i.e. basic, APL (for us old folks), python, . . . tend to be lighter in they often require less boilerplate.  Some of the newer programming languages that are compiled have picked up on this, so the trend may change.

    Finally (from my perspective), languages like python and apl make it very easy to work with matrices, arrays, . . .

    I suspect that all programming languages will move in this direction, so that there is less marginal benefit in using languages like python for quick development/experimentation.

    All the best . . .

    Phil Troy

    ------------------------------
    Phil Troy
    Cote Saint-Luc QC
    ------------------------------



  • 6.  RE: Significance of python compared to C#

    Posted 01-10-2018 21:54
    1. The Read, evaluate, print, loop (REPL) (an enhanced interactive prompt for those more familiar with Matlab or Mathematica) is essential for data analysis. A lot of the work is in data is trial and error, and being able to interactively try things is a big advantage that all of the main data analysis environments have.
    2. To expand on the comment of Erik Burd on the active communities and libraries, a second big advantage that the main data analysis environments (Python, R, Matlab) are the quantitative and scientific libraries/packages/modules available.  All are built on a foundation of linear algebra and numerical analysis libraries, and this means that other technical libraries are available using them.  So R becomes probably the best environment for statistics and data analysis.  Matlab if you are in electrical or chemical engineering.  But Python has an additional advantage over R in that it is a well used language in a wide range of domains.  So I think of Python as second to R for statistical and data analysis work, but because there are python libraries available for things such as symbolic math, optimization, simulation, and a wide range of subject specific libraries in a wide range of sciences, and that it has a wider range of facilities for managing data, working with GUI and the internet, I will often use Python because the project I'm working with is bigger than just statistics and data analysis.  SciPy has a section on their website devoted to listing all of the other (than the base numerical libraries detailed at SciPy.org - SciPy.org) scientific libraries that are available in Python (Topical Software - SciPy.org), and because of this range, whenever I find myself in a situation where I need to do analytics and ____, I often find myself choosing Python.

    [Background note: I used Python in my dissertation, but for teaching undergraduate and graduate analytics courses, I have settled on R as my language for teaching]

    ------------------------------
    Louis Luangkesorn
    Assistant Professor
    Department of Industrial Engineering
    University of Pittsburgh
    Pittsburgh PA
    ------------------------------



  • 7.  RE: Significance of python compared to C#

    Posted 01-16-2018 12:18
    This is only related to this thread for the subject line.

    Ladies- I just found the coolest website for dresses, and no matter your preference, you could buy a python (Python Code Fit & Flare Dress) or C (C Language Code Stripes Fit & Flare Dress) language dress! Plus a lot of other cool nerdy prints, I'm going to stock up! :)

    ------------------------------
    Mary Leszczynski
    Design & Brand Manager
    INFORMS
    Catonsville MD
    ------------------------------



  • 8.  RE: Significance of python compared to C#

    Posted 01-17-2018 06:33
    Those are great!  Thanks!

    ------------------------------
    Betty Love
    University of Nebraska - Omaha
    Omaha NE
    ------------------------------



  • 9.  RE: Significance of python compared to C#

    Posted 01-18-2018 06:45
    Hello:

    I feel that the points by @Erik Burd , especially 3-5 are important. The availability of libraries and support for solving problems is important for the research community to stick with a language. The introduction of a new language, for example the languages like Julia and all needs to compete with the community who creates this large library base of R and Python.
    In addition to that Python community accepts their limitation and finds solutions for that, as the 5th one pointed out. Many python libraries use C++ code for its implementation.

    ​​

    ------------------------------
    Sreekanth V K
    Research Scholar
    Indian Institute of Technology Kharagpur
    Kharagpur
    ------------------------------



  • 10.  RE: Significance of python compared to C#

    Posted 01-21-2018 19:29
    One reason C# hasn't caught on for high-performance data analysis is that the Linux/Unix implementations of C# haven't been competitive with other tools until recently. I'm not a C# programmer, but I understand from a bit of Googling that recent .NET Core releases are close to competitive on Linux with the Windows implementations.

    <NerdAlert>

    Sorry, I just can't resist pointing out that the C dress's code for printing the Fibonacci sequence is just about the slowest possible algorithm. It computes fibonacci(n) recursively as the sum of fibonacci(n-1) and fibonacci(n-2), which takes time exponential in n. An iterative computation would be much faster (linear in n), and a correctly implemented recursive algorithm would be faster still (O(log n)). The correct recursion with caching of intermediate results would be even faster.

    The naive recursive algorithm takes 1 minute and 21 seconds to compute fibonacci(50) and 2 minutes and 15 seconds to compute fibonacci(51) on my Intel i7 (3.6GHz) laptop. The iterative method and the recursive method below compute fibonacci(93) (the largest that fits in a 64-bit unsigned integer) instantaneously. If one wanted to compute larger Fibonacci numbers with a bigint library, the difference between iterative and good recursive algorithms would become apparent

    unsigned long int fibonacci(int n)
    {
        unsigned long int f1, f2;

        if ( n == 0 ) return 0;
        if ( n == 1 || n == 2 ) return 1;
        if ( n % 2 == 1 ) {
           f1 = fibonacci((n + 1) / 2);
           f2 = fibonacci((n - 1) / 2);
           return f1 * f1 + f2 * f2;
        } else {
           f1 = fibonacci(n / 2 + 1);
           f2 = fibonacci(n / 2 - 1);
           return f1 * f1 - f2 * f2;
        }
    }
    </NerdAlert>

    ------------------------------
    Matthew Saltzman
    Clemson University
    Clemson SC
    ------------------------------



  • 11.  RE: Significance of python compared to C#

    Posted 01-22-2018 08:19
    This <NerdAlert> is very interesting.   I coded up two obvious fast versions and the slow version you mentioned (see code pasted below).   The two fast versions solve the n=171 problem in 0.001 seconds on my laptop computer while the slow version solves the n=50 problem in 60.2 seconds (and the n=171 instance would take billions of years).

    PS.  When I was an undergrad at RPI, the word NERD was spelled KNURD because that's drunk backwards.  :-)

    unsigned long int fib_fast0(int n)
    {
        int i;
        unsigned long f0=1, f1=1, f2=1;
        for (i=3; i<=n; i++) {
            f0 = f1+f2;
            f2 = f1;
            f1 = f0;
        }
        return f0;
    }

    unsigned long int fib_fast1(int n)
    {
        unsigned long int f1, f2;
        if ( n == 0 ) return 0;
        if ( n == 1 || n == 2 ) return 1;
        if ( n % 2 == 1 ) {
            f1 = fib_fast1((n + 1) / 2);
            f2 = fib_fast1((n - 1) / 2);
            return f1 * f1 + f2 * f2;
        } else {
            f1 = fib_fast1(n / 2 + 1);
            f2 = fib_fast1(n / 2 - 1);
            return f1 * f1 - f2 * f2;
        }
    }

    unsigned long int fib_slow(int n)
    {
        unsigned long int f1, f2;
        if ( n == 0 ) return 0;
        if ( n == 1 || n == 2 ) return 1;
        return fib_slow(n-1) + fib_slow(n-2);
    }

    ------------------------------
    Robert Vanderbei
    Professor
    Princeton University
    Princeton NJ
    ------------------------------



  • 12.  RE: Significance of python compared to C#

    Posted 01-22-2018 09:50
    If you do long unsigned (64-bit) arithmetic, the largest correct answer you can get is fibonacci(93) = 12200160415121876738 . Anything bigger overflows. One could get 80 bits using long double, but one would need to worry about extracting exact integer values from the floating-point representation. That would get you up to fibonacci(115).

    Back in the days of the Pentium III (early ones were 650MHz), I tried this in Maple with arbitrary-precision arithmetic. The naive recursion was hopeless fairly quickly, but fibonacci(100000) (one hundred thousand, not a typo) took about 45 seconds. The fast recursion took 15 seconds. The fast recursion with Maple's "options remember" to cache intermediate results was nearly instantaneous, even then.

    ------------------------------
    Matthew Saltzman
    Clemson University
    Clemson SC
    ------------------------------



  • 13.  RE: Significance of python compared to C#

    Posted 01-22-2018 10:48
    Good point.   On my computer, I get the overflow answer for 93 but the right answer for 91 and 92:

    91:  4660046610375530309
    92:  7540113804746346429
    93: -6246583658587674878

    Of course, getting the wrong answer doesn't affect the cpu time.   :-)


    --Bob

    Robert Vanderbei

    "All of the true things that I am about to tell you are shameless lies."  -- K. Vonnegut






  • 14.  RE: Significance of python compared to C#

    Posted 01-23-2018 10:25
    Bob and Matt:

    Or you can just use Python.  You then get arbitrary precision integers for free.

    In Python on my laptop, fib_fast0(171) takes on average 11.1 microseconds, and fib_fast1(171)  takes on average 83.2 microseconds.

    And for reference, the value of fib(171) is 244006547798191185585064349218729154

    While one would expect fib_fast1() to be faster because of the theoretical O(log(n)) behavior, I think the overhead of the function calls in the recursion as well as the division operations is just too expensive.



    ------------------------------
    -Irv Lustig
    Optimization Principal
    Princeton Consultants
    ------------------------------



  • 15.  RE: Significance of python compared to C#

    Posted 01-23-2018 15:51

    Irv:

    I think you're mistaken about the complexity of fib__fast1().  Since it does two recursive calls, the recurrence is T(n) = c + 2T(n/2), which is O(n) rather than O(log n).  The following is a Ruby implementation that is O(log n), which should be pretty easy to port to Python if you are interested.   (Both Python and Ruby offer arbitrary length integer arithmetic.)

        def matrix_fib(n)
          if n == 1
            [0,1]
          else
            result = matrix_fib(n/2)
            c = result[0] * result[0] + result[1] * result[1]
            d = result[1] * (result[1] + 2 * result[0])
            n.even? ? [c,d] : [d,c+d]
          end
        end

        def fib(n)
          raise "fib not defined for negative numbers" if n < 0
          n.zero? ? n : matrix_fib(n)[1]
        end

    This is based on expressing the Fibonacci transformation as a matrix operation which is applied to a vector of 2 input values.  All Fibonacci values can then be expressed as the corresponding power of the transform matrix, and matrix exponentiation can be done in log(n) time by squaring the result of the halved power.

    On my laptop this is giving runtimes of around 12 microseconds for fib(171), 24 microseconds for fib(1000), and 90 microseconds for fib(10000).  I suspect the bulk of the time is spent in the bigint functions operating behind the scenes.

    ------------------------------
    Paul Sanchez
    Naval Postgraduate School
    Monterey CA
    ------------------------------



  • 16.  RE: Significance of python compared to C#

    Posted 01-24-2018 12:18

    Paul:

    I based my comment about O(log(n)) performance on something previously written by Matt.

    Here's an interesting graph, comparing the 3 implementations, all in Python.  fast0 and fast1 are the ones done by Matt, while matrix is the one you supplied.  It's a clear winner.  Times are computed using Python timeit capabilities, and represent the average of 10 runs.

    I think you are correct in that the computation time for doing the additions (and memory copies) of the big int values is causing the quadratic (?) performance of the fast0 algorithm, which has a linear number of operations.

    Plot of Fibonacci computation times



    ------------------------------
    -Irv Lustig
    Optimization Principal
    Princeton Consultants
    ------------------------------



  • 17.  RE: Significance of python compared to C#

    Posted 01-25-2018 10:33
    Python is awesome.

    ------------------------------
    Robert Vanderbei
    Professor
    Princeton University
    Princeton NJ
    ------------------------------



  • 18.  RE: Significance of python compared to C#

    Posted 01-23-2018 16:02
    Irv-

    Fair point. I started with C because I was critiquing the C "dress code".

    On my laptop, the crossover point where the Python recursive version (without caching) takes over is around n = 50,000 at about 50 millilseconds. For n = 500,000, the sequential version takes about two seconds and the recursive version takes under 200 milliseconds.

    I won't subject the list to the value of fib(500000), which has 104,495 digits.

    ------------------------------
    Matthew Saltzman
    Clemson University
    Clemson SC
    ------------------------------



  • 19.  RE: Significance of python compared to C#

    Posted 01-23-2018 20:13
    Matt-

    Here's a Python 3 implementation of the matrix exponentiation algorithm I previously posted in Ruby:

        def matrix_fib(n):
            if n <= 1: return [0, 1]
            result = matrix_fib(n // 2)
            c = result[0] * result[0] + result[1] * result[1]
            d = result[1] * (result[1] + 2 * result[0])
            return [c, d] if n % 2 == 0 else [d, c + d]

        def fib(n):
            return n if n <= 0 else matrix_fib(n)[1]

    On my machine it's about 100 times as fast as what you reported for n = 50000.  I'm curious if that's due to the hardware, the Python version, or the algorithm.

    ------------------------------
    Paul Sanchez
    Naval Postgraduate School
    Monterey CA
    ------------------------------



  • 20.  RE: Significance of python compared to C#

    Posted 01-29-2018 10:49
    Paul-

    I suspect it's an artifact of how I'm doing the timing. I wasn't doing anything careful, just used the Unix time command:

    $ time ./fib.py 50000

    where fib.py defines functions for the various methods and then just invokes

    fib_mat(int(sys.argv[1]))

    where fib_mat is your fib() function. Using your method for n = 50000, I got about 30 ms user time and for n = 500000, I got about 50 ms. (Compare with 50 ms and 180 ms, respectively, for the recursive code and 60 ms and 1.9 sec for the sequential.)

    My Python is 2.7.14 and my CPU is a Core i7-7600 2.8GHz, FWIW.



    ------------------------------
    Matthew Saltzman
    Clemson University
    Clemson SC
    ------------------------------



  • 21.  RE: Significance of python compared to C#

    Posted 01-18-2018 15:01
    Mary - Love the dresses! Very creative!


    ------------------------------
    Erik Burd
    University of San Francisco
    San Francisco CA
    ------------------------------