I load tested (using wrk) nginx serving a two line, static HTML file, against a basic Cowboy service that parsed parameters, did an ETS lookup, and rendered several hundred json elements, on my EC2 micro.
nginx came in at 14612 requests per second, and the dynamic code under Cowboy option came in at 13336 per second.
The same server was pulling 147 requests per second for a "hello world" rails application (Unicorn, production mode). Cowboy really isn't joking when they talk about performance.
I guess you can either see it as how much faster Cowboy is or how much more useful Rails is. It's nice if you have a lot of Erlang code already, but if you're making a web application from scratch, performance is pretty far down the list and I would stick to the half dozen stacks that are proven. Erlang can make sense sometimes, but it's probably not(!) comparable to Rails in those cases.
Edit: Eh, I forgot a "not" in the last sentence, which is now added.
> Erlang can make sense sometimes, but it's probably not(!) comparable to Rails in those cases.
I have no dog in this fight, but I've seen many people say that -despite the newness of the library "ecosystem"- they get an awful lot done with Elixir/Phoenix, and have a lot of fun doing it.
(Remember that Elixir is -like Erlang- a BEAM language. This means that Elixir gets the ease-of-understanding and interesting features of its syntax and has access to all Erlang software ever written. Similarly, any and all Elixir code can be called from Erlang with little trouble.)
Edit: (Also remember that the BEAM equivalent of Ruby is Elixir, and the (most popular?) BEAM equivalent of Rails is (AIUI) Phoenix. Phoenix uses Cowboy as its HTTP(S)/SPDY server.)
Yeah don't get me wrong, this wasn't some blog spam about "why I don't use Rails".
In fact I recommend it for a lot of cases. Problems that were solved years ago in Rails, like sane and default CSRF protection, are showing up in newly written PHP7 and Node applications that I'm seeing, because the things that Rails does are generally there for your own good.
I'm also aware it's not a direct comparison because Rails actually had an edge as far as its codebase.
The point I believe we can still take away from the original comment is - Cowboy is fast.
I'm very much looking forward to v2, the way it handles parameters is a much more natural fit imo.
The Phoenix Framework uses Cowboy at its core along with a Rack like middleware called Plug. There isn't much bloat in the stack so I'd assume performance is quite close to bare bones Cowboy. It doesn't quite have everything that Rails does, but it's getting there and fast!
Almost. I believe that half of the people that approach Phoenix coming from Rails stop when they realize there isn't any registration and authentication framework as good as Devise. The closest match is Addict but it covers only the basics and requires too much work. If one has to write his own product that could be OK. If one has to code for a customer it erodes the profit.
1) The Elixir community is still in its infancy and the community is growing rapidly and flourishing. If the concern is there isn't an equivalent Elixir library for X gem, it's only a matter of time before it is. So please join!
2) I attended the Phoenix training at ElixirConf this week and an easy answer to registration and auth is "hey, write a plug". Once you dive into how straightforward plugs (1) and what (init/1, call/2) do, it's really simple.
You will appreciate the explicit nature of plugs compared to say, Rack middleware...
I'm rewriting a Rails app in Phoenix and overall, I've found it to be a really pleasant experience.
I'm not so concerned by the lack of libraries as with that no one seem to know the state of Elixir, Phoenix or BEAM. Security vulnerabilities are found all the time in established stacks and things like robustness, reliability, maintenance and growth are ongoing issues. Yes, BEAM is proven in enterprise use, but most of us don't have those resources. The first time I tried Elixir the welcome site broke, which wasn't a big deal, but it still didn't exactly instill a lot of confidence.
> Security vulnerabilities are found all the time in established stacks and things like robustness, reliability, maintenance and growth are ongoing issues.
If you stick to writing code only in a BEAM language, you're insulated from all errors except for those of system resource exhaustion (edit: or -obviously- system hardware failure ;) ) or programmer logic errors. [0]
The parts of Elixir and other such projects that are written in another language that compiles down to native code, [1] are vulnerable to the usual array of issues endemic to the language selected.
> Yes, BEAM is proven in enterprise use, but most of us don't have those resources.
I'm not sure what you mean by this, and what concerns are implied by this statement. Would you care to expand this statement?
The WhatsApp folks have found that they got a lot done with Erlang with a small staff and (what is reported to be) very few machines when compared to the amount of work performed.
[0] Erlang can neither save you from your own faulty logic, nor can it add resources to your system. ;)
[1] Erlang provides a mechanism called NIF that lets you write performance-critical (or whatever) code in C or another language, and link it in to Erlang. From there, you can access your other-language code with from your Erlang code what appears to be a regular Erlang function call.
I guess in case of a web application it wouldn't be so much BEAM itself as the libraries. It hard to know if e.g. the crypto library is secure and used correctly. The same goes for serialization and other things that normally leads to problems further up the stack.
"Would you care to expand this statement?"
It seem proven enough if you have dedicated highly skilled developers. Not necessarily if you're a smaller startup that have to do a lot of other things and are relying on the ecosystem. With one of the more well know stacks you have a long history of not only security fixes, but knowledge about how to do things.
It kind of boils down to if I would implement say a payment system with the stack could I feel confident we wouldn't get compromised? Currently I have a lot higher confidence in e.g. Python + Django + Nginx than Elixir + Phoenix + Cowboy.
I understand and largely agree with your core points and concerns. I, myself am cautious and suspicious of new and/or unfamiliar software stacks. :)
So, the following disjointed commentary might be entirely superfluous. It also makes no mention of Elixir or Phoenix, as I've not yet used them:
AFAIK, everything shipped in Erlang/OTP has a test suite that you can run and -if you've the time and technical chops- inspect and evaluate for completeness and correctness.
Given that the Erlang community tends to be concerned about safety and reliability, I have substantially more confidence in the correctness of some random Erlang library on Github than of some random JavaScript (or -ye gods- Perl) library. ;)
Given that neither Cowboy nor Ranch appear to make use of any NIFs, I would be fairly confident in their safety in the face of garbage or malicious input.
The erlang-questions mailing list is a good place to ask questions of folks who use Erlang professionally and non-professionally for a wide variety of things. I've heard that the Elixir equivalents are similarly helpful.
Yeah, the routing system is. Fred Hebert (aka ferd, the author of "Learn You Some Erlang" and "Stuff Goes Bad: Erlang in Anger") is one of the folks working at Heroku on exactly that.
> One lesson I took from the first .com wave in the early 2000 was never to use a scripting language for application software again.
A couple of things:
Both Erlang and Python compile down to bytecode. Python compiles the first time you run a given script, Erlang won't execute Erlang source code; you must compile it to run it.
As go1979 mentions, Erlang has HiPE, which compiles Erlang down to the host platform's native code. If you find that your program is spending a lot of time in the Erlang code in a given module, you can -at any time- recompile that module with HiPE and reload it with either a system restart, or the hot code reloading machinery.
As an additional escape hatch, Erlang makes it rather easy to push performance-critical code out to a faster language -like C or C++ or whatever- and call that directly from Erlang. [0] Such code is called an NIF [1].
If most or all of what you're doing is hard-core number crunching, Erlang is probably not appropriate. If what you're doing requires tricky logic, needs to contain failures, or you're building a concurrent or distributed system, Erlang may very well be the right tool for the job.
Erlang's HiPE compiler is apparently AOT. The challenge seems to be dynamic typing. At points, it has to guess types. I assume compiled Python would have similar issues.
In the distant past, some guy performed such a benchmark on then-current versions of several popular web frameworks in Ruby, Go, maybe Python, and Elixir. Phoenix wasn't the fastest of the bunch, but it was also not the slowest, and -IIRC- provided the most consistent response time under heavy load.
I was either unaware (or had forgotten that) he had taken a second stab at the benchmark. I do hope he updates the document with the root cause of the enormous variance in response time.
I note with interest that the two frameworks which are faster than Phoenix (in both throughput and latency) are Gin, written in Go, and Play, written in Scala. In this run of the benchmark, they also have more consistent latency than Phoenix, although the author notes a suspected problem with his Erlang installation; previous runs had consistency better than Play, but still not as good as Gin.
I note with amusement that:
> Play consistently appeared to generate hundreds of socket read errors (see the detailed output), so I believe it should probably be semi-disqualified from the results for now.
I'd be interested to see another JVM framework tested. Play is really rather complicated, and it's written in Scala, which is also rather complicated. I'd love to see a comparison with a Sinatra-for-Java type framework like Spark, or perhaps Undertow (since it does so well on the TechEmpower benchmark). Oh well, maybe it's pull request time ...
Until he figures out what was up with his terrible Elixir consistency numbers, I would treat the second round of results with some suspicion. Maybe something changed in Phoenix/Elixir/Erlang, or maybe something's up with his machine.
If you do submit a PR, I do hope that you have the time and gumption to re-run all of his round 2 benchmarks, in addition to any additional ones you care to see. :)
Cowboy is great. It is a very perfomant, well designed and implemented piece of software. Also is mostly the work of one person -- Loïc Hoguin and his one-man 99s (Nine Nines) company.
Y'know, I tried using Erlang/OTP with rebar3 and cowboy to make a web app. And it turns out that I couldn't even figure out where to put my code, much less how to run it.
It seems a lot of people are having success with Erlang but I can't figure out how these applications actually run; it almost seems like there isn't a single step-by-step tutorial on the Internet for setting up an Erlang system using "best practices."
Am I just missing something? Because I feel like I'm hugely missing out.
It might be worth looking at how Phoenix[0] does it (or, you know, just consider using Elixir/Phoenix). Phoenix runs on cowboy out of the box and makes setting up a web app a snap:
$ mix phoenix.new my_app
which then sets up the directory structure for you to just run
$ mix phoenix.server
and have cowboy serving your webapp. The guides then tell you which files to add new routes, controllers, models, etc (in a way similar to Django or Rails). All that said, it's in Elixir at that point, but maybe you could still infer how the applications and gen servers and whatnot are structured from it.
It may be worth asking josevalim and chrismccord in #elixir-lang for tips on setting it up, since they obviously must know how to set up cowboy pretty well to build a system on top of it.
I think Erlang and things like cowboy and OTP are absolutely amazing for the web, but Elixir has made things 100x easier to get up and running with them.
You can also clone the repo and checkout the examples:
Make sure you have erlang installed
$ cd examples/hello_world
$ make
$ ./_rel/hello_world_example/bin/hello_world_example start
OR
$ ./_rel/hello_world_example/bin/hello_world_example console
This gets a Cowboy application up and running, but it doesn't tell you what to do to write your own Cowboy application, and doesn't give you a very firm understanding of either how the hello_world example works, or how to modify it to suit your needs.
As someone who's been down this road in the past, unless you're already rather familiar with Erlang and OTP's conventions, the gap between the User's Guide and the API documentation is far too large to make connecting information you learn from one with information you learn from the other easy.
I've summarized my personal struggle with the Cowboy user's guide and documentation in my other comment in the thread. Please read that comment before you downvote this one. :)
As far as I know there's no such thing as a `Cowboy application`. They're all Erlang/OTP applications. Sure, there's a learning curve associated with OTP apps, but since Erlang is a bit of an exotic language, I think it's understandable.
If you'd like to learn about the concepts behind an OTP application, you can read this guide: http://learnyousomeerlang.com/building-otp-applications or the Erlang documentation for OTP apps (which is pretty good I think).
Another viable option in my opinion would be to just look into the Phoenix framework for the Elixir language. Under the hood it uses cowboy. And it feels a lot like Rails.
I guess you didn't read my other comment in this thread. Please read it. ;)
> As far as I know there's no such thing as a `Cowboy application`.
From one pedant to another: "I know that." >:)
However, there is (to put it loosely in OTP terms) a set of Cowboy behaviours. Your application is expected to implement one or more of these. While the API for these behaviours is reasonably documented, the Cowboy equivalent of the -say- gen_server "User's Guide" is so sparse as to be almost non-existent.
I think one reasonable strategy of learning how some piece of software works or is meant to be used – especially when documentation is poor – is to just look at the source code (tests included).
I'm just getting started with the Phoenix framework, and I'm learning Elixir at the same time and reading through the source code helps me a lot, because:
1. I can check out how things are implemented
2. I get to read some idiomatic Elixir which helps me learn the language better
One can get reasonable understanding of software at the interplay between reading the docs and reading the source code.
I generally agree that -in the absence of sufficient documentation- reading the tests and source for the project is generally the best way to learn how to use a project.
However. If the language the project is written in is wildly unfamiliar to you, and you've never had experience with a similar language, then this technique is often... less than satisfying.
From your comments, it appears that you have more than a passing familiarity with Erlang/OTP. You have to admit that it would be substantially harder to understand both Elixir and Phoenix without your understanding of Erlang. :)
cowboy's documentation is pretty poor and there's no good 'getting started with cowboy' thing i can point you at, but this is pretty good for erlang in general:
Have you seen [1]? I found it very useful when getting starting with cowboy, though I had prior experience with Erlang.
It walks through a simple "Hello world" application starting with creation from template (erlang.mk), and discusses static files, SSL, websocket.. A very good first guide in my opinion. Though it doesn't really cover other techniques like how to architect your application or interact with a database.
Yes, I had the the same experience.
I started to learn Erlang for growing my personal skills, and because I love functional programming. There are a lot of books about Erlang itself, but when I started to try some http service I found a lack of documentations and step by step tutorial
My first attempt at making a simple Cowboy application was less than successful. I found the documentation spotty, inconsistent, and very difficult to understand. After about 45 minutes of really frustrated dicking around, I figured out how to make a simple HTTP server that allowed me to handle simple HTTP requests. I had to write code to switch on the passed-in HTTP verb, but everything worked, more or less. I could serve HTTP requests, but my code crashed after every request. (This was -of course- not fatal, but a bit of a performance killer.) Pipelined requests didn't work due to server connection resets. (Again, not fatal, just perf-diminishing.) I was utterly unable to get cowboy_rest to work.
Several months later, with a lot of additional Erlang experience under my belt, I come back to the documentation, and it's -somehow- 60->70% comprehensible. I figured out what was wrong with my simple HTTP server. (I was calling my request handler in init, rather than returning {ok, Req, []}, and letting Cowboy call my request handler later.) I still can't get cowboy_rest to work, but (OTOH) I haven't spent more than five minutes trying to get it to work since my first attempt way back when.
I don't know how much experience you have with Erlang, so I'll give you some basic advice that might be entirely unhelpful. Feel free to ask additional questions. Also feel free to ask questions on the erlang-questions mailing list. [0] Everyone who regularly posts there is very helpful. Many (most?) of them use Erlang professionally.
Anyway. Advice:
Projects/frameworks like OTP and Cowboy provide a callback API that application code is expected to conform to. Cowboy's documentation for this API is kind of all over the place, [1] but OTP's is well organized. [2]
Because Erlang tries to eschew global state as much as it can, OTP has a convention that one of the values in the tuple returned from an callback is the state for that particular instance of your application code. This State variable seems to always be the last element in the returned tuple, and the last argument to your callback function. State can be any Erlang term; it only has meaning to your application code. If your callbacks don't care about state, then you can pass anything as state. Many other frameworks have adopted this convention; it's a pretty good one.
Cowboy appears to be a pretty good HTTP/SPDY server. The beginner's documentation needs a LOT of love. (Even now, I had difficulty trying to piece together that walk through the docs in the footnote.) Given that it's a one-man shop, it's surprising that the docs are in as good a shape as they are; documentation isn't easy. :) But, still...
Anyway, questions? Comments? Ask away either here, or on the erlang-questions mailing list. :)
[2] For example, the docs for OTP's application API: http://www.erlang.org/doc/man/application.html or the gen_server API: http://www.erlang.org/doc/man/gen_server.html The parts before the "CALLBACK FUNCTIONS" section are functions you use to manipulate an application or gen_server. The bits after are functions you implement in your module that the application or gen_server machinery will call.
Just curious, was erlang.mk around and did you use the bootstrapped release / cowboy_http template [1] when creating your first application? Based on some things you mention, it makes it sound like you were writing from scratch while referencing the docs. :-)
I also got started from the cowboy user guide (with prior Erlang experience), but found it relatively easier, as I used the template, then cross-referenced the idiomatic template with the docs as I worked through them (tweaking 'knobs' as I went along), which worked well in helping me understand the design.
Along with the Erlang mailing list you've mentioned, I would also like to mention #ninenines (and #erlang for more general questions) on freenode. Very helpful communities!
* It enforces warnings-as-errors. This is a personal preference, but I hate this while developing. I want to iterate, not have to clean up unused variables that I already know about.
* It -until very recently- failed so, so, so hard when run with parallel make. I -at first- idly wondered if -somehow- Cowboy application builds were just non-deterministic. I then remembered that I aliased 'make' to 'make -j3'. Building with 'make -j1' fixed the build and put me squarely in the Rebar camp.
Rebar is just a far better fit for how I work.
> ...did you use the cowboy_http / bootstrapped release template...
Nah.
When learning new things, if I'm not in a rush, I tend -at first- to avoid templates. I've been burned by boilerplate code in the past, and have had boilerplate code cover up the true crawling horror that was more than a couple of APIs.
I'd rather initially do things "from scratch" [0] so that I can better evaluate the thing I'm learning, and the quality of its supporting documentation, or -if the documentation is inadequate- the explanatory power of its test suite and other sample code.
I did -however- read Cowboy's Getting Started guide, along with a huge chunk of the documentation. I actually started with the API documentation, figuring that it was going to be like erlang.org's documentation. Whoops! [1]
[0] And by that, I don't mean "from first principles". I'm quite happy to read an example, partially or fully digest it, then paste it into my text editor.
[1] Again, I know how good erlang.org's documentation is and how very much work it is to make such high-quality documentation. I don't mean to slight Loïc Hoguin. :)
Glad to see another person attempting it. There are a few (including a commercial product, a Kickstarter project, and me) but most have been abandoned, AFAIK.
ChicagoBoss is like a Rails for Erlang. Uses Cowboy as httpd. I've only played with it a little, so I can't say how well it holds up in production, but it sure seems neat.
nginx came in at 14612 requests per second, and the dynamic code under Cowboy option came in at 13336 per second.
The same server was pulling 147 requests per second for a "hello world" rails application (Unicorn, production mode). Cowboy really isn't joking when they talk about performance.