Performance issue

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

Performance issue

jitferry
I am evaluating Thymleaf and did some prototype to compare JSP, Thymleaf and FreeMarker, all backed by Spring MVC. I really like the way Thmleaf works and the programming model, but it seems to have some performance problems.
I implemented a page with a table containing about 100 rows with all 3 view technologies, and then use ApachBench to test the performance.
With 10000 requests and concurrency of 100, Thymleaf was about 10 times slower than the other two. It's a surprise that FreeMarker was about 10-20% faster than JSP. The memory usage of Thmeleaf was about double the other two. I am not so surprised by the higher memory usage since it's DOM based, but the rendering speed is a major concern. I thought the rendering part should only be a small portion of the overall response time, so then I hooked up a DB query to support the page, but I didn't see much difference.
Reply | Threaded
Open this post in threaded view
|

Re: Performance issue

danielfernandez
Administrator

Hello,

First of all: Thymeleaf is slower than JSP. Or at least it should be, and that is completely normal. As with everything, there is a trade-off and Thymeleaf just offers things JSP does not.

Think that each JSP file ends up being compiled at the web server, and turns into an in-memory Servlet object with a huge "service()" method, probably heavily inlined and JIT-ted by the virtual machine after a couple of executions. It's really hard to beat that in terms of performance --your FreeMarker results look quite intriguing to me, to be honest... you may have hit a performance sweet spot at FreeMarker there.

Are you using JSTL and the Spring MVC taglibs with JSP? And what about forms? Typical web apps have forms in many of their pages, and correctly bound forms --bound to a form-backing bean and adequately displaying its data in the correct format, I mean-- tend to be a performance hole compared to simply listing data from a DB...

As for Thymeleaf, I presume you are using Thymeleaf 1.x, right? Which version, specifically? Thymeleaf performance in Spring MVC environments improved quite a lot in 1.1.3, but anyway it would be fantastic if you could do your benchmarking again with 2.0.0-beta2, which is already available at the maven repositories and which sports a complete rewriting of the engine architecture, including abandoning the standard DOM API and boosting performance *a lot*. Version 2.0.0 GA will be available in a couple of days, but beta2 is feature-complete, so you should obtain accurate results with this version.

And last... would you be so kind to share your code? We still don't have JSP vs Thymeleaf benchmarking artifacts (only comparing diverse versions of Thymeleaf), so it would be great to have a look at the code you used for obtaining these figures... and see if we can spot any performance improvement points there...

Thanks,
Daniel.
Reply | Threaded
Open this post in threaded view
|

Re: Performance issue

danielfernandez
Administrator

Oh, and just one more thing: Thymeleaf performs a lot of its initialization operations in a lazy manner during its first execution of the Template Engine (instead of doing it at container startup), so I think it would be much fairer if you let the first execution of each framework out of the benchmark.

Just accessing the benchmarked URL with a browser before launching the "ab" tool should be enough to let the framework initialize without this penalizing its benchmark performance.

And of course --although I imagine you've already done this--: please set all logs in thymeleaf to WARN or ERROR. Thymeleaf can be extremely verbose and logging could severely affect your performance figures.

Thanks again,
Daniel.
Reply | Threaded
Open this post in threaded view
|

Re: Performance issue

danielfernandez
Administrator
Hi again,

I've just published the new Thymeleaf 2.0 benchmark, comparing it to 1.1.5: http://www.thymeleaf.org/benchmarking20.html

Regards,
Daniel.

Reply | Threaded
Open this post in threaded view
|

Re: Performance issue

Zemi
Administrator
Hi all,

as I have some code comparing different view technologies for the Spring I/O talk, I have run a simple benchmark against this pages.

The samples have no database access at all, and they simply render some list values, so this is by no means a significative test for a normal usage.

Some benchmark information:
- 50 concurrent threads
- 50000 requests per page
- Oracle JVM 1.7.0_01
- Apache Tomcat 7.0.23  with 2 Gb RAM
- Linux 64 bits
- Spring 3.1.0
- Apache Jmeter 2.6

These are test results:




Full source code is provided thymeleafexamples-views.tgz

Reply | Threaded
Open this post in threaded view
|

Re: Performance issue

danielfernandez
Administrator
Oh, that's pretty interesting: seems that JSP taglibs affect JSP performance enough to make it perform worse than Velocity/FreeMarker...
Reply | Threaded
Open this post in threaded view
|

Re: Performance issue

Zemi
Administrator
In my opinion, according to these figures, view technology performance is no significative for an enterprise application, where you usually have complex database access or some kind of remoting tecnologies like web services.

For some specific application domains, maybe it could be something to pay attention, but not for enterprise applications.

Reply | Threaded
Open this post in threaded view
|

Re: Performance issue

jitferry
My original numbers were based on Thymeleaf 1.4, and I forgot to turn on caching and turn off debug logging, after I made these changes, the performance improved a lot, but still about 5 times slower than JSP/FreeMarker, and then I tried v2.0-beta (it seemed the API changed quite a bit as my custom attribute processor not working with 2.0), but did not see much performance gain.

Yes, if some complicated DB queries were added, there were not much performance differences.

Then I saw the latest post of the benchmark results, so I started to wonder why my results were so much different. Then I realized I defined some "fragment" and "include" this in my page to make reusable code. The fragment is fairly simple only generating a link and a text. But after i remove the "include", it made a big difference, now it's only about 2 times slower than JSP/FreeMarker.
Reply | Threaded
Open this post in threaded view
|

Re: Performance issue

danielfernandez
Administrator
Hi,

There are still two points in your results that look a bit odd to me:

1. It is strange that you are not getting any performance gains in 1.1.4 vs 2.0.0-beta2. In fact it is very strange!

2. It is strange that you are getting no performance gains switching version BUT you are getting them when removing the include. In fact, if you look at the benchmark in http://www.thymeleaf.org/benchmarking20.html, the "GTVG-main" test was included in the benchmark because the main page of the GTVG application has a "th:include" (fragment include) and also a "th:utext" (code include)... and the performance improvements with respect to 1.1.5 are *huge* (from 7.49ms to 0.34ms)!

Are you sure your cache is working both for your "main" template and also for the template you include your fragment from? What method are you using for inclusion: "th:fragment", "XPath/DOM Selectors"...?

You can be sure your template cache works alright if you set the "org.thymeleaf.TemplateEngine.cache.TEMPLATE_CACHE" logger to TRACE and you look at the messages there... and don't forget to set it back to WARN in order to measure times! ;-)

Anda again, it would be easier if you shared your benchmarking code with us, so that we could be sure of what we're looking at...

Thanks for your comments!

Daniel.
Reply | Threaded
Open this post in threaded view
|

Re: Performance issue

Zemi
Administrator
Hi again,

I did some more test in order to add information about jitferry issue.

I re-test rendering larger lists and I find some figures more similar to jitferry's.

My benchmark information:
- 25 concurrent threads
- 50000 requests per page
- Oracle JVM 1.7.0_01
- Apache Tomcat 7.0.23  with 2 Gb RAM
- Linux 64 bits
- Spring 3.1.0
- Apache Jmeter 2.6

Test results:



As you can see, Thymeleaf performance get worse when rendering larger lists (with th:each processor).

To be fair, we should test with more complex pages, including conditions, i18n, formatting and random data.

I'm also surprising for freemarker's performance boots when rendering larger lists, maybe it is using some internal cache.

Again, full source code is provided thymeleafexamples-views.tgz

Reply | Threaded
Open this post in threaded view
|

Re: Performance issue

Zemi
Administrator
And again, my opinion is that from a architectural perspective, these performance differences are no significant.
What is more significant for the sucess of a project it is programmers' productivity and the code simpleness.
Reply | Threaded
Open this post in threaded view
|

Re: Performance issue

danielfernandez
Administrator
In reply to this post by Zemi
Hmm... it could be that given the longer executions the VM is JIT-ting something and therefore you start getting lower times with more results.

Is the template cache enabled in thymeleaf 2.0? and also... are you discarding the first execution of each framework in order to exclude one-time initialization times?

Regards,
Daniel.
Reply | Threaded
Open this post in threaded view
|

Re: Performance issue

Zemi
Administrator
>  Hmm... it could be that given the longer executions the VM is JIT-ting something

Yes, it's perfectly possible that memory starvation is affecting the test results because I'm requesting 125.000 pages which is a lot for my laptop.

On the other hand, I test all view technologies in the same test.

> Is the template cache enabled in thymeleaf 2.0?

Yes, I think so. But I didn't activate the logger to double-check this, it is in my TO-DO list.
 
> are you discarding the first execution

Yes, I am.

Reply | Threaded
Open this post in threaded view
|

Re: Performance issue

Eric Francis
Hello, was this thread ever revisited? Has there been more discussion about thyemleaf performance vs JSP?

Thanks,
Eric
Reply | Threaded
Open this post in threaded view
|

Re: Performance issue

EGHM
Were the JSPs precompiled?  If not I'ld like to see the precompiled performance included.