Replacing .Net remoting with web services

pokorra

Since January 2013 I have been working on this, beside many other things.
The goal is to replace .Net remoting as the communication between OpenPetra client and server, by using simple web service calls.
This reduces the complexity, and improves speed at the same time. We can use https for securing the connection.
We still have generated glue code, which not only supports our WebConnectors, but even our UIConnectors (remoting properties of objects).
For Standalone, we directly link the server to the client, so no two applications are needed anymore.
For development, we have our own simple ASP server, Ict.Tools.TinyWebServer.exe (see github.com/tpokorra/openpetragi ... yWebServer), which is basically a wrapper for HttpListener, and works fine on Windows. Alternatively the Mono XSP Server can be used as well.

My work is mainly in this branch at the moment:
github.com/tpokorra/openpetragi ... 4_remoting

This basically works, and should be tested:

building Windows Standalone (nant buildWindowsStandalone)
** download and test installer at ci-win.openpetra.org/view/All/j ... heus/9216/
building Linux server installer (nant buildServerCentOSPostgresqlOBS)
** see build.opensuse.org/package/show ... postgresql
** install from software.opensuse.org/download/p ... postgresql
** it will install the server with lighttpd and fastcgi-mono-server4, and install a demo database
building Windows Client installer (nant buildWindowsRemoteClient)
building Linux client installer (nant buildLinuxClientOBS)
** see build.opensuse.org/package/show ... tra-client
** install from software.opensuse.org/download/p ... tra-client

There are still some things todo, some warnings to get rid off, I think the keepalive functionality is still to be implemented again.
But basically it works, and I would appreciate feedback.

The code is not based on the latest trunk, but I will merge trunk into that branch when the next release is done (ie. when trunk is declared stable enough).

All the best,
Timotheus

joejoe2010

Hey Timo,

thanks for explaining this a bit.

I still have 2 questions.

For Standalone, we directly link the server to the client, so no two applications are needed anymore.

So the standalone version does not use the TinyWebServer?

What happens if the server-client connection breaks down at some time and then the user at the client presses the save button - is there any caching on the client side or does the information then get lost? (I think I already asked something similar on another post).

Greetings
Joachim

pokorra

Hello Joachim,
yes, the standalone will just be one simple .net application, no server in the background. It will contain the Ict.Petra.Server.*.dlls, and use them directly.

There is no caching: It would be too complicated.
I guess most applications on the web do not support caching, when the Internet connection is broken.
And in a multi-user environment, caching unsaved data on the client does not really make sense.
I think there are retries for 10 seconds, to call a web service. This could be configured to a longer time value.

All the best,
Timotheus

pokorra

Hello Christian,
just to let you know, my remoting branch currently does have problems with session cookies.
I am still experimenting with it. All tasks related to the Progress Dialog are not reliable.
I will have to take a break from the branch, I am way behind with other work, and will be available sometime in November again.

Please continue the discussion here, instead of email.

Thanks,
Timotheus

alanjfpaterson

After the weekly meeting on Thursday - and in particular concerning the provision of a Windows development web server - I was telling Christian about the Microsoft Visual Studio way of handling this issue. Going back to the early days of ASP.NET they had a product called Cassini (There was a space satellite named Cassini that went to the far reaches of the solar system I think). It was written in .NET and used System.Web and .NET sockets and was sort of to show what could be done. They made this .NET 1.1/2.0 source code publicly available (under Microsoft Public License) back in 2004 and I remember downloading it and trying it. There was an exe and a DLL and the DLL could be instantiated in its own AppDomain by the ASP.NET runtime using System.Web.CreateApplicationHost. Each instance of the exe could act as the server for one web site on a different port of the local PC. I adapted the program so that you got a nicer GUI which could run in the system tray and which enabled you to specify multiple web sites through the single GUI. I did have ideas that it might be useful at MKS but in fact it never went anywhere (and MKS never knew about it because I toyed with it in my own time).

Microsoft continued to develop the product and the source code continued to be public right up to .NET 3.5. The GUI was always very basic but the point was that it could be launched from the command line. A derivative of the public product was integrated into Visual Studio so that a web project could either use IIS or the development web server as the means of debugging a web project.

Over the weekend I dug out the code that I had written 10 years ago and got it going in .NET 4 using the original Microsoft code. I had a simple web service that I created with two methods for adding and subtracting two int's. That all worked fine with the web server and I could step through the web service code in the debugger. I also have the source pages for the Church web site on my laptop (which are just static pages) and that too is fine. If the site doesn't have a default page then you get a nice directory listing. It handles scripts and images - it basically has to handle anything that a developer might want to put on a web site. I made the GUI look nice and it is now called the 'Open Petra Web Server'. It even has help! I would like to show screenshots but I don't know how to put them on the forum.

The thing that appeals to me is that we have the whole source code for the project. It does not contain http logging but that would be easy to add as an option. By default it only accepts connections from the local host but optionally it can be used to accept trusted connections and would be fine on a trusted intranet. It is not for use over the public internet though!!

The (current) Visual Studio product does not have a GUI at all. It is a pure system tray application that is launched with a command line - which might be more suitable for us. But if you want to have a web server that is aware of multiple Open Petra branches we might prefer to have a GUI that supports that. The key thing is that the GUI's are just code on top of the DLL that does the main work and that would not change. And the nice thing is that since the code is all .NET and is self-contained it might apply equally to Windows and Linux - although I guess you have other options for Linux.

On Windows there are not many options. People obviously liked the Cassini product and a few groups have taken the original code and extended it (so there are products like UltiDev and CassiniDev) but I'm not really sure that they offer anything that we need and the code is not public although the product is free to download. I think it is important that we should have the source code for anything we use. Oh - and Microsoft make IIS Express available for free to go with the Express editions of Visual Studio. This gives you the latest version of 'Cassini' too.

So I am letting you know this in case you think it might be useful. If you think it would be useful - at least for Windows users - I would certainly like to update my version of the Microsoft code to the final publicly available code, which will only take about 1 hour, and then work out the best way of integrating the GUI with our automated nant-style configure, build and run process.

pokorra

Hello Alan,
this sounds interesting.

Please can you have a quick look at bazaar.launchpad.net/_tpokorra/o ... WebServer/ and tell me what the difference is between your server and my server?
Does your server survive changes of dlls (or even text files) in the delivery/bin folder? That is a downside of the solution that I am using.
Alternatively we could use the mono xsp server on windows as well, that should work fine.

Thanks,
Timotheus

christiankatict

Hi Alan,

I have at least anecdotal evidence that the webserver we currently use doesn't allow the server-side of OpenPetra to continue to run once the workstation on which it has been running has been put into 'hibernate mode' and then resumed from that mode - though that might be down more to the server-side implementation of OpenPetra than the actual webserver, we haven't investigated that closer. (However, it's not critical that it doesn't continue to work as it is unlikely that a workstation/server that hosts the server-side of OpenPetra would be put into 'hibernate mode', but still it isn't nice that the server-side of OpenPetra crashes badly, should that occur.)

Could you check if 'your' webserver continues happily after such an event?

Thanks,

ChristianK

alanjfpaterson

Hi Timo,

I just had a look at the TinyWebServer code and as you might expect there are both similarities and differences. Cassini has a 'Server' class which knows about the port, virtual/physical paths and a 'Host' class which gets instantiated by calling System.Web.CreateApplicationHost. Interestingly we both do the trick of copying the assembly to the web app 'bin' folder to avoid having to put anything in the GAC. The .NET 3.5 version of Cassini has an even cleverer way of handling this difficulty so it neither needs the GAC nor the bin folder but that is nothing to do with your question. I am still using the .NET 2 Cassini code so we both copy our file to 'bin'.

The big difference is that Cassini does not derive from or use the HttpListener class - it was written in the days before that class existed, so it has to go right back to the sockets level and work up from there. Ultimately, of course it hands off Request processing to HttpRuntime.ProcessRequest() after it has done all the parsing of the Request. Compared to TinyWebServer there is more smart stuff in the request processing so that it handles more than aspx/asmx and so I think will return 'better' response headers. It gives directory listings etc. It has more code in the overridden HttpRuntime methods. It also has the capability to automatically restart the server class in the event that the Host AppDomain unloads for any reason - by using a WaitCallback.

I think the key difference from the point of view of the 'surviving dll changes' issue is that the HttpListener needs a Context and it would appear that the context gets nuked when the files in the physical path change. Since Cassini does not use HttpListener there is nothing to do with 'Contexts'.

So here is what I did:.....

I started my Cassini app, which started my 2 websites. I browsed to my web service (which had an Add and a Subtract method) and invoked the Add method from the browser. All OK. Then I went to my web service code and added a Multiply method and built the new dll to the bin folder. Then browsed to the web service again and - Lo I now had three methods. I invoked the new method and got the answer. So yes it does look like there is no problem with the HttpRuntime coping with that. Which is kind of what I expected because the Visual Studio development server just runs in the background once you have started it for the first time.

Now regarding Christian's question, I am not quite sure what the test about Hibernation is. I guess you mean hibernate as in - run on battery so that the computer goes into power-saving hibernate, and then come back and open the web page again? I will try that next.

christiankatict

Hi Alan,

yes, that is what I meant with putting the server machine into 'hibernation'. You could also try the 'standby' mode, which is available only on some machines, depending on the motherboard, other hardware, BIOS and drivers (often that mode is available on laptop PC's).

The issue I saw with the webserver that we are using was that a 'Windows Crash Dialog' of some sort told me that the Ict.Tools.TinyWebServer process has crashed and needs to be closed immediately once I resumed the server machine from hibernation. So I didn't even get the chance to try and run a web request against it!

Thanks,

ChristianK

alanjfpaterson

Hello again,

Further to my previous post I have tried a couple more things ...

First I ran the server then viewed my asmx page then selected 'Sleep' from the Shutdown menu and went for a coffee. Came back, activated the laptop, logged in and browsed to the page again - success!

Next I chose 'Hibernate' from the shutdown menu - which totally powers off the PC but saves a working image to disk. Then I powered the PC back on and the previous desktop was restored. Again I could just browse to the web service.

So it all looks good.

Timo - I'm not sure what you can run on your system? Do you have a Windows machine? Would my .NET4 exe just run on Linux? I can email you something to try if you let me know what you need....

christiankatict

Hi Alan,

That sounds very good indeed!

I assume that in the two scenarios hibernate/standby both the client and the webserver were running on the same machine - is that right?

What could also easily be tried is if you perform hibernate/standby in the evening and resume the operation in the morning - is the server still working, is the client still working? This is about finding out whether there are timeouts in place, and if so, one could look into increasing those so that such a setup would support what a worker might do during a work week. Again, it's not critical and there is no urgent need for this to be tested, but it should be easy enough to test.

Thanks,

ChristianK

pokorra

Timo - I'm not sure what you can run on your system? Do you have a Windows machine? Would my .NET4 exe just run on Linux? I can email you something to try if you let me know what you need....

Hello Alan,
I do work with Windows Server 2008 R2, and with Ubuntu Linux.
On Linux, I am completely fine to use mono XSP for the webserver.

If your webserver can be licensed in a way that we can include it in our OpenPetra code base, or offer it as a download on Sourceforge, I am happy to use your webserver on Windows.

You could create a branch off my remoting branch, and see what happens if you start your webserver in
bazaar.launchpad.net/_tpokorra/o ... s.xml#L127

All the best,
Timotheus

alanjfpaterson

Hello Timo,

I made a branch from your branch as you suggested. It took me a little while to work out how the setup worked (like I couldn't find any asmx files but then I realised they got copied as part of the <target> specification). So it all makes sense now. All I did in the end was comment out some lines in the default.targets.xml file so I no longer do anything with copying/launching the TinyWebServer as part of the startServer-internal. Instead I start my alternative server in some folder where I happened to build it and point it to the delivery folder with port 9000. startServer-internal just copies the asmx files to delivery as usual and everything runs up.

My server is just left to run all the time. There are menu options to start and stop it, but there really isn't any need once it is running.

I'm not sure what I need to do to test things. (I get a failure running the common/server tests but I think that is because your branch isn't really that up to date. However I can go to all the usual Partner/Finance screens that I am familiar with.

I can also do a complete rebuild of the full solution and open the client again with no problem. (Its funny not having a command window that you have to close when you rebuild the solution!!)

So I'm not sure what to try now, but feel free to suggest something.

Regarding the server code:

We have all the source for all of it
It builds to an exe and a DLL and a Help folder with .htm and .css files in
The DLL source code contains the Microsoft Public Licence code, so I would suggest that would go quite well in our ThirdParty folder
The exe source code could go somewhere else but has a dependency on the dll.

The issue with using the server is this: It needs to run from somewhere outside the build location (same as the OPDA) because building the solution will fail if the exe/dll is in use. But if I/we were to update the source code and that was merged into a users branch they would need to know to copy by hand the new build to the location where they run the server from. It is exactly the same as working with the OPDA. We want the source code in trunk somewhere, but we want to run a copy of the executables outside trunk. But I don't think its a big problem.

On a side note I was relieved to find that for this work we use http and not https because the server does not support SSL (at the moment). It could do in the future - it would only need a new class adding. What is your thinking on how people will use the server on Windows machines? I guess for any kind of 'proper' installation users would have Windows Server so would then use IIS which is robust and secure over the internet. For a local office, so long as the server PC had Windows 7 Professional they would also have full IIS. So really we are only talking about needing something for developers or people with Windows7 Home (or whatever its called) that does not come with IIS - and if they are a single entity they will use the standalone build anyway. Is that right? I just want to understand what our perceived requirements are.

So let me know how you would like to take it from here. Maybe the next thing is for me to send you my executables so you can duplicate what I have just done.

I think the web service is a great way to go ....

Alan

pokorra

Hello Alan,
I would prefer that nant startServer will actually start the server. Otherwise we have to train the developers where to find/put the server exe.
I think there is no problem if the server is part of buildtools, we can build it when compiling OpenPetra.Tools.solution. We usually don't build the tools during normal development.
We can run the server from delivery/, instead of delivery/bin. copying it when nant startServer starts the server.
That is what I have done with my TinyWebServer.exe, and it works fine to build from within OpenPetra.Tools solution.

Starting and stopping of the server from developers assistant might still make sense, in my opinion.
Also for the NUnit tests, we need to be able to start and stop the server from nant, on a given port, so that we can run multiple instances on jenkins.

All NUnit tests should work, I have worked so that they worked. Only the new test for client keep alive fails at the moment.
What errors do you get?
nant test-without-display should work anyway, no server involved.
nant test-client: might not work since the server is not running?

SSL: you are right, that is not needed. For a proper installation, we use the "official" webservers available on Windows and Linux. Of course, in the end that is a decision of the product owners.

I guess the best is that you commit your work to your branch, and I merge your branch into mine.
No need for sending executables, since we have a distributed code versioning system! :)

Thanks for your review and contribution,
Timotheus

alanjfpaterson

Hi Timo,

I did what you suggested and made a branch to my PC from your branch. I sorted everything out which took a bit of time because my project was a regular Visual Studio one in 'My Documents', but I managed to copy it into the BuildTools section of the branch. I had to make a few tweaks to the project generation to get rid of warnings like 'using System.Web' since my project names did not quite match the ones that were excluded from the warning.

I also changed the openPetra.default.targets.xml that starts the server, so finally everything seems to work well.

But now of course I cannot upload anything from my PC to your branch as I have no write privilege.

So now I have made a branch from your branch on launchpad into a new one in my area of launchpad and am now making a branch to my PC from that.

On Thursday I will be able to copy my changes to that branch which will mean that you can access it (I hope) from the launchpad server.

Sorry for the delay, but I was able to do all the necessary things like generate solution, start the server, run the client, access the screens, generate the solution again, compile and still leave the server running all this time and when I 'started' the server (which does nothing if it is already running) everything still worked, so it is robust.

Regarding the nant test, the issue seemed to be to do with the [Setup] code for a number of the tests. I fixed it by adding the line

TAppSettingsManager SettingsManager = new TAppSettingsManager()
as the first line in the [Setup] method (often Init()). Sometimes the line was there already but it had to be moved so that it was always before any call to the new TLogging() constructor.

I have no idea why I had to do this when, apparently, you did not need to, except maybe you had a config file that I did not have. Anyway, as a result I was able to run most of the test-without-display tests, including many server tests, until I got to some GL ones that failed with foreign key constraint errors. But by then the server had done a lot of work with the database.

So it is too late to do any more now but I will commit this new branch to launchpad on Thursday and give you some more information about using the new web server.

Alan

alanjfpaterson

Good morning Timo,

I have checked all my code in to this branch:
lp:_{alan-paterson/openpetraorg/20130115_alan_net_remoting}

There are very few commits, which are all from today. Please read through the commit comments so you can see what they are about - and in particular the default.targets commit.

So I am hoping you can merge these files across into your branch.

I am really excited about your work to move to web services. As a development environment it makes life so much simpler. We can just work inside our chosen solution and we no longer have to think if we must stop the OP server before building because there are no code dependencies on any running files. It is wonderful. There is also no need to start the server before running the client (except for the first time). Thereafter you just start the client (through nant, OPDA or the development IDE) and it just runs. It's very much more productive!!

So I do hope that this enables you to move forward on the last bits of the project. For everything I have tried I could use the branch as it is. So I hope this helps you to do the progress bar stuff and whatever else is left.

I haven't checked in my changes I had to make to get the tests to run - because I don't understand why I had to make them! As I said in my last post, the problem was with the command line I think, so I am assuming that it is a config file issue. I could get all the tests (that I tried) to run if the [Setup] test fixture method instantiated a TAppSettingManager as the first action. This was nothing to do with whether or not the test involved the server or not. But I am sure that by the time you finish the branch it will be sorted. I could definitely run all the tests if I used the NUnit GUI.

Oh yes - one more thing. Stopping the server has a different meaning now. The nant stopPetraServer task has no effect on this web server. It is designed to run all the time. There is a GUI for stopping/starting but it doesn't really ever need to be used. So all the tests that include start and stop don't really do anything now. I think this is a fundamental consequence of using web services. If we used IIS we wouldn't start/stop the server service. It would just be left to run, which is another reason I really like this solution. It will be very liberating to get away from a server executable that contains code that is built as part of all our development work.

I love it!!

Alan

pokorra

Hello Alan,

I finally got round to look at your branch code.launchpad.net/_alan-paters ... t_remoting
I have now merged it into my branch: code.launchpad.net/_tpokorra/op ... t_remoting

I am working with the remoting on another branch for OM Germany, where we are using the webservice remoting already in production.
I got fed up with my TinyWebServer, and hope your server will be easier for me to use. It looks good so far.

One thing I am missing, but I can live without it: There is no console output of the webserver anymore.

Looking forward to see you all at ICCM next week!

All the best,
Timotheus

christiankatict

Hi Timotheus,

pokorra wrote
For a proper installation, we use the "official" webservers available on Windows and Linux.
...

I guess for Linux you mean Apache (with mod_mono for ASP.NET support)?
(The mono ASP.NET page (http://mono-project.com/ASP.NET) mentions other option too: "FastCGI hosting: use the FastCGI hosting if you have a web server that supports the FastCGI protocol (for example Nginx) for extending the server. You also may use a web server that only has support for CGI using cgi-fcgi.")

For Windows the 'obvious' choice would be Microsoft IIS, but TimI suggested that we should check whether we couldn't use Apache on Windows instead, as it is an open source solution.
I did a quick Internet research on that: The result from that is that it seems that individual people have somehow managed to get a configuration working, but the latest updates/posts on this topic are from 2005 or 2008, so nobody seems to have maintained that since.
I couldn't find anything that looked 'official', really. What I found, though, were posts on Forums that one shouldn't do that but use IIS, mainly because MS never supported ASP.NET on anything other than IIS (for production use, that is) and because IIS is native to Windows, whereas Apache isn't. Another argument was that IIS is free as is Apache (though IIS is of course not open source).
The mono people themselves link to a port of mod_mono to Windows, but this port (mod_mono-win32) has only got one maintainer (which poses a risk) and has not seen maintenance since 2008. Additionally it is considered 'work in progress' by the mono people (see 'mod_mono on Windows' heading here http://mono-project.com/Mod_mono).
Then there is also the 'mod_aspdotnet' project on SourceForge but it also hasn't been maintained since 2008 and seems not to support ASP.NET 2.0, so that also seems risky.

My preliminary conclusion is then: Apache on Windows for the hosting of ASP.NET Web Services seems not to be an option --- not for development and also not for production use.

What are your thoughts on this, Timotheus? (Anybody else is invited to post an answer to this topic, though!)

Thanks,

ChristianK

christiankatict

Following up my own post:
I dug a tad deeper and found out that the Nginx webserver is available on Windows.
Though people have appeared to make it work (see http://stackoverflow.com/questions/16067226/asp-net-with-nginx-lighttpd-mono-fastcgi-on-windows and http://www.codeproject.com/Articles/388040/FastCGI-NET-and-ASP-NET-self-hosting) it would probably not be a good idea to go down that path as the Nginx' website states on the Windows port page (http://nginx.org/en/docs/windows.html):

'Version of nginx for Windows uses the native Win32 API (not the Cygwin emulation layer). Only the select() connection processing method is currently used, so high performance and scalability should not be expected. Due to this and some other known issues version of nginx for Windows is considered to be a beta version.'

(emphasis added by me).

Kind regards,

ChristianK

pokorra

Hello Christian,

On Linux, I am using Lighttpd with fastcgi-mono-server4, see bazaar.launchpad.net/_tpokorra/o ... er.sh#L134

For Windows:
For Development: I am fine with the solution from Alan (based on Cassini). I merged it recently into my remoting branch, and have been working with it for the last weeks.
For Standalone, you don't need any server anymore. Ah, this needs to be tested as well, some things don't work yet. Can't remember which.
For Production in a Windows Server environment, I think you should go with the full Microsoft Stack (IIS etc).
You could look at Xampp and apachefriends.org/, but running Mono on Windows does not really make sense to me.
I still have not heard an official announcement from OM that OpenPetra needs to run on Windows Server? Is that because SLS development will not be continued?

hope this helps,
Timotheus