Master large and mid-size Visual Studio solutions

    Modularization with Visual Studio, Master mid-size and large solution files


    Motivation

    Solutions which contain significant amount of projects or a larger code base may be time consuming in loading and compiling the code within Visual Studio. Generally, mid-size, large or even huge solutions running on modest hardware may run the risk of Visual Studio crashes because of high memory footprint, if for example working with complex XAML files in designer mode.


    Ways of Structuring Solution Files

    Large amounts of code can be structured in different ways:


    • Single large solution file

      Loading and compilation time as well as memory footprint will increase massively with every additional project. If working in C++ mixed mode for example, the overall development performance of Visual Studio 2008/2010 slows down significantly. With this approach, the Visual Studio solution is the modularization unit, all projects are contained in one large solution as outlined in the following picture:


      Single large solution file

    • Multiple solution files (e.g. one sln file for each feature of the software project)

      Maintenance becomes an issue during the software project evolution. With this approach, the Visual Studio solution is the modularization unit, but the code is organized in many solutions, so that each module corresponds to one solution. In an ideal world, each project would be referenced only once by one modularization unit as shown in the following picture:


      Multiple solution file


      However some frequently used projects are often referenced by many solutions, so that the references among the solutions look like the following


      Multiple solution file: violation of DRY principle


      This violates the DRY principle (Don't Repeat Yourself), since the same settings have to be specified multiple times.


    • One big master solution with multiple smaller generated solutions

      It is a modification of the "multiple solutions" approach. There are some tools for automatic solution generations. Personally I prefer to have the control over sln and project files by the development team, not auto-magically generated. Also, it's often not possible to have a high amount of small and isolated solutions due to inter-project dependencies (see the picture above).



    Mitigations

    Since the approach of working with many solution files is not optimal especially due to maintenance reasons (e.g. the build and CI processes can be challenging with many solutions), I was looking for mitigations of the "one big solution" approach. Some mitigations would be:
    • Maintaining multiple copies of suo files (which contain the loaded/unloaded status of the projects)
      • The suo files are binary and can't be shared among team members. They also contain a lot of features like bookmarks, break points, list of open documents, etc. which are not necessary for project loadings.
      • Loading/unloading multiple projects in the Solution Explorer at the same time is quite time consuming.
    • Writing macros (probably specific for the current solution and development team) enabling compilation of the aspects.

    In addition the following techniques can be applied to improve the build performance:
    • Using tools for improving build performance like "IncrediBuild" or "Electric Cloud". Although this approach can decrease the compilation duration significantly, the solution load duration still remains the same. By the way, some of these tools can be really expensive.
    • Hardware / OS level: Multi core compilation, using SSD and a high amount of RAM, using of Memory RAM for 'bin/obj' directories, etc.

    Load Only the Required Projects

    Even if all or some of the mitigations mentioned above are applied, there is still one issue: In most cases, we work on a single feature or defect, so that only a few projects are relevant for the current aspect of work. But instead of concentrating on those few projects, we have to drag the burden of loading and compiling many (currently) irrelevant projects just because there are some technical dependencies or just because they are part of the same business project and are referenced by the sln file. So if loading of irrelevant projects for the current task could be avoided, this provided a huge benefit in loading and compiling times.


    If we just modularized our code within one single solution, it gave us a huge benefit in development efficiency. With this approach, it should be possible to work with different dedicated aspects of the solution, loading and compiling only the projects of the particular aspect as outlined in the following picture:


    Single solution modularization using load filters


    Since I couldn't find any convenient enough solution for the described approach, I wrote a Visual Studio extension called Funnel.


    With Funnel, you can just keep all the code in one single solution but structure the code by leveraging load filters. It's even possible to combine several load filters and so load several aspects at once. It's also possible to use instant (on-the-fly) filters with no preparation at all.


    Funnel lets you choose which projects to load each time a solution file is opened. You can just go on with instant filtering and select every single project to be loaded each time the solution is loaded. Alternatively the current projects selection can be stored as persistent load filter, so that it can be re-used later


    Example load filter overview


    In the following picture, two filters “ASP.NET MVC” and “Start UI” are combined at load time, so that only 3 projects contained by these filters are loaded and compiled. All other projects remain in the "unloaded" state as long as these filters are applied.


    Example load filter combination


    In the next picture, only the profiler relevant projects (which are part of the profile filter) are loaded.


    Example load filter definition


    Persistent filters are stored as plain XML files and can be shared by team members in a source control repository.

    Persistent filters are not mandatory, so developers can just use instant filtering specifying the projects to be loaded immediately at solution load time as shown in the following screenshot.


    Filtering Projects


    Funnel can be used free of charge for all kinds of projects: commercial and non-commercial. A getting-started document as well as release notes are available on this site.


    Hands-on Demonstration

    All the above description was quite theoretical, let us get to the world of code and tools.


    I prepared a 70-seconds screen cast with a demonstration of the Funnel extension. The code used can be downloaded from Source Forge.


    In the screen cast, we are going to load the existing SharpDevelop source code in two variations: using instant filters and using saved filters. All steps are commented.