Thursday, March 10, 2011

Using NuGet without committing packages

 

Update (8/16/2011): also check out this newer post that describes an easier way to set up this workflow.

The current NuGet workflow has always been to commit the Packages folder into source control. The reasoning is that it matches what developers typically do when they don’t have NuGet: they create a ‘Lib’ or ‘ExternalDependencies’ folder, dump binaries into there and commit them to source control to allow others to build.

While this has worked fine for some users, we have also heard from many that committing packages into source control is not what they want to do. When using a DVCS like Mercurial or Git, committing binaries can grow the repository size like crazy over time, making cloning more and more painful. In fact, this has been one of the top requests on NuGet our issue tracker.

The good news is that NuGet now offers a workflow which goes a long way to solving this problem. It isn’t 100% automated yet, but with some minimal pain you can set up your project to do this.

Running ‘nuget install’ on a packages.config file

Earlier, I blogged about how you can install NuGet packages from the command line by using NuGet.exe.

Get NuGet.exe from here if you don’t already have it, and run ‘nuget -update’ to self-update it.

This lets you install one package at a time, e.g.

D:\Mvc3Application>nuget install NHibernate -o Packages

As an aside, the -o flag is new and lets you specify where the package is installed.

But the big new thing is that you can now run it on a packages.config file. packages.config is a file that NuGet creates at the root of every project that has packages installed. So if you install the ‘EFCodeFirst.Sample’ package in your app, you’ll find a packages.config next to the .csproj file, and it will contain:

<?xml version="1.0" encoding="utf-8"?>
<packages>
<package id="EFCodeFirst" version="0.8" />
<package id="EFCodeFirst.Sample" version="0.8" />
</packages>

So this holds all the information about what packages are needed for your project. Suppose you don’t commit your Packages folder (which lives under the solution folder), and another developer clones your repository. They can now run:

D:\Mvc3Application>nuget i Mvc3Application\packages.config -o Packages
Successfully installed 'EFCodeFirst 0.8'.
Successfully installed 'EFCodeFirst.Sample 0.8'.

And the Packages will be restored! The other nice thing is that this command is smart enough not to do any expensive work if they are already installed, e.g.

D:\Mvc3Application>nuget i Mvc3Application\packages.config -o Packages
All packages listed in packages.config are already installed.

This completes very quickly with no network requests.

Integrating package restore into msbuild

Integrating this into your build is a simple matter of adding a Pre-build event.

First, I would suggest committing nuget.exe into your solution, e.g. under a Tools folder. Once you do that, you can then add the following Pre-build event:

$(SolutionDir)Tools\nuget install $(ProjectDir)packages.config -o $(SolutionDir)Packages

Note how packages.config lives under the project folder while the Packages folder lives under the solution folder.

And that’s it, you’re done! Now each time you build, NuGet will first make sure that you have all the packages that you need, and will download anything that’s missing from the live feed.

If your solution has multiple projects that use NuGet, add the same Pre-Build event to each project.

As an alternative, you can use an msbuild custom build target to achieve the same thing. Check out Danny Tuppeny's post for details on that. This worked better for him when using App Harbor.

We want your feedback

This is new, so it’s possible that it doesn’t quite work perfectly in all cases. Please let us know how it works for you: bugs, feedback, suggestion. Thanks!

65 comments:

  1. Hi David,

    This sounds like a good step forward. Or at least, it would be a good step forard if it worked! The version of NuGet.exe that you linked to (http://ci.nuget.org:8080/guestAuth/repository/download/bt4/.lastSuccessful/Console/NuGet.exe) doesn't appear to understand the '-o' option.


    Cheers,
    Damian.

    ReplyDelete
  2. Same here: Unknown option: '-o'.

    NuGet version 1.2.20311.155 ('NuGet.exe is up to date.').

    ReplyDelete
  3. Damian,

    Did you do the "run nuget -update" step?

    Mark

    ReplyDelete
  4. Wow! Didn't know of this hidden feature! I think next would be streamlining it depending on project configuration: release/debug, so that on my deployment environment I don't need to get the test related packages.

    ReplyDelete
  5. Thanks for posting this! I put it in the *.csproj as a pre-build event, and it works like a charm!

    ReplyDelete
  6. Sorry, I had messed things up a bit. If you run 'nuget update' now, you will get a version that works for this!

    ReplyDelete
  7. All that's missing an an MSBuild task that get the latest NuGet.exe so that it doesn't have to be included...

    ReplyDelete
  8. Sounds good - though including nuget.exe in the repo feels a but clunky given that's pretty much what you were trying to avoid. I guess if devs agreed to having the Tools folder at one level up, you could put ../ to avoid it going in the repo without much hassle.

    ReplyDelete
  9. Blogger fail. takes first part of an openID domain as the display name, so both someone else and myself show as "blog" and with the same icon. Nice one, Google ;)

    ReplyDelete
  10. @Danny: agreed, but I don't see how we can avoid having any binary at all come into the picture. You can certainly put nuget.exe in a different place, but in the end, it has to be available at build time to bootstrap the rest.

    ReplyDelete
  11. a link to bundler http://gembundler.com/ can elaborate on the vision for this

    ReplyDelete
  12. btw 'blog' guy, same here with 'id' :).

    ReplyDelete
  13. Thanks, David. This approach will be very useful for our team.

    ReplyDelete
  14. David,

    This is an interesting approach but I am having trouble applying it to my environment because I have two source feeds. I use a local feed for some internal libraries, and the default NuGet fedd for the remainder. I can see how to specify the source repository on the command line, but this only allows one source repository at a time.

    Do you have a suggestion on the right approach to restore packages that aren't committed when multiple source feeds are required?

    --philip.

    ReplyDelete
  15. @philip: Good question. I started a discussion on CodePlex to discuss further. Please join in :) http://nuget.codeplex.com/discussions/249628

    ReplyDelete
  16. @David

    Yeah, it's a tough one. Though I think having nuget outside the repo (and therefore shared across multiple repos) or another "well known location" is probably better than putting binaries back into the repo.

    I'm hoping this issue is somewhat solved in VS 2010 SP2 or VS 2012 by it being included with Visual Studio ;-)

    (Danny!)

    ReplyDelete
  17. I've just used this method to commit a new solution to my teams repository. VS2010 seems to hang during the 1st build process when nu-get downloads and installs the packages but this can only be expected.
    Other than that, brilliant.

    Cheers David.

    ReplyDelete
  18. @Martin: thanks for the feedback. When you say it seems to hang, do you mean the UI becomes frozen? I would think it could take a little while, but VS itself should still be responsive, and you should get some output in the Output window (not the nuget PS windows) as things happen.

    ReplyDelete
  19. Hy David
    Works good. Although we are using msbuild with buildParallel="true". This might cause unable to access file exception. Any ideas how to fix this? As workaround I set the property ContinueOnError to true:



    Daniel

    ReplyDelete
  20. @Daniel: let's follow up with more details about this issue on the nuget.org discussion board.

    ReplyDelete
  21. It would seem that dependency resolution is disabled when you attempt to install using a packages.config.

    packages.config:
    <?xml version="1.0" encoding="utf-8"?>
    <packages>
    <package id="AutoMapper" version="1.1.0.118" />
    <package id="NHibernate.Castle" version="3.1.0.4000" />
    <package id="FluentNHibernate" version="1.1.1.694" />
    <package id="structuremap" version="2.6.2" />
    </packages>

    expected output:
    Successfully installed 'AutoMapper 1.1.0.118'.
    Successfully installed 'log4net 1.2.10'.
    Successfully installed 'Castle.Core 2.5.2'.
    Successfully installed 'NHibernate.Castle 3.1.0.4000'.
    Successfully installed 'Iesi.Collections 3.1.0.4000'.
    Successfully installed 'NHibernate 3.1.0.4000'.
    Successfully installed 'FluentNHibernate 1.1.1.694'.
    Successfully installed 'structuremap 2.6.2'.

    actual output:
    Successfully installed 'AutoMapper 1.1.0.118'.
    Successfully installed 'NHibernate.Castle 3.1.0.4000'.
    Successfully installed 'FluentNHibernate 1.1.1.694'.
    Successfully installed 'structuremap 2.6.2'.

    ReplyDelete
  22. @will: yes, this is done deliberately because packages.config normally always contains the full dependency closure. How did you end up with a packages.config that doesn't include dependencies?

    ReplyDelete
  23. I created it by hand.

    Seems silly to me to disable the dependency resolution of a dependency management tool without explicitly asking it to. My expectation for using a package.config file is "Grab these packages, and any dependencies".

    ReplyDelete
  24. @Will: Maybe you are right. Could you file a bug on nuget.codeplex.com to track this suggestion? Thanks!

    ReplyDelete
  25. Sure. http://nuget.codeplex.com/workitem/909

    ReplyDelete
  26. This is awesome! I was just looking at NuGet to solve our library distribution problems, and was about to turn away from it when I saw this post -- I need a workflow such that if I add a new package to our project and check it in, that when my co-worker down the hall does a "get latest", everything will "just work". Looks like this gets us pretty close. I do need to be able to work with multiple sources (saw a comment from Phil that indicates that this might be a problem for now)

    ReplyDelete
  27. There were two shortcomings with this solution.

    We utilise packages from multiple sources, some from the public gallery, some from a local network share. The command line tool allows you to specify a source, but only one.

    Secondly, we often have multiple projects within a solution that utilise NuGet. This means multiple "packages.config" files per solution. It's annoying having to repeat this command for each project. These "packages.config" files are already referenced from "repositories.config".

    I have created two forks to address these issues.


    http://nuget.codeplex.com/SourceControl/network/Forks/dant199/NuGetMultipleSources

    This allows you to specify multiple sources in the form of a semi-colon delimited list.

    nuget install Project1\packages.config -o packages -s \\dev\packages;https://go.microsoft.com/fwlink/?LinkID=206669


    http://nuget.codeplex.com/SourceControl/network/Forks/dant199/NuGetRepositoriesConfigSupport

    This allows you to specify a path to "repositories.config", from which the paths to all "packages.config" files are extracted. The dependencies listed within each are then loaded.

    nuget install packages\repositories.config -o packages


    Is equivalent to:

    nuget install Project1\packages.config -o packages

    nuget install Project2\packages.config -o packages

    ...

    nuget install ProjectX\packages.config -o packages



    We are now using both of these features combined to negate the need to check in all packages. Now we simply must commit the "repositories.config", all the "packages.config" files and then perform an initial seed of dependencies by calling:

    nuget install packages\repositories.config -o packages -s \\dev-server\packages;https://go.microsoft.com/fwlink/?LinkID=206669



    I hope this can be of use.

    ReplyDelete
  28. Thanks Dan. I see you also posted to the forum (http://nuget.codeplex.com/discussions/249628), which is a better place to discuss back and forth with everyone. Let's continue the discussion over there :)

    ReplyDelete
  29. I am also seeing Visual Studio freeze using this approach on the initial build. I even get the little pop-up that says "Visual Studio is busy". It's faster on subsequent builds, but it's still *really* slow. It adds about 30 seconds to the overall build for a 6 project solution (5 of which have packages.config files).

    I really like the idea, but the performance will have to be improved quite a bit for this to be viable.

    ReplyDelete
  30. @Matt: that's strange. When you say you 'also' get the freeze, are you referring to others reporting similar things? I don't believe I saw that. In theory, it should be super fast after the initial package install, since it has basically no work to do. Is it also for you if you just run nuget.exe from the command line?

    ReplyDelete
  31. @David, I'm referring to the comment by Martin Evans above where he said VS would hang on the first build after adding the pre-build event. I'm seeing the same hang (VS is completely unresponsive). It's *really* bad the first time while it downloads packages, but it's still very noticeable on subsequent runs even after it's downloaded packages.

    I tried running NuGet.exe from the command-line, and it's also slow, taking about 5 seconds for each packages.config file. My command line looks like:

    Nuget.exe install path/to/packages.config -o packages

    The output simply says "All packages listed in packages.config are already installed."

    Is it querying the repository for some reason even though the packages are already installed?

    ReplyDelete
  32. @Matt: Hmmm, I don't seem to see this. It takes under a second for me from the command line, and it only makes one small request to http://packages.nuget.org/v1/FeedService.svc/ (which probably could be eliminated). Would you mind starting a discussion on http://nuget.codeplex.com so we can more easily discuss with others? One thing to try is to run fiddler while running nuget.exe and check for what requests are made.

    ReplyDelete
  33. @David I answered my own question, it is indeed querying the repository even though the packages are already installed.

    After digging some more, unchecking Internet Options -> LAN Settings -> Automatically Detect Settings (from within IE) speeds things up greatly. Now each call to NuGet.exe takes only about half a second to complete.

    ReplyDelete
  34. @David Yup, I'll post a discussion over there with my observation about the LAN Settings.

    I think it'd be great to remove the request to the NuGet feed if possible. If you're doing hardcore TDD, any additional time during the build is bothersome, and any time that can be shaved off between making a code change and getting feedback from your tests is much appreciated. :)

    ReplyDelete
  35. @Matt: agreed. I'm actually not sure what causes this request. We'll need to investigate.

    ReplyDelete
  36. David,

    Can this step be performed from the NuGet Package Console?

    ReplyDelete
  37. I've reproduced your demo with my code and I'm unclear what the proposed steps are for when the NuGet package has a new version.

    In your example I'm guessing that your your reference to EFCodeFirst 0.8 looks like '..\Packages\EFCodeFirst .0.8\lib\EFCodeFirst.dll'

    So, lets say EFCodeFirst gets updated to 1.0

    Your solution will continue to build and output the message "All packages listed in packages.config are already installed." because the packages.config is hard coded to look for 0.8

    So now you update your packages.config to 1.0?

    That change will allow NuGet to go out and get that version and update it for you but now all your references are 'old' because 1.0 got downloaded to 'Package\EFCodeFirst.0.8\lib' folder.

    It feels like I'm missing a core concept - please advise.

    ReplyDelete
  38. @Bill: you would perform the update using NuGet from VS using the update-package command. Generally, an update needs to be an explicit decision, so you wouldn't want that to happen just as a result of restoring packages.

    ReplyDelete
  39. Running update-package goes out and gets EFCodeFirst 1.0 and installs it to 'Package\EFCodeFirst.1.0\lib' then I watched the references go yellow and then come back.

    Unloading the project to look at the hint paths reveals that they are still pointed at the 0.8 folder instead of the 1.0 folder.

    Thoughts?

    ReplyDelete
  40. @Bill: Hmmm, I don't think that should happen. But note that EFCodeFirst 1.0 is actually an empty package that just has a dependency on the new 'EntityFramework' package, so it's sort of a special situation.

    ReplyDelete
  41. @Bill: I couldn't repro this on a new project. I ran:
    - Install-Package EFCodeFirst -Version 0.8
    - Update-Package EFCodeFirst
    And the reference got updated correctly. If you find a consistent repro, please file a bug on http://nuget.codeplex.com/ so we can investigate properly.

    ReplyDelete
  42. I'm just using EFCodeFirst as an example.

    I'm really using a Package that I've created with a couple dlls.

    ReplyDelete
  43. I see what I was doing wrong. I wasn't letting nuget do the first install. I was using the command line to download the package and then adding the references manually via "Add Reference" and the browse tab.

    I redid my test scenario and it is working as intended now.

    Thanks for your help!

    ReplyDelete
  44. After trying to execute install command i've got this message "Unable to find version '1.8.0' of package 'jQuery.Validation'." Nuget's repo latest version is 1.8.0.1. Maybe there is some option for "update to latest" or it is necessary to edit packages.config file and manually specify latest version from nuget?

    ReplyDelete
  45. @Sasha: that's strange. Looks like 1.8 is still available, but 1.8.0 is not (they're not treated as the same). Not sure how your packages.config ended up with a 1.8.0 reference.

    ReplyDelete
  46. I found a way to get this working on AppHarbor using an MSBuild target instead of a pre-build step:

    Setting up NuGet to Automatically Fetch Packages When Deploying to AppHarbor Without Committing Binaries http://j.mp/NuGetAppHarbor

    ReplyDelete
  47. Thanks Danny, I added a mention pointing to your post.

    ReplyDelete
  48. I had to wrap quotes around the "$(SolutionDir)Packages" and other macros because the spaces were causing a 9009 error in the build. After that, it was great.

    ReplyDelete
    Replies
    1. I'm late to the discussion, but for reference this was the full command I used:

      "$(SolutionDir).nuget\nuget" install "$(ProjectDir)packages.config" -o "$(SolutionDir)Packages"

      * note that `nuget.exe` showed up in the `.nuget` folder after I manually enabled package restore (via keyboard shortcut, because the menu item isn't showing up) -- see http://docs.nuget.org/docs/Workflows/Using-NuGet-without-committing-packages

      Delete
  49. This works very well for me on the build servers. However, when a developer explicitly installs or updates a package through VS, the files are automatically added to source control (TFS). And step that needs to be undone manually everytime. Is there away to prevent the "Manage NuGet packages.." feature from adding the binaries to source control?

    ReplyDelete
  50. @Alex: I agree this is a pain point. I just filed a bug to track it: http://nuget.codeplex.com/workitem/1193. Vote it up! :)

    ReplyDelete
  51. Is there any override, so that if I accidently remove the DLL's from my project, NuGet won't think it's installed and will go and get everything that

    ReplyDelete
  52. @pms: if you remove the whole folder that holds a package, the technique above will restore it. If that is not what you meant, please provide more details about your scenario.

    ReplyDelete
  53. Thanks very much. Worked perfectly. :)

    ReplyDelete
  54. This all works great for referenced binaries. How can one avoid having to check 'content' from a nuget package into source control? Refreshing packages using 'install package.config -o packages' does not appear to refresh deployed 'content'.

    ReplyDelete
  55. @aquajunky: correct, content files are not covered by this workflow and must be checked in. I think this came up once before and there was some discussion, but currently NuGet does not provide a way to restore those files (short of uninstalling/reinstalling the package).

    ReplyDelete
  56. Thanks for this post. The "new" way really was not working out, but this way worked like a dream...and really will help us out a lot!

    ReplyDelete
  57. Great work in general. Thanks a lot.
    If you have some free time please check if we are re-inventing the wheel here.

    http://texcellency.wordpress.com/2011/11/06/powershell-script-to-update-nuget-packages-easily/

    ReplyDelete
  58. This comment has been removed by the author.

    ReplyDelete
  59. To integrate nuget with msbuild we can use Custom.After.Microsoft.Common.targets file. when you build, Microsoft.Common.targets is looking for it.

    here is what I use

    <Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
    <PropertyGroup>
    <NugetExecutable>nuget.exe</NugetExecutable>
    <PackagesConfig>packages.config</PackagesConfig>
    </PropertyGroup>

    <ItemGroup Label="List of projects in solution with nuget cofiguration">
    <PackagesToResolve Include="$(SolutionDir)**\$(PackagesConfig)" />
    <NugetRepositories Include="d:\packages" />
    <NugetRepositories Include="\\server\packages" />
    <NugetRepositories Include="https://go.microsoft.com/fwlink/?LinkID=206669" />
    </ItemGroup>

    <PropertyGroup>
    <BeforeBuildDependsOn>
    GetPackagesFromNugetRepository;
    </BeforeBuildDependsOn>
    </PropertyGroup>

    <Target Name="BeforeBuild" DependsOnTargets="$(BeforeBuildDependsOn)" />

    <Target Name="GetPackagesFromNugetRepository" Inputs="@(PackagesToResolve)" Outputs="%(Identity).Dummy" >
    <Message Importance="high" Text="Getting dependencies for %(PackagesToResolve.FullPath)" />
    <Exec Command='$(NugetExecutable) install "%(PackagesToResolve.FullPath)" -o "$(SolutionDir)packages" -s "@(NugetRepositories)"' />
    </Target>
    </Project>

    ReplyDelete
  60. Thank you, Thank you, Thank you!

    ReplyDelete
  61. Hi David,
    We are using NuGet without checking in the libraries and also without the Visual Studio extensions. We specify the version number of the packages in the packages.config manually. In this way it works fine with Jenkins.

    In the packages.config file version attribute specify the exact version number of the package. Is it also possible to configure in a way that latest version of the package will be downloaded? The allowedVersions attribute seems to be working only with the Visual Studio plugin but not with the command line NuGet.exe

    It's surprising to see some features are available in the extension but not in the command line.

    ReplyDelete
  62. @iyigun: the package restore workflow is only about re-downloading the exact same packages that were previously installed in the project, and is not about upgrading packages. There were various related discussions about this on the NuGet forum, so I'd suggest taking follow up discussions there as more people will see it than on this post. Thanks!

    ReplyDelete