Update (8/16/2011): also check out this newer post that describes an easier way to set up this workflow.
The current NuGet workflow has always been to commit the Packages folder into source control. The reasoning is that it matches what developers typically do when they don’t have NuGet: they create a ‘Lib’ or ‘ExternalDependencies’ folder, dump binaries into there and commit them to source control to allow others to build.
While this has worked fine for some users, we have also heard from many that committing packages into source control is not what they want to do. When using a DVCS like Mercurial or Git, committing binaries can grow the repository size like crazy over time, making cloning more and more painful. In fact, this has been one of the top requests on NuGet our issue tracker.
The good news is that NuGet now offers a workflow which goes a long way to solving this problem. It isn’t 100% automated yet, but with some minimal pain you can set up your project to do this.
Running ‘nuget install’ on a packages.config file
Earlier, I blogged about how you can install NuGet packages from the command line by using NuGet.exe.
Get NuGet.exe from here if you don’t already have it, and run ‘nuget -update’ to self-update it.
This lets you install one package at a time, e.g.
D:\Mvc3Application>nuget install NHibernate -o Packages
As an aside, the -o flag is new and lets you specify where the package is installed.
But the big new thing is that you can now run it on a packages.config file. packages.config is a file that NuGet creates at the root of every project that has packages installed. So if you install the ‘EFCodeFirst.Sample’ package in your app, you’ll find a packages.config next to the .csproj file, and it will contain:
<?xml version="1.0" encoding="utf-8"?> <packages> <package id="EFCodeFirst" version="0.8" /> <package id="EFCodeFirst.Sample" version="0.8" /> </packages>
So this holds all the information about what packages are needed for your project. Suppose you don’t commit your Packages folder (which lives under the solution folder), and another developer clones your repository. They can now run:
D:\Mvc3Application>nuget i Mvc3Application\packages.config -o Packages Successfully installed 'EFCodeFirst 0.8'. Successfully installed 'EFCodeFirst.Sample 0.8'.
And the Packages will be restored! The other nice thing is that this command is smart enough not to do any expensive work if they are already installed, e.g.
D:\Mvc3Application>nuget i Mvc3Application\packages.config -o Packages All packages listed in packages.config are already installed.
This completes very quickly with no network requests.
Integrating package restore into msbuild
Integrating this into your build is a simple matter of adding a Pre-build event.
First, I would suggest committing nuget.exe into your solution, e.g. under a Tools folder. Once you do that, you can then add the following Pre-build event:
$(SolutionDir)Tools\nuget install $(ProjectDir)packages.config -o $(SolutionDir)Packages
Note how packages.config lives under the project folder while the Packages folder lives under the solution folder.
And that’s it, you’re done! Now each time you build, NuGet will first make sure that you have all the packages that you need, and will download anything that’s missing from the live feed.
If your solution has multiple projects that use NuGet, add the same Pre-Build event to each project.
As an alternative, you can use an msbuild custom build target to achieve the same thing. Check out Danny Tuppeny's post for details on that. This worked better for him when using App Harbor.
We want your feedback
This is new, so it’s possible that it doesn’t quite work perfectly in all cases. Please let us know how it works for you: bugs, feedback, suggestion. Thanks!
Hi David,
ReplyDeleteThis sounds like a good step forward. Or at least, it would be a good step forard if it worked! The version of NuGet.exe that you linked to (http://ci.nuget.org:8080/guestAuth/repository/download/bt4/.lastSuccessful/Console/NuGet.exe) doesn't appear to understand the '-o' option.
Cheers,
Damian.
Same here: Unknown option: '-o'.
ReplyDeleteNuGet version 1.2.20311.155 ('NuGet.exe is up to date.').
Damian,
ReplyDeleteDid you do the "run nuget -update" step?
Mark
Wow! Didn't know of this hidden feature! I think next would be streamlining it depending on project configuration: release/debug, so that on my deployment environment I don't need to get the test related packages.
ReplyDeleteThanks for posting this! I put it in the *.csproj as a pre-build event, and it works like a charm!
ReplyDeleteSorry, I had messed things up a bit. If you run 'nuget update' now, you will get a version that works for this!
ReplyDeleteAll that's missing an an MSBuild task that get the latest NuGet.exe so that it doesn't have to be included...
ReplyDeleteSounds good - though including nuget.exe in the repo feels a but clunky given that's pretty much what you were trying to avoid. I guess if devs agreed to having the Tools folder at one level up, you could put ../ to avoid it going in the repo without much hassle.
ReplyDeleteBlogger fail. takes first part of an openID domain as the display name, so both someone else and myself show as "blog" and with the same icon. Nice one, Google ;)
ReplyDelete@Danny: agreed, but I don't see how we can avoid having any binary at all come into the picture. You can certainly put nuget.exe in a different place, but in the end, it has to be available at build time to bootstrap the rest.
ReplyDeletea link to bundler http://gembundler.com/ can elaborate on the vision for this
ReplyDeletebtw 'blog' guy, same here with 'id' :).
ReplyDeleteThanks, David. This approach will be very useful for our team.
ReplyDeleteDavid,
ReplyDeleteThis is an interesting approach but I am having trouble applying it to my environment because I have two source feeds. I use a local feed for some internal libraries, and the default NuGet fedd for the remainder. I can see how to specify the source repository on the command line, but this only allows one source repository at a time.
Do you have a suggestion on the right approach to restore packages that aren't committed when multiple source feeds are required?
--philip.
@philip: Good question. I started a discussion on CodePlex to discuss further. Please join in :) http://nuget.codeplex.com/discussions/249628
ReplyDelete@David
ReplyDeleteYeah, it's a tough one. Though I think having nuget outside the repo (and therefore shared across multiple repos) or another "well known location" is probably better than putting binaries back into the repo.
I'm hoping this issue is somewhat solved in VS 2010 SP2 or VS 2012 by it being included with Visual Studio ;-)
(Danny!)
I've just used this method to commit a new solution to my teams repository. VS2010 seems to hang during the 1st build process when nu-get downloads and installs the packages but this can only be expected.
ReplyDeleteOther than that, brilliant.
Cheers David.
@Martin: thanks for the feedback. When you say it seems to hang, do you mean the UI becomes frozen? I would think it could take a little while, but VS itself should still be responsive, and you should get some output in the Output window (not the nuget PS windows) as things happen.
ReplyDeleteHy David
ReplyDeleteWorks good. Although we are using msbuild with buildParallel="true". This might cause unable to access file exception. Any ideas how to fix this? As workaround I set the property ContinueOnError to true:
Daniel
@Daniel: let's follow up with more details about this issue on the nuget.org discussion board.
ReplyDeleteIt would seem that dependency resolution is disabled when you attempt to install using a packages.config.
ReplyDeletepackages.config:
<?xml version="1.0" encoding="utf-8"?>
<packages>
<package id="AutoMapper" version="1.1.0.118" />
<package id="NHibernate.Castle" version="3.1.0.4000" />
<package id="FluentNHibernate" version="1.1.1.694" />
<package id="structuremap" version="2.6.2" />
</packages>
expected output:
Successfully installed 'AutoMapper 1.1.0.118'.
Successfully installed 'log4net 1.2.10'.
Successfully installed 'Castle.Core 2.5.2'.
Successfully installed 'NHibernate.Castle 3.1.0.4000'.
Successfully installed 'Iesi.Collections 3.1.0.4000'.
Successfully installed 'NHibernate 3.1.0.4000'.
Successfully installed 'FluentNHibernate 1.1.1.694'.
Successfully installed 'structuremap 2.6.2'.
actual output:
Successfully installed 'AutoMapper 1.1.0.118'.
Successfully installed 'NHibernate.Castle 3.1.0.4000'.
Successfully installed 'FluentNHibernate 1.1.1.694'.
Successfully installed 'structuremap 2.6.2'.
@will: yes, this is done deliberately because packages.config normally always contains the full dependency closure. How did you end up with a packages.config that doesn't include dependencies?
ReplyDeleteI created it by hand.
ReplyDeleteSeems silly to me to disable the dependency resolution of a dependency management tool without explicitly asking it to. My expectation for using a package.config file is "Grab these packages, and any dependencies".
@Will: Maybe you are right. Could you file a bug on nuget.codeplex.com to track this suggestion? Thanks!
ReplyDeleteSure. http://nuget.codeplex.com/workitem/909
ReplyDeleteThis is awesome! I was just looking at NuGet to solve our library distribution problems, and was about to turn away from it when I saw this post -- I need a workflow such that if I add a new package to our project and check it in, that when my co-worker down the hall does a "get latest", everything will "just work". Looks like this gets us pretty close. I do need to be able to work with multiple sources (saw a comment from Phil that indicates that this might be a problem for now)
ReplyDeleteThere were two shortcomings with this solution.
ReplyDeleteWe utilise packages from multiple sources, some from the public gallery, some from a local network share. The command line tool allows you to specify a source, but only one.
Secondly, we often have multiple projects within a solution that utilise NuGet. This means multiple "packages.config" files per solution. It's annoying having to repeat this command for each project. These "packages.config" files are already referenced from "repositories.config".
I have created two forks to address these issues.
http://nuget.codeplex.com/SourceControl/network/Forks/dant199/NuGetMultipleSources
This allows you to specify multiple sources in the form of a semi-colon delimited list.
nuget install Project1\packages.config -o packages -s \\dev\packages;https://go.microsoft.com/fwlink/?LinkID=206669
http://nuget.codeplex.com/SourceControl/network/Forks/dant199/NuGetRepositoriesConfigSupport
This allows you to specify a path to "repositories.config", from which the paths to all "packages.config" files are extracted. The dependencies listed within each are then loaded.
nuget install packages\repositories.config -o packages
Is equivalent to:
nuget install Project1\packages.config -o packages
nuget install Project2\packages.config -o packages
...
nuget install ProjectX\packages.config -o packages
We are now using both of these features combined to negate the need to check in all packages. Now we simply must commit the "repositories.config", all the "packages.config" files and then perform an initial seed of dependencies by calling:
nuget install packages\repositories.config -o packages -s \\dev-server\packages;https://go.microsoft.com/fwlink/?LinkID=206669
I hope this can be of use.
Thanks Dan. I see you also posted to the forum (http://nuget.codeplex.com/discussions/249628), which is a better place to discuss back and forth with everyone. Let's continue the discussion over there :)
ReplyDeleteI am also seeing Visual Studio freeze using this approach on the initial build. I even get the little pop-up that says "Visual Studio is busy". It's faster on subsequent builds, but it's still *really* slow. It adds about 30 seconds to the overall build for a 6 project solution (5 of which have packages.config files).
ReplyDeleteI really like the idea, but the performance will have to be improved quite a bit for this to be viable.
@Matt: that's strange. When you say you 'also' get the freeze, are you referring to others reporting similar things? I don't believe I saw that. In theory, it should be super fast after the initial package install, since it has basically no work to do. Is it also for you if you just run nuget.exe from the command line?
ReplyDelete@David, I'm referring to the comment by Martin Evans above where he said VS would hang on the first build after adding the pre-build event. I'm seeing the same hang (VS is completely unresponsive). It's *really* bad the first time while it downloads packages, but it's still very noticeable on subsequent runs even after it's downloaded packages.
ReplyDeleteI tried running NuGet.exe from the command-line, and it's also slow, taking about 5 seconds for each packages.config file. My command line looks like:
Nuget.exe install path/to/packages.config -o packages
The output simply says "All packages listed in packages.config are already installed."
Is it querying the repository for some reason even though the packages are already installed?
@Matt: Hmmm, I don't seem to see this. It takes under a second for me from the command line, and it only makes one small request to http://packages.nuget.org/v1/FeedService.svc/ (which probably could be eliminated). Would you mind starting a discussion on http://nuget.codeplex.com so we can more easily discuss with others? One thing to try is to run fiddler while running nuget.exe and check for what requests are made.
ReplyDelete@David I answered my own question, it is indeed querying the repository even though the packages are already installed.
ReplyDeleteAfter digging some more, unchecking Internet Options -> LAN Settings -> Automatically Detect Settings (from within IE) speeds things up greatly. Now each call to NuGet.exe takes only about half a second to complete.
@David Yup, I'll post a discussion over there with my observation about the LAN Settings.
ReplyDeleteI think it'd be great to remove the request to the NuGet feed if possible. If you're doing hardcore TDD, any additional time during the build is bothersome, and any time that can be shaved off between making a code change and getting feedback from your tests is much appreciated. :)
@Matt: agreed. I'm actually not sure what causes this request. We'll need to investigate.
ReplyDeleteDavid,
ReplyDeleteCan this step be performed from the NuGet Package Console?
I've reproduced your demo with my code and I'm unclear what the proposed steps are for when the NuGet package has a new version.
ReplyDeleteIn your example I'm guessing that your your reference to EFCodeFirst 0.8 looks like '..\Packages\EFCodeFirst .0.8\lib\EFCodeFirst.dll'
So, lets say EFCodeFirst gets updated to 1.0
Your solution will continue to build and output the message "All packages listed in packages.config are already installed." because the packages.config is hard coded to look for 0.8
So now you update your packages.config to 1.0?
That change will allow NuGet to go out and get that version and update it for you but now all your references are 'old' because 1.0 got downloaded to 'Package\EFCodeFirst.0.8\lib' folder.
It feels like I'm missing a core concept - please advise.
@Bill: you would perform the update using NuGet from VS using the update-package command. Generally, an update needs to be an explicit decision, so you wouldn't want that to happen just as a result of restoring packages.
ReplyDeleteRunning update-package goes out and gets EFCodeFirst 1.0 and installs it to 'Package\EFCodeFirst.1.0\lib' then I watched the references go yellow and then come back.
ReplyDeleteUnloading the project to look at the hint paths reveals that they are still pointed at the 0.8 folder instead of the 1.0 folder.
Thoughts?
@Bill: Hmmm, I don't think that should happen. But note that EFCodeFirst 1.0 is actually an empty package that just has a dependency on the new 'EntityFramework' package, so it's sort of a special situation.
ReplyDelete@Bill: I couldn't repro this on a new project. I ran:
ReplyDelete- Install-Package EFCodeFirst -Version 0.8
- Update-Package EFCodeFirst
And the reference got updated correctly. If you find a consistent repro, please file a bug on http://nuget.codeplex.com/ so we can investigate properly.
I'm just using EFCodeFirst as an example.
ReplyDeleteI'm really using a Package that I've created with a couple dlls.
I see what I was doing wrong. I wasn't letting nuget do the first install. I was using the command line to download the package and then adding the references manually via "Add Reference" and the browse tab.
ReplyDeleteI redid my test scenario and it is working as intended now.
Thanks for your help!
After trying to execute install command i've got this message "Unable to find version '1.8.0' of package 'jQuery.Validation'." Nuget's repo latest version is 1.8.0.1. Maybe there is some option for "update to latest" or it is necessary to edit packages.config file and manually specify latest version from nuget?
ReplyDelete@Sasha: that's strange. Looks like 1.8 is still available, but 1.8.0 is not (they're not treated as the same). Not sure how your packages.config ended up with a 1.8.0 reference.
ReplyDeleteI found a way to get this working on AppHarbor using an MSBuild target instead of a pre-build step:
ReplyDeleteSetting up NuGet to Automatically Fetch Packages When Deploying to AppHarbor Without Committing Binaries http://j.mp/NuGetAppHarbor
Thanks Danny, I added a mention pointing to your post.
ReplyDeleteI had to wrap quotes around the "$(SolutionDir)Packages" and other macros because the spaces were causing a 9009 error in the build. After that, it was great.
ReplyDeleteI'm late to the discussion, but for reference this was the full command I used:
Delete"$(SolutionDir).nuget\nuget" install "$(ProjectDir)packages.config" -o "$(SolutionDir)Packages"
* note that `nuget.exe` showed up in the `.nuget` folder after I manually enabled package restore (via keyboard shortcut, because the menu item isn't showing up) -- see http://docs.nuget.org/docs/Workflows/Using-NuGet-without-committing-packages
@davescruggs: good catch!
ReplyDeleteThis works very well for me on the build servers. However, when a developer explicitly installs or updates a package through VS, the files are automatically added to source control (TFS). And step that needs to be undone manually everytime. Is there away to prevent the "Manage NuGet packages.." feature from adding the binaries to source control?
ReplyDelete@Alex: I agree this is a pain point. I just filed a bug to track it: http://nuget.codeplex.com/workitem/1193. Vote it up! :)
ReplyDeleteCheers, will do!
ReplyDeleteIs there any override, so that if I accidently remove the DLL's from my project, NuGet won't think it's installed and will go and get everything that
ReplyDelete@pms: if you remove the whole folder that holds a package, the technique above will restore it. If that is not what you meant, please provide more details about your scenario.
ReplyDeleteThanks very much. Worked perfectly. :)
ReplyDeleteThis all works great for referenced binaries. How can one avoid having to check 'content' from a nuget package into source control? Refreshing packages using 'install package.config -o packages' does not appear to refresh deployed 'content'.
ReplyDelete@aquajunky: correct, content files are not covered by this workflow and must be checked in. I think this came up once before and there was some discussion, but currently NuGet does not provide a way to restore those files (short of uninstalling/reinstalling the package).
ReplyDeleteThanks for this post. The "new" way really was not working out, but this way worked like a dream...and really will help us out a lot!
ReplyDeleteGreat work in general. Thanks a lot.
ReplyDeleteIf you have some free time please check if we are re-inventing the wheel here.
http://texcellency.wordpress.com/2011/11/06/powershell-script-to-update-nuget-packages-easily/
This comment has been removed by the author.
ReplyDeleteTo integrate nuget with msbuild we can use Custom.After.Microsoft.Common.targets file. when you build, Microsoft.Common.targets is looking for it.
ReplyDeletehere is what I use
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<PropertyGroup>
<NugetExecutable>nuget.exe</NugetExecutable>
<PackagesConfig>packages.config</PackagesConfig>
</PropertyGroup>
<ItemGroup Label="List of projects in solution with nuget cofiguration">
<PackagesToResolve Include="$(SolutionDir)**\$(PackagesConfig)" />
<NugetRepositories Include="d:\packages" />
<NugetRepositories Include="\\server\packages" />
<NugetRepositories Include="https://go.microsoft.com/fwlink/?LinkID=206669" />
</ItemGroup>
<PropertyGroup>
<BeforeBuildDependsOn>
GetPackagesFromNugetRepository;
</BeforeBuildDependsOn>
</PropertyGroup>
<Target Name="BeforeBuild" DependsOnTargets="$(BeforeBuildDependsOn)" />
<Target Name="GetPackagesFromNugetRepository" Inputs="@(PackagesToResolve)" Outputs="%(Identity).Dummy" >
<Message Importance="high" Text="Getting dependencies for %(PackagesToResolve.FullPath)" />
<Exec Command='$(NugetExecutable) install "%(PackagesToResolve.FullPath)" -o "$(SolutionDir)packages" -s "@(NugetRepositories)"' />
</Target>
</Project>
Thank you, Thank you, Thank you!
ReplyDeleteHi David,
ReplyDeleteWe are using NuGet without checking in the libraries and also without the Visual Studio extensions. We specify the version number of the packages in the packages.config manually. In this way it works fine with Jenkins.
In the packages.config file version attribute specify the exact version number of the package. Is it also possible to configure in a way that latest version of the package will be downloaded? The allowedVersions attribute seems to be working only with the Visual Studio plugin but not with the command line NuGet.exe
It's surprising to see some features are available in the extension but not in the command line.
@iyigun: the package restore workflow is only about re-downloading the exact same packages that were previously installed in the project, and is not about upgrading packages. There were various related discussions about this on the NuGet forum, so I'd suggest taking follow up discussions there as more people will see it than on this post. Thanks!
ReplyDelete