Tuesday, May 24, 2011

Thoughts on installing and updating NuGet packages outside of Visual Studio

One thing we hear occasionally from users is that it would be nice to be able to install NuGet packages from the command line (e.g. this issue, that one, and this thread). There are good reasons why this isn’t supported today, which I will discuss in this post.

What does installing a package even mean?

Before we go further, we need to be clear about exactly what we mean by ‘installing a package’. The reason this needs to be clarified is that there are really two definitions, which can cause confusion.

  1. Getting the bits onto the machine: ‘installing a NuGet package’ is sometimes used to mean the act of getting the contents of the package onto your hard drive.
  2. Making a project use a package: more commonly, it refers to not only downloading the package bits, but also ‘applying’ them to a project.

#1 is something that is fully supported today outside of Visual Studio using nuget.exe (see my previous post). NuGet also supports restoring packages outside of VS so you don’t have to commit them.

But for the sake of this post, I am strictly referring to #2, and that’s what I mean any time I use the term ‘installing a package’ below.

Now that we have this out of the way, let’s discuss why installing a package outside Visual Studio is non-trivial, as well as why it is in most cases not useful at all, although a case can be made for updating packages.

What makes installing a package outside Visual Studio non-trivial

Installing a NuGet package into a project (e.g. a csproj file) is a rich operation which does a lot more than just copying files. Here is a rough list of what NuGet can do when you install a package from VS (whether using the Package Manager Console or the NuGet Dialog):

  1. Add references to assemblies contained in the package
  2. Add references to framework assemblies in the GAC
  3. Add content files to an app (e.g. JavaScript files, code files, …)
  4. Add assembly binding redirects to deal with version incompatibilities
  5. Perform config transformations, typically to add settings related to the package
  6. Bring in tools that can then be run from Package Manager Console
  7. Run PowerShell scripts which can do arbitrary things by automating the DTE object model

Now let’s think about what it would take to perform those operations outside of VS.

The first 3 involve making modifications to the csproj file. When done within VS, it happens automatically by calling DTE methods, but outside of VS it would need to be done using custom parsing logic. While it’s clearly possible, it needs to be done carefully to avoid corrupting the csproj file. e.g. a GAC reference should not be added if it’s already there.

#4 to #6 should not be too different from doing it in VS.

#7 is basically impossible, since you cannot really ‘fake’ the DTE to let those script run.

So conceivably, with some good amount of work, we could support all scenarios except #7. It would be a little quirky as some packages would not fully work, but in many cases it would work.

But let’s now discuss how useful it would be.

Why installing packages outside of Visual Studio rarely makes sense

So let’s say we had this feature and it fully worked. What would it let you do that you can’t do today?

You could use the command line outside VS to install a Foo package in your project, but that in itself is rarely useful. e.g. suppose the package brings in a Foo.dll. You now have that assembly added as a reference in your project, but you don’t have any code using it. You now need to go in VS to write code against that new assembly, so it would have been simpler to just add it from VS in the first place!

And that’s generally the case for most packages: the logical step after installing them is to go to VS and actually use them, which mostly negates any benefits you may find by installing it outside of VS.

Admittedly, there are exceptions, like the Elmah package which is more or less ‘ready to run’ after you install it. But for the wide majority of packages, there is no direct ‘install and run’ workflow.

What about package updates?

If package installs don’t make sense outside of VS, what about package updates?

So you have this Foo packages that you installed from VS, but now you want to update it to a new versions from the command line. Does that make sense?

I think it does make a lot more sense than the install scenario, because by that point, you (presumably) already wrote some code that uses the package. So by updating it, you might get a newer Foo.dll with bug fixes, but all the code you wrote is still valid and ready to run against.

In particular, update could work well in the constrained scenario where the new version on the package just updates an assembly but doesn’t do much else.

On the other hand, it would be hard to support in the general case, since in theory, the updated package can be completely different from the older one. e.g. suppose the new package contains some new install-time PowerShell scripts. We’d be right back with the same tough issues discussed above.

Where do we go from here? You tell us!

My take is that we need to experiment with supporting package update outside on VS for at least a subset of scenarios. The big question is deciding how far this needs to go to reach sufficiently useful state.

The first step would be to start with the ‘only the assembly changed’ scenario, which is relatively simple, and probably is the 90+% case.

If you have some thoughts on this, we’d love to hear them! Would you use such feature, and would limiting it to updating assembly references be enough for your needs?

30 comments:

  1. I can't say I've ever needed to use NuGet outside of Visual Studio, however, you bring up some interesting points about Automation.

    I could see NuGet being integrated in to automated build processes. Getting the latest version of package X before performing a deploy.

    ReplyDelete
  2. @vinvpa: note that I don't think anyone should just 'update and deploy', as any updated version of a package can break your app. So it's more: 'update, run suites and make sure everything works, and then deploy' :)

    ReplyDelete
  3. We are definitely in need of the "update packages outside of Visual Studio" scenario. We are currently considering abandoning nuget since it does not support this scenario. I'll see if I can describe our situation, and maybe you can provide some suggestions on what we should be doing differently to fit the "nuget way".

    We have a long dependency chain of libraries that needs to be pulled into our application. Consider:

    APP requires LibA. LibA requires LibB.

    I need to make a change in LibB. I get the code, make the change in Visual Studio, run tests, commit, create updated nuget package, and push it to our local repo.
    I need to rebuild LibA with the new LibB, but do not necessarily need to make any code changes in LibA, assuming compatibility did not break.
    Ideally, I should be able to go to my LibB working folder, get the latest code, use nuget from the command line to update its copy of LibB, run build and test suite, and if all is good, push a new LibA package to our local repo.
    Finally, I go to my APP folder and nuget update to get the latest LibA and LibB. This step could be done from within VS, since I likely have it open anyway.

    The key is that for the interim libraries (LibA in this example, but the chain could be longer, so there could be more), I should not have to open Visual Studio. I'm just rebuilding to update the dependencies.

    ReplyDelete
  4. I knew I would confuse one of those steps...

    The sentence that starts with "Ideally" should read:
    "Ideally, I should be able to go to my LibA working folder..."

    ReplyDelete
  5. @joshf: yep, scenario makes sense. So in your case, the packages you deal with only contain assemblies and nothing else, right? That's the 'constrained' scenario for which I think we *can* handle update.

    ReplyDelete
  6. Yes, we only care about the constrained scenario. Seems like a metadata flag for "requires VS" would be helpful.

    ReplyDelete
  7. +1 for a flag in nuget packages that says "requires VS"

    ReplyDelete
  8. also it would be good to have packages that "cannot be run in VS".

    This may sound like a strange request but there are some things cannot use nuget because of the file locking issues of visual studio. for example msbuild tasks.

    In these cases the package should get a file path to the project

    ReplyDelete
  9. I think you make an important distinction between the two types of "installs". I wonder if it would help to rename the "install" verb of nuget.exe to make it more clear that it's different from "install-package". "Nuget.exe get"? Sounds kind of redundant, but I think it's more accurate. There is might be a better term, but I think it would help to not overload the install term.

    ReplyDelete
  10. @Simon: you might want to start a thread on nuget.codeplex.com to explain your scenario in more details.

    @Andy: yes, maybe it was a mistake to call it 'Install' in both cases when they're clearly difference. Though the downside of changing it now is that it might break a lot of people.

    ReplyDelete
  11. @David Josh and I are on the team, as is Dru Sellers. Josh and Dru were working on this project and ran into the challenges Josh mentioned above.

    Dru also posted some of the other issues recently on twitter:

    http://www.tweetdeck.com/twitter/drusellers/~ETFDh

    problem 1: updating a package from the command line from a local repository that has a dependency on a remote repository currently fails

    http://www.tweetdeck.com/twitter/drusellers/~jPYWk

    problem 2: if you exclude version numbers, the command line becomes too stupid to know how to update. its like it can't look inside the nupkg to see the version number

    Josh F. thought that problem #2 has already been addressed by a recent commit to Nuget.

    ReplyDelete
  12. @Chad: the issue with a local repository package that depends on a remote repo is fixed in 1.4 (which is not yet released but available to test).

    Not quite sure I understand what #2 means. BTW, when referring to 'command line', please specify whether you mean the PowerShell console in VS, or the cmd command line using nuget.exe.

    ReplyDelete
    Replies
    1. I still have problem 2 that Chad describes with NuGet 2.1
      when using cmd to install with -excludeversion the update command stops working

      Delete
  13. @chad and @david #2 is the one Dru and I went through and I logged a bug for. There was a recent commit that says this has been fixed, but I don't know that its been verified.

    ReplyDelete
  14. Sorry for the brevity, but how can you do a post like this and not mention OpenWrap and how it has solved this need? Unless you don't know about it, which I doubt.

    ReplyDelete
  15. @igitur: the main difference is that OW only supports a subset of the NuGet functionality (from my list above). It's things like config transforms and scripts that make the problem harder. NuGet will likely later add update support for those constrained scenario as outlined above. As for *installs* from the command line, it simply is not an interesting feature for our users (for reasons state above).

    ReplyDelete
  16. You're asking the wrong question. It's not "how to install a NuGet package outside of Visual Studio" (implying that you're only exiting Visual Studio out of convenience or laziness) but "how to install a NuGet package *without* Visual Studio". Visual Studio costs money (I know about Express, but that's beside the point) and is only available on Windows.

    There are other IDEs and operating systems out there. The .NET community is greater than Microsoft, and of all people and companies, you'd think Microsoft itself would appreciate that.

    ReplyDelete
  17. @asbjornu: The update workflow that we're thinking about will indeed work *without* VS.

    BTW, note that 3rd party IDE SharpDevelop also supports NuGet.

    ReplyDelete
  18. Hi David,

    I use a *real* private nuget server at several customers. I have tied a *nuget update* into the build process/server.

    Package version != Assembly version

    This allows to download new builds of an assembly by its package without changing the *.csproj files - in the build process I only use MsBuild and NOT Visual Studio

    ReplyDelete
  19. Hi,

    Regarding step #3 "Add content files to an app", would it be possible to only copy the files when restoring a package with "nuget install" ?

    I've a package with big files in Content/App_Data (dictionary files from telerik), and i'd like not to check them in my source code control. However as "nuget install" does not copy files from the content folder, developers have to copy these files manually to the project after they've restored the package.

    ReplyDelete
  20. @Pascal: That's an interesting idea that had not come up. Could you open a bug on nuget.codeplex.com to track it? So to be clear, the restore would not have to touch the project file, but simply copy over the content files if they're missing.

    ReplyDelete
  21. @David: Thanks for you answer. I've opened an issue on codeplex : http://nuget.codeplex.com/workitem/1239

    ReplyDelete
  22. Does anything change in this space since May 2011? Is NuGet 2.0 still not able to update packages outside VS?

    ReplyDelete
  23. @sergey I don't think there are any significant changes. You might want to start a thread on http://nuget.codeplex.com/ to discuss latest.

    ReplyDelete
  24. I would like to ability to add project references outside of VS for the following reason.

    My company has a lot of shared libraries that depend on each other in some cases. I am using TFS Nugetter to build and publish nuget packages from TFS. I want to ensure that the developers can't queue a build (package) unless the project can build and run on all the newer versions. This ensures that all the newer versions of the libraries work with all the newer versions of its dependencies. If the build fails, then you need to update your nuget reference in VS and fix the compiler errors/unit tests.

    I have been looking at the NuGet source and I think I found an easy way to reuse NuGet source to modify proj files outside of VS (kinda).

    System.Type t = System.Type.GetTypeFromProgID("VisualStudio.DTE.10.0", true);
    var dte = (DTE)System.Activator.CreateInstance(t, true);
    dte.Solution.Open(@"C:\Users\paul.knopf\Documents\Visual Studio 2010\Projects\SLNMemory\SLNMemory.sln");

    Basically, open an in-memory version of visual studio, run the nuget commands, then save.

    In a build step, after GetWorkspace, I would like to run this in-memory vs to update all nuget references to the latest version.

    What do you think? It would definitely be slow, but we would be on the same code base and have all the functionality we need.

    ReplyDelete
  25. @Paul: would you mind starting a discussion on http://nuget.codeplex.com/ instead? This way it can be seen more widely by others and make it easier to discuss. Thanks!

    ReplyDelete
  26. What is the best way to promote this feature? I'm currently using Jenkins to automate non-TFS builds in my organization. When I added an integration build (for test purposes) for an IT tool managed by TFS I ran smack into the update scenario you described above.

    I would love to be able to run a simple update on our nuget package. In my case I work in the group that creates the nuget package which our IT department uses in production. So we want to make sure that whatever we change doesn't break their app.

    I'm deploying native dlls alongside our assembly, so I would need a solution that involves generalizing the post-build step for copying the native dlls to the bin directory on build.

    ReplyDelete
  27. Note regarding my post above: I'm already using PowerShell script hooks to deploy a PostBuild event when the package is installed in VS. This was the "official" method recommended for native deployment.

    ReplyDelete
  28. @Will: please note that this is a very old post and that a lot has happened since. Can you please post to http://nuget.codeplex.com/ instead to make sure it hits all the right people? Thanks!

    ReplyDelete
  29. Okay, no problem. I am building a code generator that will support NuGet. It looks like I will be doing all this logic myself after the nuget.exe downloads the files. Thanks for your help.

    ReplyDelete